Description

A senior data mesh platform engineer is a technical role responsible for designing and building the underlying infrastructure and tooling that enables different business domains within an organization to independently manage and access their data within a data mesh architecture. This person will work in the central platform team in the SA Data division of Investec which maintains and improves Investec's core data mesh infrastructure and code.  The focus is on creating scalable, performant and reliable systems, to ingest, process, and distribute data while adhering to data mesh principles like domain ownership and data as a product.

As a Data (Platform) Engineer, you will play a crucial role in designing, building, and maintaining the data infrastructure that powers our organization's data-driven initiatives

Your expertise in data engineering, DevOps, Azure cloud technologies, and software development will be essential in driving innovation and enabling efficient data processing and analytics at scale.

Experience, skill and capability

• Developing reusable technology and solutions that enable teams to ingest, store, transform, and serve their own data products.
• Managing and optimizing cloud-based data infrastructure utilizing Infrastructure-as-Code principles.
• Implementing monitoring, alerting, and responsible for automation solutions to ensure the reliability and performance of the data platform.
• Implementing data security best practices and ensuring compliance with relevant regulations.
• Implementing access controls, encryption, and other security measures to protect sensitive data.
• Embracing continuous learning and experimentation to stay updated on emerging technologies, from R&D on new technologies to assisting domains in solving complex data challenges.
• Strong programming skills in Python (Scala is a plus), SQL, C#. Proficient in software engineering best practices.
• Expert in batch, distributed data processing, and near real-time streaming data pipelines with technologies like Event Hubs, Stream Analytics, API's and structured streaming in Azure Databricks
• Experience in Data Lake / Big Data Analytics platform implementation with cloud-based solutions in Azure.
• Strong overall Azure Databricks experience including experience implementing medallion architectures using Delta lake format.
• Strong Proficiency in Azure DevOps, GIT, CI/CD pipeline management.
• Strong expertise in Infrastructure-as-Code (IaC) deployment using BICEP and proficiency in industry-best practices.
• Be able to mentor and lead junior resources where applicable
• Work alongside and assist business units in solving complex problems and bridge the gap from technical to business and vice versa.
• Be able to work in a collaborative team with strong collaborative skills
• Have a passion for learning and embracing exciting and new technologies

The Investec Culture
At Investec we look for intelligent, energetic people filled with passion, integrity and curiosity. We value individuals who in turn value our culture that is, a flexible attitude comfortable to live with ambiguity and willing to challenge the status quo. Diversity, talent and leadership are respected in pursuit of the growth of our business. People who can manage themselves and build strong relationships in order to get things done, will perform in out of the ordinary ways in our environment.