Software Engineer - Data - Data Mesh & Lakehouse - Lawrence Harvey
Software Engineer - Data | Data Mesh & Lakehouse
About the Role
We are looking for a Software Engineer - Data to join a central Data Delivery team building the foundation of a global Data Mesh platform based on a Lakehouse architecture .
You will work on the engineering layer responsible for ingesting, processing and provisioning source-aligned data products into central data catalogs, enabling teams across the organisation to consume trusted data for analytics, reporting, machine learning and other data-driven applications.
A key part of the platform is enabling scalable data consumption through zero-copy data sharing , while maintaining strong governance, security and data quality standards.
What You'll Be Working On
- Design and build scalable batch and streaming data pipelines .
- Develop ingestion solutions for source-aligned data products.
- Work with Apache Spark and Databricks within a modern Lakehouse environment.
- Implement data ingestion strategies including Full Loads, Delta Loads and Change Data Capture (CDC) .
- Build and maintain streaming pipelines using technologies such as Kafka, Flink or Confluent .
- Manage datasets stored across Google Cloud Storage and Azure Blob Storage .
- Enable secure zero-copy data sharing through technologies such as Databricks Unity Catalog and BigQuery.
- Implement data governance and access-control models including RBAC and attribute-based access control .
- Design solutions capable of handling complex schema evolution , including backward and forward compatibility.
Tech Environment
Data & Processing: Apache Spark, Databricks, BigQuery
Streaming & Ingestion: Kafka, Flink, Confluent, Airbyte
Cloud & Storage: GCP, Google Cloud Storage, Azure, Azure Blob Storage
Orchestration & Platform: Airflow, Kubernetes
Governance: Databricks Unity Catalog, RBAC, ABAC/CBAC, Data Contracts
CI/CD: GitLab, Azure DevOps, JFrog Artifactory
Quality & Security: SonarQube, Snyk
What We're Looking For
- Strong professional experience in Data Engineering or Software Engineering focused on data platforms .
- Hands-on experience with Databricks and Apache Spark .
- Experience designing distributed data pipelines in cloud environments.
- Strong understanding of Lakehouse architectures .
- Experience or strong knowledge of Data Mesh principles and Data Products .
- Experience with batch and streaming ingestion patterns.
- Knowledge of CDC and incremental data processing strategies .
- Experience dealing with schema evolution in production data pipelines.
- Understanding of modern data governance and access-control models .
- Experience with Airflow, Kubernetes and CI/CD .
- Comfortable working collaboratively through code reviews and technical design discussions.
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
