Position OverviewWe are seeking a Senior Databricks Engineer to lead the design and implementation of a modern Databricks Lakehouse architecture.This role is focused on migrating legacy data solutions into scalable, governed, and performance-optimized pipelines using Databricks, PySpark, and Delta Lake.The engineer will play a key role in defining and building Bronze, Silver, and Gold data layers, enabling both batch and near real-time processing, and improving the reliability and usability of enterprise data platforms.This position will work closely with data engineers, architects, and stakeholders while providing technical leadership and guidance to more junior team members.Key ResponsibilitiesArchitecture & EngineeringArchitect and implement end-to-end data solutions within the Databricks Lakehouse environmentDesign and build ingestion, transformation, and serving pipelines using PySpark, Spark SQL, and Delta LakeEstablish and enforce medallion architecture practices across Bronze, Silver, and Gold layersMigration & ModernizationLead the migration of legacy Azure SQL, SSIS, and ADF pipelines into Databricks-native workflowsConvert existing T-SQL and ETL logic into scalable Spark-based processingSupport modernisation of data platforms with a focus on governance, scalability, and maintainabilityStreaming, Governance & OptimizationImplement Auto Loader, Delta Lake Change Data Feed (CDF), and streaming/batch ingestion patternsDefine and enforce governance using Unity Catalog, including RBAC and data lineageOptimise performance through Photon, partitioning, Z-Order indexing, and cluster tuningValidation & Technical LeadershipGuide validation and reconciliation strategies between legacy SQL outputs and Delta-based pipelinesPerform code reviews and promote engineering best practicesMentor mid-level and junior engineers across design, development, and troubleshooting activitiesRequired Experience & QualificationsBachelor's degree in Computer Science, Engineering, Information Systems, or equivalent experience6+ years of experience in data engineering, with 3+ years of hands-on Databricks experienceStrong expertise in PySpark, Spark SQL, and Delta LakeProven experience migrating legacy data platforms such as SQL Server, SSIS, and ADFStrong understanding of ETL/ELT design, data modeling, and pipeline architectureExperience designing data solutions using Bronze / Silver / Gold layersHands-on experience with the Azure ecosystem, including ADLS, ADF, and related servicesStrong knowledge of performance tuning techniques for distributed data processingExperience leading technical implementation efforts and supporting junior engineersPreferred QualificationsExperience with Unity Catalog, Lakehouse Federation, or CDC frameworksDatabricks certifications (Associate or Professional level)Exposure to large-scale regulated datasets such as healthcare, claims, or financial dataExperience with both batch and streaming data pipelinesFamiliarity with enterprise-grade data governance and lineage practices
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
