- Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.
- Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
- Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
- Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
- Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.
- Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
- Implement data governance, access control, and lineage using Unity Catalog.
- Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
- Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.
- Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.
- Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
- Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).
- Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
- Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
- Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.
- Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
- Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
- Languages: Python, SQL; Scala (nice to have).
- DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
- Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.
- Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.
- Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
- Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
- Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
- Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.
- Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
- Implement data governance, access control, and lineage using Unity Catalog.
- Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
- Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.
- Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.
- Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
- Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).
- Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
- Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
- Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.
- Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
- Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
- Languages: Python, SQL; Scala (nice to have).
- DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
- Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.
- Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.
- Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
- Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
- Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
- Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.
- Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
- Implement data governance, access control, and lineage using Unity Catalog.
- Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
- Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.
- Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.
- Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
- Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).
- Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
- Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
- Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.
- Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
- Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
- Languages: Python, SQL; Scala (nice to have).
- DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
- Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
Similar jobs
