Location: New York(Hybrid)
Employment Type: Contract
Experience: 10+ Years
Job Summary
We are seeking an experienced Lead Databricks Developer to design, develop, and lead the implementation of enterprise-scale data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in Databricks, Apache Spark, Delta Lake, cloud data platforms, and modern data engineering practices, along with the ability to lead technical teams and drive end-to-end delivery of scalable data solutions.
Key Responsibilities
- Lead the design, development, and implementation of enterprise data engineering solutions using Databricks.
- Build and optimize scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
- Develop and manage Delta Lake architectures for reliable, high-performance data processing.
- Design and implement batch and streaming data pipelines for enterprise analytics.
- Build and optimize Lakehouse architectures supporting BI, AI, and machine learning workloads.
- Collaborate with Solution Architects, Data Architects, Data Scientists, and business stakeholders to define technical solutions.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Implement data quality, governance, lineage, security, and monitoring best practices.
- Integrate data from multiple enterprise systems, APIs, databases, and cloud platforms.
- Develop reusable frameworks, coding standards, and CI/CD pipelines for data engineering.
- Mentor junior developers, perform code reviews, and provide technical leadership.
- Support production deployments, troubleshooting, and performance tuning.
- Drive Agile delivery while ensuring high-quality software engineering practices.
Required Qualifications
- 8+ years of experience in Data Engineering with at least 4+ years of hands-on Databricks development.
- Strong expertise in Databricks Lakehouse Platform.
- Advanced experience with Apache Spark (PySpark and/or Scala).
- Strong knowledge of Delta Lake, Spark SQL, and distributed data processing.
- Experience building enterprise-scale ETL/ELT pipelines.
- Strong proficiency in Python and SQL.
- Experience with Azure Databricks, AWS Databricks, or GCP Databricks.
- Hands-on experience with cloud storage technologies such as ADLS, S3, or Google Cloud Storage.
- Experience integrating structured and unstructured data sources.
- Knowledge of data modeling, data warehousing, and dimensional modeling.
- Experience with Git, CI/CD pipelines, and DevOps practices.
- Strong understanding of data governance, security, and performance optimization.
- Experience leading technical teams and mentoring developers.
- Strong communication and stakeholder management skills.
- Experience working in Agile/Scrum environments.
Preferred Qualifications
- Experience with Azure Data Factory, Azure Synapse, AWS Glue, or similar cloud data services.
- Knowledge of Kafka, Event Hubs, or other streaming technologies.
- Experience supporting AI/ML data platforms and feature engineering.
- Databricks Certified Data Engineer certification.
- Experience implementing enterprise data modernization and cloud migration initiatives.
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
Similar jobs
Show more