Principal Databricks Data Engineer

Principal Databricks Data Engineer

Experience Required: 12–18 Years

Key Requirements

• 12–18 years of overall Data Engineering experience
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
• 5+ years of hands‑on experience with Databricks and Apache Spark at scale
• Strong experience modernizing legacy Cloudera platforms including:
  • CDH/CDP
  • Hive
  • HBase
  • Impala
  • Spark
• Modernize Cloudera platforms to Databricks Lakehouse architecture
• Redesign ingestion, transformation, and consumption patterns from HDFS‑based architecture to Cloud Object Storage and Delta Lake
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
• Ensure data reconciliation, audit integrity, and consistency during migration
• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
• Implement layered data architecture including:
  • Raw / Landing Layer
  • Curated / Conformed Layer
  • Semantic / Consumption Layer
• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
• Strong experience with finance and risk data models including:
  • General Ledger
  • Sub‑ledger
  • Financial Hierarchies
  • Credit Risk
  • Liquidity Risk
  • Market Risk
• Enable reporting capabilities including:
  • Aggregation
  • Drill‑down
  • Drill‑back
• Build and manage semantic and consumption layers for BI, reporting, and analytics
• Define business metrics, dimensions, hierarchies, and KPIs
• Experience with:
  • Databricks SQL
  • Delta Tables
  • dbt or similar frameworks
• Develop and optimize large‑scale data pipelines using:
  • PySpark
  • Spark SQL
  • Delta Lake
• Implement Medallion Architecture including:
  • Bronze Layer
  • Silver Layer
  • Gold Layer
• Optimize workloads using:
  • Z‑ORDER
  • OPTIMIZE
  • Caching
  • Cluster Configuration Tuning
• Implement:
  • Data Governance
  • Data Quality Frameworks
  • Reconciliation Controls
  • Exception Handling
• Establish data lineage and metadata management
• Ensure data security, access control, and compliance standards
• Experience with AWS or Azure cloud platforms
• Experience with CI/CD pipelines using:
  • Git
  • Terraform
  • Jenkins
  • Azure DevOps
• Familiarity with:
  • Apache Airflow
  • Databricks Workflows
• Experience with dbt is an advantage
• Act as a technical authority and lead enterprise architecture decisions
• Mentor senior engineers and establish engineering standards
• Collaborate with finance, risk, analytics, and governance stakeholders
• Translate complex data structures into business‑ready insights

Nice‑to‑Have Skills

• Experience in BFSI, Capital Markets, or Regulatory Reporting
• Exposure to:
  • SAP Finance
  • Oracle Financials
  • SAP S/4HANA
• Experience supporting AI/ML workloads
• Databricks and Cloud Certifications

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives
• Shape enterprise finance and risk data platforms
• Support regulatory, management, and analytical reporting systems

Essential Skills

• Databricks
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Lakehouse Architecture
• Cloudera (CDH/CDP)
• Hive
• HBase
• Impala
• Data Warehouse & Data Lake
• Medallion Architecture
• Databricks SQL
• dbt
• AWS / Azure
• Airflow / Databricks Workflows
• CI/CD
• Terraform
• Jenkins
• Git
• Finance & Risk Data Models
• Enterprise Data Architecture

• Sign In
• Create Account

Sign in

To continue your application

Sign in with Google

Sign in with Linkedin

---

or continue with email

Email address

Password

Forgot your password?

Resume or CV

Improve your chances of getting an interview

Drag & drop or browse

.pdf, .doc, .docx (Max 10MB)

Continue →

By continuing you agree to our Terms & Privacy Policy.