Lead Data Engineer

Argyllinfotech (Illinois, USA) Follow 4 hours ago

Job Title: Lead Data Engineer (AWS | Databricks | Python | Spark)

Location: 100% Remote (Preferred: Chicago, IL / Chicagoland Area)
Client Location: Chicago, IL
Duration: 6+ Months Contract (Contract-to-Hire)
Start Date: 07/20/2026
End Date: 12/31/2026
Position Type: Contract (C2C)
No. of Positions: 1

Position Overview

We are seeking a highly skilled
Lead Data Engineer to join a growing Data & Analytics team responsible for building and modernizing an enterprise-scale
AWS + Databricks Lakehouse platform. This is a hands-on technical leadership role where you will architect, develop, and optimize scalable data solutions that power analytics, reporting, and AI initiatives across the organization.

The ideal candidate has extensive experience designing enterprise data platforms, building robust ETL/ELT pipelines, and working with cloud-native technologies such as
Databricks, Apache Spark, Delta Lake, Unity Catalog, AWS, SQL, and Python.

This position offers the opportunity to influence enterprise data architecture while mentoring engineering teams and collaborating closely with business stakeholders.

Key Responsibilities

Design, develop, and maintain scalable enterprise data pipelines using Databricks and AWS.
Lead the architecture and implementation of modern Lakehouse solutions.
Build high-performance ETL/ELT pipelines supporting both batch and streaming workloads.
Develop data ingestion, transformation, and orchestration frameworks.
Optimize Spark workloads for performance, scalability, and cost efficiency.
Implement Delta Lake best practices for reliable and governed data management.
Design and maintain enterprise data models supporting analytics and reporting.
Collaborate with business stakeholders to translate requirements into scalable technical solutions.
Establish engineering standards, coding best practices, and data governance processes.
Monitor pipeline performance and troubleshoot production issues.
Mentor junior engineers while remaining actively involved in hands-on development.
Support CI/CD automation and DevOps best practices across the data platform.

Required Technical Skills

Databricks

Apache Spark
Delta Lake
Unity Catalog
Databricks Workflows
Lakehouse Architecture
Notebook Development
Performance Optimization

AWS Cloud

Amazon S3
IAM
VPC & Networking
Cloud Storage
Security & Access Management
Cloud-native Architecture

Programming

Python
SQL
PySpark
Spark SQL
Data Processing Frameworks

Data Engineering

ETL / ELT Development
Data Pipelines
Data Integration
Data Transformation
Batch Processing
Streaming Data Processing
Distributed Computing

Data Architecture

Lakehouse Architecture
Medallion Architecture
Data Modeling
Data Warehousing
Metadata Management
Data Governance
Data Quality

DevOps & Automation

CI/CD Pipelines
Git
Azure DevOps or GitHub Actions
Infrastructure Automation
Monitoring & Logging

Required Qualifications

10+ years of overall IT experience.
5+ years of hands-on Data Engineering experience.
Strong experience designing enterprise-scale ETL/ELT solutions.
Expert-level experience with Databricks, Apache Spark, Delta Lake, and Unity Catalog.
Strong AWS experience including S3, IAM, networking, and cloud architecture.
Advanced SQL and Python programming skills.
Experience building batch and streaming data pipelines.
Strong understanding of distributed computing concepts.
Experience implementing Lakehouse and Medallion architecture.
Experience with Agile software development methodologies.
Strong problem-solving and analytical skills.
Excellent communication and stakeholder management abilities.