AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a
Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards.
Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards.
WHAT YOU WILL DO
- Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale.
- Make architectural decisions around partitioning, file formats, schema and data-type strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake.
- Own the design of scheduled batch workflows (DAGs) on the client's Airflow setup, defining pipeline structure, dependencies, and triggering strategies, and driving architectural discussions around them.
- Drive the use of the client's AWS data stack, including an S3-backed data lake, Athena, and EKS/Kubernetes.
- Partner directly with the client's DevOps team to clarify functional and non-functional requirements.
- Review pull requests and enforce code quality standards.
- Guide senior developers and ensure alignment with the client's engineering practices.
MUST HAVES
-
7+ years of engineering experience , with a proven track record designing and implementing
ETL pipelines and making architectural decisions for large-volume data systems.
7+ years of engineering experience , with a proven track record designing and implementing
ETL pipelines and making architectural decisions for large-volume data systems.
- Hands-on experience with
OLAP-style analytical data architecture , with experience in
Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar technologies .
OLAP-style analytical data architecture , with experience in
Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar technologies .
- Hands-on experience designing
data lakes backed by object storage such as S3 or equivalent , including partitioning strategies, file formats such as Parquet/ORC, and cost/performance tradeoffs.
data lakes backed by object storage such as S3 or equivalent , including partitioning strategies, file formats such as Parquet/ORC, and cost/performance tradeoffs.
- Deep familiarity with
DAG-style workflow definition and triggering , with substantial experience in
Airflow or comparable orchestrators such as Dagster, Prefect, Luigi, or Step Functions .
DAG-style workflow definition and triggering , with substantial experience in
Airflow or comparable orchestrators such as Dagster, Prefect, Luigi, or Step Functions .
- Practical experience across the
AWS data ecosystem , including S3-backed data lakes, serverless query engines such as Athena or equivalent, and EKS/Kubernetes.
AWS data ecosystem , including S3-backed data lakes, serverless query engines such as Athena or equivalent, and EKS/Kubernetes.
- Strong backend proficiency in
Python , with experience using
FastAPI or Flask .
Python , with experience using
FastAPI or Flask .
- Comfortable working with
REST and GraphQL .
REST and GraphQL .
- Experience with
Docker and PostgreSQL for transactional and application layers.
Docker and PostgreSQL for transactional and application layers.
- Highly comfortable working in Mac/Linux terminal-centric environments.
- Practical, hands-on experience with
AI-assisted development tools such as Claude Code , combined with the critical judgment to challenge AI-generated output when it compromises long-term maintainability.
AI-assisted development tools such as Claude Code , combined with the critical judgment to challenge AI-generated output when it compromises long-term maintainability.
- Leadership experience setting standards for responsible use of AI tooling, including identifying risky AI-driven shortcuts during code review.
- Strong communication and technical judgment, with the ability to defend technical decisions, challenge quick fixes with sound reasoning, and balance long-term maintainability with pragmatic delivery.
- Upper-Intermediate English level.
NICE TO HAVES
- Direct production experience with Athena.
- Working knowledge of TypeScript and React, sufficient to guide integrations and review frontend-adjacent pull requests.
- Production experience building AI features using AWS Bedrock, LangChain, Pydantic AI, or similar technologies.
- Experience with monorepo tooling such as Nx or modern package managers such as Poetry, UV, or Yarn.
- Experience with Redis and caching layers or SageMaker.
- Experience with marketing data structures, campaign management APIs, or digital advertising metrics.
PERKS AND BENEFITS
-
Growth without limits : build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
Growth without limits : build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
-
Competitive compensation : get recognition that reflects your skills and impact, with regular performance and compensation reviews
Competitive compensation : get recognition that reflects your skills and impact, with regular performance and compensation reviews
-
Flexibility : work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
Flexibility : work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
-
Meaningful, modern projects : build impactful products using modern technologies alongside global teams and leading brands
Meaningful, modern projects : build impactful products using modern technologies alongside global teams and leading brands
-
Collaborative culture : join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
Collaborative culture : join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
-
Well-being & support : access local well-being programs and people-focused support tailored to your location
Well-being & support : access local well-being programs and people-focused support tailored to your location
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
Similar jobs
Show more
