Lead Data engineer

VSG Business Solutions LLC (Charlotte NC, USA) Follow 17 hours ago

Our challenge:

We are seeking an experienced Lead Data Engineer to design, develop, and optimize scalable data platforms and pipelines that support enterprise analytics, reporting, machine learning, and business intelligence initiatives. The ideal candidate will have strong expertise in modern data engineering technologies, cloud platforms, data warehousing, and big data ecosystems, along with proven leadership experience in driving data engineering projects and mentoring teams.

The Role

Responsibilities:

Lead the design, development, and implementation of scalable data pipelines and data integration solutions.
Architect and maintain enterprise data platforms, data lakes, and data warehouses.
Build and optimize batch and real-time data processing frameworks.
Collaborate with business stakeholders, analysts, data scientists, and application teams to understand data requirements.
Develop and maintain ETL/ELT processes ensuring data accuracy, consistency, and reliability.
Implement data quality, governance, lineage, and monitoring frameworks.
Optimize data models, database performance, and query execution.
Drive cloud data platform adoption and modernization initiatives.
Ensure compliance with security, regulatory, and data privacy standards.
Lead code reviews, technical design sessions, and architecture discussions.
Mentor and guide data engineers while driving engineering best practices.
Support production systems, troubleshoot complex data issues, and implement preventive measures.

Requirements:

Experience in Data Engineering, Data Warehousing, or Big Data technologies.
Strong proficiency in Python and SQL.
Hands-on experience with Apache Spark, Databricks, Kafka, and Airflow.
Strong experience in ETL/ELT development and data pipeline orchestration.
Expertise in cloud platforms such as AWS, Azure, or GCP.
Experience with data warehouses including Snowflake, Redshift, Synapse, or BigQuery.
Strong understanding of data modeling, dimensional modeling, and database design principles.
Experience with CI/CD, Git, Docker, and Kubernetes.
Knowledge of Data Lake, Lakehouse, and modern data architecture patterns.
Strong problem-solving, analytical, and leadership skills.
Experience working in Agile/Scrum environments.

Preferred Skills

Experience with Data Mesh, Data Fabric, or modern distributed data architectures.
Exposure to AI/ML data pipelines and MLOps frameworks.
Knowledge of metadata management, data lineage, and data governance tools.
Experience with Infrastructure as Code (Terraform, CloudFormation, etc.).
Hands-on experience with streaming technologies and real-time analytics.
Cloud certifications (AWS, Azure, or GCP).
Databricks, Snowflake, or other relevant data platform certifications.
Experience leading geographically distributed teams.
Familiarity with Master Data Management (MDM) solutions.
Excellent stakeholder management and communication skills.