This role is for one of the Weekday's clients
Min Experience: 3+ years
Location: Bengaluru
JobType: full-time
We are looking for a highly skilled Big Data Engineer with 3+ years of experience in building scalable data pipelines and distributed systems. The ideal candidate will have strong expertise in Apache Spark (Scala), experience working across on-premise and AWS environments, and a solid understanding of large-scale data processing in AdTech ecosystems.
This role involves working on high-volume datasets (billions of records), optimizing distributed jobs, and contributing to the design of robust data infrastructure powering analytics and identity-driven use cases.
Requirements
Key Responsibilities:
- Design, develop, and optimize large-scale batch data pipelines using Apache Spark (Scala)
- Process and transform high-volume datasets (TBs of data) in distributed environments
- Build and maintain data pipelines across hybrid infrastructure (On-Prem + AWS)
- Work with object storage systems such as S3 and MinIO for efficient data access and storage
- Develop reusable and scalable data processing frameworks
- Optimize Spark jobs for performance (memory tuning, partitioning, shuffling, etc.)
- Manage and orchestrate workloads using HashiCorp Nomad
- Integrate data pipelines with PostgreSQL and other downstream systems
- Ensure data quality, consistency, and reliability across pipelines
- Troubleshoot production issues and perform root cause analysis
- Contribute to system design discussions, especially for high-scale AdTech use cases (identity resolution, user profiling, etc.)
Required Skills:
- Strong programming experience in Scala
- Good working knowledge of Python (for auxiliary tasks, scripting, or ML integration)
- Deep expertise in Apache Spark (Core and SQL)
- Strong understanding of distributed data processing and large-scale systems
- Experience working with AWS (S3, EMR or equivalent ecosystem) and on-prem clusters
- Hands-on experience with object storage systems (S3 / MinIO)
- Experience with HashiCorp Nomad or similar orchestration tools
- Solid understanding of data modeling and ETL pipeline design
- Experience working with PostgreSQL or similar relational databases
- Strong debugging and performance tuning skills for Spark jobs
- Familiarity with Unix/Linux environments and shell scripting
Good to Have:
- Experience in AdTech, Identity Graph, or User Profiling systems
- Exposure to machine learning pipelines or feature engineering workflows
- Experience with data lake architectures
- Understanding of cost optimization and resource management in AWS
Tech Stack Summary:
- Languages: Scala (Primary), Python (Secondary)
- Processing: Apache Spark (Core and SQL)
- Infrastructure: AWS and On-Prem
- Storage: S3, MinIO
- Orchestration: HashiCorp Nomad
- Database: PostgreSQL
Must-have skills
Spark, SQL, Scala
Good-to-have skills
Python, AWS, Big Data
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
