Senior Data Engineer - Python & Microservices

We are looking for an experienced Senior Python Data Engineer with strong expertise in Python development, microservices architecture, data engineering, and backend API development. The ideal candidate will have hands-on experience building scalable Python-based applications and data services, designing robust REST APIs, performing complex data transformations, and developing high-performance backend frameworks to support enterprise data-driven applications.

This role requires strong proficiency in Python web frameworks, asynchronous programming, microservices architecture, ETL processes, database management, and modern software engineering best practices. Exposure to big data technologies, DevOps practices, and cloud-native architectures will be highly advantageous.

Requirements

Key Responsibilities

• Design, develop, and maintain scalable Python-based web frameworks and backend services.
• Develop robust RESTful APIs using Python frameworks such as FastAPI, Flask, or Django.
• Build and maintain microservices-based applications supporting enterprise data platforms.
• Design and implement data transformation pipelines for data ingestion, processing, and enrichment.
• Develop backend systems to efficiently serve datasets through APIs and other interfaces.
• Collaborate with cross-functional teams to deliver scalable and secure data-driven applications.
• Design and maintain relational and NoSQL database schemas.
• Develop efficient ETL (Extract, Transform, Load) processes for enterprise data requirements.
• Optimize application and database performance to ensure scalability and reliability.
• Implement asynchronous programming techniques using asyncio and optimize concurrent connections and I/O operations.
• Ensure data integrity, consistency, and quality across data transformation pipelines.
• Participate in code reviews and contribute to software engineering best practices.
• Troubleshoot, debug, and resolve production issues across applications and data pipelines.
• Collaborate closely with stakeholders and engineering teams throughout the SDLC.

Required Technical Skills

Core Python Development

• Strong hands-on expertise in Core Python.
• Experience building enterprise-grade Python applications.
• Strong understanding of:
• Python programming concepts
• Object-oriented programming
• Asynchronous programming (asyncio)
• Concurrent connections handling
• I/O optimization techniques
• Clean code and software design principles

Python Web Frameworks

Hands-on experience with one or more of the following:

• FastAPI
• Flask
• Django
• Pyramid

Strong understanding of:

• RESTful API development
• Routing
• Authentication mechanisms
• Testing frameworks
• Database integrations
• Framework architecture and scalability

Microservices Architecture

Strong experience with:

• Microservices-based application design
• Service-oriented architectures
• API integrations
• Distributed systems
• Enterprise backend services

Knowledge of:

• API Gateways
• Service communication patterns
• Scalable backend architectures

Security & API Management

Experience implementing:

• OAuth
• JWT Authentication
• API Security Best Practices
• Authorization mechanisms
• Encryption standards
• Secure API development practices

Data Engineering & Data Transformations

Strong hands-on experience with:

• Data transformation pipelines
• ETL processes
• Data cleansing and enrichment
• Data aggregation techniques
• Data quality and integrity management

Experience in:

• Dataset preparation
• Data processing workflows
• Enterprise data platforms

Database Management

Experience working with:

Relational Databases

• PostgreSQL
• MySQL
• SQL Server (preferred)

NoSQL Databases

• MongoDB
• Redis (good to have)

Strong knowledge of:

• Database schema design
• Performance optimization
• Transactions management
• Efficient data retrieval techniques

ORM Frameworks

Hands-on experience with:

• SQLAlchemy
• Django ORM

Ability to perform:

• CRUD operations
• Query optimization
• Transaction handling
• Data modelling

DevOps & CI/CD

Experience with:

• CI/CD pipeline development and maintenance
• Git-based workflows
• Version control best practices
• Branching strategies
• Code reviews
• Production deployments

Big Data Technologies (Preferred)

Exposure to:

• Apache Spark
• Hadoop
• Apache Kafka

Understanding of:

• Data warehousing concepts
• Large-scale data processing architectures

Scripting Skills

Strong proficiency in:

• SQL
• Shell Scripting
• Python scripting

Cloud & Containerization (Good to Have)

Exposure to:

• AWS
• Microsoft Azure
• Google Cloud Platform (GCP)
• Docker
• Kubernetes

Data Governance (Good to Have)

Knowledge of:

• Data governance principles
• Data security best practices
• Compliance frameworks
• Enterprise data management practices

Preferred Additional Skills

• Experience with R Programming Language.
• Experience working on enterprise-scale data engineering initiatives.
• Exposure to cloud-native architectures and containerized deployments.

Required Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management capabilities.
• Strong debugging and troubleshooting skills.
• Ability to work effectively in Agile and collaborative environments.
• Strong ownership mindset and attention to detail.
• Ability to adapt quickly to evolving business and technical requirements.
• Strong documentation and technical communication skills.