Job Summary:
or continue with email
The Senior DataOps/Cloud Data Engineer is responsible for designing, developing, and optimizing data pipelines and models primarily using Azure Data Factory and Databricks within cloud environments. This role focuses on migrating and managing large-scale data from Oracle databases to Lakehouses following Medallion Architecture principles, ensuring robust data integration, quality, and security. The engineer will collaborate with cross-functional teams to transfer knowledge, maintain documentation, and support enterprise data solutions in a highly regulated, on-premises and cloud hybrid environment.
Responsibilities:- Design, develop, and optimize Azure Data Factory and Databricks pipelines for data ingestion and transformation from Oracle databases to Lakehouses.
- Create and optimize data models based on Medallion Architecture and relational database principles.
- Translate existing Informatica ETL workflows to Azure Data Factory and Databricks ELT processes.
- Develop and maintain data connections between cloud and on-premises systems for downstream consumers.
- Implement data quality checks including validation, profiling, cleansing, and monitoring across pipelines.
- Apply data governance and information architecture standards, including data anonymization and masking for sensitive data.
- Manage cloud data services such as Data Lakehouse, key vaults, virtual machines, and storage accounts.
- Develop and maintain documentation for designs, support, releases, and training materials, and conduct knowledge transfer sessions.
- Support continuous integration and deployment (CI/CD) pipelines and automate data provisioning tasks.
- Monitor and tune DataOps performance and troubleshoot production issues.
- Collaborate with project managers and stakeholders to ensure timely communication and project delivery.
- Extensive experience with ETL/ELT tools including Informatica, Azure Data Factory, and Databricks (Python, Lakeflow, SQL optimization).
- Proficiency in programming languages: Python, SQL, T-SQL, PL/SQL, and experience with SSIS and Microsoft Fabric.
- Strong knowledge of Medallion Architecture for data modeling and pipeline design.
- Experience with cloud data platforms, including Data as a Service (DaaS), Database as a Service (DBaaS), and Data Warehouse as a Service (DWaaS).
- Expertise in data pipeline orchestration, deployment, automation, and Delta Lake management.
- Experience implementing data governance, security, and compliance practices including Microsoft Entra ID integration.
- Strong analytical, problem-solving, and communication skills.
- Experience with Agile and SDLC methodologies, including testing phases (SIT, SAT, UAT).
- Experience using Azure DevOps for project and code management.
- Ability to create detailed technical documentation and training materials.
- Certifications in Databricks and/or Microsoft Fabric.
- Experience with advanced data visualization and KPI development.
- Familiarity with data anonymization and masking techniques.
- Experience with AODA/WCAG compliance standards.
- Previous public sector experience in large organizations.
- Experience leading teams and providing technical guidance.
- Must work onsite at the office location 5 days per week.
- Knowledge transfer activities require thorough documentation and one-on-one sessions before project completion or consultant departure.
- Must comply with strict data security and privacy requirements.
- Ability to handle large volumes of structured and unstructured data in OLAP and OLTP environments.
- Full-time, standard business hours with potential Agile sprint cycles.
- Regular communication and coordination with project teams and stakeholders to meet deadlines.
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
Similar jobs
Show more
