Job Title: Data Engineer
Job ID: 2023-11994
Job Location: Morris Plains, NJ (remote to start)
Job Travel Location(s):
# Positions: 2
Employment Type: W2
Candidate Constraints:
Duration: Long Term
# of Layers: 0
Work Eligibility: All Work Authorizations are Permitted
Key Technology: Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
Job Responsibilities- Work with business and technical leadership to understand requirements.
- Design to the requirements and document the designs.
- Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark
- Do data modeling as needed for the requirements.
- Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive
- Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or Hadoop Edge Nodes
- Implement Hadoop job orchestration using Shell scripting, Apache Oozie, CA7 Enterprise Scheduler and Airflow
- Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment
- Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.
Skills and Experience Required
- Strong development experience in Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
- Experience of Ab Initio is a bonus.
- Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures
- Excellent work experience on Hadoop as data warehouse/Data Lake implementations
- Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory)
- Unix/Linux Shell scripting (KSH) and basic administration of Unix servers
- CA7 Enterprise Scheduler
- Experience with AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch)
- Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration)
- Experience in Jira and Confluence
- Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
- Health care domain knowledge is a plus.
#J-18808-Ljbffr
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
Similar jobs
Show more