StreamSets Developer

Centstone (US, USA) Follow 19 hours ago

Contract Azure Real-Time Data Data Ingestion Data Integration Event-driven

Apply Now

StreamSets Developer

Location: Remote
Job Type: Long-Term Contract
Experience: 5+ Years

Job Description

We are seeking an experienced StreamSets Developer to design, develop, implement, and support enterprise-scale data ingestion and integration pipelines. The ideal candidate will have strong hands-on experience with StreamSets Data Collector (SDC), ETL/ELT, SQL, APIs, databases, cloud platforms, and real-time data integration.

Key Responsibilities

Design, develop, configure, and maintain data pipelines using StreamSets Data Collector (SDC).
Build scalable ETL/ELT pipelines for batch and real-time data processing.
Integrate data from databases, REST APIs, files, cloud platforms, and enterprise applications.
Configure StreamSets origins, processors, destinations, and pipeline parameters.
Develop data transformation, cleansing, validation, enrichment, and routing logic.
Monitor and troubleshoot pipeline performance and data integration issues.
Implement error handling, logging, recovery mechanisms, and data quality checks.
Work with JSON, XML, CSV, Avro, and Parquet data formats.
Integrate StreamSets with relational/NoSQL databases, data warehouses, and cloud platforms.
Optimize pipelines for scalability, reliability, performance, and throughput.

Required Skills

Strong hands-on experience with StreamSets Data Collector (SDC).
Proven experience developing and supporting StreamSets pipelines.
Strong ETL/ELT and data integration experience.
Strong SQL skills.
Experience with Oracle, SQL Server, PostgreSQL, or MySQL.
Experience with REST APIs and web services.
Experience with JSON, XML, CSV, Avro, and Parquet.
5+ years of experience in Data Engineering, ETL, or Data Integration.

Preferred Skills

StreamSets Transformer and StreamSets Control Hub.
AWS, Azure, or GCP.
Snowflake, Databricks, or BigQuery.
Kafka and event-driven data integration.
Hadoop/HDFS.
Python or Java.
Jenkins, GitHub Actions, GitLab, or Azure DevOps.
Docker/Kubernetes.
Data warehouses and data lakes.
Data governance, metadata management, and data quality.

Qualifications

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field preferred.
Strong analytical, troubleshooting, communication, and problem-solving skills.
Strong hands-on StreamSets development experience is required.

Apply Now

Save Job