Data Engineer
Technologie
InTech Partner
Vor 4 Tagen
Stellenbeschreibung
We’re looking for an experienced Data Engineer to join a Data Ingestion team working with large-scale real-world driving data used by downstream annotation, data science and model training teams.
The main focus of the role is improving the reliability, efficiency and throughput of high-volume data ingestion pipelines, solving production issues and making distributed data processing workflows more resilient and scalable.
🔹 What you’ll be working on
- Debugging and resolving failing or blocked data ingestion pipelines
- Investigating issues caused by corrupt, malformed or unexpected data
- Making pipelines more resilient so individual bad data segments don’t block wider workflows
- Improving ingestion of varied data formats from partners, suppliers and third-party sources
- Supporting orchestration across multi-step workflows, including dependencies, retries and queue management
- Optimising Apache Spark jobs for throughput, compute efficiency and reliability
- Reducing operational overhead caused by failed jobs, stalled pipelines and manual interventions
- Working with high-volume batch processing systems where performance, reliability and cost all matter
- Prioritising and unblocking important datasets for downstream annotation, data science and model training teams
- Strong production experience with Apache Spark
- Strong Python engineering skills
- Hands-on experience building, debugging or operating large-scale data ingestion, ETL or data processing pipelines
- Experience with distributed data processing systems
- Proven ability to optimise workloads for throughput, compute efficiency and reliability
- Experience troubleshooting production pipeline failures
- Confidence working with messy, corrupt, incomplete or inconsistent datasets
- Understanding of orchestration across multi-step pipelines and downstream dependencies
- Experience working at significant data scale, ideally PB-scale or similarly high-throughput environments
- Ability to work independently and ramp up quickly in a fast-moving technical environment
- Airflow, Flyte, Databricks Workflows or similar orchestration tools
- Databricks, Delta Lake / Delta tables
- Scala or Java, particularly in Spark environments
- Queue-based processing, retry handling and priority workflows
- High-throughput batch processing systems
- Platforms involving multiple data producers, consumers or external data sources
- Third-party, partner or supplier data with inconsistent formats or quality
- Automotive, robotics, autonomy, mapping, ML data platforms or embodied AI
- Compute and storage cost optimisation
- High-performance engineering environments involving distributed systems and throughput-focused workloads
Interessiert an dieser Stelle?