Zum Hauptinhalt springen

Data Engineer

Technologie
InTech Partner
Vor 4 Tagen

Stellenbeschreibung

We’re looking for an experienced Data Engineer to join a Data Ingestion team working with large-scale real-world driving data used by downstream annotation, data science and model training teams.

The main focus of the role is improving the reliability, efficiency and throughput of high-volume data ingestion pipelines, solving production issues and making distributed data processing workflows more resilient and scalable.

🔹 What you’ll be working on

  • Debugging and resolving failing or blocked data ingestion pipelines
  • Investigating issues caused by corrupt, malformed or unexpected data
  • Making pipelines more resilient so individual bad data segments don’t block wider workflows
  • Improving ingestion of varied data formats from partners, suppliers and third-party sources
  • Supporting orchestration across multi-step workflows, including dependencies, retries and queue management
  • Optimising Apache Spark jobs for throughput, compute efficiency and reliability
  • Reducing operational overhead caused by failed jobs, stalled pipelines and manual interventions
  • Working with high-volume batch processing systems where performance, reliability and cost all matter
  • Prioritising and unblocking important datasets for downstream annotation, data science and model training teams
🔹 What we’re looking for
  • Strong production experience with Apache Spark
  • Strong Python engineering skills
  • Hands-on experience building, debugging or operating large-scale data ingestion, ETL or data processing pipelines
  • Experience with distributed data processing systems
  • Proven ability to optimise workloads for throughput, compute efficiency and reliability
  • Experience troubleshooting production pipeline failures
  • Confidence working with messy, corrupt, incomplete or inconsistent datasets
  • Understanding of orchestration across multi-step pipelines and downstream dependencies
  • Experience working at significant data scale, ideally PB-scale or similarly high-throughput environments
  • Ability to work independently and ramp up quickly in a fast-moving technical environment
🔹 Nice to have Experience with any of the following would be a strong advantage:

  • Airflow, Flyte, Databricks Workflows or similar orchestration tools
  • Databricks, Delta Lake / Delta tables
  • Scala or Java, particularly in Spark environments
  • Queue-based processing, retry handling and priority workflows
  • High-throughput batch processing systems
  • Platforms involving multiple data producers, consumers or external data sources
  • Third-party, partner or supplier data with inconsistent formats or quality
  • Automotive, robotics, autonomy, mapping, ML data platforms or embodied AI
  • Compute and storage cost optimisation
  • High-performance engineering environments involving distributed systems and throughput-focused workloads

Interessiert an dieser Stelle?