Data Engineer with Apache Flink

Senior Data Platform Engineer — Nexus Data Platform

About the team

What you'll do

• Own pipeline slices end-to-end: data ingest into a distributed streaming platform, stream-processing transform jobs, and projection of clean, contract-compliant records into downstream stores.
• Onboard new data domains and production entities — legacy schema mapping, streaming topic contracts, and stream-processing job development.
• Build and run stream-processing jobs in the cloud — transforms along the change-data-capture → streaming → processing path and batch ingestion via managed big-data services — including checkpointing, failure recovery, and job-level runbooks.
• Deploy and troubleshoot streaming and processing workloads on a container orchestration platform — job lifecycle and day-to-day application-level troubleshooting, in partnership with SRE on the underlying cluster.
• Partner with downstream teams on data contracts: schema ownership, where consumers read from, and clear boundaries between the ETL pipeline and the teams that serve on top of it.

What we’re looking for

Required

• 5+ years experience building production grade streaming pipeline systems.
• Hands-on experience with a distributed streaming/messaging platform (Kafka + Flink/Spark or similar): topic and partition design, consumer group semantics, compaction, and schema-aware serialization.
• Production experience with a distributed stream-processing engine: SQL or DataStream-style jobs, state management, checkpointing, and operational debugging of long-running jobs.
• CDC ingestion familiarity — WAL or log-based change capture into a streaming platform.
• Demonstrated experience with SQL and schema design across relational sources.
• Demonstrated experience deploying and troubleshooting streaming workloads on a container orchestration platform — basic troubleshooting of deployments and running jobs.
• Track record shipping observable, testable pipeline components — health checks, smoke gates, benchmark-driven validation.
• Ability to work independently on assigned roadmap items while coordinating with cross-functional partners on contract boundaries.

Preferred

• Direct, hands-on experience with Apache Kafka and Apache Flink
• Familiarity with Debezium CDC capture.
• Experience projecting streaming data into an OLAP or analytical serving engine (StarRocks, ClickHouse, Druid, Pinot, or similar).
• Awareness of Gov Cloud deployment constraints and data residency considerations.
• Experience with multi-tenant SaaS data workloads (tenant partitioning, hot-partition skew).

How we work

• AI-assisted development — we use coding agents and automation to move fast; you should be comfortable reviewing and directing AI-generated changes, not writing every line by hand.
• Local-first iteration — every engineer has a local stack to develop and test pipeline changes before promoting to shared environments; schema and job work should be fast to try without waiting on central infrastructure.
• Prove it before you ship — automated smoke tests and performance benchmarks are part of how we validate changes, not a handoff to someone else at the end.
• Cross-team by design — data contracts between the pipeline and downstream teams are explicit; we coordinate early with external teams so downstream consumers stay aligned without slowing delivery.

What success looks like in the first 6 months

• At least one production data domain onboarded through Kafka and Flink end-to-end, with automated validation passing in local and shared environments.
• A partner team consuming pipeline output under a documented contract — clear access patterns and agreed SLA baselines.
• Deploy and integration workflow dependable enough for routine use, not a manual heroics exercise each time.
• Contributed runbooks, automation, or reusable patterns that make onboarding the next domain meaningfully faster.