Software Engineer, Data Core

Were looking for an experienced Software Engineer with a strong background to become an integral member of our Data-Core team, tasked with the mission of processing, structuring, and analyzing hundreds of millions of data sources.

Your role will be pivotal in creating a unified, up-to-date, and accurate utilities map, services, and applications for accelerating our mapping operations. Your contributions will directly impact our core product's success.Key ResponsibilitiesBuild and maintain components within distributed data pipelines - producer-consumer workflows over SQS, Airflow DAGs, Kubernetes-based processing.Integrate outputs from the algorithms and ML teams into production workflows - implementing ETL steps, handling data transformations, ensuring upstream results are consumed correctly.Contribute to shared Python libraries (14 packages consumed by 10+ services) - following existing patterns, writing clean code, understanding how changes affect downstream consumers.Work within stateful workflow engines - implementing new steps, fixing issues in ticket state machines and multi-stage recovery workflows.Debug pipeline issues by tracing data flow through SQS, Airflow, Kubernetes, and PostgreSQL to narrow down where things broke.Contribute to ETL and data infrastructure - multi-source ingestion, taxonomy-based normalization, and database operations.Improve reliability and observability - adding error handling, logging, monitoring, and alerting to existing services.Requirements: +4 years as a backend/software engineer with solid Python skills.Experience as a Data Infrastructure Engineer or in a similar role in managing and processing large-scale datasets.Proven experience with AI development tools - we use Claude Code as a daily tool on our team.Experience in deploying a diverse range of cloud-based technologies to support mission-critical projects, including expertise in understanding, testing, and deploying code within a Kubernetes environment.

Experience working with message queues (SQS, Kafka, RabbitMQ, or similar) - understands why messages can fail and how to handle errors in a consumer.Familiarity with Airflow (or comparable orchestration) and Kubernetes/Docker in production.Experience with AWS services (SQS, S3) or comparable cloud platforms.Solid PostgreSQL/SQL skills - comfortable writing complex queries, understanding schemas, working with multiple databases.Experience working in a large existing codebase - can read code they didn't write, follow established patterns, and contribute without breaking things.Solid debugging skills - can trace a problem through logs and multiple services rather than guessing at fixesBachelors degree in Computer Science, Engineering or similar (such as equivalent army background).This position is open to all candidates.