Senior Data Engineer - Python/ETL

Description :

Role : Level Lead / Senior (Individual Contributor / Team Lead Track)

Experience : 7 - 10 years of relevant professional experience

Location : Remote (Pune Based Preferred)

Employment Type : Contract

Industry Preference Any (Healthcare preferred Payer / Provider experience strongly preferred)

About the Role :

We are looking for a Senior Data Engineer with deep expertise in streaming architectures, graph database platforms, and large-scale data pipeline engineering. This is a high-ownership, hands-on role that sits at the intersection of real-time data infrastructure, entity resolution, and multi-database system design.

You will architect and build pipelines that drive a complex, multi-layered data platform ingesting from diverse upstream sources, resolving entities at scale, and keeping graph, relational, search, and caching layers in sync.

You will work closely with data architects, AI engineers, and product teams to deliver reliable, high-performance data infrastructure that powers downstream analytics and intelligent applications across any domain.

Key Responsibilities :

Streaming & Ingestion Architecture :

• Design and build production-grade CDC (Change Data Capture) pipelines using Apache Kafka, consuming events from PostgreSQL, SQL Server, and other RDBMS sources into a centralised knowledge graph.
• Architect multi-source ingestion connectors supporting schema evolution, backpressure handling, and at-least-once delivery guarantees across heterogeneous data sources.
• Configure and govern Confluent Schema Registry with Avro / Protobuf schemas across all Kafka topics; enforce backward and forward compatibility standards.
• Design micro-batch and streaming ETL/ELT workflows using Apache Spark or equivalent frameworks for bulk initial loads and ongoing incremental refresh patterns.
• Manage messaging workflows where required; define routing, dead-letter, and retry strategies appropriate to each integration pattern.

Graph Database Engineering :

• Design, build, and optimise graph data models on a production graph database platform; Neo4j is preferred but experience with Amazon Neptune, ArangoDB, TigerGraph, or equivalent graph databases is valued.
• Author complex graph queries and traversal patterns Cypher (Neo4j), Gremlin (Neptune/TinkerPop), or SPARQL for both operational and analytical use cases.
• Own ingestion-side write strategies for the graph layer : batch import patterns, upsert logic, index management, and performance tuning under high write throughput.
• Collaborate with senior architects to ensure graph data models honour defined schema constraints and governance standards; apply constraint validation frameworks where applicable.
• Engineer reliable data flows across complementary stores relational (PostgreSQL), search (Elasticsearch), caching (Redis), and time-series (TimescaleDB) with consistent transaction semantics.

Entity Resolution & Data Quality :

• Build probabilistic entity resolution engines for large-scale deduplication across master data domains customers, products, entities, or records leveraging record linkage concepts (Fellegi-Sunter model, blocking strategies, confidence thresholds) and libraries such as Splink, Zingg, or Dedupe.io.
• Define and enforce data quality validation rules at ingestion time; implement automated alerting for schema violations, volume anomalies, and SLA breaches.
• Design master data management patterns for cross-system entity matching and golden record creation; ensure consistency across all downstream consumers.

Data Platform & Lakehouse :

• Design and implement data lakehouse patterns (Iceberg / Parquet on S3-compatible or Azure storage) for historical data retention, cost-efficient storage, and analytical workloads.
• Build and maintain ETL/ELT pipelines using Apache Spark or dbt; define transformation logic, partitioning strategies, and incremental processing patterns.
• Ensure data lineage, audit trail, and observability are built into pipeline design from the outset using OpenTelemetry or equivalent tooling.

Technical Leadership & Collaboration :

• Contribute to a sub-team of data engineers; participate in sprint planning, design reviews, and on-call rotations for critical pipelines.
• Define and enforce coding standards, pipeline patterns, and infrastructure-as-code practices using Terraform, Docker, and Kubernetes.
• Drive proof-of-concept evaluations for new ingestion technologies, graph platforms, and data tooling relevant to the engagement.

Required Qualifications :

Experience :

• 7 to 10 years of progressive experience in data engineering or a closely related discipline.
• Demonstrated track record of delivering production-grade streaming and CDC pipeline systems in enterprise environments across any industry vertical.
• Hands-on experience with graph database platforms in production Neo4j preferred; Amazon Neptune, ArangoDB, TigerGraph, or equivalent is acceptable.
• Practical experience with entity resolution, fuzzy matching, or master data management at scale (500K records).
• Solid experience with multi-database architectures combining graph, relational, and search layers.
• Candidates from any industry are welcome; experience in regulated or data-intensive domains (financial services, retail, logistics, telecoms, healthcare) is advantageous.

Technical Skills :

• Streaming & CDC : Apache Kafka
• Graph Databases : Production experience with at least one graph database platform Neo4j (preferred), Amazon Neptune, ArangoDB, or TigerGraph; proficiency in the associated query language (Cypher, Gremlin, or SPARQL).
• Supporting Databases : PostgreSQL (relational), Elasticsearch (search), Redis (caching).
• Programming : Python (Advanced) pipeline automation, data workflow scripting, testing; SQL at expert level for complex transformations and query optimisation.
• Entity Resolution : Probabilistic record linkage concepts; practical experience with Splink, Zingg, Dedupe.io, or a comparable library.
• Data Engineering : High-volume ETL/ELT pipeline design; Apache Spark for distributed processing; data lakehouse patterns (Iceberg, Parquet, Delta Lake).
• Cloud & Infrastructure : AWS or Azure production delivery on at least one platform; Docker, Kubernetes, Terraform.
• Familiarity with semantic or schema standards OWL 2, RDF, SHACL, JSON-LD sufficient to write conformant graph data models against a defined schema.
• Experience with OpenTelemetry, distributed tracing, or observability tooling for pipeline monitoring and incident response.
• Prior work in compliance-driven data environments with audit trail, data masking, or access control requirements.
• Exposure to graph analytics and visualisation tooling such as Neo4j Bloom, Gephi, or equivalent.
• Experience with data governance platforms such as Microsoft Purview, Collibra, or Alation.

Preferred Qualifications :

• Bachelors or Masters degree in Computer Science, Information Systems, or a related engineering discipline.