Lead Data Engineer

Role Overview

We are looking for a Lead Data Engineer with strong experience building scalable batch, near-real-time, and streaming data platforms on Microsoft Azure.

The role requires hands-on expertise in Python, PySpark, advanced SQL, Azure data services, and Medallion Architecture, along with a strong understanding of Apache Spark internals and distributed data processing.

Hands-on experience deploying Spark workloads on Kubernetes or Azure Kubernetes Service (AKS) is a strong plus, not a hard requirement. Candidates with deep Spark engineering fundamentals but limited Kubernetes exposure are encouraged to apply — structured onboarding support will be provided to build this skill on the job.

Key Responsibilities

• Design and build batch, near-real-time, and streaming data pipelines on Azure.
• Develop Bronze, Silver, and Gold data layers using Medallion Architecture.
• Develop and optimize PySpark applications for large-scale transformation, cleansing, enrichment, reconciliation, and validation.
• Implement incremental loads, CDC, watermarking, deduplication, schema evolution, retries, and recovery.
• Optimize Spark jobs — partitioning, shuffles, joins, file sizes, and query performance.
• Integrate data from REST APIs, SFTP, databases, files, enterprise systems, and Azure Event Hubs.
• Implement monitoring, logging, alerting, audit controls, and data-quality checks.
• Build reusable Python, PySpark, and SQL components, and CI/CD pipelines for data workloads.
• Partner with the platform engineering team to deploy and operate containerized PySpark workloads on Kubernetes/AKS, progressively taking ownership of Spark driver/executor configuration and resource allocation on Kubernetes.
• Review technical designs, mentor data engineers, and establish implementation standards across the team.

Required Skills

• 6+ years of hands-on data engineering experience.
• Strong experience with Microsoft Azure data platforms.
• Advanced Python, PySpark, and SQL skills.
• Strong hands-on experience with Apache Spark, Azure Data Lake Storage Gen2, Azure Event Hubs, and Azure DevOps/Git.
• Strong understanding of Spark drivers, executors, resource allocation, partitioning, caching, broadcast joins, shuffle optimization, and skew handling.
• Experience with batch, streaming, ETL, ELT, and event-driven processing patterns.
• Experience implementing Medallion Architecture.
• Experience with REST APIs, SFTP, JSON, CSV, Parquet, Delta Lake, and relational databases.
• Experience with CDC, incremental processing, schema enforcement, and schema evolution.
• Strong understanding of data modelling, schema design, partitioning, and storage optimization.
• Experience implementing pipeline observability, data validation, monitoring, alerting, and error recovery.

Preferred Skills (Nice to Have — Training Provided)

• Hands-on experience deploying or operating Apache Spark workloads on Kubernetes or Azure Kubernetes Service (AKS).
• Experience with Docker and containerized application deployment.
• Familiarity with Kubernetes Jobs, CronJobs, ConfigMaps, Secrets, resource limits, node pools, and autoscaling.