Senior Data Engineer

About the Role

Build and optimize Databricks pipelines (PySpark, SQL, Delta Lake) that transform Raw data into curated and business-ready layers. Own production deployments through Azure DevOps (ADO), including code reviews, testing, and releases; track work in ADO Boards. Monitor, troubleshoot, and tune jobs for performance, reliability, and cost.

Implement data quality checks and governance using Unity Catalog. Partner with business users and the Platform team to turn requirements into technical solutions. Maintain clear documentation and support runbooks.

Responsibilities

• Build and optimize Databricks pipelines (PySpark, SQL, Delta Lake) that transform Raw data into curated and business-ready layers.
• Own production deployments through Azure DevOps (ADO), including code reviews, testing, and releases; track work in ADO Boards.
• Monitor, troubleshoot, and tune jobs for performance, reliability, and cost.
• Implement data quality checks and governance using Unity Catalog.
• Partner with business users and the Platform team to turn requirements into technical solutions.
• Maintain clear documentation and support runbooks.

Required Skills

• Strong hands-on experience with Databricks (notebooks, jobs, workflows, clusters).
• Proficiency in PySpark, Python, and advanced SQL.
• Solid understanding of Delta Lake, data modeling, and ETL/ELT patterns.
• Experience supporting production workloads, with strong debugging and data validation skills.
• Working knowledge of Git and CI/CD, ideally Azure DevOps (Repos, Pipelines, Boards).