Lead Data Engineer - Python/Azure Databricks

Experience: 10 Years

Location: Kochi / Coimbatore / Chennai / Mumbai / Pune

LinkedIn Profile: Mandatory

About the Role:

We are looking for an experienced Lead Data Engineer to design, develop, and support enterprise-grade data platforms and pipelines. The ideal candidate should have strong expertise in Azure Data Engineering, Databricks/Microsoft Fabric, PySpark, SQL, data migration, and modern data architecture, along with hands-on experience in production and L2/L3 support.

Key Responsibilities:

• Design and develop scalable enterprise data pipelines using Azure data technologies.
• Build and optimize data solutions using Azure Data Factory, Azure Databricks/Microsoft Fabric, PySpark, and Delta Lake.
• Implement Medallion Architecture and modern data engineering best practices.
• Lead data migration initiatives and ensure data quality, reliability, and performance.
• Develop effective data models and optimize complex SQL queries and data workloads.
• Perform performance tuning, job monitoring, troubleshooting, and failure recovery.
• Provide L2/L3 production support, including RCA, incident resolution, and job failure troubleshooting.
• Participate in on-call support and Hypercare activities during critical releases and migrations.
• Ensure data governance, security, documentation, and compliance standards.
• Mentor data engineers and provide technical guidance across projects.

Mandatory Skills:

• Advanced SQL
• Azure Data Factory (ADF)
• Azure Databricks / Microsoft Fabric
• PySpark
• Delta Lake
• Medallion Architecture
• Data Migration
• Data Modeling
• Production Support / L2-L3 Support
• RCA & Job Failure Troubleshooting
• On-call Support & Hypercare

Qualification:

• Bachelors or Masters degree in Computer Science, Information Technology, Engineering, or a related field.

Ideal Candidate:

• 10 years of experience in Data Engineering with strong Azure expertise.
• Proven experience handling enterprise-scale data platforms and production environments.
• Strong problem-solving, troubleshooting, communication, and mentoring skills.