Azure Databricks Engineer - Python/PySpark

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize large-scale data processing solutions using Python and Spark.
• Develop robust ETL/ELT pipelines for batch and near-real-time data processing.
• Work with Azure Data Lake Storage (ADLS) and other Azure data services.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Optimize Spark jobs, Databricks notebooks, clusters, and SQL queries for performance and cost efficiency.
• Develop reusable and maintainable data engineering frameworks and components.
• Implement data quality, governance, security, and monitoring practices.
• Work with structured and unstructured datasets across multiple data sources.
• Collaborate with solution architects and engineering teams to translate business requirements into technical solutions.
• Troubleshoot production issues and perform root-cause analysis.
• Implement CI/CD processes for Databricks code and data pipelines.
• Follow engineering best practices around version control, testing, documentation, and deployment.
• Mentor junior and mid-level data engineers and contribute to technical decision-making.

Required Skills &

Experience :

• 8 years of experience in Data Engineering, Big Data, or related technologies.
• Strong hands-on experience with Azure Databricks.
• Excellent knowledge of PySpark and Apache Spark.
• Strong programming experience in Python.
• Strong experience developing ETL/ELT data pipelines.
• Hands-on experience with Azure Data Lake Storage (ADLS Gen2).
• Good knowledge of Azure data services and cloud architecture.
• Strong SQL skills and experience working with relational and analytical databases.
• Experience with Delta Lake and Delta Tables.
• Strong understanding of data modeling, data warehousing, and distributed data processing.
• Experience with performance tuning and optimization of Spark/Databricks workloads.