Data Engineer - ETL/Python

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Build robust ETL/ELT workflows for batch and real-time data processing.
• Develop data transformations using PySpark, Python, and SQL.
• Design and implement data lake/lakehouse and data warehouse solutions.
• Integrate data from multiple structured and unstructured sources.
• Optimize data pipelines, queries, and Spark workloads for performance and cost.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Work with cloud-based data storage and Azure data services.
• Collaborate with architects, data scientists, analysts, and application teams.
• Follow engineering best practices around CI/CD, security, governance, and documentation.

Required Skills :

• 8 to 15 years of experience in data engineering.
• Strong hands-on experience with Azure Databricks.
• Strong hands-on experience with Azure Data Factory (ADF).
• Excellent PySpark, Python, and SQL skills.
• Strong understanding of ETL/ELT concepts and data pipeline architecture.
• Experience with Data Lake / Lakehouse architecture.
• Experience with data modeling and performance optimization.
• Good understanding of Microsoft Azure data services.
• Experience working in enterprise-scale data environments.