Azure Data Engineer - Python/Databricks

Azure Data Engineer

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).
• Build and optimize data processing workflows using Azure Databricks and Python.
• Develop ETL/ELT processes to extract, transform, and load data from multiple sources.
• Work with structured and unstructured data to support analytics and reporting requirements.
• Implement data transformation, cleansing, validation, and integration processes.
• Optimize data pipelines and Spark jobs for performance, scalability, and cost efficiency.
• Integrate data from various sources into Azure-based data platforms.
• Ensure data quality, consistency, reliability, and security across data workflows.
• Monitor and troubleshoot data pipelines, jobs, and production issues.
• Collaborate with data analysts, data scientists, architects, and business stakeholders to understand data requirements.
• Follow best practices for data governance, access management, documentation, and deployment.
• Support automated deployment and version control processes for data engineering solutions.

Required Skills and Qualifications:

• 5 - 10 years of relevant experience in data engineering or related technical roles.
• Strong hands-on experience with Microsoft Azure data services.
• Proficiency in Python for data processing and automation.
• Hands-on experience with Azure Data Factory (ADF) for building and managing data pipelines.
• Strong experience with Azure Databricks for data transformation and processing.
• Understanding of ETL/ELT concepts, data integration, and data warehousing.
• Knowledge of SQL and database concepts.
• Experience with data pipeline monitoring, troubleshooting, and performance optimization.
• Strong analytical, problem-solving, and communication skills.