Job Overview :
We are seeking an experienced Senior Data Engineer to join our dynamic team and drive our data engineering initiatives.
The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimizing data workflows, and enabling data-driven decision making across the organization.
Key Responsibilities :
- Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services.
- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark.
- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation.
- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes.
- Create and maintain data models, schemas, and documentation to ensure data quality and consistency.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.
- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed.
- Ensure data security, privacy, and compliance with industry standards and regulations.
- Mentor junior team members and provide technical guidance on data engineering best practices.
- Stay current with emerging technologies and industry trends in data engineering and cloud computing.
- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables.
- Design and implement data governance frameworks and data quality monitoring solutions.
Ace Recruit