Data Engineer
Technology
Today
Full timeFully remote
Job description
Programmers.io is currently looking for an "Azure Data Engineer"
Remote Role
.Full Time Role with benefits
Role-
Key Responsibilities
- Develop scalable ETL/ELT pipelines using Azure Data Factory.
- Build data processing and transformation solutions using PySpark and Python.
- Design and implement data lake solutions using ADLS Gen2 and Delta Lake.
- Develop optimized Spark jobs for large-volume data processing.
- Implement data quality, validation, logging, monitoring, and error-handling frameworks.
- Troubleshoot and optimize data pipelines for reliability and performance.
- Work with business, analytics, BI, and architecture teams to deliver end-to-end data solutions.
- Follow development standards, Git workflows, code reviews, and CI/CD practices.
Mandatory Skills
- 6+ years of Data Engineering experience.
- Strong hands-on PySpark development experience.
- Strong experience with Azure Data Factory (ADF).
- Hands-on experience with Delta Lake and ADLS Gen2.
- Strong Python and SQL programming skills.
- Experience with ETL/ELT, data ingestion, transformation, and performance optimization.
- Good understanding of Azure-based data engineering architecture.
Good to Have
- Azure Databricks
- Apache Spark
- Azure DevOps / CI/CD
- Medallion architecture
- Power BI / Snowflake / Microsoft Fabric exposure
Keywords
Data EngineeringPySparkAzure Data FactoryDelta LakeAzure Data Lake Storage Gen2PythonSQLETL/ELTData IngestionData TransformationApache SparkAzure DatabricksAzure DevOpsCI/CDMedallion ArchitecturePerformance OptimizationADLS Gen2Spark JobsData QualityData Validation
Interested in this role?