Atharva Chaudhari

Azure Data Engineer with ~4 years of experience building scalable, cloud-native data platforms on Azure. Focused on end-to-end ETL/ELT pipelines structured on Medallion Architecture (Bronze → Silver → Gold), along with data warehouses and multi-vendor reconciliation frameworks using Azure Synapse Analytics, Azure Data Factory, Delta Lake, PySpark, Microsoft Fabric, and Azure Databricks. Pipelines regularly process millions of rows per run — with a consistent 20–30% improvement in execution time after optimization.

Experienced with CI/CD automation on Azure DevOps and hands-on delivery across the full data engineering lifecycle.

Azure Data Engineer at Kanini Software Solutions (2022-03 – Present)

• Architected end-to-end Azure Synapse Analytics pipelines (full & incremental loads) using Linked Services, Datasets, and ADF orchestration to ingest data from Azure SQL, Blob Storage, and ADLS Gen2 — processing millions of rows per execution.
• Designed and implemented Medallion Architecture (Bronze → Silver → Gold) on ADLS Gen2 with Delta Lake, organizing raw ingestion, cleansed transformation, and curated analytics layers to improve data reliability and reusability across pipelines.
• Developed PySpark transformation workflows in Synapse Spark Notebooks to reconcile multi-vendor investment market data with client datasets, improving data accuracy and automating discrepancy detection.
• Built a centralized enterprise data warehouse using Azure Dedicated SQL Pools and designed client-specific data marts, enabling faster self-service analytics and reporting.
• Implemented Mapping Data Flows in ADF to migrate and transform data from ADLS Gen2 to SQL Server with complex multi-step transformation logic.
• Configured Azure Integration Runtime to handle secure, high-throughput data movement between ADLS and Dedicated SQL Pool.
• Implemented CI/CD pipelines via Azure DevOps to automate deployment of ADF and Synapse artifacts across Dev, Test, and Production environments.
• Optimized ETL pipelines by tuning Spark configurations and refactoring transformation logic, achieving a 20–30% reduction in pipeline execution time.
• Handled large-scale datasets in Parquet, CSV, JSON, and XML; performed root cause analysis for data quality issues, schema drift, and pipeline failures.
• Ran POC evaluations of Microsoft Fabric (OneLake, lakehouse architecture) and Azure Databricks (Delta Live Tables, Unity Catalog) to determine which platform best fits the team's next phase — findings shared as formal migration recommendations.

B.E. in Mechanical Engineering – Savitribai Phule Pune University (2021)