Software Development Engineer - 1, Whiteklay Technologies Pvt Ltd – Pune July 2022 - Present
- Working on Star Health and Allied Insurance Co Ltd Project (Big Data Engineering Project)
- Developed robust data ingestion pipelines using Azure Synapse Analytics Pipelines and Copy Data Activity to extract and load batch data (Full Refresh, SCD0, SCD1) from diverse sources (Oracle, PostgreSQL, MySQL, SQL
Server, MongoDB, SFTP) into Azure Data Lake Storage Gen2 (ADLS Gen2).
- Implemented real-time data streaming using Azure Event Hub to capture and ingest Change Data Capture
(CDC) data into Azure Data Lake Storage Gen2 (ADLS Gen2)
- Developed comprehensive efficient data transformation ,reports and aggregates using PySpark/Scala Spark to meet specific client requirements and provide actionable insights
- Proactively resolved production pipeline failures, achieving a significant reduction from 76% to 98%,
minimizing downtime and ensuring continuous data availability.
- Engineered and implemented Delta Lake utilities for Repartitioning and Vacuuming, resulting in a 30% boost in query performance.
- Implemented Rollup/Reconciliation processes, enhancing data accuracy by 60% through the identification and resolution of source system discrepancies (timezone, late data, primary key conflicts).
- Executed the migration of Synapse Spark notebooks to Databricks Notebooks, delivering a 14% reduction in operational costs.
Intern, Eaton Technologies Pvt Ltd – Pune Jan 2022 - Jun 2022
- Developed Desktop Web Applications for internal company activities and management using Microsoft Power
Apps. Developed workflows through Microsoft Power Automate for seamless task automation
