Azure Data Engineer -- AL GDO - Application Development at Virtusa Consulting services Pvt Ltd (2024-05 – Present)
Designed, developed, and maintained end-to-end ETL/ELT pipelines processing large volumes of data on Azure Databricks using Medallion Architecture
- Designed, developed, and maintained 50+ end-to-end ETL/ELT pipelines that processes 100 - 200GB of data on Azure Databricks using Medallion Architecture (Raw, Stage, Gold, Mart Layers), achieving 30% improvement in processing efficiency
- Developed Ingestion jobs using Azure Data Factory to ingest on-premise data to cloud from 13+ countries and stored in ADLS Gen2 Raw layer
- Created Python notebooks to read the data from Raw Layer and merge similar datasets and store in stage layer as a Delta Table while partitioning on Country and Year columns
- Created CORE DATA MODELS(CDM) which consists of 50+ datasets as a Star Schema/Snowflake schema in gold layer
- Implemented SCD Type 1 and Type 2 for dimension tables to efficiently store Active + Inactive records
- Split CORE DATA MODELS(CDM) into 4 different data models for efficient processing which reduced the resource consumption and cost reduction by 40%
- Built Mart layer data models sourcing aggregated data from CDMs, optimizing for BI consumption by 5000+ Power BI report users across 13+ Countries
- Created views for all the datasets in Delta Lake in Azure Synapse Serverless SQL pool to refresh PBI semantic models
- Developed and deployed Power BI Premium datasets with optimized relationships and country-level row-level security (RLS). Automated the model deployments using Azure DevOps CI/CD
Azure Data Engineer -- AL GDO – Application Maintenance at Virtusa Consulting services Pvt Ltd (2023-10 – 2024-04)
Proactively monitored and maintained ADF pipelines with high SLA adherence, resolved incidents and implemented change requests
- Proactively monitored 30+ ADF pipelines using Azure Monitor Alerts Maintaining 99% SLA adherence across critical data workflows
- Resolved more than 100 service-now incidents – including pipeline failures, data reconciliation errors, and access issues within SLA time period
- Implemented 60+ change requests across ADF pipelines and Azure SQL configurations, ensuring zero unplanned downtime during change windows and maintaining ITSM compliance with ServiceNow workflows
Jr Data Engineer -- SHIFT-Finance Ops Controller at Virtusa Consulting services Pvt Ltd (2022-09 – 2023-08)
Architected and delivered ELT pipelines for financial data migration with significant efficiency improvements
- Partnered with Finance and BI teams to architect and deliver 15+ ELT pipelines in Azure Data Factory, migrating on-premise financial data to Azure SQL Database and cutting manual reporting effort by 6 hours/week
- Managed version control and release pipelines using Azure DevOps, enabling automated deployments across DEV/PPRD/PROD environments following branching strategies
- Designed and optimized Azure SQL Database Core Data Models (CDMs) using indexing strategies and query tuning, reducing average query execution time by 60% and supporting quick results in PowerApps
- Built event-driven and scheduled ADF pipeline triggers using tumbling window and event-based mechanisms, ensuring real-time ingestion SLAs for 5 critical Finance Ops datasets