Sumanth kumar Vanam

Azure Data Engineer with 5 years of experience in designing, developing, and optimizing scalable ETL/ELT pipelines and Lakehouse solutions using Azure Data Factory, ADLS Gen2, Azure Databricks, PySpark, Spark SQL, Delta Lake, and Azure Synapse Analytics. Experienced in building data pipelines for large-scale data processing and implementing Bronze-Silver-Gold Medallion Architecture. Hands-on experience with batch and micro-batch processing, incremental loading, CDC, watermarking, schema evolution, data quality, SCD Type 1/2, dimensional modeling, and Spark performance optimization.

Skilled in Azure DevOps, Git, CI/CD, production support, troubleshooting, root-cause analysis, and Agile/Scrum methodologies.

Data Engineer - Wipro Technologies - Bangalore, India

(2025-07)

Project: Retail Lakehouse Analytics Platform | Client: Sainsbury’s

• Developed scalable batch and micro-batch ingestion pipelines using Azure Data Factory, Databricks Auto Loader, and Event Hubs to ingest ~1.5 TB+ daily POS, e-commerce, inventory, and loyalty data across 10+ sources into ADLS Gen2.
• Implemented Bronze-Silver-Gold Lakehouse architecture using Delta Lake enabling ACID transactions, versioning, auditability, and time-travel capabilities.
• Built PySpark transformation workflows for data cleansing, deduplication, Standardization, joins and aggregation workflows with incremental loading, watermarking, and schema evolution handling.
• Optimized Spark and Delta workloads using partitioning, broadcast joins, Z-Ordering, OPTIMIZE, VACUUM, and small-file optimization techniques.
• Designed curated Gold-layer datasets and Star Schema models with fact and dimensional tables to support Azure Synapse analytics for sales, demand forecasting, and customer trend reporting.
• Implemented CI/CD using Azure DevOps for ADF pipelines and Databricks notebooks, with pipeline monitoring, logging, alerting, and automated job scheduling.
• Collaborated with business stakeholders on data requirements, data models, data quality, governance, documentation, architecture diagrams, and operational runbooks.

Associate Data Engineer - Capgemini Technology Services India Limited - Pune, India

(2021-10 - 2025-06)

Project: Deutsche Bank Data Integration & Analytics (DBDI) | Client: Deutsche Bank

• Designed, developed, and maintained enterprise ETL/ELT pipelines using Azure Data Factory, ADLS Gen2, Azure Synapse, and Azure Databricks.
• Migrated legacy Hive scripts to PySpark transformations on Databricks and validated data processing through notebook-based development and testing.
• Developed Bronze-to-Silver PySpark transformations to standardize and integrate data from multiple sources into common analytical structures.
• Supported migration from batch processing to micro-batch processing to enable near-real-time data availability for analytics.
• Implemented and enhanced PySpark-based data quality and validation checks across Silver-layer datasets to improve data consistency.
• Resolved Tier-2/Tier-3 production pipeline issues within SLA timelines and documented root causes and resolutions.
• Reduced manual touch points through metadata-driven orchestration and automated workflows.
• Developed ADLS file-compaction scripts to address small-file fragmentation and improve storage efficiency.
• Optimized Spark SQL, DataFrame transformations, joins, and partition strategies to improve processing efficiency and reduce memory-related issues.
• Participated in CI/CD activities and knowledge-sharing sessions, supporting documentation and team enablement.

Bachelor of Technology (B.Tech) - Electronics & Communication Engineering - Sreenivasa Institute of Technology and Management Studies, Chittoor, Andhra Pradesh