Bharadwaz

Senior Data Engineer with 5 years of experience designing and delivering scalable, cloud-native data platforms using Azure Databricks, Pyspark, and Delta Lake. Proven expertise in building metadata-driven frameworks, modernizing large-scale pipelines, and enabling high-performance analytics for enterprise systems.

Big Data Engineer at Deloitte (2021-03 – Present)

• Led and owned the design and implementation of a reusable, metadata-driven Azure Databricks ingestion framework, built using Pyspark and Delta Lake (Unity Catalog), enabling automated full and incremental data loads from BigQuery and downstream integrations with Teradata and SQL Server, scaling to 85+ tables with audit, control, restartability and data validation.
• Led end-to-end modernization of a large-scale data processing pipeline using Pyspark, enabling dynamic segmentation across 20M+ records and 90+ categories, reducing processing time from 10+ days to under 2 hours, and significantly improving downstream analytics and data-driven decision-making.
• Designed and implemented highly scalable data pipelines in Azure Databricks, efficiently processing 18M+ records and 45–50 GB of nested JSON data using Delta Lake architecture to deliver optimized performance for large-scale workloads.
• Developed an enterprise-wide modernization assessment framework to categorize and analyze legacy scripts, enabling accurate effort estimation and strategic planning for large-scale migration from on-prem systems to cloud platforms.
• Built a robust Pyspark and Spark SQL–based processing framework, incorporating advanced optimization techniques such as partitioning, compaction, archival, purge and Delta Lake lifecycle management to improve performance and storage efficiency.
• Automated end-to-end workflows using shell scripting and Databricks orchestration, integrating data across Teradata, Delta Lake, and SQL Server while ensuring operational reliability and scalability.
• Recognized with multiple Applause and Outstanding Awards for successful enterprise go-lives, delivering high-impact data solutions while driving architecture, solution design and technical excellence across projects.

Big Data Intern at Digital Lync (2019-05 – 2019-06)

• Built data ingestion pipelines between relational databases and HDFS and developed Tableau dashboards to support business analytics and reporting.

Bachelor of Technology in Information Technology – G. V. P. College of Engineering (A) (2016-08 – 2020-09)