Pavan Kumar Vakalapudi

Data Engineer with 4+ years of experience building scalable data pipelines using Python, SQL, PySpark, Databricks and Azure. Experienced in building production grade data pipelines, performance tuning and delivering reliable data solutions.

Data Engineer at Fractal Analytics (2024-04 – Present)

• Designed and implemented an end-to-end ETL pipeline using Azure Data Factory for orchestration, Azure Databricks for spark based transformations and Azure Data Lake Storage Gen2 for data storage.
• Led performance optimization of data pipelines, cutting runtime by 40% and saving $12000 annually in cloud computing costs.
• Built incremental load patterns using watermark columns and Delta merges on Databricks, reducing daily ingestion windows from 4+ hours to under 45 minutes.
• Engaged with 5+ business stakeholders to gather and clarify project requirements which helped in improving team efficiency and performance.
• Managed five concurrent projects, debugging and resolving issues reported by business stakeholders.

Data Engineer at Cognizant (2022-12 – 2024-04)

• Implemented a reusable code that automates data validation between two tables, which helped in reducing the manual effort by 50 hours.
• Switched scheduled pipelines from all purpose to job clusters, reducing compute costs by 50-70% per run and saving around $3000 annually.
• Optimized Databricks pipeline execution by implementing multi-task workflows and replacing Azure Data Factory individual notebook activities, reducing compute costs and improving overall pipeline efficiency.

Analyst Intern at Cognizant (2022-03 – 2022-08)

• Actively pursued training opportunities in Spark to broaden my knowledge and stay updated with the latest advancements in big data analytics.

Bachelors of Technology in Computer Science and Engineering – GMR Institute of Technology (2018-07 – 2022-05)