Skip to main content
Pavan Kumar Vakalapudi

Pavan Kumar Vakalapudi

Data Engineer | 4 Years Experience | Azure Databricks, PySpark, SQL, ADF, Data Modelling | Building Scalable ETL/ELT Pipelines

Talent
BengaluruMember since August 27, 2026
Data EngineerAzure DatabricksPySparkSQLADFData ModellingBuilding Scalable ETL/ELT PipelinesApache SparkPythonEtl
·

Hire this person

Send a job offer directly to this candidate

Data EngineerAzure DatabricksPySparkSQLADF

About

Data Engineer with 4+ years of experience building scalable data pipelines using Python, SQL, PySpark, Databricks and Azure. Experienced in building production grade data pipelines, performance tuning and delivering reliable data solutions.

Experience

Data Engineer at Fractal Analytics (2024-04 – Present)

  • Designed and implemented an end-to-end ETL pipeline using Azure Data Factory for orchestration, Azure Databricks for spark based transformations and Azure Data Lake Storage Gen2 for data storage.
  • Led performance optimization of data pipelines, cutting runtime by 40% and saving $12000 annually in cloud computing costs.
  • Built incremental load patterns using watermark columns and Delta merges on Databricks, reducing daily ingestion windows from 4+ hours to under 45 minutes.
  • Engaged with 5+ business stakeholders to gather and clarify project requirements which helped in improving team efficiency and performance.
  • Managed five concurrent projects, debugging and resolving issues reported by business stakeholders.

Data Engineer at Cognizant (2022-12 – 2024-04)

  • Implemented a reusable code that automates data validation between two tables, which helped in reducing the manual effort by 50 hours.
  • Switched scheduled pipelines from all purpose to job clusters, reducing compute costs by 50-70% per run and saving around $3000 annually.
  • Optimized Databricks pipeline execution by implementing multi-task workflows and replacing Azure Data Factory individual notebook activities, reducing compute costs and improving overall pipeline efficiency.

Analyst Intern at Cognizant (2022-03 – 2022-08)

  • Actively pursued training opportunities in Spark to broaden my knowledge and stay updated with the latest advancements in big data analytics.

Education

Bachelors of Technology in Computer Science and Engineering – GMR Institute of Technology (2018-07 – 2022-05)

Skills

Data Engineer4 Years ExperienceAzure DatabricksPySparkSQLADFData ModellingBuilding Scalable ETL/ELT Pipelines

Reviews

Related