Skip to main content
Santhosh Kumar

Santhosh Kumar

Azure Data Engineer

Talent
ChennaiMember since April 21, 2026
PythonSQLScala SparkSpark/PySparkKafkaDatabricksAzure Data FactoryTerraformCI/CDAzure DevOpsDatabricks Asset BundlesGitADLSPowerBIDelta LakeAuto LoaderAutosysAirflowPoetryAZSQLDjango
·

Hire this person

Send a job offer directly to this candidate

PythonSQLScala SparkSpark/PySparkKafka

About

Data Engineer with 4+ years of experience in designing and implementing batch/real-time data solutions on Azure cloud platform. Seeking a Data Engineer role that allows me to learn new technologies and enhance my problem-solving abilities.

Experience

Azure Data Engineer at KamerAI (2025-06 – Present)

Azure Data Engineer at Colan Infotech (2022-03 – 2025-05)

L3 Data Engineer at PepsiCo (2025-06 – Present)

L3 Data Engineer at PepsiCo

  • Converted a high-frequency CSV ingestion ADF pipeline into a Spark Structured Streaming workflow using Auto Loader, which resulted in $10,593.6 monthly savings in Databricks and VM costs.
  • Replaced the traditional watermark-based ingestion logic with Change Data Feed (CDF) incremental loading, enabling the pipeline to process only recently changed data instead of scanning entire terabyte-scale source tables, resulting in $4,401.16 monthly cost savings.
  • Reduced the runtime of a legacy Databricks job from 4.5 hours to 2.2 hours without increasing infrastructure expenses, which allows the pipeline to refresh 10-11 times a day.
  • Handled high-priority production incidents and re-written/optimized over 6 legacy applications, resulting in cumulative monthly cost savings of $11,649.19 and worked alongside a team of 16 members, providing support to L1 and L2 teams in addressing production issues.

Senior Data Engineer at SES Satellites (2022-03 – 2025-05)

Senior Data Engineer at SES Satellites

  • Designed and implemented an end-to-end project lifecycle for Spark Structured Streaming jobs across multiple environments.
  • Reduced Databricks DBU and Azure VM costs by reconfiguring and reorganizing job clusters and streaming jobs, which results in savings of > $93,000 a year without impacting job performance.
  • Migrated the entire project from Scala to Python/PySpark to unify the codebase, Utilized Databricks Asset Bundles (DABs) for redesigning with best practices.
  • Developed a custom Spark Streaming Listener function to log streaming metrics into Log Analytics Workspace.

Education

Bachelor of Computer Applications in Computer Applications – Thiruvalluvar University (2018-06 – 2021-03)

Skills

Azure Data Engineer

Reviews

Related