Santhosh Kumar
Azure Data Engineer
Talent
ChennaiMember since April 21, 2026
PythonSQLScala SparkSpark/PySparkKafkaDatabricksAzure Data FactoryTerraformCI/CDAzure DevOpsDatabricks Asset BundlesGitADLSPowerBIDelta LakeAuto LoaderAutosysAirflowPoetryAZSQLDjango
·
Hire this person
Send a job offer directly to this candidate
PythonSQLScala SparkSpark/PySparkKafka
About
Data Engineer with 4+ years of experience in designing and implementing batch/real-time data solutions on Azure cloud platform. Seeking a Data Engineer role that allows me to learn new technologies and enhance my problem-solving abilities.
Experience
Azure Data Engineer at KamerAI (2025-06 – Present)
Azure Data Engineer at Colan Infotech (2022-03 – 2025-05)
L3 Data Engineer at PepsiCo (2025-06 – Present)
L3 Data Engineer at PepsiCo
- Converted a high-frequency CSV ingestion ADF pipeline into a Spark Structured Streaming workflow using Auto Loader, which resulted in $10,593.6 monthly savings in Databricks and VM costs.
- Replaced the traditional watermark-based ingestion logic with Change Data Feed (CDF) incremental loading, enabling the pipeline to process only recently changed data instead of scanning entire terabyte-scale source tables, resulting in $4,401.16 monthly cost savings.
- Reduced the runtime of a legacy Databricks job from 4.5 hours to 2.2 hours without increasing infrastructure expenses, which allows the pipeline to refresh 10-11 times a day.
- Handled high-priority production incidents and re-written/optimized over 6 legacy applications, resulting in cumulative monthly cost savings of $11,649.19 and worked alongside a team of 16 members, providing support to L1 and L2 teams in addressing production issues.
Senior Data Engineer at SES Satellites (2022-03 – 2025-05)
Senior Data Engineer at SES Satellites
- Designed and implemented an end-to-end project lifecycle for Spark Structured Streaming jobs across multiple environments.
- Reduced Databricks DBU and Azure VM costs by reconfiguring and reorganizing job clusters and streaming jobs, which results in savings of > $93,000 a year without impacting job performance.
- Migrated the entire project from Scala to Python/PySpark to unify the codebase, Utilized Databricks Asset Bundles (DABs) for redesigning with best practices.
- Developed a custom Spark Streaming Listener function to log streaming metrics into Log Analytics Workspace.
Education
Bachelor of Computer Applications in Computer Applications – Thiruvalluvar University (2018-06 – 2021-03)
Skills
Azure Data Engineer

