Sankalp

Data Engineer with 1.5+ years of experience designing and deploying scalable ETL pipelines, cloud-native data lakes, and distributed data processing systems on AWS. A proven track record of automating data workflows, improving pipeline reliability, and delivering analytics-ready datasets that drive business decisions. Hands-on expertise in PySpark, Python, SQL, and modern data orchestration. Seeking to build and scale high-impact data infrastructure at a world-class technology organization.

Jr. Technical Consultant — Data Engineer at F1 IT InfoTech (2024-07 – 2025-12)

Data engineer responsible for migrating production data and implementing validation frameworks

• Supported migration of 32TB of production data from Vertica to Databricks, ingesting from sources including Salesforce and MySQL, improving processing speed and platform scalability.
• Designed and implemented a chunk-based migration strategy — splitting historical data month-by-month via a Vertica → S3 → Databricks pipeline to handle memory constraints of 60 cores and 128GB RAM.
• Built a multi-layer data validation framework including row count checks, day-wise and month-wise record comparisons, and SHA-256 hash-based validation to detect row-level mismatches — results stored in Delta tables for ETL and business teams.
• Refactored SQL scripts by converting UPDATE statements to MERGE operations and adapting functions to Databricks syntax, ensuring compatibility across pre-prod and prod environments.
• Optimized JDBC batch sizes and scheduling when available connections dropped from 10 to 2, maintaining extraction throughput without pipeline failures.

Big Data Trainee at USM's Shriram Mantri Vidyanidhi Info Tech Academy (VITA) (2023-09 – 2024-02)

Big Data trainee architecting and deploying cloud ETL pipelines and leading cross-functional teams

• Architected and deployed end-to-end cloud ETL pipelines on AWS using Glue and Lambda to process 23GB of data, enabling real-time data ingestion and accelerating reporting turnaround for business stakeholders.
• Led a cross-functional team of 8 engineers in delivering a production-grade Big Data solution — managed sprint planning, code reviews, and on-time delivery.
• Established a CI/CD deployment pipeline using GitHub Actions, reducing release time and ensuring zero-downtime deployments for the project.

PG Diploma in Big Data Analytics – CDAC, Mumbai (2023-01 – 2024-12)

B.Tech. in Computer Science & Engineering – Dr. Ram Manohar Lohia Avadh University (2019-01 – 2023-12)