Shivam Agarwal

Data Engineer with 3 years of experience designing and delivering production-grade ETL/ELT pipelines using Azure Databricks, PySpark, Delta Lake, and Azure Data Factory. Deep expertise in Apache Spark internals — DAG execution, Catalyst optimizer, Tungsten engine, and shuffle tuning. Hands-on experience building scalable data warehouse solutions using Medallion Architecture, dimensional data modeling, and star schema design.

Proven track record in Spark version migrations, incremental pipeline design, and performance optimization.

Data Engineer — Retail Analytics Platform at Persistent Systems (2024-04 – Present)

• Implemented ETL/ELT batch pipelines on Azure Databricks and PySpark, for retail sales, customer, and product datasets.
• Built Medallion Architecture pipeline (Bronze → Silver → Gold) with schema enforcement, idempotent incremental ingestion using file-based watermarks, and automated data quality validations.
• Designed Gold-layer data warehouse with star schema fact and dimension tables (including SCDs) using Spark SQL, enabling downstream BI consumption and analytical reporting.
• Improved end-to-end pipeline processing time by 30% through partitioning strategies, broadcast joins, caching, file compaction, and systematic Spark UI profiling.
• Implemented large-scale deduplication and data quality checks on raw ingests from ADLS Gen2, improving downstream data reliability and reducing reconciliation effort.

Data Engineer — Global Media Data Integration Platform at Persistent Systems (2023-10 – 2024-03)

• Contributed to a multi-country media ETL platform processing viewership and demographic data across 10+ regions, supporting audience analytics and reporting.
• Led migration from Python 2 → 3 and Spark 2 → 3; leveraged Adaptive Query Execution (AQE) and optimised join strategies to improve throughput and reduce shuffle overhead.
• Defined data contracts and enforced schema consistency across regional data sources, improving data reliability for downstream consumers.
• Collaborated with cross-functional teams to align pipeline outputs with business reporting requirements across multiple geographies.

Data Engineering Intern at Persistent Systems (2023-01 – 2023-05)

• Performed data cleaning, transformation, and exploratory data analysis on structured datasets using Python, SQL, and Pandas as part of production data pipeline preparation.
• Built foundational knowledge in distributed data processing, data structures, and algorithm complexity — applied directly in subsequent DE roles.

B.Tech in Computer Science & Engineering – Meghnad Saha Institute of Technology, Kolkata (2019 – 2023)