Dhruv Arora

Data Engineer with 2.5+ years of experience building and scaling production-grade ETL pipelines across on-prem and cloud environments. Developed and maintained 25+ data pipelines processing 2.3+ TB daily, enabling reliable data availability for business reporting and analytics. Improved pipeline performance by 30–40%, reducing data latency from 4+ hours to under 2.5 hours, which directly enabled faster daily reporting and reduced decision-making lag for business teams.

Designed and implemented a metadata-driven ETL framework to standardize ingestion from multiple Oracle sources into S3, reducing onboarding time for new pipelines by 50%. Implemented data quality validation and logging frameworks, reducing data inconsistencies by 60%, significantly improving trust in dashboards used by stakeholders. Experienced in SQL, Python, PySpark, and distributed systems, with hands-on exposure to AWS, Azure, Airflow, Talend, and Databricks.

Data Engineer (Associate Consultant) at KPMG India (2026-04 – Present)

• Unified Python Framework For Shared Drive Ingestion: Architected and executed the migration of 250+ legacy Talend ETL jobs into a unified, metadata-driven Python ingestion framework.
• Eliminated operational chaos by centralizing execution and automating credential management — reducing credential rotation effort from 10 days to 5 minutes.
• Accelerated onboarding SLA for new ingestion pipelines from 3 days to 6 hours, enabling rapid business responsiveness.
• Designed vendor-agnostic architecture ensuring flexibility and reversibility — minimal effort required even if cloud provider changes, compared to Talend's rigid setup.
• Implemented centralized logging and monitoring, providing full traceability of ingestion runs and execution IDs, improving audit readiness and stakeholder confidence.
• Drastically reduced maintenance overhead by 20x and decreasing MTTR by 100% and projected 60% cost savings over Talend ETL.
• Developed end-to-end data pipelines following Medallion Architecture using Delta Lake and AWS S3, enabling structured data flow from raw to business-ready layers (Bronze, Silver, Gold) for analytics and data science modeling purposes.
• Implemented Delta Lake on Databricks with optimized data storage and query performance through partitioning, Z-Ordering in delta lake tables
• Created and deployed Spark Databricks Notebooks on Databricks clusters for large-scale data processing and ETL workflows.
• Optimized Spark configurations (executor memory, cores, parallelism) inside databricks cluster to improve job performance and reduce cluster cost.
• Project: Data Migration from Hadoop to Cloud (Databricks & Snowflake): Migrated Spark ETL jobs from Hadoop HDFS to Cloud Databricks to Snowflake, setting up infrastructure for Databricks and Snowflake.
• Scheduled ETL workflows using PySpark on Databricks and orchestrated them using open-source Apache Airflow.
• Built scalable ETL pipelines to extract, transform, and load data from various sources like SQL Server, Excel, and flat files into Snowflake using Spark on Databricks.
• Developed an ETL pipeline using Databricks notebooks to extract data from relational databases using Spark JDBC connectivity. Utilized Spark for data transformation, including data cleaning, aggregation, and normalization. Loaded the transformed data into Snowflake using the Spark Snowflake connector, enabling efficient querying and reporting.
• Implemented Spark performance tuning techniques like broadcast joins, salting, caching, and persistence, reducing job execution time from 4 hours to 2.4 hours
• Optimized Spark ETL workflows by applying broadcast joins to efficiently merge large financial transaction tables with smaller reference/master datasets, reducing shuffle operations and improving query performance by 40%.

Data Engineer (Analyst) at KPMG India (2024-01 – 2026-03)

• SAP Data Migration Pipeline: Designed and implemented a config-driven data ingestion framework to onboard SAP data (CSV-based extracts) into raw and curated layers (Parquet)
• Built incremental processing logic without native CDC using md5 checksum and merge-based upserts to handle late-arriving and duplicate data
• Developed a robust ingestion layer with file validation, schema checks, and quarantine handling for corrupt or incomplete files
• Ensured idempotent data pipelines by implementing file tracking and merge strategies, preventing duplicate processing and ensuring data consistency
• Implemented data validation and reconciliation checks between Oracle and SAP datasets to support a safe and controlled migration
• Enabled config-driven cutover and rollback strategy using metadata (DynamoDB), minimizing downstream disruption across 50+ dependent pipelines
• On Demand Data Pipeline: Architected a scalable, event-driven data platform for real-time ingestion and processing of compressed data streams
• Leveraged Amazon SQS with Lambda triggers to achieve asynchronous, decoupled, and resilient processing
• Designed stateful orchestration using AWS Step Functions, enabling retry logic, failure handling, and workflow visibility
• Built idempotent processing mechanisms using process_id to avoid duplicate data ingestion
• Developed schema-aware parsing logic for handling diverse file formats within nested gzip structures
• Implemented observability layer using metadata tables and CloudWatch for end-to-end pipeline monitoring
• Unified Framework for Talend ETL Jobs: Architected a scalable metadata layer unifying ETL configurations across multiple sources, eliminating duplication and reducing operational overhead by 30–40%
• Enabled parallel job orchestration using execution grouping, significantly improving resource utilization and reducing batch processing latency by 30%
• Developed a centralized observability framework for ETL pipelines, improving failure detection, auditability, and reducing mean time to resolution (MTTR) by 50%

Bachelor of Engineering – Delhi Technological University (2019 – 2023)