Harshit Goel

Architected a Full Sample ETL pipeline processing 40M household records, increased data throughput by 10x while limiting operational cost reduction by 50% through resource efficiency. Managed an massive data ecosystem of 1.5 Petabytes with an ingestion rate of 100 TB per month, recorded at sub-minute frequency. Integrated an automated DQC layer using Python and SQL to validate respondent-level weights, trend stability, and deduplication.

Standardized data across National and Local reporting scales while driving a 30% reduction in production incidents

Data Engineer - 2 at NIELSEN (2024-09 – Present)

• Architected a Full Sample ETL pipeline processing 40M household records, increased data throughput by 10x while limiting operational cost reduction by 50% through resource efficiency.
• Managed an massive data ecosystem of 1.5 Petabytes with an ingestion rate of 100 TB per month, recorded at sub-minute frequency.
• Integrated an automated DQC layer using Python and SQL to validate respondent-level weights, trend stability, and deduplication. Standardized data across National and Local reporting scales while driving a 30% reduction in production incidents
• Optimized Airflow DAGs to enable 20+ simultaneous extracts, eliminating the need for additional cluster provisioning, reduced compute costs by 50% for historical data processing.
• Spearheaded cross-departmental alignment during the discovery phase, conducting deep-dive requirement analysis to define the project's architecture.

Data Engineer at NAGARRO (2022-01 – 2024-09)

• Reduced job run time by 40% after implementing caching strategies in Pyspark code for silver layer.
• Designed and implemented auditing, logging and error handling modules in the data pipeline, boosting the silver layer framework's efficiency by 25%.
• Engineered a comprehensive ETL pipeline with Dataiku, achieving a 45% acceleration in data processing and significantly improving data availability for analytics.
• Efficiently processed 10k daily records in medallion architecture and analyzed over 500GB of data using AWS Athena, deriving actionable insights that informed strategic decisions.
• Strategically optimized AWS resource allocation, employing cost-effective strategies that led to a 25% reduction in AWS expenses without compromising on performance.
• Conducted comprehensive volume testing on data pipelines with over 10 million records, enhancing job performance by 30% through targeted optimizations and efficiency improvements.
• Attained data analysis on 5 million data and built an impactful business analysis dashboard using Power BI for a Diamond processing giant which increased data transparency by 40%.

Bachelor of Technology in Computer Science – DR. A.P.J ABDUL KALAM TECHNICAL UNIVERSITY, Ghaziabad (UP) (2018-08 – 2022-06)