Siddharth Chatterjee

Data Engineer with 4 years of experience designing and owning scalable data pipelines across large-scale batch and near real-time data processing systems at Deloitte USI. Experienced in building distributed data solutions using Spark and Google Cloud Platform (BigQuery, GCS), with a strong focus on performance optimization, reliability, and production-grade system design. Demonstrated impact in data migrations, pipeline optimization, and enabling reliable analytics for high-volume data systems.

Consultant — Data Engineer 2 at Deloitte USI (2022-08 – Present)

Promoted within 2 years for strong ownership and delivery impact. Owned end-to-end data pipelines for large-scale financial datasets (accounts, transactions, reporting data; multi-TB scale), including ingestion, transformation, validation, and publishing layers, ensuring SLA compliance and system reliability.

• Led migration of 25+ Hive tables (7 TB) to BigQuery on GCP, redesigning storage formats and ingestion workflows, improving query performance and simplifying downstream analytics.
• Designed reusable ingestion frameworks supporting multiple data formats (Avro, JSON, Parquet), reducing onboarding time by 40%.
• Built multi-stage ETL pipelines (ingestion → transformation → publishing) processing multi-terabyte datasets across scheduled workflows.
• Optimized distributed PySpark pipelines through partition tuning, caching, and execution planning, reducing runtime by up to 60%.
• Addressed Hadoop small file challenges via compaction strategies, improving storage efficiency and cluster stability.
• Contributed to near real-time data pipelines using Apache Kafka and Spark Structured Streaming, working with event ingestion and fault-tolerant processing.
• Orchestrated workflows using Autosys, managing dependencies across pipeline stages.
• Built data validation frameworks with automated quality checks, improving pipeline reliability.
• Monitored pipelines using Acceldata, ensuring data freshness and SLA compliance.
• Enabled onboarding of 650,000+ financial accounts into analytics systems, supporting reporting and operational decision-making across financial datasets.
• Developed dbt workflows to standardize transformation logic and improve maintainability.
• Led debugging and resolution of production pipeline failures, ensuring minimal downtime.

B.Tech in Electronics & Computer Engineering – VIT Chennai (2018 – 2022)