Anoop Singh

Results-driven Data Engineer with 3 + years of experience designing and deploying scalable ETL / ELT pipelines, Delta Lake architectures, and real-time data ingestion frameworks on Azure and AWS. Proficient in PySpark, SQL, Databricks, Azure Data Factory, and Apache Spark, with deep expertise in data modeling, SCD implementations, upsert/merge strategies, and performance tuning. Reduced ETL processing times by 40 %, migrated 20 TB+ of legacy data to cloud data lakes, and improved data availability by 35 %.

Triple

Databricks certified, with a track record of delivering high-quality, compliant data solutions across enterprise environments.

Data Engineer - PIB Group India - Gandhinagar, Gujarat

(2025-07)

• Architected and deployed end-to-end ELT pipelines on Databricks using PySpark and SQL to ingest, transform, and deliver 50 M – 150 M records / day from heterogeneous sources into Delta Lake, ensuring schema evolution and data consistency.
• Implemented Delta Lake MERGE operations (upserts) and SCD Type 2 patterns for slowly-changing dimension management, maintaining full historical lineage.
• Optimised Spark execution plans using Spark UI, broadcast joins, partition pruning, and adaptive query execution (AQE), reducing mean time to resolution for production failures by 50 %.
• Built a library of reusable PySpark utility modules and pipeline templates (ingestion, transformation, audit), cutting new pipeline development effort by 30 %.
• Automated ETL audit logging and alerting system using Python and email/webhook notifications, enabling proactive anomaly detection and SLA-breach prevention.
• Designed Delta Lake table layouts with strategic partitioning, Z-ordering, OPTIMIZE, and VACUUM, improving downstream query performance by 2 – 3 x.

Data Engineer - Digivate Labs - Gurugram, Haryana

(2023-10 - 2025-06)

• Engineered scalable ELT pipelines using PySpark, SQL, and Azure Data Factory, integrating 15+ heterogeneous data sources (REST APIs, RDBMS, flat files, event streams) and improving data availability for analytics teams by 35%.
• Reduced ETL processing time by 40% through Spark performance tuning—including partition optimisation, broadcast joins, window functions, predicate pushdown, and caching strategies.
• Designed and implemented SCD Type 1 & 2 patterns and MERGE/upsert logic in Databricks Delta Lake, ensuring accurate historical tracking for enterprise dimensional models.
• Led migration of 20 TB+ legacy on-premise data to cloud data lakes (ADLS Gen2, S3), restructuring storage layers (Bronze/Silver/Gold medallion architecture) to accelerate BI decision cycles.
• Optimised data warehouse schemas on Azure Synapse and Amazon Redshift using star/snowflake modelling, columnstore indexing, distribution keys, and query rewrite, reducing average query latency by 30 %.
• Automated data quality validation framework using Great Expectations and custom PySpark checks, improving data accuracy by 50% and eliminating manual reconciliation overhead.
• Implemented GDPR and SOC 2 compliance controls—column-level encryption, row-level security, data masking, and audit trails—across Azure and AWS multi-cloud environments.
• Built real-time pipeline health dashboards (Azure Monitor / custom Python + alerts), achieving 99.9 % pipeline uptime through proactive monitoring and rapid incident response.

Database Administrator - RR Finance - Delhi, India

(2023-02 - 2023-10)

• Administered SQL Server environments—user provisioning, backup & recovery strategies, and high-availability configuration—for transactional systems processing thousands of daily operations.
• Rewrote and optimised T-SQL stored procedures and complex queries, reducing critical report execution times by up to 60% through index tuning, query plan analysis, and set-based refactoring.
• Enforced data integrity, consistency, and security policies, ensuring zero data-loss incidents across production databases.

Bachelor of Technology - Civil Engineering - IIMT Engineering College, Meerut (2015 - 2019)