Darshan Jain

Data Engineer with 2+ years of experience building, optimizing, and troubleshooting production batch and streaming pipelines processing 1M+ records daily. AWS Certified Data Engineer - Associate (DEA-C01) with hands-on expertise in Python, PySpark, SQL, AWS Glue, S3, Redshift, EMR, EventBridge, Kinesis, MSK and Airflow.

Experience in distributed ETL/ELT, incremental processing, dimensional modeling, SCD Type 1/2, data quality, workflow orchestration, performance optimization, and production incident resolution.

Programmer Analyst - Cognizant Technology Solutions

(2024-08)

• Processed 1M+ healthcare records daily across 20+ ETL workflows for multiple data sources using distributed PySpark transformations on AWS Glue.
• Optimized PySpark transformations using partition pruning, broadcast joins, predicate pushdown, repartitioning, and shuffle reduction, reducing ETL execution time by up to 40% and improving Glue resource utilization.
• Orchestrated AWS Glue, Step Functions, and MWAA workflows to meet daily SLAs; performed CloudWatch-based RCA and collaborated with business stakeholders to resolve production pipeline and data-quality incidents.
• Built reusable Full Load, Delta Load, and SCD Type 2 PySpark frameworks for change detection, historical tracking, record linking, and consistent downstream dimensional loads.
• Performed schema validation, data quality checks, reconciliation, and business-rule validation across ETL pipelines; optimized Redshift SQL and transformation logic, improving execution performance by 20–25%.
• Delivered account-inactivation, Leap data-integration, and record hard-linking enhancements, improving master-data quality and downstream reliability.
• Expanded ownership from individual ETL workflows to end-to-end pipeline design and delivery across 5+ data sources within the first year, serving as a key point of contact for production RCA and issue resolution.

B.Tech - Computer Science and Engineering - K. J. Somaiya Institute of Technology, Mumbai (2023-06)