Gaurav Singh

Senior Data Engineer with 6 years of experience designing and optimizing scalable data pipelines across Manufacturing, Insurance, Banking, and Automotive domains. Skilled in cloud data engineering, ETL/ELT processes, batch-time, real-time streaming, and data warehousing. Hands-on expertise in AWS (Glue, Lambda, Redshift, DynamoDB, S3), Snowflake, PySpark, and SQL. Proven track record of reducing costs, improving performance, and leading teams to deliver high-impact data solutions.

Senior Data Engineer at Quantiphi Inc. (2023-10 – Present)

AI-first data and cloud engineering company

• Migrated over 200 GB of weekly migration data and 150 SAS codes to AWS Cloud, achieving a data integrity rate of 99.5% and reducing pipeline execution time by 2/3rd
• Utilized PySpark, SQL, and Python to convert 60k lines of SAS code, enhancing efficiency and processing speed
• Designed and maintained data pipelines using AWS Lambda, Glue, DynamoDB, and Kafka, supporting both real-time and batch processes
• Implemented AWS DynamoDB for NoSQL data transformation, ensuring efficient data retrieval and processing
• Led a team of 3-5 data engineers in the development and optimization of scalable data pipelines for various clients, mentoring junior members on best practices
• Conducted code reviews and knowledge-sharing sessions, improving team productivity
• Created Redshift stored procedures to detect dormant accounts and generate fraud alerts, contributing to improved client decision-making and revenue
• Designed and developed scalable ETL pipelines using Informatica Intelligent Cloud Services (IICS) for batch and incremental data processing
• Built parameterized mappings, reusable mapplets, and taskflows to orchestrate complex data transformations across multiple source systems
• Implemented SCD Type 1 & Type 2 logic for dimensional data modeling in enterprise data warehouse solutions
• Optimized IICS mappings by reducing transformation complexity and pushdown optimization to Snowflake
• Designed interactive Power BI dashboards with advanced DAX measures for KPI tracking and executive reporting
• Built star-schema based data models and optimized relationships for high-performance reporting

Data Engineer at Tata Consultancy Services (2022-07 – 2023-11)

Global IT and consulting services firm. On Third-Party role - Stratosphere IT services Pvt Ltd.

• Architected and implemented robust real-time and batch data pipelines processing an average of 25 GB of data using Hadoop, Hive, and Spark technologies
• Designed and maintained data models using dimensional modeling principles, enhancing query performance and consistency
• Implemented data governance and lineage tracking, ensuring compliance with industry regulations and standards
• Leveraged PySpark for data transformation, cleansing, and feature engineering, reducing processing times by 30%
• Optimized data warehouse performance on SQL-based systems, improving query response times by 42% through bucketing and partitioning
• Utilized Python scripting to automate data processing within AWS pipelines, reducing processing time by 30%
• Led the L2 production support team, managing incident resolution and ensuring 99.9% pipeline uptime through proactive monitoring and quick issue resolution
• Handled critical client queries and support tickets, collaborating with business users to resolve data discrepancies, restore jobs, and maintain SLA compliance

Software Engineer at Motherson sumi infotech & Design (2019-09 – 2022-07)

IT solutions provider for automotive and manufacturing.

• Followed OLAP processes to extract key insights from Hive/Hadoop Data Warehouses
• Developed and maintained dashboards and reports using Power BI, providing valuable business insights
• Automated data extraction from Hadoop processes, reducing manual effort by 70%
• Designed and maintained data models and schemas for reporting requirements
• Managed Data Pipelines using Airflow with Spark DataFrames and AWS services
• Created complex Power BI reports with dynamic measures, calculated columns, and interactive visuals to support business intelligence needs
• Automated data extraction from Hadoop processes for Power BI reporting, reducing manual effort by 70% and ensuring timely data refreshes

Bachelor of Technology in Information Technology – Ajay Kumar Garg Engineering College (2015-01 – 2019-01)