Shridharjadhav1122

Results-driven Data Engineer with 4+ years of experience, currently specializing in AWS Cloud, building scalable, reliable data pipelines and ETL/ELT workflows. Hands-on expertise in Amazon S3, AWS Glue, EMR, Redshift, Lambda, Step Functions, Athena, and CloudWatch for end-to-end data engineering solutions. Strong proficiency in PySpark, Python, and SQL for large-scale data processing, transformation, and performance optimization.

Experienced in designing event-driven architectures and orchestrating workflows using AWS Step Functions and Apache Airflow. Prior 1 years of experience in Azure ecosystem, working with Azure Databricks, Azure Data Factory (ADF), ADLS Gen2, and Snowflake, building scalable ELT pipelines and modern data warehouse solutions. Solid understanding of data warehousing, data modelling, and pipeline optimization, with a focus on cost-efficient, high-performance cloud solutions.

Experienced in Agile environment, delivering reliable and business-aligned data solutions.

AWS Data Engineer at VENTAS SOFTECH PVT LTD (2023-08 – Present)

Client name: Emblem health Care

• Built end-to-end data engineering pipelines on AWS to ingest, process, and load incremental data into fact and dimension tables in Amazon Redshift.
• Implemented CDC (Change Data Capture) framework to process incremental data loads efficiently using checkpointing mechanisms.
• Designed an event-driven architecture where file uploads to Amazon S3 (Bronze layer) trigger AWS Lambda, initiating AWS Step Functions workflows.
• Orchestrated multiple AWS Glue jobs using Step Functions to process data through Bronze → Silver → Gold layers.
• Used AWS Glue Data Catalog for metadata management and schema inference.
• Performed data cleansing, deduplication, and transformations using AWS Glue (PySpark) and stored processed data in S3 (Parquet format).
• Applied business transformations and built Gold layer datasets with partitioning strategy for optimized querying.
• Implemented Slowly Changing Dimensions (SCD Type 2) for historical data tracking in data warehouse.
• Loaded curated data into Amazon Redshift using optimized loading techniques for downstream analytics and reporting.
• Utilized Amazon Athena for querying and analyzing data directly from S3.
• Configured SNS notifications for failure alerts and workflow monitoring in Step Functions.
• Automated job scheduling using EventBridge for batch pipeline execution.
• Developed Lambda functions (Python, Boto3) for orchestration, triggering workflows, and handling event-based processing.
• Performed performance tuning and optimization of Glue jobs, Athena queries, and Redshift workloads.
• Monitored pipelines and system performance using Amazon CloudWatch logs and alarms.
• Implemented cost optimization strategies for S3 storage, Glue jobs, and Redshift usage.
• Collaborated with analytics teams to provide clean and reliable datasets for reporting and dashboarding.

Data Engineer at VENTAS SOFTECH PVT LTD (2022-07 – 2023-08)

Client name: Cdiscount Ltd

• Developed ETL pipelines using Azure Databricks (PySpark) to ingest data from SFTP into ADLS Gen2, applying business transformations.
• Orchestrated end-to-end workflows using Azure Data Factory (ADF) with scheduling and dependency management.
• Built ELT pipelines to load data from ADLS into Snowflake, leveraging staging, COPY commands, and optimized warehouse usage.
• Processed and transformed data (CSV, JSON, Parquet) and optimized Spark jobs for performance and scalability.
• Implemented ADF triggers (event & schedule-based) for automated pipeline execution.
• Monitored pipelines using Azure Monitor & Log Analytics and resolved failures efficiently.
• Developed reusable, parameterized ADF pipelines for multiple data sources.
• Wrote complex SQL queries and transformations in Snowflake for analytics use cases.
• Created and managed Snowflake schemas, tables, and views.
• Implemented data validation and quality checks across ETL stages.
• Optimized ETL workflows and Snowflake queries for performance and cost efficiency.
• Collaborated with cross-functional teams to deliver scalable data solutions.

B.E in Computer Science Engineering – Hindusthan college of engineering and technology, Coimbatore (2018-08 – 2022-06)