Skip to main content
Sahil Ghaywat

Sahil Ghaywat

Data Engineer

Talent
MumbaiMember since May 21, 2026
Data EngineerPythonAws Lambdaaws glueaws redshiftaws s3aws rdsaws step functionsEtl Developerpysparkapache sparkcloudwatchetl pipelinesaws data engineerpyspark developerci/cdhadoophdfs
·

Hire this person

Send a job offer directly to this candidate

Data EngineerPythonAws Lambdaaws glueaws redshift

About

Data Engineer with 4.4 years of experience in designing scalable ETL pipelines, big data processing, and cloud-based data warehousing solutions using AWS services such as Glue, S3, Lambda, Redshift, Step Functions, and RDS. Strong expertise in PySpark, Spark SQL, Python, and SQL for building high-performance data pipelines and optimizing large-scale data transformations. Experienced in data migration, incremental loading, data quality validation, and Redshift performance optimization in enterprise data engineering environments.

Experience

Data Engineer - Exner Technologies Private Limited - Pune, Maharashtra

(2022-01 - 2026-04)

  • Designed and implemented a scalable cloud-based data warehousing solution by migrating data from multiple on-premise databases including MySQL, Oracle, and PostgreSQL into Amazon Redshift to support centralized analytics and reporting.
  • Developed scalable ETL pipelines using AWS Glue and PySpark to ingest and transform structured enterprise data into Amazon S3 staging layers for seamless data migration.
  • Automated end-to-end ingestion workflows using AWS Lambda and AWS Step Functions for orchestration and event-driven processing.
  • Implemented incremental data loading strategies to efficiently process large datasets and reduce pipeline execution time.
  • Designed optimized Redshift schemas and leveraged COPY command for high-performance bulk data ingestion from Amazon S3.
  • Improved query performance using distribution keys, sort keys, partitioning strategies, and query optimization techniques.
  • Configured IAM roles and secure cross-service access policies for Glue, Lambda, S3, and Redshift integration.
  • Monitored production workflows using CloudWatch logs and resolved ETL failures, deadlocks, and parallel execution bottlenecks.
  • Engineered automated ETL workflows using AWS Glue, PySpark, and AWS Lambda for managing the data migration and processing of pharmaceutical datasets from Amazon S3 into PostgreSQL (AWS RDS).
  • Developed robust data validation and cleansing framework including duplicate removal, null handling, and custom PySpark UDF implementations.
  • Implemented referential integrity checks and orphan record handling mechanisms for incomplete product and inventory feeds.
  • Optimized Glue job performance by tuning worker configurations, Spark transformations, and partition processing for large-scale datasets.
  • Generated optimized Parquet datasets in Amazon S3 to improve downstream analytical query performance.
  • Built reporting datasets for business use cases including top-selling products, monthly sales trends, and inventory analytics.
  • Maintained ETL orchestration using AWS Step Functions and monitored workflow executions through CloudWatch.
  • Developed incremental data ingestion and ongoing data migration pipelines to load retail sales and product datasets into Amazon S3 raw and curated layers.
  • Built AWS Glue PySpark ETL jobs to cleanse, standardize, and transform high-volume retail transaction datasets during phase-wise data migration.
  • Implemented AWS Glue Data Quality validation rules to ensure consistency, completeness, and accuracy of incoming data feeds.
  • Designed optimized Redshift tables and loading strategies to support faster analytical reporting and dashboard queries.
  • Generated aggregated business metrics including category-wise sales, regional performance analysis, and monthly revenue trends.
  • Stored transformed datasets in optimized Parquet format to improve storage efficiency and query performance.
  • Enhanced pipeline reliability and maintainability through modular ETL design and automated workflow monitoring.

Education

Bachelor of Computer Applications - Computer Applications - Savitribai Phule Pune University (SPPU)

Skills

Data Engineer

Reviews

Related