Sahil Ghaywat

Data Engineer with 4.3 years of experience in designing scalable ETL pipelines, big data processing, and cloud-based data warehousing solutions using AWS services such as Glue, S3, Lambda, Redshift, Step Functions, and RDS. Strong expertise in PySpark, Spark SQL, Python, and SQL for building high-performance data pipelines and optimizing large-scale data transformations. Experienced in data migration, incremental loading, data quality validation, and Redshift performance optimization in enterprise data engineering environments.

Data Engineer - Exner Technologies Private Limited - Pune, Maharashtra

(2022-01 - 2026-04)

• Designed and implemented a scalable cloud-based data warehousing solution by migrating data from multiple on-premise databases including MySQL, Oracle, and PostgreSQL into Amazon Redshift to support centralized analytics and reporting.
• Developed scalable ETL pipelines using AWS Glue and PySpark to ingest and transform structured enterprise data into Amazon S3 staging layers for seamless data migration.
• Automated end-to-end ingestion workflows using AWS Lambda and AWS Step Functions for orchestration and event-driven processing.
• Implemented incremental data loading strategies to efficiently process large datasets and reduce pipeline execution time.
• Designed optimized Redshift schemas and leveraged COPY command for high-performance bulk data ingestion from Amazon S3.
• Improved query performance using distribution keys, sort keys, partitioning strategies, and query optimization techniques.
• Configured IAM roles and secure cross-service access policies for Glue, Lambda, S3, and Redshift integration.
• Monitored production workflows using CloudWatch logs and resolved ETL failures, deadlocks, and parallel execution bottlenecks.
• Engineered automated ETL workflows using AWS Glue, PySpark, and AWS Lambda for managing the data migration and processing of pharmaceutical datasets from Amazon S3 into PostgreSQL (AWS RDS).
• Developed robust data validation and cleansing framework including duplicate removal, null handling, and custom PySpark UDF implementations.
• Implemented referential integrity checks and orphan record handling mechanisms for incomplete product and inventory feeds.
• Optimized Glue job performance by tuning worker configurations, Spark transformations, and partition processing for large-scale datasets.
• Generated optimized Parquet datasets in Amazon S3 to improve downstream analytical query performance.
• Built reporting datasets for business use cases including top-selling products, monthly sales trends, and inventory analytics.
• Maintained ETL orchestration using AWS Step Functions and monitored workflow executions through CloudWatch.
• Developed incremental data ingestion and ongoing data migration pipelines to load retail sales and product datasets into Amazon S3 raw and curated layers.
• Built AWS Glue PySpark ETL jobs to cleanse, standardize, and transform high-volume retail transaction datasets during phase-wise data migration.
• Implemented AWS Glue Data Quality validation rules to ensure consistency, completeness, and accuracy of incoming data feeds.
• Designed optimized Redshift tables and loading strategies to support faster analytical reporting and dashboard queries.
• Generated aggregated business metrics including category-wise sales, regional performance analysis, and monthly revenue trends.
• Stored transformed datasets in optimized Parquet format to improve storage efficiency and query performance.
• Enhanced pipeline reliability and maintainability through modular ETL design and automated workflow monitoring.

Bachelor of Computer Applications - Computer Applications - Savitribai Phule Pune University (SPPU)