
Sahil Ghaywat
Data Engineer
Hire this person
Send a job offer directly to this candidate
About
Data Engineer with 4.4 years of experience in designing scalable ETL pipelines, big data processing, and cloud-based data warehousing solutions using AWS services such as Glue, S3, Lambda, Redshift, Step Functions, and RDS. Strong expertise in PySpark, Spark SQL, Python, and SQL for building high-performance data pipelines and optimizing large-scale data transformations. Experienced in data migration, incremental loading, data quality validation, and Redshift performance optimization in enterprise data engineering environments.
Experience
Data Engineer - Exner Technologies Private Limited - Pune, Maharashtra
(2022-01 - 2026-04)
- Designed and implemented a scalable cloud-based data warehousing solution by migrating data from multiple on-premise databases including MySQL, Oracle, and PostgreSQL into Amazon Redshift to support centralized analytics and reporting.
- Developed scalable ETL pipelines using AWS Glue and PySpark to ingest and transform structured enterprise data into Amazon S3 staging layers for seamless data migration.
- Automated end-to-end ingestion workflows using AWS Lambda and AWS Step Functions for orchestration and event-driven processing.
- Implemented incremental data loading strategies to efficiently process large datasets and reduce pipeline execution time.
- Designed optimized Redshift schemas and leveraged COPY command for high-performance bulk data ingestion from Amazon S3.
- Improved query performance using distribution keys, sort keys, partitioning strategies, and query optimization techniques.
- Configured IAM roles and secure cross-service access policies for Glue, Lambda, S3, and Redshift integration.
- Monitored production workflows using CloudWatch logs and resolved ETL failures, deadlocks, and parallel execution bottlenecks.
- Engineered automated ETL workflows using AWS Glue, PySpark, and AWS Lambda for managing the data migration and processing of pharmaceutical datasets from Amazon S3 into PostgreSQL (AWS RDS).
- Developed robust data validation and cleansing framework including duplicate removal, null handling, and custom PySpark UDF implementations.
- Implemented referential integrity checks and orphan record handling mechanisms for incomplete product and inventory feeds.
- Optimized Glue job performance by tuning worker configurations, Spark transformations, and partition processing for large-scale datasets.
- Generated optimized Parquet datasets in Amazon S3 to improve downstream analytical query performance.
- Built reporting datasets for business use cases including top-selling products, monthly sales trends, and inventory analytics.
- Maintained ETL orchestration using AWS Step Functions and monitored workflow executions through CloudWatch.
- Developed incremental data ingestion and ongoing data migration pipelines to load retail sales and product datasets into Amazon S3 raw and curated layers.
- Built AWS Glue PySpark ETL jobs to cleanse, standardize, and transform high-volume retail transaction datasets during phase-wise data migration.
- Implemented AWS Glue Data Quality validation rules to ensure consistency, completeness, and accuracy of incoming data feeds.
- Designed optimized Redshift tables and loading strategies to support faster analytical reporting and dashboard queries.
- Generated aggregated business metrics including category-wise sales, regional performance analysis, and monthly revenue trends.
- Stored transformed datasets in optimized Parquet format to improve storage efficiency and query performance.
- Enhanced pipeline reliability and maintainability through modular ETL design and automated workflow monitoring.
Education
Bachelor of Computer Applications - Computer Applications - Savitribai Phule Pune University (SPPU)
