Sabaresh Ramesh

AWS Data Engineer with 4 years of experience in designing and developing scalable ETL/ELT pipelines and cloud-based data solutions using AWS Glue, AWS Lambda, Amazon S3, Amazon Athena, AWS IAM, and AWS CloudFormation. Hands-on experience integrating external platforms such as CEIPAL with Amazon S3 through API-based data ingestion processes using Python, Apache Kafka, and Amazon Kinesis. Skilled in developing automated ETL pipelines using AWS Glue, PySpark, Spark SQL, and AWS Glue Crawlers for schema detection, data transformation, cleansing, and metadata management.

Proficient in SQL, Python, PySpark, Amazon Athena, and Power BI for data analysis, dashboard development, reporting automation, and large-scale data processing. Experienced in building automated batch and incremental processing pipelines, implementing duplicate data removal, workflow orchestration using Apache Airflow, and optimizing ETL performance for scalable analytics solutions.

Data Engineer at VDart Inc (2021-12 – 2026-02)

• Designed and developed an end-to-end AWS Data Lake architecture workflow integrating external CEIPAL APIs with Amazon S3 using Python-based API ingestion processes and automated ETL pipelines.
• Built scalable data ingestion architecture using AWS Lambda, Apache Kafka, and Amazon Kinesis for handling batch and near real-time data processing workflows.
• Developed AWS Glue ETL jobs using PySpark DataFrames and Spark SQL for large-scale data transformation, cleansing, schema mapping, standardization, and duplicate data removal.
• Created a multi-layered Amazon S3 Data Lake architecture consisting of Raw Layer, Processed Layer, and Reporting Layer for efficient data storage and analytics processing.
• Implemented Full Load and Incremental Load processing strategies to efficiently process new and updated records while reducing redundant data execution.
• Utilized AWS Glue Crawlers for automated schema detection, metadata catalog creation, and centralized data management for analytics and reporting workflows.
• Automated workflow orchestration, job scheduling, dependency handling, and monitoring using Apache Airflow and AWS Lambda for reliable ETL pipeline execution.
• Configured AWS IAM role-based access controls and utilized AWS CloudFormation templates for secure infrastructure automation and deployment of AWS resources.
• Queried transformed and processed datasets using Amazon Athena and optimized PySpark processing using partitioning and performance tuning techniques for large-scale datasets.
• Implemented logging, monitoring, and error-handling mechanisms using Amazon CloudWatch and developed Power BI dashboards for automated reporting, business insights, and analytics visualization.

Data Engineer at VDart Inc (2021-12 – 2026-02)

Project 1 – API Integration & Reporting Automation (Internal)

• Designed and developed an end-to-end AWS Data Lake architecture workflow integrating external CEIPAL APIs with Amazon S3 using Python-based API ingestion processes and automated ETL pipelines.
• Built scalable data ingestion architecture using AWS Lambda, Apache Kafka, and Amazon Kinesis for handling batch and near real-time data processing workflows.
• Developed AWS Glue ETL jobs using PySpark DataFrames and Spark SQL for large-scale data transformation, cleansing, schema mapping, standardization, and duplicate data removal.
• Created a multi-layered Amazon S3 Data Lake architecture consisting of Raw Layer, Processed Layer, and Reporting Layer for efficient data storage and analytics processing.
• Implemented Full Load and Incremental Load processing strategies to efficiently process new and updated records while reducing redundant data execution.
• Utilized AWS Glue Crawlers for automated schema detection, metadata catalog creation, and centralized data management for analytics and reporting workflows.
• Automated workflow orchestration, job scheduling, dependency handling, and monitoring using Apache Airflow and AWS Lambda for reliable ETL pipeline execution.
• Configured AWS IAM role-based access controls and utilized AWS CloudFormation templates for secure infrastructure automation and deployment of AWS resources.
• Queried transformed and processed datasets using Amazon Athena and optimized PySpark processing using partitioning and performance tuning techniques for large-scale datasets.
• Implemented logging, monitoring, and error-handling mechanisms using Amazon CloudWatch and developed Power BI dashboards for automated reporting, business insights, and analytics visualization.

Data Engineer at VDart Inc (2021-12 – 2026-02)

Project 2 – Duplicate Data Removal & ETL Optimization (Internal)

• Developed scalable AWS Glue ETL pipelines using PySpark to identify, standardize, cleanse, and remove duplicate records from enterprise datasets.
• Built distributed batch processing workflows using PySpark DataFrames and Spark SQL for handling high-volume data processing.
• Worked with Apache Kafka and Amazon Kinesis concepts for scalable data ingestion and streaming pipeline integration.
• Automated ETL workflow orchestration and scheduling using Apache Airflow.
• Integrated AWS Lambda functions for automated validation, notifications, and ETL trigger mechanisms.
• Used advanced SQL queries and Spark SQL transformations for filtering, aggregation, deduplication, and data standardization processes.
• Implemented Incremental Load processing and S3 partitioning strategies to improve ETL execution performance and reduce processing time.
• Developed reusable and scalable AWS Glue jobs for automated data transformation and batch processing workflows.
• Configured AWS IAM policies and secure access management between AWS services and ETL components.
• Queried transformed datasets using Amazon Athena for reporting, validation, and analytical purposes.
• Automated infrastructure setup and deployment activities using AWS CloudFormation templates.
• Implemented logging, monitoring, and error-handling mechanisms using Amazon CloudWatch for reliable ETL execution.
• Improved data quality, reporting accuracy, and workflow efficiency through ETL optimization and automation techniques.

Master of Business Administration in HR and Finance – K. RAMAKRISHNAN COLLEGE OF ENGINEERING (2019-08 – 2021-06)

Bachelor of Engineering in EEE – MOOKAMBIGAI COLLEGE OF ENGINEERING (2014-08 – 2018-04)