JanakiRaman T

Data Engineer

Talent

BengaluruMember since August 16, 2026

PySparkHadoopSqoopHiveAWSAWS S3AWS EMRAWS Step FunctionsAWS LambdaScalaData MigrationHDFSPartitioningSQL

Hire this person

Send a job offer directly to this candidate

PySparkHadoopSqoopHiveAWS

About

Having 5 years of experience in designing, developing, and optimizing big data solutions, focusing on data migration, integration, transformation, and testing for scalable, high-performance applications. Experienced in developing scalable PySpark applications on AWS EMR, processing large datasets efficiently with distributed computing frameworks. Proficient in leveraging AWS S3 for storage and integration with PySpark jobs, enabling fault-tolerant and scalable data pipelines in the cloud.

Strong expertise in orchestrating PySpark workflows using AWS Step Functions, automating data processing tasks and managing complex job dependencies.

Experience

Data Engineer - ASICS TECHNOLOGIES PVT LTD

(2021-10)

Developed and maintained scalable data pipelines using PySpark on AWS EMR, processing large datasets stored in AWS S3 and Hadoop HDFS.
Optimized PySpark applications for performance, focusing on resource allocation, data shuffling, and memory management in distributed environments like AWS and Hadoop clusters.
Integrated AWS services (S3, Lambda, and Step Functions) with PySpark to automate data workflows and orchestrate large-scale data processing tasks.
Designed and implemented efficient Hive tables and partitioning strategies to optimize query performance and ensure smooth data processing in Hadoop.
Leveraged Sqoop for importing and exporting large datasets between relational databases and Hadoop ecosystems, ensuring data integrity and consistency.
Created and optimized custom PySpark transformations for batch data processing, enabling real-time analytics and reporting in a distributed Hadoop environment.
Managed and monitored Hadoop clusters to ensure smooth operation and optimal resource utilization for Spark and Hive jobs.
Utilized Hive's SQL-like queries to perform data aggregation and transformation on large datasets stored in HDFS for business insights.
Integrated Hadoop with AWS S3 for scalable data storage and efficient querying using Hive and Spark, enabling easy access to historical and real-time data.
Worked closely with data scientists and business analysts to understand data requirements and deliver high-quality, accurate data pipelines using Hadoop, PySpark, and Sqoop.