Praveen D

Having 9 years of hands-on experience in designing and developing Big Data applications using Hadoop ecosystem technologies including HDFS, Hive, Sqoop, Apache Spark, and AWS. Strong real-time experience across data storage, querying, processing, and analytical workflows in Big Data environments. Proven ability to design and implement solutions for complex large-scale data processing requirements.

Experienced in processing large volumes of structured, semi-structured, and streaming data efficiently. Hands-on experience in data architecture, including data ingestion, ETL pipeline design, and optimization. Extensive experience working on Hadoop migration projects and proof-of-concepts (POCs).

Expertise in developing and optimizing Hive-based data processing jobs for analytical use cases. Strong experience with Sqoop for importing and exporting data between HDFS and relational databases. Proficient in handling multiple file formats including JSON, XML, AVRO, and text files.

Hands-on experience deploying and executing Spark jobs on AWS EMR clusters. Experienced in implementing performance optimization and simplification initiatives to improve application efficiency and scalability. Strong understanding of Hadoop ecosystem installation, configuration, and usage, including HDFS, Hive, Sqoop, and Spark.

Skilled in understanding business requirements, collaborating with stakeholders, and performing source-to-target data mapping and design reviews.

Experience working in Agile development environments, collaborating closely with IT and business teams. Strong database experience with SQL Server and MySQL, along with advanced SQL querying skills. Excellent problem-solving and analytical abilities, with a continuous improvement and innovation mindset. Strong interpersonal, communication, and collaboration skills, with the ability to work effectively in team environments.

AWS Cloud Spark Developer - TATA Consultancy Services

(2021-04)

Project: Enterprise Data Processing & Analytics Platform. Client: AIG.

• Designed and developed large-scale distributed data processing applications using Apache Spark (PySpark, Spark SQL, RDDs, DataFrames) for batch analytics.
• Built and implemented end-to-end ETL pipelines to process structured and semi-structured datasets from multiple sources.
• Performed data cleansing, validation, and preprocessing using Spark transformations and actions.
• Utilized Spark RDD and DataFrame APIs to implement complex transformations including filtering, mapping, grouping, aggregations, and joins.
• Optimized Spark applications using data partitioning, caching, persistence, shuffle tuning, and memory optimization, improving job performance and resource utilization.
• Developed and deployed PySpark jobs on AWS EMR clusters, processing large datasets stored in Amazon S3 and HDFS.
• Integrated Spark applications with AWS services including S3, EMR, EC2, supporting scalable data lake architectures.
• Implemented robust error handling, logging, and debugging mechanisms for Spark jobs running on EMR clusters.
• Managed and orchestrated Spark job executions using AWS Step Functions.
• Designed and developed Apache Airflow DAGs to orchestrate complex Spark workflows on AWS EMR.
• Integrated Airflow with AWS services such as S3, EMR, Lambda, and SNS to enable event-driven and scheduled pipelines.
• Worked extensively with Hive, creating and managing managed, external, and partitioned tables based on business requirements.
• Implemented Hive partitioning and bucketing strategies to improve query performance.
• Handled Hive schema evolution using Avro file formats.
• Tuned Hive queries by adjusting memory allocation, parallelism, compression, and execution settings.
• Troubleshot and resolved Hive-related issues such as data skew, table corruption, and query performance bottlenecks.
• Collaborated with cross-functional teams to understand business requirements and deliver scalable data solutions in an Agile environment.

SQL Developer - TATA Consultancy Services

(2017-01 - 2021-03)

Project: credit reporting. Client: DTCC.

• Developed and maintained complex SQL queries, views, and stored procedures to support daily, weekly, and monthly business reports.
• Extracted, transformed, and aggregated data from multiple source systems to build report-ready datasets.
• Designed summary tables and materialized views to improve reporting performance and reduce query execution time.
• Optimized SQL queries using indexing, partitioning, and execution plan analysis, resulting in significant performance improvements.
• Implemented data validation and reconciliation checks to ensure accuracy and consistency of reported metrics.
• Supported BI tools such as Power BI by providing optimized datasets and backend SQL logic.
• Automated report generation and data refresh processes using scheduled SQL jobs.
• Monitored production reporting jobs, resolved failures, and performed root cause analysis for data issues.
• Collaborated with business analysts and stakeholders to translate reporting requirements into efficient SQL solutions.
• Participated in UAT and production deployments for reporting enhancements and new reports.

M.tech - VIT university (2014 - 2016)

B.tech - RVR & JC college of engineering (2009 - 2013)

intermediate - Narayana junior college (2007 - 2009)

High School - Shoghi High School (2007)