Viky

Exhibit a strong commitment to delivering impactful data solutions as a Big Data Engineer, leveraging 4+ years of experience in the comprehensive design, development, and maintenance of extensive business applications across various sectors. Results-driven Big Data Engineer with 4 years of experience developing scalable data solutions for cloud and on-premises environments.

Designed

PySpark pipelines on AWS, resulting in 30% reduction in processing time ($15K/month savings) through dynamic resource allocation and Parquet compression. Led migration of 10+ TB of data from SQL Server to Hadoop using Sqoop and PySpark, leading to 25% improvement in query performance. Proficient in Hadoop, Spark (RDD, DataFrame, SQL), Hive, Sqoop, and various cloud services.

Skilled in data serialization formats, schema evolution, and performance enhancement methods for data integrity and compatibility. Collaborative leader with interdisciplinary teamwork skills and strategic problem-solving approach to complex data challenges.

Big Data Engineer at Programmr Technologies Pvt. Ltd. (2021-11 – Present)

• Designed PySpark/Glue ETL pipelines processing 3-4TB/day (90% automation), while reducing AWS costs by 20% through Parquet compression and cluster tuning
• Optimized AWS EMR clusters (28-node R5.8x), reducing 5-6TB dataset processing from 7 to 4.5 hours (35% faster)
• Spearheaded end-to-end workflows with Airflow DAGs, achieving 99% uptime and reducing failure resolution time to <30 minutes via schema validation
• Implemented real-time data processing (Kafka/PySpark) for IoT analytics, cutting latency by 40%
• Executed Delta Lake for reliable data governance and schema evolution
• Migrated legacy SQL Server workloads to Snowflake, improving query speeds by 25%
• Collaborated on data lake integrations to converted and saved processed data in Parquet format to S3 and Snowflake, facilitating efficient analytics downstream
• Led Sqoop to transferred over 10 TB of enterprise data between Hadoop and relational databases, guaranteeing smooth integration and minimal interruptions
• Utilized Hive Query Profiler and partitioning techniques to boost query speeds by 35%
• Focused on schema management by creating Avro schemas for Hive tables and overseeing schema evolution to support field additions, type modifications, and nullability constraints
• Enhanced PySpark batch jobs on EMR using Parquet columnar storage, cutting costs by 20% through compression
• Performance optimization tools, including Hive Query Profiler, Hive Query Plan Visualization, and Hive Load Testing Tools, along with their capabilities and drawbacks
• Led creating Avro schema for Hive tables and overseeing schema evolution to adapt to modifications in data structure and format
• Created and enhanced extensive distributed data workflows utilizing PySpark on AWS for effective batch processing
• Connected AWS S3 with PySpark tasks for the storage and retrieval of substantial datasets, encompassing both intermediate and results
• Combining PySpark tasks with AWS S3, HDFS, and EC2 to handle data lakes and guarantee dependable storage and access of intermediate and final results
• Minimizing serialization costs and enhancing data compression by choosing the best formats and codecs suited to specific workload types
• Facilitating data-driven decision-making by providing quicker access to high-quality data across platforms integrated in the cloud and distributed systems

Master of Science – BPP University (2021-10)

Bachelor of Engineering – Gujarat Technology University (2018-06)