Kajal Kumari

Data Engineer with 4 years of experience building large-scale distributed data pipelines using PySpark, AWS EMR, and Apache Airflow. Proficient in developing Spark DataFrame and RDD-based workflows for structured and semi-structured data, including transformations, joins, aggregations, and window functions. Experienced in implementing Spark RDD-based workflows with Python.

Ability to troubleshoot common issues with Spark DataFrame, such as data processing errors, performance bottlenecks, and scalability limitations.

Optimized

PySpark applications running on AWS EMR for efficient data processing. Debugged complex Spark data transformations in PySpark jobs on AWS EMR. Expertise in querying Hive tables using SQL-like syntax and performing data analysis using tools like Apache Spark.

Skilled in integrating Hive tables with other big data technologies, such as Hadoop. Familiarity with Hive megastore and its role in managing table metadata and schema evolution. Knowledge of Hive table formats, including ORC, Parquet, and Avro, and their advantages and disadvantages for different use cases.

Proficient in writing Sqoop commands to transfer data between Hadoop and various databases such as MySQL and SQL Server.

Data Engineer (PySpark) - Mavenir

(2024-08)

• Expertise in using Spark SQL to process large-scale structured and semi-structured data sets, including querying, filtering, mapping, reducing, grouping, and aggregating data.
• Designed and developed serverlessETL pipelines using AWS Glue
• Created Glue Crawlers to catalog data from S3 into AWS Glue Data Catalog
• Built Glue Jobs using PySpark for large-scale batch data processing
• Implemented data transformations, cleansing, and enrichment using Spark SQL
• Optimized analytics using Amazon Athena and Redshift, improving query performance by 40%.
• Designed and implemented end-to-end ETL pipelines using AWS Glue (PySpark), S3, and Amazon Redshift for large-scale analytics.
• Designed and implemented scalable ETL pipelines using AWS Glue, PySpark, and Lambda to process structured and semi-structured data from multiple sources.
• Designed and optimized Amazon Redshift schemas using star and snowflake models.
• Orchestrated complex data workflows using Apache Airflow (MWAA) with dependency management and retries.
• Scheduled and monitored ETL jobs using Glue Triggers and CloudWatch.
• Loaded large-scale datasets into Redshift using the COPY command with S3 and IAM roles.
• Set up CloudWatch logs, alarms, and SNS notifications to monitor pipeline failures.
• Proficient in developing and implementing Spark DataFrame-based data processing workflows using Scala, Java, or Python programming languages.
• Experienced in optimizing Spark DataFrame performance by tuning various configuration settings, such as memory allocation, caching, and serialization.
• Experienced in optimizing Spark RDD performance by tuning various configuration settings, such as memory allocation, caching, and serialization.

Data Engineer (PySpark) - Mavenir

(2022-08 - 2024-08)

• Knowledge of Hive query tuning best practices, such as minimizing data transfers, avoiding unnecessary data conversions, and using appropriate data formats and compression codes.
• Experienced in handling schema compatibility issues in Hive, such as adding or removing fields, changing field types or names, and handling default values and nullability.
• Skilled in configuring Sqoop jobs for incremental data transfers using the Sqoop incremental import feature.
• Familiarity with Spark DataFrame schema and data type operations, such as adding, renaming, and dropping columns, casting data types, and handling null values
• Designed and implemented scalable data pipelines using Amazon Web Services (AWS) services, including S3, Glue, Redshift, Lambda, and IAM.
• Loaded processed data into Amazon Redshift using the COPY command with IAM roles for secure access.
• Experience working with Spark SQL in production environments and implementing performance monitoring and alerting systems to detect and resolve performance issues proactively.
• Designed and implemented an end-to-end cloud data migration pipeline to move large-scale structured telecom data from on-premise systems/HDFS to AWS S3 and Amazon Redshift for analytical reporting and business insights.
• Built data pipelines with AWS EMR and PySpark, reading from and writing to AWS S3.
• Debugged PySpark code to resolve errors and improve efficiency on AWS EMR.

BCA - Computer Applications - BRABU (Bihar) (2018 - 2021)

12th - BRABU (2017 - 2018)

10th - BSEB (2016 - 2017)