Anshuman Prakash

Data Engineer with 4+ years of experience in PySpark, ETL, Hadoop, and backend development using C# and .NET core, working on Market Risk systems for reporting and data validation. Experienced in building PySpark ETL pipelines, processing large datasets, integrating multiple data sources, and troubleshooting Spark applications. Skilled in Python, PySpark, Spark SQL, Hadoop, C#/.NET, SQL Server, Oracle, Kafka and Solace, with experience in performance tuning, CI/CD, database optimization, and production support.

Software Developer – BA4 (Promoted from BA3 in Nov 2024) at BARCLAYS (2022-08 – Present)

• Developed a PySpark based ETL application on an in-house Hadoop cluster for risk report generation, saving 4-5 hours of manual effort daily and replacing MicroStrategy to reduce tool dependency and business costs.
• Added features to perform data joins and multi source ingestion using configurable JSON rules and Spark SQL, supporting CSV, Avro, Parquet, SQL Server, PostgreSQL, Oracle, Hive/Impala, MongoDB, Cube, and other sources.
• Built PySpark data quality pipelines to detect CDE errors and data mismatches across systems, processing 2+ billion rows daily using configurable rules and data contracts.
• Developed a log anaylzer tool to process logs and present useful information through an interactive UI using streamlit.
• Diagnosed and resolved Spark failures, Hadoop cluster performance issues, and application defects using Spark UI and log analysis.
• Developed C#/.NET microservices to trigger ETL operations and consume ETL outputs for Market Risk reporting via WebHDFS, integrating REST APIs, Solace, and Kafka to process 140,000+ events daily.
• Well versed in CI/CD pipelines using GitLab, Jenkins, and Nolio with SonarQube integration, eliminating manual deployments and code quality checks and reducing release time from 1-2 hours to 15 minutes.
• Migrated the data access layer from Entity Framework/SQL Server to Oracle via ODBC, optimizing stored procedures and reducing parallel load times by 2-3 hours daily.
• Worked in an Agile/Scrum environment, participating in sprint planning, daily stand-ups, retrospectives, and backlog refinement.
• Resolved critical memory leaks through improved memory management and garbage collection, preventing application downtime during large scale data processing.
• Supported production during disaster recovery events, application migrations, and AutoSys automation with minimal user impact.

B.Tech in CSE (Cybersecurity) – SRM Institute of Science and Technology (2018 – 2022)