Akhilesh kumar Gupta

Data Engineer with 2+ years of experience specializing in Python, PySpark, SQL, Databricks, Azure Data Factory (ADF), Delta Lake, Azure Data Lake, and Generative AI. Experienced in designing scalable ETL pipelines, real-time data ingestion, streaming architectures, and enterprise data platforms using Delta Lake and Unity Catalog. Proficient in developing LLM-powered RAG applications using LangChain, Databricks Mosaic AI, Vector Search, and MLflow to build AI-driven business solutions.

Hands-on experience in workflow automation, monitoring, data quality optimization, and delivering reliable data platforms by collaborating with cross-functional business and technical teams.

Assistant Manager at PGP Glass Limited (2024-06 – Present)

Data & AI Engineering

• Logbook Digitization & Data Pipelines: Digitized 700+ physical logbooks across 28+ departments; resolved UAT issues for enterprise rollout, eliminating 300 hrs/month of manual effort and cutting paper operations by approximately 90%. Architected automated Delta Lake ingestion pipelines on Databricks enabling real-time plant-wide analytics.
• Streaming ETL & Auto Loader Architecture: Architected real-time streaming ETL pipelines using PySpark, Delta Lake, and Azure Blob Storage with Auto Loader for 200+ JSON datasets (schema evolution, deduplication). Migrated from DLT to Bronze-Silver Medallion architecture, improving ingestion efficiency by 45-50% and reducing pipeline failures by approximately 35%.
• IoT Sensor Integration & Unity Catalog: Designed MQTT-based real-time IoT data ingestion into Azure Blob Storage and engineered curated Delta tables in Unity Catalog, enabling scalable analytics and data science workloads.
• SQL Server to Delta Lake Migration: Migrated 22+ tables from legacy SQL Server into Delta Lake using Bronze-Silver Medallion architecture, modernizing enterprise data pipelines and enabling near-real-time visibility across 70+ production lines.
• Databricks Performance Optimization: Optimized Spark jobs through partition tuning, cluster right-sizing, caching strategies, and Delta Lake optimizations, reducing compute costs by approximately 20% and improving overall pipeline performance.
• Digital AI Avatar Trainer: Engineered a production-grade AI Digital Trainer platform using Python, Flask, Azure AI Speech Avatar, Azure App Service, Azure Blob Storage, and Databricks Model Serving (Claude Sonnet), delivering 23+ interactive AI training modules with Azure AD SSO and LLM-powered voice/text Q&A; automated enterprise AI training, eliminating the need for a team of 5 trainers and saving approximately 40+ man-hours per week through self-paced digital learning.

B.Tech in Computer Science and Engineering – Maulana Azad National Institute of Technology (NIT), Bhopal (2020 – 2024)