Avanish Gupta

Senior Data Engineer with 5+ years of experience designing cloud-native data platforms, scalable ETL/ELT pipelines, and Databricks Lakehouse solutions on AWS. Expertise in Python, PySpark, Spark SQL, Data Modeling, Data Architecture, and CI/CD with a proven track record of delivering high-performance, cost-optimized, and governed data platforms that power analytics, AI-assisted automation, and enterprise decision-making.

Senior Data Engineer at Bristol Myers Squibb (2025-07 – Present)

• Architected and implemented an enterprise Databricks Lakehouse platform using Medallion Architecture and Unity Catalog, governing 20,000+ data assets through secure RBAC, centralized metadata, and scalable data governance.
• Architected an AI-assisted Data Conformance Engine using Python, AWS Glue, Lambda, EventBridge, and SNS, automating vendor biomarker validation and reducing manual effort by 80% while improving enterprise data quality.
• Designed reusable cloud-native data platform components and standardized ingestion patterns that improved pipeline scalability, data governance, and onboarding across multiple enterprise data products.

Data Engineer II at Bristol Myers Squibb (2023-09 – 2025-06)

• Designed configurable, event-driven ETL pipelines using Python, AWS Lambda, and EventBridge, enabling scalable metadata-driven ingestion and eliminating custom onboarding for over 75% of vendors.
• Modernized legacy ETL workloads by migrating 40% of pipelines to serverless AWS Glue, improving scalability, maintainability, operational efficiency, and cloud cost optimization.
• Developed scalable PySpark transformation pipelines on Databricks, implementing reusable transformation logic that improved data accessibility and downstream analytics performance.
• Implemented least-privilege IAM access across 50+ IAM roles and 12+ Amazon S3 buckets, strengthening data security, governance, and reducing storage costs by 60%.

Data Engineer at Noodle.AI (2022-06 – 2023-08)

• Designed and optimized distributed PySpark ETL pipelines on Databricks processing 10M+ IoT records/day, enabling ML-driven manufacturing analytics and reducing production defects by 20%.
• Optimized multi source ETL workloads through partition pruning, query tuning, execution plan optimization, and workload balancing, reducing processing time by 90% and cloud compute costs by 70%.
• Migrated enterprise workflows to Airflow (Astronomer), implementing centralized orchestration, retry policies, SLA monitoring, and dependency management for reliable pipeline execution.

Application Development Engineer I at NCR Corporation (2021-02 – 2022-05)

• Developed PySpark and SQL ETL pipelines on Databricks, integrating Amazon S3 with enterprise data warehouses to enable scalable reporting and analytics.
• Contributed to Agile software delivery using Git, SVN, JIRA, and CI/CD practices, improving release quality and development efficiency.

Bachelor of Engineering in Computer Science – Chandigarh University (2017-08 – 2021-08)