Anup Regmi

Senior Data Engineer with 6+ years of experience designing and building scalable data platforms and high-performance data pipelines across healthcare, retail, and financial domains. Strong expertise in developing ETL/ELT pipelines using Python, SQL, PySpark, and Apache Spark, supporting large-scale data processing across Azure and AWS environments. Experienced in designing modern lakehouse architectures using Databricks (Delta Lake), Snowflake, and Apache Iceberg for enterprise analytics and reporting.

Built batch and real-time data pipelines using Apache Kafka, AWS Kinesis, and Spark Structured Streaming, enabling scalable data processing. Skilled in orchestration and workflow management using Apache Airflow, Azure Data Factory, and Databricks Workflows, with a strong foundation in data warehousing and dimensional modeling (Snowflake, Azure Synapse, Redshift). Experienced in implementing data governance, security, and data quality frameworks in regulated environments, with additional expertise in cloud engineering using Terraform, Docker, Kubernetes, and CI/CD pipelines.

Senior Data Engineer at Change Healthcare (2023-06 – Present)

• Architected and owned an enterprise-scale Azure data platform processing 5TB+ daily healthcare data (claims, eligibility, clinical), enabling regulatory and operational analytics.
• Architected end-to-end ETL/ELT pipelines and ingestion frameworks using Azure Data Factory, Databricks, and Snowflake, enabling scalable batch and real-time processing.
• Designed and implemented a Medallion (Bronze/Silver/Gold) lakehouse architecture using Azure Databricks (Delta Lake) and ADLS Gen2, improving data reliability and reducing transformation latency by 35%.
• Developed enterprise-grade Snowflake data models and dbt transformation frameworks (SCD Type 2, incremental pipelines) to support compliance reporting and analytics.
• Optimized Snowflake workloads through clustering, warehouse scaling, and query tuning, improving performance by 3x and reducing compute costs by 20%.
• Engineered batch and real-time pipelines using PySpark, Kafka, Spark Structured Streaming, and CDC, reducing processing time by 40%.
• Built scalable data ingestion frameworks integrating EMR systems, claims, and provider data, improving ingestion efficiency by 30%.
• Established HIPAA-compliant data governance and security using Azure RBAC, Key Vault, Unity Catalog, and data quality frameworks (Great Expectations, dbt tests).
• Implemented data validation, monitoring, and quality frameworks across pipelines, ensuring high data accuracy and reliability for downstream analytics.
• Integrated cross-cloud data pipelines between AWS (S3, Lambda) and Azure environments, enabling seamless data exchange and unified reporting.
• Led migration of legacy on-prem SQL systems to Azure Synapse and Snowflake with zero downtime, improving scalability and reliability.
• Supported ML-enabled healthcare analytics by building feature engineering pipelines and scalable data workflows for model deployment and inference systems, enabling downstream automation and decision support.
• Mentored 4+ engineers on Spark optimization, distributed systems, and cloud architecture best practices, improving team delivery efficiency.

Data Engineer at Home Depot (2021-01 – 2023-05)

• Designed and implemented scalable AWS-based data pipelines processing millions of daily POS and inventory transactions, enabling real-time retail analytics and forecasting.
• Built ingestion frameworks using AWS Glue, improving ingestion throughput by 40% for high-volume retail datasets.
• Developed distributed transformation pipelines using Apache Spark on AWS EMR, converting raw data into curated, analytics-ready datasets.
• Architected Amazon S3 data lakes with optimized partitioning and lifecycle policies, reducing storage costs by 25%.
• Modeled enterprise Snowflake data warehouses and developed dbt transformation pipelines with testing and documentation standards, supporting merchandising analytics and demand forecasting for 100+ stakeholders.
• Developed real-time data pipelines using AWS Kinesis, Lambda, and Spark Structured Streaming, enabling faster analytics and reporting.
• Optimized Amazon Redshift and Snowflake performance through query tuning and workload management, improving dashboard response times by 2–3x.
• Integrated hybrid cloud reporting workflows using Azure Data Factory, enabling cross-platform analytics.
• Improved platform reliability and security using AWS CloudWatch, IAM, KMS encryption, and VPC configurations, ensuring system stability and compliance.
• Led large-scale data migration using AWS DMS, ensuring zero data loss and minimal downtime during transition.

Analytics Engineer at JPMorgan (2018-02 – 2020-12)

• Developed distributed data pipelines using Hadoop (HDFS, MapReduce) to process large-scale financial transactions, supporting regulatory and risk reporting.
• Built ingestion frameworks using Apache Sqoop and Apache Flume, integrating structured banking data into centralized platforms.
• Designed and optimized Apache Hive data warehouse schemas, improving performance for compliance and audit reporting.
• Implemented large-scale transformation workflows using MapReduce and Apache Pig, improving processing efficiency for financial datasets.
• Leveraged Apache Kafka to ingest real-time financial events, enabling near real-time fraud detection and monitoring.
• Applied partitioning and compression techniques in HDFS, reducing storage footprint by 20% and improving query performance.
• Designed scalable data marts supporting compliance and operational analytics across multiple business units.
• Implemented data validation and reconciliation frameworks, ensuring high data accuracy for regulatory reporting.
• Contributed to modernization initiatives by introducing Apache Spark into legacy Hadoop workflows, improving performance and scalability.
• Collaborated with cross-functional teams (risk, analytics, business stakeholders) to deliver enterprise data solutions in agile environments.

Bachelors in Electrical and Computer Engineering – Texas A&M University