Vishal Nagar

Data Engineer with nearly 3 years of experience designing and maintaining scalable ETL/ELT pipelines and cloud data platforms. Skilled in Azure Databricks, PySpark, SQL, Python, Delta Lake, AWS, and Azure. Experienced in SQL Server and ADF migrations, Lakehouse architecture, metadata-driven pipelines, MDM integration, and automated data validation. Strong background in production troubleshooting, pipeline optimization, and enterprise healthcare data engineering.

Data Engineer / Data Analyst at Skyralytics (2026-01 – Present)

• Design and develop production-grade ETL/ELT pipelines using Azure Databricks, PySpark, Spark SQL, Delta Lake, and Unity Catalog, implementing scalable Bronze, Silver, and Gold Lakehouse architecture.
• Migrate legacy SQL Server and Azure Data Factory (ADF) pipelines to Databricks, modernizing data ingestion, transformation, and processing workflows while preserving business logic and data integrity.
• Build reusable, metadata-driven, multi-tenant data pipelines with parameterized notebooks, dynamic configurations, and Databricks Jobs orchestration to support multiple healthcare clients.
• Integrate complex healthcare EMR data from NetSmart and Axxess, developing standardized datasets for admissions, referrals, patient episodes, accounts, census, contacts, and service locations.
• Implement incremental processing, Delta MERGE operations, composite-key matching, source deduplication, and change detection to maintain reliable and consistent data across pipelines.
• Develop complex business transformations and MDM integrations, including sales-representative attribution, referral-source mapping, and reconciliation of source-system identifiers with canonical OneSky IDs.
• Build automated data-quality and reconciliation frameworks using row-level comparisons, hash validation, duplicate detection, key uniqueness checks, and source-to-target data verification.
• Troubleshoot production pipeline failures, schema inconsistencies, MERGE conflicts, and data-quality issues while collaborating with engineering and CRM teams to deliver validated datasets.

Data Engineer / MLOps at Novartis (2024-01 – 2025-12)

• Designed and maintained scalable data and machine learning pipelines using Databricks, Python, PySpark, SQL, Delta Lake, and Unity Catalog for enterprise analytics and AI/ML applications.
• Developed a reusable Python-based framework for ETL processing, feature engineering, model training, and inference, standardizing pipeline development across multiple projects.
• Automated CI/CD pipelines using Bitbucket, Jenkins, and Databricks, enabling reliable code deployment and environment promotion across development, validation, and production.
• Implemented automated ML retraining, configuration-driven inference workflows, and model monitoring using MLflow and Databricks Lakehouse Monitoring to track data quality, drift, and production performance.
• Optimized Spark and SQL workloads through query tuning, partitioning, Delta Lake design, and cluster configuration to improve processing efficiency and resource utilization.
• Deployed and maintained Langfuse-based LLM observability on AWS using EC2, EKS, RDS, ClickHouse, Kubernetes, and Helm, supporting monitoring and troubleshooting of enterprise RAG and GenAI applications.
• Developed analytics-ready datasets and reporting pipelines supporting Power BI and Tableau dashboards, while managing data governance, permissions, and lineage through Unity Catalog.
• Resolved production data pipeline failures, deployment issues, and data-quality inconsistencies while collaborating with data engineering, MLOps, and analytics teams.

Advanced Diploma in Cloud Architecture and Administration – Seneca College, Toronto, ON

Advanced Diploma in Big Data Solutions Architecture – Conestoga College, Kitchener, ON

Bachelor in Computer Engineering – A.D. Patel Institute of Technology, India