Hridaya Yadav

Data Engineer with 4 years building production-grade pipelines across logistics, life sciences, and financial domains. Specializes in Databricks (PySpark, Delta Lake, Auto Loader, Structured Streaming) and Azure Data Factory — designing medallion architectures, metadata-driven incremental frameworks, and performance-optimized ingestion systems that directly power executive dashboards and real-time financial reporting. Proven track record of eliminating manual ETL bottlenecks, preventing downstream data failures, and delivering audit-ready pipelines for regulated industries.

Data Engineer at R Systems (2025-07 – Present)

Client: 3PL Logistics Provider

Payables Analytics Engine at R Systems (2025-09 – 2026-01)

• Eliminated manual gold-layer loading by building a metadata-driven incremental pipeline on Databricks — processing vendor transactions from D365 Finance and BTMS, enabling same-day executive dashboards for cash flow and vendor payment analytics
• Implemented watermark-based change detection with dynamic SQL switching between FULL and INCREMENTAL loads, ensuring zero reprocessing of unchanged records across multi-table JOIN aggregations
• Built comprehensive audit logging and automated alerting, providing full lineage and traceability required for financial compliance reporting

Data Engineer at Sequent Tech (2024-06 – 2025-07)

Life Sciences & Healthcare

Census Data Ingestion at Sequent Tech (2025-01 – 2025-04)

• Built real-time patient census ingestion using Databricks Structured Streaming into Delta tables, ensuring continuous data availability for clinical reporting with zero message loss
• Applied Auto Loader optimization, strategic repartitioning, and file size tuning to eliminate small-file accumulation — improving downstream query performance and preventing job degradation over time
• Implemented checkpointing for fault-tolerant state management, guaranteeing exactly-once processing across restarts

Commission Analytics System at Sequent Tech (2024-10 – 2025-01)

• Automated monthly, quarterly, and yearly commission calculation workflows for hospital staff in ADF — replacing a manual process and eliminating calculation errors that previously caused payment disputes
• Delivered timely incentive reporting directly tied to patient admissions and revenue data, improving staff trust in finance data accuracy

Enterprise Data Integration at Sequent Tech (2024-08 – 2024-11)

• Designed end-to-end medallion pipeline (Raw → Integration → Reporting) on Databricks using Auto Loader for life sciences data from ADLS Gen2, supporting admissions, referrals, and discharge dimensions
• Eliminated duplicate records at ingestion using window-function deduplication before landing into curated Delta zones — preventing downstream analytical errors
• Applied Salting on high-cardinality keys (admission_id, client_id) to resolve data skew; added schema inference + drift validation to handle evolving source schemas without pipeline failures
• Enforced data quality via expectation-based assertions on critical fields (referral_id), with automated alerting on null/missing values

Data Engineer at Gemini Solutions (2022-06 – 2024-06)

Financial Data

Historical Data Migration Framework at Gemini Solutions (2023-11 – 2024-02)

Built parameterized ADF pipeline to backfill years of Bloomberg ticker data via Azure Batch + Python — reducing backfill delivery time by 70% compared to the prior manual process and ensuring completeness for historical financial modelling

Bloomberg Ticker Ingestion at Gemini Solutions (2023-07 – 2023-09)

Developed Azure Function triggering REST calls to Bloomberg endpoints, parsing JSON responses, and loading enriched data into Delta tables via Databricks — feeding financial modelling dashboards on Azure SQL Database

Rollover Automation at Gemini Solutions (2022-12 – 2023-02)

Automated daily missing-ticker detection using ADF Lookup and Filter activities, reducing manual reconciliation effort by 70% and preventing downstream failures caused by data gaps

B.Tech in Information Technology – ABES Engineering College