Farhan Khan

Senior Data Engineer with 9+ years of end-to-end experience designing scalable data platforms, modernizing legacy systems, and building real-time/batch data pipelines across healthcare, financial services, and insurance. Expert in cloud-native engineering across AWS, Azure, and GCP, distributed systems, Spark-based architectures, streaming systems, lakehouse modernization, LLM/NLP pipelines, and HIPAA-compliant healthcare data engineering. Proven success delivering high-performance ETL/ELT, lakehouse architectures (Delta Lake, Snowflake), and streaming platforms (Kafka, Kinesis, Pub/Sub) processing millions of daily events.

Strong background integrating EHR/EMR data (Epic, Cerner, Allscripts), implementing data governance & observability, and enabling analytics for enterprise-scale decision-making. Recognized for driving cost optimization (30–40%), cutting pipeline failures by 90%, and enabling real-time insights via dashboarding, AI/ML integrations, and automated workflows using dbt, Airflow, Cloud Composer, MWAA, and Dataform.

Sr Data Engineer | Data Analyst at Somatus (2023-11 – Present)

Architected and delivered enterprise-scale, HIPAA-compliant data platforms used across clinical operations, population health, finance, and research analytics. Led multi-cloud modernization initiatives, real-time streaming architectures, and data quality/governance programs that improved reliability, observability, and decision-making across the organization.

• Architected and engineered multi-cloud data pipelines leveraging GCP (BigQuery, Dataflow, Dataproc), AWS (EMR, Glue, S3, Lambda), and Azure (Synapse, ADF, Databricks, ADLS), enabling unified ingestion of structured, semi-structured, streaming, and EHR datasets across the enterprise.
• Designed and implemented Delta Lake and Snowflake lakehouse architectures that reduced query latency by 40%, improved schema evolution, and supported both batch and streaming use cases used by data scientists, analysts, and clinical teams.
• Built high-performance ingestion frameworks for HL7, FHIR, CCD, and EHR extracts, enhancing downstream model accuracy and reducing manual clinical data reconciliation.
• Optimized BigQuery performance by implementing table partitioning, clustering, materialized views, and predicate pushdown—leading to consistent 30–50% cost and scan-time reduction.
• Leveraged Athena dynamic partition pruning and file compaction strategies to achieve 5x query performance for large-scale patient metrics and claims datasets.
• Designed and deployed real-time streaming architectures using Kafka 2.8, Spark Structured Streaming 3.4, Azure Event Hubs, Kinesis, and Pub/Sub, enabling ingestion and transformation of 25M+ healthcare events per day.
• Built real-time patient monitoring pipelines to ingest IoT device and telemetry feeds through Azure Event Hubs → Stream Analytics → ADLS → Databricks, enabling low-latency clinical alerts.
• Implemented exactly-once processing semantics, watermarking, checkpointing, and backpressure handling to guarantee data accuracy in high-volume streaming pipelines.
• Developed automated stream health dashboards using Datadog, CloudWatch, and metrics instrumentation to alert on lag, throughput, and consumer failures, cutting operational incidents by 40%.
• Designed robust ETL/ELT workflows for Epic Clarity, Epic Caboodle, Cerner, Allscripts, reducing delivery time for critical datasets (encounters, labs, orders, claims) by 30%.
• Built reusable and modular dbt transformation layers with Git-based CI/CD that improved maintainability and reduced deployment failures by 30%+.
• Created enterprise-wide dimensional and Data Vault models supporting care management, population health, quality reporting, risk stratification, and financial insights.
• Created standardized ETL frameworks that follow the rules of SQL Server SSIS. This makes it possible to use modular, reusable pipeline components and makes it easier to keep everything up to date and consistent across environments.
• Implemented Monte Carlo Data Observability, enabling anomaly detection, data drift monitoring, schema change alerts, and incident triaging, reducing downstream failures by 90%.
• Integrated OpenLineage into Airflow, dbt, and Dataflow workloads, creating full end-to-end lineage across ingestion, transformations, and BI layers—significantly improving audit readiness.
• Built validation frameworks for HIPAA, HEDIS, NCQA, CMS requirements, ensuring regulatory compliance and reducing manual audit preparation time.
• Engineered scalable APIs with MuleSoft, Python (FastAPI/Flask), and AWS Lambda, enabling high-performance interoperability across Salesforce, EHR platforms, CRM systems, and enterprise data warehouses.
• Reduced data synchronization latency by 40% between Salesforce Data Cloud, AWS Redshift, and on-prem systems.
• Built automation scripts and event-driven pipelines (Cloud Functions, Lambda, EventGrid triggers) that reduced manual operational overhead for file-based and event-based workflows.
• Created and deployed high-performance RESTful APIs for accessing and reporting data, which work with scalable microservices architectures.
• Delivered analytics solutions using Tableau, Power BI, Looker, and Qlik, enabling real-time visibility into operational metrics, patient quality indicators, revenue cycle trends, and clinical productivity.
• Built executive-level EHR analytics dashboards supporting IRS, CMS, HHSC, compliance audits, and clinical quality reporting.
• Partnered with clinicians, business stakeholders, and data scientists to deliver high-value datasets used for predictive modeling, provider performance, risk scoring, and care management programs.
• Reduced manual reporting by 40% through automated refresh pipelines and governance workflows.
• Served as a technical lead for multi-cloud data engineering initiatives, reviewing architecture proposals, mentoring junior engineers, and driving coding standards.
• Worked with InfoSec and Compliance Teams to implement HIPAA-compliant encryption, RBAC, PHI masking, data retention, audit logging, and secure access patterns across data platforms.
• Improved SLAs for critical datasets by implementing checkpointing, file compaction, orchestration retries, and latency monitoring across Airflow and MWAA environments.

Senior Data Engineer/Big Data Engineer at S&P Global (2020-01 – 2023-10)

Led large-scale cloud modernization, data engineering, and analytics initiatives across Azure and AWS for a global insurance and financial services organization. Built high-performance pipelines, reduced compute costs across multiple cloud platforms, and enabled real-time analytics for underwriting, risk modeling, financial forecasting, and marketing intelligence.

• Spearheaded migration of legacy on-prem analytics workloads to Azure ADF, ADLS, Azure SQL, and Synapse, achieving zero downtime and ensuring continuity of financial reporting systems.
• Designed hybrid cloud pipelines spanning AWS S3 → Azure Databricks → SQL Server/Snowflake, enabling a unified cross-cloud data integration strategy.
• Built scalable ingestion frameworks using ADF, Databricks, Glue, Dataflow, automating ingestion of structured, semi-structured, streaming, and third-party vendor data.
• Reduced BigQuery scan costs by 35% through strategic use of clustering, partitioning, materialized views, and query refactoring.
• Cut Snowflake and Redshift compute expenses by 25% by tuning WLM, optimizing clustering keys, and implementing auto-suspend/auto-resume strategies.
• Built automated cloud cost-governance pipelines to monitor table scans, query costs, pipeline runtimes, and resource utilization across AWS and Azure.
• Built and managed the Databricks lakehouse architecture, which includes the structure of workspaces, cluster policies, job orchestration, and security models.
• Optimized Databricks Spark workloads and Delta Lake storage, which lowered the cost of computing and sped up queries on large datasets.
• Led the move of old data platforms to Databricks, bringing batch and streaming pipelines up to date with a single lakehouse architecture.