Vaishali Reddy

Senior Data Engineer with 8+ years of experience building data pipelines, data warehouses, and analytics platforms across healthcare, life sciences, and pharmaceutical domains. 2+ years of hands-on experience building solutions on Google Cloud Platform using BigQuery, Dataflow, Dataproc, Cloud Composer, Pub/Sub, Cloud Storage, and Cloud Functions. Designed and established standardized, reusable ELT patterns in PySpark and BigQuery SQL, with layered raw, staging, and curated models for analytics-ready data. Built batch and near real-time ingestion pipelines using Kafka, Pub/Sub, and Dataflow for claims, eligibility, and event-driven data.

Orchestrated complex workflows in Apache Airflow and Cloud Composer with dependency checks, retries, and recovery steps. Strong background in data warehousing and dimensional modeling, designing star and snowflake schemas and data marts for claims reporting, member analytics, and operational dashboards. Advanced SQL skills across BigQuery, Snowflake, Synapse, and SQL Server, including complex queries, reconciliation logic, and performance tuning.

Deep healthcare data experience with claims, member, provider, clinical, and real-world evidence datasets, plus HL7 and FHIR interoperability. Implemented HIPAA-compliant data protection using IAM, VPC Service Controls, encryption, audit logging, and data masking for PHI. Built data quality, validation, reconciliation, and lineage frameworks using Python, SQL, and dbt.

Multi-cloud experience across GCP, AWS, and Azure, including Hadoop-to-cloud migrations to Dataproc, Snowflake, and Azure Data Lake. Automated infrastructure and deployments using Terraform, Jenkins, Harness, Docker, and Kubernetes. Experienced working in Agile product delivery teams, mentoring engineers, reviewing code, and presenting technical designs to technical and business audiences.

Sr. Data Engineer - Highmark Health - Pittsburg, PA

(2026-04)

• Design and build scalable data pipelines on GCP using Dataflow, Dataproc, Cloud Storage, Cloud Functions, and BigQuery to process healthcare claims, member, provider, and clinical datasets for enterprise analytics.
• Established standardized, reusable ELT patterns in PySpark and BigQuery SQL for ingestion, transformation, and validation, reducing new pipeline build time and keeping design consistent across teams.
• Build batch and near real-time ingestion pipelines using Pub/Sub, Kafka, and Dataflow to deliver claims and eligibility events into BigQuery.
• Orchestrate daily and intraday workflows in Cloud Composer (Airflow) with dependency checks, retries, and recovery steps.
• Develop layered ELT models in BigQuery (raw, staging, curated), loading source data first and transforming it inside the warehouse into analytics-ready tables.
• Design dimensional models and data marts using star and snowflake schemas for claims reporting, member analytics, and operational dashboards.
• Owned end-to-end pipeline architecture on GCP across ingestion, transformation, modeling, and serving layers for claims, member, and provider data.
• Optimized BigQuery warehouse cost and compute efficiency through partitioning, clustering, and materialized views.
• Mentored engineers, lead design reviews, and explain architecture tradeoffs to technical and business stakeholders.
• Partnered cross-functionally with analytics and data science teams to design curated BigQuery datasets for reporting and modeling.
• Write advanced SQL for claims adjudication analysis, member eligibility tracking, and provider performance reporting.
• Build data quality and reconciliation checks in Python and SQL to verify record counts, financial totals, and field-level accuracy between source and target systems.
• Protect PHI under HIPAA using IAM, VPC Service Controls, customer-managed encryption keys, Cloud Audit Logs, and Sensitive Data Protection for masking.
• Optimize BigQuery performance and cost through partitioning, clustering, and materialized views, and tune Dataproc Spark jobs through partitioning, broadcast joins, and executor sizing.
• Automate deployment of Dataflow jobs, Composer DAGs, Cloud Functions, and Terraform infrastructure through CI/CD pipelines in Jenkins and Harness.
• Work in an Agile product delivery model with architects, analysts, and business stakeholders, turning reporting and regulatory requirements into stable, supportable pipelines.
• Review code and guide other engineers on pipeline design standards, and present technical designs to both technical and business audiences.

Sr. Data Engineer - Clarivate Analytics

(2024-09 - 2026-03)

• Developed Python services collecting publication, patent, and clinical trial data from publisher, patent office, and trial registry REST APIs.
• Built Pub/Sub and Kafka ingestion flows delivering content update events into Cloud Storage and BigQuery.
• Created Cloud Functions to validate vendor file arrivals and trigger downstream workflows.
• Built Dataproc PySpark jobs processing large volumes of scientific publication, citation, and patent records.
• Processed Parquet, Avro, JSON, and XML files with PySpark while handling schema evolution and tracking rejected records.
• Implemented Dataflow transformations standardizing author, institution, drug, and company identifiers across regional and third-party sources.
• Migrated legacy Hadoop workloads to Dataproc, reducing operational effort while keeping existing transformation and reconciliation logic.
• Built ELT models in BigQuery SQL tracking clinical trial status, drug pipeline stages, regulatory approvals, and competitive intelligence metrics.
• Designed dimensional models for real-world evidence reporting across de-identified claims, prescriptions, patient cohorts, and provider activity.
• Applied BigQuery materialized views to speed up recurring citation metrics and customer usage reports.
• Built Vertex AI pipelines classifying research documents and matching duplicate author and organization records.
• Orchestrated daily Dataproc, Dataflow, and BigQuery workflows in Cloud Composer (Airflow) with dependency checks and recovery steps.
• Configured VPC Service Controls, IAM permissions, and encryption for licensed content and de-identified patient datasets, enforcing data-use and privacy restrictions.
• Improved developer experience by standardizing Terraform modules and CI/CD pipelines for cluster, storage, and deployment setup.
• Used generative AI and LLM tools to automate documentation, test creation, and SQL development, speeding up delivery for the team.
• Automated provisioning of Dataproc clusters, storage buckets, service accounts, and networking with Terraform modules.
• Deployed pipeline changes through Jenkins, Git, Docker, and Kubernetes across development, test, and production environments.
• Monitored Spark and Dataflow jobs with Cloud Monitoring and tuned executor settings to improve processing efficiency.
• Built Looker datasets supporting product analytics, customer usage tracking, and life sciences market insights.
• Prepared source mappings, deployment notes, test evidence, and recovery procedures for quarterly enterprise releases.

Data Engineer - Merck Pharma (CTS - New Jersey) - New Jersey

(2021-07 - 2023-09)

• Collaborated with business stakeholders to gather and translate business requirements into scalable data engineering and analytics solutions.
• Developed and maintained dbt-based data transformation and validation workflows, implementing reusable models, data quality checks, lineage, and transformation logic to support reliable and governed analytical datasets.
• Designed and maintained Azure Data Factory pipelines integrating Azure Data Lake Storage, Snowflake, and downstream analytics platforms to support scalable batch and incremental data processing.
• Developed PySpark applications in Databricks for ETL transformations, cleansing, and enrichment of healthcare datasets.
• Designed and managed ETL pipelines using Azure ADF and Airflow, orchestrating batch and incremental loads.
• Modeled enterprise data marts using star-schema and dimensional design principles.
• Implemented real-time ingestion using Kafka + Azure Event Hubs.
• Migrated legacy Hadoop workloads into cloud-native pipelines (Snowflake, ADF, ADLS).
• Implemented data quality, validation, lineage, and reconciliation scripts using SQL and Python.
• Ensured compliance with HIPAA, GDPR, and enterprise data governance standards.
• Worked with global teams (US, India, Europe) to coordinate development, UAT, and production deployments.
• Deployed containerized ETL and microservice workloads on Azure Kubernetes Service (AKS), enabling auto-scaling, load balancing, and efficient orchestration of data processing tasks.
• Optimized PySpark jobs by fine-tuning partitioning, caching, and broadcast joins to improve execution performance on Databricks clusters.
• Implemented data interoperability pipelines supporting HL7 and FHIR messages, ensuring compliance with healthcare integration standards.
• Deployed containerized ETL workloads on Azure Kubernetes Service (AKS), handling deployment and production troubleshooting.
• Built automated data testing and lineage with dbt tests.