Bhargavi N

Results-driven Data Engineer with around 9+ years of experience building scalable and automated data ecosystems across healthcare, retail, and finance. Expertise in AWS, ETL design, PySpark, and orchestration tools like Airflow. Skilled in data modeling, data governance, and performance tuning, delivering secure, optimized data pipelines and actionable business insights.

Developed and optimized PySpark-based transformations in Palantir Foundry, handling incremental loads, schema evolution, and large-scale joins across distributed datasets. Improved data reliability and performance for mission-critical reporting and decision-making. Built and maintained end-to-end data pipelines in Palantir Foundry using code repositories and pipeline orchestration, enabling automated data refreshes, strong lineage, and governed access to trusted datasets for analytics and business users.

Extensively worked on AWS Cloud services like EC2, VPC, IAM, RDS, ELB, EMR, EKS, Auto-scaling, S3, CloudFront, Glacier, Elastic Beanstalk, Lambda, Elastic Cache, Route53, OpsWorks, CloudWatch, CloudFormation, Redshift, DynamoDB, SNS, SQS, SES, Kinesis, Firehose, Cognito IAM. Hands-on experience in Unified Data Analytics with Databricks, Databricks Workspace Interface, Managing Databricks Notebooks, Delta Lake with Python, and Delta Lake with Spark SQL. Expertise in using Hadoop infrastructures such as MapReduce, Pig, Hive, Zookeeper, Sqoop, Oozie, Flume, Drill, and Spark for data storage and analysis, and programming with Scala, Python, SQL, and NoSQL databases such as HBase, MongoDB, Cassandra.

Efficient in writing Infrastructure as Code (IaC) in Terraform, Azure Resource Management (ARM), and AWS CloudFormation. Created reusable Terraform modules in both Azure and AWS Cloud platforms. Built CDC pipelines using AWS DMS and streaming frameworks, implementing watermarking, bookmarks/checkpointing, and idempotent MERGE upserts.

Modeled data warehouses in Amazon Redshift and Snowflake using star/snowflake schemas, SCD Type 1/2 dimensions, conformed facts, and surrogate keys to eliminate filter ambiguity. Tuned performance through predicate pushdown, column pruning, broadcast joins, skew mitigation, and repartition/coalesce strategies targeting 128–256 MB file sizes. Orchestrated workflows with Apache Airflow using task groups, retries/backoff, dataset-aware scheduling, backfills, and SLA-backed alerting to Slack/Email.

Enforced data quality and contracts through pandas/dbt tests, row counts, hash totals, referential integrity checks, schema drift monitors, and DLQ + replay paths. Hardened security and compliance with IAM least-privilege, KMS encryption, Secrets Manager, Lake Formation LF-tags, PII/PHI masking, and audit trail enforcement. Automated delivery pipelines via CI/CD using CodePipeline, CodeBuild, and Infrastructure as Code (Terraform/CloudFormation) with blue/green deployments.

Delivered measurable outcomes: faster refresh windows, stable SLAs, reduced computing/storage c

Senior Data Engineer - PG&E - Lehi Utah

(2026-03)

Designed and developed scalable data pipelines using Snowflake, Informatica IDMC, Airflow, and UC4 to support enterprise utility data processing and reporting needs.

• Built and maintained Snowflake medallion architecture layers, including raw/bronze, refined/silver, and curated/gold data models for analytics and downstream consumption.
• Developed complex Snowflake SQL procedures, views, tasks, and data transformation logic to process large volumes of structured and semi-structured data.
• Integrated data from multiple source systems using Informatica IDMC and loaded curated datasets into Snowflake for business intelligence and operational reporting.
• Created and optimized ELT workflows in Snowflake by applying partitioning, clustering, query tuning, and warehouse sizing best practices to improve performance and reduce cost.
• Automated and scheduled production data workflows using Airflow and UC4, ensuring reliable job orchestration, dependency management, and failure recovery.
• Worked with Snowlite/Snowflake-based internal tooling to validate, monitor, and support data loads across different environments.
• Implemented data quality checks, reconciliation logic, and exception handling to ensure accuracy, completeness, and consistency across Snowflake medallion layers.
• Collaborated with business analysts, application teams, and reporting users to understand requirements and deliver reliable curated datasets for analytics and compliance reporting.
• Supported production releases, troubleshooting, incident resolution, and performance tuning for snowflake pipelines in a senior data engineering capacity.
• Designed process documentation and internal training sessions for data governance and pipeline optimization.
• Maintained coordination with offshore and onsite teams, ensuring timely sprint deliverables.

Senior Data Engineer - MSC Industrial Supply - Melville, NY

(2023-08 - 2026-01)

Built and maintained end-to-end data pipelines in Palantir Foundry, transforming raw source data into reliable, analytics-ready datasets. Focused on creating seamless data flows that served as the backbone for business intelligence, analytics, and reporting teams.

• Designed and implemented bronze, silver, and gold datasets in Foundry to establish a clear Medallion architecture. This framework improved data quality and standardization while significantly boosting downstream performance for complex analytical queries.
• Developed complex PySpark and SQL transformations within Foundry to handle sophisticated business logic. Implemented robust logic to support incremental loads and schema evolution, ensuring the system remained resilient to upstream data changes.
• Implemented Foundry Ontology models to represent core business entities, bridging the gap between technical tables and business concepts. This enabled intuitive data relationships and empowered stakeholders with true self-service analytics capabilities.
• Orchestrated data workflows using Foundry pipelines, managing the scheduling and execution of tasks across multiple domains. Ensured that ingestion, transformation, and delivery processes were synchronized and met rigorous enterprise SLA standards.
• Monitored and supported production Foundry pipelines, taking proactive steps to troubleshoot failures and validate outputs. Focus remained on ensuring SLA-driven data availability to prevent disruptions in critical business decision-making.
• Performed rigorous data validation and quality checks within Foundry pipelines to maintain high standards of accuracy. Developed automated checks to verify the completeness and consistency of enterprise datasets before they reached the end user.
• Analyzed data lineage and dependencies in Foundry to conduct thorough impact assessments for proposed system changes. This visibility allowed for effective change management and helped prevent cascading failures in downstream reporting.
• Built CDC-style (Change Data Capture) ingestion patterns within Foundry to maintain real-time synchronization with upstream transactional systems. This ensured that analytical datasets accurately reflected the most current state of the business operations.
• Optimized Foundry transformations for performance by fine-tuning Spark logic and implementing efficient partitioning strategies. Specifically addressed join patterns and shuffle operations to reduce resource consumption and execution time.
• Collaborated closely with product owners and business teams to translate abstract requirements into scalable Foundry data models. Acted as a technical advisor to ensure that data structures were aligned with long-term strategic business goals.