Data Engineer - PivotX Advisors (Client: Firstsource | Cigna & Comcast)
(2026-06)
Co-developing the backend of an enterprise file processing and monitoring platform serving Cigna and Comcast, enabling automated ingestion and transformation of SLA/TAT report files.
- Building a Python backend system that actively monitors configured SharePoint folders for new files, automatically triggering ingestion and processing workflows upon file detection.
- Implemented a multi-agent AI pipeline that processes complex CSV and Excel configuration files to intelligently extract and map required schema information, supporting user-defined transformations including column renaming, column pivoting, and custom field mappings.
- Developed robust file processing error handling covering schema mismatch detection, schema change management, and corrupt file handling to ensure pipeline reliability in production.
- Built metrics configuration capabilities allowing business users to define and extract KPI and SLA/TAT metrics directly from ingested data columns without engineering intervention.
Data Engineer - PivotX Advisors (Client: Hitachi Digital Services)
(2025-09 - 2026-05)
Contracted to deliver data engineering solutions for Hitachi Digital Services, a Fortune 500 global technology company. Selected as the only fresher entrusted with the HR data engineering stream among the contractor team, and the only fresher to receive a performance bonus in recognition of the quality and impact of work delivered.
- Designed and implemented a config-driven HR data pipeline on Apache Airflow (GCP Cloud Composer) that reads active configurations from a BigQuery control table, scans raw datasets for unprocessed tables by prefix, and processes them in chronological order to ensure data integrity across multiple HR worker lifecycle use cases including new joiners, terminations, rehires, cross-company transfers, and garden leave scenarios.
- Implemented intelligent error handling and pipeline safety mechanisms — on successful processing, raw tables are archived and removed from the raw layer; on failure, tables are moved to an error archive, an HTML error report is sent via GCP Application Integration, and the configuration is automatically deactivated (is_active = 0) to prevent corrupt data from propagating downstream until the issue is resolved.
- Built comprehensive audit and observability infrastructure including batch-level and activity-level logging to BigQuery, trace ID based GCP Cloud Logging integration, structured error publishing via Pub/Sub, and execution duration tracking across all pipeline runs.
- Developed BigQuery stored procedures that process raw HR tables into an ODS layer, implementing deduplication and ranking logic using advanced multi-column window functions (ROW_NUMBER, RANK, LAG, LAST_VALUE) within CTEs, along with a reprocessing procedure for replaying records from specific time periods.
- Built BigQuery views on top of the ODS layer consolidating multiple HR lifecycle use cases into a unified Okta integration consumption layer, with MDM config-table driven date-range filtering to control which records are provisioned for each operating company.
- Engineered Airflow DAG concurrency control using DagRun state inspection to prevent duplicate pipeline executions, and implemented environment-aware scheduling across dev, staging, and prod environments.
- Built a comprehensive automated email alerting and reporting system via GCP Application Integration REST API — covering pipeline failure alerts with structured HTML error tables, daily data quality reports, CC mapping gap alerts, Okta provisioning summaries (success/failed/pending), and multi-environment scheduled summary reports.
Data Engineering Intern - BinaryChakra
(2025-01 - 2025-07)
- Built large-scale web scrapers using Scrapy (structured sites) and Selenium (dynamic JavaScript-heavy sites) to collect datasets from multiple external sources for downstream analytics.
- Designed data pipelines to transform and store scraped data in Supabase (PostgreSQL) and Google BigQuery, enabling efficient querying and downstream consumption.
- Built Shopify App Store and Partner Directory scraper extracting app metadata, reviews, and installation stats; delivered insights dashboard identifying market gaps and top-performing apps.
- Applied NLP techniques using spaCy and TextBlob for sentiment analysis, complaint categorization, and customer insight extraction from product review data.
- Fine-tuned Azure OpenAI models for domain-specific tasks with automated training pipelines, configurable hyperparameters, and integrated deployment monitoring.
