Shruthi

Data Engineering professional with 8+ years of experience building reliable, production-grade data pipelines, ETL/ELT workflows, and data integration solutions across enterprise and financial data environments. Strong hands-on expertise in Python, SQL, Databricks, Apache Spark (PySpark) and modern Lakehouse patterns (Bronze/Silver/Gold). Strong hands-on experience developing scalable Python and Java frameworks for enterprise data processing and ETL applications.

Experienced in building cloud-native data engineering solutions using AWS Glue, Amazon S3, AWS EMR, AWS Lambda, Amazon Redshift, Athena, and Step Functions. Hands-on experience writing PyTest unit and integration test cases for Python and PySpark applications. Experienced in developing and consuming RESTful Web Services and integrating enterprise applications using APIs.

Strong knowledge of Unix Shell scripting for deployment automation, monitoring, and production support. Proven track record of designing end-to-end ETL/ELT pipelines using Azure Data Factory / Fabric Data Factory, Databricks Workflows, and orchestration tools. Deep understanding of Spark internals including RDD transformations/actions, caching/persistence, partitioning, shuffle behavior, and performance tuning for large workloads.

Experienced in creating Delta Lake tables with ACID compliance, schema evolution, MERGE/UPSERT, time travel, and optimized file layouts for scalable analytics. Strong SQL engineering background—writing and tuning complex queries, data validation logic, and warehouse-friendly transformations for reporting and downstream consumption. Hands-on experience building deployment automation with CI/CD (Azure DevOps / GitHub Actions) using YAML pipelines, approvals, RBAC, and environment-based releases (Dev/Test/Prod).

Skilled in building secure, production-grade data solutions using Key Vault, secrets management, service principals, and access controls (RBAC/Unity Catalog patterns). Designed ingestion frameworks pulling data from ERP systems (SAP, Oracle, Infor M3), APIs, files (CSV/JSON/XML), and databases into cloud platforms.

Experience implementing data quality and reconciliation checks (row counts, hash totals, schema checks, duplicate detection, referential checks) to ensure trusted data products. Comfortable with interactive development in Databricks notebooks / Jupyter, building reusable utilities and parameterized jobs for scalable execution. Familiar with orchestration patterns using Airflow (including Dockerized deployments) and legacy scheduling ecosystems (e.g., Oozie/Control-M concepts).

Strong

DevOps mindset—version control, branching strategies, code reviews, unit testing, job monitoring, alerting, and operational runbooks for supportability.

Experience with data governance and metadata—documenting KPI definitions, lineage, and standards for consistent reporting across teams. Collaborative communicator who partners with stakeholders to translate business requirements into performant data

Senior Data Engineer - Daikin - Waller, TX

(2024-01)

• Designed and delivered scalable Databricks + PySpark pipelines processing large operational datasets to power analytics and reporting.
• Implemented Lakehouse medallion architecture (Bronze/Silver/Gold) with standardized ingestion, transformations, and curated datasets.
• Built and maintained Delta Lake tables using MERGE/UPSERT patterns, handling late-arriving data, schema evolution, and incremental loads.
• Developed reusable Python frameworks for data validation, transformation utilities, logging, and parameterized job execution.
• Developed reusable Python and Java frameworks for enterprise ETL processing, service integration, logging, and exception handling.
• Developed PyTest automation suites for validating Python and PySpark applications, improving code quality and deployment reliability.
• Configured and supported Markit EDM workflows for enterprise data ingestion, validation, enrichment, matching, and distribution — ensuring high-quality, trusted data products for analytics and downstream consumption.
• Designed and maintained Markit EDM data models and integration processes to onboard data from internal systems, external vendors, flat files, and APIs into the EDM platform with consistent schema handling and transformation logic.
• Built exception handling, reconciliation, and monitoring processes within Markit EDM to detect and route data quality failures, reducing manual intervention and improving SLA adherence for downstream reporting pipelines.
• Implemented data enrichment and matching rules in Markit EDM to standardize and cross-reference records across multiple source systems, improving data consistency and reference data accuracy across the enterprise.
• Managed end-to-end data distribution workflows from Markit EDM to downstream analytical systems and reporting layers, ensuring timely, validated data delivery aligned to business SLAs.
• Collaborated with cross-functional stakeholders including data engineers, analysts, and product teams to translate business requirements into Markit EDM workflow configurations and integration specifications.
• Tuned Spark workloads by optimizing partitioning, caching, broadcast joins, cluster sizing, and shuffle configs, improving runtime and stability.
• Performed peer code reviews, architecture validation, and coding standard compliance using Git-based pull request workflows.
• Automated deployment validation through PyTest and CI/CD pipelines.
• Created ADF/Fabric-style ingestion patterns using Copy activities, parameterized pipelines, triggers, and metadata-driven configurations.
• Integrated data from enterprise sources including databases, flat files, and REST endpoints; standardized ingestion with consistent schema handling.
• Built cloud-native ETL solutions using AWS Glue, Amazon S3, Lambda, EMR, Step Functions, and Redshift for large-scale data processing.
• Developed and consumed RESTful Web Services for integrating enterprise applications and external data providers.
• Implemented CI/CD for data assets using Azure DevOps YAML pipelines for notebooks/jobs/pipeline JSON with gated approvals and release controls.
• Improved observability by implementing job-level metrics, failure alerts, and runbook documentation for production support.
• Worked closely with BI teams to deliver analytics-ready datasets that reduced report refresh times and improved data trust.
• Led software architecture design for enterprise Lakehouse platform applying design principles including loose coupling, high cohesion, and fault tolerance across ingestion, transformation, and serving layers.
• Drove migration of legacy Hadoop/Hive ecosystems to cloud-native Databricks/Spark architecture, enabling modern CI/CD-driven deployments and reducing operational overhead.
• Established Secure Application Development standards for the data engineering team, incorporating threat modeling, credential vaulting, network isolation, and audit logging into all pipeline development workflows.
• Maintained and extended SSIS-based legacy integrations during cloud transition phases, ensuring business continuity while modern pipelines were built and validated in parallel.
• Authored comprehensive Functional System Design Documents for all major data products, covering architectural diagrams, data flow specs, business rule mappings, SLA definitions, and rollback procedures.
• Designed Stored Procedures and SQL Server transformation logic for downstream analytical layers, ensuring performance, correctness, and alignment with data warehouse standards.
• Championed Secure SDLC methodologies across the team — conducting code reviews with security lens, enforcing RBAC at Unity Catalog level, integrating automated secrets scanning in CI/CD gates.
• Architected scalable ingestion frameworks using software design patterns.