Shiva Kumar

Senior Data Engineer with 5+ years of experience in architecting, building, and operating cloud-native, distributed data platforms in banking and enterprise environments. Extensive hands-on expertise in Python, SQL, Apache Spark, Databricks, Azure, AWS, and Snowflake, with deep focus on end-to-end data architecture, including source ingestion, change data capture (CDC), batch and real-time streaming pipelines, distributed processing, and optimized storage layers. Proven ability to design fault-tolerant, scalable data pipelines using Spark, Kafka, and cloud-native orchestration frameworks, ensuring high availability, data consistency, and SLA compliance.

Strong background in data modeling (dimensional, denormalized, and analytical schemas), partitioning strategies, and file format optimization (Parquet, ORC, AVRO, Delta) to support large-scale analytical workloads. Experienced in implementing data quality frameworks, validation rules, and monitoring/alerting mechanisms to ensure data reliability across complex pipelines. Adept at performance tuning through optimized joins, caching strategies, indexing, and resource configuration across Spark and cloud data warehouses.

Hands-on experience with data governance and security, including role-based access control (RBAC), row-level security (RLS), data masking, and auditability to meet regulatory and compliance requirements. Skilled in establishing CI/CD pipelines for data platforms, automating deployments, and enforcing coding and architectural standards. Recognized for partnering closely with product owners, data scientists, analysts, and business stakeholders to translate complex requirements into robust, production-grade data solutions, while mentoring engineers and driving best practices in Agile delivery environments.

Data Engineer - Molina Healthcare

(2023-08)

• Designed and implemented a Healthcare data platform ingesting patient, provider, medical and pharmacy claims, and eligibility data to support HEDIS quality measures, RAF risk adjustment analytics, and operational dashboards.
• Architected end-to-end batch ingestion pipelines using Azure Data Factory to extract data from SQL Server systems into ADLS Gen2.
• Utilized Databricks to write PySpark transformations like groupBy (), filter () and transform () to process large-volume structured and semi-structured datasets, applying business rules and data validations.
• Implemented the Delta within Azure Databricks to write only the incremental delta records into the refine folder of the HES Data lake to be consumed for the Azure Cognitive Search (ACS) canonical model.
• Implemented incremental load strategies using watermarking and changing data capture (CDC) techniques to reduce processing time and avoid full data reloads.
• Employed Pandas and NumPy within Azure Databricks, integrating with services using Azure Data Lake Storage and Azure Synapse Analytics for efficient processing of large datasets and advanced analytics capabilities.
• Enabled historical and trend analysis by implementing slowly changing dimension (SCD) handling in data models.
• Conducted performance tuning for long-running jobs by optimizing joins, caching strategies, and resource configurations. Optimized Spark workloads, reducing pipeline execution time by over 30%.
• Built reusable Spark frameworks to standardize data cleansing, schema enforcement, and error handling across pipelines.
• Integrated Kafka for near real-time ingestion of transaction and event data, enabling faster availability for downstream analytics.
• Modeled curated datasets using dimensional and flattened schemas to support reporting, risk analytics, and regulatory use cases.
• Developed automated data quality checks to validate record counts, null thresholds, referential integrity, and business rules.
• Implemented logging, auditing, and alerting mechanisms to track pipeline failures and data anomalies.
• Collaborated with compliance and security teams to enforce data access controls and masking of sensitive financial information.
• Designed and implemented row-level security (RLS) to ensure data access based on user roles and regulatory requirements.
• Published curated datasets for analytics teams and enabled self-service reporting through Power BI.
• Set up CI/CD pipelines using Git and Jenkins to automate deployment of data pipelines and code changes.
• Supported production issues by analyzing logs, identifying root causes, and implementing long-term fixes.
• Partnered with business analysts to translate reporting requirements into scalable data models.
• Documented architecture, data flows, and operational runbooks for ongoing support and knowledge transfer.
• Participated in Agile ceremonies, providing estimates, tracking progress, and delivering incremental enhancements.

Data Analyst - HDFC Bank

(2020-04 - 2021-12)

• Designed a real-time data processing platform to support fraud detection and risk monitoring for high-volume financial transactions.
• Applied window functions rank, dense rank, and aggregation logic accordingly to identify suspicious activity patterns and anomalies.
• Implemented batch pipelines using Streams & Tasks to persist enriched transaction data into Snowflake for analytics.
• Tuned SQL queries and indexes to improve reporting performance for high-volume datasets.
• Built data validation checks null, datatype, to ensure data completeness and consistency across streaming and batch layers.
• Built streaming ingestion pipelines using Kafka to capture transaction events from core banking applications.
• Integrated streaming outputs with downstream data stores for both real-time dashboards and historical analysis.
• Automated orchestration of batch jobs using Apache Airflow dags, task, operators, ensuring reliable scheduling and dependency management.
• Implemented retry logic, dead-letter queues, and error handling to improve pipeline resilience by using Airflow on_failure_callback.
• Designed interactive reports in Microsoft Excel with PowerPivot and advanced formulas (INDEX-MATCH, SUMIFS, VLOOKUP), improving KPI tracking and performance analysis for operations teams.
• Developed Python-based feature extraction pipelines to support fraud scoring and risk models.
• Enabled near real-time dashboards using Tableau for fraud analysts and operations teams.
• Retrieved KPI metrics using fact & dimensional tables to track fraud rates, alert volumes, and investigation outcomes.
• Designed data schemas optimized both real-time access and long-term reporting.
• Worked closely with data scientists to operationalize model inputs and outputs within data pipelines.
• Supported A/B testing of fraud rules and thresholds by maintaining versioned datasets.
• Reduced end-to-end data latency by over 40% through streaming optimizations.
• Provided on-call production support for critical fraud detection pipelines.
• Documented data flows, pipeline logic, and operational procedures for audit and compliance reviews.
• Collaborated with cross-functional teams to continuously refine fraud detection metrics and data logic.
• Ensured high availability and reliability of pipelines supporting 24/7 fraud monitoring operations.