Nithya Kumari

Accomplished Senior Data Engineer with 6+ years architecting enterprise-scale data platforms across AWS, Azure, and GCP. Expert in building real-time pipelines using Apache Kafka, Spark, and Airflow, migrating legacy systems to cloud warehouses like Snowflake and Redshift. Proficient in Python, SQL, dimensional modeling, and implementing data governance frameworks using modern DataOps practices.

Senior Data Engineer - Toronto, Ontario, Canada

• Engineered scalable data pipelines using Python and Apache Spark to process terabytes of data daily, orchestrating workflows through Apache Airflow while ensuring data quality validation with Great Expectations framework
• Implemented real-time streaming solutions leveraging Apache Kafka and Spark Streaming to ingest customer events, transforming raw data into actionable insights stored in Delta Lake for analytics consumption
• Designed dimensional models following Star Schema principles using Erwin for enterprise data warehouse on Snowflake, optimizing query performance through clustering keys and materialized views for reporting teams
• Migrated legacy ETL processes from Informatica PowerCenter to cloud-native AWS Glue jobs, reducing processing time by implementing partitioning strategies on S3 with Parquet format for storage efficiency
• Built end-to-end data orchestration framework using Apache Airflow and dbt to manage dependencies across Redshift transformations, scheduling incremental loads from multiple source systems into warehouse tables
• Developed streaming applications with Kafka Streams and Apache Flink to process clickstream data in real-time, writing aggregated metrics to PostgreSQL and publishing alerts through RabbitMQ queues
• Architected multi-cloud data platform integrating AWS S3, Azure ADLS Gen2, and GCP Cloud Storage using Apache NiFi for seamless data movement while maintaining lineage through Apache Atlas metadata repository
• Optimized BigQuery datasets by implementing partition pruning and clustering strategies, creating federated queries across Cloud SQL instances while monitoring performance through Stackdriver and Cloud Logging dashboards
• Automated infrastructure provisioning using Terraform scripts for AWS EMR clusters, configuring IAM roles and VPC security groups while orchestrating deployments through GitHub Actions CI/CD pipelines for consistency
• Created serverless data processing workflows using AWS Lambda functions triggered by S3 events, transforming JSON payloads and loading results into Aurora databases with error handling via CloudWatch alarms
• Established data governance framework using Collibra for metadata management, implementing access controls through AWS Lake Formation while cataloging assets in Glue Data Catalog for discoverability across teams
• Built distributed processing jobs using PySpark on Azure Databricks clusters, reading from Event Hubs streams and writing to Synapse Analytics dedicated pools after applying business transformation logic
• Configured Kubernetes deployments on EKS for containerized Docker applications, managing secrets through AWS IAM and monitoring cluster health using Prometheus and Grafana visualization dashboards for observability
• Developed change data capture pipelines using Apache Kafka connectors to replicate transactional data from Oracle and SQL Server databases into Snowflake staging tables for downstream analytics processing
• Implemented data quality checks using Monte Carlo for anomaly detection across Redshift tables, configuring alerting rules that notified teams via webhooks when metrics exceeded threshold boundaries defined
• Orchestrated batch processing jobs on GCP Dataproc clusters using Apache Beam SDK, reading from Pub/Sub topics and writing aggregated results to Cloud Spanner for low-latency transactional queries
• Designed RESTful APIs using Python and deployed on Azure Functions with OAuth2 authentication, connecting to Azure SQL Database and caching frequent queries in Blob Storage for performance
• Created data lineage tracking across Teradata migrations to Snowflake using Alation catalog, documenting transformation logic and maintaining version control through Git repositories for audit compliance requirements
• Built real-time dashboards in Tableau and Power BI connecting to AWS Athena queries on S3 data lakes, enabling business users to analyze metrics with row-level security through IAM integration
• Configured streaming ingestion from Kinesis data streams into Redshift using AWS Lambda micro-batching, applying Encryption at Rest and In Transit while logging activities through CloudTrail for compliance
• Developed Scala applications on Apache Spark for processing semi-structured data from HDFS, implementing custom partitioners and serializers to optimize shuffle operations across YARN managed cluster resources
• Established CI/CD pipelines using Jenkins and GitLab CI to automate deployment of SQL stored procedures to MySQL and PostgreSQL databases, running integration tests before production releases
• Implemented data mesh architecture on Azure Synapse Analytics with domain-oriented ownership, using Data Factory for cross-domain data sharing while maintaining quality standards through Great Expectations test suites
• Configured disaster recovery strategies for AWS RDS Aurora clusters across regions, automating failover procedures using Bash scripts and validating replication lag through CloudWatch custom metric monitoring
• Migrated monolithic ETL workflows to microservices architecture on GCP GKE, deploying Docker containers managed by Helm charts while implementing distributed tracing using X-Ray for debugging performance bottlenecks
• Developed streaming analytics using Confluent Platform schema registry for Apache Kafka message validation, processing events with Apache Storm topologies and persisting results into HBase for operational queries
• Coordinated Agile sprints following Scrum methodology to deliver data platform features, collaborating with DevOps teams on ECS container deployments while maintaining documentation in ER/Studio for architecture standards
• Performed complex joins across multiple large datasets ensuring performance and scalability, developed Spark SQL and DataFrame-based transformations for business logic, built analysis-ready datasets with validation checks (nulls, duplicates, reconciliation)
• Strong analytical experience working with large-scale operational and manufacturing datasets, performing data validation, completeness checks, and root-cause analysis to improve data quality and reliability
• Extensive experience using advanced SQL for ad-hoc analysis, anomaly detection, and data investigation across complex datasets
• Experience working with operational and production data, ensuring data accuracy, traceability, and consistency across systems
• Proven ability to collaborate with business and operational teams to resolve data issues and improve data processes
• Experience working with Lakehouse architecture using Medallion (Bronze, Silver, Gold) layers for structured data processing and analytics