Nihal Kumar Mothukuri

Designed and implemented cross-project data pipelines that consumed streaming messages from Pub/Sub, performed transformations, and published processed data to downstream Pub/Sub topics for consumption by multiple GCP projects. Built and managed Apache Airflow DAGs using Cloud Composer and Python to orchestrate complex data workflows, enabling reliable scheduling, monitoring, and dependency management of large-scale data pipelines. Implemented complex SQL transformations in BigQuery including joins, window functions, and aggregations to process high-volume transactional datasets supporting business analytics and reporting.

Data Engineer at American Express (2025-09 – Present)

• Designed and implemented cross-project data pipelines that consumed streaming messages from Pub/Sub, performed transformations, and published processed data to downstream Pub/Sub topics for consumption by multiple GCP projects.
• Built and managed Apache Airflow DAGs using Cloud Composer and Python to orchestrate complex data workflows, enabling reliable scheduling, monitoring, and dependency management of large-scale data pipelines.
• Implemented complex SQL transformations in BigQuery including joins, window functions, and aggregations to process high-volume transactional datasets supporting business analytics and reporting.
• Led migration of legacy on-premise data pipelines to the GCP-based Lumi ecosystem, redesigning ETL workflows using Dataflow, Pub/Sub, and BigQuery to improve scalability and reliability.
• Performed large-scale data migration and validation in BigQuery by implementing reconciliation checks, row-count validation, and checksum comparisons to ensure data accuracy between source and target systems.
• Developed automated data validation frameworks using Python and SQL in BigQuery to verify schema consistency, detect anomalies, and maintain high data quality across migrated datasets.
• Optimized BigQuery storage and query performance during migration by implementing partitioning, clustering, and efficient query patterns for large financial datasets.
• Implemented data validation and quality checks using Python and SQL to ensure accuracy, integrity, and consistency of financial transaction data across multiple GCP environments.
• Developed Apache Beam pipelines on Dataflow to process streaming events, perform encryption/decryption operations, and enrich data by querying BigQuery before publishing transformed messages to downstream systems.

Data Engineer at Amazon (2022-07 – 2025-03)

• Developed scalable ETL pipelines using Python and PySpark to process large-scale international shopping transaction data across multiple Amazon marketplaces, improving data processing efficiency by 40%.
• Built batch and real-time data ingestion pipelines using Apache Kafka and AWS Glue to integrate customer activity, order, and product catalog data into the centralized data lake on AWS S3.
• Implemented complex data transformations and aggregations using Apache Spark and SQL to support analytics for cross-border pricing strategies, inventory optimization, and international customer behavior insights.
• Designed and maintained dimensional data models in Amazon Redshift to support BI dashboards and reporting for international marketplace performance and sales analytics.
• Optimized Spark jobs and large-scale SQL queries using partitioning, caching, and performance tuning techniques, reducing processing time for multi-terabyte datasets by 35%.
• Developed automated workflow orchestration using Apache Airflow to schedule and monitor ETL pipelines, ensuring reliable daily processing of international marketplace datasets.
• Implemented data quality checks and validation frameworks using Python and SQL to ensure accuracy, completeness, and consistency of customer and order data across multiple regions.
• Collaborated with product managers, data scientists, and analytics teams to deliver curated datasets supporting recommendation systems, international pricing models, and customer experience improvements.

Data Engineer at Wipro Technologies (2017-10 – 2020-10)

• Designed and developed scalable ETL pipelines using Python and PySpark to process large-scale upstream and downstream operational datasets, improving data processing efficiency by 35%.
• Built real-time and batch data ingestion pipelines using Apache Kafka and AWS Glue to integrate data from drilling systems, production databases, and third-party APIs into the enterprise data lake.
• Implemented complex data transformations and aggregations using Apache Spark and SQL to support analytics for reservoir monitoring, production optimization, and operational reporting.
• Designed and maintained cloud-based data lake architecture on AWS S3 with partitioning and optimized storage formats to support structured and semi-structured datasets.
• Optimized Spark jobs and SQL queries through partitioning, caching, and performance tuning techniques, reducing large-scale data processing time by 40%.

Master of Science in Computer Science – California State University, Long Beach (2021-01 – 2022-05)

Bachelor of Engineering in Information Technology – Osmania University, Hyderabad, India (2013-08 – 2017-05)