Paul Chong

Senior Data Engineer with 9 years of experience specializing in AI data platforms, RAG pipelines, and governed data infrastructure at petabyte scale. Experienced in building cost-efficient data systems for large-scale AI shopping and enterprise data marketplace platforms, delivering 42% lower AI infrastructure cost and reducing data access approval time from 3.2 days to 1.2 hours. Deep expertise in PySpark, AWS, OpenSearch, Kafka/MSK, and data quality frameworks with a focus on reliable, governed, and scalable AI data systems.

Senior Data Engineer | Amazon Rufus - Amazon - Remote

(2023-01)

• Led the design and delivery of petabyte-scale RAG data pipelines for Amazon Rufus using AWS EMR, PySpark, S3, DynamoDB, OpenSearch, and Lambda, enabling product search, comparison, and AI-generated shopping answers across 200M+ monthly queries.
• Built near-real-time product embedding pipelines that ingest product listings, reviews, Q&A, and image metadata using PySpark micro-batches and OpenSearch vector indexing, improving retrieval freshness and answer relevance for Rufus shopping experiences.
• Architected an agentic Auto-Buy data pipeline using AWS MSK, Lambda, DynamoDB transactions, Step Functions, and SNS, processing real-time price-change events with strong ordering, idempotency, and purchase validation controls.
• Optimized AI data infrastructure cost and performance by migrating EMR Spark workloads to Graviton3, enabling dynamic resource allocation, and replacing full scans with S3 Select, reducing infrastructure cost per Rufus query by 42%.
• Evaluated Apache Airflow as an orchestration alternative to AWS Step Functions; led proof-of-concept migration for 3 pipelines, reducing DAG complexity by 25% and improving cross-team pipeline visibility.
• Developed a knowledge graph data quality framework using Great Expectations, Deequ, AWS Glue, and SNS alerts to validate product entities, relationship freshness, and required attributes, reducing Rufus hallucination-related data incidents by 82%.
• Mentored 3 junior engineers, led design reviews with ML, product, and retail teams, and authored internal RFCs on cost-effective RAG data pipelines for large-scale AI shopping systems.

Data Engineer | Amazon DataZone - Amazon - Remote

(2020-01 - 2023-12)

• Built core data marketplace infrastructure for Amazon DataZone using AWS Step Functions, Lambda, DynamoDB, OpenSearch, Glue, Redshift, and S3, helping 1,200 internal teams discover, govern, and access 120,000+ datasets.
• Designed serverless metadata crawlers across Glue, Redshift, and S3 using Lambda, Step Functions, DynamoDB state tracking, and EventBridge scheduling, improving catalog freshness and reliability for enterprise data discovery.
• Developed subscription and approval workflows using API Gateway, Lambda, DynamoDB, SNS, IAM, and CloudTrail, reducing data access approval time from 3.2 days to 1.2 hours through automated provisioning.
• Implemented glossary synchronization pipelines using AWS Glue PySpark, S3, PostgreSQL, and CloudWatch Events, standardizing business definitions across data teams and improving consistency in reporting and analytics.
• Implemented dbt models to standardize transformations on Redshift, establishing reusable metric definitions across 5 data domains and reducing ad hoc SQL inconsistencies flagged in data quality reviews.
• Contributed to reusable DataZone blueprint templates using CloudFormation, AWS Service Catalog, CodeBuild, and Python validation scripts, accelerating analytics environment setup for governed data workspaces.
• Led crawler reliability improvements, authored internal postmortems, ran knowledge-sharing sessions on serverless metadata pipelines, and mentored one new graduate engineer on AWS data platform development.

Data Engineer | Koalas - Databricks - New York, NY

(2016-01 - 2020-12)

• Built core Koalas DataFrame and Series APIs using Python, pandas, PySpark, Spark SQL functions, and Koalas InternalFrame, helping data scientists scale pandas-style workloads on Apache Spark without rewriting code.
• Implemented high-use DataFrame methods including .filter, .drop_duplicates, and .rename, translating pandas semantics into Spark logical plans while improving pandas API coverage from 40% to 55%.
• Developed Series operations including .value_counts, .unique, and .isnull using PySpark SQL functions and pytest coverage, improving usability for distributed data exploration and reducing migration friction from pandas to Spark.
• Improved open-source engineering velocity by triaging GitHub issues, migrating CI from Travis CI to GitHub Actions, and reducing test runtime from 45 minutes to 15 minutes through parallelized Spark-version testing.

Bachelor of Science - Computer Science - Calvin University (2013-01 - 2016-01)