Jy Kim

Data Engineer with extensive experience in scalable ETL pipeline development, distributed system architecture design, and data quality assurance. Skilled in data mining, validation, DW/DM/DL(TB) and governance, delivering robust and high-performance pipelines for analytics and machine learning applications. Experienced in team leadership, workflow automation, and distributed data optimization across both batch and real-time environments.

Data Engineer - NewSelect - Seoul, Korea

(2026-03)

• Built GCP-based data platform using Airflow, BigQuery, GCS, Python, and Docker.
• Designed end-to-end pipelines integrating multiple e-commerce platforms, marketplace, and OMS/ERP data sources.
• Established layered warehouse architecture (raw → ods → dw → dm) and common product/order models.
• Optimized BigQuery performance and cost using partitioning, clustering, and snapshot/current models.
• Automated data quality checks including row count, null, aggregation, and change detection.
• Improved team productivity by 30–40%, reduced manual operations by 70%, and cut validation effort by 80–90%.

Big Data Engineer | Specialist - CJ Crop (OliveNetworks) - Seoul, Korea

(2023-01 - 2026-01)

• Designed and optimized distributed ETL pipelines integrating data from multiple CJ affiliates (TVing, ENM, CGV, Logistics, Foodvile) using Airflow, PySpark, Hive, and AWS Glue.
• Increased DW/DM/DL ingestion performance by 30% via Spark AQE, broadcast joins, partitioning, and Tez-based Hive optimization.
• Built and maintained data warehouse (Dw) pipelines on AWS S3 and Databricks, enabling scalable ingestion and unified storage for structured/unstructured data.
• Implemented automated HDFS file integrity validation and backup assurance with snapshots and checksums.
• Developed data quality governance workflows fully integrated into ETL pipelines.
• Built data mining pipelines for waybil classification ML models with clean, governed datasets.
• Operated global MES pipelines for CJ factories (US, Philippines) on Databricks.
• Mentored junior engineers and led optimization initiatives for distributed pipeline reliability.

Big Data Engineer - WEMAKEPRICE - Seoul, Korea

(2021-06 - 2023-01)

• Built batch ranking and recommendation pipelines (trending keywords, keyword-product matching) using PySpark and Airflow.
• Applied behavioral analytics and data mining techniques to optimize search and recommendation relevance.
• Automated RDB-to-HDFS ingestion with Sqoop, ensuring performance tuning and data validation checks across large-scale jobs.
• Centralized ETL orchestration in Airflow, reducing manual intervention and job failures.
• Data Analytics Engineer: Enhanced search pipelines with data cleansing, typo correction, stopword filtering, and brand keyword extraction.

Data Engineer - Korea Information Engineering Co. - Seoul, Korea

(2019-02 - 2021-06)

• Developed ETL pipelines for e-commerce transactions (orders, refunds, deliveries) using Python and MySQL.
• Designed Mysql DW/DM architecture to support analytics, reporting, and decision-making.
• Implemented web scraping automation for ingesting external partner data.
• Ensured data validation and reconciliation for financial dashboards, improving reporting reliability.

B.S. - Computer Engineering (SW) - University of Ulsan (2015 - 2019)