Eric Tran

Senior Data Engineer with 12+ years of experience across Coinbase and Databricks, progressing from full-stack and backend engineering into distributed data platforms, streaming systems, and AI-ready infrastructure. Hands-on with Python, SQL, Spark/PySpark, Kafka, Airflow, Snowflake, Databricks, Delta Lake, and AWS across batch, CDC, and near-real-time workloads. Experienced in data modeling, quality, governance, lineage, observability, cost optimization, and infrastructure automation.

Senior IC who combines hands-on implementation with architecture ownership, mentoring, and cross-team technical influence.

Senior Software Engineer - Data Platform & AI at Databricks (2022-06 – Present)

• Designed an internal AI/RAG data platform on top of Delta Lake, Unity Catalog, Mosaic AI Vector Search, MLflow, and Model Serving, giving application teams a reusable production architecture for ingestion, retrieval, evaluation, governance, and deployment rather than rebuilding those capabilities for each use case.
• Built PySpark and Spark SQL pipelines processing millions of document chunks per cycle from S3-backed Delta datasets. Stable document/chunk IDs, content hashes, and incremental processing avoided full-corpus reprocessing and cut unnecessary parsing and embedding work by roughly 60% on frequently refreshed datasets.
• Defined platform-wide data contracts covering source identity, document versions, ACL metadata, chunk lineage, embedding versions, and vector-index state, making replay, deletes, schema changes, permissions, and reproducibility explicit rather than hidden inside individual pipelines.
• Combined pipeline-level schema, completeness, freshness, and reconciliation checks with MLflow evaluation for retrieval quality and RAG regressions; a technically successful Spark job could still be blocked from promotion when downstream retrieval behavior degraded.
• Established Unity Catalog RBAC and lineage conventions across structured and unstructured assets, carrying source permissions through document, chunk, and vector-index metadata so restricted content was enforced at retrieval time.
• Orchestrated ingestion, document processing, embedding, index synchronization, and evaluation through Databricks Lakeflow Jobs; Terraform, Asset Bundles, and GitHub Actions promoted the same reviewed configuration across dev, staging, and production, reducing new-environment setup from roughly 1-2 days to a few hours.
• Owned operational health and cost/performance tuning for the reference workloads - source freshness, Spark skew, parser failures, embedding retries, vector-index lag, retrieval regressions, and serving dependencies. Partition sizing, incremental reprocessing, batching, and workload-specific freshness SLOs kept expensive AI processing tied to actual business freshness requirements.
• Led RFCs and architecture reviews with Data Engineering, ML Engineering, Security/Governance, and application teams, mentoring engineers on failure handling and architectural tradeoffs and turning recurring solutions into documented platform standards. The work built on Databricks products; not claiming ownership of the product teams that developed Delta, Unity Catalog, Vector Search, or Lakeflow.

Senior Software Engineer - Data Platform at Coinbase (2020-05 – 2022-05)

• Worked on Coinbase's Kafka/MSK data platform while the broader environment processed billions of events per day, focusing on reliable CDC ingestion and a consistent onboarding path for transactional data sources.
• Standardized Kafka Connect configurations around topic naming, business keys, offsets, secrets, observability, and Schema Registry compatibility, replacing source-by-source conventions with a repeatable platform pattern.
• Solved the snapshot-to-stream boundary by recording a safe source position, intentionally overlapping historical and live ingestion, and relying on deterministic, idempotent merge semantics to prevent silent gaps or duplicate state.
• Built connector and consumer telemetry that separated idle producers, unhealthy connectors, broker pressure, and downstream consumer lag; this made production incidents diagnosable before teams defaulted to increasing capacity.
• Used Spark Structured Streaming for Kafka-to-Delta CDC workloads where micro-batch processing, Delta integration, replay semantics, and existing Spark operational expertise mattered more than introducing a separate low-latency streaming runtime. Pilot datasets reached roughly 10-15 minute freshness.
• Generalized source-specific logic behind a configuration-driven ingestion contract-topic, decoder, key columns, target table, append/merge mode, checkpoint, and trigger interval, so additional databases could reuse the same pipeline architecture.
• Codified Kafka topics, ACLs, connectors, and environment configuration with Terraform, making platform changes code-reviewed and repeatable and reducing manually maintained production configuration.
• Contributed to Coinbase's Airflow 1.10 - 2.x modernization, building compatibility checks and migration tooling and helping move 40+ production workflows through staged testing and cutover while domain teams retained ownership of their business logic.

Software Engineer - Backend & Data Platform at Coinbase (2014-06 – 2020-04)

• Began in full-stack engineering across customer-facing applications, backend APIs, and transactional services, then moved progressively toward backend reliability, observability, and data-platform problems as Coinbase scaled.
• Built Airflow pipelines for 20+ transactional and operational datasets, establishing a repeatable ingestion path from production systems into Snowflake for Product, Finance, Compliance, and analytics.
• Reworked expensive full-refresh jobs around incremental watermarks, overlap windows, staging, and deterministic deduplication; several high-volume pipelines ran roughly 40-60% faster while preserving safe retries, replay, and backfills.
• Introduced pipeline-level quality controls for freshness, uniqueness, nullability, schema drift, source-to-target reconciliation, and financial balances, moving many failures from downstream dashboards into observable data-pipeline checks.
• Developed 30+ Snowflake fact, dimension, and analytical models, centralizing transaction, account, and customer logic in reviewed SQL rather than duplicating definitions across reports and dashboards.
• Resolved semantic disagreements with Product, Finance, and Compliance around transaction state, settlement timing, and customer attributes before encoding those definitions into shared analytical datasets and Looker models.
• Migrated a high-volume logging workload away from Coinbase's shared Elasticsearch environment during periods when the broader platform generated terabytes of logs per hour, using workload isolation, buffered ingestion, Datadog/Kibana monitoring, load testing, and staged cutover rather than simply adding cluster capacity.

Bachelor in Computer Science – University of Texas - Austin (2010 – 2014)