Naeem Yeager

Staff engineering leader with 10+ years designing and scaling cloud-native data platforms, distributed processing systems, and serverless infrastructure across real estate, healthcare, and fintech. Proven track record architecting Spark/Scala/PySpark pipelines, real-time streaming, AI workflow orchestration, and AWS-native microservices—driving ~50% faster pipeline execution and ~40% lower infrastructure costs while mentoring teams to deliver production-ready, observable systems.

Technical Team Lead / Staff Data Engineer - HouseCanary

(2022-06)

• Architected and led migration from legacy monolithic ETL infrastructure to a fully serverless Spark-based processing platform using AWS Glue, Lambda, EMR, and Redshift.
• Designed scalable Spark, Scala, and PySpark-based distributed data pipelines processing high-volume national property datasets.
• Reduced end-to-end pipeline execution times by 50% through Spark execution tuning, workload redesign, and distributed compute optimization.
• Architected and developed an internal AI orchestration platform enabling secure development, deployment, and coordination of domain-specific AI agents across Data Factory systems.
• Designed modular microservice-oriented infrastructure supporting AI agent execution, workflow orchestration, observability, and Slack-native operational tooling.
• Led development of an AI-assisted onboarding and configuration framework automating MLS integration workflows and execution management.
• Built extensible workflow execution systems enabling reproducible event-driven and file-driven automation pipelines.
• Developed reusable infrastructure abstractions supporting scalable Lambda-based deployment patterns across hundreds of internal services.
• Implemented centralized observability, execution tracing, and operational telemetry systems for AI workflows and distributed platform infrastructure.
• Managed and mentored distributed engineering teams while serving as technical lead across multiple platform initiatives.

Senior Data Engineer / Systems Architect - Phrase Health - Philadelphia

(2021-06 - 2022-05)

• Designed and implemented distributed Spark and PySpark pipelines for batch and near-real-time clinical data transformation and synchronization.
• Built scalable ingestion and transformation frameworks (Spark/Scala) with dataset lineage and versioning to support reproducible analytics.
• Developed reusable upsert/merge libraries optimized for distributed PostgreSQL operations and warehouse integration.
• Engineered a versioned lakehouse architecture to enable historical lineage, reproducible processing, and reliable rollback.
• Optimized Spark compute and resource allocation strategies, reducing infrastructure costs by ~40%.

Senior Software Engineer / Data Platform Engineer - Capital One - New York

(2017-05 - 2021-05)

• Developed distributed Spark and Scala data processing systems powering large-scale internal data platform workloads.
• Built enterprise data discovery infrastructure for dataset indexing, ingestion, and retrieval to accelerate analytics and ML adoption.
• Designed and maintained scalable ETL pipelines and event-driven processing patterns to ensure reliable data delivery.
• Implemented low-latency real-time streaming services to surface newly created datasets for downstream consumers.
• Led logging and observability standardization, improving telemetry, debugging workflows, and operational visibility across the platform.

Director of Training & Program Development - Code for Progress - Washington

(2016-01 - 2017-05)

• Led curriculum strategy, hiring, and instructional staff management to deliver community-focused software engineering training programs.
• Designed and launched training initiatives and external partnerships to expand program reach and improve participant outcomes.