William Sims

Senior Data Engineer with a decade experience of designing and building high-volume data platforms, realtime pipelines, and analytics systems that drive mission-critical decisions across e-commerce, government, and enterprise environments. Expert at transforming messy, high-scale data into reliable, SLA-driven workflows — spanning streaming architectures, distributed storage, dbt modeling, and cloud orchestration. Recognized for elevating engineering quality through sound technical judgment, hands-on optimization, and simplifying complex systems, enabling teams to deliver faster and with confidence.

Senior Data Engineer - Shopify - San Francisco, CA

(2022-02)

• Engineered ELT pipelines for merchant, catalog, order, and financial data across tens of thousands of stores, ensuring reliable, up-to-date analytics for product, finance, and operations teams.
• Streamlined high-volume ingestion pipelines processing billions of events daily, with SLA-driven orchestration, automated backfills, and late-arrival handling for peak traffic stability.
• Built analytics-ready dbt models and curated datasets supporting experimentation, reporting, and product analytics across merchant and operational domains.
• Developed automated data quality checks and alerting workflows for schema drift, freshness, and pipeline failures, reducing reporting inconsistencies and operational incidents.
• Built analytics-ready datasets and KPI reporting layers consumed by finance, operations, and BI teams through Power BI and Looker dashboards.
• Partnered with finance and product stakeholders to deliver operational reporting and financial analytics models supporting executive decision-making.
• Supported experimentation and A/B testing analysis by building analytics-ready datasets and trusted KPI models for product and operations teams.
• Re-architected Kafka consumer microservices, boosting throughput from 25 → 2,500 msgs/s, delivering near real-time merchant insights and reducing downstream processing delays.
• Tuned ClickHouse clusters for high-concurrency OLAP queries, cutting dashboard latency using optimized partitions, skip indexes, and table engine configurations.
• Partnered with product and engineering teams to instrument new application events and improve data quality monitoring, enabling more reliable downstream analytics and faster issue detection.
• Transitioned Airflow workflows to Mage, lowering pipeline failure rates and accelerating recovery across hundreds of scheduled jobs.
• Built dbt transformation layers with hundreds of staging, intermediate, and mart models, improving test coverage, lineage visibility, and developer efficiency.
• Developed real-time event-processing and segmentation services with Kafka, PySpark, and AWS components, powering personalization and internal behavioral analytics.
• Implemented incremental ingestion pipelines with idempotency, schema-drift handling, and audit logs, stabilizing data flow from multiple transactional systems.
• Created internal monitoring tools for data freshness, volume, and schema integrity, reducing incidents and strengthening confidence in downstream reporting.
• Contributed to cloud infrastructure automation and deployment workflows using GitHub Actions and infrastructure-as-code practices to improve pipeline reliability and delivery speed.
• Collaborated with product and engineering leads to design data models supporting merchant features, billing workflows, catalog search, and operational reporting.
• Established standardized data contracts and schema governance across ingestion and transformation layers, minimizing duplication and preventing downstream issues.
• Mentored junior and mid-level engineers, promoting best practices in pipeline design, dbt modeling, distributed debugging, and code review standards.

Senior Data Engineer - Palantir Technologies - Denver, CO

(2019-08 - 2022-01)

• Led design and development of a multi-sector government analytics platform processing terabyte-scale datasets across Tax, Education, and Health domains, orchestrated via ~200 Airflow DAGs across Raw → HLP → Dimension → Gold → QA layers.
• Migrated legacy SQL Server and ETL workflows into Azure-native lakehouse pipelines using Databricks, Delta Lake, and cloud-based orchestration patterns.
• Built a generic ETL framework standardizing ingestion, validation, and transformation, reducing new-pipeline development from weeks → days and end-to-end runtime from 10 → 5 hours.
• Designed medallion-style Bronze, Silver, and Gold data layers to improve data quality, governance, and downstream analytical consistency.
• Engineered ingestion pipelines from MySQL, MS SQL, CSV, Excel, REST APIs, web scraping, and PDF extraction, enabling unified cross-department analytics access.
• Designed SCD Type 2/4 dimensional models supporting YoY/MoM reporting, historical tracking, and longitudinal analysis across millions of citizen-level records.
• Developed a geographic name-normalization layer reconciling inconsistent region, district, and locality naming, eliminating recurring manual cleanup cycles.
• Built an LLM-assisted product-name normalization pipeline for Arabic vendor variants, improving match accuracy across tens of thousands of SKUs.
• Applied systematic query optimization including indexing, partitioning, skip indexes, and join rewrites, reducing heavy analytical query runtimes by minutes and stabilizing peak-load performance.
• Automated RBAC, anonymization, and sensitive-field masking, cutting manual governance work by 90% while ensuring compliance across agencies.
• Implemented automated DDL backups and schema-drift detection, saving 4+ hours per week and preventing downstream breakages during vendor updates.
• Deployed an on-premises data lakehouse with Azure Databricks, Delta Lake, Trino, and ClickHouse, supporting batch and interactive analytical workloads.
• Built CI/CD pipelines using Azure DevOps, ensuring consistent deployment of notebooks, jobs, and SQL models across dev/stage/prod environments.
• Collaborated with policy analysts, data scientists, and agency stakeholders to translate regulatory requirements into scalable data models and reproducible workflows.
• Led sprint planning, backlog refinement, and cross-team architecture reviews, aligning delivery across multi-agency, multi-year analytics initiatives.

Data Engineer - Slalom Consulting - Seattle, WA

(2017-10 - 2019-07)

• Designed and delivered end-to-end data platforms for enterprise clients, building ingestion and transformation pipelines processing millions of records daily across operational and analytical workloads.
• Built and maintained Azure Data Factory pipelines orchestrating ingestion, transformation, and loading workflows across enterprise data platforms.
• Implemented low-latency streaming pipelines with Spark Structured Streaming, reducing data availability delays from multi-hour batch windows → sub-minute refresh cycles for event-driven systems.
• Deployed and maintained Azure-hosted database environments, supporting multiple production workloads with routine tuning, automated backups, and secure access configuration.
• Built Azure DevOps CI/CD pipelines, cutting deployment time from 30–40 minutes of manual steps → 5–7 minute fully automated pipelines.
• Developed a dynamic SQL Builder enabling business teams to generate custom KPIs through a UI, eliminating dozens of recurring ad-hoc SQL requests per month.
• Created reusable ingestion templates for APIs, CSVs, and relational systems, reducing new-pipeline setup from days → hours.
• Integrated MLflow for model versioning and experiment tracking, supporting concurrent machine learning model iterations for data science teams.
• Partnered with analysts and architects to translate business requirements into scalable data workflows for reporting, forecasting, and operational dashboards.
• Participated in architecture reviews, sprint planning, and delivery checkpoints across multi-client consulting projects, ensuring consistency and quality in solution delivery.

Software Engineer Intern - InfluxData - San Francisco, CA

(2016-06 - 2017-09)

• Contributed to TimescaleDB's PostgreSQL extension layer, implementing C/C++ and Python features utilized in production by internal teams and open-source users.
• Added inheritance support to the catalog layer, enabling flexible schema modeling for workloads with 10k–100k related entities.
• Implemented mathematical function operators in the query execution path, boosting analytical query performance on large numeric arrays by 1.4×.
• Optimized metadata lookups and execution-path branching, reducing complex query overhead by 8–12 ms per call in microbenchmarks.
• Collaborated with senior engineers on open-source workflows, performing code reviews, regression testing, and performance benchmarking across PostgreSQL 9.x releases.

Bachelor of Science - Computer Science - University of Illinois (2012-04 - 2016-05)