Casey Boyle

Data engineer with 10+ years building large-scale batch and streaming pipelines on AWS, GCP and Databricks. Led a migration of 1,000+ Spark jobs to PySpark and Spark SQL, and built CDC streaming and self-service lakehouse infrastructure on GCP. Previously scored 12M+ domains daily at SecurityScorecard and processed tens of billions of geolocation records per day at Gravy Analytics. GCP Professional Data Engineer.

Data Engineer - Rearc

(2024-06)

• Migrated 1,000+ Java Spark (RDD) jobs to PySpark and Spark SQL on Databricks, replacing job-specific code with a config-driven (YAML/JSON) framework and reusable transformation templates
• Automated build, test and deployment with GitLab CI/CD, Databricks Asset Bundles and pytest
• Built pipelines across Snowflake, S3 and Delta tables, with access governed through Unity Catalog
• Built real-time CDC streaming from Cloud Spanner Change Streams through Dataflow (Apache Beam, Java) to Pub/Sub for external clients
• Added Pub/Sub ordering keys to guarantee chronological delivery, with separate ordered and unordered topics for different consumers
• Deployed infrastructure declaratively with Config Connector in a Bazel-built Java monorepo
• Built self-service tooling for a large-scale lakehouse migration that generates Kubernetes and GCP resource specs from validated Java configs
• Enabled per-service provisioning of Cloud Storage, BigQuery and Cloud Spanner with CMEK encryption via Cloud KMS

Software Developer - Gravy Analytics - Remote

(2023 - 2024)

• Maintained the core ETL platform processing tens of billions of geolocation data points per day with Spark on AWS (S3, EMR, DynamoDB)

Senior Software Engineer - SecurityScorecard - Remote

(2021 - 2023)

• Built Spark (Scala), Airflow and Databricks pipelines on AWS that scored 12M+ domains daily
• Productionized machine learning models for cybersecurity ratings as Spark applications
• Built a change-detection workflow that publishes alert events to SQS for customer notifications
• Migrated data from HDFS to S3, Redis and ScyllaDB with Airflow-orchestrated Databricks Spark jobs and staged cutover validation
• Designed date-partitioned S3 data layouts and tuned Spark jobs (partitioning, caching, cluster sizing) to reduce skew, small files and cost
• Mentored junior engineers and conducted technical interviews

Software Engineer - SecurityScorecard - Remote

(2017 - 2021)

• Delivered end-to-end platform features as a full-stack engineer, from data pipelines and APIs to the customer-facing UI

Quality Reliability Engineer, Data Quality Analyst - SecurityScorecard - Remote

(2015 - 2017)

• Automated bulk data validation in Python and attributed internet assets to organizations

B.S. - Psychology - University of San Francisco (2007)