Joe Schaefer
Principal Engineer | Systems, Performance & Reliability | Team-Building IC
Hire this person
Send a job offer directly to this candidate
About
Principal-level individual contributor who stands up engineering teams and ships P&L outcomes: cost out of cloud bills, nines onto SLAs, and products into production. Builds the platform, the practice, and the people who keep both running—Perl, Python, C/C++/XS, Apache, GitOps, and 2026 AI-assisted delivery.
Experience
President / Principal Engineer at SunStar Systems, Inc. (1999 – Present)
- Designed and shipped Orion, a secure Apache-based CMS/wiki; Incremental Build Technology is pending patent. Integrated Oracle Cloud + OCI AI as four-continent HA infrastructure aimed at 99.99% SLA.
- Delivered contracting outcomes in software, infosec, and IT for clients from startups to enterprises—revenue through finished systems, not retainers for activity.
- Open-sourced the Orion SSG toolchain (Apache License 2.0) and CPAN/systems libraries (sealed.pm, Algorithm::LCS::XS, pty, orthrus) that other teams can adopt without a services dependency.
Principal Optimization Engineer at Clear Demand, Inc. (2024-07 – 2026-02)
- Scaled the RSIM price-optimization engine by several orders of magnitude (allocation arena + GKE swap), improving reliability while saving ~$60K/year in cloud compute.
- Added fork() and POSIX threads for multiprocess throughput; shipped Python bindings to librsim.so so batch-terminal tribal knowledge could be retired.
- Built the linter / CI / AI / Markdown / unit+regression harness for RSIM+CEVAL—the release path that protected the patent-pending optimization work.
- Coauthored pending patent on joint price-optimization methods.
GitLab Instructor at 321 Gang (2023-01 – 2024-01)
- Stood up a billable GitLab practice (bid and delivery) and earned GitLab Professional Services Engineer certification.
Staff Core SRE at Sharefile (2023-03 – 2023-07)
- Ported Elasticsearch 2.x to OpenSearch 7.x and migrated the EU platform—cutover completed on a four-month subcontract.
Performance Architect at Royal Caribbean (2022-06 – 2022-07)
- Designed LoadRunner-based performance test architecture for an IBM mainframe → SaaS migration and evaluated vendor SOWs against performance risk.
Vice President, Lead Cloud Engineer at Blackstone (2021-03 – 2022-04)
- Saved more than $1M/year in cloud spend by converting Kubernetes Guaranteed pods to Burstable without dropping the workloads those pods ran.
- Completed mass migration of datacenter assets to AWS (EKS, GitOps, CI/CD, GitLab); left monitoring and delivery on a cloud footing instead of a dual-running estate.
- Found a mission-critical file-descriptor leak in Datadog’s Kubernetes cron tooling and shipped automated weekly cross-account EKS repair so availability did not wait on a vendor patch.
Lead Platform Performance Architect; Lead Data Quality Engineer at Magic Leap (2018-01 – 2020-03)
- Raised cluster SLO from <90% to >99.9% and instituted HA, change-control policy, and Ansible Vault + Rancher Secrets across the performance-tuned MSA cluster.
- Stood up division-wide Jenkins CI for all Gerrit codebases and IaC/CaC CD (daily staging, weekly production).
- Wrote the ML2 memory-management architecture document that set the next-generation product’s MM constraints.
- Ported bpftrace into Debian-on-Android (adeb) for AMD64 and ARM64; delivered live mutex hold/wait and leak analysis used to close real performance defects.
- Built a 4-person Data Quality team around an SLI/error-budget contract; replaced OpenGrok with a peak-performance AOSP search utility.
Director of IT, Chief Scientist at Bio-Tech Medical Software (Biotrack / Forian) (2014-08 – 2017-12)
- Re-architected per-state traceability to HA (redundant firewalls/switches, load-balancing, failover, no RDBMS SPOF); doubled deployed servers and introduced Puppet.
- Built a 3-engineer systems team and a Git-branch + Puppet-environment model that scaled as the company won new state contracts.
- Met 99.9% uptime SLAs, passed KirkpatrickPrice SOC 2, and ran Qualys vulnerability scanning plus Icinga2/Xymon/Tarsnap→Glacier backup.
- Replaced copy-paste traceability code with a modular MVC Perl framework and Django templates so additional states were a configuration problem, not a fork.
Senior System Administrator (team lead) at The Apache Software Foundation (2006-04 – 2014-06)
- Led 4 full-time remote sysadmins supporting ~4,000 developers; owned IT budgets up to $700K/year.
- Conceived and built the Apache CMS (NoSQL, mod_perl, ZFS, LDAP) that hosted 100+ project sites including OpenOffice and Thrift.
- Co-implemented Orthrus OTP two-factor auth in C before 2FA was standard; handled a public security incident with disclosure later cited by Bruce Schneier.
- Sustained 30M+ hits/day across 3 datacenters; wrote Infrabot so on-call coordination scaled with the staff, not with heroics.
Education
Ph.D. in Mathematics – Stony Brook University (1997)
M.S. in Mathematical Physics (1994)
B.S. in Mathematics (1991)

