Sr. Software / Platform / Data Engineer

3-4 Months

About the Role: 

Build the technical implementation of a system that captures GitHub Copilot CLI agent traces across developer machines, scrubs sensitive content centrally in Azure, and serves a sanitized, queryable dataset for review. You’ll own the pipeline end-to-end — from the endpoint capture script to the Azure ingestion, scrub, and viewer.

You are a fit - If you can take a written design and ship the working pipeline mostly on your own, and you treat privacy as a hard requirement, not an afterthought.

What does day-to-day look like:

• Stand up the OpenTelemetry capture path: endpoint env-var/.bashrc setup, OTLP export, and a collector pipeline (receivers → processors → exporters).

• Build the central ingestion + privacy scrub block in Azure (secret + PII detection/redaction) before any data reaches a reviewer.

• Wire up the sanitized stores and a read-only trace viewer, with task↔session↔trace correlation.

Required Skills:

• Bachelor’s degree in Computer Science, Engineering, or a related field — or equivalent practical experience.

• 5+ years in software / platform / data engineering, shipping production systems.

• 3+ years hands-on with Azure (or a comparable major cloud with clear transfer to Azure PaaS), including 2+ years building observability or telemetry pipelines with direct OpenTelemetry experience — OTLP, collectors, spans/traces, semantic conventions.

• Azure PaaS: Container Apps/AKS, API Management, ADX/Kusto or App Insights, ADLS Gen2/Blob, Key Vault, Entra ID.

• Solid Python or Node.js, plus Bash scripting.

• Practical data privacy / PII & secret scrubbing experience (regex, entropy, redaction/tokenization).

• Data-pipeline and cloud-security fundamentals (RBAC, private networking, audit).

Nice to Have:

• Grafana / telemetry-backend setup; LLM-agent or GenAI observability exposure; cross-cloud identity federation (AAD ⇄ GCP IAM).