Observability Engineer (KUBERNETES & CONTAINERS, GITOPS & AUTOMATION, QUERY LANGUAGES, CLOUD & INFRASTRUCTURE)

Observability Engineer (KUBERNETES & CONTAINERS, GITOPS & AUTOMATION, QUERY LANGUAGES, CLOUD & INFRASTRUCTURE)

Toronto, On

Hybrid - 4 days

ABOUT THE ROLE

We are seeking an experienced Observability Engineer to join our Enterprise Kubernetes Platform team at a leading financial services organization. You'll own the complete observability stack across 50 production Kubernetes clusters, providing metrics, logging, tracing, and alerting capabilities that ensure exceptional reliability and performance for mission-critical applications.

This role combines deep technical expertise in modern observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting, and self-healing infrastructure.

========================================================================

WHAT YOU'LL DO

========================================================================

OBSERVABILITY STACK OWNERSHIP

--------------------------------------------------------------------------------

• Design, deploy, and maintain enterprise-scale observability infrastructure including Prometheus, Grafana, Thanos, Loki, and modern collection agents • Manage observability deployments using GitOps principles and infrastructure as code • Implement long-term metrics storage solutions with cloud object storage • Maintain and upgrade observability components across development, QA, UAT, production, and DR environments • Configure distributed observability architecture spanning multiple data centers and cloud providers METRICS & MONITORING

--------------------------------------------------------------------------------

• Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized applications • Create ServiceMonitors, PodMonitors for automated metrics collection • Develop rules for intelligent alerting with minimal false positives • Configure multi-cluster metrics federation and aggregation • Optimize metrics cardinality, storage e_iciency, and query performance • Implement recording rules for pre-aggregated metrics and SLI calculations DASHBOARDS & VISUALIZATION

--------------------------------------------------------------------------------

• Build comprehensive Grafana dashboards for infrastructure health, application performance, and business metrics • Create reusable dashboard templates and libraries for development teams • Implement dashboard-as-code • Configure multiple datasources • Design executive dashboards with SLO/SLI tracking and business KPIs • Implement role-based access control and multi-tenancy in Grafana LOGGING INFRASTRUCTURE

--------------------------------------------------------------------------------

• Deploy and manage centralized logging solutions (Loki, ELK, Splunk, or similar)
• Configure log collection agents (Promtail, Fluentd, FluentBit, Vector, etc.) • Design log retention policies balancing cost, compliance, and operational needs • Create LogQL/Lucene queries and log-based alerts • Implement log correlation with metrics and traces for unified troubleshooting • Build log aggregation pipelines with parsing, filtering, and enrichment ALERTING & INCIDENT MANAGEMENT

--------------------------------------------------------------------------------

• Configure intelligent alerting with Alertmanager or equivalent platforms • Design alert rules with appropriate severity levels, thresholds, and SLOs • Implement alert routing to Slack, PagerDuty, ServiceNow, email, and webhooks • Create automated runbooks and remediation workflows • Develop alert inhibition, silencing, and grouping strategies • Tune alerting to achieve signal-to-noise ratio improvements • Integrate with incident management and on-call rotation systems DISTRIBUTED TRACING

--------------------------------------------------------------------------------

• Implement distributed tracing solutions using OpenTelemetry, Jaeger, or Tempo • Instrument applications for trace collection and correlation • Configure trace sampling strategies for cost and performance optimization • Build trace-based dashboards for latency analysis and dependency mapping • Integrate tracing with metrics and logs for comprehensive observability AI/ML FOR OBSERVABILITY

--------------------------------------------------------------------------------

• Implement AI-powered anomaly detection for metrics and logs • Build predictive alerting using machine learning models to forecast issues before they occur • Develop intelligent alert correlation and root cause analysis systems • Integrate LLM-based tools for log analysis and troubleshooting assistance • Implement AIOps capabilities for automated incident triage and resolution • Use AI to optimize alert thresholds and reduce false positives • Build natural language query interfaces for observability data • Implement Model Context Protocol (MCP) for AI agent integration with observability platforms AUTOMATION & PLATFORM INTEGRATION

--------------------------------------------------------------------------------

• Automate observability deployment using GitOps workflows (FluxCD, ArgoCD) • Integrate with CI/CD pipelines for automated testing and validation • Build self-service portals for teams to create dashboards and alerts • Develop APIs and CLIs for observability automation • Integrate with secrets management solutions (Vault, AWS Secrets Manager) • Configure LDAP/AD/SSO authentication for observability platforms • Automate compliance reporting and audit logging ENABLEMENT & COLLABORATION

--------------------------------------------------------------------------------

• Onboard application teams to observability platforms • Provide guidance on instrumentation best practices • Create documentation, training materials, and self-service guides • Conduct workshops • Support development teams during incidents with observability insights • Collaborate with SRE, DevOps, and platform engineering teams PERFORMANCE & COST OPTIMIZATION

--------------------------------------------------------------------------------

• Monitor and optimize observability stack resource consumption • Implement autoscaling for stateless observability components • Tune data retention, compaction, and downsampling strategies • Conduct capacity planning for metrics and log storage growth • Optimize query performance and dashboard response times • Implement cost allocation and chargeback for multi-tenant environments =======================================================================

REQUIRED QUALIFICATIONS

=======================================================================

EXPERIENCE

--------------------------------------------------------------------------------

• 4-6 years of experience in observability, monitoring, SRE, or platform engineering roles • 3 years hands-on production experience with Prometheus and Grafana • 2 years working with Kubernetes and containerized environments • Strong expertise with PromQL for metrics querying and alerting • Experience deploying and managing observability stacks at scale (1000 nodes) • Proven track record of reducing MTTR through e_ective observability CORE TECHNICAL SKILLS

--------------------------------------------------------------------------------

OBSERVABILITY PLATFORMS

• Prometheus (including Prometheus Operator) • Grafana (dashboards, alerting, plugins) • Thanos, Cortex, or Mimir for long-term storage • Alertmanager or equivalent alerting platforms

LOGGING

• Loki, Elasticsearch/ELK Stack, Splunk, or CloudWatch Logs • Log collection agents (Promtail, Fluentd, FluentBit, Vector) • LogQL, Lucene, or equivalent query languages

TRACING

• OpenTelemetry (OTEL Collector, instrumentation) • Jaeger, Zipkin, Tempo, or AWS X-Ray • Trace sampling and correlation strategies KUBERNETES & CONTAINERS:
• Kubernetes architecture and operations (1.24 ) • Custom Resource Definitions (CRDs) and Operators • ServiceMonitor, PodMonitor, PrometheusRule resources • Kubernetes metrics (kube-state-metrics, node-exporter, cAdvisor) GITOPS & AUTOMATION:
• GitOps workflows (FluxCD, ArgoCD, or similar) • Infrastructure as Code (Kustomize, Helm, Terraform) • CI/CD platforms (GitHub Actions, GitLab CI, Jenkins) • Scripting (Python, Bash, Go) QUERY LANGUAGES:
• PromQL (expert level required)
• LogQL or Lucene
• Basic SQL for data analysis

CLOUD & INFRASTRUCTURE:

• AWS, Azure, or GCP cloud platforms
• Object storage (S3, GCS, Azure Blob)
• Multi-cloud and hybrid architectures
• vSphere or on-premises virtualization (nice to have) AI/ML & EMERGING TECHNOLOGIES

--------------------------------------------------------------------------------

• Experience with AI/ML frameworks for observability (Prophet, TensorFlow,

PyTorch)

• Anomaly detection algorithms and time-series forecasting • LLM integration for log analysis and troubleshooting (GPT, Claude, etc.) • Model Context Protocol (MCP) for AI agent integration • AIOps platforms (Moogsoft, BigPanda, Datadog Watchdog, etc.) • Natural language processing for log parsing and analysis • Familiarity with vector databases for semantic search (Pinecone, Weaviate) • Experience with AI-powered root cause analysis tools • Knowledge of prompt engineering for observability use cases PREFERRED QUALIFICATIONS

--------------------------------------------------------------------------------

• Kubernetes certifications (CKA, CKAD, or CKS) • Grafana certification or equivalent training • Knowledge of service mesh observability ( • Experience with SRE practices, SLIs, SLOs, and error budgets • Background in financial services or regulated industries • Familiarity with compliance requirements (SOX, PCI-DSS, etc.) • Contributions to open-source observability projects • Experience with APM tools