# Mission — Observability & Monitoring

## Why this course exists

**Course 6 of 7** in the deployment/infra track. Courses 1–5 built and shipped the system: a
**container** (C1) running as Kubernetes **objects** (C2), generated by **Helm** (C3), routed by
**Istio** (C4), delivered by **CI/CD** (C5). Now it's running in production — and something's
slow, or erroring, at 2am. **Observability is how you find out what, where, and why**, without
SSH-ing into a box.

Several threads from earlier courses already point here: `istio_requests_total` (C4) is
*scraped by* this stack and drives KEDA and alerts; the CI deploy alerts fire from here; the
Istio access logs and Zipkin traces (C4) land here. This repo runs a full **self-hosted** stack
— **Prometheus + Thanos + Grafana + OpenTelemetry Collector + Jaeger + Kiali + Grafana OnCall**,
with **GCP Cloud Logging** for logs. This course is how it all fits together.

## Who the learner is

- Backend engineer on the **conversationmgmt / notification / spike** team — strong Go, gRPC,
  Kafka, Postgres, Airflow; now solid on containers, K8s, Helm, Istio, and CI/CD (eleven courses
  in).
- **Newer to observability as a discipline.** Metrics/logs/traces, PromQL, SLOs, tracing,
  alerting are new as a *whole*. Assume strong backend maturity, NOT SRE fluency.
- Directly relevant: their services' `/metrics`, traces, and logs flow through this stack; when
  one of their services misbehaves in prod, this is where they'll look.

## What "success" looks like

By the end, the learner can, from memory:

1. Explain observability vs monitoring and the **three pillars** (metrics, logs, traces) — and
   the golden signals / RED / USE.
2. Explain Prometheus — metric types, the pull/scrape model, and PromQL — and how this repo
   scrapes services (the `prometheus.io/scrape` annotation).
3. Write basic PromQL (`rate`, `histogram_quantile`, aggregation) and read recording/alerting
   rules.
4. Explain **Thanos** (long-term storage, global query, dedup, downsampling) and **Grafana**
   (dashboards, datasources).
5. Explain **alerting & on-call** (Alertmanager / Grafana OnCall, severity, escalation) and
   **SLI/SLO/error budgets**.
6. Explain **distributed tracing** (spans, OTel, the Collector → Jaeger pipeline) and
   **structured logging** (Cloud Logging) — and how the three pillars correlate.
7. Read **Kiali** and debug an incident by moving across metrics → traces → logs.

## Grounding

Every concept lands on real config under `deployments/helm/platforms/monitoring/` and the
`internal/golibs/{metrics,tracer,logger}` packages — mapped in
[reference/repo-observability-map.md](./reference/repo-observability-map.md). External claims cite
the official Prometheus / OpenTelemetry / Grafana / Thanos docs ([RESOURCES.md](./RESOURCES.md))
— never my memory.

## The seven-course plan (this track)

1. Containers & Local Dev ✅
2. Kubernetes ✅
3. Helm ✅
4. Istio & Service Mesh ✅
5. CI/CD & Delivery ✅
6. **Observability & Monitoring** ← *you are here*
7. Infrastructure as Code — Terraform/GCP/GKE, the clusters this stack monitors