Course 6 of 7 · The deployment/infra track

Observability & Monitoring, from newcomer to interview-ready

How you see what your running system is doing — metrics, traces, and logs — when something breaks at 2am.

Courses 1–5 built and shipped the system. Now it's in production and something's slow. Observability is how you find out what, where, and why — without SSH-ing into a box. This repo runs a full self-hosted stack: Prometheus + Thanos + Grafana + OpenTelemetry + Jaeger + Kiali + Grafana OnCall, with GCP Cloud Logging. Several threads you already know feed it: istio_requests_total is scraped here, the deploy alerts fire here, the Istio traces & access logs land here. Three parts, built one at a time. Lessons are short — one win each. Read them in order.

How to use this Do one lesson, take its quiz from memory (no peeking), then skim the matching row of the cheat sheet. Come back a day later and re-take it — spacing beats cramming. Your eleventh course; you know the rhythm. Stuck? Ask me — I'm your teacher, not just the author.
Where observability sits in the stack Everything before this ran your system. Observability watches it: the metrics it emits (Prometheus), the traces of requests through it (OTel/Jaeger), and the logs of what happened (Cloud Logging) — surfaced in Grafana & Kiali, paging you via alerts.

Part 1 — Fundamentals & metrics available now

The three pillars, and the metrics pillar in depth — Prometheus & PromQL.

1 · What observability is

The three pillars (metrics/logs/traces) + the golden signals.

ready

2 · Metrics & Prometheus

Counter/gauge/histogram/summary; the pull model; scraping.

ready

3 · How this repo scrapes

The prometheus.io/scrape annotation, /metrics, golibs/metrics.

ready

4 · PromQL & rules

rate, histogram_quantile; recording vs alerting rules.

ready

Part 2 — Scale, dashboards & alerting available now

Storing metrics forever, visualizing them, and turning them into pages.

5 · Thanos

Long-term storage & a global query view across Prometheus.

ready

6 · Grafana

Dashboards & datasources — the visualization layer (dashboards-as-code).

ready

7 · Alerting & on-call

Alertmanager → Grafana OnCall — routing by label, escalation to Twilio.

ready

8 · SLI / SLO / error budgets

Turning metrics into reliability targets and burn rates.

ready

Part 3 — Traces, logs & the whole picture available now

The other two pillars, and putting all three together to debug an incident.

9 · Distributed tracing

Spans & traces — following one request across services.

ready

10 · OTel Collector & Jaeger

The receiver→processor→exporter pipeline; tail sampling; the trace backend.

ready

11 · Logs

Structured logging & GCP Cloud Logging; correlating with traces.

ready

12 · Kiali & the whole picture

The mesh graph, and debugging across all three pillars.

ready

Reference shelf

Cheat sheet

Dense revision sheet + PromQL + interview one-liners.

Glossary

The canonical vocabulary, opinionated.

Repo map

Ground truth: Prometheus/Thanos/Grafana/OTel/Jaeger config.

Resources

Prometheus/OTel/Grafana docs, the SRE book, the PCA cert.
All 12 lessons are built — the Observability course is complete. From the three pillars through Prometheus/Thanos/Grafana/alerting to tracing, logs, and Kiali, you can now follow any signal from your Go code to the dashboard that wakes you. Next: Course 7 (Infrastructure as Code) — the final course. Or — twelve courses' worth of lessons, still no retention check — ask me to run a mock interview and I'll record where you're solid.