Course 6 of 7 · The deployment/infra track
Observability & Monitoring, from newcomer to interview-ready
How you see what your running system is doing — metrics, traces, and logs — when something breaks at 2am.
Courses 1–5 built and shipped the system. Now it's in production and something's slow.
Observability is how you find out what, where, and why — without SSH-ing into
a box. This repo runs a full self-hosted stack: Prometheus + Thanos + Grafana + OpenTelemetry
+ Jaeger + Kiali + Grafana OnCall, with GCP Cloud Logging. Several threads you already
know feed it: istio_requests_total is scraped here, the deploy alerts fire here,
the Istio traces & access logs land here. Three parts, built one at a time. Lessons are
short — one win each. Read them in order.
Part 1 — Fundamentals & metrics available now
The three pillars, and the metrics pillar in depth — Prometheus & PromQL.
1 · What observability is
The three pillars (metrics/logs/traces) + the golden signals.
ready2 · Metrics & Prometheus
Counter/gauge/histogram/summary; the pull model; scraping.
ready3 · How this repo scrapes
The prometheus.io/scrape annotation, /metrics, golibs/metrics.
4 · PromQL & rules
rate, histogram_quantile; recording vs alerting rules.
Part 2 — Scale, dashboards & alerting available now
Storing metrics forever, visualizing them, and turning them into pages.
5 · Thanos
Long-term storage & a global query view across Prometheus.
ready6 · Grafana
Dashboards & datasources — the visualization layer (dashboards-as-code).
ready7 · Alerting & on-call
Alertmanager → Grafana OnCall — routing by label, escalation to Twilio.
ready8 · SLI / SLO / error budgets
Turning metrics into reliability targets and burn rates.
readyPart 3 — Traces, logs & the whole picture available now
The other two pillars, and putting all three together to debug an incident.
9 · Distributed tracing
Spans & traces — following one request across services.
ready10 · OTel Collector & Jaeger
The receiver→processor→exporter pipeline; tail sampling; the trace backend.
ready11 · Logs
Structured logging & GCP Cloud Logging; correlating with traces.
ready12 · Kiali & the whole picture
The mesh graph, and debugging across all three pillars.
readyPart 4 — Senior observability judgment new
The senior stretch — signal selection, alert quality, SLO maturity, sampling/cost trade-offs, review judgment, and interview-grade answers.
13 · Signal judgment
Choose useful telemetry and avoid cardinality traps that make the platform noisier and more expensive.
new14 · Alert quality
Separate useful pages from noisy ones and reason about routing and escalation design.
new15 · SLO maturity
Explain threshold-based reliability targets honestly and name the gap to fuller burn-rate operations.
new16 · Sampling & cost
Reason about telemetry cost, evidence quality, and cross-pillar debugging under imperfect data.
new17 · Reviewing observability designs
Review metrics, alerts, traces, logs, and dashboards like a senior engineer.
new18 · Senior interview questions
High-signal observability questions with the right answer shape.
new19 · Mock senior interview pack
A retrieval-first mock round with scoring and strong answer points.
newReference shelf
Cheat sheet
Dense revision sheet + PromQL + interview one-liners.
Glossary
The canonical vocabulary, opinionated.
Repo map
Ground truth: Prometheus/Thanos/Grafana/OTel/Jaeger config.
Resources
Prometheus/OTel/Grafana docs, the SRE book, the PCA cert, plus senior observability judgment topics.
Senior backend playbook
The compressed version of the new senior part: signal judgment, alert quality, SLO maturity, sampling/cost trade-offs, review judgment, and interview answers.