# Senior observability playbook

Compressed reference for the senior-only observability extension. Pair with [cheat-sheet.md](./cheat-sheet.md), [GLOSSARY.md](./GLOSSARY.md), and [repo-observability-map.md](./repo-observability-map.md).

## The senior shift
Intermediate understanding is: "I know the repo’s Prometheus, Thanos, Grafana, alerting, SLO concepts, tracing, logs, and Kiali stack."

Senior understanding is: "I can explain which signals are worth keeping, where alert quality breaks down, how honest our SLO maturity really is, what telemetry costs us, and how to review the observability design for usefulness rather than volume."

## Six senior lenses
1. **Signal choice beats signal volume** — instrumentation must answer real production questions without cardinality blowups.
2. **Alert quality is human quality** — a technically correct rule can still be a bad page.
3. **SLO maturity has levels** — threshold alerts are useful, but not the same as full error-budget operations.
4. **Sampling is an evidence policy** — it changes what the trace view can and cannot prove.
5. **The three pillars are a workflow** — metrics quantify, traces localize, logs explain.
6. **Review is about usefulness and honesty** — ask what the telemetry answers, what it costs, and where it still leaves you blind.

## Senior review checklist
- Does this metric use stable labels, or is it quietly creating high-cardinality cost?
- Does this alert deserve a page, or only a dashboard/ticket?
- Do the routing labels clearly identify the right owner and urgency?
- Are we describing threshold-based alerting as if it were full SLO maturity?
- What incidents could trace sampling still hide from us?
- Do the logs still explain the concrete failure context after metrics and traces narrow it down?
- Does this dashboard help answer a debugging sequence, or only look visually rich?

## Interview answer shapes
### Why is “measure everything” not mature observability?
Because telemetry must stay queryable, affordable, and useful. More signals are only valuable if they help answer real incidents without causing cardinality or noise explosions.

### What makes an alert high-quality?
It is actionable, urgent enough for its channel, and routed to the right human through labels, grouping, and escalation design.

### How would you describe the repo’s SLO maturity honestly?
It uses SLO-like p95/error-rate alert thresholds today, but does not yet have full first-class SLO objects, explicit error budgets, or burn-rate operating workflows.

### Why does sampling matter beyond cost?
Because it determines which traces remain as detailed debugging evidence and which incidents must be reconstructed from metrics and logs instead.

### What is the real job of a senior observability reviewer?
To ask what production question the telemetry answers, how reliable that answer is under pressure, and what cost, routing weakness, or blind spot the design still carries.