Lesson 17 · Senior observability judgment

Reviewing observability designs like a senior engineer

How to review metrics, alerting, tracing, logging, and dashboards for operational usefulness, truthfulness, and cost-aware design.

Your win: review observability designs for hidden weakness, false confidence, and unnecessary cost instead of stopping at “we added telemetry.”

In plain English Plain English: senior observability review asks whether the telemetry will actually help a human understand and respond to the next real incident.

Why observability review is different from instrumentation review

Instrumentation review asks whether signals were emitted. Observability review asks whether the resulting system is strong enough to support diagnosis, ownership, and action under pressure. That is a wider question.

This distinction matters because a design can be technically “instrumented” and still weak operationally. The metrics might be too noisy, the labels too expensive, the pages misrouted, the traces sampled in an unhelpful way, or the logs too unstructured to explain failure. Senior review is where those weaknesses get noticed before they become production habits.

The review question What production question is this telemetry supposed to answer, how reliably can it answer it, and what operational cost or blind spot comes with that design?

Where strong review comments usually land

metrics that add labels faster than they add usable signal
alerts that technically detect a problem but do not cleanly identify an owner or urgency
SLO language that overstates platform maturity
trace sampling stories that hide what evidence is actually preserved
logs that are structured enough to ship but not structured enough to correlate or explain
dashboards that look rich but do not help answer a debugging sequence

Backend use case This lesson helps you review repo changes around Prometheus metrics, alert rules, Alertmanager routing, OTel Collector sampling, Cloud Logging fields, Grafana dashboards, and Kiali/Jaeger integration with sharper questions.

Common mistake Assuming telemetry is valuable just because another graph, exporter, or panel was added.

What the strongest review mindset sounds like

The strongest comments in this area are rarely “add more telemetry.” They are usually sharper than that: “this label will create avoidable cardinality,” “this page has no clear owner,” “this dashboard looks rich but still doesn’t answer the incident flow,” or “this trace policy changes what evidence survives.”

That is the mindset to practice. You are not grading observability by volume. You are grading it by usefulness, honesty, and operational consequence.

Use the observability map as the review anchor

The best reviews in this area check claims against the real stack topology and runtime flow, not just the intention behind the change.

→ Repo observability map
→ docs/observability/lessons/0012-kiali-whole-picture.html

Check yourself (from memory)

Q1. What is the most senior observability review question?

Senior review is about usefulness, truthfulness, and cost — not raw volume.

What should a senior engineer ask when reviewing an observability design?

recall, then click to reveal

What question the telemetry answers, how dependable that answer is during an incident, and what cost, routing weakness, or blind spot the design still carries.

Want a compact observability-review checklist for PRs in this repo? Ask me.

Sources. Repo observability map; existing observability lessons; official docs.