Lesson 14 · Senior observability judgment

Alert quality, noisy pages, and escalation design trade-offs

How to tell the difference between a useful page and a distracting one, and why routing and escalation design matter as much as the PromQL condition itself.

Your win: explain how a senior engineer evaluates alert quality, chooses what should page vs ticket, and reasons about routing and escalation honestly instead of treating every threshold breach as equally important.

In plain English Plain English: a good alert is not the one that notices everything. It is the one that wakes the right person for the right problem at the right urgency.

Why noisy alerting is an observability failure

It is tempting to think alerting quality is mostly about PromQL thresholds. But a technically correct rule can still be operationally bad if it pages too often, pages the wrong team, or pages for symptoms that no human needs to react to immediately.

This is one of the clearest places where observability stops being just measurement and becomes human-systems design. The page is part of the product. If the wrong person gets woken up, or the right person gets woken up for noise often enough, the monitoring stack is doing damage even when the rule expression is mathematically correct.

The alert rule An alert is high-quality when it is actionable, urgent enough for its channel, and routed to the human most able to respond.

Why routing labels matter as much as expressions

A strong alert expression without clear routing labels still creates operational confusion. Someone has to own the page. Someone has to know whether it belongs to backend, platform, notification, spike, or another team. That ownership story is carried in labels and routing design, not just the numeric threshold.

The repo is a good teaching case because the routing tree is explicit. Alerts carry labels like severity and app, Alertmanager groups and routes them, and Grafana OnCall turns that into a schedule and escalation path. The signal is not complete until it can reach the right human correctly.

Backend use case In this repo, Prometheus rules fire with routing labels, Alertmanager groups and deduplicates them, and team-specific Grafana OnCall webhooks plus Slack/Twilio escalation decide how humans are actually notified.
Common mistake Treating an alert as “done” once the expression works, without checking whether the ownership and escalation path are also correct.

What stronger alerting judgment sounds like

A mature answer here does not just define warning vs critical. It explains the cost of false urgency, the operational value of grouping and routing, and the difference between something worth paging for now vs something that should become a dashboard trend, ticket, or office-hours fix.

That is the shift to listen for in your own answer: are you describing a PromQL condition, or are you describing a paging system that has to work for tired humans at 2am? The latter is the senior version.

Read this next

Read alerting as a routing system, not only a query system

Use the Alertmanager and OnCall docs with the repo map so the paging story stays operationally grounded.

→ Alertmanager docs
→ Grafana OnCall docs
→ Repo observability map

Check yourself (from memory)

Q1. What makes an alert operationally high-quality?

High-quality alerting is about human response quality, not just signal detection.
Why are routing labels part of alert quality?
recall, then click to reveal
Because an alert is only useful if it reaches the right owner with the right urgency. Labels and routing design determine that operational path.
Want a “should this be a page, a ticket, or just a dashboard?” drill? Ask me.

Sources. Alertmanager docs; Grafana OnCall docs; repo observability map.