Lesson 7 · Scale, dashboards & alerting
Alerting & on-call
From a firing rule to the right human's phone — routing, grouping, and escalation.
Your win: explain the two stages after a rule fires — Alertmanager (grouping/routing) and Grafana OnCall (paging/escalation) — and read how this repo routes an alert to the owning team.
An alert firing isn't the end — it's the start
Lesson 4 got a rule to fire. But a raw stream of firing alerts is useless: it needs to be grouped (don't send 500 separate pages for one outage), deduplicated, and routed to whoever owns that service — then escalated if nobody responds. This repo splits that into two tools:
Alertmanager — grouping & routing
Alertmanager receives fired alerts and handles grouping, deduplication,
silencing, and routing to receivers.1
Routing is a tree matched on the alert's labels — which is why Lesson 4's rules all
carried severity and app.
[alertname, job], defaults to Slack
#prod-monitoring, repeats every 8h (:47-49). Then it
fans out by the app label to the owning team, e.g.
app=~"(notification|spike|conversationmgmt)" → the communication
team's receiver (:71-73), with nested matchers on
consumer_name/destination_service_name/grpc_server_method
for finer routing. So the label you set on a rule is the address of the team it pages.
Grafana OnCall — schedule & escalate
Alertmanager decides which team; Grafana OnCall decides which person, right now, and what to do if they don't answer.
oncall.manabie.io/integrations/v1/alertmanager/<token>/
(prod-values.yaml:392-466, receivers
grafana-oncall-platform, -communication, -lesson, …).
OnCall (chart oncall-1.12.0) then runs the rotation and escalation, with
Twilio for SMS + phone calls and Slack. Crucial nuance: the schedules and
escalation policies live in the OnCall database/UI, not in the repo — the repo pins the
wiring (which alert → which team's OnCall integration), and OnCall owns "who's on
tonight and how loudly to wake them."
#prod-monitoring (the default receiver) so
there's a shared timeline; there's an Opsgenie key too. And send_resolved: true
means OnCall/Slack also get told when the alert clears — so a page auto-resolves
instead of leaving someone wondering.
Alertmanager + Grafana OnCall
Routing/grouping/silencing, and on-call schedules + escalation policies.
→ prometheus.io — Alertmanager
→ grafana.com — Grafana OnCall
Check yourself (from memory)
Q1. Alertmanager's job (vs the rule that fired) is to…
Q2. An alert reaches the right team via…
app, severity);
that's why rules carry those labels. e.g. app=notification → communication team.
Q3. On-call schedules & escalation policies live…
prod-values.yaml:45-387):
root groups by [alertname, job], default Slack #prod-monitoring, repeat 8h; fans
out by the app label → team (e.g. app=notification/spike/conversationmgmt →
COMMUNICATION team), nested by consumer/service/method. GRAFANA ONCALL: each team = a webhook
receiver (oncall.manabie.io/integrations/…, :392-466); OnCall runs
the SCHEDULE + ESCALATION (Slack → Twilio SMS → phone). Schedules/policies live in the OnCall
DB (NOT the repo) — repo pins the WIRING. send_resolved: true auto-clears pages.1. Prometheus — Alertmanager; in-repo platforms/monitoring/prometheus/prod-values.yaml:45-466.