Lesson 8 · Scale, dashboards & alerting

SLI / SLO / error budgets

Turning metrics into a reliability target — and alerting on how fast you're spending it.

Your win: define SLI, SLO, and error budget precisely, explain burn-rate alerting, and see how this repo expresses reliability as alert rules rather than formal SLO objects.

Three definitions, in order

Term	What it is	Example
SLI — Service Level Indicator	a measured number: how well the service is doing, from the user's view	% of requests < 300ms (or non-5xx)
SLO — Service Level Objective	the target for that SLI	99.5% of requests < 300ms
Error budget	the allowed failure: `100% − SLO`	0.5% may be slow/failing

An SLA is the contractual version (with penalties); SLOs are your internal target, usually stricter.¹ The SLI is built from the metrics you already have — e.g. a ratio of "good" requests to total, straight out of the RED metrics from Lesson 3.

The error budget changes how you think

Why the error budget is the clever bit "100% reliable" is impossible and not even desirable (it'd mean never shipping). An SLO of 99.5% says 0.5% failure is acceptable — that's your error budget, and you can spend it: ship risky changes, run experiments. Blow the budget and you slow down and stabilize. It turns "reliability vs velocity" from an argument into a number both sides agree on. That reframing is the whole point of SLOs.

Alert on burn rate, not raw errors

The naive alert "error rate > 1%" pages you at 3am for a blip that doesn't threaten the SLO. The SRE approach alerts on burn rate — how fast you're consuming the error budget.² A burn rate of 1 means you'll exactly exhaust the budget by the window's end; a burn rate of 14 means you'll burn it in hours — page now. Slow burns get a ticket, not a page. This is how mature teams cut alert fatigue.

Anchor — this repo has no SLO objects; it uses alert rules Honest framing: there's no formal SLO CRD or SLO dashboard here. "SLOs" are expressed implicitly as the p95-latency and error-rate alerting rules you saw in Lesson 4 (platforms/monitoring/prometheus/values.yaml:720-791, the backend-services group: HighNumberOfSlowGrpcRequests on p95, HighNumberOfFailedGrpcRequests on error rate). The threshold in a rule is a de facto objective; there's no explicit error-budget/burn-rate machinery. So the correct interview line is: "we monitor SLO-style thresholds via Prometheus alert rules, but don't yet track formal error budgets or burn-rate alerts — that'd be a natural next step." Knowing the gap is as valuable as knowing the ideal.

Google SRE — SLIs, SLOs, and error budgets

The canonical definitions, the error-budget philosophy, and burn-rate alerting.

→ sre.google — Service Level Objectives
→ sre.google — Alerting on SLOs (burn rate)

Check yourself (from memory)

Q1. An SLI is…

SLI = what you measure (e.g. % fast requests). SLO = the target for it. Error budget = 100 − SLO.

Q2. An error budget of 0.5% comes from an SLO of…

Error budget = 100% − SLO = 100 − 99.5 = 0.5%. It's the failure you're allowed to spend.

Q3. You alert on burn rate (not raw error %) to…

Burn rate reflects how fast you're spending the budget — fast burn pages now, slow burn is a ticket. Cuts alert fatigue.

SLI/SLO/error budget/burn rate — and this repo's reality.

recall, then click to reveal

SLI = a MEASURED indicator (e.g. % requests < 300ms), built from your metrics. SLO = the TARGET (99.5%). ERROR BUDGET = 100% − SLO (allowed failure, 0.5%) — you can SPEND it (ship risk) until it's gone, then stabilize → turns reliability-vs-velocity into a number. SLA = the contractual version. BURN RATE = how fast you spend the budget; alert on burn rate (fast=page now, slow=ticket) not raw error% → less alert fatigue. REPO: NO formal SLO object/dashboard — "SLOs" are the p95/error-rate ALERT RULES (backend-services, values.yaml:720-791); no explicit error-budget/burn-rate machinery yet. Honest line: "SLO-style thresholds via alert rules; formal error budgets would be the next step."

✅ Part 2 complete You've scaled and surfaced the metrics pillar: Thanos (store forever, query globally, L5), Grafana (visualize, dashboards-as-code, L6), alerting & on-call (fire → route → page, L7), and SLI/SLO/error budgets (L8). Part 3 is the other two pillars — traces and logs — and correlating all three to debug an incident.

Ready for Part 3 (tracing, OTel/Jaeger, logs, Kiali)? Or a mock interview on Parts 1–2 — SLI/SLO/error-budget is a guaranteed SRE question. Ask me.

1. Google SRE — Service Level Objectives.

2. Google SRE Workbook — Alerting on SLOs.