Lesson 4 · Fundamentals & metrics
PromQL & rules
Turning raw counters into rates and percentiles — and into the alerts that page you.
Your win: write the PromQL you'll actually use — rate,
aggregation, histogram_quantile — and read this repo's recording and alerting
rules, including the RED alerts on your own services.
The three PromQL patterns that cover 80%
PromQL is a big language, but a handful of patterns do most of the work:1
| Pattern | Does |
|---|---|
rate(counter[5m]) | per-second increase of a counter over 5m — you almost never graph a raw counter |
sum by (service) (rate(...[5m])) | aggregate away noisy labels, keep the ones you care about |
histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m]))) | p95 latency from a histogram's buckets |
rate(), always
A counter like requests_total just climbs forever — the raw value is meaningless
("42 million requests since when?"). rate(...[5m]) gives you the current
speed — requests/sec, averaged over 5 minutes — which is what you actually want to see and
alert on. And it handles counter resets (pod restarts) correctly. If you're graphing a counter
without rate, you're almost certainly doing it wrong.
Recording rules — precompute the expensive stuff
A recording rule evaluates a PromQL expression on a schedule and saves the result as a new time series.2 Use it when a costly query (a big aggregation) is reused across many dashboards/alerts — compute it once, read it many times.
job:kube_job_status_start_time:max,
prometheus/values.yaml:537,557). Most of the heavy lifting is in
alerting rules, evaluated directly. So "we precompute everything with recording rules" would be
wrong here — a good reminder to check before assuming.
Alerting rules — a condition that pages you
An alerting rule is a PromQL condition that, when true for: a
duration, fires an alert with labels (severity, team) and annotations
(description).3 The for: avoids paging on a
momentary blip.
backend-services group, built on the metrics from Lesson 3:
HighNumberOfFailedGrpcRequests— error rate fromgrpc_io_server_completed_rpcs(the E in RED).HighNumberOfSlowGrpcRequests— a p95 fromhistogram_quantile(0.95, …grpc_io_server_server_latency_bucket…)(the D).
istio-basic group on
istio_requests_total (5xx > 7%, unimplemented/unavailable gRPC) — the mesh metric
you've followed since Course 4. Each rule carries severity + app
labels; those labels are how the alert gets routed to the owning team (Lesson 7).
# the shape of an alerting rule
- alert: HighNumberOfSlowGrpcRequests
expr: histogram_quantile(0.95, sum by (le, grpc_server_method)
(rate(grpc_io_server_server_latency_bucket[5m]))) > 3000
for: 5m
labels: { severity: warning, app: backend }
annotations: { description: "p95 gRPC latency above 3s for {{ $labels.grpc_server_method }}" }
Prometheus — Querying + Alerting rules
PromQL basics/functions (rate, histogram_quantile) and how
recording/alerting rules are defined.
→ prometheus.io — Querying (PromQL)
→ prometheus.io — Alerting rules
Check yourself (from memory)
Q1. You graph a counter with rate() because…
rate(...[5m]) gives per-second speed and handles
resets — what you actually want from a counter.
Q2. p95 latency from a histogram uses…
histogram_quantile(0.95, sum by (le)(rate(x_bucket[5m])))
— the reason to use histograms over summaries.
Q3. An alerting rule's for: clause…
rate(counter[5m]) (per-sec speed — ALWAYS wrap a
counter; handles resets), sum by (label)(...) (aggregate), histogram_quantile(0.95,
sum by (le)(rate(x_bucket[5m]))) (p95). RECORDING RULE = precompute an expensive query
into a new series (repo: only cronjob helpers — used sparingly). ALERTING RULE = a PromQL
condition true for: a duration → fires an alert with severity+
app LABELS (route) + annotations. REPO: backend-services
(values.yaml:720-791) = RED alerts on YOUR services'
grpc_io_server_completed_rpcs (errors) + _server_latency_bucket p95
(duration); istio-basic (:793-839) on istio_requests_total
(5xx>7%). Labels → team routing (Lesson 7).1. Prometheus — Querying basics.