Lesson 2 · Fundamentals & metrics

Metrics & Prometheus

The four metric types, the pull model, and why a metric is nearly free while a log is not.

Your win: name Prometheus's four metric types and when to use each, explain the pull/scrape model (and why it's pull), and understand the labels/cardinality trap.

A metric is a labelled number over time

A metric is a named numerical measurement with labels (dimensions), sampled over time — e.g. http_requests_total{code="200", service="bob"}. Each unique set of label values is its own time series. Metrics are cheap: you're storing numbers, not text, so you can keep them at high resolution and alert on them.

The four metric types

Prometheus has four exposition types, and picking the right one is a real skill:¹

Type	Behaviour	Use for
Counter	only ever increases (resets on restart)	totals: requests, errors, bytes
Gauge	goes up and down	current values: memory, queue depth, in-flight
Histogram	counts observations into buckets + sum + count	latency/size — compute percentiles across instances
Summary	client-side quantiles over a window	quantiles when you can't aggregate

Histogram vs summary — the classic interview question Both track distributions (latency), but: a histogram buckets observations server-side, so you can aggregate across instances and compute a p95 with histogram_quantile (Lesson 4). A summary computes quantiles in the client — you cannot average them across pods. Rule of thumb: prefer histograms for anything you'll aggregate. (This repo's request latency is a histogram — grpc_io_server_server_latency_bucket.)

The pull model — Prometheus scrapes you

Prometheus is pull-based: it discovers targets and scrapes their HTTP /metrics endpoint on an interval, rather than targets pushing to it.² This sounds backwards but has real advantages:

The target doesn't need to know Prometheus exists — it just exposes /metrics.
A target being down is itself signal — the scrape fails and the synthetic metric up goes to 0, so "is it alive?" is free.
You control the load (scrape interval), not a flood of pushers.

The exception: short-lived jobs → pushgateway Pull can't scrape something that already exited (a batch job, a test run). For those, a pushgateway lets the job push its metrics to a place Prometheus can scrape. This repo uses one for BDD test metrics — the one legit place to push.

Anchor — Prometheus here, and the 2-hour surprise This repo runs plain Prometheus (the community chart, not the Operator — so scrape config and rules live inline in platforms/monitoring/prometheus/values.yaml, not ServiceMonitor CRDs). One number worth remembering: local Prometheus retains data for only 2 hours (prometheus/values.yaml:58). Everything longer-term is shipped to Thanos (Lesson 5) via a sidecar in the Prometheus pod. So "Prometheus stores the metrics" is only true for the last 2h — a detail that surprises people and matters when you query old data.

Prometheus — Metric types + Overview

The four types with examples, and the pull/scrape architecture.

→ prometheus.io — Metric types
→ prometheus.io — Overview (pull model)

Check yourself (from memory)

Q1. A metric that only ever increases is a…

Counter = monotonic (requests_total, errors_total). Gauge goes up and down.

Q2. Prefer a histogram over a summary because a histogram…

Buckets are server-side, so you can sum across pods and compute p95. Summary quantiles are client-side, un-aggregatable.

Q3. A benefit of Prometheus's pull model is…

up == 0 on a failed scrape = free liveness signal. Exited jobs need a pushgateway.

The four metric types + the pull model + the repo's retention.

recall, then click to reveal

METRIC = a labelled number over time; each unique label combo = a time series. TYPES: COUNTER (↑ only — totals), GAUGE (↕ — current values), HISTOGRAM (bucketed observations + sum + count — aggregatable percentiles via histogram_quantile), SUMMARY (client-side quantiles — NOT aggregatable). Prefer HISTOGRAM for anything you'll aggregate. PULL MODEL: Prometheus SCRAPES targets' /metrics on an interval — the target needn't know Prometheus; a failed scrape → up==0 = free liveness; you control load. Exception: short-lived jobs PUSH to a PUSHGATEWAY (repo: BDD tests). REPO: plain Prometheus (community chart, inline config, NOT Operator); retention = only 2h (values.yaml:58) → long-term to THANOS via a sidecar.

Want to see the text exposition format, or a real high-cardinality outage story? Ask me.

1. Prometheus — Metric types.

2. Prometheus — Overview (pull model).