Lesson 3 · Fundamentals & metrics

How this repo scrapes

From a pod annotation to a time series — and where your service's numbers actually come from.

Your win: trace how Prometheus discovers and scrapes this repo's services (the prometheus.io/scrape annotation), and know what metrics your Go services expose — the RED metrics and the custom business ones your team already emits.

Service discovery by annotation

How does Prometheus find ~25 services across autoscaling pods? Not a hard-coded list — it discovers pods from the Kubernetes API and filters by an annotation. A pod that carries prometheus.io/scrape: "true" (plus port/path) gets scraped; others are ignored.

Anchor — the kubernetes-pods scrape job platforms/monitoring/prometheus/values.yaml:424-467 defines the job. Via relabeling it: keeps only pods with prometheus_io_scrape=true (:430), maps prometheus_io_scheme→__scheme__, prometheus_io_path→__metrics_path__, and __address__ + prometheus_io_port→__address__ (:436-448), then attaches namespace/pod labels. Those annotations come from the Helm library's _workload_metadata.tpl (Course 3) — the same file that set the Istio sidecar tuning. So every service opts into scraping just by having those annotations, no per-service Prometheus config.
Relabeling — the workhorse you'll see everywhere "Relabeling" is Prometheus's rule engine for rewriting labels before scraping: keep/drop targets, rename labels, build the address. It's how one generic job scrapes every annotated pod. Worth knowing the word — it's a favourite interview topic and the source of most "why isn't my target being scraped?" confusion.

What your Go services expose

Anchor — /metrics on :8888 Every bootstrapped service serves /metrics on port 8888 (internal/golibs/bootstrap/monitor.go:75-79). The registry is an OTel Prometheus exporter that bridges OpenCensus stats (interceptors/telemetry.go:274-321) — so the metric names are OpenCensus-style (grpc_io_server_*), a detail worth being precise about.
The RED metrics — free, via interceptors You don't hand-write request metrics. The gRPC interceptor stack registers OpenCensus views (telemetry.go:34-52) that emit, for every RPC: That's the whole RED triad (Lesson 1), for free, on every service — and it's exactly what the backend-services alert rules fire on (Lesson 4).
Anchor — your team's custom business metrics Beyond RED, services register domain metrics via the WithPrometheusCollectors hook (in each cmd/server/<svc>/gserver.go), using the internal/golibs/metrics helper. Your team's: Each has a matching alert rule. So your services already emit both the generic RED signals and business-specific counters — this course is teaching you to read metrics you author.
Read this next

Prometheus — Configuration (scrape & relabeling)

Kubernetes SD, the scrape config, and relabel_configs (keep/drop/replace).

→ prometheus.io — Configuration
→ istio.io — Envoy stats (the mesh side)

Check yourself (from memory)

Q1. Prometheus decides which pods to scrape by…

The kubernetes-pods job keeps pods with prometheus_io_scrape=true via relabeling. (No Operator/ServiceMonitor here.)

Q2. The RED metrics on this repo's services come from…

OpenCensus views (grpc_io_server_completed_rpcs + latency histogram) give Rate/Errors/Duration for every RPC.

Q3. "Relabeling" in a scrape config is used to…

It rewrites labels before scraping — keeping annotated pods, building the address, attaching pod/namespace labels.
How this repo scrapes, and what the Go services expose.
recall, then click to reveal
DISCOVERY by ANNOTATION: the kubernetes-pods job (prometheus/values.yaml:424-467) uses RELABELING to keep pods with prometheus_io_scrape=true, map scheme/path/port→__address__, attach pod/namespace labels — annotations from _workload_metadata.tpl (C3). No Operator/ ServiceMonitor. SERVICES expose /metrics on :8888 (bootstrap/monitor.go:75), via an OTel-Prometheus exporter BRIDGING OpenCensus (names grpc_io_server_*). RED for free via gRPC interceptor VIEWS (telemetry.go:34-52): grpc_io_server_completed_rpcs (Rate/Errors) + grpc_io_server_server_latency_bucket (Duration). CUSTOM business metrics via WithPrometheusCollectors (repo: notification/spike/conversationmgmt counters), each with a matching alert.
Want to see your service's actual /metrics output, or how a custom collector is registered in gserver.go? Ask me.

1. Prometheus — Configuration; in-repo platforms/monitoring/prometheus/values.yaml:424-467, internal/golibs/bootstrap/monitor.go:75.