Lesson 6 · Scale, dashboards & alerting

Grafana

The window onto everything — and why the dashboards live in git, not in a UI.

Your win: explain what Grafana is (and isn't), what a datasource is, and how this repo treats dashboards as code — plus the detail that Grafana queries Thanos, not raw Prometheus.

Grafana is a window, not a store

A common misconception: Grafana stores your metrics. It doesn't. Grafana is a visualization layer — it queries other systems (its datasources) and draws dashboards.1 The data lives in Prometheus/Thanos (metrics) and Jaeger (traces); Grafana just makes it legible.

Anchor — the datasources (Thanos is default!) platforms/monitoring/grafana/values.yaml:549-571 provisions three: Thanos (thanos-query-frontend:9090, isDefault: true), Prometheus, and Jaeger. The surprising part is which is default: Thanos, not Prometheus. That follows directly from Lesson 2/5 — Prometheus keeps only 2h, so a dashboard that wants any history must go through Thanos Query. When you open a Grafana graph here, you're querying Thanos's global, long-term view by default.

Dashboards as code

You can click dashboards together in Grafana's UI — but then they live only in a database, undiffable and unreviewed. This repo does the opposite: dashboards are provisioned from files in git.

Anchor — provisioned JSON + generated per-service dashboards grafana/values.yaml:661-880 configures file-based dashboard providers (folders: backend, istio, kafka, kubernetes, …), each loading JSON from grafana/dashboards/** rendered into ConfigMaps. Per-service dashboards are generated — dashboards/feature-services/backend-<svc>-gen.json — and there's a grafonnet/jsonnet builder library at internal/golibs/grafanabuilder/. So a dashboard change is a PR (reviewable, diffable, versioned), not a click someone forgets they made. Same discipline as Helm charts and Terraform: the config is code.
Reaching Grafana — behind the JWT gateway Grafana (and Prometheus/Jaeger/Kiali) aren't on the public internet unguarded — they sit behind the per-env monitoring Istio gateway with JWT RequestAuthentication (Course 4's edge, applied to dashboards). The prod Grafana also runs the grafana-oncall-app plugin — the bridge to Lesson 7's paging.
Read this next

Grafana — dashboards & data sources

What a datasource is, provisioning dashboards from files, and panel basics.

→ grafana.com — Data sources
→ grafana.com — Dashboards

Check yourself (from memory)

Q1. Grafana's role is to…

It's a visualization layer querying datasources (Thanos/Prometheus/Jaeger). It doesn't store data itself.

Q2. Grafana's default metrics datasource here is…

Thanos Query is default — Prometheus keeps only 2h, so history goes through Thanos.

Q3. This repo's dashboards are…

Dashboards-as-code: file-provisioned JSON (per-service ones generated), reviewable via PR. Not UI clicks.
What Grafana is, its datasources, and dashboards-as-code here.
recall, then click to reveal
GRAFANA = a VISUALIZATION layer, NOT a store — it queries DATASOURCES and draws dashboards. Repo datasources (grafana/values.yaml:549-571): THANOS (thanos-query-frontend:9090, isDefault), Prometheus, Jaeger. Default = THANOS because Prometheus keeps only 2h. DASHBOARDS AS CODE: file-provisioned JSON from grafana/dashboards/** (values.yaml:661-880), per-service ones GENERATED (backend-<svc>-gen.json), grafonnet builder in golibs/grafanabuilder — a dashboard change is a reviewable PR, not a UI click. Grafana sits behind the monitoring Istio gateway (JWT) + runs the grafana-oncall-app plugin.
Want to see a generated per-service dashboard, or how grafonnet builds one from code? Ask me.

1. Grafana — Data sources; in-repo platforms/monitoring/grafana/values.yaml:549-880.