Lesson 5 · Scale, dashboards & alerting

Thanos

Where the metrics actually live — long-term storage and one query across every cluster.

Your win: explain the two problems Thanos solves (long-term storage & global query), name its components, and read this repo's hub-and-spoke setup.

The two things plain Prometheus can't do

Prometheus keeps metrics on local disk — this repo, for only 2 hours (Lesson 2). And each Prometheus sees only its own cluster. Neither is enough:

You need history — "was latency worse than last month?" needs months of data, not 2 hours.
You need one view — with a Prometheus per cluster/org, you can't query "error rate across all of prod" from one place.

Thanos adds a layer that gives Prometheus long-term object storage, a global query view, deduplication (for HA pairs), and downsampling.¹

The components

Component	Job
Sidecar	sits next to Prometheus, uploads its 2h TSDB blocks to object storage
Store Gateway	serves those historical blocks back from object storage
Compactor	compacts + downsamples old data and enforces retention
Query	fans out across all sidecars + stores → one global, deduplicated view

Downsampling — why old data stays queryable and cheap You don't need per-15s resolution for a year-old graph. The Compactor downsamples: keep raw data for a while, then 5-minute rollups, then 1-hour rollups. A "last 90 days" query reads coarse data fast, instead of billions of raw points. That's how Thanos makes long-term retention affordable and fast.

Anchor — Thanos here (GCS + retention) Config in platforms/monitoring/thanos/production-values.yaml: object store is the GCS bucket manabie-thanos (:4). The Compactor keeps raw data 60d, 5-min-downsampled 60d, 1-hour-downsampled 90d (:24-26). The sidecar runs inside the Prometheus pod (prometheus/values.yaml:74-118), uploading blocks to GCS. Prometheus is the short-term buffer; GCS + Thanos is the real store.

Anchor — the global hub & the fan-out Remember the topology (repo map): the manabie cluster is a global hub that runs Thanos Query but no Prometheus. Its Query fans out to each workload cluster's sidecar over TLS via static endpoints (thanos/production-values.yaml:60-63): thanos-sidecar.staging / prod.tokyo / prep.tokyo / jp-partners .manabie.io. So one PromQL query at the hub reaches every cluster's live data + the historical GCS blocks — the single global view across the whole env×org fleet.

Thanos — components & design

Sidecar/store/compactor/query, object storage, downsampling, and global query.

→ thanos.io — Components (quick tutorial)
→ thanos.io — Design

Check yourself (from memory)

Q1. The two problems Thanos primarily solves are…

Prometheus keeps little data (2h here) and sees one cluster; Thanos adds object-store history + a global view (+ dedup/downsampling).

Q2. The component that downsamples old data + sets retention is the…

The Compactor downsamples (raw→5m→1h) and enforces retention. The Store Gateway just serves blocks; Query fans out.

Q3. In this repo, a single global metrics view comes from…

The manabie hub's Thanos Query queries every cluster's sidecar (static TLS endpoints) + GCS blocks.

What Thanos solves, its components, and this repo's setup.

recall, then click to reveal

Plain Prometheus keeps little data (repo: 2h) & sees ONE cluster. THANOS adds: LONG-TERM STORAGE (object store), GLOBAL QUERY (across many Prometheis), DEDUP (HA pairs), DOWNSAMPLING. COMPONENTS: SIDECAR (uploads Prom's TSDB blocks to object store), STORE GATEWAY (serves historical blocks FROM the store), COMPACTOR (downsamples raw→5m→1h + retention), QUERY (fans out over sidecars+stores → one deduplicated view). REPO: GCS bucket manabie-thanos; retention raw60d/5m60d/1h90d; sidecar INSIDE the Prom pod. The manabie cluster is a global HUB running Thanos Query that fans out to each workload cluster's thanos-sidecar.* over TLS → one query across the whole fleet.

Want to see how dedup works for HA Prometheus pairs, or why static endpoints instead of DNS discovery? Ask me.

1. Thanos — Design; in-repo platforms/monitoring/thanos/production-values.yaml.