# Cheat sheet — Observability & Monitoring

Dense revision sheet. PromQL + interview one-liners at the bottom.

---

## The three pillars
| Pillar | What | Answers | Repo |
|---|---|---|---|
| **Metrics** | numbers over time | *is* something wrong? (alert) | Prometheus + Thanos |
| **Traces** | one request across services | *where*? | OTel Collector → Jaeger |
| **Logs** | event records | *why*? (context) | zap JSON → GCP Cloud Logging |
- **Golden signals:** latency, traffic, errors, saturation. **RED** = Rate/Errors/Duration
  (requests). **USE** = Utilization/Saturation/Errors (resources).

## Prometheus
- **Pull model:** Prometheus *scrapes* targets' `/metrics` HTTP endpoints on an interval.
- **Metric types:** COUNTER (↑ only), GAUGE (↕), HISTOGRAM (buckets + sum + count → percentiles),
  SUMMARY (client quantiles).
- **Labels = dimensions;** each unique combo = a time series → beware **high cardinality**.
- Repo: retention **2h only** → long-term to Thanos; scrape via `prometheus.io/scrape` annotation.

## PromQL essentials
```
rate(http_requests_total[5m])                      # per-second rate of a counter
sum by (service) (rate(...[5m]))                   # aggregate, keep the service label
histogram_quantile(0.95, sum by (le)(rate(x_bucket[5m])))   # p95 latency
count(up == 0)                                     # how many targets are down
```
- **Recording rule** = precompute an expensive query into a new series (dashboards/alerts reuse).
- **Alerting rule** = a condition true `for:` a duration → fires an alert with labels/annotations.

## Thanos (scale Prometheus)
- **Sidecar** (in the Prom pod) uploads TSDB blocks to **object storage** (GCS `manabie-thanos`).
- **Store Gateway** serves historical blocks from GCS. **Compactor** downsamples + sets retention
  (repo: raw 60d / 5m 60d / 1h 90d). **Query** = global fan-out across clusters + dedup.
- Repo: the **`manabie` hub** runs Thanos-Query fanning out to each workload cluster's sidecar.

## Grafana
- Visualization only (not a store). **Datasources:** Thanos-Query (**default**), Prometheus, Jaeger.
- Repo: dashboards are **code** (JSON provisioned; per-service generated).

## Alerting → on-call (repo flow)
```
Prometheus rule fires  →  Alertmanager (group/dedup/route by `app`+`severity` labels)
                          →  Slack #prod-monitoring  +  per-team grafana-oncall-<team> webhook
                             →  Grafana OnCall  →  escalation (Slack → Twilio SMS/phone)
```
- Alertmanager: routing/grouping/silencing. OnCall: schedules + escalation (runtime state, not in repo).

## SLI / SLO / error budget
- **SLI** = measured indicator (e.g. % requests < 300ms). **SLO** = target (99.5%).
- **Error budget** = 100% − SLO. **Burn rate** = how fast you spend it (alert on this).
- Repo: no SLO CRD — "SLOs" are the p95/error-rate alert rules.

## Tracing
- **Span** = one timed op; **trace** = the span tree for one request. Follow it to find *where*.
- **OTel Collector** pipeline: RECEIVERS (otlp/jaeger/zipkin) → PROCESSORS (batch, tail_sampling)
  → EXPORTERS (→ Jaeger). Repo: Istio → zipkin:9411; Go → OTLP/Jaeger-thrift; 100% app sampling,
  the collector's **tail_sampling** does the real reduction.
- Repo Jaeger = **all-in-one** (single binary), not distributed.

## Logs
- **Structured** = JSON fields (queryable), not free text. Repo: zap → stdout JSON → GKE →
  **GCP Cloud Logging** (`severity`/`time`/`msg` fields). No Loki.

---

## This repo (anchors)
| Thing | Where |
|---|---|
| Deploy / global hub | `platforms/monitoring/skaffold.monitoring.yaml:23-84` |
| Scrape job (annotation relabel) | `monitoring/prometheus/values.yaml:424-467` |
| 2h retention → Thanos | `monitoring/prometheus/values.yaml:58` |
| istio_requests_total alerts | `monitoring/prometheus/values.yaml:793-839` |
| backend RED alerts (`grpc_io_server_*`) | `monitoring/prometheus/values.yaml:720-791` |
| Thanos GCS + retention/downsample | `monitoring/thanos/production-values.yaml:1-4,24-26` |
| Alertmanager routing → OnCall | `monitoring/prometheus/prod-values.yaml:45-466` |
| OTel Collector pipeline | `monitoring/opentelemetry-collector/values.yaml:6-203` |
| Go tracing (100%) | `internal/golibs/interceptors/telemetry.go:218-272` |
| `/metrics` :8888 + RED views | `internal/golibs/bootstrap/monitor.go:75-79`; `telemetry.go:34-52` |
| Logger (zap → Cloud Logging) | `internal/golibs/logger/logger.go:42-94` |

## Interview one-liners
- **Metrics vs logs vs traces?** *"Metrics say IF something's wrong (alert); traces say WHERE
  (which service/span); logs say WHY (the context). You debug by moving across them."*
- **Why pull, not push?** *"Prometheus scrapes `/metrics` endpoints it discovers — targets don't
  need to know about Prometheus; a target being down is itself signal (`up == 0`)."*
- **Histogram vs summary?** *"Histogram buckets observations server-side — you can aggregate and
  compute quantiles across instances with `histogram_quantile`. Summary computes quantiles
  client-side — can't aggregate. Prefer histograms."*
- **What does Thanos add?** *"Long-term storage in object storage, a global query across many
  Prometheis, dedup for HA pairs, and downsampling. Our Prometheus keeps only 2h; Thanos has the
  rest."*
- **SLI/SLO/error budget?** *"SLI = what you measure, SLO = the target, error budget = 100−SLO —
  the allowed failure. Alert on burn rate, not raw errors."*
- **How does a trace get built?** *"Each service creates spans with a shared trace ID (propagated,
  here via B3); the collector assembles them into a tree in Jaeger so you see one request across
  services."*