# Repo map — Observability

Ground truth for the course. Paths from the repo root; most config is under
<span class="filepath">deployments/helm/platforms/monitoring/</span>. Line numbers drift.

---

## 0. Topology — one stack, a global hub

`skaffold.monitoring.yaml` deploys the stack as Helm releases into the **`monitoring`** namespace
(Kiali → `istio-system`), with per-cluster profiles. **The `manabie` cluster is the global hub:**
it *removes* prometheus/jaeger/kiali and keeps **Thanos-Query + Grafana + Grafana OnCall**. Each
**workload cluster** (tokyo, jp-partners, staging, dorp) runs its own **Prometheus + Thanos
sidecar + OTel Collector + Jaeger + Kiali**. The hub's Thanos Query fans out across them for a
single global view. Releases: `skaffold.monitoring.yaml:23-84`.

---

## 1. Prometheus — plain (not Operator) *(Lessons 2–4)*
- Community chart `prometheus-27.18.0.tgz` — **no ServiceMonitor/PrometheusRule CRDs**; scrape
  config + rules are inline in `serverFiles` in
  <span class="filepath">platforms/monitoring/prometheus/values.yaml</span>.
- **Retention = 2h only** (`values.yaml:58`); TSDB block pinned to 2h for Thanos (`:67-68`).
  Long-term is delegated to Thanos.
- **Scrape job `kubernetes-pods`** (`values.yaml:424-467`): relabels on the annotation triplet —
  keep `prometheus_io_scrape=true` (`:430`), `prometheus_io_scheme`→`__scheme__`,
  `prometheus_io_path`→`__metrics_path__`, `__address__` + `prometheus_io_port`→`__address__`
  (`:436-448`). Other jobs: apiservers/nodes/cadvisor/service-endpoints/pushgateway/blackbox +
  static `otel-collector` (scrapes collector app metrics on `:8889`, `:521`) and
  `otel-collector-internal` (`:8888`, `:527`).
- **Thanos sidecar runs INSIDE the Prometheus pod** (`values.yaml:74-118`, thanos v0.32.5, gRPC
  `:10901`, uploads TSDB blocks to GCS).
- **Recording rules:** only cronjob ones (`values.yaml:537,557`).
- **Alerting rules** (all in `serverFiles.alerting_rules.yml`):
  - `istio-basic` (`:793-839`) — all on `istio_requests_total`: unimplemented/unavailable gRPC,
    5xx > 7%, ingress-traffic-missing.
  - `backend-services` (`:720-791`) — **RED on the Go SDK metrics**:
    `HighNumberOfFailedGrpcRequests` on `grpc_io_server_completed_rpcs`,
    `HighNumberOfSlowGrpcRequests` p95 on `grpc_io_server_server_latency_bucket`, HTTP variants.
  - `istio-infras` (proxy CPU/mem), `kubernetes-apps/storage/resources`, and domain groups for
    notification/spike/chat/kafka/nats/etc. Each rule carries `severity` + `app` labels (the
    routing keys, topic 4).

## 2. Thanos — long-term + global query *(Lesson 5)*
- Chart `thanos-12.16.1.tgz`; config `thanos/production-values.yaml`. Object store: **GCS bucket
  `manabie-thanos`** (`:1-4`).
- Enabled components: **Compactor + Query + Query-Frontend + Store Gateway** (ruler/receive/
  bucketweb disabled).
- **Compactor** (`:18-35`): retention raw=60d, 5m-downsampled=60d, 1h-downsampled=90d (`:24-26`).
- **Store Gateway** (`:100-116`): reads historical blocks from GCS (WI SA).
- **Query** (`:37-65`): the global fan-out — **static cross-cluster endpoints** (`:59-64`):
  `thanos-sidecar.staging/prod.tokyo/prep.tokyo/jp-partners.manabie.io` + `thanos-store`, over TLS.
- **Query-Frontend** (`:67-98`): Grafana's entrypoint (`thanos-query-frontend:9090`).

## 3. Grafana — everything provisioned *(Lesson 6)*
- Datasources (`grafana/values.yaml:549-571`): **Thanos** (`thanos-query-frontend:9090`,
  **isDefault**), Prometheus, Jaeger. *Grafana queries Thanos by default* (Prometheus keeps only 2h).
- Dashboards = **code** (file provider, `:661-880`); per-service **generated**
  `dashboards/feature-services/backend-<svc>-gen.json`; a grafonnet/jsonnet builder lib at
  `internal/golibs/grafanabuilder/`.
- Prod (`grafana/production-values.yaml`): Grafana 11.6.14, 2 replicas, **grafana-oncall-app**
  plugin, Cloud SQL proxy to the OnCall DB, WI SA `grafana@…`.
- Ingress: per-env `{env}-gateway.yaml` (Istio Gateway + VirtualService + **JWT
  RequestAuthentication**). Hosts: prometheus/pushgateway/thanos-sidecar/jaeger/kiali (`stag-gateway.yaml:85-291`).

## 4. Alerting — Alertmanager → Grafana OnCall *(Lesson 7)*
Flow: **Prometheus rules → Alertmanager → per-team webhook → Grafana OnCall → escalation.**
- Alertmanager disabled in base (`prometheus/values.yaml:14-15`), fully configured per-env in
  <span class="filepath">prometheus/prod-values.yaml:35-467</span>.
- **Routing tree** (`prod-values.yaml:45-387`): root `group_by: [alertname, job]`, default
  receiver Slack `#prod-monitoring`, `repeat_interval: 8h`. Fans out by the **`app` label** →
  team, nested by `consumer_name`/`destination_service_name`/`grpc_server_method`/`pod`.
- **Receivers** (`:392-466`): Slack + per-team `grafana-oncall-<team>` webhooks to
  `oncall.manabie.io/integrations/v1/alertmanager/<token>/`. `send_resolved: true`. Opsgenie key too.
- **Grafana OnCall** (chart `oncall-1.12.0.tgz`, prod hub only): engine + Celery workers +
  RabbitMQ + Redis; **Twilio (SMS + phone calls)** + Slack (via plugin); Postgres on Cloud SQL.
  **On-call schedules + escalation policies are runtime state in the OnCall DB, not in the repo**
  — the repo pins the Alertmanager→OnCall wiring.

## 5. Tracing — OpenCensus+OTel bridge → OTel Collector → Jaeger *(Lessons 9–10)*
- **OTel Collector** (`opentelemetry-collector-0.126.0.tgz`, config `.../opentelemetry-collector/values.yaml`):
  - Receivers (`:6-24`): otlp `:4317`/`:4318`, jaeger thrift `:14268`, **zipkin `:9411`** (Istio).
  - Processors (`:26-100`): `batch` + **`tail_sampling`** (`:32-77`) — drop hasura/cypress/grafana,
    drop `/Subscribe*` streams, keep latency > 800ms, keep 5xx/gRPC-error/timeout/ERROR spans.
  - Exporters (`:138-155`): otlp → **`jaeger-all-in-one:4317`**; prometheus `:8889` (span→metrics).
- **Jaeger** = **jaeger-all-in-one** (`jaegertracing/all-in-one:1.52`, Badger 10Gi, UI `:16686`) —
  NOT distributed. (A `jaeger/` chart dir exists but is unused.)
- **Go spans** = OpenCensus + OTel **bridge**, not pure OTel SDK: `internal/golibs/tracer/grpc.go`
  (ocgrpc B3); `internal/golibs/interceptors/telemetry.go:218-272` (`NewTelemetry`: Jaeger-thrift
  if the endpoint is `http`, else OTLP gRPC); sampler `ParentBased(TraceIDRatioBased(rate))`,
  **rate hardcoded 1.0 = 100%** (`bootstrap/monitor.go:103`); B3 propagator. Config key
  `remote_trace.otel_collector_receiver`.
- **Istio** Envoy → **Zipkin** to `opentelemetry-collector.monitoring:9411` @ **100%**
  (`platforms/istio/istiod-values.yaml:18-21`).

## 6. Kiali *(Lesson 12)*
- Chart `kiali-server/`, namespace `istio-system`; auth `anonymous`, image `v1.50.0`. Reads
  **Prometheus** (`values.yaml:89-94`) + **Jaeger** (`:103-114`) + links Grafana. Ingress host
  `kiali.<env>.manabie.io` → `kiali:20001`.

## 7. Go service metrics — `/metrics` on `:8888` *(Lesson 3)*
- Endpoint: `internal/golibs/bootstrap/monitor.go:75-79`
  (`StartMetricHandlerWithProvider("/metrics", ":8888", …)`).
- Registry: `PrometheusProvider` = OTel Prometheus exporter **bridging OpenCensus stats**
  (`interceptors/telemetry.go:274-321`) + Go runtime collector.
- **RED gRPC metrics via OpenCensus views** (`telemetry.go:34-52`):
  `grpc_io_server_completed_rpcs` (rate/errors) + `grpc_io_server_server_latency_bucket`
  (duration, buckets 5…10000ms). These are what the `backend-services` alerts fire on.
- Custom-collector helper: `internal/golibs/metrics/metrics.go` (`RegisterGauge/CounterFunc/Histogram`).
- **Learner services' custom business metrics** (each with a matching alert):
  - notification: `manabie_notification_error_pushed_counter`,
    `notificationmgmt_request_send_email_status_counter`
    (`internal/notification/infra/metrics/notification_metrics.go:43,50`).
  - spike: `manabie_email_event_counter` (`internal/spike/modules/email/metrics/email_metrics.go:23`).
  - conversationmgmt: `agora_chat_concurrent_connections_counter`
    (`internal/conversationmgmt/pkg/metric/chat_collector.go:21`).
- BDD tests push to **`prometheus-pushgateway`** (`features/main.go:168`, `run.bash:227`).

## 8. Logs — Cloud Logging via stdout JSON (zap), NO Loki *(Lesson 11)*
- Logger = **zap** (`internal/golibs/logger/logger.go`): non-local builds a **JSON** encoder to
  stdout, with field mapping for **GCP Cloud Logging** (`LevelKey: "severity"`, `TimeKey: "time"`,
  level→severity mapping `:64-94`). GKE's node agent forwards container stdout → Cloud Logging;
  no in-cluster shipper.
- Istio access logs: JSON to `/dev/stdout` (`istiod-prod-values.yaml:20-50`).
- **NO Loki** — confirmed (only a vendored grafonnet lib mentions it, not a deployment).

## 9. Grafana OnCall — see topic 4. *(Lesson 7)*

## 10. Deploy / SLOs *(Lessons 6, 8)*
- All Helm in `monitoring` (Kiali `istio-system`), SOPS secrets. **No SLO CRD** — "SLOs" are
  expressed as the p95/error-rate **alert rules** in `prometheus/values.yaml`. Dashboards are code.

---

## Anchors cheat-list

| Thing | Where |
|---|---|
| Deploy topology / hub | `skaffold.monitoring.yaml:23-84` |
| Prometheus scrape job | `prometheus/values.yaml:424-467` (annotation relabel) |
| 2h retention → Thanos | `prometheus/values.yaml:58,67-68` |
| Thanos sidecar (in Prom pod) | `prometheus/values.yaml:74-118` |
| istio-basic alerts | `prometheus/values.yaml:793-839` (`istio_requests_total`) |
| backend RED alerts | `prometheus/values.yaml:720-791` (`grpc_io_server_*`) |
| Thanos GCS + retention | `thanos/production-values.yaml:1-4,24-26` |
| Thanos global fan-out | `thanos/production-values.yaml:59-64` |
| Grafana datasources (Thanos default) | `grafana/values.yaml:549-571` |
| Alertmanager routing tree | `prometheus/prod-values.yaml:45-387` |
| OnCall webhooks per team | `prometheus/prod-values.yaml:398-466` |
| OTel Collector pipeline | `opentelemetry-collector/values.yaml:6-155,173-203` |
| Jaeger all-in-one | `jaeger-all-in-one/values.yaml:7-11` |
| Go tracing init (100%) | `interceptors/telemetry.go:218-272`; `bootstrap/monitor.go:103` |
| `/metrics` :8888 + RED views | `bootstrap/monitor.go:75-79`; `telemetry.go:34-52` |
| Logger (zap → Cloud Logging) | `internal/golibs/logger/logger.go:42-94` |
| Kiali | `kiali-server/values.yaml:89-114` |