Lesson 1 · Fundamentals & metrics

What observability is

Understanding a running system from the outside — the three pillars, and the signals that matter.

Your win: explain observability vs monitoring, the three pillars (metrics, logs, traces) and what each is for, and the golden signals — the vocabulary the rest of the course builds on.

Observability vs monitoring

It's 2am and your service is slow. You can't SSH into a pod and poke around — there are dozens, autoscaling, ephemeral. Observability is the property of being able to understand a system's internal state from its external outputs — enough to answer questions you didn't think to ask in advance.¹

The distinction interviewers listen for Monitoring answers known questions — predefined dashboards and alerts for failure modes you anticipated ("is CPU high?"). Observability lets you explore unknown ones — "why is this user's request slow, only on Tuesdays?" — by having rich enough telemetry to slice and follow. Monitoring is a subset; observability is the capability.

The three pillars

Observability rests on three kinds of telemetry, and the trick is knowing what each is for:²

Pillar	What it is	Answers
Metrics	numerical measurements over time (a counter, a latency histogram)	Is something wrong? — cheap to store, you alert on these
Traces	the path of one request across services (a tree of spans)	Where is it wrong? — which service/call slowed or failed
Logs	timestamped event records (structured JSON)	Why is it wrong? — the detailed context to fix it

The debugging loop the pillars enable Metrics alert you → traces localize it → logs explain it. An alert fires on an error-rate metric; you open a trace of a failing request to see it died in the payment call; you read that service's logs at that timestamp to find the actual error. Each pillar hands off to the next. Learning to move across them is the skill.

The golden signals

You can't measure everything. Google's SRE book distils it to four golden signals that catch most problems: latency, traffic, errors, saturation.³ Two popular framings pick subsets:

RED (request-centric): Rate, Errors, Duration — for a service handling requests.
USE (resource-centric): Utilization, Saturation, Errors — for a resource (CPU, disk, queue).

Anchor — this repo's stack, and a loop you've been living This repo runs a full self-hosted stack for the pillars: Prometheus + Thanos (metrics), OpenTelemetry + Jaeger (traces), GCP Cloud Logging (logs), surfaced in Grafana + Kiali and paging via Grafana OnCall — all under deployments/helm/platforms/monitoring/. And a thread from three courses ago closes here: istio_requests_total (the mesh metric, Course 4) that already scales your service (KEDA, Course 2) and fires deploy alerts (Course 5) is produced in the mesh and scraped by this very stack — and its backend-services RED alerts run on your Go services' metrics (prometheus/values.yaml:720-791). You've been generating telemetry all along; now you'll learn to read it.

Google SRE — Monitoring distributed systems + the three pillars

The golden signals from the source, and the metrics/logs/traces model.

→ sre.google — Monitoring (golden signals)
→ opentelemetry.io — Observability primer

Check yourself (from memory)

Q1. Observability differs from monitoring in that it…

Monitoring answers known questions; observability gives rich enough telemetry to answer unknown ones.

Q2. You alert on problems using mainly…

Metrics say if something's wrong (cheap, alertable); traces localize, logs explain.

Q3. The four golden signals are…

Latency, traffic, errors, saturation. RED and USE are request- and resource-centric subsets.

Observability vs monitoring, the three pillars, and the golden signals.

recall, then click to reveal

OBSERVABILITY = understand a system's internal state from its EXTERNAL outputs, enough to answer UNANTICIPATED questions. MONITORING = predefined dashboards/alerts for KNOWN failure modes (a subset). THREE PILLARS: METRICS (numbers over time — is it wrong? ALERT on these), TRACES (one request across services — where?), LOGS (event records — why?). Debug loop: metrics ALERT → traces LOCALIZE → logs EXPLAIN. GOLDEN SIGNALS: latency, traffic, errors, saturation. RED = Rate/Errors/Duration (requests); USE = Utilization/Saturation/Errors (resources). Repo stack: Prometheus+Thanos (metrics), OTel+Jaeger (traces), Cloud Logging (logs), Grafana+Kiali, Grafana OnCall — and istio_requests_total (C2/C4/C5) is scraped HERE.

Want the "unknown-unknowns" idea made concrete, or why cardinality makes logs-as-metrics expensive? Ask me — metrics & Prometheus are Lesson 2.

1. OpenTelemetry — Observability primer.

2. The three pillars (metrics/logs/traces) — see OpenTelemetry — Signals.

3. Google SRE — Monitoring Distributed Systems (golden signals).