Lesson 12 · Traces, logs & the whole picture
Kiali & the whole picture
The mesh map — and putting all three pillars together to debug a 2am incident.
Your win: explain what Kiali adds, and — the payoff of the whole course — walk an incident from alert to root cause by moving across metrics, traces, and logs.
Kiali — the mesh's live map
Kiali is the Istio dashboard: a live service graph (who calls whom, with traffic health / error rates on each edge) plus config validation.1 It doesn't collect its own data — it reads the stack you've built.
Anchor — Kiali reads Prometheus + Jaeger
platforms/monitoring/kiali-server/values.yaml: Kiali (namespace
istio-system) points at Prometheus
(prometheus-server.monitoring…, :89-94) for the
traffic metrics that colour the graph, at Jaeger
(jaeger-all-in-one.monitoring…:16685, :110-112) for
traces, and links Grafana. So Kiali is a consumer of everything in this course — it
turns istio_requests_total (Lesson 1!) into a coloured arrow between two services.
The whole picture — one incident, three pillars
Here's why you built all of this. A realistic 2am flow, using every course lesson:
1. PAGE Grafana OnCall wakes you (L7): "HighNumberOfSlowGrpcRequests, app=communication"
└─ the alert rule (L4) fired on your service's p95 latency metric (L3)
2. METRICS Grafana dashboard (L6, Thanos datasource L5): notificationmgmt p99 spiked at 01:40
└─ but WHICH call is slow? metrics can't say →
3. TRACES Jaeger (L10): open a slow trace → the push.Send span is 95ms, the DB span 8ms (L9)
└─ so it's the downstream push provider → but WHY? →
4. LOGS Cloud Logging (L11): filter the log for that trace's request → "FCM timeout, retry 3/3"
5. KIALI the mesh graph shows the communication→FCM edge red — confirms the blast radius
── root cause: the push provider is timing out. Metrics said WHAT, traces WHERE, logs WHY. ──
The three pillars are a workflow, not three tools
Metrics alert and quantify (is it wrong, how much). Traces
localize (which hop). Logs explain (the actual error). You
move across them — and the glue is correlation: shared labels
(service, app) and the trace ID linking a trace to its logs. A tool that only had one pillar
would leave you stuck at "something's slow" with no way to reach "the push provider timed out."
Anchor — the full stack, one paragraph
Your Go service emits RED metrics on
:8888 (L3) and B3-propagated spans (L9);
Envoy adds istio_requests_total + Zipkin spans (L1/L9); logs go JSON→stdout→Cloud
Logging (L11). Prometheus scrapes the metrics (2h) → Thanos stores them long-term + globally
(L5) → Grafana visualizes via the Thanos datasource (L6); alert rules (L4) → Alertmanager →
Grafana OnCall pages you (L7). The OTel Collector fans spans (otlp/zipkin) through
tail_sampling into Jaeger (L10); Kiali reads Prometheus + Jaeger to draw the mesh. That's the
entire self-hosted observability stack — and you can now trace any signal from your code to
the dashboard.
Read this next
Kiali + the three pillars, together
The mesh graph, and correlating metrics/traces/logs to debug.
Check yourself (from memory)
Q1. Kiali draws its service graph by reading…
Kiali consumes Prometheus metrics (e.g.
istio_requests_total) + Jaeger traces — it builds on this course's stack.
Q2. In a debug flow, traces are the pillar that…
Metrics alert (what), traces localize (where — which span/hop),
logs explain (why). Traces are the middle step.
Q3. The glue that links a trace to its log lines is…
Correlation via the trace ID (and shared service labels) lets
you jump metrics→traces→logs for one request.
Kiali, and the three-pillar incident workflow.
recall, then click to reveal
KIALI = the Istio dashboard: live SERVICE GRAPH (who calls whom, edge
error rates) + config validation; it READS Prometheus (
kiali-server/values.yaml:89-94)
+ Jaeger (:110-112) — a consumer of this course's stack. THE THREE PILLARS = A
WORKFLOW: METRICS alert + quantify (WHAT), TRACES localize (WHERE — which hop), LOGS explain
(WHY). Move ACROSS them; the glue is CORRELATION (shared service/app labels + the trace ID
linking a trace to its logs). Incident flow: OnCall page (L7) ← alert rule (L4) on a metric
(L3) → Grafana/Thanos dashboard (L5/6) → Jaeger trace (L9/10) finds the slow span → Cloud
Logging (L11) gives the error → Kiali confirms the blast radius.
🎓 Course complete — all 12 lessons
From the three pillars through Prometheus/PromQL, Thanos, Grafana, alerting/on-call and SLOs,
to tracing, the OTel/Jaeger pipeline, logs, and Kiali — you can now read every signal this
system emits and follow it from your Go code to the dashboard that wakes you. The big loop is
closed:
istio_requests_total (C2 KEDA / C4 mesh / C5 alerts) is this stack's
metric. Course 7 (Infrastructure as Code) is the finale of the track — the
Terraform that provisions the clusters, the GCS buckets (incl. Thanos's), and the identities
everything runs on.
Ready for Course 7 (Infrastructure as Code) — the last course? Or — twelve
courses' worth of lessons, still no retention check — let me run a mock interview
on observability and record where you're solid. Ask me.
1. Kiali — Docs; in-repo platforms/monitoring/kiali-server/values.yaml:89-112.