Following one request across every service it touches — the pillar that answers where.
Your win: explain spans and traces, how a trace is stitched together across
services (context propagation), and how this repo's Go services and Istio both emit traces.
The problem metrics can't solve
Metrics told you the p99 is 3 seconds (Lesson 4). But a request touches
gateway → bob → notificationmgmt → the DB — which hop is slow? Metrics
aggregate away the individual request. Distributed tracing follows one
request through every service and shows you exactly where the time went.1
Spans and traces
A span = one timed operation (an RPC, a DB call), with a name, start/end,
attributes, and a link to its parent.
A trace = the tree of spans for one request — a root span (the incoming
request) with children for each downstream call.
How the spans find each other — context propagation
The magic is that a shared trace ID travels with the request: each service,
on making a downstream call, injects the trace/span IDs into the request headers, and
the next service extracts them to make its spans children of the same trace. The wire
format for those headers is a propagation standard — this repo uses
B3. Without propagation you'd get disconnected single-service spans, not a
joined-up trace. This is the whole trick of distributed tracing.
Anchor — how Go services trace (OpenCensus + OTel bridge)
Set up in internal/golibs/interceptors/telemetry.go: it builds
the trace exporter (:112 Jaeger collector endpoint, or OTLP gRPC
for staging), a sampler ParentBased(TraceIDRatioBased(rate))
(:119), and the B3 propagator
(:131, b3.New(...)). Note it's the OpenCensus + OTel
bridge, not the pure OTel SDK — which is why the metric names were
grpc_io_server_* (Lesson 3). A gRPC interceptor
(bootstrap/monitor.go) attaches it so every RPC becomes a span,
with the trace ID propagated via B3 headers to the next service.
Anchor — Istio traces too (the sidecar)
You get traces even without app code: every Envoy sidecar (Course 4) emits a span per hop, in
Zipkin format, to the collector at
opentelemetry-collector.monitoring:9411 at 100% sampling
(platforms/istio/istiod-values.yaml:18-21). So a trace here is
assembled from both Envoy spans (the network hops) and the Go app spans (the in-process
work) — B3 keeps them on the same trace ID.
Read this next
OpenTelemetry — Traces (concept)
Spans, traces, context propagation, and sampling — the vendor-neutral model.
A span = one timed operation; a trace = the tree of spans for
one request across services. It answers where.
Q2. Spans across services join one trace via…
Each service injects/extracts the trace ID in request headers
(B3 here). No propagation → disconnected spans.
Q3. In this repo, a trace is assembled from…
Both: Envoy emits per-hop Zipkin spans, Go emits in-process
spans; B3 keeps them on one trace ID.
Spans, traces, propagation — and how this repo traces.
recall, then click to reveal
Metrics aggregate away individual requests → can't say WHICH hop is slow.
DISTRIBUTED TRACING follows ONE request across services. SPAN = one timed operation (name,
start/end, attributes, parent link); TRACE = the TREE of spans for one request. PROPAGATION:
a shared TRACE ID travels in request HEADERS — each service injects/extracts it so downstream
spans join the same trace; wire format = B3 here (no propagation → disconnected spans). REPO:
Go = OpenCensus+OTel BRIDGE (telemetry.go: Jaeger/OTLP exporter :112, sampler
ParentBased(TraceIDRatioBased) :119, B3 propagator :131) via a gRPC interceptor; Istio Envoy
emits ZIPKIN spans to opentelemetry-collector.monitoring:9411 @100%
(istiod-values.yaml:18-21). A trace = Envoy hops + Go app spans, joined by B3.
Want to see a B3 header on the wire, or how a span's parent link is set in the interceptor?
Ask me — the collector that assembles them is Lesson 10.