Lesson 10 · Traces, logs & the whole picture

OTel Collector & Jaeger

Where all the spans go — a pipeline that receives, samples, and stores traces.

Your win: explain the OpenTelemetry Collector's receiver → processor → exporter pipeline, why tail sampling matters, and read this repo's collector config and Jaeger backend.

The Collector is a pipeline

Lesson 9's spans (from Go apps and Envoy) need somewhere to go. The OpenTelemetry Collector is a vendor-neutral pipeline that ingests telemetry, processes it, and exports it onward — decoupling "who produces spans" from "where they're stored".1 Three stages:

StageDoes
Receiversingest spans in some protocol (OTLP, Jaeger, Zipkin)
Processorsbatch, sample, transform, enrich
Exporterssend to a backend (Jaeger) or another system
Anchor — the collector pipeline platforms/monitoring/opentelemetry-collector/values.yaml:

Tail sampling — the clever cost lever

Why sample at the tail, not the head Storing 100% of traces is expensive; storing a random 1% (head sampling — decide up front) throws away the interesting ones. Tail sampling waits until a trace is complete, then decides based on what happened — so it can keep the ones that matter and drop the boring ones. This repo's services sample at 100% up front, but the collector's tail_sampling does the real reduction: keep traces with latency > 800ms, or a 5xx/gRPC-error/timeout, and drop the noise (hasura/cypress/grafana, /Subscribe streams) (values.yaml:32-77). You end up storing the slow and failing traces — exactly the ones you'd want when debugging.
Anchor — Jaeger is "all-in-one" The trace backend is jaeger-all-in-one (jaegertracing/all-in-one:1.52, jaeger-all-in-one/values.yaml:7-11) — a single binary bundling collector + storage (Badger, 10Gi) + the query UI (:16686) — not the distributed multi-component Jaeger. Honest scale note: all-in-one is simple and fine for this volume, but it's a single point of storage; a higher-scale setup would split Jaeger into separate collector/query/storage. Knowing that distinction is a good interview signal.
Read this next

OpenTelemetry Collector + Jaeger

The receiver/processor/exporter model, tail sampling, and the Jaeger backend.

→ opentelemetry.io — Collector
→ jaegertracing.io — Jaeger docs

Check yourself (from memory)

Q1. The OTel Collector's three stages are…

Ingest (receivers) → process (batch/sample) → send (exporters). Repo: [otlp, zipkin] → [batch, tail_sampling] → Jaeger.

Q2. Tail sampling beats head sampling because it…

It waits for the complete trace, so it can keep slow/failing ones and drop the boring ones — not a blind random cut.

Q3. Istio's Envoy sends its spans to the collector's…

Envoy emits Zipkin-format spans → the collector's zipkin receiver on :9411, then into the same traces pipeline.
The Collector pipeline, tail sampling, and the Jaeger backend here.
recall, then click to reveal
OTel COLLECTOR = a vendor-neutral pipeline decoupling producers from backends: RECEIVERS (ingest — otlp/jaeger/ZIPKIN) → PROCESSORS (batch, tail_sampling) → EXPORTERS (send — to Jaeger). Repo pipeline (values.yaml:173-182): traces: [otlp, zipkin] → [batch, tail_sampling] → jaeger-all-in-one:4317; also a prometheus exporter (:8889, spans→metrics). TAIL SAMPLING (vs head): decide AFTER the trace completes → keep latency>800ms + 5xx/gRPC-error/timeout, drop noise (hasura/cypress//Subscribe). Apps sample 100% up front; the collector does the real reduction. Backend = JAEGER ALL-IN-ONE (single binary: collector+Badger storage+UI:16686, image 1.52) — simple, single storage point; distributed Jaeger would split it for scale.
Want to see the full tail_sampling policy list, or why a collector between apps and Jaeger is worth it? Ask me.

1. OpenTelemetry — Collector; in-repo platforms/monitoring/opentelemetry-collector/values.yaml.