Lesson 16 · Senior observability judgment

Sampling, cost, and debugging across metrics, traces, and logs

How to reason about the cost and visibility trade-offs in telemetry systems while still keeping the three-pillar debugging workflow effective.

Your win: explain why observability systems trade visibility for cost, how sampling changes what you can and cannot know, and how to debug across metrics, traces, and logs without pretending the data is perfect.

In plain English Plain English: telemetry is never free, so observability design is partly about deciding what evidence you can afford to keep in enough detail.

Why cost belongs in the observability conversation

Observability tools often look like pure visibility systems, but they are also cost systems. Retention, scrape volume, number of series, sampling rate, log volume, and trace storage all change what the platform can afford to keep.

This matters because observability that becomes too expensive or too noisy does not stay observability for long. It becomes a system people are afraid to query, afraid to expand, or forced to trim down in a hurry. Senior design tries to avoid that cliff before it arrives.

The cost rule Good observability design keeps enough high-value evidence to debug real incidents, while constraining the telemetry shapes that make storage, query, or operator burden explode.

Sampling changes what your traces can prove

Sampling is a good example of this tension. If you keep every trace forever, cost rises fast. If you keep too little, traces become less trustworthy as debugging evidence. So the important question is not “is sampling good?” The important question is “what does this sampling policy preserve, and what kinds of incidents could it still hide?”

The repo’s setup is a strong teaching case: application-side sampling is effectively 100%, Istio also emits heavily, and the collector applies tail sampling logic to keep the traces that matter most. That is a practical design, but it still needs to be explained as a trade-off system rather than magical full visibility.

Backend use case In this repo, traces come through OpenCensus + OTel bridging and Istio Zipkin input, then the collector uses tail_sampling rules to keep more interesting traces while the rest of the stack leans on metrics and logs for broader coverage.

Common mistake Talking as if one trace view always proves the full production story, without considering sampling or the limits of each pillar.

Why the three pillars still matter even more here

Sampling is exactly why the earlier “metrics → traces → logs” workflow matters so much. If traces are selective, you cannot treat them as perfect truth. You use metrics to see the bigger shape, traces to localize what the retained evidence shows, and logs to reconstruct the concrete failure context.

That is the stronger answer to practice: not “traces are sampled,” but “because traces are sampled, you need a cross-pillar debugging workflow that respects what each pillar can still prove.” That sounds much closer to the strongest earlier lessons.

Read sampling as an evidence policy

Use the OTel and repo docs together so trace retention and debugging claims stay grounded in what the system actually keeps.

→ OTel Collector docs
→ Jaeger docs
→ Repo observability map

Check yourself (from memory)

Q1. What is the senior way to describe sampling?

The strong answer connects cost control to evidence quality and debugging limits.

Why does sampling matter to debugging quality?

recall, then click to reveal

Because a sampled trace view does not preserve every request equally. Sampling policy determines which incidents remain visible in detail and which ones may need to be reconstructed through metrics and logs instead.

Want a “what can we still prove with this telemetry policy?” drill? Ask me.

Sources. OTel Collector docs; Jaeger docs; repo observability map.