Lesson 13 · Senior observability judgment

Choosing signals, cardinality control, and instrumentation judgment

How to decide what to instrument, what not to instrument, and how to avoid turning useful telemetry into an expensive or noisy liability.

Your win: explain how a senior engineer chooses metrics and labels that are useful for debugging and alerting without creating runaway cost, query pain, or dashboards nobody can trust.

In plain English Plain English: good instrumentation is not “measure everything.” It is “measure the things that help you answer the next production question.”

Why this page matters more than it first looks

Most engineers learn observability by learning tools: Prometheus, Grafana, Jaeger, Cloud Logging. That is necessary, but it is not yet the senior layer. The senior layer starts one step earlier: what evidence should exist in the first place?

That question sounds simple until you live through the bad version of it. A team keeps adding labels, new counters, and clever dimensions. Dashboards become crowded. Queries slow down. Costs rise. Everyone feels like they have “more observability,” but the platform actually becomes harder to operate. That is why signal choice is a judgment problem, not just an instrumentation task.

The signal rule Instrumentation is good when it helps explain real production behavior with stable, queryable dimensions — not when it records every possible detail indiscriminately.

Why instrumentation decisions age differently under load

Junior instrumentation decisions often sound reasonable in isolation. Add a label here. Emit another counter there. Track every request by user, classroom, conversation, tenant, or raw URL. Each one sounds useful when you imagine a future debugging session.

But observability systems care about more than whether telemetry exists. They care whether telemetry stays queryable, affordable, and interpretable after real traffic, real dashboards, and real operators start depending on it. That is the hidden maturity test: not “could this label ever help?” but “will this signal still be operationally worth it at scale?”

Cardinality is the trap behind the trap

The easiest way to make observability look richer while quietly weakening it is to attach highly variable labels to metrics. User IDs, request IDs, raw URLs, or other near-unique values feel descriptive, but they explode time-series count and make the system harder and more expensive to operate.

That is why strong metric design prefers stable dimensions: service, method, status code, environment, destination, queue, or operation class. You want labels that help group behavior, not labels that turn every event into its own snowflake.

Backend use case In this repo, the important metric shapes are service-level and method-level RED metrics like grpc_io_server_completed_rpcs, latency histograms, istio_requests_total, and a small number of business counters — not per-request or per-user metric labels.

Common mistake Treating observability maturity as “we added more labels,” when the real question is whether those labels stay operationally useful under real query load and retention pressure.

What a senior answer sounds like

A strong answer here does not stop at “avoid high cardinality.” It explains why. You want to say that instrumentation has to support alerting, dashboards, and debugging with stable dimensions, while avoiding metric shapes that create storage cost, query latency, and operator confusion.

That answer sounds stronger because it connects the design choice to the real operational consequences. Earlier lessons taught you how the stack works. This lesson teaches you how to decide what evidence is worth feeding into that stack.

Revisit the Prometheus data model with cardinality in mind

Read the metric and label model again, this time asking which dimensions stay stable enough to support real operations.

→ prometheus.io — Data model
→ Repo observability map

Check yourself (from memory)

Q1. What makes a metric label a dangerous choice?

The core issue is unstable, near-unique dimensions that make the system expensive and less usable.

What is the senior goal of instrumentation?

recall, then click to reveal

To emit telemetry that helps answer real production questions with stable, queryable dimensions, without creating cardinality, cost, or noise that weakens the platform.

Want a “would this metric design be safe?” review drill? Ask me.

Sources. Prometheus data model; repo observability map.