Lesson 7 · Traffic, the edge & resilience

Resilience

Timeouts, retries, and circuit breaking — the mesh features that stop one slow service taking down the rest.

Your win: explain the four resilience tools Istio gives you for free — timeouts, retries, circuit breaking, outlier detection — and be honest about which ones this repo actually turns on.

Why the mesh is the right place for this

In a microservice call graph, one slow or failing dependency can cascade: callers pile up waiting, exhaust connections, and fall over in turn. Istio lets you add resilience declaratively, without app code — the Envoy sidecar enforces it.¹

Tool	Where	Does
timeout	VirtualService	bound how long a call may take; fail fast instead of hanging
retries	VirtualService	re-attempt failed calls (with per-try timeout, retry conditions)
circuit breaking	DestinationRule `connectionPool`	cap concurrent connections/requests; shed load past the limit
outlier detection	DestinationRule	eject Pods that error too often from the LB pool, temporarily

Circuit breaking = outlier detection + connection limits Istio's "circuit breaker" isn't a single switch. It's the combination of connectionPool limits (reject new load when saturated) and outlierDetection (pull a consistently-failing Pod out of rotation so healthy ones absorb the traffic). Together they stop a struggling backend from dragging everything down. A favourite interview question — name both halves.

Retries can amplify an outage — be careful Naïve retries turn one failed request into several, multiplying load on an already-struggling service (a "retry storm"). That's why real retry config pairs a small attempt count with a perTryTimeout and specific retryOn conditions (e.g. only connect-failure, not application 500s). Retries are a tool, not a default-on freebie.

What this repo actually does (the honest part)

Anchor — the resilience knobs are mostly OFF here Being accurate matters more than looking thorough. In this repo:

The VirtualService route tables (backend/bob/values.yaml apiHttp) set match/route/rewrite/ fault but no timeout or retries — calls use Envoy's defaults.
tom's DestinationRule (Lesson 6) sets a load-balancer but no connectionPool or outlierDetection — so no mesh-level circuit breaking is configured.

So the correct interview line is: "Istio gives us timeouts/retries/circuit breaking for free, but our services rely on Envoy defaults and app-level handling rather than tuning them in the mesh." Knowing what's not configured is as valuable as knowing what is — and it's a concrete, low-risk improvement you could propose.

There is one fault-injection use The repo does use fault.abort — but for access control, not chaos testing: bob returns 403 to block Internal RPCs at the edge (bob/values.yaml:42-46, Lesson 4). Same primitive, different purpose.

Istio docs — Circuit breaking + request timeouts

The outlier-detection/connection-pool circuit breaker and how timeouts/retries are set on a VirtualService.

→ istio.io — Circuit Breaking
→ istio.io — Request Timeouts

Check yourself (from memory)

Q1. Istio's circuit breaking is built from…

connectionPool sheds load when saturated; outlierDetection ejects failing Pods. Both live in the DestinationRule.

Q2. A risk of aggressive retries is…

Retries multiply load on a struggling service; pair a small count with perTryTimeout and specific retryOn conditions.

Q3. In this repo, mesh timeouts/retries/circuit breaking are…

Route tables set no timeout/retries; tom's DR has no connectionPool/outlier. The capability exists but isn't tuned here.

The four resilience tools — and which this repo uses.

recall, then click to reveal

Istio adds resilience DECLARATIVELY (Envoy enforces, no app code): TIMEOUT (VS — bound call duration, fail fast); RETRIES (VS — re-attempt, with perTryTimeout + retryOn; beware RETRY STORMS); CIRCUIT BREAKING = connectionPool limits (shed load when saturated) + outlierDetection (eject failing Pods) — both in the DestinationRule; these two together ARE the circuit breaker. REPO (honest): route tables set NO timeout/retries (Envoy defaults); tom's DR has NO connectionPool/outlier → no mesh circuit breaking configured. The only fault use is fault.abort 403 to BLOCK Internal RPCs at the edge (access control, not chaos). Line: "the mesh gives us these free, but we rely on Envoy defaults + app-level handling."

Want to design a safe retry/timeout policy for one of your gRPC services, or see an outlier-detection config? Ask me.

1. Istio — Circuit Breaking; Request Timeouts.