Lesson 7 · Traffic, the edge & resilience

Resilience

Timeouts, retries, and circuit breaking — the mesh features that stop one slow service taking down the rest.

Your win: explain the four resilience tools Istio gives you for free — timeouts, retries, circuit breaking, outlier detection — and be honest about which ones this repo actually turns on.

Why the mesh is the right place for this

In a microservice call graph, one slow or failing dependency can cascade: callers pile up waiting, exhaust connections, and fall over in turn. Istio lets you add resilience declaratively, without app code — the Envoy sidecar enforces it.¹

Tool	Where	Does
timeout	VirtualService	bound how long a call may take; fail fast instead of hanging
retries	VirtualService	re-attempt failed calls (with per-try timeout, retry conditions)
circuit breaking	DestinationRule `connectionPool`	cap concurrent connections/requests; shed load past the limit
outlier detection	DestinationRule	eject Pods that error too often from the LB pool, temporarily

Circuit breaking = outlier detection + connection limits Istio's "circuit breaker" isn't a single switch. It's the combination of connectionPool limits (reject new load when saturated) and outlierDetection (pull a consistently-failing Pod out of rotation so healthy ones absorb the traffic). Together they stop a struggling backend from dragging everything down. A favourite interview question — name both halves.

Retries can amplify an outage — be careful Naïve retries turn one failed request into several, multiplying load on an already-struggling service (a "retry storm"). That's why real retry config pairs a small attempt count with a perTryTimeout and specific retryOn conditions (e.g. only connect-failure, not application 500s). Retries are a tool, not a default-on freebie.

What this repo actually does (the honest part)

Anchor — the resilience knobs are mostly OFF here Being accurate matters more than looking thorough. In this repo:

The VirtualService route tables (backend/bob/values.yaml apiHttp) set match/route/rewrite/ fault but no timeout or retries — calls use Envoy's defaults.
tom's DestinationRule (Lesson 6) sets a load-balancer but no connectionPool or outlierDetection — so no mesh-level circuit breaking is configured.

So the correct interview line is: "Istio gives us timeouts/retries/circuit breaking for free, but our services rely on Envoy defaults and app-level handling rather than tuning them in the mesh." Knowing what's not configured is as valuable as knowing what is — and it's a concrete, low-risk improvement you could propose.

There is one fault-injection use The repo does use fault.abort — but for access control, not chaos testing: bob returns 403 to block Internal RPCs at the edge (bob/values.yaml:42-46, Lesson 4). Same primitive, different purpose.

Backend use case Backend use case: this helps distinguish between Istio’s resilience feature surface and the repo’s real posture, which relies more on defaults than explicit tuning in many places.

Common mistake Common mistake: answering with generic Istio resilience features without checking whether the local mesh configuration actually uses them.

Istio docs — Circuit breaking + request timeouts

The outlier-detection/connection-pool circuit breaker and how timeouts/retries are set on a VirtualService.

→ istio.io — Circuit Breaking
→ istio.io — Request Timeouts

In plain English Plain English: Istio can express timeouts, retries, and circuit-breaking behavior, but the important question is whether the repo actually turns those knobs on.

Check yourself (from memory)

Q1. Istio's circuit breaking is built from…

connectionPool sheds load when saturated; outlierDetection ejects failing Pods. Both live in the DestinationRule.

Q2. A risk of aggressive retries is…

Retries multiply load on a struggling service; pair a small count with perTryTimeout and specific retryOn conditions.

Q3. In this repo, mesh timeouts/retries/circuit breaking are…

Route tables set no timeout/retries; tom's DR has no connectionPool/outlier. The capability exists but isn't tuned here.

The four resilience tools — and which this repo uses.

recall, then click to reveal

Istio adds resilience DECLARATIVELY (Envoy enforces, no app code): TIMEOUT (VS — bound call duration, fail fast); RETRIES (VS — re-attempt, with perTryTimeout + retryOn; beware RETRY STORMS); CIRCUIT BREAKING = connectionPool limits (shed load when saturated) + outlierDetection (eject failing Pods) — both in the DestinationRule; these two together ARE the circuit breaker. REPO (honest): route tables set NO timeout/retries (Envoy defaults); tom's DR has NO connectionPool/outlier → no mesh circuit breaking configured. The only fault use is fault.abort 403 to BLOCK Internal RPCs at the edge (access control, not chaos). Line: "the mesh gives us these free, but we rely on Envoy defaults + app-level handling."

Want to design a safe retry/timeout policy for one of your gRPC services, or see an outlier-detection config? Ask me.

1. Istio — Circuit Breaking; Request Timeouts.