Lesson 12 · Security, telemetry & operations

Operations & the mesh's shape

Reaching outside the mesh, limiting the blast radius, and the gotchas — plus what this mesh is really for.

Your win: explain ServiceEntry (reach external services) and the egress-limiting Sidecar resource, know the top operational gotchas and the istioctl tools to debug them, and summarise this repo's whole mesh posture.

Reaching outside the mesh — ServiceEntry

By default the mesh only knows about in-cluster services. A ServiceEntry adds an external host to the mesh registry so sidecars can route to it with mesh policies (retries, TLS origination, telemetry).¹

Anchor — cross-cluster ServiceEntries The only ServiceEntries here are in deployments/helm/platforms/gateway/templates/service-entry.yaml — location: MESH_EXTERNAL entries for cross-cluster reach: prod-tokyo calling into the renseikai/ga clusters, and synersia→tokyo admin, each paired with a DestinationRule doing tls.mode: SIMPLE + sni. So one org's mesh can call another org's public endpoint as if it were a mesh service.

Limiting the blast radius — the Sidecar resource

Every sidecar, by default, gets config for every service in the mesh — which bloats Envoy's memory as the mesh grows. A Sidecar resource narrows what a workload's proxy knows about, especially its egress targets.²

Anchor — an egress allow-list deployments/helm/platforms/gateway/templates/sidecar.yaml (gated on allowxDS, for prod/dorp tokyo|jprep) sets an egress.hosts allow-list — a namespace's workloads may only reach ./*, istio-system/*, sibling env-org namespaces, keda/*, monitoring/*. It shrinks Envoy's config and tightens what a workload can talk to — a partial nod toward the zero-trust the mesh otherwise skips (Lesson 9).

Operational gotchas (the ones that page you)

Jobs that never finish — a run-to-completion Job with an injected sidecar hangs, because Envoy never exits. Fix: sidecar.istio.io/inject: "false" (Lesson 2 — the repo does this for Jobs/migration hooks).
Startup races — the app sends a request before Envoy is ready. Fix: holdApplicationUntilProxyStarts: true (set in this repo's mesh config).
Config that "doesn't apply" — usually a namespace missing the injection label, or a VirtualService whose host doesn't match. Debug with istioctl analyze and istioctl proxy-config routes <pod> to see the live Envoy config.
EnvoyFilter broke on upgrade — version coupling (Lesson 8). Pin Istio (1.18.5 here) and re-test filters on upgrade.

This repo's mesh, in one paragraph Istio 1.18.5, installed via Helm (base/istiod/ingressgateway), revision-based injection. Used for traffic routing (VirtualServices from apiHttp/ webHttp values, DestinationRules for LB affinity), edge TLS (ingress Gateway, SIMPLE), telemetry (istio_requests_total → Prometheus → KEDA + alerts; Zipkin traces), and a couple of EnvoyFilters (Salesforce/Hasura Lua). It deliberately skips in-mesh mTLS and backend AuthorizationPolicy — auth is in-app, security is at the edge. A pragmatic "routing + edge + observability" mesh, not a zero-trust one. Knowing exactly which features a real mesh does and doesn't use is the mark of someone who's operated one.

Istio docs — ServiceEntry & Sidecar (+ istioctl)

External service registration, per-workload config scoping, and the debugging CLI.

→ istio.io — ServiceEntry · Sidecar
→ istio.io — istioctl analyze

Check yourself (from memory)

Q1. A ServiceEntry is used to…

It adds an external host to the mesh registry (repo: cross- cluster, MESH_EXTERNAL) so sidecars route to it with mesh policy.

Q2. A Sidecar resource's egress allow-list…

It scopes the proxy's egress (and shrinks its config) — a partial blast-radius / zero-trust measure.

Q3. A Job with an injected sidecar hangs because…

A run-to-completion Job waits on the long-running sidecar. Fix: sidecar.istio.io/inject: "false" (Lesson 2).

ServiceEntry, the Sidecar resource, the gotchas, and this mesh's posture.

recall, then click to reveal

SERVICEENTRY = add an EXTERNAL host to the mesh registry so sidecars route to it with mesh policy (repo: MESH_EXTERNAL CROSS-CLUSTER reach, tokyo↔renseikai/ ga, + paired DR tls SIMPLE/sni). SIDECAR resource = scope a workload's proxy config, esp. EGRESS allow-list (repo: prod/dorp tokyo|jprep — reach only ./*, istio-system, siblings, keda, monitoring) → shrinks Envoy mem + limits blast radius. GOTCHAS: injected Jobs HANG (sidecar never exits → inject:"false"); startup races (holdApplicationUntilProxyStarts); config "not applying" (missing inject label / host mismatch → istioctl analyze/proxy-config routes); EnvoyFilter version-break (pin 1.18.5). POSTURE: routing + edge TLS + telemetry + a few EnvoyFilters; NO in-mesh mTLS / backend authz (in-app + edge instead). Pragmatic, not zero-trust.

🎓 Course complete — all 12 lessons From "what a service mesh is" (sidecar/Envoy + istiod) through traffic management (Gateway/VirtualService/DestinationRule + resilience + EnvoyFilter) to security & observability (mTLS/authz — and the honest opt-out — plus istio_requests_total → KEDA). You can now read this repo's mesh accurately, place any feature in the stack, and — the rarer skill — say what a real mesh does and doesn't use. Course 5 (CI/CD) is next: how all this Helm/Istio config actually reaches the clusters.

Ready for Course 5 (CI/CD)? Or — ten courses' worth of lessons, still no retention check — let me run a mock interview on Istio and record where you're solid. Ask me.

1. Istio — ServiceEntry.

2. Istio — Sidecar.