Coder Legion - Recent posts tagged istio

Patterns, Anti-Patterns, and Operational Reality

Wed, 02 Sep 2026 11:41:09 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous posts covered what the architecture does and why each layer exists. This post covers what matters when the system is running: the patterns that compose cleanly, the mistakes that widen blast radius, and the operational work that falls out of the design.

Pattern: The Sidecar Contract

The simplest pattern in this architecture is the service contract. A protected application service reads two sidecar-projected headers (x-ms<N>-user, x-ms<N>-role) and does its domain logic. It doesn't validate JWTs. It doesn't call Keycloak. It doesn't manage TLS certificates. It doesn't need to know whether the user authenticated with a session cookie or a bearer token.

async def get_ms2_headers(
    x_ms2_user: str | None = Header(None),
    x_ms2_role: str | None = Header(None),
    x_request_id: str | None = Header(None),
):
    if not x_ms2_user or not x_ms2_role:
        raise HTTPException(status_code=401, detail="Missing required legacy headers")
    return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}

This has practical consequences. To add a protected service, you deploy it with a service account, add destination-side header stripping, configure RequestAuthentication, write an AuthorizationPolicy, and read the two projected headers. Gateway-facing services also need a route-to-audience entry in Auth Service. The application still avoids an auth library, OIDC issuer config, and a runtime dependency on Keycloak.

When the identity infrastructure changes (new signing algorithm, new identity provider, new token format), application code doesn't change. The mesh config updates, and services keep reading the same two headers. Resource checks still live in application code, but the identity input stays stable.

The service does not trust arbitrary incoming headers. Each destination sidecar has a Lua filter that strips existing x-ms<N>-user and x-ms<N>-role headers before JWT validation runs. After stripping, outputClaimToHeaders writes fresh values from the verified token. If a compromised internal service sends a request with a spoofed x-ms2-user header, the destination sidecar removes it and replaces it with the value from the signed JWT. The protected service only sees headers authored by its own sidecar.

The tradeoff is dependency on the mesh. If sidecar injection fails, the service has no independent way to verify identity. PeerAuthentication: STRICT and workload AuthorizationPolicy are what make that failure visible instead of silently accepting plaintext traffic. Security is centralized at the infrastructure layer, so the infrastructure has to be treated as part of the application contract.

Pattern: Namespace Segregation by Function

Four project namespaces, plus istio-system for the mesh control plane and ingress gateway:

zt-apps — application services (Profile Aggregator, Employee Records, Device Inventory, Holiday Calendar, Office Directory, Auth Service)
zt-identity — Keycloak (OIDC provider)
zt-security — Vault, Cerbos
zt-data — PostgreSQL

Namespaces are RBAC boundaries, network policy boundaries, and resource quota boundaries. An attacker with kubectl access scoped to zt-apps cannot read Secrets in zt-security, where the POC stores the Vault root token for tests. Auth Service's scoped Vault token does live in zt-apps, because it needs signing authority at runtime. That token can sign through Transit and read public keys. It cannot rotate keys, configure Vault, or access other secrets engines. Istio's SPIFFE identity includes the namespace, so AuthorizationPolicy can distinguish "any workload in zt-apps" from "Auth Service running as auth-service-sa."

With everything in default, accidental privilege broadening is easier. RBAC rules, Secrets, service accounts, and network policy exceptions all share the same namespace scope, so blast radius becomes harder to reason about during reviews.

Pattern: Signed Tokens Over Forwarded Credentials

Auth Service (auth-service) validates the user's session cookie or bearer token and then mints a short-lived, purpose-specific mesh token. Downstream services do not receive the original credentials. The session cookie and Keycloak JWT stop at the gateway/Auth Service boundary.

Why this matters:

If a downstream service is compromised, the original credential is absent. Depending on the compromise point, the attacker may see projected identity headers or the short-lived mesh assertion in transit, but they do not get the user's session cookie or Keycloak bearer token. The mesh assertion has a 5-minute TTL and still has to match the destination's aud, act, and source-principal checks.
The mesh token carries the platform claims this architecture uses: user, role, audience, delegation, groups, department, and request metadata. It does not carry session IDs or refresh tokens.
Each token is scoped to a specific set of destination services via the aud claim. A token minted for the profile flow (Profile Aggregator, Employee Records, Device Inventory) is invalid at Holiday Calendar or Office Directory.

The mistake this avoids is forwarding the original bearer token or session cookie to every downstream service. That gives each service in the call chain access to authentication material that should have stopped at the boundary.

Pattern: Fail-Closed by Default

Protected paths are designed so that missing configuration or failed infrastructure denies access instead of opening it.

Protected workloads have scoped ALLOW AuthorizationPolicy resources. In Istio, traffic is allowed by default when no policy targets a workload. Once an ALLOW policy applies, unmatched requests are denied.
Auth Service down? ExtAuthz cannot approve protected routes, so the gateway returns 401, 403, or 503 depending on where the failure surfaces.
Vault unreachable? Auth Service cannot mint a new mesh token. The JWKS endpoint returns 500 if it cannot read public keys from Vault, while sidecars may keep using cached keys until refresh.
Cerbos unreachable? Employee Records and Device Inventory fail closed and return 403. Holiday Calendar and Office Directory allow read-only public-data actions during Cerbos outages and deny writes. That is a product decision, not a universal security property.
Sidecar not injected? PeerAuthentication: STRICT means mesh peers reject plaintext connections to that pod. Traffic from outside the mesh still needs Kubernetes NetworkPolicy or another boundary if the cluster allows it.

This is verified by a negative-path test that scales the Auth Service deployment to zero replicas and confirms that protected requests do not return 200:

# Scale auth to 0, confirm requests fail-closed
kubectl scale deployment auth-service -n zt-apps --replicas=0
sleep 10
response=$(curl -s -o /dev/null -w "%{http_code}" https://app.localtest.me/api/profile/1)
assert_not_equal "$response" "200"
kubectl scale deployment auth-service -n zt-apps --replicas=1

For protected routes, a failure that returns application data is a security bug.

Anti-Pattern: mTLS-Only Security

mTLS verifies that the calling pod is who it claims to be (via its SPIFFE certificate) and encrypts the connection. That's necessary but not sufficient.

If Holiday Calendar is compromised and your only security layer is mTLS, the attacker has a valid mesh certificate. They can make encrypted, authenticated connections to any other service in the mesh. mTLS tells Employee Records "this connection is from Holiday Calendar," but without AuthorizationPolicy, Employee Records accepts it. The encrypted tunnel between a compromised pod and a sensitive service is still a compromised path.

mTLS is the foundation. It's the transport layer identity that AuthorizationPolicy references. But treating it as the entire security model means any single pod compromise gives the attacker lateral movement across the entire mesh.

Anti-Pattern: Security in Application Code

One common alternative is implementing identity verification and coarse authorization in every service. Each service validates JWTs, wires its own policy checks, and owns the failure behavior for missing or malformed credentials.

The problems compound at scale:

One service has a bug in JWT validation. Maybe it doesn't check exp. Maybe it accepts tokens from the wrong issuer. That service is now a bypass.
The signing algorithm needs to change. You roll it out to 47 services. Three of them are on the old version for a week because the team is busy. During that week, your security posture is inconsistent.
A new developer adds an endpoint and forgets the auth middleware. The code review misses it. That endpoint is now unprotected, and the mesh has no way to catch it because it doesn't know the endpoint should be protected.

With mesh-based security, identity verification and coarse enforcement (mTLS, AuthorizationPolicy, header projection) are expressed in mesh config and enforced by the sidecar. The application developer does not need to remember those checks on every endpoint because they run before application code. Resource-level authorization (Cerbos calls) and response shaping (field masking) still live in application code, but the identity those decisions rely on comes from the sidecar.

Anti-Pattern: Flat Trust Zones

The cluster-internal trust assumption is simple: anything inside the cluster can talk to anything else. The perimeter becomes the main security boundary.

A single compromised pod then becomes a useful foothold. Container escapes, supply-chain issues, and vulnerable dependencies all put the attacker inside the trusted zone. Without service-to-service policy, the network gives them too many next hops.

The tiered architecture limits those next hops. Tier 1 can reach the specific Tier 2 services it needs. Approved services can reach PostgreSQL. Reverse and lateral paths are not part of the allow-list. Istio AuthorizationPolicy enforces that at each destination workload.

Operational Reality: Debugging Auth Failures

Auth failures are easiest to debug by locating the first boundary that rejected the request. The same HTTP status can come from different layers, so start with where the request stopped.

401, 403, or 503 from the gateway: ExtAuthz denied the request, Auth Service was unavailable, Vault signing failed, or the route was not in the coarse policy map. Check the ingress gateway response code first, then Auth Service logs.

401 from the destination service: The sidecar validated the JWT but the service didn't find projected headers. This usually means RequestAuthentication is misconfigured (wrong fromHeaders name, wrong outputClaimToHeaders claim path) or the JWT is missing the expected claims.

403 before the destination service logs anything: The mTLS identity or JWT claims did not match AuthorizationPolicy. Check istioctl proxy-config listener <pod> and the policy when conditions. A common cause is a token whose aud claim does not include the destination service name.

Cerbos denial: The user passed the route and network checks, but the resource-level policy denied the action. Check Cerbos with the exact principal, resource, action, and resource attributes.

Empty result set (RLS): The query returns zero rows. Check what app.current_user_id was set to on the transaction and compare with the RLS policy conditions.

The rule of thumb: start from the gateway and move inward. If the request never reaches the service, investigate ExtAuthz, routing, JWT validation, and AuthorizationPolicy. If it reaches the service, investigate projected headers, Cerbos inputs, and RLS transaction context.

Operational Reality: Sidecar Overhead

Every pod in this mesh runs an Envoy sidecar. That adds CPU and memory per pod, latency per request path, and one more place to inspect when behavior does not match the application logs.

For this POC on a Kind cluster, the overhead is negligible. In production, it is a real cost. Istio ambient mesh (ztunnel + waypoint proxies) addresses part of that cost by moving L4 processing to a node-level daemon and making L7 processing opt-in.

The tradeoff is where L7 policy runs. Sidecar-per-pod gives each workload local L7 enforcement. Ambient mesh moves L4 security and policy to ztunnel; L7 features such as path and method matching, CUSTOM authorization, JWT claim handling, and header projection require waypoint proxies. This architecture depends on L7 behavior at the gateway and protected service hops, so an ambient version would need waypoints for those workloads. The resource savings come from workloads that only need L4 mTLS, peer identity, and connection-level authorization.

Operational Reality: Testing Security Properties

The tests that matter most verify that incorrect behavior is denied. The main negative-path script checks these cases:

Unauthenticated access to protected routes returns 401/403.
External requests with spoofed internal headers are rejected.
Direct access to Tier 2 services from the gateway is impossible.
A non-Profile Aggregator pod calling Employee Records in-cluster is rejected.
Tokens with wrong audience claims are rejected.
Expired tokens are rejected.
Profile Aggregator cannot connect directly to PostgreSQL.
Auth Service outage fails closed for protected routes.

Each test encodes a specific assumption about the security model. If any test returns application data where denial is expected, a security property has been violated. The repository also has a broader 12-category attack-test harness for spoofing, JWT abuse, token replay, path normalization, SSRF, RLS leakage, ExtAuthz fail-open behavior, Cerbos spoofing, mTLS, and Istio config audit. For this POC, I treat these as local release-gate scripts; production needs them wired into CI before mesh config changes can be treated as guarded.

The rotation test verifies the mint-and-rotate path: mint a token, record its kid, rotate the Vault Transit key, mint another token, and confirm the new kid changed. That proves Auth Service fetches the current key version at mint time. It does not, by itself, prove every sidecar refreshed JWKS without a temporary rejection window.

What I'd Do Differently

Vault storage: emptyDir means Vault's state is lost on pod restart. The bootstrap job re-initializes everything, which works for a POC but is unacceptable for production. Persistent storage with auto-unseal is the minimum for real use.

JWKS caching: Auth Service fetches public keys from Vault on every JWKS request. Under load, this is unnecessary. A short-lived cache with forced invalidation on rotation would reduce Vault load. The cache duration should be tied to the token TTL and the rotation process, because stale JWKS is part of the key-rotation failure mode.

Token TTL tuning: 5 minutes is conservative. For internal-only calls from Profile Aggregator to Employee Records, a 60-second TTL would be sufficient and would reduce the replay window. The tradeoff is more frequent token minting for slower operations.

Ambient mesh: I would evaluate ambient mesh service by service. Workloads that only need mTLS and peer authorization are good candidates for ztunnel without sidecars. Workloads that depend on JWT validation, header projection, path/method policy, or ExtAuthz need waypoints. In this architecture, most protected paths depend on L7 behavior, so ambient mesh would reduce overhead only where the workload boundary can stay at L4.

Wrapping Up

Five layers. Eight request checkpoints. The config is complex, but the idea is simple: every hop verifies independently, each layer covers a different failure mode, and application code stays focused on domain logic.

The patterns that make it work are all forms of separation: security enforcement from application code, identity creation from identity verification, coarse authorization from fine-grained authorization, network boundaries from data boundaries. Each separation limits how far one mistake can propagate.

The anti-patterns all collapse those separations: treating one layer as sufficient, spreading identity verification across services, or trusting cluster networking as the security boundary.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

Signing Mesh Tokens with Vault

Wed, 02 Sep 2026 02:07:34 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post traced one request's identity flow from gateway to database. It left one piece unexplained: how Auth Service actually signs the identity token that carries a user through the mesh. That signing step is the system's root of trust. Steal the signing key and you can forge a token for any user and any role, and every downstream service will believe it.

A quick recap, so this post stands on its own. The system is an HR employee directory running on Istio. A gateway-facing service, Profile Aggregator, fans out to internal-only services like Employee Records (HR data) and Device Inventory (hardware assignments). Auth Service (auth-service) sits at the gateway: it authenticates the external user and mints a short-lived signed JWT, x-mesh-identity (the "mesh token"), that downstream service sidecars verify on every request. It signs that token with Vault, HashiCorp's secrets manager, using Vault's Transit engine, so the private key never touches application code. This post covers why that matters, how it works, and what happens when things go wrong.

The Problem: Secrets in Pods

The default Kubernetes pattern for secrets is to mount them into pods. Environment variables, files in /var/run/secrets, ConfigMaps with sensitive values. Assume any compromised pod leaks whatever secrets it holds. If your signing key is in the Auth Service environment, a container escape gives the attacker the key. They can mint tokens for any user, any role, indefinitely, from anywhere, until you detect the breach and rotate.

The blast radius of a compromised pod should be limited to what that pod can do right now, not what it can do forever with stolen material.

What Vault Solves, and What It Does Not

Token signing is only one use case. In a production cluster, Vault is the central control point for secret access, audit, and rotation across the system. The larger pattern is that applications should not carry durable secret material when the platform can issue short-lived authority from workload identity.

The production version removes static Vault tokens from application config. Auth Service runs under a dedicated Kubernetes service account with a projected, short-lived service-account token. Vault Kubernetes Auth maps that service account to the narrow signing policy. A Vault Agent sidecar authenticates, renews the resulting Vault token, and keeps the raw token inside the Agent boundary. Auth Service calls Vault through the Agent instead of reading a long-lived token from its own environment.

There is still a credential. The difference is that it is platform-issued workload identity, not a manually provisioned Vault token sitting in app config. The remaining credential is issued by Kubernetes, rotated by the platform, scoped by Vault policy, and auditable at Vault.

This POC uses the simpler version: Auth Service receives VAULT_TOKEN directly. That keeps the signing flow easy to see, but it is a demo shortcut, not the endpoint of the pattern.

Vault also does not make a compromised pod harmless. If the Auth Service pod is compromised, the attacker can use whatever runtime authority that pod has until you revoke or cut it off. What Vault changes is the value of that authority. Instead of stealing the RSA private key and signing offline forever, the attacker gets access to a narrow, revocable, auditable signing path. The private key stays inside Vault, every signing operation goes through a central API, and rotation is controlled in one place.

Vault Transit: Signing Without Holding Keys

Vault's Transit engine is a cryptography-as-a-service API. You send it bytes, it signs them with a key it manages, and returns the signature. The key material never leaves Vault's control.

In this architecture, Auth Service uses Transit for one specific operation: signing mesh identity tokens with RS256 (RSA-2048, PKCS#1 v1.5).

The flow:

Auth Service builds the JWT header and payload (claims assembled from the authentication result).
Auth Service encodes the signing input: base64url(header).base64url(payload).
Auth Service sends those bytes to Vault: POST /v1/transit/sign/mesh-identity.
Vault signs with the current RSA private key, returns vault:v<version>:<base64_signature>.
Auth Service strips the prefix, re-encodes as base64url, and assembles the final JWT.

def sign_payload(payload_bytes: bytes) -> str:
    """Signs a payload using Vault Transit and returns the raw signature"""
    url = f"{settings.VAULT_URL}/v1/transit/sign/mesh-identity"
    headers = {"X-Vault-Token": settings.VAULT_TOKEN}

    encoded_payload = base64.b64encode(payload_bytes).decode("utf-8")
    data = {
        "input": encoded_payload,
        "hash_algorithm": "sha2-256",
        "signature_algorithm": "pkcs1v15",
    }

    response = _client.post(url, headers=headers, json=data)
    response.raise_for_status()
    result = response.json()

    # Vault returns the signature as "vault:v<version>:<base64sig>"; strip the prefix for JWS.
    sig = result["data"]["signature"]
    parts = sig.split(":")
    if len(parts) >= 3:
        return parts[2]
    return sig

Two Vault round-trips happen per token mint: one to fetch the current key version (for the kid header claim), one to sign. In this POC, Auth Service mints a fresh mesh token for every protected gateway request. That keeps the behavior simple and security-first: every request gets a new iat, exp, jti, and signing decision, and key rotation is observed immediately.

The tradeoff is that Vault sits in the synchronous request path. Under production load, you may choose to cache mesh tokens inside Auth Service until their 5-minute expiry, keyed by session, subject, roles, audience, and delegation context. The token still stays inside the platform boundary: it is not returned to the browser, stored in a user cookie, or exposed to frontend code. Caching removes most signing calls for repeated requests from the same user flow.

The cost is freshness: revocation, role changes, and key rotation take effect at token-expiry boundaries instead of per request.

Caching also constrains the token shape. Keep request-specific data such as x-request-id outside the token so the same mesh assertion can safely travel with multiple requests. That would require changing the current POC token, because this implementation includes request_id in the JWT payload.

The JWKS Endpoint: Public Key Distribution

Every service sidecar needs to verify mesh tokens. Auth Service exposes GET /auth/jwks, which fetches all active public key versions from Vault and formats them as a standard RFC 7517 JWKS response:

def get_jwks() -> dict:
    """Format Vault public keys as JWKS"""
    keys = vault_client.get_public_keys()
    jwks = {"keys": []}
    for version, key_info in keys.items():
        public_key = serialization.load_pem_public_key(key_info["public_key"].encode("utf-8"))
        if isinstance(public_key, RSAPublicKey):
            numbers = public_key.public_numbers()
            jwks["keys"].append({
                "kty": "RSA",
                "kid": str(version),
                "use": "sig",
                "alg": "RS256",
                "n": int_to_base64url(numbers.n),
                "e": int_to_base64url(numbers.e),
            })
    return jwks

There's no version filtering in this code, and none is needed. get_public_keys() reads the keys map from vault read transit/keys/mesh-identity, and Vault only returns the working set there: versions from min_decryption_version through the latest. Anything below min_decryption_version is archived and dropped from that response, so it never reaches the JWKS output. The filtering happens inside Vault, not in application code. That makes min_decryption_version the single switch controlling which keys sidecars will accept, and it's why rotation and invalidation are two separate operations, covered next.

This cluster runs istiod with PILOT_JWT_ENABLE_REMOTE_JWKS enabled, so each Envoy sidecar fetches this endpoint directly over mTLS, rather than istiod fetching it once and pushing the keys. A sidecar caches the response and refreshes on its own schedule, on the order of 5 minutes. During rotation, there's a window where a sidecar still holds the old key set. More on that below.

Key Rotation

Rotation happens in two steps: create a new key version, then invalidate old versions.

#!/bin/bash
# Step 1: Rotate — creates a new key version, old versions remain valid
vault write -f transit/keys/mesh-identity/rotate

# Step 2: Invalidate — bumps min_decryption_version so old keys can't verify
vault write transit/keys/mesh-identity/config \
  min_decryption_version=$NEW_LATEST

The --grace flag in the rotation script introduces a 300-second (5-minute) wait between these steps. Why 5 minutes? Because mesh tokens have a 5-minute TTL. After one full TTL passes, all tokens signed with the old key have expired. No valid old-key tokens remain in flight, so invalidating the old key causes no disruption.

if [ "$1" = "--grace" ]; then
    echo "Grace period: waiting 300s for in-flight tokens to expire..."
    sleep 300
fi
vault write transit/keys/mesh-identity/config \
    min_decryption_version=$NEW_LATEST

Bumping min_decryption_version removes the old version from the JWKS endpoint, but sidecars don't pick that up instantly. Each one validates tokens against the JWKS it last fetched. A failed validation does not trigger an immediate refetch. The sidecar keeps using that cached key set until its next scheduled refresh, on the order of the 5-minute cache interval above.

That cache lag is exactly why the grace period exists. By waiting one full 5-minute token TTL before bumping the minimum, every token signed with the old key has already expired. After that, it no longer matters whether a given sidecar has refreshed its JWKS. If it still has the old public key cached, there are no valid old-key tokens left to accept. If it has refreshed, it only sees the new key. Skip the grace period and you risk rejecting still-valid old-key tokens the moment a sidecar refreshes mid-flight. Either way, bumping the minimum only archives a version; it stays recoverable by lowering min_decryption_version again, and is gone for good only once it drops below min_available_version.

The grace period covers the old key. The new key has a separate window. Auth Service sets kid to the latest version on every mint, so it starts issuing tokens signed with the new key the instant you rotate. A sidecar whose JWKS cache is still stale has no public key for that kid yet, and rejects those freshly-minted tokens with a 403. So rotation should also trigger a JWKS cache invalidation, forcing every sidecar to refetch immediately and pick up the new public key before any new-key token reaches it. That closes the window so clients never see a 403.

Vault Token Scoping

In this POC, Auth Service holds a Vault token with exactly two capabilities:

update on transit/sign/mesh-identity (can sign)
read on transit/keys/mesh-identity (can read public keys and current version)

It cannot rotate keys, cannot modify key configuration, cannot access any other Vault path. This is the minimum privilege needed for signing and JWKS serving.

The token has a 32-day renewable period. Rotation of the Vault token itself (revoking and reissuing) is a separate operational concern from key rotation. In the production model described earlier, this same policy would be attached to the Vault token issued through Kubernetes Auth and managed by Vault Agent.

Blast Radius: What Happens When Things Break

Auth Service compromised:

The attacker has the pod's signing authority: the direct Vault token in this POC, or access to the Agent-mediated signing path in the production model. They can sign arbitrary payloads under mesh-identity and read public keys. They can mint tokens for any user, any role, with any audience. That breaks the user-identity layer for any service that accepts x-mesh-identity.

It does not automatically bypass every mesh policy. Downstream AuthorizationPolicy rules still check the caller's mTLS workload identity, and the token still has to carry the expected aud (audience: which services may accept the token) and act (the delegation claim naming the authorized intermediary) claims for that route. A forged token from the wrong pod can still fail the source-principal check. The real danger is that the compromised Auth Service can mint tokens that look legitimate to the JWT layer, including tokens with audiences and delegation claims chosen by the attacker.

But the damage is bounded. The attacker cannot rotate keys or modify Vault configuration. You revoke the direct Vault token, or revoke the Agent-issued token and cut off the Kubernetes Auth path for that workload, and their signing ability stops. In-flight tokens they've already minted expire within 5 minutes. You rotate the key and bump min_decryption_version, which drops the compromised version from the JWKS endpoint so sidecars stop accepting anything signed with it once they refresh their cache.

Detection: Vault audit logs show every signing request with the token accessor. Anomalous patterns (sudden spike in signing, unusual audiences) are detectable if audit logging is enabled.

Vault compromised:

This is the worst case, but the severity depends on the compromise depth. If the attacker gains root token access or breaks into the Vault API, they can sign arbitrary tokens through Transit. There's still an audit trail (Vault audit log records each API call), and you recover by rotating the Transit key and revoking the compromised token.

If the attacker extracts the actual key material (storage backend compromise, memory dump, unsealed Vault snapshot), they can sign tokens offline, from anywhere, without making Vault API calls. No audit trail. Tokens they produce are indistinguishable from legitimate ones until you rotate to an entirely new key.

In this POC, Vault uses emptyDir storage (data lives only in the pod's ephemeral filesystem). In production, Vault would use encrypted backend storage with auto-unseal via a cloud KMS. The private key's residency model depends entirely on how Vault is deployed.

A downstream service compromised (Employee Records, Device Inventory, etc.):

The attacker can read projected headers for any request that arrives at that service. They see user IDs and roles for requests they handle. They cannot mint new mesh tokens (no Vault access) and cannot call other internal-only (Tier 2) services (AuthorizationPolicy blocks it).

For database access, the compromised service has its scoped DB role (Employee Records can access hr.* tables, Device Inventory can access it.*). A malicious process could call set_config('app.current_user_id', ...) with an arbitrary value to bypass row-level security (RLS) within its own schema. RLS stops application bugs, not a fully malicious process controlling its own DB session. The containment here is schema isolation (Employee Records can't query it.* tables) and AuthorizationPolicy (Employee Records can't reach Device Inventory's database path).

The Anti-Patterns

Local key pairs: Auth Service generates and holds an RSA key pair. Compromise means the attacker has the key forever (until you detect and rotate). No audit trail of signing operations. No centralized rotation procedure.

Shared HMAC secrets: All services share a symmetric key for signing/verification. Any compromised service can mint tokens. The blast radius of a single pod compromise becomes total. Rotation requires coordinated rollout to every service simultaneously.

Secrets in environment variables: The key material is readable by any process in the container (via /proc/self/environ), visible in crash dumps, and exposed through the container runtime debug surface. Even secretKeyRef values end up as plaintext in the process environment once the pod starts. This is the Kubernetes default for "sensitive config."

Each of these works fine at small scale with a trusted team. They become unacceptable when the question shifts from "can someone on our team see this secret" to "what happens when one pod in a 50-service mesh is compromised."

What's Next

The next post covers authorization in depth: how network-level, request-level, resource-level, and field-level authorization compose into a system where each layer catches what the one above it misses.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

The Identity Flow: From Gateway to Database

Tue, 01 Sep 2026 22:13:32 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post laid out the five security layers as a static architecture. This post follows a single request through them, down to the question that matters in practice: what does a service actually trust, and why is trusting it safe?

The system is an HR employee directory, built to be attacked. Five microservices sit behind the mesh, numbered ms1 through ms5: Profile Aggregator (ms1) is the gateway-facing orchestrator; Employee Records (ms2) holds PII and salary data and Device Inventory (ms3) holds hardware assignments, both internal-only; Holiday Calendar (ms4) and Office Directory (ms5) serve holiday and office-location data, most of it public. Auth Service mints a signed identity token, x-mesh-identity, that carries the user through the mesh, and Vault signs it. This post traces one concrete request: an employee, Alice, requests her own profile with GET /api/profile/{id}. That request reaches Profile Aggregator, which fans out to Employee Records and Device Inventory. I'll follow the ms1 to ms2 hop, Profile Aggregator calling Employee Records, so every ms2-* resource below is that running example, not a special case.

Employee Records reads the user's identity from a plain HTTP header, eg x-ms2-user. The header is named per service (ms2 reads x-ms2-user, ms3 reads x-ms3-user) so one service can never consume another's identity headers. Employee Records does no JWT parsing and makes no callback to an auth service. It reads that header and treats the value as the authenticated user. That should look fragile, because headers are trivially spoofable. Any external client can send x-ms2-user: alice, and so can any compromised pod inside the cluster.

It works because of everything that happens before a service reads that header. A single request moves through eight steps from the gateway to the database, each adding one guarantee:

Header stripping at the gateway.
Authentication of the caller's credentials.
Route-level authorization against a coarse policy map.
Token minting, signed via Vault.
Token validation and header projection at the destination sidecar.
AuthorizationPolicy, binding peer identity to token claims.
The service contract, the two headers the application actually reads.
Database identity context for row-level enforcement.

Six of those steps run in the mesh before the application sees the request. By the time Employee Records reads x-ms2-user, its value has been stripped of anything the caller sent and re-derived from the signed token the destination sidecar verified. The last two steps are the service's own contract and the database scoping it then sets up.

The Trust Boundary

The Istio Ingress Gateway is where external input is converted into internal identity. Everything before it is untrusted. An external client can send any header it wants, including headers that look exactly like internal identity assertions. The gateway's job is to strip those, authenticate the caller, and produce a cryptographically signed token that downstream services can verify independently.

After the gateway, a service trusts only the headers its own destination sidecar wrote. The sidecar does the verification itself: it strips any inbound copy of the projected headers (like x-ms2-user), validates the JWT signature, checks the issuer, and verifies expiry, then writes the claims into fresh headers. The application reads those headers directly and runs no verification of its own, because the sidecar already ran it.

That trust is narrowly scoped. It covers only the headers the sidecar itself wrote after validating the token. Anything that merely arrived on the request, from an external client or an internal pod, is ignored. So if a compromised internal pod sends a spoofed x-ms2-user header, the destination sidecar strips it before the JWT filter runs and re-projects the value from the verified token. The spoofed header never reaches the application.

This model has a clear contract: the gateway is responsible for identity creation, sidecars are responsible for identity verification, and services are responsible for business logic. If any of those three fail in isolation, the other two still limit the blast radius.

Step 1: Header Stripping

Before authentication, before routing, before anything else, a Lua EnvoyFilter removes every internal trust header from the incoming request:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-prestrip
  namespace: istio-system
spec:
  priority: -100
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.ext_authz"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.prestrip
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                local headers_to_remove = {}
                for key, value in pairs(request_handle:headers()) do
                  local lower_key = string.lower(key)
                  if string.match(lower_key, "^x%-platform%-") or
                     string.match(lower_key, "^x%-role%-") or
                     lower_key == "x-mesh-identity" or
                     string.match(lower_key, "^x%-ms%d%-user") or
                     string.match(lower_key, "^x%-ms%d%-role") then
                    table.insert(headers_to_remove, lower_key)
                  end
                end
                for _, key in ipairs(headers_to_remove) do
                  request_handle:headers():remove(key)
                end
              end

The filter runs at priority -100 with INSERT_BEFORE the ExtAuthz filter. The order is load-bearing: if ExtAuthz ran first and injected x-mesh-identity based on a valid session, but a spoofed x-ms2-user header also survived from the client, the downstream service would read the attacker-controlled value.

The two-pass approach (collect keys, then remove) is required because mutating headers during iteration is undefined behavior in Envoy's Lua API.

What gets stripped: x-platform-*, x-role-*, x-mesh-identity, and all x-ms<N>-user / x-ms<N>-role headers. Case-insensitive matching ensures no variant slips through.

Step 2: Authentication (ExtAuthz)

After stripping, the request hits the ExtAuthz filter. This calls Auth Service (auth-service), which resolves the caller's identity from one of two sources:

Bearer token: Extracted from the Authorization header, validated against Keycloak's JWKS endpoint. Roles come from realm_access.roles in the Keycloak JWT.

Session cookie: An opaque zt_session cookie, looked up in PostgreSQL. The session record holds the user's roles from the original OIDC flow.

Both paths produce the same output: a subject identifier, username, email, roles, groups, and department. The subject is deterministic across both auth paths. It's computed as uuid5(NAMESPACE_URL, "istio-security://users/<username>"), which means the same user always gets the same sub regardless of whether they authenticated via cookie or bearer token. This is important for RLS (which filters by sub) and for audit trails.

Step 3: Route-Level Authorization

Before minting a token, Auth Service checks a coarse policy map. This is the first authorization gate:

# Coarse route/role policy map (ext_authz.py)
if path.startswith("/api/profile/"):
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin"}
    decision["audience"] = ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"]
elif path.startswith("/api/holidays") and method == "GET":
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin", "public_data_admin", "security_auditor"}
    decision["audience"] = ["ms4-holiday-calendar"]
elif path.startswith("/api/holidays") and method in ("POST", "PUT", "DELETE", "PATCH"):
    allowed_roles = {"public_data_admin", "hr_admin"}
    decision["audience"] = ["ms4-holiday-calendar"]
else:
    decision["reason"] = "Route not defined in coarse policy map"
    return decision

user_roles = set(decision["roles"])
if not allowed_roles.intersection(user_roles):
    decision["reason"] = "Unauthorized role for route"
    return decision

If the user's role doesn't match the route, the request is denied with 401. No token is minted. The audience field determines which services this token will be valid for. A token for the profile endpoint is valid at Profile Aggregator (ms1-profile-aggregator), Employee Records (ms2-employee-details), and Device Inventory (ms3-hardware-assets). A token for the holidays endpoint is valid only at Holiday Calendar (ms4-holiday-calendar). This prevents a valid token from one flow being replayed against a different service.

Any path not in the policy map is denied by default. GET /api/offices is deliberately absent, because those reads are public: the gateway's AuthorizationPolicy exempts them from ExtAuthz, so an unauthenticated read reaches Office Directory (ms5-office-locations) without a token and is treated as anonymous/public. Writes to /api/offices do go through the map above and require public_data_admin.

Step 4: Token Minting

If authentication and route authorization both pass, Auth Service mints a signed mesh token. The JWT is built manually (no library) and signed via Vault Transit:

header = {"alg": "RS256", "typ": "JWT", "kid": str(vault_key_version)}

payload = {
    "iss": "auth-service",
    "sub": subject_uuid,
    "preferred_username": username,
    "roles": ["employee", "manager"],
    "roles_csv": "employee,manager",
    "aud": ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"],
    "act": {"sub": "ms1-profile-aggregator"},
    "exp": now + 300,  # 5 minutes
    "jti": str(uuid4()),
    "request_id": request_id
}

The signing input (header_b64.payload_b64) is sent to Vault Transit's /v1/transit/sign/mesh-identity endpoint. Vault signs it with the RSA-2048 private key and returns the signature. Auth Service never holds the private key.

Key details:

roles_csv exists because Istio's outputClaimToHeaders can only project string claims, not arrays.
act (the delegation claim from RFC 8693) is set to ms1-profile-aggregator when Profile Aggregator is in the audience. Auth Service sets this at mint time because it already knows (from the route-policy map) that the profile flow will pass through Profile Aggregator to reach Employee Records and Device Inventory. The act claim asserts that Profile Aggregator is the authorized intermediary for this request. When Profile Aggregator forwards the token downstream, the receiving AuthorizationPolicy verifies both the mTLS identity (is this actually Profile Aggregator?) and the act claim (was Profile Aggregator designated as the intermediary?). If a different service somehow obtained this token, the mTLS check would fail.
kid is fetched live from Vault on every mint. When keys rotate, new tokens immediately reference the new version.
TTL is 5 minutes. Short enough to limit replay windows, long enough that a fan-out from Profile Aggregator to Employee Records and Device Inventory completes comfortably.
The JWT is assembled manually rather than via a library like PyJWT. This is because the signing step goes through Vault Transit's API (not a local key), and standard JWT libraries assume they hold the private key. Building the token by hand keeps the signing interface clean: encode header and payload, send bytes to Vault, attach the returned signature.

Step 5: Token Validation and Header Projection

The signed token travels as the x-mesh-identity header. The destination sidecar handles it in two stages before the application sees the request.

First, a Lua EnvoyFilter strips any inbound x-ms2-user / x-ms2-role headers. It runs at priority: 10 with INSERT_BEFORE the JWT authentication filter, so spoofed values are removed before any claim is projected:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: header-projection-ms2
  namespace: zt-apps
spec:
  priority: 10
  workloadSelector:
    labels:
      app: ms2-employee-details
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.jwt_authn"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.projection
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                request_handle:headers():remove("x-ms2-user")
                request_handle:headers():remove("x-ms2-role")
              end

This is the destination-side counterpart to the gateway pre-strip described in The Trust Boundary: the same strip-then-project pattern, now enforced at the workload. Any inbound copy of these headers is removed before the JWT filter runs.

Then the RequestAuthentication resource validates the token and projects the verified claims:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: mesh-identity-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  jwtRules:
    - issuer: "auth-service"
      jwksUri: "http://auth-service.zt-apps.svc.cluster.local:8000/auth/jwks"
      forwardOriginalToken: false
      fromHeaders:
        - name: "x-mesh-identity"
      outputClaimToHeaders:
        - header: "x-ms2-user"
          claim: "sub"
        - header: "x-ms2-role"
          claim: "roles_csv"

The sidecar fetches public keys from the Auth Service JWKS endpoint, validates the JWT signature, checks iss and exp, and then projects sub into x-ms2-user and roles_csv into x-ms2-role. Because the strip filter already cleared any inbound copies, these headers can only carry claims from a validated token. These are the only headers the application reads.

One Istio nuance is easy to miss: RequestAuthentication validates a JWT when one is present. It does not, by itself, mean "a JWT is required." The requirement comes from the AuthorizationPolicy in the next step. That policy checks claims that only exist after successful validation, so a missing or invalid token has no matching rule and is denied.

Profile Aggregator has forwardOriginalToken: true because it needs to propagate x-mesh-identity to Employee Records and Device Inventory. All other services have it set to false since they're terminal and don't make downstream calls that need identity propagation.

Step 6: AuthorizationPolicy

After token validation, Istio's AuthorizationPolicy enforces token presence, token scope, and peer identity:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-ms1-to-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa
      to:
        - operation:
            ports: ["8000"]
      when:
        - key: request.auth.claims[aud]
          values: ["ms2-employee-details"]
        - key: request.auth.claims[act][sub]
          values: ["ms1-profile-aggregator"]

Three things must be true simultaneously: the request must come from ms1-profile-aggregator-sa over mTLS, the token must be valid for ms2-employee-details, and the token must name ms1-profile-aggregator as the authorized intermediary. The peer identity comes from the client certificate. The aud and act values come from the signed JWT. These are independent cryptographic assertions. Compromising one doesn't give you the other.

The aud claim is a list because a single profile request legitimately reaches more than one service. Istio's claim matching checks whether the configured value is present in that claim, so a token minted for Profile Aggregator, Employee Records, and Device Inventory can match ms2-employee-details here. A token minted only for Holiday Calendar would not match.

This is confused-deputy prevention: stopping a privileged intermediary from being tricked into using its authority on behalf of an attacker. If Profile Aggregator's service account is somehow used by a different process, the process would still need a validly-signed token that authorizes Profile Aggregator to call Employee Records for this user flow. If an attacker replays a legitimate token from a different pod, the mTLS identity check fails. Both bindings must match.

The deny behavior depends on having an AuthorizationPolicy that targets the workload. Istio allows traffic by default when no policy applies. In this architecture, every protected workload has at least one scoped ALLOW policy, so unmatched traffic is denied. A request with no token, a bad signature, the wrong audience, or the wrong caller identity has no matching rule and never reaches the application.

Fine-grained authorization (can this user see this specific record, and which fields?) also runs at this point, via Cerbos. That's the resource and field-level layer, covered in a later post. This post stays on the identity flow.

Step 7: The Service Contract

After all of this, what does the service actually see? Two headers:

async def get_ms2_headers(
    x_ms2_user: str | None = Header(None),
    x_ms2_role: str | None = Header(None),
    x_request_id: str | None = Header(None),
):
    if not x_ms2_user or not x_ms2_role:
        raise HTTPException(status_code=401, detail="Missing required legacy headers")
    return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}

The service reads x-ms2-user and x-ms2-role. It returns 401 if they're missing. That's the entire identity contract. The 401 message calls them "legacy" headers, but the name only refers to this plain-header interface: the service reads simple headers as if it had no awareness of the mesh, and the mesh does the real verification upstream. The service does no JWT parsing, OIDC configuration, or Vault calls of its own. It trusts that if these headers are present, the sidecar has already done the verification.

Profile Aggregator forwards x-mesh-identity and x-request-id to downstream services. It explicitly does not forward x-ms2-user or x-ms2-role since the receiving sidecar projects those independently from the JWT. Profile Aggregator cannot influence what identity Employee Records sees.

Step 8: Database Identity Context

The final step. Before executing any query, the service sets the transaction context:

async def set_rls_context(session: AsyncSession, user_id: str, roles: str, request_id: str):
    await session.execute(
        text("SELECT set_config('app.current_user_id', :user_id, true)"),
        {"user_id": user_id},
    )
    await session.execute(
        text("SELECT set_config('app.current_roles', :roles, true)"),
        {"roles": roles},
    )
    if request_id:
        await session.execute(
            text("SELECT set_config('app.request_id', :request_id, true)"),
            {"request_id": request_id},
        )

PostgreSQL RLS policies read app.current_user_id and app.current_roles to filter rows; app.request_id is set for audit correlation. The true third argument makes each setting transaction-local, so it doesn't leak across pooled connections.

At this point, the database sees the same user identity that the sidecar projected from the verified token. In normal operation, the service is a pass-through for identity. It reads the sidecar-projected header and sets it on the transaction.

A fully compromised service process could call set_config with an arbitrary value. RLS doesn't protect against a malicious process controlling its own DB session. But the layers above constrain this: the compromised service only has its own scoped DB role (Employee Records can only access hr.* tables, Device Inventory only it.*), and AuthorizationPolicy prevents it from reaching other services' databases. The blast radius of a compromised Employee Records service is limited to the HR data it already has access to, scoped by RLS to the identities it actually receives requests for.

Failure Behavior

The secure path matters, but so does the failure path. A missing or invalid mesh token fails at AuthorizationPolicy because the required claims are absent. If Auth Service is down, ExtAuthz cannot approve the request and no mesh token is minted. If Vault is unreachable, Auth Service cannot sign a token. If the sidecar cannot refresh JWKS, it continues using its cached keys until the cache expires; after that, validation fails closed for tokens it cannot verify.

The one operational caveat is key rotation. Istio sidecars cache JWKS, so there is a short window where a sidecar may still verify tokens with a cached old public key. The next post covers that rotation window and why the token TTL is only 5 minutes.

The Complete Chain

Gateway strips external headers -> Auth Service validates credentials -> Auth Service checks route-level policy -> Auth Service mints a signed token -> sidecar validates the token and projects claims -> AuthorizationPolicy checks peer identity and token claims -> service reads projected headers -> database enforces row-level visibility.

Eight steps. By the time Employee Records reads x-ms2-user, that value has passed through header stripping, credential validation, route authorization, token signing, sidecar validation, and peer binding. The service receives a narrow identity contract and passes that identity into PostgreSQL for row-level enforcement.

What's Next

The next post covers the signing infrastructure: why Vault Transit instead of local keys, how key rotation works without downtime, and what the blast radius looks like when different components are compromised.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth