Coder Legion - Recent posts tagged cache-layer

Semantic Caching for LLM Apps Without Serving the Wrong Answer

Wed, 23 Sep 2026 14:39:11 +0000

Semantic Caching for LLM Apps Without Serving the Wrong Answer

A customer asks your support assistant: "Can I get a refund if I cancel within 14 days?" Your semantic cache finds a stored question that is almost word-for-word the same — "Can I get a refund if I cancel after 14 days?" — scores it above your threshold, and returns the stored answer before the model is even called. The answer is fluent, confident, and exactly wrong.

That exchange is the whole risk of semantic caching. The idea itself is sound: when a new question means the same thing as one you have already answered, reuse the answer and skip the model — no output tokens, no generation latency. The naive build (embed the question, take the nearest neighbour, serve it above 0.9) is also how you ship wrong answers at scale, because embedding similarity measures how close two questions are in topic, not whether they have the same answer.

You can't tune your way out of that with a better number. The vCache paper (ICLR 2026) found that "correct and incorrect cache hits have highly overlapping similarity distributions, suggesting that fixed thresholds are either unreliable or must be set extremely high to avoid errors." Separate research on universal text embeddings found "a significant lack of negation awareness", with models "often interpreting negated text pairs as semantically similar." Negation, numbers and conditions are precisely the words that decide a support answer.

So this article treats similarity as a way to find candidates, never as the verdict: scoped keys, two thresholds, a deterministic guard for the words embeddings miss, a cheap verifier for the grey zone, invalidation tied to your sources, and a shadow-mode rollout that measures the false-hit rate before one cached answer reaches a user. It is the caching layer we cover when teaching how to ship an AI product to real traffic at Cursuri-AI.ro, written out in code.

Three caches that get confused

	Exact-match response cache	Semantic response cache	Provider prompt caching
Key	Hash of the normalized request	Embedding of the question, inside a scope	Exact prefix of the prompt
What it skips	The whole model call	The whole model call	Re-processing the cached prefix
Saves	Input, output, latency	Input, output, latency	Most of the input cost of the prefix
Can it return a wrong answer?	Only if the key is incomplete	Yes, by design	No, the model still generates
Lives in	Your infrastructure	Your infrastructure	The provider

Prompt caching isn't a competitor. It makes re-reading a stable prefix cheap: on Anthropic's current lineup a cache read costs 10% of the base input price, 5% on Claude Opus 5.5 and 2.5% on Claude Fable 5.1. But the model still runs and still writes every output token. A semantic cache skips the model entirely.

That shifts the business case. When your prompts already cache well, the input side of a request is cheap, and a semantic cache mostly buys you output tokens and latency. Use both layers: a semantic-cache miss still goes to the model with a cached prefix.

First, an expected-value check

A semantic cache has a benefit you can calculate and a cost teams usually leave out: the wrong answers it serves. Put both in one function before you build anything.

def monthly_value(
    requests: int,
    hit_rate: float,              # share of requests answered from cache
    false_hit_rate: float,        # share of cache hits that were wrong
    usd_per_generation: float,    # average cost of a fresh answer
    usd_per_wrong_answer: float,  # support ticket, refund, lost customer: your number
    usd_per_lookup: float,        # embedding call + index query
) -> tuple[float, int]:
    hits = requests * hit_rate
    wrong = hits * false_hit_rate
    value = (hits * usd_per_generation
             - requests * usd_per_lookup
             - wrong * usd_per_wrong_answer)
    return value, round(wrong)

Run it with illustrative numbers: 200,000 questions a month, a 25% hit rate, $0.02 per generated answer, and a wrong answer that costs $5 of support time. At a 2% false-hit rate the cache saves $1,000 of generation and serves 1,000 wrong answers worth $5,000: a net loss of about $4,000 before lookup costs. At 0.2% it serves 100 wrong answers and comes out about $500 ahead. The hit rate sets how much you can save. The false-hit rate decides whether you save anything at all.

Where the arithmetic tends to work: support and FAQ assistants, documentation Q&A, internal help desks, product questions — high repetition, long answers, one correct answer for everyone in a scope. That is the territory of AI-assisted customer support, and it is where the rest of this design pays for itself. Where it doesn't work: coding agents, anything personalized, multi-step tool use, creative generation (users expect variety), and questions about live state such as "is the API down right now?".

The design: similarity proposes, rules decide

Every request goes through the same six steps:

Normalize the question (case, whitespace; multi-turn chats are covered further down).
Scope it: compute a key from everything that can change the correct answer — tenant, locale, plan, model, prompt version, knowledge-base version. Lookups never cross scopes.
Exact match first: a hash lookup catches verbatim repeats with no similarity risk.
Nearest neighbour within the scope.
Guard: below the low threshold, a miss. Different critical tokens, a miss. Above the high threshold, a hit. In between, verify.
On a miss, generate the answer, then decide at write time whether it may be cached at all.

The table

Postgres with pgvector is enough for this; use Redis or a vector store if that is what you already run. Storage is not the hard part.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE answer_cache (
    id              BIGSERIAL    PRIMARY KEY,
    scope_key       TEXT         NOT NULL,  -- hash of tenant, locale, plan, model, prompt and KB versions
    question        TEXT         NOT NULL,  -- normalized question
    question_hash   TEXT         NOT NULL,  -- exact-match layer
    question_sig    TEXT         NOT NULL,  -- critical tokens, see below
    embedding       VECTOR(1024) NOT NULL,  -- match your embedding model's dimension
    answer          TEXT         NOT NULL,
    source_doc_ids  TEXT[]       NOT NULL DEFAULT '{}',
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ  NOT NULL,
    hit_count       INT          NOT NULL DEFAULT 0,
    UNIQUE (scope_key, question_hash)
);

CREATE INDEX ON answer_cache USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON answer_cache USING gin (source_doc_ids);

One pgvector detail bites multi-tenant caches. As the README puts it, "with approximate indexes, filtering is applied after the index is scanned". With many scopes, the nearest neighbours the HNSW scan returns can all belong to other tenants, the WHERE scope_key = … filter removes them, and you get zero rows: a miss that isn't one. Turn on iterative index scans (pgvector 0.8.0+, SET hnsw.iterative_scan = relaxed_order), use partial indexes for a few large scopes, or partition when there are many.

The lookup

import hashlib
from dataclasses import dataclass

HIGH, LOW = 0.95, 0.88   # placeholders, not recommendations: calibrate them (see below)

@dataclass
class Candidate:
    id: int
    question: str
    answer: str
    similarity: float

def normalize(q: str) -> str:
    return " ".join(q.lower().split())

def lookup(db, scope_key: str, question: str) -> tuple[str, Candidate | None]:
    q = normalize(question)
    exact = db.fetchone(
        "SELECT id, question, answer FROM answer_cache "
        "WHERE scope_key = %s AND question_hash = %s AND expires_at > now()",
        (scope_key, hashlib.sha256(q.encode()).hexdigest()),
    )
    if exact:
        return "hit", Candidate(exact["id"], exact["question"], exact["answer"], 1.0)

    vec = embed(q)   # normalized vector; pgvector's psycopg adapter registered
    row = db.fetchone(
        """
        SELECT id, question, question_sig, answer,
               1 - (embedding <=> %s) AS similarity
          FROM answer_cache
         WHERE scope_key = %s AND expires_at > now()
         ORDER BY embedding <=> %s
         LIMIT 1
        """,
        (vec, scope_key, vec),
    )
    if row is None or row["similarity"] < LOW:
        return "miss", None
    if row["question_sig"] != critical_signature(q):
        return "miss", None          # same topic, different negation, number or condition
    cand = Candidate(row["id"], row["question"], row["answer"], row["similarity"])
    return ("hit" if cand.similarity >= HIGH else "verify"), cand

<=> is pgvector's cosine distance, so 1 - distance is the similarity you compare against thresholds. The function returns one of three states instead of a boolean, and that third state is the point of the design.

Guard the words embeddings miss

Embeddings are weakest exactly where support answers are most sensitive: negation, numbers and conditions. Within 14 days versus after 14 days, with versus without, EU versus US. A cheap, deterministic check runs before any model is involved: pull out the tokens that flip meaning, and require them to match.

import re

NEGATIONS = re.compile(r"\b(not|no|never|without|except|unless|cannot|can't|don't|doesn't|isn't|won't)\b")
NUMBERS = re.compile(r"\d+(?:[.,]\d+)?")
CONDITIONS = re.compile(r"\b(before|after|within|over|under|more than|less than|at least|at most)\b")
DOMAIN_TERMS = {"eu", "us", "uk", "basic", "pro", "enterprise", "ios", "android"}  # your vocabulary

def critical_signature(q: str) -> str:
    words = set(re.findall(r"[a-z0-9']+", q))
    parts = [
        ",".join(sorted(set(NEGATIONS.findall(q)))),
        ",".join(sorted(set(NUMBERS.findall(q)))),
        ",".join(sorted(set(CONDITIONS.findall(q)))),
        ",".join(sorted(words & DOMAIN_TERMS)),
    ]
    return "|".join(parts)

The refund example from the opening now fails the guard (within ≠ after) and becomes a miss, without any model deciding anything.

This check is deliberately blunt. It will turn some genuine paraphrases into misses ("I can't log in" and "login is not working" carry different negation tokens), and that is the right trade: a miss costs one generation, a false hit costs a wrong answer. Have whoever owns the product catalog own DOMAIN_TERMS too, so a new plan name doesn't quietly become a blind spot.

The grey zone: verify, don't guess

Between the two thresholds, ask a small, fast model one narrow question: does the stored answer fully answer the new question? Structured outputs keep the reply machine-readable.

import json
import anthropic

client = anthropic.Anthropic()
VERIFIER_MODEL = "claude-haiku-4-5-20251001"   # the smallest current model you trust

VERIFY_SYSTEM = (
    "You decide whether a stored answer can be reused for a new question. "
    "reusable is true only if the stored answer fully and correctly answers the new "
    "question: nothing the new question asks is missing, and no condition in the new "
    "question (dates, amounts, negations, products, regions) changes the answer."
)

REUSABLE = {
    "type": "object",
    "properties": {"reusable": {"type": "boolean"}},
    "required": ["reusable"],
    "additionalProperties": False,
}

def still_answers(new_question: str, cand: Candidate) -> bool:
    try:
        resp = client.messages.create(
            model=VERIFIER_MODEL,
            max_tokens=100,
            system=VERIFY_SYSTEM,
            output_config={"format": {"type": "json_schema", "schema": REUSABLE}},
            messages=[{"role": "user", "content": (
                f"<new_question>{new_question}</new_question>\n"
                f"<stored_question>{cand.question}</stored_question>\n"
                f"<stored_answer>{cand.answer}</stored_answer>"
            )}],
        )
        text = "".join(b.text for b in resp.content if b.type == "text")
        return json.loads(text)["reusable"] is True
    except (anthropic.APIError, json.JSONDecodeError, KeyError):
        return False    # any doubt is a miss, including an outage or a retired model ID

At $1/$5 per million tokens, Haiku 4.5 turns a verification into a short prompt and a one-field answer: a small fraction of a full generation on a frontier model, on the model Anthropic lists as its fastest. A verified hit is still slower than a clean one, since it adds a round trip, so narrow the grey zone once your data shows where it is. Keep the verifier's model ID in one constant, because small models get retired like any other, and notice that every failure path returns a miss. A verifier that fails open turns an outage into a stream of unchecked cache hits.

Put together, the request path is short:

def answer(db, request) -> str:
    scope = scope_key_for(request)
    state, cand = lookup(db, scope, request.question)
    if state == "hit" or (state == "verify" and still_answers(request.question, cand)):
        db.execute("UPDATE answer_cache SET hit_count = hit_count + 1 WHERE id = %s", (cand.id,))
        return cand.answer
    result = generate(request)                 # your normal model call
    if cacheable(result):
        store(db, scope, request.question, result)
    return result.text

def store(db, scope: str, question: str, result, ttl_hours: int = 24) -> None:
    q = normalize(question)
    db.execute(
        """
        INSERT INTO answer_cache (scope_key, question, question_hash, question_sig,
                                  embedding, answer, source_doc_ids, expires_at)
        VALUES (%s, %s, %s, %s, %s, %s, %s, now() + make_interval(hours => %s))
        ON CONFLICT (scope_key, question_hash) DO NOTHING
        """,
        (scope, q, hashlib.sha256(q.encode()).hexdigest(), critical_signature(q),
         embed(q), result.text, result.source_doc_ids, ttl_hours),
    )

source_doc_ids records which documents the answer was built from. It looks like bookkeeping until the day a policy page changes; the invalidation section below is where it pays off.

Calibrate thresholds on your own data

HIGH and LOW are not constants you copy from a blog post, this one included. They depend on your embedding model, your domain and your users. Measure them:

Take a few weeks of logged questions. For each one, find its nearest earlier neighbour in the same scope and record the similarity.
Label a few hundred of those pairs: same answer or different answer. Humans are best; an LLM judge with human spot-checks works if you audit it.
Pick the lowest threshold that meets your error budget, and look at how much traffic it leaves you.

def pick_threshold(
    pairs: list[tuple[float, bool]],    # (similarity, same_answer)
    max_false_hit: float = 0.01,
    min_support: int = 50,
) -> tuple[float, float] | None:
    for t in sorted({s for s, _ in pairs}):
        served = [same for s, same in pairs if s >= t]
        if len(served) < min_support:
            return None                  # too little data above this point to trust
        if served.count(False) / len(served) <= max_false_hit:
            return t, len(served) / len(pairs)   # threshold, share of candidates served
    return None

If it returns None on a few hundred labeled pairs, that is a finding, not a bug: no single threshold keeps your error budget on this data, which is exactly the overlap vCache describes. You then need the verifier for everything above LOW, or per-question thresholds, or no semantic layer at all (exact-match only). Treat this the way you would treat any retrieval quality problem: a labeled dataset and a metric, not intuition about what 0.9 "means".

Scope or leak: the cache is a cross-user channel

A semantic cache that serves user B an answer generated for user A is, by construction, a way to move data between users. If A's answer was built from A's account data, A's private documents or A's conversation, that is a data breach delivered with low latency.

Two rules close it.

The scope key includes everything that can change the answer. Tenant, locale, the user's plan (entitlements change answers), model ID, system prompt version, and a version of the knowledge base the answer was built from:

def scope_key_for(request) -> str:
    parts = [request.tenant_id, request.locale, request.plan,
             MODEL_ID, PROMPT_VERSION, KB_SNAPSHOT_ID]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()

Model and prompt versions in the key mean a deploy that changes either one starts a fresh cache automatically, and the old entries simply expire.

Cacheability is decided at write time, from what the generation actually touched. Before generating, you often can't tell whether a question needs private data; afterwards, you know exactly what the model called and read:

def cacheable(result) -> bool:
    return (
        result.stop_reason == "end_turn"
        and not result.used_account_tools        # looked up this user's orders, invoices…
        and not result.read_private_documents    # retrieval hit anything not visible to the whole scope
        and not result.used_conversation_history
        and not result.flagged_by_guardrails
        and bool(result.text.strip())
    )

The same channel runs the other way, too. If someone can get a harmful answer generated once — a prompt-injected link, an offensive reply — a shared cache will serve it to everyone who asks something similar. Cache only answers that passed your output guardrails; consider promoting an answer to the shared cache only after the same question has come from several distinct users with no negative feedback; and show a "cached answer" marker with a regenerate button, which also gives you a false-hit signal at no extra cost. Poisoning, cross-tenant leakage and injection belong on the same threat model when you're securing an LLM application; the cache is just one more component that stores model output.

Invalidation: tie entries to their sources

A cached answer is correct until the thing it was based on changes. Three mechanisms, from coarse to precise:

TTL by content class. Prices and policies: hours to a day. How-to documentation: days to weeks. Nothing lives forever.
Versions in the scope key. A new model, prompt or knowledge-base snapshot means a new scope; nothing needs deleting.
Purge by source. Store the IDs of the documents each answer was built from, and delete the dependent entries when one changes:

def purge_for_document(db, doc_id: str) -> int:
    return db.execute(
        "DELETE FROM answer_cache WHERE source_doc_ids @> ARRAY[%s]::text[]",
        (doc_id,),
    ).rowcount

Hook it to whatever already re-indexes your RAG corpus: the moment a document is re-embedded, the answers built on its old version go away. Of the three, this is the one teams skip, and it is the one that explains why the bot quoted last quarter's refund policy.

Multi-turn conversations

"And for the Enterprise plan?" means nothing without the previous turn, and embedding it on its own will match whatever else was asked about Enterprise. Two safe options: cache only first-turn questions that stand on their own, or rewrite each follow-up into a standalone question with a small model before lookup. The rewrite costs a call on every multi-turn request, which eats into the savings, so measure whether it pays. Never key the cache on the raw last message of a conversation.

Roll it out in shadow mode

Don't find out your false-hit rate from customers. Run the cache in shadow first: on every request, do the lookup, still generate a fresh answer, serve the fresh one, and log both.

def handle(db, request) -> str:
    scope = scope_key_for(request)
    state, cand = lookup(db, scope, request.question)
    result = generate(request)
    if state != "miss":
        log_shadow(request.question, cand, result.text, state)   # judged offline
    if cacheable(result):
        store(db, scope, request.question, result)
    return result.text

Offline, have a judge (with human spot-checks) decide whether each would-be cached answer was equivalent to the fresh one. After a week or two you have a measured false-hit rate per state, hit and verify, at your current thresholds, and monthly_value() stops being a guess. Switch serving on only when the numbers clear your budget, and keep a kill switch. In production, track four things: hit rate, measured false-hit rate (keep sampling), latency for hits and misses, and negative feedback on cached versus fresh answers.

Cached answers are personal data processing too

Anything you store about users' questions and your answers to them falls under the GDPR if it can relate to a person. Three consequences for the cache. Shared entries must never contain personal data, which the write-time rule above enforces. TTLs are your retention policy, so write them down as one. And if you keep any per-user cache, it belongs in your erasure flow: "delete my data" has to reach the cache, not only the main database. Designing data protection into AI systems from the start is cheap; bolting it on after an audit is not.

FAQ

What similarity threshold should I use for semantic caching?
There is no universal value. Similarities depend on the embedding model and the domain, and correct and incorrect hits overlap. Calibrate on labeled pairs from your own traffic, use two thresholds with a verifier in between, and accept that some workloads have no safe threshold at all.

Is semantic caching the same as prompt caching?
No. Prompt caching lives at the provider, discounts re-reading an identical prompt prefix, and still generates a fresh answer, so it can't return a wrong one. Semantic caching lives in your infrastructure and skips generation entirely. They stack: a semantic miss still benefits from a cached prefix.

Should I use Redis or Postgres for a semantic cache?
Whichever you already operate. Both can store vectors and run approximate nearest-neighbour search. The hard parts are scoping, thresholds and invalidation, and none of those depend on the store.

Can I semantically cache an AI agent's responses?
Not end to end. Agent runs call tools, read live state and cause side effects. Cache below the agent instead: exact-match caching of read-only tool results, with short TTLs, carries far less risk.

How do I know the cache is serving wrong answers?
Shadow mode before launch, then continuous sampling: a judge compares a sample of cached answers against fresh generations every week. Add a visible "cached" marker with a regenerate option, and watch whether users regenerate cached answers more often than fresh ones.

The short version

Embedding similarity tells you two questions are about the same thing, not that they have the same answer.
Do the expected-value check first: the false-hit rate decides whether the cache saves money at all.
Scope every lookup by tenant, plan, locale, model, prompt and knowledge-base version. Decide cacheability at write time from what the generation touched.
Guard negations, numbers and conditions deterministically, and verify the grey zone with a small model.
Calibrate thresholds on labeled pairs from your own traffic, and expect overlap.
Purge answers when their source documents change, roll out in shadow mode, and keep a kill switch.

A semantic cache built this way is less aggressive than the one in most tutorials. It serves fewer hits, and every hit it serves has cleared more than a similarity score.

I build and teach production AI systems at Cursuri-AI.ro, Eastern Europe's AI education platform — hands-on courses on shipping LLM features that stay correct, secure and affordable under real traffic.

CAG: The Simpler Way to Ground Your LLM

Sun, 28 Jun 2026 05:09:39 +0000

If you've been building AI applications recently, you've probably come across Retrieval-Augmented Generation (RAG). It has become the go-to way of giving LLMs access to external knowledge.

But RAG isn't the only option.

As context windows continue to grow, another approach is becoming increasingly practical: Cache-Augmented Generation (CAG).

Before we begin, a small disclaimer. This article intentionally argues in CAG's favor. Think of it as a friendly debate where CAG finally gets a chance to speak while RAG takes a short coffee break.

Why RAG Became So Popular

RAG solved a real problem.

Instead of expecting an LLM to know everything, we store information in a vector database. When a user asks a question, we retrieve the most relevant pieces and send them to the model.

A typical RAG pipeline looks like this:

Query → Embed → Search → Rank → Retrieve → Generate

It's a proven approach and works really well, especially when your knowledge base is large or changes frequently.

The only downside is that every question has to go through this retrieval process before the model can generate an answer.

That means more infrastructure, more moving parts, and a little extra latency.

Meet CAG

CAG takes a much simpler approach.

Instead of searching for information every time someone asks a question, it loads the required knowledge into the model's context once and keeps using it.

The workflow becomes:

Load knowledge → Cache context → Generate

That's the entire idea.

No vector search.

No retrieval step.

No ranking.

The model already has the information it needs.

Why Is This Possible Now?

A couple of years ago, CAG wasn't practical.

Context windows were simply too small.

Today, that's no longer true.

Many modern models support hundreds of thousands and sometimes even millions of tokens.

That changes the question from:

"How do I retrieve the right documents?"

"Can I fit my knowledge into the context window?"

For many internal tools, company documentation, onboarding guides, product manuals, and API references, the answer is surprisingly often yes.

RAG vs CAG

Both approaches solve the same problem, but in different ways.

Choose RAG when:

Your knowledge base is too large to fit into the model's context.
Information changes frequently.
Different users need different subsets of knowledge.
You need real-time data.

Choose CAG when:

Your documentation comfortably fits in the context window.
Most of the information is relatively static.
Low latency is important.
You want a simpler architecture with fewer components.

Neither approach is "better."

The right choice depends on your use case.

A Simple Example

A traditional RAG pipeline might look like this:

query = "What's our refund policy?"

embedding = embed(query)
chunks = vector_db.search(embedding, top_k=5)

context = "\n".join(chunks)

response = llm.generate(
    f"Context:\n{context}\n\nQuestion: {query}"
)

A CAG implementation is much simpler:

with open("knowledge_base.txt") as f:
    knowledge = f.read()

system_prompt = f"""
You are an assistant.

Use the following knowledge when answering questions.

{knowledge}
"""

response = llm.generate(
    system=system_prompt,
    user="What's our refund policy?"
)

The biggest difference isn't the amount of code.

It's that there is no retrieval happening during inference.

Why Not Use Both?

In practice, many applications don't have to choose one over the other.

A hybrid approach often works best.

Keep your stable documentation in the model's cached context using CAG.

Retrieve only the information that changes frequently using RAG.

This gives you fast responses for most questions while still allowing access to fresh information whenever needed.

Final Thoughts

As developers, we sometimes assume that every LLM application needs a vector database.

But that's not always true anymore.

Before building a RAG pipeline, ask yourself one simple question:

Does my knowledge base actually fit inside the model's context window?

If it does, CAG could be a simpler solution that's easier to build, easier to maintain, and often faster to serve.

If it doesn't, RAG is still an excellent choice.

The goal isn't to replace RAG.

It's to recognize that modern context windows have changed what's possible, and CAG deserves a place in the conversation.

TL;DR

RAG retrieves relevant information before every query.
CAG loads knowledge into the model's context upfront.
Modern context windows make CAG practical for many use cases.
If your knowledge fits in context, CAG is worth considering before reaching for a vector database.
For large or frequently changing knowledge bases, RAG remains the better choice.

Sometimes the simplest architecture is the one that gets out of the model's way.

How to Actually Decouple i18n Updates from Application Deploys

Sun, 10 May 2026 12:37:22 +0000

Edge-Native i18n with Astro & Cloudflare Workers - Part 3

In a previous article, I made a claim: translations are data, not code. Your Worker shouldn't care whether you support two languages or fifty. Fixing a typo in a German heading shouldn't require a software release.

This is a condensed version of a full technical deep-dive. The complete implementation - including production wrangler tail logs, the full fetcher.ts and bundle-translations.ts walkthrough, DX considerations, and a six-axis trade-off framework - is available at edgekits.dev.

I built an architecture around that claim. Translations live in Cloudflare KV. A SHA hash of the entire translation bundle - TRANSLATIONS_VERSION - bakes into the Worker as a build-time constant and embeds into every cache key:

const cacheId = `${PROJECT.id}:i18n:v${TRANSLATIONS_VERSION}:${lang}:${namespaces.join(',')}`

The theory: change a translation, regenerate the hash, and all old cache entries become addressed by a stale key that nothing will ever ask for again. Clean, deterministic, content-driven invalidation.

Then I deployed to production and noticed something uncomfortable.

I wanted to tweak the hero heading on the Spanish landing page. The only way to push that change was npm run i18n:migrate and wrangler deploy. Because the hash constant lived inside the Worker bundle, updating the hash meant rebuilding the entire application - every time, for every translation change.

The architecture shipped translations as data. But it invalidated them as code.

The Mental Model Mismatch

Here's what the architecture looked like on paper:

edit JSON  →  i18n:migrate  →  users see update

Here's what it actually required:

edit JSON  →  i18n:migrate  →  wrangler deploy  →  users see update
                                      ↑
                           this step is not optional

TRANSLATIONS_VERSION is a constant compiled into the Worker bundle. The only way to change its value at runtime is to rebuild the Worker and redeploy. So the cache keys in production don't change until you ship a new Worker - meaning all existing cache entries keep serving the old content, regardless of what you pushed to KV.

The translation update and the cache invalidation are two physically separate events. One is a KV write. The other is a code deployment. The architecture, despite its elegance, silently requires both.

For a solo project where you control both code and content, this is easy to miss - the deploy happens naturally during the normal development loop. But the moment a non-developer should be able to update translations - a content editor, a translator, a marketing teammate - the coupling becomes a real problem. You can't hand someone a workflow that requires a full application redeploy.

Three Dead Ends

I went through three intermediate approaches before arriving at the right one. The short version:

wrangler deploy --var TRANSLATIONS_VERSION=<hash> looks like it separates the version from the bundle. It doesn't. wrangler.jsonc variables take precedence over CLI flags, so the CLI override is silently discarded. The hash still lives in the bundle.

Version stored in KV, read at request time. This works, but it adds a mandatory KV read to every single request - even fully cached ones. The hot path, which should be a single cache lookup, now becomes a sequential chain: read KV for version, build cache key, check cache. The version KV entry can be cached too, but then you've got a double cache lookup, and the TTL on the version cache becomes a new source of propagation lag.

Version in KV with short TTL. Scales poorly across Cloudflare's global edge. Each of the ~300 edge nodes maintains its own cache independently. A short TTL means each node re-fetches the version on its own schedule, and the propagation window for a content update is bounded by that TTL - not by actual propagation speed. You've traded one coupling for another.

All three approaches share the same underlying assumption: the cache key should carry a version marker, and invalidation happens by rotating that marker. That assumption is the problem.

The Breakthrough: Static Keys + Explicit Invalidation

Every dead-end I hit was a variation on passive invalidation: nothing actively removes stale entries; we stop addressing them. The cache fills up with ghosts. Old content drifts around at the edge until Cloudflare's LRU policy decides to evict it.

The alternative is active invalidation: the cache key stays stable across content changes, and when translations update, we explicitly tell Cloudflare to delete the affected entries.

The new cache key shape:

i18n:<locale>:<namespace>

Wrapped into the URL shape that Cloudflare's Cache API expects:

https://<PROJECT.id>/<encoded-identifier>

One key per locale:namespace pair. Stable forever. The same key that stores Spanish landing translations today will store them in a year - whatever version "today" happens to be.

There's a secondary win here. The old architecture cached combined namespace sets: i18n:v<hash>:es:common,landing,newsletter was a single cache entry. A different page requesting common,landing got its own entry - with overlapping content. Same translations, cached multiple times under different keys.

With per-namespace static keys, each namespace is its own cache entry. A page that needs common, landing, and newsletter does three parallel cache lookups. Any other page requesting common gets a cache hit on that namespace - shared across requests, no duplication.

The Hot Path

A user hits /es/blog/some-article. The page needs common, blog, and newsletter. The fetcher issues three parallel cache lookups:

const cacheResults = await Promise.all(
  namespaces.map(async (ns) => {
    const req = buildTranslationCacheRequest(locale, ns)
    const cached = await cache.match(req)
    return { ns, data: cached ? await cached.json() : null, hit: !!cached }
  })
)

Full hit - all three cached - returns immediately. Zero KV reads.

Partial miss - say blog was recently invalidated - filters down to the missing namespace and issues one KV batch call:

const missing = cacheResults.filter((r) => !r.hit).map((r) => r.ns)
const kvKeys = missing.map((ns) => buildTranslationKvKey(locale, ns))
const kvBatch = await env.TRANSLATIONS.getMultiple(kvKeys, { type: 'json' })

One KV batch regardless of how many namespaces are missing. Results merge with the cache hits, get written back to cache, return.

This is better than both previous architectures on every metric I care about: zero KV reads on the hot path, one batch on partial miss, no cache bloat from combined keys, no version tracking overhead at request time.

The Cloudflare Purge API

If cache keys are stable, how does a translation update ever reach users?

The answer is a Cloudflare platform feature most developers know exists but have never used from code: the Purge API.

The request shape:

POST https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/purge_cache
Authorization: Bearer <API_TOKEN>
Content-Type: application/json

{
  "files": [
    "https://edgekits.dev/i18n%3Aen%3Alanding",
    "https://edgekits.dev/i18n%3Aes%3Alanding"
  ]
}

You hand it a list of URLs. It returns success and deletes those exact entries from the edge cache, globally, within about a second. The next request to each URL results in a cache miss, the Worker falls through to KV, and fresh content is re-cached under the same stable key.

Notice that the URLs in the purge request are the same URLs used as cache keys. The https://<PROJECT.id>/<encoded-key> shape is shared between cache.put inside the Worker and purge_cache in the migration script. The fetcher writes cache entries under a URL; the migration script deletes them under the same URL. One formula, two contexts. Any drift between them and the purge silently misses - which is why both sides import from the same translations-keys.ts module.

One constraint worth calling out clearly: the Purge API requires your domain to be proxied through Cloudflare - the orange cloud icon in your DNS records. On *.workers.dev subdomains, cache.put silently discards and cache.match always returns undefined. The Workers Cache API is tied to zone-level caching that doesn't exist for the shared subdomain. If your project isn't on a proxied custom domain, this architecture's signature feature is inactive.

Incremental Purging: The Hash File Strategy

Purging every possible locale:namespace URL on every migration run would work but produces a cache stampede - simultaneously invalidating content across all edge nodes triggers a burst of KV reads to re-populate entries that didn't change.

For a project with 5 locales and 10 namespaces, fixing a typo in en/landing.json would purge all 50 cache entries. 49 of those purges are wasted.

The fix: track which namespaces actually changed between migrations.

After every successful migration, compute a SHA hash of each locale:namespace JSON payload and store the results in .i18n-hashes.json. On the next run, recompute and diff:

const currentHashes: Record<string, string> = {}
for (const locale of locales) {
  for (const ns of namespaces) {
    currentHashes[`${locale}:${ns}`] = sha256(
      JSON.stringify(translations[locale][ns])
    )
  }
}

const previousHashes = readHashFile() ?? {}
const changedKeys = Object.keys(currentHashes).filter(
  (key) => currentHashes[key] !== previousHashes[key]
)

changedKeys is the exact list of pairs whose content differs from the last migration. Feed those into buildTranslationCacheUrl and you've got the precise list of URLs to purge. A single typo fix produces one or two API calls. Everything else stays warm.

One important detail: .i18n-hashes.json is gitignored local state. If the Purge API call fails - rate limit, expired token, network error - we deliberately don't update the hash file. On the next i18n:migrate run, the same namespaces still look "changed" and the retry happens automatically. Graceful recovery without manual intervention.

The full migration pipeline:

1. Read all locale JSON files
2. Compute current hashes per locale:namespace
3. Push translations to KV
4. Diff against previous hashes → changedKeys
5. If empty → skip purge
6. Build Purge URLs for changedKeys
7. Call Purge API in chunks of 30 (free tier rate limit)
8. On success → write updated hash file
9. On failure → log warning, skip hash update (retry next run)

After i18n:migrate completes successfully, the edge cache reflects the latest translations within seconds. No Worker redeploy. No version constant. No coupling.

When to Use This vs. the Original Architecture

Both the content-hash approach (Part 1) and this explicit-purge approach are valid - they solve the same problem under different constraints.

Use the original content-hash architecture when:

You don't want to manage an additional Cloudflare API token.
Translations change rarely - once a month or only at release time.
It's a solo project where "run two commands" isn't a real friction.
You're on *.workers.dev or a non-proxied domain.

Use the Purge API architecture when:

Your domain is proxied through Cloudflare (orange cloud DNS).
Translations change independently from code - content editors, translators, a CMS.
You want a non-developer to deploy content changes without touching the codebase.
Translation velocity matters: "seconds to propagate" vs. "wait for a deploy."

The key question is team composition and update frequency. If translations change once a month on release day, the redeploy requirement of the original architecture is nearly invisible - you're deploying anyway. If a marketing teammate needs to adjust hero copy weekly, the redeploy bottleneck becomes genuinely painful. Part 3 reduces that to one command.

For edgekits.dev, I knew I wanted a content workflow where non-developer updates were possible and translation iteration was cheap. The original architecture couldn't deliver that. For your project, the answer depends on which constraints actually dominate.

The Pattern Underneath

The refactor in this article was, underneath the implementation details, an exercise in putting each piece of state in the right layer.

Environment variables are for how the Worker is configured. They live with the Worker. They change when the Worker changes.

KV is for durable data the Worker reads. It lives independently of the Worker. It can change without a redeploy.

The Cache API is for transient acceleration. It's downstream of KV, populated by the Worker to save round-trips.

When TRANSLATIONS_VERSION lived in an env var, we were saying "this version is part of the Worker's identity" - and paying for that every time the version should change without the Worker itself changing. Putting versioning information in a build constant, then trying to update content independently of builds, was always going to produce friction. The state was in the wrong layer.

Static keys + explicit Purge API invalidation put each piece where it belongs: content in KV, cache entries downstream of KV, invalidation as an explicit out-of-band operation. Once the pieces sit in their right layers, the system composes cleanly and the coupling dissolves.

Get the Code
You don't have to build this from scratch. The entire architecture discussed today is available as an open-source starter kit: https://github.com/EdgeKits/astro-edgekits-core

The full deep-dive and complete implementation code for translations-keys.ts, fetcher.ts, and the migration script, production wrangler tail logs showing cache hit rates, and a detailed six-axis trade-off framework - is at edgekits.dev.

Performance Optimization: The Basics That Scale

Tue, 11 Nov 2025 21:09:58 +0000

Most developers wait until performance becomes a problem before doing anything about it. By then, you're firefighting instead of building. Here are the optimizations you should implement from day one - they cost almost nothing upfront but save you from major headaches later.

Database & Queries

Add indexes on columns you query frequently. Your database has to scan every row without them, and that gets slower as your data grows.
Select only what you need. SELECT * pulls data you don't use, wastes bandwidth, and makes your queries slower.
Avoid N+1 queries. If you're fetching data in a loop, you're probably making one database call per iteration when you could make just one.
Paginate early. Don't wait until you have thousands of records to add pagination. Implement it when you have ten.

Caching Strategy

Cache what's unavoidably complex. If a query or request takes time and the result doesn't change often, cache it. Don't cache just because you can.
Use a CDN for static assets. Images, CSS, and JavaScript files should be served from edge locations close to your users, not from your application server.

Network & Infrastructure

Put your application close to users. Deploy to regions where your users are - latency matters more than most developers think.
Put services close to each other. If your API and database are in different regions, every request crosses continents unnecessarily.
Fail fast. Set appropriate timeouts so slow operations don't block everything else. If something's going to fail, let it fail quickly.

Application Architecture

Process heavy work asynchronously. Sending emails, generating reports, or processing uploads shouldn't happen in the request-response cycle—use queues.
Measure before you optimize. You can't fix what you don't measure. Track response times and error rates from day one.

These aren't premature optimization. They're sensible defaults that prevent problems before they start.

The time to implement them is now, when your application is small and changes are easy. Trying to retrofit them into a large codebase under production load is significantly harder.

Do it right from the start, and you won't have to do it again later.

Intelligent Prediction State Management: Building Scalable Storage Abstractions for AI Applications

Thu, 11 Sep 2025 04:44:17 +0000

Creating flexible, environment-adaptive storage systems that seamlessly switch between Cloudflare KV and Durable Objects while maintaining consistent state management across distributed AI processing pipelines

The Challenge

Modern AI applications face complex state management challenges that traditional database approaches struggle to address effectively. When building systems that process long-running AI tasks like image generation, video processing, or complex language model operations, developers encounter several critical problems:

Storage Flexibility Requirements: Different deployment environments and scaling requirements demand different storage solutions. What works in development may not be optimal for production, and what performs well under light load may fail under heavy traffic.

Distributed State Consistency: AI processing often involves multiple services, edge locations, and asynchronous operations. Maintaining consistent state across these distributed components while ensuring data integrity becomes increasingly complex.

Performance vs. Persistence Trade-offs: Fast access to prediction status is crucial for user experience, but long-term persistence requirements vary. Some applications need millisecond response times, while others prioritize data durability and consistency.

Environment Adaptability: Applications need to adapt seamlessly to different infrastructure constraints. Development environments might use simple key-value stores, while production systems require more sophisticated distributed storage solutions.

Architecture Design Philosophy

The solution lies in building a storage abstraction layer that separates business logic from storage implementation details. This architecture enables applications to adapt to different environments and requirements while maintaining consistent behavior and API contracts.

Interface-Driven Design

Clean interfaces provide the foundation for flexible storage implementations:

interface IPredictionStorage {
  initialize(options: InitializePredictionOptions): Promise<KVStorageResult<void>>;
  get(predictionId: string): Promise<KVStorageResult<PredictionStatus>>;
  update(predictionId: string, updates: Partial<PredictionStatus>): Promise<KVStorageResult<PredictionStatus>>;
  delete(predictionId: string): Promise<KVStorageResult<void>>;
  list(options?: ListPredictionOptions): Promise<KVStorageResult<ListPredictionResult>>;
}

interface PredictionStatus {
  status: 'starting' | 'processing' | 'succeeded' | 'failed' | 'canceled';
  predictionId: string;
  createdAt: number;
  updatedAt: number;
  payload?: Record<string, any>;
  prompt?: string;
  model?: string;
  imageUrl?: string;
  imagePath?: string;
  error?: string;
  estimatedTime?: number;
  generationTime?: number;
}

This interface design ensures that switching between storage implementations requires no changes to business logic, enabling true storage abstraction.

Factory Pattern Implementation

The factory pattern centralizes storage instantiation and environment detection:

export class PredictionStorageFactory {
  private static instance: IPredictionStorage | null = null;

  static create(): IPredictionStorage {
    if (this.instance) {
      return this.instance;
    }

    const provider = this.getCurrentProvider();
    
    switch (provider) {
      case StorageProvider.DO:
        this.instance = new DOPredictionStorage();
        break;
        
      case StorageProvider.KV:
      default:
        this.instance = new KVPredictionStorage();
        break;
    }

    return this.instance;
  }

  static getCurrentProvider(): StorageProvider {
    return (process.env.PREDICTION_STORAGE_PROVIDER || 'kv') as StorageProvider;
  }
}

This factory pattern provides several key benefits:

Environment Detection: Automatically selects the appropriate storage implementation based on configuration without requiring code changes.

Singleton Pattern: Ensures consistent storage instances across the application lifecycle, preventing configuration drift and resource waste.

Testing Flexibility: Enables easy mocking and testing by providing a reset mechanism for test environments.

Core Implementation Components

Unified State Manager

The state manager provides a high-level API that abstracts storage complexity:

export class PredictionStatusManager {
  private static storage: IPredictionStorage = PredictionStorageFactory.create();

  static async initializePredictionStatus(options: {
    predictionId: string;
    payload: Record<string, any>;
    prompt: string;
    model: string;
    estimatedTime?: number;
  }): Promise<KVStorageResult<void>> {
    return this.storage.initialize(options);
  }

  static async getPredictionStatus(predictionId: string): Promise<KVStorageResult<PredictionStatus>> {
    return this.storage.get(predictionId);
  }

  static async markAsProcessing(predictionId: string): Promise<KVStorageResult<PredictionStatus>> {
    return this.updatePredictionStatus(predictionId, { status: 'processing' });
  }

  static async markAsSucceeded(
    predictionId: string,
    imageUrl: string,
    imagePath: string,
    generationTime?: number
  ): Promise<KVStorageResult<PredictionStatus>> {
    return this.updatePredictionStatus(predictionId, {
      status: 'succeeded',
      imageUrl,
      imagePath,
      generationTime,
    });
  }

  static async markAsFailed(
    predictionId: string,
    error: string
  ): Promise<KVStorageResult<PredictionStatus>> {
    return this.updatePredictionStatus(predictionId, {
      status: 'failed',
      error,
    });
  }
}

The state manager pattern provides several advantages:

Simplified API: Business logic uses intuitive methods like markAsSucceeded rather than complex storage operations.

Atomic Operations: Each state transition is handled as a complete unit, ensuring data consistency.

Audit Trail: All state changes are tracked with timestamps and metadata for debugging and analytics.

Multi-Provider Storage Strategy

Cloudflare KV Implementation

The KV storage implementation optimizes for global distribution and eventual consistency:

export class KVPredictionStorage implements IPredictionStorage {
  private static readonly KEY_PREFIX = 'prediction:';
  private static readonly DEFAULT_TTL = 3600; // 1 hour

  async initialize(options: InitializePredictionOptions): Promise<KVStorageResult<void>> {
    const { predictionId, payload, prompt, model, estimatedTime } = options;

    const status: PredictionStatus = {
      status: 'starting',
      predictionId,
      createdAt: Date.now(),
      updatedAt: Date.now(),
      payload,
      prompt,
      model,
      estimatedTime,
    };

    return await putKV(
      `${KVPredictionStorage.KEY_PREFIX}${predictionId}`,
      status,
      { expirationTtl: KVPredictionStorage.DEFAULT_TTL }
    );
  }

  async update(
    predictionId: string,
    updates: Partial<PredictionStatus>
  ): Promise<KVStorageResult<PredictionStatus>> {
    const existingResult = await this.get(predictionId);
    if (!existingResult.success || !existingResult.data) {
      return existingResult;
    }

    const updatedStatus: PredictionStatus = {
      ...existingResult.data,
      ...updates,
      updatedAt: Date.now(),
    };

    const saveResult = await putKV(
      `${KVPredictionStorage.KEY_PREFIX}${predictionId}`,
      updatedStatus,
      { expirationTtl: KVPredictionStorage.DEFAULT_TTL }
    );

    return saveResult.success ? 
      { success: true, data: updatedStatus } : 
      { success: false, errorCode: saveResult.errorCode };
  }
}

KV storage provides specific advantages:

Global Distribution: Data replicates automatically across Cloudflare's global network for low-latency access worldwide.

Automatic Expiration: Built-in TTL support ensures efficient cleanup of temporary prediction data without manual maintenance.

High Availability: Eventually consistent model provides excellent availability characteristics for read-heavy workloads.

Durable Objects Implementation

The Durable Objects implementation focuses on strong consistency and stateful operations:

export class DOPredictionStorage implements IPredictionStorage {
  private async getBaseDORpcService() {
    const { env } = await getCloudflareContext({ async: true });
    const { BASE_DO_SERVICE } = env;
    
    if (!BASE_DO_SERVICE) {
      throw new Error('BASE_DO_SERVICE binding not found');
    }

    return BASE_DO_SERVICE;
  }

  private getDOName(predictionId: string): string {
    return `prediction-${predictionId}`;
  }

  async initialize(options: InitializePredictionOptions): Promise<KVStorageResult<void>> {
    const { predictionId, payload, prompt, model, estimatedTime } = options;

    const status: PredictionStatus = {
      status: 'starting',
      predictionId,
      createdAt: Date.now(),
      updatedAt: Date.now(),
      payload,
      prompt,
      model,
      estimatedTime,
    };

    const doData = {
      status,
      metadata: {
        createdAt: status.createdAt,
        updatedAt: status.updatedAt,
        status: status.status,
      },
    };

    try {
      const rpcService = await this.getBaseDORpcService();
      const doName = this.getDOName(predictionId);
      
      await rpcService.set(doName, doData, 3600000); // 1 hour TTL
      
      return { success: true };
    } catch (error) {
      return {
        success: false,
        errorCode: ECommonErrorCode.INTERNAL_SERVER_ERROR,
      };
    }
  }

  async update(
    predictionId: string,
    updates: Partial<PredictionStatus>
  ): Promise<KVStorageResult<PredictionStatus>> {
    try {
      const rpcService = await this.getBaseDORpcService();
      const doName = this.getDOName(predictionId);
      const existingData = await rpcService.get(doName);
      
      if (!existingData || !existingData.status) {
        return {
          success: false,
          errorCode: ECommonErrorCode.SERVICE_UNAVAILABLE,
        };
      }

      const updatedStatus: PredictionStatus = {
        ...existingData.status,
        ...updates,
        updatedAt: Date.now(),
      };

      const updatedData = {
        status: updatedStatus,
        metadata: {
          createdAt: existingData.metadata.createdAt,
          updatedAt: updatedStatus.updatedAt,
          status: updatedStatus.status,
        },
      };

      await rpcService.set(doName, updatedData, 3600000);

      return {
        success: true,
        data: updatedStatus,
      };
    } catch (error) {
      return {
        success: false,
        errorCode: ECommonErrorCode.INTERNAL_SERVER_ERROR,
      };
    }
  }
}

Durable Objects provide distinct benefits:

Strong Consistency: Each prediction gets its own isolated instance with guaranteed consistency for all operations.

Stateful Processing: Can maintain complex state transitions and perform multi-step operations atomically.

Geographic Affinity: Instances are created close to users for optimal performance while maintaining consistency.

State Machine Pattern Integration

Predictable State Transitions

The architecture implements a clear state machine that governs prediction lifecycle:

enum PredictionStatus {
  Starting = 'starting',
  Processing = 'processing', 
  Succeeded = 'succeeded',
  Failed = 'failed',
  Canceled = 'canceled',
}

// Valid state transitions
const VALID_TRANSITIONS = {
  starting: ['processing', 'failed', 'canceled'],
  processing: ['succeeded', 'failed', 'canceled'],
  succeeded: [], // Terminal state
  failed: [], // Terminal state  
  canceled: [], // Terminal state
};

Atomic State Updates with Validation

State transitions include validation and metadata tracking:

async function validateAndUpdateStatus(
  predictionId: string,
  newStatus: PredictionStatus,
  metadata?: Record<string, any>
): Promise<KVStorageResult<PredictionStatus>> {
  const current = await PredictionStatusManager.getPredictionStatus(predictionId);
  
  if (!current.success || !current.data) {
    return current;
  }

  // Validate transition
  const currentStatus = current.data.status;
  const validTransitions = VALID_TRANSITIONS[currentStatus] || [];
  
  if (!validTransitions.includes(newStatus)) {
    return {
      success: false,
      errorCode: ECommonErrorCode.INVALID_PARAMETER,
      message: `Invalid transition from ${currentStatus} to ${newStatus}`,
    };
  }

  // Perform atomic update
  return PredictionStatusManager.updatePredictionStatus(predictionId, {
    status: newStatus,
    ...metadata,
    updatedAt: Date.now(),
  });
}

This validation layer prevents invalid state transitions and maintains data integrity across the entire prediction lifecycle.

Integration Patterns

API Route Integration

REST endpoints use the abstracted state manager for consistent behavior:

export async function GET(
  request: NextRequest,
  { params }: { params: { predictionId: string } }
): Promise<NextResponse> {
  try {
    const { predictionId } = await params;

    const result = await PredictionStatusManager.getPredictionStatus(predictionId);

    if (!result.success || !result.data) {
      return NextResponse.json(
        { error: 'Prediction not found' },
        { status: 404 }
      );
    }

    const status = result.data;

    return NextResponse.json({
      success: true,
      data: {
        predictionId: status.predictionId,
        status: status.status,
        imageUrl: status.imageUrl,
        error: status.error,
        estimatedTime: status.estimatedTime,
        generationTime: status.generationTime,
        createdAt: status.createdAt,
        updatedAt: status.updatedAt,
      },
    });
  } catch (error) {
    return NextResponse.json(
      { error: 'Internal server error' },
      { status: 500 }
    );
  }
}

Webhook Processing Integration

Webhook handlers use the same abstraction for status updates:

async function handleSuccessfulPrediction(predictionId: string, event: WebhookEvent) {
  try {
    // Extract result data
    const imageUrl = Array.isArray(event.output) ? event.output[0] : event.output;
    const generationTime = event.metrics?.predict_time * 1000;

    // Download and store the result
    const imageResponse = await fetch(imageUrl);
    const imageBuffer = await imageResponse.arrayBuffer();
    
    const uploadResult = await uploadFile({
      path: 'predictions',
      fileName: `${Date.now()}-${predictionId}.png`,
      file: imageBuffer,
      contentType: 'image/png',
    });

    const finalImageUrl = generatePublicUrl(uploadResult.filePath);

    // Update prediction status atomically
    await PredictionStatusManager.markAsSucceeded(
      predictionId,
      finalImageUrl,
      uploadResult.filePath,
      generationTime
    );

    console.log('Prediction completed successfully:', predictionId);

  } catch (error) {
    // Handle failure with automatic status update
    await PredictionStatusManager.markAsFailed(
      predictionId,
      `Processing failed: ${error.message}`
    );
    
    throw error;
  }
}

Asynchronous Task Management

The state manager integrates seamlessly with background job processing:

async function initializeAsyncPrediction(
  predictionId: string,
  payload: Record<string, any>,
  model: string
): Promise<void> {
  // Initialize prediction state
  await PredictionStatusManager.initializePredictionStatus({
    predictionId,
    payload,
    prompt: generatePrompt(payload),
    model,
    estimatedTime: getEstimatedTime(model),
  });

  // Submit to AI service
  const aiResponse = await submitToAIService({
    predictionId,
    payload,
    model,
    webhookUrl: `${process.env.API_BASE_URL}/api/webhooks/ai-service`,
  });

  // Update status to processing
  if (aiResponse.success) {
    await PredictionStatusManager.markAsProcessing(predictionId);
  } else {
    await PredictionStatusManager.markAsFailed(
      predictionId,
      `AI service submission failed: ${aiResponse.error}`
    );
  }
}

Performance and Reliability Characteristics

Storage Performance Comparison

Cloudflare KV Metrics:

Read latency: 10-50ms globally
Write latency: 100-300ms (eventual consistency)
Throughput: 10,000+ reads/second per key
Automatic global replication
Built-in TTL and cleanup

Durable Objects Metrics:

Read latency: 5-20ms within region
Write latency: 20-50ms (strong consistency)
Throughput: 1,000+ operations/second per object
Single-region strong consistency
Manual cleanup required

Environment Adaptation Strategies

The architecture adapts automatically to different deployment scenarios:

function getOptimalStorageProvider(): StorageProvider {
  // Development: Use KV for simplicity
  if (process.env.NODE_ENV === 'development') {
    return StorageProvider.KV;
  }

  // High-consistency requirements: Use DO
  if (process.env.REQUIRE_STRONG_CONSISTENCY === 'true') {
    return StorageProvider.DO;
  }

  // High-throughput requirements: Use KV
  if (process.env.OPTIMIZE_FOR_THROUGHPUT === 'true') {
    return StorageProvider.KV;
  }

  // Default to KV for most production scenarios
  return StorageProvider.KV;
}

Production Benefits and Scale

Deployment Flexibility

This architecture has been successfully deployed at AI Jersey Design, an AI jersey design generator that processes thousands of image generation requests daily. The system demonstrates several key production benefits:

Environment Adaptability: Development environments use KV storage for simplicity, while production can switch to Durable Objects when strong consistency is required, all without code changes.

Performance Optimization: The abstraction layer enables A/B testing different storage strategies to optimize for specific workload patterns.

Operational Simplicity: Unified APIs reduce operational complexity while maintaining the flexibility to optimize storage choices based on requirements.

Scale Performance Metrics

Throughput Capacity:

KV implementation: 15,000+ status checks per minute
DO implementation: 8,000+ status updates per minute
Average response time: 45ms (KV), 25ms (DO within region)
99.7% uptime across both storage implementations

Resource Efficiency:

Memory usage: 12MB per 1,000 active predictions (KV), 8MB (DO)
Automatic TTL cleanup reduces storage costs by 40%
Zero manual maintenance required for state cleanup

Developer Experience:

Single API for all storage operations
Environment-specific optimization without code changes
Built-in error handling and retry mechanisms
Comprehensive TypeScript support for type safety

How IBM Mainframe Cache Architecture Outperforms Traditional Server CPUs?

Thu, 19 Jun 2025 04:42:42 +0000

Introduction

In the world of high-performance enterprise computing, IBM mainframes are renowned for their unmatched reliability, throughput, and scalability. At the core of this superiority lies a fundamental difference in architectural design—a sophisticated, multi-layered cache hierarchy that significantly outpaces traditional server CPUs. While x86 architectures dominate commodity computing, IBM’s z-series mainframes, such as the z15 and z16, bring an advanced memory subsystem to the table that dramatically enhances performance for mission-critical workloads.

This article delves into the cache hierarchy of IBM mainframes, explaining its structure, benefits, and why it consistently outperforms traditional server processors in real-world applications.

Understanding Cache Hierarchy in CPUs

Before exploring the IBM mainframe, it’s essential to understand how CPU caches work in general.

What is Cache?

CPU caches are small, fast memory layers located closer to the processor cores than main memory (RAM). Their purpose is to store frequently accessed data and instructions, reducing the time it takes for the CPU to fetch them.

Traditional x86 Server Cache Hierarchy

Most x86 server CPUs (e.g., Intel Xeon, AMD EPYC) implement a three-level cache:

L1 Cache: Per core, very fast but small (32KB–64KB).
L2 Cache: Per core, larger but slower (256KB–1MB).
L3 Cache: Shared among cores on a CPU socket (up to 96MB in AMD EPYC).

These caches are typically built using SRAM, known for its speed and cost efficiency for small capacities.

️ The IBM Mainframe Cache Hierarchy: A Four-Tiered Powerhouse

IBM’s mainframes elevate cache design to a new level by implementing a four-level hierarchy with an additional L4 system cache, rarely seen in traditional CPUs.

1. L1 Cache (Per Core)

Split into instruction (L1I) and data (L1D) caches.
Offers ultra-low latency access (typically 1–2 cycles).
Holds most frequently used data, such as loop counters or stack variables.

2. L2 Cache (Per Core)

Slightly larger (256KB–2MB range), also private to each core.
Used for holding recently accessed variables, array data, and small working sets.

3. L3 Cache (Per Chip)

Shared among all cores on a chip.
Implemented using embedded DRAM (eDRAM) for high density and power efficiency.
Much larger (e.g., 128MB per chip), serving as a large buffer for workloads like database queries or transaction processing.

4. L4 Cache (System-Level Cache)

The defining innovation in IBM’s architecture.
Shared across the entire Central Processor Complex (CPC).
Can be as large as 960MB+, acting as a last-resort cache before memory.
Accelerates access for workloads spanning multiple cores, chips, or logical partitions (LPARs).

Performance Impact: How the Cache Hierarchy Translates to Real-World Gains

IBM’s cache system offers several tangible benefits that dramatically impact workload performance:

✔️ 1. Higher Throughput and Lower Latency

The L4 cache absorbs many L3 cache misses, significantly reducing round trips to DRAM.
This is critical in online transaction processing (OLTP) systems where latency is measured in microseconds.

L4 cache serves as a shared resource between cores, sockets, and logical partitions.
Enables faster context switches, shared-memory communication, and fewer performance bottlenecks across concurrent workloads.

✔️ 3. Workload Isolation and Predictability

Each core has private L1/L2 caches, while L4 provides a shared but controlled buffer.
Supports workload isolation—ideal for cloud environments and mainframe-as-a-service where performance predictability is crucial.

✔️ 4. Better Cache Hit Rates

Larger caches (L3+L4) mean less frequent data eviction and re-fetching.
Particularly beneficial for analytics workloads or mainframe batch jobs with large working sets.

IBM vs Traditional x86 Servers: Side-by-Side Comparison

Feature	IBM z15/z16 Mainframe	Intel Xeon / AMD EPYC Servers
Cache Levels	L1–L4	L1–L3
L4 Cache	Present (shared across CPC)	Not present
Cache Size	Up to 960MB (L4), 128MB (L3)	Up to 96MB (L3)
Cache Type	eDRAM (for L3, L4)	SRAM
Performance Target	High throughput, isolation, reliability	High performance, cost efficiency
Workload Suitability	OLTP, hybrid cloud, security-critical	Web servers, databases, HPC workloads

Use Case Impact: Who Benefits the Most?

IBM’s cache architecture shines in industries and applications where:

Latency and consistency are non-negotiable.
Concurrent workloads run simultaneously.
Security and uptime are critical.

Industries:

Banking and Finance (real-time transactions)
Insurance and Claims Management
Government and Defense
Large Retail and Logistics
Healthcare Information Systems

Applications:

z/OS with DB2 or IMS databases
Secure APIs and encryption services
Batch analytics and mainframe-based ETL
Cloud-native workloads on LinuxONE

Architectural Trade-offs

Advantage	Trade-off
Better performance under scale	More silicon real estate required
Superior workload isolation	Higher hardware costs
System-wide data sharing via L4	Complexity in cache coherence management
Lower memory access latency	Power usage from large eDRAM cache

Conclusion

The cache hierarchy in IBM mainframes is a critical differentiator that contributes directly to their legendary performance, reliability, and scalability. The introduction of a dedicated L4 system cache, shared across the processor complex, allows IBM to deliver high throughput while minimizing latency and improving workload isolation.

While traditional server CPUs offer excellent performance per dollar, they fall short in scenarios where predictability, fault isolation, and sheer transactional throughput are paramount.

For organizations dealing with mission-critical workloads, IBM’s architectural investment in cache design isn’t just an engineering marvel—it’s a business imperative.

Coder Legion - Recent posts tagged cache-layer

Semantic Caching for LLM Apps Without Serving the Wrong Answer

Semantic Caching for LLM Apps Without Serving the Wrong Answer

Three caches that get confused

First, an expected-value check

The design: similarity proposes, rules decide

The table

The lookup

Guard the words embeddings miss

The grey zone: verify, don't guess

Calibrate thresholds on your own data

Scope or leak: the cache is a cross-user channel

Invalidation: tie entries to their sources

Multi-turn conversations

Roll it out in shadow mode

Cached answers are personal data processing too

FAQ

The short version

CAG: The Simpler Way to Ground Your LLM

Why RAG Became So Popular

Meet CAG

Why Is This Possible Now?

RAG vs CAG

A Simple Example

Why Not Use Both?

Final Thoughts

TL;DR

How to Actually Decouple i18n Updates from Application Deploys

The Mental Model Mismatch

Three Dead Ends

The Breakthrough: Static Keys + Explicit Invalidation

The Hot Path

The Cloudflare Purge API

Incremental Purging: The Hash File Strategy

When to Use This vs. the Original Architecture

The Pattern Underneath

Performance Optimization: The Basics That Scale

Database & Queries

Caching Strategy

Network & Infrastructure

Application Architecture

Intelligent Prediction State Management: Building Scalable Storage Abstractions for AI Applications

The Challenge

Architecture Design Philosophy

Interface-Driven Design

Factory Pattern Implementation

Core Implementation Components

Unified State Manager

Multi-Provider Storage Strategy

Cloudflare KV Implementation

Durable Objects Implementation

State Machine Pattern Integration

Predictable State Transitions

Atomic State Updates with Validation

Integration Patterns

API Route Integration

Webhook Processing Integration

Asynchronous Task Management

Performance and Reliability Characteristics

Storage Performance Comparison

Environment Adaptation Strategies

Production Benefits and Scale

Deployment Flexibility

Scale Performance Metrics

How IBM Mainframe Cache Architecture Outperforms Traditional Server CPUs?

Introduction

Understanding Cache Hierarchy in CPUs

What is Cache?

Traditional x86 Server Cache Hierarchy

️ The IBM Mainframe Cache Hierarchy: A Four-Tiered Powerhouse

1. L1 Cache (Per Core)

2. L2 Cache (Per Core)

3. L3 Cache (Per Chip)

4. L4 Cache (System-Level Cache)

Performance Impact: How the Cache Hierarchy Translates to Real-World Gains

✔️ 1. Higher Throughput and Lower Latency

✔️ 2. System-Wide Data Sharing

✔️ 3. Workload Isolation and Predictability

✔️ 4. Better Cache Hit Rates

IBM vs Traditional x86 Servers: Side-by-Side Comparison