Coder Legion - Recent posts tagged caching

Polar Bear Cache: Local Caching Without Repeated Remote Calls

Mon, 26 Oct 2026 10:07:01 +0000

Shared storage provides the source of truth. Local caches provide speed. Events help keep cache state in sync.

Project: Polar Bear Cache — GitHub repository

In my previous discussions about microservices, I explored how shared storage and event-driven synchronization can work together to manage distributed session state.

This time, I want to focus on another part of the problem: caching frequently accessed data across multiple service instances.

The Problem: Faster Reads, Distributed Cache Invalidation

In a microservices architecture, multiple service instances often need the same data, especially data that is read frequently but updated relatively infrequently.

A distributed cache can reduce database load, but accessing it still involves network round trips and potentially serialization and deserialization overhead.

So, what if each service instance could keep frequently accessed data in its own local cache, while receiving notifications when that data changes elsewhere?

That is the idea behind Polar Bear Cache, an open-source Java caching library designed to work with Spring's familiar cache annotations.

The Design: Local Cache + Event-Driven Invalidation

Each service instance maintains its own local cache. When data changes, an event service can notify other instances that specific cache entries need to be invalidated.

The key principle is simple:

Broadcast invalidation events, not the entire cached data.

Polar Bear Cache integrates with Spring's cache annotations:

@Cacheable — Returns a cached value when available, or executes the underlying method to load it when the value is missing.
@CachePut — Updates the local cache and, when event-service integration is configured, notifies other instances to invalidate the corresponding entry.
@CacheEvict — Evicts cached data locally and, when event-service integration is configured, notifies other instances to invalidate their corresponding entries.

For example, when a product is updated:

The application updates the source of truth and its local cache.
It publishes a cache-invalidation event through the event service.
Other service instances receive the event and evict the corresponding local entry.
Their next read loads the data again from the source of truth.

This allows repeated reads to stay local in the common case, while cross-instance communication is used to coordinate invalidation when data changes.

When Does This Approach Make Sense?

This design is particularly suitable for:

Read-heavy workloads with relatively infrequent updates.
Applications running multiple service instances, each with its own local cache.
Use cases where brief delays in cache invalidation are acceptable.

There is an important trade-off: event-driven invalidation is asynchronous. Between a data update and the time other instances process the invalidation event, stale data may still be served.

Polar Bear Cache does not guarantee strong consistency across service instances.

Depending on the application, you may also need to consider event-delivery reliability, duplicate events, cache expiration, and recovery strategies when a service instance misses an event.

The goal is not to eliminate every consistency trade-off, but to make local caching practical in distributed applications without requiring every read to go through a remote cache.

Why “Polar Bear”?

The name comes from a somewhat optimistic engineering idea:

Local caching
→ Fewer remote cache requests and network round trips
→ Potentially more throughput from the same resources
→ Potentially fewer servers needed for a given workload
→ Lower energy consumption and greenhouse gas emissions
→ Maybe we can help save polar bears. 🐻❄️🧊

Of course, the actual impact depends on workload characteristics, infrastructure, and deployment choices. Performance improvements do not automatically translate into lower energy consumption, but I like the idea that better resource efficiency could benefit both applications and the environment.

I'd Love to Hear Your Thoughts

I'm interested in how other engineers approach local caching in distributed systems.

How do you handle cache invalidation across service instances?
Do you prefer event-driven invalidation, short TTLs, versioned keys, or a combination?
What failure scenarios would you consider essential to address in this design?

If you're interested in the implementation, feedback and technical discussion are very welcome.

Less remote access, more ice. 🐻❄️🧊

Java #Spring #Microservices #Caching #DistributedSystems #OpenSource

Caching Strategies Demystified: Cache-Aside, Write-Through, and Write-Behind

Fri, 09 Oct 2026 10:55:06 +0000

In software development, caching is the practice of storing frequently used data in a high-speed, temporary storage layer so future requests can be served faster. Caching strategies are the architectural patterns that dictate how your application coordinates the movement of data between this high-speed temporary memory (the cache) and your permanent, slower storage system (the database). Choosing the right strategy ensures your application stays incredibly fast while keeping data accurate and consistent.

The Notebook and the Library Analogy

Imagine you are a researcher studying in a massive university library (the database) containing millions of books.

Cache-Aside (Lazy Loading): You keep a blank personal notebook (the cache) on your desk. When you need a fact, you look in your notebook first. If it is there (a cache hit), you read it instantly. If not (a cache miss), you walk into the library archives, search the shelves, find the fact, write it down in your notebook, and then read it.
Write-Through: Every time you discover a new piece of information, you write it in your personal notebook and immediately send a copy to the library's archiving desk to ensure both records are updated simultaneously before you proceed.
Write-Behind (Write-Back): When you are brainstorming rapidly, you scribble your notes exclusively in your notebook. At the end of the day, you hand your notebook to a library assistant who updates the main archives in the background while you sleep.

Why Caching Strategies Matter in the Tech Industry

Without these strategies, modern platforms would collapse. If a viral social media post forces millions of users to fetch the same user profile directly from a database, the database will exhaust its connection pool, slow down, and eventually crash.

Software engineers use these patterns strategically to balance speed and data integrity:

Cache-Aside prevents databases from being choked by heavy read traffic while keeping implementation simple.
Write-Through avoids data discrepancies, ensuring that critical data (like bank account balances) is never out of sync between the memory and the disk.
Write-Behind is utilized in write-heavy systems (like logging user activity or IoT sensor tracking) to group thousands of database writes into batch updates, protecting disk-based storage from wearing out or bottlenecking.

Implementing Cache-Aside in Node.js & Express

Here is how you can implement a standard Cache-Aside strategy using Express.js and MySQL. In this scenario, we check an in-memory cache object before querying our relational database.

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

// Mocking our fast memory cache (e.g., Redis)
const memoryCache = new Map();

// MySQL connection pool configuration
const dbPool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  database: 'production_db'
});

// Route utilizing Cache-Aside
app.get('/api/users/:id', async (req, res) => {
  const userId = req.params.id;
  const cacheKey = `user:${userId}`;

  try {
    // 1. Check if the data exists in the high-speed cache
    if (memoryCache.has(cacheKey)) {
      console.log('Cache Hit! Serving from memory.');
      return res.json(memoryCache.get(cacheKey));
    }

    // 2. Cache Miss: Retrieve data from the slow MySQL database
    console.log('Cache Miss. Fetching from MySQL database...');
    const [rows] = await dbPool.query('SELECT id, name, email FROM users WHERE id = ?', [userId]);

    if (rows.length === 0) {
      return res.status(404).json({ error: 'User not found' });
    }

    const userData = rows[0];

    // 3. Write data to the cache so the next request is instant
    memoryCache.set(cacheKey, userData);

    return res.json(userData);
  } catch (error) {
    return res.status(500).json({ error: 'Internal Server Error' });
  }
});

app.listen(3000, () => console.log('Server running on port 3000'));

Key Takeaway

Caching is not a one-size-fits-all solution; it is a careful compromise between performance and accuracy. While caching can make your application feel instantaneous, selecting the wrong update strategy can lead to stale data displays or lost records, making it vital to match your architectural pattern to your application's specific read-and-write ratios.

Originally published on my blog. You can read the alternative breakdown here.

Semantic Caching for LLM Apps Without Serving the Wrong Answer

Wed, 23 Sep 2026 14:39:11 +0000

Semantic Caching for LLM Apps Without Serving the Wrong Answer

A customer asks your support assistant: "Can I get a refund if I cancel within 14 days?" Your semantic cache finds a stored question that is almost word-for-word the same — "Can I get a refund if I cancel after 14 days?" — scores it above your threshold, and returns the stored answer before the model is even called. The answer is fluent, confident, and exactly wrong.

That exchange is the whole risk of semantic caching. The idea itself is sound: when a new question means the same thing as one you have already answered, reuse the answer and skip the model — no output tokens, no generation latency. The naive build (embed the question, take the nearest neighbour, serve it above 0.9) is also how you ship wrong answers at scale, because embedding similarity measures how close two questions are in topic, not whether they have the same answer.

You can't tune your way out of that with a better number. The vCache paper (ICLR 2026) found that "correct and incorrect cache hits have highly overlapping similarity distributions, suggesting that fixed thresholds are either unreliable or must be set extremely high to avoid errors." Separate research on universal text embeddings found "a significant lack of negation awareness", with models "often interpreting negated text pairs as semantically similar." Negation, numbers and conditions are precisely the words that decide a support answer.

So this article treats similarity as a way to find candidates, never as the verdict: scoped keys, two thresholds, a deterministic guard for the words embeddings miss, a cheap verifier for the grey zone, invalidation tied to your sources, and a shadow-mode rollout that measures the false-hit rate before one cached answer reaches a user. It is the caching layer we cover when teaching how to ship an AI product to real traffic at Cursuri-AI.ro, written out in code.

Three caches that get confused

	Exact-match response cache	Semantic response cache	Provider prompt caching
Key	Hash of the normalized request	Embedding of the question, inside a scope	Exact prefix of the prompt
What it skips	The whole model call	The whole model call	Re-processing the cached prefix
Saves	Input, output, latency	Input, output, latency	Most of the input cost of the prefix
Can it return a wrong answer?	Only if the key is incomplete	Yes, by design	No, the model still generates
Lives in	Your infrastructure	Your infrastructure	The provider

Prompt caching isn't a competitor. It makes re-reading a stable prefix cheap: on Anthropic's current lineup a cache read costs 10% of the base input price, 5% on Claude Opus 5.5 and 2.5% on Claude Fable 5.1. But the model still runs and still writes every output token. A semantic cache skips the model entirely.

That shifts the business case. When your prompts already cache well, the input side of a request is cheap, and a semantic cache mostly buys you output tokens and latency. Use both layers: a semantic-cache miss still goes to the model with a cached prefix.

First, an expected-value check

A semantic cache has a benefit you can calculate and a cost teams usually leave out: the wrong answers it serves. Put both in one function before you build anything.

def monthly_value(
    requests: int,
    hit_rate: float,              # share of requests answered from cache
    false_hit_rate: float,        # share of cache hits that were wrong
    usd_per_generation: float,    # average cost of a fresh answer
    usd_per_wrong_answer: float,  # support ticket, refund, lost customer: your number
    usd_per_lookup: float,        # embedding call + index query
) -> tuple[float, int]:
    hits = requests * hit_rate
    wrong = hits * false_hit_rate
    value = (hits * usd_per_generation
             - requests * usd_per_lookup
             - wrong * usd_per_wrong_answer)
    return value, round(wrong)

Run it with illustrative numbers: 200,000 questions a month, a 25% hit rate, $0.02 per generated answer, and a wrong answer that costs $5 of support time. At a 2% false-hit rate the cache saves $1,000 of generation and serves 1,000 wrong answers worth $5,000: a net loss of about $4,000 before lookup costs. At 0.2% it serves 100 wrong answers and comes out about $500 ahead. The hit rate sets how much you can save. The false-hit rate decides whether you save anything at all.

Where the arithmetic tends to work: support and FAQ assistants, documentation Q&A, internal help desks, product questions — high repetition, long answers, one correct answer for everyone in a scope. That is the territory of AI-assisted customer support, and it is where the rest of this design pays for itself. Where it doesn't work: coding agents, anything personalized, multi-step tool use, creative generation (users expect variety), and questions about live state such as "is the API down right now?".

The design: similarity proposes, rules decide

Every request goes through the same six steps:

Normalize the question (case, whitespace; multi-turn chats are covered further down).
Scope it: compute a key from everything that can change the correct answer — tenant, locale, plan, model, prompt version, knowledge-base version. Lookups never cross scopes.
Exact match first: a hash lookup catches verbatim repeats with no similarity risk.
Nearest neighbour within the scope.
Guard: below the low threshold, a miss. Different critical tokens, a miss. Above the high threshold, a hit. In between, verify.
On a miss, generate the answer, then decide at write time whether it may be cached at all.

The table

Postgres with pgvector is enough for this; use Redis or a vector store if that is what you already run. Storage is not the hard part.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE answer_cache (
    id              BIGSERIAL    PRIMARY KEY,
    scope_key       TEXT         NOT NULL,  -- hash of tenant, locale, plan, model, prompt and KB versions
    question        TEXT         NOT NULL,  -- normalized question
    question_hash   TEXT         NOT NULL,  -- exact-match layer
    question_sig    TEXT         NOT NULL,  -- critical tokens, see below
    embedding       VECTOR(1024) NOT NULL,  -- match your embedding model's dimension
    answer          TEXT         NOT NULL,
    source_doc_ids  TEXT[]       NOT NULL DEFAULT '{}',
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ  NOT NULL,
    hit_count       INT          NOT NULL DEFAULT 0,
    UNIQUE (scope_key, question_hash)
);

CREATE INDEX ON answer_cache USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON answer_cache USING gin (source_doc_ids);

One pgvector detail bites multi-tenant caches. As the README puts it, "with approximate indexes, filtering is applied after the index is scanned". With many scopes, the nearest neighbours the HNSW scan returns can all belong to other tenants, the WHERE scope_key = … filter removes them, and you get zero rows: a miss that isn't one. Turn on iterative index scans (pgvector 0.8.0+, SET hnsw.iterative_scan = relaxed_order), use partial indexes for a few large scopes, or partition when there are many.

The lookup

import hashlib
from dataclasses import dataclass

HIGH, LOW = 0.95, 0.88   # placeholders, not recommendations: calibrate them (see below)

@dataclass
class Candidate:
    id: int
    question: str
    answer: str
    similarity: float

def normalize(q: str) -> str:
    return " ".join(q.lower().split())

def lookup(db, scope_key: str, question: str) -> tuple[str, Candidate | None]:
    q = normalize(question)
    exact = db.fetchone(
        "SELECT id, question, answer FROM answer_cache "
        "WHERE scope_key = %s AND question_hash = %s AND expires_at > now()",
        (scope_key, hashlib.sha256(q.encode()).hexdigest()),
    )
    if exact:
        return "hit", Candidate(exact["id"], exact["question"], exact["answer"], 1.0)

    vec = embed(q)   # normalized vector; pgvector's psycopg adapter registered
    row = db.fetchone(
        """
        SELECT id, question, question_sig, answer,
               1 - (embedding <=> %s) AS similarity
          FROM answer_cache
         WHERE scope_key = %s AND expires_at > now()
         ORDER BY embedding <=> %s
         LIMIT 1
        """,
        (vec, scope_key, vec),
    )
    if row is None or row["similarity"] < LOW:
        return "miss", None
    if row["question_sig"] != critical_signature(q):
        return "miss", None          # same topic, different negation, number or condition
    cand = Candidate(row["id"], row["question"], row["answer"], row["similarity"])
    return ("hit" if cand.similarity >= HIGH else "verify"), cand

<=> is pgvector's cosine distance, so 1 - distance is the similarity you compare against thresholds. The function returns one of three states instead of a boolean, and that third state is the point of the design.

Guard the words embeddings miss

Embeddings are weakest exactly where support answers are most sensitive: negation, numbers and conditions. Within 14 days versus after 14 days, with versus without, EU versus US. A cheap, deterministic check runs before any model is involved: pull out the tokens that flip meaning, and require them to match.

import re

NEGATIONS = re.compile(r"\b(not|no|never|without|except|unless|cannot|can't|don't|doesn't|isn't|won't)\b")
NUMBERS = re.compile(r"\d+(?:[.,]\d+)?")
CONDITIONS = re.compile(r"\b(before|after|within|over|under|more than|less than|at least|at most)\b")
DOMAIN_TERMS = {"eu", "us", "uk", "basic", "pro", "enterprise", "ios", "android"}  # your vocabulary

def critical_signature(q: str) -> str:
    words = set(re.findall(r"[a-z0-9']+", q))
    parts = [
        ",".join(sorted(set(NEGATIONS.findall(q)))),
        ",".join(sorted(set(NUMBERS.findall(q)))),
        ",".join(sorted(set(CONDITIONS.findall(q)))),
        ",".join(sorted(words & DOMAIN_TERMS)),
    ]
    return "|".join(parts)

The refund example from the opening now fails the guard (within ≠ after) and becomes a miss, without any model deciding anything.

This check is deliberately blunt. It will turn some genuine paraphrases into misses ("I can't log in" and "login is not working" carry different negation tokens), and that is the right trade: a miss costs one generation, a false hit costs a wrong answer. Have whoever owns the product catalog own DOMAIN_TERMS too, so a new plan name doesn't quietly become a blind spot.

The grey zone: verify, don't guess

Between the two thresholds, ask a small, fast model one narrow question: does the stored answer fully answer the new question? Structured outputs keep the reply machine-readable.

import json
import anthropic

client = anthropic.Anthropic()
VERIFIER_MODEL = "claude-haiku-4-5-20251001"   # the smallest current model you trust

VERIFY_SYSTEM = (
    "You decide whether a stored answer can be reused for a new question. "
    "reusable is true only if the stored answer fully and correctly answers the new "
    "question: nothing the new question asks is missing, and no condition in the new "
    "question (dates, amounts, negations, products, regions) changes the answer."
)

REUSABLE = {
    "type": "object",
    "properties": {"reusable": {"type": "boolean"}},
    "required": ["reusable"],
    "additionalProperties": False,
}

def still_answers(new_question: str, cand: Candidate) -> bool:
    try:
        resp = client.messages.create(
            model=VERIFIER_MODEL,
            max_tokens=100,
            system=VERIFY_SYSTEM,
            output_config={"format": {"type": "json_schema", "schema": REUSABLE}},
            messages=[{"role": "user", "content": (
                f"<new_question>{new_question}</new_question>\n"
                f"<stored_question>{cand.question}</stored_question>\n"
                f"<stored_answer>{cand.answer}</stored_answer>"
            )}],
        )
        text = "".join(b.text for b in resp.content if b.type == "text")
        return json.loads(text)["reusable"] is True
    except (anthropic.APIError, json.JSONDecodeError, KeyError):
        return False    # any doubt is a miss, including an outage or a retired model ID

At $1/$5 per million tokens, Haiku 4.5 turns a verification into a short prompt and a one-field answer: a small fraction of a full generation on a frontier model, on the model Anthropic lists as its fastest. A verified hit is still slower than a clean one, since it adds a round trip, so narrow the grey zone once your data shows where it is. Keep the verifier's model ID in one constant, because small models get retired like any other, and notice that every failure path returns a miss. A verifier that fails open turns an outage into a stream of unchecked cache hits.

Put together, the request path is short:

def answer(db, request) -> str:
    scope = scope_key_for(request)
    state, cand = lookup(db, scope, request.question)
    if state == "hit" or (state == "verify" and still_answers(request.question, cand)):
        db.execute("UPDATE answer_cache SET hit_count = hit_count + 1 WHERE id = %s", (cand.id,))
        return cand.answer
    result = generate(request)                 # your normal model call
    if cacheable(result):
        store(db, scope, request.question, result)
    return result.text

def store(db, scope: str, question: str, result, ttl_hours: int = 24) -> None:
    q = normalize(question)
    db.execute(
        """
        INSERT INTO answer_cache (scope_key, question, question_hash, question_sig,
                                  embedding, answer, source_doc_ids, expires_at)
        VALUES (%s, %s, %s, %s, %s, %s, %s, now() + make_interval(hours => %s))
        ON CONFLICT (scope_key, question_hash) DO NOTHING
        """,
        (scope, q, hashlib.sha256(q.encode()).hexdigest(), critical_signature(q),
         embed(q), result.text, result.source_doc_ids, ttl_hours),
    )

source_doc_ids records which documents the answer was built from. It looks like bookkeeping until the day a policy page changes; the invalidation section below is where it pays off.

Calibrate thresholds on your own data

HIGH and LOW are not constants you copy from a blog post, this one included. They depend on your embedding model, your domain and your users. Measure them:

Take a few weeks of logged questions. For each one, find its nearest earlier neighbour in the same scope and record the similarity.
Label a few hundred of those pairs: same answer or different answer. Humans are best; an LLM judge with human spot-checks works if you audit it.
Pick the lowest threshold that meets your error budget, and look at how much traffic it leaves you.

def pick_threshold(
    pairs: list[tuple[float, bool]],    # (similarity, same_answer)
    max_false_hit: float = 0.01,
    min_support: int = 50,
) -> tuple[float, float] | None:
    for t in sorted({s for s, _ in pairs}):
        served = [same for s, same in pairs if s >= t]
        if len(served) < min_support:
            return None                  # too little data above this point to trust
        if served.count(False) / len(served) <= max_false_hit:
            return t, len(served) / len(pairs)   # threshold, share of candidates served
    return None

If it returns None on a few hundred labeled pairs, that is a finding, not a bug: no single threshold keeps your error budget on this data, which is exactly the overlap vCache describes. You then need the verifier for everything above LOW, or per-question thresholds, or no semantic layer at all (exact-match only). Treat this the way you would treat any retrieval quality problem: a labeled dataset and a metric, not intuition about what 0.9 "means".

Scope or leak: the cache is a cross-user channel

A semantic cache that serves user B an answer generated for user A is, by construction, a way to move data between users. If A's answer was built from A's account data, A's private documents or A's conversation, that is a data breach delivered with low latency.

Two rules close it.

The scope key includes everything that can change the answer. Tenant, locale, the user's plan (entitlements change answers), model ID, system prompt version, and a version of the knowledge base the answer was built from:

def scope_key_for(request) -> str:
    parts = [request.tenant_id, request.locale, request.plan,
             MODEL_ID, PROMPT_VERSION, KB_SNAPSHOT_ID]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()

Model and prompt versions in the key mean a deploy that changes either one starts a fresh cache automatically, and the old entries simply expire.

Cacheability is decided at write time, from what the generation actually touched. Before generating, you often can't tell whether a question needs private data; afterwards, you know exactly what the model called and read:

def cacheable(result) -> bool:
    return (
        result.stop_reason == "end_turn"
        and not result.used_account_tools        # looked up this user's orders, invoices…
        and not result.read_private_documents    # retrieval hit anything not visible to the whole scope
        and not result.used_conversation_history
        and not result.flagged_by_guardrails
        and bool(result.text.strip())
    )

The same channel runs the other way, too. If someone can get a harmful answer generated once — a prompt-injected link, an offensive reply — a shared cache will serve it to everyone who asks something similar. Cache only answers that passed your output guardrails; consider promoting an answer to the shared cache only after the same question has come from several distinct users with no negative feedback; and show a "cached answer" marker with a regenerate button, which also gives you a false-hit signal at no extra cost. Poisoning, cross-tenant leakage and injection belong on the same threat model when you're securing an LLM application; the cache is just one more component that stores model output.

Invalidation: tie entries to their sources

A cached answer is correct until the thing it was based on changes. Three mechanisms, from coarse to precise:

TTL by content class. Prices and policies: hours to a day. How-to documentation: days to weeks. Nothing lives forever.
Versions in the scope key. A new model, prompt or knowledge-base snapshot means a new scope; nothing needs deleting.
Purge by source. Store the IDs of the documents each answer was built from, and delete the dependent entries when one changes:

def purge_for_document(db, doc_id: str) -> int:
    return db.execute(
        "DELETE FROM answer_cache WHERE source_doc_ids @> ARRAY[%s]::text[]",
        (doc_id,),
    ).rowcount

Hook it to whatever already re-indexes your RAG corpus: the moment a document is re-embedded, the answers built on its old version go away. Of the three, this is the one teams skip, and it is the one that explains why the bot quoted last quarter's refund policy.

Multi-turn conversations

"And for the Enterprise plan?" means nothing without the previous turn, and embedding it on its own will match whatever else was asked about Enterprise. Two safe options: cache only first-turn questions that stand on their own, or rewrite each follow-up into a standalone question with a small model before lookup. The rewrite costs a call on every multi-turn request, which eats into the savings, so measure whether it pays. Never key the cache on the raw last message of a conversation.

Roll it out in shadow mode

Don't find out your false-hit rate from customers. Run the cache in shadow first: on every request, do the lookup, still generate a fresh answer, serve the fresh one, and log both.

def handle(db, request) -> str:
    scope = scope_key_for(request)
    state, cand = lookup(db, scope, request.question)
    result = generate(request)
    if state != "miss":
        log_shadow(request.question, cand, result.text, state)   # judged offline
    if cacheable(result):
        store(db, scope, request.question, result)
    return result.text

Offline, have a judge (with human spot-checks) decide whether each would-be cached answer was equivalent to the fresh one. After a week or two you have a measured false-hit rate per state, hit and verify, at your current thresholds, and monthly_value() stops being a guess. Switch serving on only when the numbers clear your budget, and keep a kill switch. In production, track four things: hit rate, measured false-hit rate (keep sampling), latency for hits and misses, and negative feedback on cached versus fresh answers.

Cached answers are personal data processing too

Anything you store about users' questions and your answers to them falls under the GDPR if it can relate to a person. Three consequences for the cache. Shared entries must never contain personal data, which the write-time rule above enforces. TTLs are your retention policy, so write them down as one. And if you keep any per-user cache, it belongs in your erasure flow: "delete my data" has to reach the cache, not only the main database. Designing data protection into AI systems from the start is cheap; bolting it on after an audit is not.

FAQ

What similarity threshold should I use for semantic caching?
There is no universal value. Similarities depend on the embedding model and the domain, and correct and incorrect hits overlap. Calibrate on labeled pairs from your own traffic, use two thresholds with a verifier in between, and accept that some workloads have no safe threshold at all.

Is semantic caching the same as prompt caching?
No. Prompt caching lives at the provider, discounts re-reading an identical prompt prefix, and still generates a fresh answer, so it can't return a wrong one. Semantic caching lives in your infrastructure and skips generation entirely. They stack: a semantic miss still benefits from a cached prefix.

Should I use Redis or Postgres for a semantic cache?
Whichever you already operate. Both can store vectors and run approximate nearest-neighbour search. The hard parts are scoping, thresholds and invalidation, and none of those depend on the store.

Can I semantically cache an AI agent's responses?
Not end to end. Agent runs call tools, read live state and cause side effects. Cache below the agent instead: exact-match caching of read-only tool results, with short TTLs, carries far less risk.

How do I know the cache is serving wrong answers?
Shadow mode before launch, then continuous sampling: a judge compares a sample of cached answers against fresh generations every week. Add a visible "cached" marker with a regenerate option, and watch whether users regenerate cached answers more often than fresh ones.

The short version

Embedding similarity tells you two questions are about the same thing, not that they have the same answer.
Do the expected-value check first: the false-hit rate decides whether the cache saves money at all.
Scope every lookup by tenant, plan, locale, model, prompt and knowledge-base version. Decide cacheability at write time from what the generation touched.
Guard negations, numbers and conditions deterministically, and verify the grey zone with a small model.
Calibrate thresholds on labeled pairs from your own traffic, and expect overlap.
Purge answers when their source documents change, roll out in shadow mode, and keep a kill switch.

A semantic cache built this way is less aggressive than the one in most tutorials. It serves fewer hits, and every hit it serves has cleared more than a similarity score.

I build and teach production AI systems at Cursuri-AI.ro, Eastern Europe's AI education platform — hands-on courses on shipping LLM features that stay correct, secure and affordable under real traffic.

The Difference Between a Cache and a Lie

Wed, 19 Aug 2026 15:51:36 +0000

A cache is a promise with an expiration date.

That sounds obvious until a production system starts depending on cached state as if it were truth. Then the cache is no longer an optimization. It is a second source of truth with worse durability, weaker ownership, and fewer people watching it.

That is where cache bugs become painful.

The hard part of caching is not storing a value. The hard part is deciding what the value is allowed to mean.

A Cache Should Have a Narrow Job

The safest caches answer narrow questions:

Have we recently computed this result?
Have we recently seen this object absent?
Have we recently fetched this configuration?
Can we avoid doing the same expensive work again?

The more dangerous caches answer authority-shaped questions:

Does this object exist?
Is this user allowed?
Is this state current?
Has this workflow completed?

Sometimes a cache must sit near one of those questions. That is fine. But the design needs to say exactly how wrong the cache is allowed to be and what happens when it is wrong.

The Three Questions Every Cache Needs

Before adding a cache, I like to ask three questions.

First:

What is the source of truth?

If the team cannot answer this clearly, the cache will eventually become part of the data model by accident.

Second:

What is the failure mode when the cache is unavailable?

Fail-open and fail-closed are both valid choices, but they belong to different classes of systems. A performance cache in front of an idempotent delete path can usually fail open. An authorization decision probably cannot.

Third:

What is the failure mode when the cache is stale?

This is where many designs get uncomfortable. A stale cache entry might mean an extra database read, which is fine. It might mean stale search results for a minute, which may be acceptable. It might mean granting access after permission was revoked, which is not the same kind of risk.

TTL Is Not Invalidation

TTL is a recovery mechanism. It is not a complete correctness story.

A TTL says:

If we miss an invalidation, the system will eventually heal.

It does not say:

The system is correct until the TTL expires.

If stale data can cause a correctness bug, the system still needs an invalidation path, a version check, a source-of-truth read, or a design that makes stale data harmless.

TTL is useful because production systems are imperfect. It should not be the only thing standing between the user and a wrong answer.

Fail-Open vs Fail-Closed

The right behavior depends on what the cache protects.

For a performance cache:

cache hit   -> use cached result
cache miss  -> recompute or read from source
cache error -> bypass cache

This keeps correctness independent of cache availability.

For a safety or authorization cache, bypassing the cache may be wrong. If the cache is the only fast path to know whether an operation is allowed, fail-open can turn an outage into a security issue. In those systems, the right answer may be to fail closed, degrade the feature, or require a source-of-truth check.

The cache design should make that tradeoff explicit.

When a Cache Becomes a Lie

A cache becomes a lie when the rest of the system forgets that it is allowed to be wrong.

Common warning signs:

no documented source of truth
no owner for invalidation
no metric for hit rate, miss rate, or stale reads
business logic that only exists in cache-writing code
cache entries with no version, generation, or TTL
code that treats cache errors as successful reads
tests that mock the cache but never test cache loss

The worst cache bugs are not caused by Redis being down. They are caused by Redis being trusted too much.

The Lesson

A good cache makes a system faster without becoming more important than the data it stores.

It is allowed to be missing. It is allowed to be cold. It is allowed to be flushed. It may be useful when present and boring when absent.

The moment a cache cannot be lost, rebuilt, bypassed, or corrected, it is no longer just a cache.

It is part of the system of record, whether the architecture diagram admits it or not.

Why I Prefer Boring Caches

Tue, 11 Aug 2026 14:30:00 +0000

I like caches that are easy to delete.

That may sound strange, because the whole point of a cache is that it improves something important: latency, cost, load, or availability. But the safest caches are the ones the system can survive without. They make the steady state better without becoming the only thing holding correctness together.

That usually means the cache is boring.

Exact keys. Simple values. Clear TTLs. Obvious invalidation. Metrics. A fallback path. No clever inference that future maintainers have to rediscover during an incident.

What Makes a Cache Boring

A boring cache has a small contract.

Given this exact key, return the recently stored value if it exists.

It does not try to become a database. It does not hide business rules inside serialization code. It does not require five services to update it in exactly the right order. It does not make correctness depend on a best-effort write that nobody can replay.

The boring cache is an optimization. The source of truth remains somewhere else.

Exact Beats Clever

Approximate structures are powerful, but they are easy to misuse.

Bloom filters, Cuckoo filters, sampled caches, and lossy sketches all have places where they shine. They are excellent when the direction of error is safe. They are risky when a false positive or an operationally incomplete view can suppress work that should have happened.

Before using an approximate cache, ask:

What does a false positive do?
What does a false negative do?
Can the structure be rebuilt completely?
What happens if updates are missed during an outage?
Is the approximation mathematical, operational, or both?

The last question matters. A data structure may have no false negatives in theory while the system still has false negatives because the structure did not observe every write.

That is not a data structure problem. It is a system design problem.

TTLs Are a Safety Net

I like TTLs because they give a cache a half-life.

They make stale state temporary. They make missed invalidations survivable. They make emergency flushes less scary because the system was already designed for cache entries to disappear.

But TTLs should be chosen with the error mode in mind.

A five-minute stale product description may be harmless. A five-minute stale permission decision may be unacceptable. A one-hour absence marker may be fine if it only suppresses repeated no-op work and is cleared by the create path.

The TTL is not just a performance setting. It is a correctness boundary.

Observability Is Part of the Design

A cache without metrics is a superstition.

At minimum, I want to know:

hit rate
miss rate
error rate
latency
evictions or expirations
size or key count
fallback rate
invalidation rate

The most useful cache metric is often not hit rate. It is fallback behavior. If the cache fails, does the system keep working? If the hit rate drops, do downstream dependencies absorb the load? If invalidation spikes, is that expected or is a writer misbehaving?

The cache should tell operators when it is helping and when it is merely present.

The Fallback Path Must Stay Healthy

The fallback path is production code.

If the cache normally handles 95 percent of reads, it is tempting to ignore the uncached path. That path may be slower, less tested, or underprovisioned. Then the cache has an incident and the fallback path collapses under traffic it has not seen in months.

A boring cache design keeps the fallback path honest:

load test without the cache
sample bypasses in production when safe
alert on fallback latency
keep source-of-truth reads efficient enough to survive cache loss
document the emergency flush procedure

The cache is only optional if the fallback really works.

The Lesson

Clever caches often move complexity. Boring caches remove it.

They are exact where correctness needs exactness. They are bounded where memory or staleness matters. They are observable enough to prove they help. They degrade in a way the rest of the system understands.

The best cache does not make people ask, "What does this thing know?"

It makes people say, "If it is there, use it. If it is not, carry on."

Five Caching Strategies Every Backend Dev Should Know

Sun, 19 Apr 2026 19:33:53 +0000

Five Caching Strategies Every Backend Dev Should Know

If you build backend systems, you eventually run into caching.

Someone says “just put Redis in front of it” and suddenly your app is faster… until you start seeing stale data, cache misses, and weird bugs when you deploy.

This post explains five core caching strategies in plain language:

Cache-Aside
Read-Through
Write-Through
Write-Back
Write-Around

You’ll see how each one works, what can go wrong, and when to use it.

A cache is just a faster, smaller copy of some data. The hard part is keeping that copy useful: when to fill it, when to read from it, and when to update or skip it.

1. Cache-Aside (Lazy Loading)

Idea: The application talks to the cache and the database. It “checks the cache first, then falls back to the DB” and fills the cache on a miss.

If the key is in the cache (a hit), just return it. Super fast.


If the key is missing, the app queries the database instead.


The app stores the result in the cache and returns it to the user.

Why people love it

Simple and flexible: logic lives in your code; you control what goes in the cache.
Resilient: if the cache dies, the app can still read from the DB (just slower).
Memory-efficient: only data that is actually read gets cached.

What can go wrong

The first request for a key is slow (cold cache).
If the DB is updated outside your code path (e.g. admin tool, another service), the cache can serve stale data until TTL expires or you manually invalidate.

When to use

Great default for read-heavy apps (blogs, product catalogs, many dashboards).
When you’re just adding caching to an existing codebase and want full control.

2. Read-Through

Idea: The application pretends the cache is the main database.

It calls the cache for every read. On a miss, the cache layer itself fetches from the DB, stores the result, and returns it.

With Cache-Aside, your app code calls both cache and DB. With Read-Through, your app only calls the cache; a loader behind the cache talks to the DB.

Pros

Cleaner application code: your code only knows “get from cache”.
Cache libraries or infrastructure can handle loading, retries, and metrics.

Cons

If some writes go directly to the DB (bypassing the cache API), the cache can easily become stale.
You need a cache provider or library that supports read-through loading.

When to use

Larger teams where you want centralized cache logic, not ad-hoc caching scattered everywhere.
Situations where infra/platform team owns the cache layer.

3. Write-Through

So far we talked about reads. Now let’s talk about writes: how we keep cache and DB in sync.

Write-Through means:

On a write, update the cache and the database synchronously. Only when both succeed is the write considered “done”.

Flow

User updates something (e.g. changes quantity in a cart).
App writes the new value into the cache.
App also writes the same value into the database.
Only if both succeed do we return “OK” to the user.

Pros

Strong consistency: cache and DB always match for that key.
Reads from the cache always see the latest value.

Cons

Slower writes: every write has to hit both cache and DB before returning.
If DB is slow, your API write latency is also slow.

When to use

When correctness matters more than raw speed:
- balances in a wallet / bank,
- inventory in an e‑commerce site,
- data where a stale read would be really bad.

4. Write-Back (Write-Behind)

Write-Back trades strict consistency for speed.

On a write, update only the cache and return success quickly. Later, an async process flushes those changes to the database.

Flow

User sends a write.
App writes to cache and returns 200 OK immediately.
A background worker or the cache itself periodically writes batched updates into the DB.

Pros

Very fast writes: app isn’t blocked waiting for DB.
Great for high-throughput workloads:
- counters (views, likes),
- logs / telemetry,
- social feed events.

Cons

Risk of data loss: if the cache crashes before flushing to DB, those changes are gone.
DB is eventually updated, not immediately.

When to use

Write-heavy workloads where losing a tiny fraction of data is acceptable:
- metrics, dashboards, click-streams,
- non-critical logs.

5. Write-Around

Write-Around takes a different approach:

Writes skip the cache and go straight to the DB. The cache is only filled later when data is read.

Flow

A write goes directly into the database.
The cache is not updated.
On the next read, the app (or cache) sees a miss:
- loads from DB,
- populates the cache,
- and returns the value.

Pros

Prevents “polluting” the cache with data that is written but rarely read.
Good when you have big write batches (imports, ETL jobs) that users won’t read immediately.

Cons

The first read after a write is always a cache miss (slower).
More frequent DB reads if many keys are only read once.

When to use

Large imports, archival data, logs that might be read only occasionally.
Systems where memory is tight and you want the cache to focus on truly hot data.

Side-by-side comparison

Here’s a quick comparison of these five strategies using the labels you provided.

How teams typically choose

To connect this with the interactive content you built:

These charts show two things:

Cache-Aside dominates in real systems because it’s simple and resilient.
Write-Back gives the fastest writes but carries the most risk; Write-Through is safest but slower on each write.

Which caching strategy should you pick? (a practical rule-set)

If you only remember one thing from this post, remember this: you pick a caching strategy based on the cost of a stale read and the cost of a slow write.

Here’s a fast, real-world decision guide:

Most read-heavy apps (catalogs, blogs, dashboards):
- Start with Cache-Aside.
- Add TTLs and a plan for invalidation if data changes.
You want a clean abstraction and centralized caching (platform/infra owned):
- Use Read-Through so product code only depends on the cache API.
A stale read would be painful (wallet balances, inventory, “did my payment go through?”):
- Prefer Write-Through (consistency > speed).
Writes are hot and you can tolerate some eventual consistency (counters, analytics, clickstream):
- Consider Write-Back, but only with durability guardrails (below).
You write lots of data that is rarely read (imports, ETL, logs):
- Use Write-Around to avoid polluting the cache.

Two failure modes teams hit in production (and what to do)

Caching rarely fails in obvious ways. It fails with “the DB is melting” or “why is data stale?”. Two patterns show up everywhere.

1) Cache stampede (thundering herd)

Symptom: a hot key expires (or cache restarts), and suddenly thousands of requests miss at once and slam the DB.

Mitigations:

Request coalescing / single-flight: only one request recomputes the value; others wait and reuse it.
Stale-while-revalidate: keep serving a slightly stale value while one background refresh updates it.
Jittered TTLs: add randomization to expiration times so many keys don’t expire at the same second.

2) Hot key / hotspotting

Symptom: one key becomes so popular (home feed, trending item, auth config) that a single cache shard or a single DB row gets hammered.

Mitigations:

Shard the key (e.g. trending:v1:0..N) and merge results.
Use a two-layer cache: per-instance in-memory (tiny TTL) + shared Redis.
Precompute or cache at a higher level (edge/CDN) if the data is public.

What to monitor so caching doesn’t become “magic”

Whatever strategy you choose, watch these:

Cache hit rate (overall and by endpoint)
DB QPS (does caching actually reduce it?)
p95/p99 latency (cache helps only if tail latency improves)
Eviction rate / memory pressure (are you constantly evicting hot data?)
Stale reads / correctness signals (app-specific: inventory mismatches, stale dashboards, etc.)

If you can’t measure these, caching will eventually surprise you.

Putting it all together (for a newbie)

If you’re just starting with caching:

Start with Cache-Aside for reads. It’s simple, safe, and easy to reason about.
Add Read-Through if you want a cleaner abstraction and your cache provider supports it.
For writes:
- Use Write-Through when you absolutely need the cache and DB to agree on every write.
- Use Write-Back when you care a lot about write speed and can tolerate a little risk.
- Use Write-Around when you write lots of data that users won’t read immediately.
Whatever you choose, always:
- Set reasonable TTLs,
- Have a plan for invalidating keys,
- And monitor hit rate and latency so you can adjust over time.

With just these five strategies, you can handle most caching problems you’ll run into as a backend engineer.