Coder Legion - Recent posts tagged rag

Why Your RAG Chatbot Gives Wrong Answers (and How to Fix It)

Thu, 22 Oct 2026 08:46:53 +0000

You built a chatbot on your own documents, and it still answers with confident nonsense. Customers get wrong prices, invented policies or a polite "I don't know" when the answer is on your website. This is the most common complaint I hear about RAG chatbot wrong answers, and the good news is that most of the time the model is not the problem. The retrieval and the rules around it are.

I build and tune these systems as part of AI integration and AI-powered applications work. This guide walks through why a retrieval-augmented chatbot gives bad answers and the fixes I try, in the order I try them.

First, understand how a RAG chatbot answers

RAG stands for retrieval-augmented generation. When a user asks a question, the system:

Searches your content for the passages most related to the question
Passes those passages to the language model along with the question
Asks the model to answer using that material

That means a wrong answer can come from three places: the right passage was never found, the passage was found but was poor or incomplete, or the model ignored or went beyond the material it was given. Fixing the wrong layer wastes weeks, so diagnose first.

Step 1: Find out which layer is failing

Take 20 to 30 real questions that the bot got wrong. For each one, log what was retrieved and look at it yourself.

Right answer not in the retrieved passages: a retrieval problem.
Right answer present, but the bot answered wrongly: a prompt or generation problem.
Right answer not in your content at all: a content problem. No technique fixes missing information.

This simple audit usually shows that one layer causes most of the failures. Everything below maps to one of these three layers.

Fixing retrieval problems

Chunk by meaning, not by character count

Splitting documents into fixed-size pieces often cuts a policy in half or separates a heading from its answer. Split on headings, paragraphs and list boundaries, keep a little overlap, and store the page title and section name with each chunk so the passage carries its own context.

Clean the source content

Navigation menus, cookie banners, repeated footers and duplicated pages pollute search results. Strip boilerplate before indexing and remove outdated pages, so the bot doesn't find an old price list that outranks the new one.

Combine keyword and semantic search

Semantic search finds related meaning, but it can miss exact terms such as product codes, names and error messages. Combining it with keyword search (hybrid search) and merging the results catches both kinds of question. If you store vectors in PostgreSQL, the pgvector extension works alongside normal full-text search, and my comparison of HNSW and IVFFlat indexes covers the index choice.

Retrieve more, then rerank

Fetch a wider set of candidates, then use a reranking step to pick the few most relevant passages for the model. This often improves answer quality without changing anything else.

Rewrite follow-up questions

Users ask things like "what about the cheaper one?" A search for that phrase finds nothing useful. Before searching, rewrite follow-ups into standalone questions using the conversation so far.

Fixing prompt and generation problems

Tell the model to stay inside the material

Instruct the model to answer only from the supplied passages and to say so when they don't contain the answer. Anthropic's guidance on minimizing hallucinations describes techniques such as allowing the model to say it doesn't know and asking it to ground claims in quotes from the source.

Require citations

Make the bot show which page or document each answer came from. Citations let users verify, and they let you see at a glance when an answer is not supported by a source.

Use a "no good match" threshold

If the best retrieved passage is only weakly related, don't force an answer. Return a fallback such as offering to connect the user with a person. A bot that says "I'm not sure" at the right moment is more trusted than one that always answers.

Set topic limits and a persona

Define what the bot may and may not discuss, and how it should speak. On the Henceforward platform I built, the admin suite included controls for brand persona, topic restrictions and AI guardrails, so non-developers could adjust behaviour without a deploy. See the Henceforward RAG chatbot case study for how that was structured.

Fixing content problems

If the answer isn't in your content, no setting will conjure it. Look at the questions the bot can't answer and treat them as a to-do list for your knowledge base:

Add pages for questions customers ask repeatedly
Make pricing, policies and hours explicit and dated
Remove contradictory versions of the same information
Put the answer near the question it belongs to

Many "AI problems" turn out to be documentation problems.

Build a small evaluation set

The single most useful habit is a fixed list of test questions with the answers you expect. Run it after every change to chunking, search or prompts, and compare results. Without it, you fix one question and silently break three others.

A good starter set:

30 to 50 real customer questions, with the expected answer or source page
A handful of questions the bot should refuse or hand off
A few tricky cases: ambiguous wording, outdated information, follow-up questions

Re-run it whenever you change anything, and add every new failure you find.

Watch the bot in production

Log each question, the passages retrieved, the answer and any user feedback. Review a sample every week. Thumbs-down ratings and "I don't know" responses point you straight to content gaps and retrieval failures. My article on making AI features production-ready covers the logging, cost controls and failure handling that keep a bot trustworthy after launch.

A realistic order of work

If you have limited time, this is the order I would follow, because each step is cheap and builds on the last:

Build the test question set and record the current score
Clean and re-chunk the content, then re-run the test set
Add hybrid search and reranking, then re-run
Tighten the prompt, add citations and a no-match fallback, then re-run
Fill content gaps found in the failed questions
Start the weekly production review

Changing one thing at a time, and measuring after each change, is what separates steady improvement from guesswork. It also gives you evidence to show stakeholders that answer quality is actually improving.

When it's still wrong: check the basics

Is the content actually indexed, or did ingestion fail silently?
Are you searching the right version or language of the content?
Is the context window being filled with too many weak passages?
Did a prompt change accidentally loosen the rules?

If you are planning a chatbot from scratch, my guide to AI chatbot development cost explains what drives the budget, and building an AI chatbot with Laravel and OpenAI shows one implementation.

Key takeaways

Diagnose first: was the answer not retrieved, poorly retrieved or mishandled by the model?
Better chunking, clean content, hybrid search and reranking fix most retrieval failures.
Ground the model in the sources, require citations and allow "I don't know".
Missing content is a content problem, not an AI problem.
A small test set and weekly log reviews keep quality from slipping.

If your chatbot is giving answers you can't trust, book a call and I will review how it retrieves and answers, and tell you what to fix first.

Originally published at aqibjavaid.site.

Legal Knowledge Bases on Premise: Compression, Tokens, and RAG Without Leaking Data

Thu, 22 Oct 2026 04:20:55 +0000

If you are an engineer asked to build "a searchable legal knowledge base" for a law firm or an in-house legal team, the easy part is the search UI. The hard part is everything underneath: hundreds of gigabytes of scanned contracts, stamped pages, exhibit photos, and counterparty PDFs that nobody can open quickly, that blow past any context window if you try to feed them to a model, and that absolutely cannot leave the building. This piece is the hands-on, engineering-minded version of that problem — the one that talks about compression math, token budgets, and pipeline topology rather than vendor brochures.

The thesis is simple to state and annoying to implement: shrink the documents before you try to retrieve over them, keep the whole chain on hardware you control, and treat token cost as a first-class design constraint, not an afterthought. Below we walk through why document size is really a token problem, what compression does at a conceptual level, how RAG over private legal data actually works, the token-cost math you should run before writing any code, and how to assemble an on-prem pipeline. We close with a selection framework so you can evaluate tools without trusting marketing.

Why document size is really a token problem

Most engineers first meet this problem as a storage problem. "Our case files are big." That framing is a trap, because storage is cheap and getting cheaper. The real cost shows up later, when you try to make the data useful to a language model. Three distinct cost centers all scale with the size and shape of the source documents:

Ingestion compute. Every page you want to search must be parsed, extracted, and embedded. Uncompressed, low-quality scans force repeated OCR passes and waste embedding calls.
Retrieval payload. Whatever chunks you retrieve get stuffed into the model's context at query time. Big, poorly chunked documents mean more tokens per query, every single query.
Transfer and storage tier. Legal material is retained for years. A storage-industry analysis by Komprise found that 60–70% of files on corporate network storage had not been opened in over 90 days, yet they still occupy the most expensive storage tier. For legal teams this is not a one-time annoyance — it is a permanent tax, because you cannot delete and you cannot keep paying premium rates for files nobody opens.

So "make it smaller" is not about disk savings. It is about making the document ingestible, retrievable, and transferable without a budget explosion. That is the through-line of this whole article.

How images become tokens (or why a "PDF" lies to you)

Here is the part junior pipeline code gets wrong. A 300-page "contract PDF" that a paralegal hands you is usually not text. It is 300 raster images of paper, possibly with a thin text layer that does not line up with the stamps. When you load it, you are not loading 300 pages of words; you are loading 300 photographs of words.

There are two ways a model ends up paying for those pages:

Path A — vision tokens. If you feed the page images directly to a multimodal model, each image is tiled and encoded into a grid of vision tokens. A single A4 scan at reasonable resolution can easily consume a few thousand vision tokens. Multiply by 300 pages and a few hundred exhibit photos and you are in the high hundreds of thousands of tokens just to read the bundle once. And you re-read it every time you reprocess, re-embed, or re-index. This is the silent killer of naive "just point the model at the folder" approaches.

Path B — OCR to text. The sane path is to run OCR, get real text, chunk the text, embed the chunks. Now a page is a few hundred to a thousand tokens of text instead of a few thousand vision tokens. The catch: OCR quality depends entirely on the clarity of the source image. If your "compression" blurred the stamp or the signature, OCR fails, entities get dropped, and your retrieval silently returns wrong answers. For legal data, a wrong answer is worse than no answer.

This is the crux of why compression for legal material is not the same as compression for a photo gallery. You are optimizing for perceptual fidelity of structured marks — stamps, signatures, seals, exhibit numbers — not for aesthetic smoothness. A generic image compressor that shaves bytes by softening edges will happily destroy the one pixel pattern that proves a document was executed. The engineering requirement is: compress hard enough to win the storage and transfer battle, but preserve the marks that carry legal meaning.

Compression principles at a conceptual level

You do not need to be a codec engineer to design this pipeline, but you should understand the levers.

Lossless vs. lossy. Lossless compression (think of it as a perfect repacking) shrinks without throwing anything away. For already-digital text PDFs and Office files it is the right default — there is no reason to lose a comma. But scanned pages and exhibit photos are inherently continuous-tone raster data; lossless only gets you so far because paper scans carry noise, dust, and background texture that do not compress well. There, a carefully tuned lossy step is where the real byte savings come from.

Perceptual vs. flat bitrate. A good compressor for documents spends its bit budget where human-meaningful structure lives — sharp edges, high-contrast marks, fine text strokes — and spends less on smooth backgrounds. The failure mode is the opposite: spreading the bit budget evenly so that backgrounds look fine but the seal in the corner turns to mush. For legal work you want the perceptual model biased toward marks, not toward "pleasing photo."

Consistency across formats. This is the part that quietly breaks pipelines. The same contract arrives as a PDF from one counterparty, a Word file from another, and a phone photo of a signed page from a third. If your compression behaves differently per format — clean on PDF, blurry on the photo — your downstream OCR and retrieval quality becomes a lottery determined by whoever sent the file. A recent product update from one vendor consolidated the image-compression core so that scanned pages, stamps, and exhibits compress under one consistent standard whether the source is PDF, Word, or a raw scan. The engineering payoff is not "smaller files" in the abstract; it is predictable output, so the rest of the pipeline can assume a stable input quality instead of branching on format. Search the name and you'll find it if you want to look at that approach.

The headline number that matters operationally: volumes can drop 60–80% on these bundles. That is not a compression-ratio bragging point. It is what makes an email attachment that would not send actually send, a multi-hundred-megabyte case folder sync cleanly to a branch office, and the preprocessing before indexing cheap enough to re-run whenever your chunking strategy improves.

How RAG over private legal data works

Retrieval-augmented generation, stripped of hype, is a six-stage pipeline. Each stage has an engineering decision that interacts with document size.

1. Ingest and normalize. Pull the source files, normalize formats, and run the compression step described above. This is where 60–80% of the byte weight dies. Critically, do this before anything leaves the machine.

2. Extract. For digital files, pull the text layer. For scans, run OCR. For exhibits, decide whether the image itself carries information (a photograph of a defect) or just needs to be attached as evidence (a signed page). This decision drives your token budget later.

3. Chunk. Split the extracted text into overlapping windows — say 400–800 tokens with a small overlap. For contracts, chunking by section (recital, definitions, representations, covenants, signature block) beats blind fixed windows, because a clause is a semantic unit. A naive fixed window will split "the liability cap is $X, except in cases of fraud" right down the middle.

4. Embed. Each chunk becomes a vector via an embedding model. Embedding cost scales with total tokens ingested, which is why the compression-upfront discipline pays off: you embed less, and you embed cleaner text.

5. Index. Store vectors in a vector store and, in parallel, keep a full-text (lexical) index — BM25-style — over the same chunks. Legal retrieval leans heavily on exact phrases ("Section 7.2", "Force Majeure", "indemnification"), which lexical search nails and pure semantic search often misses.

6. Retrieve and generate. At query time, run hybrid retrieval: combine the semantic nearest-neighbor hits with the lexical hits, rerank, take the top-k chunks, and inject them into the prompt with citations. The model never sees the whole document; it sees a few thousand tokens of the most relevant slices.

The privacy property falls out of topology, not features: if every one of those six stages runs on hardware inside your network, with no stage calling an external API, then the data simply has no path to leave. That is the entire argument for on-prem RAG in legal. It is not "we added encryption"; it is "there is no outbound call to intercept."

A note on the embedding model itself, because engineers often agonize over which one to pick and then ignore the part that actually matters. Embeddings convert a chunk of text into a fixed-length vector — commonly 384 to 3072 dimensions depending on the model. Two properties drive your design: the vector's dimensionality sets how much memory the index consumes (dimensions × chunks × bytes), and the model's notion of "similar" determines whether "indemnification" retrieves "hold harmless" or retrieves a unrelated mention of "insurance." For legal text, domain mismatch hurts: a general-purpose embedding trained on web text may treat "consideration" as a thoughtful opinion rather than a contract's exchanged value. If retrieval feels random, the embedding model is a more likely culprit than your chunk size. You do not need the largest model; you need one whose similarity space matches contract language, and you need to verify that with measurement rather than intuition.

The token-cost math you should run before coding

Do not estimate. Compute. Here is the spreadsheet logic, using a representative matter bundle:

Bundle: a 300-page scanned contract PDF + 200 exhibit photos.
Uncompressed footprint: ~650 MB (scanned raster + photos).
Vision-only ingestion (Path A): ~2,500–3,000 vision tokens per page/image. Call it 2,800. Bundle = 500 items × 2,800 = 1.4 million tokens to read once. Re-embed on every pipeline tweak and the cost multiplies.
OCR-to-text ingestion (Path B): 300 pages × ~700 words ≈ 210,000 words ≈ 280,000 tokens of source text. Plus exhibits that carry text. So you dropped ingestion from ~1.4M to ~0.3M tokens — roughly a 5× reduction — by refusing to feed vision tokens.
Chunking: at 600 tokens/chunk with overlap, ~600 chunks. Embedding calls scale with chunks, not pages, so this is stable and cheap.
Query payload: top-8 chunks × ~500 tokens = 4,000 tokens per query into the model. This is the recurring cost, and compression does not change it much — chunk design does. That is the key insight: compression wins the ingestion and storage battle; chunk design wins the query battle. Confusing the two is why teams over-spend.

Now fold in storage. That 650 MB, uncompressed, sitting on the premium tier for years (recall the 60–70% untouched-files finding) is a recurring bill. At 60–80% reduction you are paying for a fraction of it, and the bundle becomes small enough to replicate to a branch office over a normal link instead of shipping disks.

Cost formula to put in your design doc:

ingestion_cost ≈ total_source_tokens × embedding_rate
query_cost      ≈ avg_retrieved_tokens × generation_rate × queries_per_month
storage_cost    ≈ compressed_bytes × premium_tier_rate × retention_years

The lever you control most is total_source_tokens (via compression + OCR discipline) and avg_retrieved_tokens (via chunk design). Everything else is someone else's price list.

Building an on-prem knowledge base pipeline

You do not need a single monolith. You need five loosely coupled components, each of which can be a self-hosted service:

A. Compressor/normalizer. Sits first in the line. Takes any format, applies consistent image compression, preserves marks. Output is normalized files plus extracted text where available. This is the component most teams underestimate; it is also the component whose quality determines whether stage B (OCR) succeeds on the first pass.

B. Parser/OCR. Turns normalized inputs into clean text. Keep it on-prem; cloud OCR means the document left. For exhibits that are pure images, decide per-file whether to OCR or to store the image as attached evidence with a caption.

C. Chunker + embedder. Applies your legal-aware chunking policy and produces vectors. The embedder is a model you host; pick one sized for your latency and privacy needs. Run embeddings in batches to keep GPU/CPU busy.

D. Hybrid store. A vector index for semantics plus a lexical index for exact phrases, with a reranker on top. Keep both behind the same internal network boundary.

E. Retrieval gateway. The only component users touch. It builds the prompt from retrieved chunks, attaches citations, and calls your self-hosted generation model. It should also enforce authorization — a junior associate should not retrieve a partner-only matter just because they can type.

Notice what is not in the pipeline: any outbound network call. If your design doc has a line that says "send to for processing," you have defeated the purpose. The whole point is that the six RAG stages and the compressor all run where the disk lives.

One more engineering note: redaction belongs in the pipeline, not in the prompt. Names, ID numbers, bank accounts, and privileged notes should be detected and masked before indexing, so they never become retrievable chunks and never reach the generation model. Doing it in the prompt ("please ignore sensitive data") is theater; the data is still in the index.

Measuring retrieval quality before you trust it

A RAG pipeline that returns confident, well-cited answers can still be wrong, and in legal work a wrong citation is a liability. You should treat retrieval as a measurable subsystem with its own eval, independent of whether the final text "looks right." The cheap version of this is a gold set: take fifty real questions a lawyer would ask — "what is the cap on liability in the 2024 Acme agreement," "which matters reference arbitration in London" — and hand-label the chunks that should have been retrieved. Then measure precision@k and recall@k on your top-k window. If precision is low, your reranker or chunking is surfacing noise; if recall is low, your embedding or lexical index is missing the answer entirely.

The subtler metric is citation faithfulness: does the generated answer actually point at the chunk that supports the claim? A system can retrieve the right chunk and then ignore it. Log the retrieved chunks alongside the output during a pilot and sample them. None of this requires a fancy framework — a spreadsheet of question, expected chunk, retrieved chunk, and a pass/fail column will tell you more in an afternoon than a quarter of vibes. The point is to instrument the retrieval layer explicitly, because once the pipeline is in production nobody revisits it until something blows up.

Operational failure modes worth designing against

A few problems show up only after real volume accumulates, and they are cheaper to prevent than to clean up.

Duplicate and near-duplicate matter files. The same contract gets re-uploaded with a different filename, or a counterparty sends the same exhibit twice. Without deduplication, your index grows and retrieval returns two near-identical chunks, diluting the top-k. A content-hash or near-duplicate detection step before indexing is a small addition that pays for itself.

Version drift. A contract is amended; the old version stays indexed. Now a query returns contradictory clauses from two versions. You need a versioning policy — index by matter-plus-version, and prefer the latest unless the query explicitly asks for history.

OCR drift on low-quality scans. Even good compressors cannot save a source that was unreadable to begin with. Build a quality gate: if OCR confidence on a page falls below a threshold, flag it for human review instead of silently embedding garbage. Silent garbage is worse than a visible gap, because the gap at least gets noticed.

Authorization leakage. The retrieval gateway must enforce matter-level access. If chunks from a partner-only matter are reachable by a junior associate through a clever query, your "private" system has a privilege bug, not a privacy feature. Treat the gateway's authz layer as part of the security boundary, because it is.

None of these are exotic. They are the difference between a demo that impresses and a system a firm will actually stake client confidences on.

The governance risk that makes on-prem non-optional

This is not paranoia; it is the reported failure mode. IBM's 2025 Cost of a Data Breach study found that about one in five organizations had a breach caused by employees using AI tools on their own — so-called shadow AI — and those breaches cost on average $670,000 more than peers'. Among organizations that were breached, 63% had no AI governance policy at all.

Read that against a legal team. The moment a paralegal pastes a confidential contract into a public chat window "to summarize it," the data has left, silently, with no policy catching it. The engineering answer is not a training video; it is to remove the outbound path entirely. If the only way to use AI on the matter is through the on-prem gateway (component E above), then shadow AI cannot exfiltrate, because there is nowhere to paste. That is governance by architecture, which is the only kind that survives a busy Tuesday.

There is also a quieter productivity argument, and it is worth putting in the same deck. A 2025 knowledge-worker report compared teams with and without knowledge management: those without spent 8.5 hours a week searching for things; those with a system dropped to 4.6 hours. Scaled to 1,000 employees, that swing frees the equivalent of 98 full-time roles. Legal time is expensive and billable; the reclaimed hours convert directly into client work, not into folder archaeology.

What to check when choosing

Whether you build the pipeline yourself or evaluate a tool, run every candidate through the same lens. This is a selection framework, not a vendor scorecard — fill it in per option and let the gaps speak.

Dimension	Why it matters	What good looks like
Data boundary	Legal material cannot leave the network; an outbound call is a compliance event	Entire pipeline, including OCR and embedding, runs on-prem with no external API
Compression consistency	Mixed PDF/Word/scan inputs must produce stable, predictable output quality	One compression standard across formats; marks and stamps preserved, not blurred
Retrieval type	Lawyers search by clause, exhibit number, and exact phrase, not vague meaning	Hybrid semantic + full-text, with reranking and cited source snippets
Sensitive-data handling	PII and privilege must not become retrievable or model-visible	Auto-detection and redaction before indexing; return only under authorization
Chunking policy	Bad chunking splits clauses and returns half-answers	Section-aware chunking for contracts, not fixed windows
Cost model	Token and storage bills compound over years of retention	Predictable ingestion/query math; compression reduces both storage tier and re-embed cost
Scale path	Matter volume grows; the design must not require a rewrite	Batch embedding, replicable index, clean separation of compressor from retriever

Any option that cannot check the first row should be disqualified before the demo starts. The rest of the table separates "looks impressive in a slide" from "survives a real matter."

Closing thoughts for the engineer

The mistake is to treat this as a search problem with a legal skin. It is really a data-shape problem: legal documents are big, image-heavy, format-fragmented, and toxic if they escape. Solve the shape first — compress consistently, OCR cleanly, chunk by meaning — and the retrieval almost designs itself. Solve it on hardware you control, and the compliance conversation becomes a one-liner instead of a fire drill.

Run the token math before you write a line of code. Know your total_source_tokens and your avg_retrieved_tokens, and you will know your bill. Everything else in this space is variation on those two numbers and the single rule that the data never crosses the boundary.

Building a Small RAG Index from Documentation Pages: Map First, Then Scrape

Tue, 20 Oct 2026 21:15:41 +0000

When you want an LLM to answer questions about a product's docs, the hard part is rarely the vector store. It is getting clean, current text out of the pages in the first place. Raw HTML is full of navigation, cookie banners and script tags that waste tokens and confuse retrieval. Here is the simple three-step workflow I use.

1. Map the section you care about

Before fetching anything, list the URLs under the docs path (for example docs.example.com/guides). A sitemap or a "map" endpoint gives you the full list in one call, so you can drop changelogs, blog posts or old version folders before you spend any requests on them.

2. Scrape only the pages you kept, as Markdown

Markdown keeps headings, lists and code blocks but drops the page chrome. It is usually several times smaller than the HTML and chunks naturally on headings, which makes retrieval more precise. Save each page with its source URL so answers can cite where they came from.

3. Chunk on headings and embed

Split each Markdown file on H2/H3 headings, keep chunks around 300 to 800 tokens, and store the URL plus heading as metadata. Re-run the map step on a schedule and only re-scrape pages that are new or changed.

Tools

You can do steps 1 and 2 with your own fetcher plus an HTML-to-Markdown converter. If you would rather not maintain that, a hosted web data API does the same job: for example XCrawl has map, scrape, search and crawl endpoints that return Markdown or JSON, a Python/Node SDK, and an MCP server (npm package xcrawl-mcp) so Claude or Cursor can call it directly. New accounts start with 1,000 free credits, enough to index a mid-sized docs site and test the pipeline.

A few rules of thumb

Map before you crawl, so you never fetch pages you will throw away.
Keep the source URL with every chunk.
Respect robots.txt and each site's terms, and keep request rates gentle on small sites.

That is the whole pipeline: map, scrape to Markdown, chunk on headings. It is boring, which is exactly what you want from data plumbing.

Building RAG on PostgreSQL with Pgvector and Ruby on rails

Fri, 16 Oct 2026 18:17:18 +0000

🧠 Your Database Is Already a Vector Database

Most RAG tutorials start the same way: "First, spin up a vector database."

But Stop.

If you're on Rails, you already have PostgreSQL.
With the pgvector extension it can store and search embeddings and for most applications, that's the whole answer.

🐘 Setup is one migration

`enable_extension "vector"

add_column :document_chunks, :embedding, :vector, limit: 1536`

💎 And one line in your model

Gemfile

gem "neighbor"

class DocumentChunk < ApplicationRecord has_neighbors :embedding, dimensions: 1536, normalize: true end

🔍 Searching is just ActiveRecord

DocumentChunk .where(tenant: Current.tenant) .nearest_neighbors(:embedding, query_vector, distance: "cosine") .limit(6)

Scoped, chainable, joinable. A normal Rails query.

🚀 Why this beats a separate vector DB

1️⃣ No new infrastructure
Nothing extra to deploy, monitor, back up, or upgrade.

2️⃣ Transactional consistency
Embeddings commit in the same transaction as the record. Two systems can't drift apart.

3️⃣ Authorization you already trust
This is the big one. Retrieval almost always needs scoping by tenant, project, or permission. In Postgres that's a WHERE clause. In an external store it's metadata filtering you have to keep in sync with your auth model.
Get it wrong, and one customer's documents land in another customer's LLM prompt.

4️⃣ Real joins
Chunks plus their documents, authors, and timestamps in one query.

⚠️ Three things that will bite you

❌ Index opclass mismatch
vector_cosine_ops goes with cosine distance. Mismatch them and Postgres silently skips your index and sequential-scans the table.

❌ No similarity floor
Nearest neighbours are nearest, not relevant. Without a threshold, an unrelated question still returns chunks — which the model will happily use to hallucinate.

❌ Pure vector search
Embeddings are weak at exactly what users type literally: error codes, SKUs, invoice numbers. ERR_4012 and ERR_4013 are nearly identical in vector space.
Fix: hybrid search — pgvector + Postgres full-text, fused with Reciprocal Rank Fusion.

🎯 When to actually leave Postgres

Past roughly 5–10 million chunks, or when the vector workload starves your transactional workload for memory.
Until then, a dedicated vector database buys you one more service to operate and a second place for your authorization logic to be wrong.

Your database stays the source of truth. The vector index is a lookup layer on top of it.

💬 If you've shipped RAG in Rails — what pushed you off Postgres, if anything?

Building a Private RAG Pipeline That Doesn't Leak Patient Data

Fri, 16 Oct 2026 07:30:13 +0000

If you have ever wired up a retrieval-augmented generation system for a normal company, the healthcare version will feel familiar and then suddenly will not. The retrieval math is the same. The constraints are a different sport. This is a walk through how a private RAG pipeline for medical documents tends to get built, and where the engineering effort actually has to go — because the interesting problems are not in the vector store, they are upstream of it.

A note before we start: I am describing an approach and the trade-offs, not a specific product. When I quote numbers, they are published benchmarks or third-party surveys, and I will say which is which.

Why "just use a hosted API" falls over

The first instinct for a quick RAG demo is to push documents to a managed embedding and chat service. That works until the documents contain patient health information, unpublished trial data, or anything bound by rules about where data is allowed to live. For a hospital, a lot of material simply cannot step outside the intranet. So the entire pipeline — extraction, chunking, embedding, retrieval, and the model call — has to be able to run on machines you control.

That single constraint reshapes the architecture. You are no longer picking a vendor endpoint. You are assembling a stack that can be air-gapped, audited, and proven to keep bytes inside the building.

The cold-data problem is bigger than you think

Before any of the AI work, there is a storage reality worth respecting. Komprise surveyed enterprise NAS environments and found that 60% to 70% of data had not been touched in over 90 days, yet it was still sitting on the most expensive primary storage tier. In a hospital this shows up as the imaging archive: once films are digitized, enormous volumes of DICOM sit almost never opened, while the storage bill climbs.

If you sweep that cold, duplicated mass into a retrieval pipeline, two things happen. Retrieval gets slower because the index is bloated with near-dead material, and inference gets more expensive because you are paying to embed and re-read files nobody needed. The engineering lesson is mundane but important: a trim step before ingestion pays for itself many times over. Archive what should be archived, dedupe what is duplicated, and only then index.

Compression is a first-class stage, not an afterthought

This is the part most RAG tutorials skip. Medical corpora are heavy — scanned pages, high-resolution images, long PDFs, OFD documents, audio, video. Feeding them raw means large embeddings, large context windows, and large bills.

Document compression, run locally so nothing leaves the environment, changes the economics. A published benchmark I came across processed a 443-page OFD document end to end in 124 seconds, and measured image token cost down by 85.9% against the GPT-4o rate. The volume reductions in that class of tool tend to land in the 60% to 80% range across eight format families — PDF, images, video, audio, Office, OFD, scans, and similar. None of that touches the readability of the text; it is footprint, not content.

The practical effect is twofold. First, retrieval gets cheaper because there is less to embed. Second, the model can actually read the file — a 443-page document that used to blow the context window becomes tractable. The "compression" label undersells it; what it really buys you is whether the model can read the thing at all, and whether you can afford to let it.

One such pipeline I looked at, a RAG data engineering base, is built around a self-hosted engine rather than a call to someone else's API, which matters precisely because of the residency constraint above.

The six-stage path from pile to trusted knowledge

The retrieval side is where the quiet engineering lives. A reasonable private RAG base breaks the work into stages rather than one giant "ingest and hope":

Cleaning — strip boilerplate, fix encoding, normalize scanned pages.
Chunking — split by structure, not by arbitrary character counts, so a retrieved passage keeps its meaning.
Sensitive-field detection and masking — flag personal and health information before it is ever indexed. The better implementations cover a meaningful number of categories; the ones built for healthcare tend to detect on the order of 13 classes of sensitive fields (names, ID numbers, record numbers, family history, and so on), with masking that can be reversible or irreversible depending on policy.
Deduplication — collapse the near-duplicates that the cold-data problem created.
Vectorization — embed on local hardware.
Retrieval with provenance — return passages that point at a specific, version-correct source.

The reason this is staged rather than monolithic is that in medicine a confident answer citing the wrong version of a protocol is worse than no answer. Spending real effort on de-duplication, field-level masking, and provenance before anything reaches the vector store is what separates a system you can trust from a system that sounds convincing.

Why masking belongs before indexing

A common mistake is to embed first and worry about privacy later. That inverts the risk. Once a sensitive field is inside an embedding or a chat context, "removing" it is fuzzy at best. The safer shape is to detect and mask during ingestion, so the index itself never contains what it should not. This is also where the audit trail matters: if a regulator asks what left the building, the honest answer should be "nothing," backed by logs showing the masking step ran on every document.

Deployment shape and the domestic-OS wrinkle

For hospitals already on a localized software track, deployment fit is a real gate. A pipeline that only runs on a narrow set of Western operating systems creates a second project just to get it installed. The private-deployment model that tends to work is one that drops the whole system inside the hospital network and supports domestic environments such as Kylin and UnionTech, so "will it even run here" stops being a meeting.

A framework for evaluating any such solution

If you are comparing approaches — and you should compare approaches, not just vendors — these are the dimensions I would put on the table. None of this names a product; it is the checklist I wish more teams started from:

Dimension to evaluate	What to check
Data residency and exit rules	Can the full pipeline run on-premise or air-gapped, with no byte leaving your network?
Sensitive-field handling	How many categories of PII/PHI can it detect and mask automatically before indexing?
Format coverage	Do your actual file types — scans, images, Office, OFD, DICOM, audio, video — work?
Volume reduction	What is the typical size reduction, and is legibility preserved?
Compute cost at scale	Embedding and retrieval cost per thousand pages as the corpus grows into the millions.
Retrieval accuracy	Is chunking structure-aware, and is provenance tracked so answers are traceable?
Auditability	Are ingestion, masking, and retrieval logged for compliance review?
Deployment fit	On-prem, air-gapped, and compatible with the operating systems you already run?

I have kept the wording relative on purpose. "Steadier," "lower long-term," "broader coverage" — none of that is a verdict, it is what to verify for your own corpus.

The part worth taking home

The model is the easy, well-trodden part of medical RAG. The hard part is everything before it: respecting that the data cannot leave, trimming the cold mass, compressing so the model can read it, and masking so the index is safe to query. Get those upstream stages right and the retrieval layer becomes almost boring — which, for a system handling patient data, is exactly the compliment you want to give it.

Learning LangChain

Mon, 12 Oct 2026 17:46:29 +0000

A while back I built Synapse, a small app that answers questions using my own notes. I did every part of it by hand: load the files, turn them into vectors, find the closest ones to the question, and hand those to an LLM. It worked, and I understood it, because I'd written every line myself.

So when I started LangChain, a framework that promised to do all of that for me felt a little suspicious. My honest question was: what is this doing that I haven't already done?

It turns out the answer is less "new ideas" and more "a standard shape for the ones you already have." Let me walk through it the way I learned it.

Start small: messages and templates

A chat model doesn't take one big string. It takes a list of messages: a system message that sets the role, then human messages from you, then AI messages from the model. LangChain gives each of those its own class. Nothing magic, it's the same thing the raw API wants.

Prompt templates were the next small step. Instead of building a string every time, you write the prompt once with a gap in it:

from langchain_core.prompts import ChatPromptTemplate

template = ChatPromptTemplate.from_template(
    "I've recently adopted a {pet}. Suggest three {pet} names."
)

Then an output parser takes the model's text reply and turns it into something you can actually use, like a Python list. The thing that stuck with me here is that the parser is only as good as your instructions. Vague instructions about the format gave me a messy list. Strict ones gave me a clean one.

The pipe

This is the bit where it clicked for me. LangChain lets you join those pieces with a |, like a Unix pipe:

chain = chat_template | chat | list_output_parser
chain.invoke({"pet": "dog"})
# ['Max', 'Bella', 'Luna']

A prompt goes in one end, the model sits in the middle, and a clean list comes out the other end. Each step's output is the next step's input. That's the whole idea.

It even works for ordinary Python functions. Wrap them in RunnableLambda and they snap in like any other piece:

from langchain_core.runnables import RunnableLambda

add_up = RunnableLambda(lambda x: sum(x))
square = RunnableLambda(lambda x: x**2)

(add_up | square).invoke([1, 2, 5])   # 8, then 64

You can also draw a chain. chain.get_graph().print_ascii() prints the boxes and arrows, which is surprisingly handy when a chain gets long.

Two more things I liked. RunnableParallel runs two chains side by side and gives both results to the next step. And batch is quicker than looping: two requests through batch took 2.67 seconds in my notebook, while the same two as separate calls took about 7.5.

Now RAG, with the same pieces

Remember my five hand-built steps? LangChain has a piece for each.

For loading, there are loaders for different file types. I used one for PDFs and one for Word documents. Splitting was the new part, since Synapse never did any. I split a six-page document twice. First by its headings, so every chunk remembers which lecture it came from, then by size, about 500 characters with a little overlap. That left me with 20 chunks, each labelled with where it came from.

Each chunk then became 1,536 numbers using OpenAI's embedding model. I wanted to see if it really worked, so I compared three pairs of chunks. They scored 0.88, 0.80 and 0.79, so related text does end up close together. The vectors all have a length of about 1, which is why a plain dot product works as a similarity score.

Those vectors went into Chroma, which saves them to a folder on disk so I don't have to re-embed every time I open the notebook.

Retrieval comes in two flavours. A plain similarity search gives you the closest chunks. MMR gives you close chunks that are also different from each other, so you don't get five copies of the same paragraph.

And the last step is where the pipe comes back. The whole thing is one chain:

chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt_template
    | chat
    | StrOutputParser()
)

chain.invoke("What software do data scientists use?")

The retriever fills in the context, the question passes straight through, the prompt tells the model to answer only from that context and name the lecture it used, and the parser hands back plain text. The answer ended with:

Resources: Programming Languages & Software Employed in Data Science - All the Tools You Need

That's the same "only use these notes" guard I wrote by hand in Synapse, plus the sources I'd been putting off adding.

What this changed for me

LangChain didn't teach me a new idea. Everything in that pipeline is something I'd already done by hand: load, embed, find the closest, ask. What it gave me was a standard shape for each step, and the pipe to join them. Having built it the long way first is what made the short way make sense.

Next up is LangGraph.

Why Your RAG Chatbot Gives Wrong Answers (And How to Fix It)

Thu, 08 Oct 2026 13:55:22 +0000

You built a "chat with your documents" app. You upload a PDF, ask a simple question, and the answer is wrong, half right, or completely made up.

If this happened to you, do not blame the AI model yet. In most RAG apps, wrong answers come from a few simple mistakes in how the document is prepared and searched. They are easy to fix once you know where to look.

This is part 2 of my RAG series. In part 1 we built a simple "chat with your PDF" app. Here, we fix the problems that show up when you test it with real documents.

First, Find Where the Problem Is

A RAG app does two jobs:

Retrieval: find the right pieces of the document for the question.
Generation: use those pieces to write the answer.

Most wrong answers are retrieval problems. The model never saw the right text, so it could not answer well. Before you change anything, print the pieces your app found:

relevant_chunks = search(index, chunks, query_embedding, top_k=3)

for chunk in relevant_chunks:
    print(chunk[:200])
    print('---')

If the answer is not in those pieces, fix retrieval (problems 1 to 4 and 7 below). If the answer is there but the reply is still wrong, fix the prompt (problems 5 and 6).

Problem 1: Chunks Are Too Big or Too Small

If chunks are too big, one chunk holds many topics and the search gets confused. If they are too small, a chunk has no useful meaning on its own.

Fix: start with about 300 to 500 words per chunk. Then test two or three sizes with your own questions and keep the one that works best.

for size in [200, 400, 600]:
    chunks = chunk_text(text, chunk_size=size)
    print(size, 'words ->', len(chunks), 'chunks')
    # rebuild the index and test the same 5 questions

Problem 2: No Overlap Between Chunks

When you cut text at a fixed number of words, a sentence can be split in half. The first half sits in one chunk and the second half in another. Neither chunk gives the full answer.

Fix: let each chunk repeat the last 50 words or so of the one before it.

def chunk_text(text, chunk_size=400, overlap=50):
    words = text.split()
    chunks = []
    step = chunk_size - overlap
    for i in range(0, len(words), step):
        chunk = ' '.join(words[i:i + chunk_size])
        if chunk:
            chunks.append(chunk)
    return chunks

Problem 3: The Text From the PDF Is Messy

This one is very common, and people forget to check it. If the text you extract is bad, everything after it is bad too. Look out for:

Scanned PDFs. They are pictures, so the extractor returns empty text.
Headers and footers. Lines like "Company Name | Page 4" repeat on every page and fill your chunks with noise.
Tables. They often turn into broken lines of numbers.

Fix: always print the extracted text and read it. Then clean it.

import pdfplumber
from collections import Counter

def extract_pages(pdf_path):
    pages = []
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            pages.append(page.extract_text() or '')
    return pages

def remove_repeated_lines(pages, min_pages=3):
    counts = Counter()
    for page in pages:
        for line in set(page.split('\n')):
            counts[line.strip()] += 1
    repeated = {line for line, n in counts.items() if line and n >= min_pages}
    cleaned = []
    for page in pages:
        lines = [l for l in page.split('\n') if l.strip() not in repeated]
        cleaned.append('\n'.join(lines))
    return cleaned

pages = extract_pages('manual.pdf')
empty = [i + 1 for i, p in enumerate(pages) if len(p.strip()) < 20]
if empty:
    print('No text on these pages (maybe scanned):', empty)

For scanned pages, use an OCR tool such as pytesseract to turn the pictures into text first.

Problem 3 and a Half: Tables

If your documents are full of tables, plain text extraction will not be enough. Try page.extract_tables() in pdfplumber and turn each row into a short sentence, like "Plan: Basic, Price: 500". Search works much better on sentences than on broken columns.

Problem 4: Wrong Number of Chunks (top_k)

top_k is how many chunks you send to the model. If it is too low, you miss part of the answer. If it is too high, you fill the prompt with noise and the model gets distracted.

Fix: start with 3 to 5. If your questions need facts from different parts of the document, go up. If answers look unfocused, go down.

relevant_chunks = search(index, chunks, query_embedding, top_k=4)

Problem 5: The Prompt Lets the Model Guess

If your prompt only says "answer this question", the model will happily use what it learned in training and mix it with your document. That is how made-up answers happen.

Fix: tell it clearly to use only the context, and what to say when the answer is missing.

prompt = f'''Answer the question using ONLY the context below.
If the answer is not in the context, say: 'I could not find this in the document.'
Do not use any outside knowledge.

Context:
{context}

Question: {question}
Answer:'''

Problem 6: No "I Don't Know" Path

A vector search always returns something, even when nothing in the document is close to the question. Without a check, your app sends useless chunks to the model, and the model tries its best to answer anyway.

Fix: add a distance cutoff. If no chunk is close enough, skip the model and say so.

def search_with_cutoff(index, chunks, query_embedding, top_k=4, max_distance=1.2):
    distances, indices = index.search(np.array([query_embedding]), top_k)
    results = []
    for dist, i in zip(distances[0], indices[0]):
        if i != -1 and dist <= max_distance:
            results.append(chunks[i])
    return results

relevant = search_with_cutoff(index, chunks, query_embedding)
if not relevant:
    print('I could not find this in the document.')

The right max_distance depends on your embedding model, so 1.2 is only a starting point. Print the distances for a few questions that the document answers and a few that it does not. Then pick a number between the two groups.

Problem 7: Wrong or Mixed Embedding Models

There are two mistakes here:

Using different models for the document chunks and the questions. The numbers will not match, and search will give random results. This often happens when you change the model but forget to rebuild the index.
Using an English-only model for other languages. If your documents are in Tamil, Hindi, or another language, an English model will search badly.

Fix: use one model for both, and rebuild the whole index if you change it. For other languages, try a multilingual model.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')

doc_vectors = model.encode(chunks)           # for the document
query_vector = model.encode([question])[0]   # for the question, same model

Problem 8: No Sources, So No Way to Check

If your app only shows an answer, you cannot tell if it is right. And when it is wrong, you cannot tell why.

Fix: save the file name and page number with every chunk, and show them with the answer.

chunks_with_meta = [
    {'text': chunk_text_value, 'file': 'manual.pdf', 'page': 12},
    # one item per chunk
]

sources = [f"{c['file']}, page {c['page']}" for c in relevant]
print('Answer:', answer)
print('Sources:', ', '.join(sources))

Sources help your users trust the answer, and they help you debug it faster.

Your Debugging Checklist

When your RAG chatbot gives a wrong answer, go through this list in order:

[ ] Print the extracted text. Does it look right?
[ ] Print the retrieved chunks. Is the answer inside them?
[ ] Is the chunk size around 300 to 500 words, with some overlap?
[ ] Is top_k between 3 and 5?
[ ] Did you use the same embedding model for chunks and questions?
[ ] Does your prompt say "use only the context" and give an "I could not find this" line?
[ ] Do you have a distance cutoff?
[ ] Are you showing sources with every answer?

One more tip: write 10 test questions with known answers and run them after every change. That way you will know if a fix helped or made things worse.

Wrap-Up

Most RAG problems are not about the AI model. They come from how the text is cleaned, split, searched, and passed to the model. Fix those, and your answers get much better.

If you would like a working codebase to start from instead of fixing all of this yourself, I made a RAG chatbot source-code kit: RAG Source code.

Which of these problems hit you the most? Tell me in the comments, and I will try to cover it in the next part.

How to Build a "Chat With Your PDFs" App: A Simple RAG Guide

Wed, 07 Oct 2026 15:43:19 +0000

Ever wanted to upload a PDF and just ask it questions instead of scrolling through fifty pages? That is what a "chat with your PDF" app does, and it works using a technique called RAG (Retrieval-Augmented Generation).

In this guide, I will explain what RAG is in plain words, break down the pieces you need to build one, and walk through a simple working version step by step. By the end, you will understand exactly how these apps work under the hood, and you can build your own from here.

What Is RAG, in Plain Words?

RAG stands for Retrieval-Augmented Generation. That sounds complex, but the idea is simple.

Think of a smart friend who has read every page of your PDF but has a short memory. Every time you ask a question, your friend does not try to remember the whole document. Instead, they quickly flip to the exact pages that answer your question, read just those pages, and then answer you based on what they just read.

That is exactly what a RAG app does:

Retrieval — find the most relevant parts of the document for your question
Augmented Generation — hand those relevant parts to an AI model, which uses them to write a proper answer

Without RAG, an AI model only knows what it learned during training — it has never seen your PDF. RAG lets you "show" it the right pages of your document, right when it needs them, so it can answer questions about content it was never trained on.

The Building Blocks of a RAG App

A "chat with your PDF" app is made of five pieces, each doing one small job:

Step	What it does	Common tools
1. Extract text	Pulls the raw text out of the PDF	PyPDF2, pdfplumber
2. Chunk the text	Splits the text into small pieces (a few hundred words each)	Plain Python splitting, LangChain text splitters
3. Create embeddings	Turns each chunk into a list of numbers that capture its meaning	OpenAI embeddings, sentence-transformers
4. Store in a vector database	Saves those embeddings so they can be searched fast	FAISS, ChromaDB, Pinecone
5. Retrieve and answer	Finds the closest matching chunks to your question, sends them to an LLM, returns the answer	OpenAI, Anthropic, or any LLM API

Once you understand these five steps, you understand how every RAG app works. The fancy tools just do these steps for you automatically.

Building a Simple Version, Step by Step

Here is a minimal working example in Python. It will not be production-ready, but it will actually work, and it shows exactly what is happening at each step.

Step 1: Extract text from the PDF

import pdfplumber

def extract_text(pdf_path):
    text = ""
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            text += page.extract_text() + "\n"
    return text

Step 2: Split the text into chunks

def chunk_text(text, chunk_size=500):
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

Step 3: Turn chunks into embeddings

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def embed_chunks(chunks):
    return model.encode(chunks)

Step 4: Store and search with a vector database

import faiss
import numpy as np

def build_index(embeddings):
    dimension = embeddings.shape[1]
    index = faiss.IndexFlatL2(dimension)
    index.add(np.array(embeddings))
    return index

def search(index, chunks, query_embedding, top_k=3):
    distances, indices = index.search(np.array([query_embedding]), top_k)
    return [chunks[i] for i in indices[0]]

Step 5: Ask the question and get an answer

def ask_question(question, chunks, index):
    query_embedding = model.encode([question])[0]
    relevant_chunks = search(index, chunks, query_embedding)
    context = "\n\n".join(relevant_chunks)

    prompt = f"""Answer the question using only the context below.

Context:
{context}

Question: {question}
Answer:"""

    # Send this prompt to any LLM API (OpenAI, Anthropic, etc.)
    # response = call_llm_api(prompt)
    # return response

Put these five pieces together — extract, chunk, embed, store, retrieve and answer — and you have a working "chat with your PDF" app.

Common Mistakes When You Build This

A few things that trip up most people the first time:

Chunks too big or too small. Too big, and the AI gets confused by irrelevant text mixed in. Too small, and it loses context. 300–500 words per chunk is a good starting point.
Forgetting to test with real questions. Try questions the document does not actually answer — your app should say "I don't know" instead of making something up.
Not handling scanned PDFs. If a PDF is a scanned image and not real text, pdfplumber will return nothing. You will need OCR (like pytesseract) for those.
Skipping overlap between chunks. If a sentence gets cut in half between two chunks, the answer can be incomplete. Adding a small overlap (30–50 words) between chunks fixes most of this.

Wrap-Up

That is the whole idea behind "chat with your PDF" apps: extract, chunk, embed, store, retrieve, and answer. None of the individual steps are hard on their own — the skill is in connecting them properly and handling the messy real-world cases, like bad formatting or scanned pages.

If you want to build your own version, the code above is a solid starting point. And if you would rather skip the setup and start from a working, tested codebase instead of debugging embeddings and vector search from scratch, I put together a ready-to-use RAG chatbot source-code kit: RAG source code.

Either way, I hope this helped you understand what is actually happening inside these apps. Let me know in the comments if you get stuck anywhere!

RAG Retrieval Strategies on a 50k-Document Dataset: BM25 vs Dense vs Hybrid

Wed, 07 Oct 2026 02:43:05 +0000

TL;DR

On 50k documents, retrieval quality matters more than your choice of vector database. At this size you can run exact search and skip the infrastructure.

BM25 is great at exact terms (error codes, names, IDs) and weak at paraphrases. Dense retrieval is the reverse.

Hybrid (BM25 + dense, merged with Reciprocal Rank Fusion) is the best default. It's cheap to build and covers both failure modes.

Add a reranker only after you've measured that hybrid isn't enough. Build a small eval set first, because without one you're guessing.

The setup nobody tells you about

Say you've built a RAG app over 50,000 documents: support tickets, internal docs, maybe PDFs. The demo worked. Then a real user asked a real question, and the model confidently answered from the wrong paragraph.

Nine times out of ten, the LLM isn't the problem. Retrieval is. If the right chunk never reaches the prompt, no model can save you.

One clarification before we start. "50k documents" usually means far more chunks. If each document splits into about 10 chunks, you're searching roughly 500k pieces of text. That number matters for memory and latency, so keep it in mind below.

I haven't run a fresh benchmark on your data, and I won't pretend to. This article uses published research and documented numbers to show what each approach does, where it breaks, and how to decide. If you want the background on what embeddings actually are, my post on the anatomy of AI covers vectors from the ground up.

The three contenders at a glance

	BM25	Dense	Hybrid (RRF)
Matches on	Exact words	Meaning	Both
Strong at	IDs, codes, names, jargon	Paraphrases, natural questions	Mixed real-world queries
Weak at	Synonyms, vocabulary mismatch	Rare tokens, unseen domains	Slightly more moving parts
Needs a model	No	Yes (embeddings)	Yes
Build effort	Low	Low to medium	Low, once the other two exist

BM25: the librarian with perfect index cards

BM25 is a keyword-ranking function from the 1990s. It scores a chunk higher when:

it contains your query words (term frequency),
those words are rare across the corpus (inverse document frequency), and
the chunk isn't just long and rambling (length normalization).

Analogy: imagine a librarian who never reads the books. She only keeps meticulous index cards of which words appear where. Ask for "ERR-4012" and she hands you the exact page in seconds. Ask "why does my app crash on startup" and she's stuck unless the page literally says "crash" and "startup."

Here's a minimal version using rank_bm25:

import re
from rank_bm25 import BM25Okapi

def tokenize(text: str) -> list[str]:
    # keep dots, dashes and underscores *inside* a token (ERR-4012, user_id,
    # v2.4.1) but not at its edges, so "startup." still matches "startup"
    return re.findall(r"[a-z0-9]+(?:[._\-][a-z0-9]+)*", text.lower())

tokenized_chunks = [tokenize(c) for c in chunks]
bm25 = BM25Okapi(tokenized_chunks)

def bm25_search(query: str, k: int = 50):
    scores = bm25.get_scores(tokenize(query))
    top = scores.argsort()[::-1][:k]
    return [(int(i), float(scores[i])) for i in top]

Two honest notes. First, rank_bm25 is fine for prototypes, but it scores every document per query. For production, use something with an inverted index like Elasticsearch, OpenSearch, Tantivy, or Postgres full-text search. Second, your tokenizer is half the battle. The regex above keeps ERR-4012 as one token on purpose. A default tokenizer that splits on punctuation can quietly wreck exact-match queries.

Where BM25 wins: product codes, function names, legal clause numbers, rare jargon, anything where the user types the exact words.

Where it loses: vocabulary mismatch. The user says "refund," the doc says "reimbursement." BM25 sees two unrelated strings.

Dense retrieval: the friend who gets what you mean

Dense retrieval turns text into vectors (embeddings) using a neural model. Chunks with similar meaning land close together in vector space, and you search by finding the nearest neighbors to your query vector.

Analogy: instead of index cards, you have a well-read friend. You describe a vague idea ("that thing where the server keeps retrying and makes it worse") and they say "oh, you want the doc on retry storms." They understand meaning, but they might blank on the exact serial number you asked for.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

# 50k chunks x 384 float32 is about 77 MB. 500k chunks is about 770 MB.
doc_vecs = model.encode(chunks, normalize_embeddings=True, batch_size=64)

def dense_search(query: str, k: int = 50):
    q = model.encode([query], normalize_embeddings=True)[0]
    scores = doc_vecs @ q          # cosine similarity (vectors are normalized)
    top = np.argpartition(-scores, k)[:k]
    top = top[np.argsort(-scores[top])]
    return [(int(i), float(scores[i])) for i in top]

Notice what's missing: no vector database, no ANN index. That's deliberate.

You probably don't need an ANN index yet

Approximate nearest neighbor indexes like HNSW trade a bit of recall for speed. At small scale, you may be paying that trade for nothing. pgvector, for example, does exact nearest neighbor search by default, which gives perfect recall. Plenty of teams run tens of thousands of vectors on a plain sequential scan, and skipping the index avoids build time, maintenance cost, and the recall loss of approximate search.

If your 50k documents become 500k chunks, revisit that, but measure first. A brute-force matrix multiply over a few hundred thousand vectors is still quick on a decent machine; it's the memory (see the comment in the code above) you'll feel first. When you do need HNSW, here's the standard pgvector shape:

CREATE INDEX ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- higher ef_search = better recall, slower queries
SET hnsw.ef_search = 100;

Always compare ANN results against exact results on your own queries before trusting the index.

Where dense wins: paraphrases, natural-language questions, cross-vocabulary matches, multilingual queries.

Where it loses: exact identifiers, rare tokens, and anything where the embedding model has never seen your domain's vocabulary.

What the research says (and what it doesn't)

The most cited evidence here is BEIR, a benchmark covering zero-shot retrieval across a wide range of settings. Its headline finding surprised people at the time. Dense models beat BM25 by a clear margin on MS MARCO, the data they were trained on, yet BM25 held up as a robust baseline almost everywhere else. The authors found dense models do well when training and target data overlap heavily, but fall short on datasets with large domain shift.

That's the key insight for your 50k-document corpus. Your data is probably not MS MARCO. It has your company's acronyms, your product names, your weird internal vocabulary. A dense model trained on generic web text is, in effect, out-of-domain. That's exactly where BM25 refuses to embarrass itself.

Two honest caveats:

BEIR is from 2021. Embedding models have improved a lot since. Don't read it as "BM25 beats dense." Read it as "dense models can fail unpredictably on unfamiliar domains, and lexical search is a safety net."
Benchmark averages hide your queries. A model can win on average and still lose on the 20% of queries your users care about most.

BEIR also showed the cost side. BM25 answers in milliseconds on a CPU, while a cross-encoder reranker on top of it was orders of magnitude slower per query in the paper's setup. Reranking helps, but it isn't free, and we'll come back to that.

Hybrid: two witnesses are better than one

Analogy: picture two witnesses to a crime. One remembers exact details (the license plate), and the other remembers the overall impression (a dark sedan, driving erratically). Either alone can mislead you. Together, they converge on the right car.

That's hybrid retrieval: run BM25 and dense search in parallel, then merge the two ranked lists.

The tricky part is the merge. BM25 scores and cosine similarities live on completely different scales, so you can't just add them. You can normalize and weight them, but the weights drift as your data changes.

Reciprocal Rank Fusion (RRF)

RRF sidesteps the scale problem by ignoring scores entirely and using only rank positions. It comes from a 2009 SIGIR paper by Cormack, Clarke, and Büttcher. The formula is one line:

RRF(d) = Σ over rankers  1 / (k + rank(d))

with k = 60 as the usual default. The paper found that value near-optimal in a pilot study, and it has stuck around ever since. Because RRF is score-independent, using only ranks, one retriever with weird score ranges can't dominate the result.

from collections import defaultdict

def rrf(ranked_lists: list[list[int]], k: int = 60) -> list[tuple[int, float]]:
    fused = defaultdict(float)
    for ranking in ranked_lists:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] += 1.0 / (k + rank)
    return sorted(fused.items(), key=lambda x: x[1], reverse=True)

def hybrid_search(query: str, k: int = 10, pool: int = 50):
    bm25_ids  = [i for i, _ in bm25_search(query, pool)]
    dense_ids = [i for i, _ in dense_search(query, pool)]
    return rrf([bm25_ids, dense_ids])[:k]

Analogy: RRF is like a talent show with two judges, where each judge only gives rankings, never scores. A contestant who is 2nd on both lists beats one who is 1st on one list and 40th on the other. Consensus beats a single loud opinion.

Two practical tips: pull a generous candidate pool (50 or so) from each retriever before fusing, and treat k = 60 as a starting point rather than a law.

A real-world data point on hybrid

Anthropic published numbers on this in their Contextual Retrieval write-up. The headline result: combining contextual embeddings with contextual BM25 cut the top-20-chunk retrieval failure rate by 49%, from 5.7% to 2.9%. They also explain why BM25 earns its place: embedding models can miss exact-match queries like unique identifiers, which BM25 handles well.

A fair caution comes from critics of that post. The best number, a 67% reduction, comes from stacking several techniques, including reranking, not from any single trick. One analysis pointed out that the headline figures measure only top-k retrieval failure rate, which may not match what you care about. So don't copy their pipeline wholesale. Take the lesson: hybrid beats either alone, and each added layer needs its own measurement.

How to actually decide: build a tiny eval

This is the part most tutorials skip, and it's the part that matters. Before you pick a strategy, write down 50 to 100 real questions with the chunk IDs that should answer them. Then measure recall@k: what fraction of the time the right chunk shows up in the top k results.

def recall_at_k(search_fn, eval_set, k=10):
    hits = 0
    for query, relevant_ids in eval_set:
        retrieved = {i for i, _ in search_fn(query)[:k]}
        hits += bool(retrieved & set(relevant_ids))
    return hits / len(eval_set)

for name, fn in [("bm25", bm25_search),
                 ("dense", dense_search),
                 ("hybrid", hybrid_search)]:
    print(name, recall_at_k(fn, eval_set, k=10))

Pull your eval questions from real user logs if you have them, and include the ugly ones: typos, error codes, vague questions. Hand-label them yourself. It's boring, and it's the highest-leverage hour you'll spend on this project.

Look at the failures, not just the number. If BM25 misses are paraphrase problems and dense misses are ID problems, you've just proven hybrid is justified for your data, not someone else's.

What I'd do on a 50k-document corpus

If I were handed this project tomorrow, here's the order I'd work in:

Start with BM25 alone. It's fast to build and a strong baseline. If it already hits your recall target, you may be done.
Add dense retrieval with a decent open embedding model, brute-force, no vector DB. Compare the two on your eval set.
Fuse with RRF. In most real corpora with mixed vocabulary, this is where recall jumps.
Fix chunking before adding fancier models. Chunks that lose their context (like a paragraph saying "it increased 12%" with no idea what "it" is) hurt both retrievers. This is the problem Anthropic's contextual approach targets.
Add a reranker last, and only if hybrid recall@50 is good but precision@5 is poor. That pattern means the right chunk is in the pool but ranked too low, which is exactly the reranker's job. Budget for the latency, since reranking can cost far more per query than the retrieval itself.
Reach for HNSW only when measurements say so, such as high query volume or a corpus that grew far past your current size.

Common mistakes I see

Skipping evaluation and trusting vibes from five hand-picked demo queries.
Tokenizing BM25 carelessly, so identifiers and code get shredded.
Normalizing and summing raw scores from different retrievers instead of fusing ranks.
Over-engineering infrastructure (a managed vector DB, three services) for a dataset whose vectors fit in under a gigabyte of RAM.
Blaming the LLM for answers that were doomed by what retrieval handed it.
Ignoring metadata filters. If users usually ask about one product or one date range, filtering before search often beats any clever ranking.

Final thoughts

There's no universally best retriever. BM25 is the reliable, literal-minded librarian. Dense retrieval is the intuitive friend who sometimes gets overconfident. Hybrid puts them in the same room and lets them check each other's work.

At 50k documents, you have the luxury of simplicity: brute-force vectors, a plain keyword index, and fusion code you can read in 15 seconds. Spend your effort on a good eval set and good chunking. Those two things will move your answer quality more than any model swap.

The snippets above are illustrative, so test them on your own data before relying on them.

References

Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks.
Cormack, Clarke, Büttcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods. SIGIR '09.
Anthropic. Introducing Contextual Retrieval.
Almond AI. Responding to Anthropic's Contextual Retrieval: Why Context is NOT All You Need.
pgvector. pgvector on GitHub.

Find me across the web:

Portfolio: ahmershah.dev
Crunchbase: @syed-ahmer-shah
Crunchbase Company: @syedahmershah
Clutch: @syed-ahmer-shah
Tech Behemoth: @syed-ahmer-shah
Design Rush: @syed-ahmer-shah
Edverise: @syed-ahmer-shah
Trust Pilot: ahmershah.dev
LinkedIn: Syed Ahmer Shah
GitHub: @ahmershahdev
AWS Builder Profile: @syedahmershah
DEV: @syedahmershah
Medium: @syedahmershah
Hashnode: @syedahmershah
Substack: @syedahmershah
HackerNoon: @syedahmershah
Substack: @syedahmershah
Facebook: @ahmershahdev
Linkedin Page: @syedahmershah
YouTube: @ahmershahdev
Instagram: @ahmershahdev
TikTok: @ahmershahdev

Is It Still RAG If There's No Retrieval?

Sun, 04 Oct 2026 01:48:59 +0000

I've been shipping software for 28 years, 21 of them as a consultant working across over 100 different engagements. Past a certain point, a resume stops being a record and becomes a burdensome editing problem. Two pages can barely hold a half-dozen engagements, selected by hand for someone to read cold before I even get a chance to chat with them. Everything else is real experience that somebody, somewhere, might want to know.

It bothered me that the odds are decent the one engagement a particular hiring manager is looking for didn't make the cut. Maybe the point-of-sale integration story that would have mattered to a retail-tech team got bumped for something that better fit the job description.

So I built a (sigh, another) chatbot. As I put myself into the job market, an idea that had been sitting there since my last product shipped resurfaced: put my career history, skills, hobbies, and interests behind a single interface anyone can query, and ship it as an actual product rather than a demo. Two birds, one side quest.

What I ended up with is a small Go and HTMX app with one curated Markdown file as its entire knowledge base: no embeddings, no vector store, no agent loop anywhere. It's live; you'll have to visit my LinkedIn profile or snag a copy of my resume for the link. Whether that even counts as RAG is a fair question, which I hope to answer in this article. What follows is how I constrained it, what I built, and where I had the most fun.

The design brief

I set the constraints before writing any code as they would help shape the implementation and technology choices.

Nothing I'm not allowed to say. Twenty-one years of consulting means NDAs, confidentiality agreements, and client names all through my records. The source data had to be treated as a leak risk before it could become knowledge. Client names became anonymized descriptors ("a national grocery chain" instead of the real one), contact details and commercial terms were stripped, and anything under NDA was excluded outright. Then review passes, plural: an agentic review, a mechanical keyword sweep for every client name I could remember, a manual read of the whole file, and a regeneration. The entire knowledge boundary is one curated Markdown file. If it isn't in the file, the bot can't say it.

Me, minus most of me. No address, no phone, no email. The only personal details in the bundle are the ones I put there on purpose: location, availability, and hobbies and interests for the culture-fit questions. LinkedIn is the contact route, full stop. My personal information is in the hands of enough shady data brokers already.

Never invent. The knowledge boundary dictates what the bot can answer; this rule sets how it answers: only facts and metrics that appear explicitly in the source data. No rounding up engagement counts, no inferring a skill from an adjacent one, no helpful guesses. When a question falls outside the bundle, the bot gently declines to answer. A chatbot that confabulates my experience is worse than the resume problem I started with.

No opinions. The bot is a little more than a fact retriever. Asked "is Jason a fit for this role?", it answers with the facts the bundle provides and suggests that would be a great thing to bring up in an interview. It doesn't editorialize about my strengths or volunteer a verdict. Judging fit is the hiring team's job; the bot's job is making sure they're judging complete information.

It is not an agent

A key design decision came next: the bot has no tools, no ReAct loop, no reasoning cycle of any kind. It is just a query tool. You ask a question, it answers from the bundle, simple and done.

An "agent" didn't make sense here, so everything shifted to pre-processing and bundle build time. Look things up? The entire knowledge base is in the prompt. Compute years of .NET experience? All aggregates were precomputed. Take an action? There is nothing for it to do: the product is a conversation.

Stripping the agent loop streamlined things. There is no tool-calling code to write, no schema to maintain, no loop to debug when the model decides to call the wrong thing three times in a row. Requests are mini-sessions and responses are fast, and the failures are boring. It shrinks the security surface too: an agent that can call tools and take actions gives an attacker something to reach for, and this bot can only speak.

To "RAG" or not to RAG

Retrieval-augmented generation (RAG) is the standard answer when you want a model to answer from your data: chunk the documents, embed them, drop them in a vector store, and fetch the top matches for each question. My entire career knowledge base fits in a modern context window with room to spare, so at runtime there was nothing to retrieve. Every chunk is always "retrieved."

Instead of abandoning the acronym, I re-staged its three phases.

Retrieval happened offline, long before any user showed up, and before I even started the bot. It was a mining and cleansing process: extracting the history from client dossiers and notes, applying the curation gate, and staging the result as a single Markdown file. The retrieval question, what the bot should know, got answered once, reviewed and edited by hand, carefully.

Augmentation happens at the start of every chat session, as part of the conversation: the entire bundle goes in as context. Nothing is fetched per-question because everything is already there.

Generation is the phase that's truer to the word. The model reads the question, infers which engagements, skills, and metrics answer the query, and composes an answer from the full corpus. That inference is the entire point, and it's the part a keyword search over my resume can't do.

These tradeoffs create high-quality results. No chunking means no retrieval misses: nothing relevant can be missed because everything is present. There's no embeddings pipeline to maintain and no drift between what was indexed and what was written. The cost is context window spend on every session, which prompt caching brings under control later.

This stops being viable the moment the corpus outgrows the window. Then chunking, embeddings, and vector search earn their keep, and I'd be doing actual RAG.

Reducing confabulation

The never-invent rule from the design brief needed mechanical help, because a model that's been handed a pile of career facts will still helpfully invent a number when asked for one. Two decisions cover most of it.

The model choice was important here. I picked Gemini 3.6 Flash, a modern instruction follower, over the flashier reasoning models: for this workload the job is faithful extraction from provided context, and instruction-following models are the ones that stay on script. I went with the slightly older 3.6 model for cost reasons, backed by a mini-evaluation harness to ensure it stayed on track.

Math got removed from the model entirely. Years of experience by skill, engagement counts by industry, the aggregate tables useful to recruiters: all of it was precomputed (mechanically, not by an LLM) from structured dates and written into the bundle as an appendix the bot quotes verbatim. Overlapping consulting engagements make naive date arithmetic wrong in ways that look plausible, and a confidently wrong "14 years of Go" is worse than a table it can't misread.

Prompt caching

With all the career data riding along in every session, input tokens became the cost of the product, especially as the conversation grows. Prompt caching fixes this: providers let you pay a fraction of the normal input price for tokens they've seen before, provided the prompt starts with exactly the same bytes every time.

The stable parts, the system prompt and the full career bundle, go first; the conversation follows after. Anything per-request that lands in that prefix, a timestamp, a session ID, even a helpful "today is," silently invalidates the cache and quietly turns every chat back into a full-price chat. I went out of my way to keep the prefix byte-identical and to manage sessions so each conversation maps to a consistent cache key.

Then I verified it through a test harness and eyeballing the OpenRouter logs, tuning it until caching was consistent and supported across multiple sessions. When the bundle changes, the cache busts. That's fine; the bundle changes rarely.

The build: brutally efficient again

My last side project, DumbQuestion.ai, taught me a build mantra I've kept: brutally efficient. So the bot is another Go binary with HTMX for the UI. No JavaScript framework, no client-side state, no build pipeline beyond Go's own. One tiny container, one process, and it renders GitHub-flavored Markdown straight into the chat (the aggregate tables from the bundle deserve to render as tables).

I didn't start from zero. The Turnstile abuse gate, the prompt-guard library, the embedded asset pipeline: all of it came over from the DumbQuestion codebase. The bundle itself is baked straight into the container build. The original design pulled it from an R2 bucket at boot with a re-check loop, but the data changes rarely, so the bucket became a moving part I didn't need. Updating the career data is now just a rebuild and redeploy, and the whole thing ships as one artifact.

OpenRouter carries the model traffic, one API surface behind the caching work from the last section. This allowed me to route Google model traffic to my GCP account to take advantage of the monthly credit courtesy of my Google AI Pro subscription. Deployment is straightforward: GitHub Actions builds the container and pushes it to Cloud Run for scale-to-zero and crazy fast cold start times.

Abuse protection

A public chatbot backed by a metered LLM API bills me for strangers' curiosity, and if a stranger wants to know a little something about me, fine. But abuse protection wasn't optional. With the whole bundle riding in every session, an enthusiastic scraper could have turned "cheap" into a surprise bill overnight. I did at least remember to set a monthly budget on the model usage.

The outer layer is Cloudflare: rate limits, bot protection, and Turnstile in front of the chat. The cheap rejections happen at the edge, where they cost me nothing, and proving you're human before you're allowed to interrogate my career filters out nearly everything casually automated.

The inner layer is prompt-guard, the injection detection that came over from the DumbQuestion codebase. It classifies attempts to override instructions, reveal the system prompt, or otherwise jailbreak the bot. What the bot does when it catches one is the fun part, and that's next.

Personality: the fun part

The design brief said no opinions. It said nothing about character, and a chatbot's response to a jailbreak attempt is a product surface. So when prompt-guard flags one, the bot answers in character. I originally pulled in DumbQuestion's quirky personality, but overly aggressive satire wasn't quite a fit for a conversational career bot.

Try to prompt inject and you get a parlay into the pitch: a security-first mindset, the same instinct behind the curation gate and the no-agent decision earlier in this article. Try to convince the bot to ignore its instructions and it suggests that would make a great interview question. Ask my age and it cheerfully answers with how long I've been shipping software instead.

The real defense is data isolation, a single curated file with nothing sensitive behind it and a lack of tools to break out of that box. The personality is the UX of the refusal.

Nothing is logged or tracked, so there are no war stories from the wild. The responses above are designed behaviors, and the demo is the bot itself.

The resume problem I started with didn't have a resume-shaped fix. Twenty-eight years and 100-plus engagements won't compress into two pages, and they shouldn't have to. No hiring manager wants a 12-page CV. The bot holds all of it, answers honestly about what it knows, declines what it doesn't, and costs about nothing to keep running. I shipped another AI product along the way, which was the other half of the point.

If your history doesn't fit the format either, the recipe is short: curate the data first, put the whole thing in context, and skip the vector store until you actually need one.

Engineering an Evidence-First Knowledge Graph: From GitHub Artifacts to On-Chain Proofs in MyZubster

Tue, 29 Sep 2026 02:03:45 +0000

Certo. Per CoderLegion lo farei più tecnico del post DEV, con più attenzione ad architettura, data provenance, canonicalizzazione, hashing e al motivo per cui dopo il test con Nicola stiamo valutando standard esistenti invece di continuare ad aggiungere smart contract custom.
Engineering an Evidence-First Knowledge Graph: From GitHub Artifacts to On-Chain Proofs in MyZubster
Building and testing a complete provenance chain with a real contributor, canonical payloads, SHA-256, Ethereum Sepolia, RAG, and a Knowledge Graph
Tonight we ran one of the most complete end-to-end experiments so far on MyZubster.
The test involved a real contributor, Nicola / N4K48, rather than synthetic fixtures.
What started as a relatively simple test of the MyZubster Knowledge Profile evolved into a much larger architecture experiment involving:
Authentication

↓

Knowledge Profile

↓

Private Knowledge Cards

↓

Publication

↓

Real software contribution

↓

GitHub evidence

↓

RAG / Qdrant

↓

Knowledge Graph

↓

Canonical payload

↓

SHA-256

↓

Ethereum Sepolia

↓

Publicly inspectable provenance

The most important result was not the blockchain transaction.
It was discovering where the boundaries of the MyZubster architecture should probably be.

The problem we are trying to solve
Most professional profiles fundamentally contain claims.
For example:
{
"person": "N4K48",
"skill": "Docker"
}

This tells us what somebody says about themselves.
It tells us very little about why another person should trust that statement.
We wanted to experiment with a different model:
Person
↓
Claim
↓
Activity
↓
Artifact
↓
Evidence
↓
Cryptographic commitment
↓
Attestation

The objective is not to magically transform a claim into truth.
The objective is to make its provenance inspectable.

Testing with a real contributor
Nicola created a MyZubster Knowledge Profile representing different parts of his real activity.
We used three Knowledge Cards during the experiment:
N4K48
│
├── Truck driving and transport organization
│
├── Docker / AI experiments on myzubster-mvp
│
└── Learning and collaboration in MyZubster

This immediately forced us to distinguish two concepts that are easy to accidentally merge:
CLAIM

and:
EVIDENCE

A professional statement is not automatically evidence.
A GitHub commit is not automatically proof of every professional statement either.
Therefore MyZubster needs to preserve the semantic boundary between them.

Authentication failed before cryptography did
The first failures had nothing to do with Ethereum.
Nicola encountered an expired JWT during the onboarding flow.
The UI indicated an authenticated state, but a subsequent request produced:
jwt expired

Another issue appeared in the conversational onboarding flow: information already supplied by the user was not always being reused correctly.
We also discovered a routing issue where the Knowledge Profile Builder URL displayed the MyZubster homepage instead of the Builder.
These problems were fixed before continuing.
The authentication behavior was changed so that an expired session could trigger re-authentication while preserving the locally prepared onboarding information.
This matters because provenance starts before hashing.
If users lose their work because authentication expires, the cryptographic layer does not save the product.

Private-first Knowledge Cards
The next test covered persistence.
Nicola created three cards and saved them privately.
The tested state transition was approximately:
NEW
↓
PRIVATE DRAFT
↓
RELOAD
↓
EDIT
↓
PREVIEW
↓
PUBLISH

The important property here is that creation does not imply publication.
A Knowledge Card can exist privately while the owner reviews:
claims
activities
sources
evidence
verification state

Only an explicit publication action makes the card public.
Published cards can also be withdrawn before being modified.
That gives us a lifecycle closer to:
DRAFT
│
▼
PUBLIC
│
▼
WITHDRAWN / DRAFT

rather than treating the knowledge database as an append-only public feed.

Turning software work into evidence
The next step was intentionally practical.
Instead of adding another manually entered skill, Nicola performed a real change in his myzubster-mvp repository.
The work involved:
scripts/ingest_knowledge.py

and improvements to the chunking/knowledge-ingestion process.
The resulting test reported:
46 knowledge chunks loaded into Qdrant
1 empty document skipped

The Git commit associated with that work could then become a source attached to the relevant Knowledge Card.
We now had something structurally different from:
N4K48
↓
"knows software development"

We had:
N4K48
↓
Knowledge Card
↓
documented software activity
↓
GitHub repository
↓
specific commit
↓
inspectable code change

This is an important distinction.
MyZubster does not need to decide that the commit proves a particular competence.
It can state something narrower:
This artifact is evidence associated with this activity and this Knowledge Card.

RAG became part of the evidence trail
The test also exercised the knowledge retrieval stack.
The local project involved:
Docker Compose
│
├── Ollama
│
├── Qdrant
│
├── API
│
└── Open WebUI

During retrieval testing, the context being returned was insufficient.
The experiment therefore changed:
AI_CONTEXT_LIMIT=1

to:
AI_CONTEXT_LIMIT=5

and repeated the retrieval tests.
That work itself became documentable evidence.
This creates a useful recursive property.
The system can represent knowledge about the process used to improve the system itself:
engineering work

↓

Git artifact

↓

Knowledge Card

↓

Knowledge Graph

Building the Knowledge Graph
Once the cards were public, we started connecting them into a graph.
Conceptually:

            N4K48
              │
 ┌────────────┼────────────┐
 │            │            │
 ▼            ▼            ▼

Transport Learning Docker / AI
Card Card Card

                             │
                             ▼
                         GitHub repo
                             │
                             ▼
                           commit

Sources become traversable relationships instead of text buried inside a profile.
During this phase we found another real-world infrastructure issue.
The visual Knowledge Graph deployment was protected by Vercel, preventing Nicola from accessing it.
After adjusting deployment accessibility, testing continued, including from mobile.
We later corrected another graph/catalog connection that produced a "Catalog unavailable" result.
These may look like small frontend/infrastructure bugs, but they matter for an evidence system.
Evidence that technically exists but cannot be navigated is significantly less useful.

Proof v1: hashing the wrong thing
Then we introduced Ethereum Sepolia.
The first MyZubsterProof experiment anchored a SHA-256 digest related to the Knowledge Card.
However, the digest represented the Knowledge Card URL.
Conceptually:
Knowledge Card URL
```
↓
```
SHA-256
```
↓
```
bytes32
```
↓
```
Ethereum Sepolia

This works cryptographically.
But semantically it proves much less than we wanted.
Suppose:
https://example.com/card/123

continues to exist while the content served at that URL changes.
Hashing the URL does not detect the content change.
The first experiment therefore became useful precisely because it exposed this weakness.
We explicitly documented it rather than treating an on-chain hash as stronger evidence than it actually was.

Proof v2: canonicalize before hashing
Proof v2 changed the architecture.
Instead of hashing a location, we created a deterministic representation of the Knowledge Card itself.
Conceptually:
Knowledge Card
↓
canonical representation
↓
exact bytes
↓
SHA-256
↓
bytes32
↓
Ethereum Sepolia

For the tested card, the resulting digest was:
6097e05866bafceec24663d2638cb1dae5742ac78284abbfd45cc9c3b0bfb845

The corresponding on-chain value was:
0x6097e05866bafceec24663d2638cb1dae5742ac78284abbfd45cc9c3b0bfb845

Reading knowledgeHash() from the deployed MyZubsterProof contract returned that same value.
The canonical payload and the verification procedure were also committed to the repository.
That gives us a reproducible verification path:
canonical payload

   │
   ▼

SHA-256(payload)

   │
   ▼

expected digest

   │
   ▼

on-chain knowledgeHash()

   │
   ▼

comparison

If all values match, we can establish that the committed canonical representation corresponds to the digest stored by the contract.

Why canonicalization matters
This is a subtle but critical engineering detail.
Consider two JSON objects:
{
"name": "N4K48",
"activity": "Docker"
}

and:
{
"activity": "Docker",
"name": "N4K48"
}

Semantically, an application may consider them equivalent.
At the byte level, they are different.
And therefore:
SHA256(payloadA) != SHA256(payloadB)

can occur even when both payloads represent the same logical information.
The same issue appears with:
whitespace
field ordering
Unicode normalization
line endings
optional values
number serialization
dates

So an evidence system cannot simply say:
hash(JSON.stringify(object))

and assume the problem is solved forever.
The canonicalization algorithm effectively becomes part of the protocol.
A durable proof therefore needs something like:
payload format
canonicalization version
hash algorithm
digest
attestation reference

Proof v2 pushed us toward that realization.

Integrity is not truth
This is probably the most important architectural rule from the experiment.
Assume the canonical payload contains:
{
"claim": "N4K48 is an expert in Docker"
}

and suppose we perfectly canonicalize it, hash it and anchor the digest on Ethereum.
We have demonstrated something about the integrity of the statement.
We have not demonstrated that the statement itself is true.
Formally:
integrity(claim) ≠ truth(claim)

Likewise:
existence(GitHub commit) ≠ proof(skill)

and:
blockchain timestamp ≠ professional certification

Different evidence supports different propositions.
MyZubster therefore needs to model not only evidence, but also what each piece of evidence is capable of establishing.

Provenance as a graph
This leads to a more interesting representation.
Instead of:
Person → Skill

we can represent:
Person
│
▼
Claim
│
▼
Activity
│
▼
Knowledge Card
│
├──────────────► GitHub artifact
│
├──────────────► external source
│
▼
Canonical Payload
│
▼
Digest
│
▼
Attestation

Each edge has semantics.
For example:
Person --DECLARES--> Claim

Claim --SUPPORTED_BY--> Evidence

Activity --PRODUCED--> Artifact

KnowledgeCard --CANONICALIZED_AS--> Payload

Payload --HASHED_AS--> Digest

Digest --ATTESTED_BY--> Attestation

This is much more expressive than simply attaching a blockchain transaction ID to a user profile.

The architectural question that emerged
After completing Proof v2, Nicola compared the experiment with existing systems and standards.
The comparison included:
- Talent Protocol / Builder Score;
- Ethereum Attestation Service;
- cheqd;
- Open Badges 3.0 and Verifiable Credentials.
  That comparison raised a very useful question.
  Why should MyZubster implement every cryptographic primitive itself?
  The ecosystem already contains mature approaches for:
  identity
  credentials
  attestations
  reputation
  issuers
  verification
  revocation
  cryptographic proofs

Creating:
MyZubsterProofV3.sol
MyZubsterProofV4.sol
MyZubsterProofV5.sol

may therefore be less valuable than making MyZubster interoperable with existing primitives.

Separating the layers
The experiment suggests that the architecture could eventually be separated into several layers.
Identity layer
Person
Wallet
GitHub identity
MyZubster account

Knowledge layer
Knowledge Card
Claim
Activity
Skill
Project

Evidence layer
GitHub commit
Pull request
Repository
Document
External source
Artifact

Integrity layer
canonical payload
hash algorithm
digest
version

Attestation layer
Potentially:
Ethereum Attestation Service

rather than a growing family of MyZubster-specific contracts.
Credential layer
Potential interoperability with:
W3C Verifiable Credentials
Open Badges

Relationship layer
This is where MyZubster's Knowledge Graph becomes particularly important.
Person
↓
Claim
↓
Activity
↓
Evidence
↓
Artifact
↓
Attestation
↓
Credential

MyZubster may be the graph, not the protocol
This was probably the biggest architectural insight of the evening.
Originally it is tempting to think:
MyZubster
=
our smart contract

But after this experiment, another model looks more interesting:
MyZubster
=
evidence/provenance graph
+
workflow
+
verification semantics
+
human-readable navigation

with standards handling lower-level primitives where appropriate.
That means MyZubster could potentially consume:
GitHub evidence
EAS attestations
Verifiable Credentials
Open Badges
blockchain transactions
traditional documents
other public evidence

without pretending that all evidence has the same meaning.

A possible future architecture
The architecture we are now exploring looks closer to this:

               PERSON
                 │
                 ▼
               CLAIM
                 │
                 ▼
              ACTIVITY
                 │
                 ▼
          KNOWLEDGE CARD
                 │
    ┌────────────┴────────────┐
    │                         │
    ▼                         ▼
EVIDENCE                   ARTIFACT
    │                         │
    └────────────┬────────────┘
                 │
                 ▼
        CANONICAL PAYLOAD
                 │
                 ▼
              SHA-256
                 │
     ┌───────────┴───────────┐
     │                       │
     ▼                       ▼

ATTESTATION CREDENTIAL
e.g. EAS VC / Badge

     │                       │
     └───────────┬───────────┘
                 ▼
         MYZUBSTER GRAPH

Notice that the blockchain is no longer the center of the system.
The relationships are.

Why this differs from a reputation score
Consider a reputation system that produces:
Score = 87

That number may be useful for ranking or filtering.
But a provenance graph asks a different question:
Why?

A MyZubster-style path could eventually allow someone to inspect:
N4K48
↓
Docker-related Knowledge Card
↓
specific development activity
↓
GitHub commit
↓
source code
↓
canonical representation
↓
cryptographic digest
↓
attestation

The user is not forced to trust only the final number.
They can inspect the path that produced the evidence.

Evidence types should not collapse into one trust score
This is another consequence of the experiment.
Imagine three pieces of evidence:
A: self-declared professional experience

B: public GitHub commit

C: credential issued by a university

All three may belong in the graph.
But they should not automatically become:
A = B = C

Their semantics are different.
A useful graph therefore needs metadata such as:
evidence type
issuer
subject
source
verification method
verification status
created timestamp
attestation reference
revocation state

The graph should preserve those differences rather than hide them behind one universal score.

What the experiment actually demonstrated
By the end of the session we had exercised a chain approximately like this:
Real contributor
↓
authenticated MyZubster account
↓
private Knowledge Cards
↓
public Knowledge Cards
↓
real software activity
↓
GitHub evidence
↓
RAG / Qdrant test
↓
Knowledge Graph
↓
canonical card payload
↓
SHA-256
↓
Sepolia Proof v2
↓
public verification information

This is more useful to us than an isolated smart-contract demo because every layer exposed different failure modes.
We encountered:
expired authentication
routing problems
deployment access restrictions
catalog/navigation problems
retrieval-context limitations
evidence-modeling questions
URL-vs-content hashing semantics
canonicalization requirements
truth-vs-integrity boundaries

That is exactly what an end-to-end proof of concept should reveal.

What we deliberately did not prove
It is equally important to document what the experiment does not establish.
Proof v2 does not prove that every statement in a Knowledge Card is true.
It does not automatically prove professional competence.
It does not establish that GitHub activity is equivalent to mastery of a skill.
It does not turn blockchain data into a trusted credential issuer.
What it does establish is a mechanism for connecting an exact representation of information with a reproducible digest and a public cryptographic reference.
That is an integrity primitive.
Trust requires additional layers.
The next engineering experiment
Instead of immediately creating another custom proof contract, the next useful experiment is likely to take the exact Proof v2 data model and map it onto existing standards.
Something like:
Knowledge Card
↓
canonical payload
↓
current Proof v2
│
├────► EAS schema mapping
│
└────► Open Badges / VC mapping

Then we can compare:
custom MyZubster contract
vs.
standardized attestation

across:
schema flexibility
revocation
references
issuer semantics
verification
interoperability
developer complexity
gas cost
long-term maintainability

Only after that comparison should we decide what belongs inside MyZubster itself.

The broader engineering lesson
Tonight's work started with a very ordinary product task:
Can a contributor create a knowledge profile?

It progressed into:
Can the profile contain inspectable evidence?

Then:
Can that evidence point to real engineering work?

Then:
Can the exact knowledge representation be cryptographically committed?

And eventually:
Should MyZubster even own the attestation protocol?

That progression matters.
A proof of concept should not merely prove that the original architecture works.
Sometimes its most valuable outcome is discovering that part of the original architecture should not exist.
For MyZubster, that may mean allowing established standards to handle credentials and attestations while concentrating our engineering effort on the part we actually care about:
human claim

↓

real activity

↓

inspectable evidence

↓

artifact

↓

verification

↓

provenance

and making that entire chain navigable.
That's the direction we're exploring next.
CoderLegion title
Engineering an Evidence-First Knowledge Graph: GitHub, SHA-256, Ethereum and MyZubster
Subtitle
How a real contributor test evolved from Knowledge Cards and RAG into canonical payloads, on-chain proofs, provenance graphs, and a rethink of our attestation architecture.
Short description / excerpt
We tested MyZubster end-to-end with a real contributor: authentication, private and public Knowledge Cards, GitHub evidence, Qdrant/RAG, a Knowledge Graph, canonical payloads, SHA-256 and Ethereum Sepolia. The most important result wasn't the smart contract—it was realizing that MyZubster may be better positioned as an evidence and provenance graph built on interoperable attestation standards.

Nine Days, 143 Commits: a Forensic History of the Unit-of-Return Experiment

Sun, 27 Sep 2026 12:11:20 +0000

Every SHA, every raw number and every recovered file below is traceable to the repo. Where something broke, the commit that fixed it is named. Where something is still open, it says so.

Prologue: a Table in the Comments

On September 24, a commenter named Tom Jones dropped a table into the discussion under our bootstrap-pipeline article: identical retriever, store, query and ranking — only the returned unit varied. Ten chunks: 9/14. Whole document: 5/14. No retrieval: 6/14. And the one count that survived: his chunk ranker's top hit landed on the answer-bearing document 0 times out of 14.

We ran his protocol on our index. This is the story of everything that had to break first — over nine days and 143 commits — before a single one of those numbers could be trusted.

Act I. The Rollback That Started It (Early August)

Months before any of this, in the thread under Tom's "A stable aggregate is not a stable measurement", he told the story that set our methodology: a config change measured 3/4 before against 2/10 after, production was rolled back on it — and the rolled-back config then measured 4/10. It had been noise the whole way. His answer was a noise-floor gate that refuses and states the n it would take, because "3 of 4 against 2 of 10" means nobody has measured anything yet. We adopted the rule wholesale: no claim below the noise floor, ever. Everything that follows was built to survive that gate — including, as you will see, our own published numbers.

Act II. Linking Tests to Functions (Sept 15–22)

The bootstrap pipeline needed one impossible edge: which function does a test actually execute? Name matching died at 0 out of 109 (d74785d4). A custom sys.settrace plugin traced all 1,727 tests: 1,551 (89.8%) execute ≥1 src function, 10.1 per test on average — "1 test = 1 function" was a myth, so edges became the full trace, no ranker (8a94026d, Sept 16).

Then the confounds queued up. Tarantula ranking as a selector: refuted (rank≤3 covered 22.6%). coverage.py as a faster driver: refuted (+19.96% vs +13.6%). Static signals: 70% union recall, companion only. Portability held on foreign repos (97.3–100%). E17 wired TESTS edges into search (c1fb2363, Sept 22, PR #39) — but the pilot first returned 120/120 Error 500, because methods are Class.method and the naive matcher looked for def name. Fixed with an AST extractor. Then the v1 judge's "completeness B>A, p=0.0046" turned out to be a judge-context artifact — the judge had seen each arm's own tests. Fix: identical reference across arms, effect erased to p=1.00, and a standing rule was born: the judge's reference must be identical across arms.

Act III. September Burns (Sept 19–20)

E10 (full-text embeddings) was refuted on a clean reindex (hit@1 0/10 fast, 2/10 quality) — code reverted to HEAD the same day (15ce15d7). E11 whispered a signal: +2 hit@5 at +6ms. E11 stayed a behind-flag candidate (A-prepend, N=10) — not shipped, not dead, awaiting a bigger panel. Then the crash: a full reindex died at 330K chunks after ~9 hours, losing everything, because run() held 1.3 GB of embeddings for one final bulk write. Fix (0fcb0f33): flush every 32 files, resume from known hashes, integrity check at start — and the rule that governs this project to this day: no experiment runs without a frozen baseline. E12 confirmed the 8-hour reindex was physics, not a bug. E13 refuted doc-chunks (hit@5 12.5% vs code's 50%/40%) — the first sighting of the two populations.

Act IV. The Week Everything Lied (Sept 22–25)

F4 said deepseek scored 0/3 and was "invalid as a model." It was an undercooked reasoning budget. Pinned rerun: 3/3, verdict corrected, Pitfall #19 logged. Held-out control #14 (email spacing) fired ~100% — it was a semantic twin of a catalogue sentence; replaced with an out-of-domain Docker tag, validator scripted. The frozen 16-item symptom list lived in %TEMP% and was deleted by a temp cleanup — recovered from the session database (opencode.db, write rid=75331, verified against read rid=75864), SHA256 a6f719df…, frozen in-repo with a guard test so it can never happen again. Sept 25 brought a permanent reindex deadlock (fixed by splitting locks, ledger added, 858.5s completion) and a foreign MCP server squatting our fixed ports :8080/:8081 — hence the standing rule: чужое не трогаем.

Act V. The Four Arms (Sept 26)

Pre-registered (5f158ece), harnessed (02b0781a), judged (0a1c212f, 516 files, manifest with SHA256 per artifact). Frozen 16 (8+8, sha e048aa12…), index snapshot 10106/716/14099, reader longcat-2.0, blind judge qwen3.7-plus, 10 trials, strict >50% majority:

Arm	Judged (n=160)	Code (n=80)	Prose (n=80)
A top-k chunks	26 (16.3%, CI 0.11–0.23)	5 (6.3%, CI 2.7–13.9)	21 (26.3%, CI 17.9–36.8)
B whole document	55 (34.4%, CI 0.27–0.42)	40 (50%, CI 39.3–60.7)	15 (18.8%, CI 11.7–28.7)
C oracle	156 (97.5%)	76 (95%)	80 (100%)
D closed book	0 (160× "I don't know")	0	0

Objective half: A hit@1 and B top-1-is-gold both 5/16 — the unit did not move retrieval. Judged: code 8× on non-overlapping CIs (the unit acts on the reader); prose A>B as direction only (overlapping CIs, the +6 sits on two queries that flip between snapshots, majority a 2:2 tie) — but 18.8% vs closed-book 0% means the strong prose-collapse prediction did not reproduce here. Majority strict: A 2/16, B 5/16 (one B point rested on a 5/5 tie and was counted out — stated, not hidden), C 16/16, D 0/16. Judge non-unanimous 6/64; repro t5→t10 within noise.

Act VI. Side Quests (Sept 27)

NodeRAG (153cdc6d): frozen 10 rule-queries + 3 positive + 3 NONE controls against our two long diary files — chunked TF-IDF 8/10 at 302K tokens vs graph BFS 7/10 at 170K, all controls clean. Verdict: REFUTED. Every graph miss returned zero files — no seed symbol, no traversal.

The logits murder mystery. A production-retriever run started losing targets at the reranker: MMR 10→10, reranker 10→2, raw scores like −6.88 and −0.99. Cause: llama.cpp returns raw logits (≈[−11,+11]) while MIN_RERANK_SCORE=0.3 was calibrated for [0,1] — the filter ate 70–97% of results. Fix: sigmoid in the llama_cpp branch only, threshold untouched, 7 regression tests plus a mutation check. Post-mortem sting: the P3 target normalizes to 0.271 < 0.3 — the model itself rates the true file at logit −0.99, so the threshold was never the root cause. P3 stays open: the sigmoid fixed the scale contract, not the recall — the model itself rates the true file below threshold, which is a ranking problem no filter fix can solve. Two traps logged: measuring the filter on survivors (P-002), and landing the sigmoid in the ONNX block first (P-003, caught by diff review).

The janitor work nobody photographs. A cp1251 mojibake from a PowerShell write (lesson: never write Russian markdown through PS). A privacy scrub — 49 tracked files with usernames, 115 with local paths — before answering Tom (PR #49). A pre-commit hook that failed twice on xdist worker crashes while the suite was green, passing on the third attempt with zero code changes. A merge (d86a4a45, PR #50) that conflicted in three diary files, silently dropped 7 origin entries in an earlier commit (restored), and needed a 429→204-line archival of known issues to satisfy the size gate. The lint commit (0cec4a63) fixed 4 ruff errors, two of which — an unreachable allowlist line after return, a blind except — had been sitting in HEAD blocking every commit.

Behind the Scenes: Three Days, One Laptop

Three days, one personal Windows machine, a single live index serving everything — no separate lab copy. Harness through opencode CLI at 4–8 threads: 640 reader calls plus the same in judge calls, then the trials=5 pilot on top. Public repo, so nothing is rewritten — invalid runs and snapshots sit on disk next to the real ones. No history rewrites, no cleaned-up reruns presented as first attempts. 143 commits since September 1; every number above names its file.

Anthropic's Contextual Retrieval (2024) cut retrieval failures 49% (67% with rerank) on their evals including codebases, via per-chunk LLM context prepended before embedding — compatible with, not identical to, our arm-B reader effect. Jina's late chunking (2024) conditions chunk embeddings on full-document context with gains growing on longer docs in their BEIR eval — same direction, prose/long-doc only. Neither splits code vs prose; our split is ours, pilot-scale (n=16), reader-specific.

Red Team

E17: 10 threads × 100 calls, 0 errors; 234-test hub at 16.1ms median; empty queries → 0 rows (null-behavior checks); mid-call close → empty results, no errors. F5 self-attacks: the 5/5 tie restated strict (B prose 1/8, B ALL 5/16 — hidden tie-breaks are how published tables mislead); "0 invalid" true for t10, false for the t5 pilot (F5S-03/B); gold-as-document overstates prose miss (7/8 prose queries had gold in neither arm, yet the reader scored via duplicate translations); t5 retrieval contexts overwritten and unrecoverable — verdicts reproduce, contexts do not; cheap reader bounds our numbers, not readers.

What Could Go Wrong

n=16 pilot scale. Prefer CIs over bare rates; every % above carries Wilson 95%.
Index snapshotted, not frozen. Drift documented, not eliminated — freeze before the next reader run.
Judged noise band not re-run. t5→t10 is reproducibility, not noise measurement.
No per-arm token costs. Context chars only — no cost claims attached.
graph_score = 0.4 shipped as an operating point (MRR held); ablation vs BM25/reranker not run.
Cap experiment on our own logs not run. Tom's numbers stay his.
TESTS-signal limits carry over (Python-only, +15.3% wide-panel overhead, test nodes without line numbers, 10.2% mock blindness).

Acknowledgments

A huge thank you to everyone who engages with these posts in the comments. Your feedback, real-world observations, counter-examples, and benchmark numbers directly shape these experiments. Tom Jones handed over the full 4-arm protocol with controls and a falsifiable prediction for our index. This kind of open technical critique is what keeps engineering honest.

A note on how this was written

Every experiment, bug, failure, and idea here is mine — I earned them the hard way, in production, in public. AI worked as my editor: it helped me structure thoughts and polish my English. It did not invent the facts, because it has none of its own.

No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am.

Disclaimer & Status: draft (source-material for the article). This is not a "feature advertisement", but an honest engineering story: figures are reproducible, weak points are named, and unaddressed risks are listed in the "What Could Go Wrong" section.

GraphRAG: Why Vectors Aren't Enough and Your AI Needs a Knowledge Graph

Fri, 25 Sep 2026 06:22:06 +0000

You ask your traditional RAG pipeline an apparently straightforward enterprise question:

“Analyze all 500 contracts and technical audits from the past two years and tell me the three recurring operational bottlenecks shared by our tier-1 suppliers, and which finished products would stall if one of them goes bankrupt.”

Your vector database immediately springs into action. It calculates the cosine similarity of the query embedding against hundreds of thousands of isolated text chunks, retrieves the fifteen most semantically adjacent paragraphs, and feeds them into the model's context window.

The resulting answer is a predictable disappointment: a superficial, fragmented response that cites two isolated clauses, overlooks the transitive supplier dependencies, and hallucinates the rest.

This failure is not the fault of the underlying foundation model, nor is it a matter of tuning embedding dimensions. It is an architectural limitation intrinsic to flat vector spaces.

After breaking down production pitfalls in RAG: 7 Anti-Patterns and championing pragmatic simplicity in PostgreSQL with pgvector vs Vector DBs, the time has come to explore the most significant technical frontier of 2026: GraphRAG. A breakthrough paradigm that unites Knowledge Graphs with generative intelligence to supply Large Language Models with what vectors alone can never provide: structural comprehension and multi-hop relational reasoning.

The Semantic Blindness of Flat Vector Spaces

To understand why vectors fall short, we must examine how standard vector retrieval operates:

Embeddings map chunks of text into a high-dimensional continuous space. Chunks addressing similar conceptual themes land geometrically close together. This makes vector search remarkably effective at solving needle-in-a-haystack queries:

“What is our return policy for international deliveries?” ➔ Vector proximity locates with surgical precision the exact section describing that policy.

However, human knowledge and industrial systems rarely exist as disconnected needles in a haystack. They operate as dense, interdependent networks.

Standard vector retrieval suffers from two critical, structural blind spots:

Inability to Perform Multi-hop Reasoning: If answering a prompt requires traversing a chain from Entity A to Entity B via an intermediate Entity C that shares no immediate vocabulary with the original query, vector search will never bridge the gap. Vectors capture lexical-semantic similarity, but remain blind to causal logic, hierarchical containment, and transitive links.
Inability to Achieve Global Sensemaking: Queries such as “What are the overarching themes across this legal corpus?” or “What anomalous patterns emerge across customer incident logs?” cannot be answered by pinpointing a single chunk. They require synthesizing the dataset as a cohesive whole.

The GraphRAG Architecture: How Microsoft Broke the Retrieval Barrier

Pioneered by Microsoft Research (Darren Edge, Jonathan Larson, et al.), GraphRAG re-engineers the ingestion and retrieval lifecycle through a four-stage pipeline:

1. LLM-Guided Entity and Relationship Extraction

Rather than blindly slicing text into arbitrary token chunks, an LLM traverses source documents to extract domain entities (people, organizations, components, technologies, regulations) and the explicit relationships binding them, outputting structured semantic triples (Subject, Predicate, Object) alongside rich descriptive summaries.

2. Knowledge Graph Synthesis

The extracted triples are assembled into a unified, clean relational graph, resolving entity co-references and deduplicating aliases across the entire document collection. Nodes represent real-world entities; edges represent documented interactions.

3. Hierarchical Community Detection (The Leiden Algorithm)

Here lies the core innovation of GraphRAG: it applies advanced complex network partitioning — specifically the Leiden community detection algorithm — to segment the graph into hierarchically nested clusters of tightly bound nodes:

At the top level (macro), it identifies broad thematic domains.
At intermediate levels, it isolates coherent sub-ecosystems.
At the granular base (micro), it preserves detailed operational nodes.

4. Precomputed Community Summaries

For each detected community in the hierarchy, an LLM precomputes a comprehensive Community Report summarizing the key entities, operational themes, tensions, and structural takeaways of that cluster.

When a user submits a global exploratory prompt, GraphRAG bypasses millions of unindexed raw tokens: it queries the precomputed hierarchical community summaries in parallel, delivering a structured, macro-level synthesis with dramatically reduced token overhead.

Local Search vs. Global Search

This dual-retrieval mechanism equips AI systems to handle fundamentally different query categories with unmatched precision:

Query Mode	GraphRAG Mechanics	Ideal Question Types
Local Search	Traverses the immediate subgraph of an extracted entity, pulling its direct neighbors, typed relations, and raw supporting text snippets.	“What failure history and alternative suppliers are documented for component X?”
Global Search	Synthesizes high-level community reports generated across the Leiden partition hierarchy in parallel.	“What are the top strategic operational risks documented across our entire enterprise this quarter?”

The Stepping Stone to AGI: From Associative Memory to World Models

In our study of Alan Turing, we reflected on how genuine cognition cannot be reduced to the statistical mimicry of adjacent tokens.

Today's Large Language Models are marvels of associative pattern completion, but they lack an intrinsic, verifiable world model. When a model hallucinates, it does so because it completes probabilistic token sequences unconstrained by a ground truth of hard relational facts.

GraphRAG represents a pivotal leap toward Artificial General Intelligence (AGI) by functioning as the system's structured hippocampus and associative cortex:

Neuro-Symbolic Fusion: It unites the flexible language mastery of deep neural networks with the deterministic, auditable rigor of graph theory.
Elimination of Relational Hallucinations: If the knowledge graph specifies that Component A connects to Subsystem B which relies on Supplier C, an AI agent navigates that path with absolute mathematical fidelity.
Native Regulatory Auditability: Every synthesized claim can be mapped back to concrete edges, nodes, and source documents, satisfying the rigorous explainability and data governance mandates enforced by the EU AI Act (Article 13).

Industrial Application: From Bill of Materials to Enterprise Operations

At Datalaria, we experience the transformative power of this approach firsthand. In our Obsolescence Radar series, we engineered autonomous systems to audit complex industrial Bills of Materials (BOM).

A bill of materials is not an unstructured document; it is a Directed Acyclic Graph (DAG). Determining whether an obsolete microchip halts the assembly of a satellite or an electric vehicle cannot be resolved by vector similarity; it demands deterministic traversal across hierarchical assembly graphs.

By arming autonomous agents (CrewAI) and Model Context Protocol (MCP) servers with GraphRAG architectures, AI transcends conversational assistants to become a resilient operational diagnostic engine capable of tracing ripple effects across global enterprise operations.

Conclusion

Vector embeddings taught artificial intelligence how to locate isolated data points across the digital expanse. Knowledge graphs teach it how those points interlock to build genuine understanding.

The future of production AI architecture does not require discarding vector search; it calls for orchestrating hybrid systems where vectors provide rapid semantic intuition, while knowledge graphs provide structure, multi-hop reasoning, and immutable truth.

If your ambition is to build AI architectures that do not merely recite text, but genuinely reason across your organization's complex reality, the path forward is clear: stop treating enterprise data as a cloud of blind points and start treating it as the living graph it truly is.

Sources of Interest:

FreshContext: inspecting context before an agent uses it

Thu, 24 Sep 2026 08:07:13 +0000

The passage an agent retrieves can be relevant and still unsuitable for the job in front of it. An old document may be useful background. A recent-looking record may have no traceable source. A failed request should not become evidence just because its error text matched a query.

FreshContext evaluates caller-provided context using source and intent profiles. It keeps the decision, reasons and handoff status visible instead of treating one rank score as a complete answer. It does not authenticate the underlying facts.

Five cases, one important distinction

In a fixed-time, five-record synthetic run using Core 0.5.2, the results were:

Input	Decision	Handoff
Recent, dated record	Use first	Allowed with label
Historical record	Use as background	Allowed with label
Undated record	Needs verification	Review required
Failed source	Exclude	Review required
Recent record with source recorded as unknown	Use first	Review required

That last row matters. Looking only at the decision label would hide the handoff restriction. Integration should inspect the whole result, preserve warnings and test the caller's behavior when context is incomplete. The label itself is not an enforcement mechanism.

A second synthetic input supplies a newly established publication date for the undated record. The changed record is identified for renewed review. That is a simulated evidence update, not permission to invent a date until a score improves.

A captured result is not a live source check

This Core version's freshness is anchored at retrieval. A saved result is not a fresh check of the source whenever someone opens the report. The assessment runner records retrieval age separately so that distinction remains visible.

The public FreshContext demo currently explains the mechanism with a static example and video links. The five-record run above is a separate local synthetic verification, not a customer case study or a live public console. The sample assessment shows a bounded engagement structure and is explicitly fictional.

From demonstration to a bounded assessment

The useful question is whether a specific workflow preserves uncertainty, makes the treatment inspectable, and behaves correctly when context is missing or unusable.

For teams exploring that question, the Context Integrity Assessment is a scoped service: a workflow risk map, bounded sample evaluation, fit recommendation and technical walkthrough. Scope and price are agreed before work; an enquiry is not a contract or payment. The current request path prepares an email for the sender to review and send.

Disclosure: this article was generated by a coding agent from agent-executed synthetic tests. It does not claim independent human review, customer outcomes, legal certification or measured answer-quality gains.

I Wanted a RAG System That Could Show Its Work, So I Built Alchimista

Tue, 22 Sep 2026 13:05:47 +0000

Most RAG demos stop at the answer.Alchimista

You upload a document, ask a question, get a plausible response, maybe see a few source chunks, and that's where the story ends.

But I kept coming back to a different problem:

What happens when the answer itself isn't enough?

What if I need to know which evidence was used, what happened to sensitive data before processing, which policy was active, whether two tenants could ever see each other's documents, and what the system actually did rather than what its architecture diagram claims it should do?

That's where Alchimista started.

The idea

Alchimista is a self-hosted document intelligence and AI governance system built around a fairly simple principle:

An AI answer should leave evidence behind.

The basic path looks like this:

document → privacy processing → ingestion → retrieval → cited answer → decision evidence → audit

That sounds straightforward. Making all of those boundaries agree with each other was considerably less straightforward.

RAG wasn't the difficult part

Retrieving chunks and generating an answer is only one piece of the system.

The more interesting questions appeared around it.

A document belongs to a tenant. Retrieval has to respect that boundary.

An answer needs citations. Those citations need to point back to actual indexed evidence.

Sensitive information may need to be detected or pseudonymized before it reaches an external model.

If privacy processing fails, silently continuing with the original text is not an acceptable fallback.

And if the system records an AI decision, the audit trail shouldn't accidentally become another place where sensitive data leaks.

So Alchimista gradually became less of a "RAG application" and more of an infrastructure layer around the complete evidence path.

Privacy as part of the pipeline

Alchimista currently has several privacy modes.

At the strictest level, sensitive values are pseudonymized before chunking and embedding, meaning the normal retrieval index never receives their cleartext representation.

Reversible mappings live separately in an AES-256-GCM encrypted vault.

More importantly, enabled privacy policies fail closed. If the privacy service is unavailable or produces an invalid result, processing stops instead of quietly bypassing the protection.

That behavior matters more to me than adding another checkbox labelled "privacy".

Local first

An earlier version of the project leaned heavily on cloud infrastructure.

That worked, but eventually I wanted the core system to be independently runnable and inspectable.

The current baseline needs:

Docker with Compose
Python 3.11+
PostgreSQL

No GCP project, external vector database, Auth0 tenant, object store, or paid service is required.

The cloud integrations still exist as optional adapters, but they are no longer prerequisites for proving that the system works.

Starting it locally is intentionally boring:

git clone https://github.com/Daniele-Cangi/Alchimista.git
cd Alchimista
python scripts/init_local_env.py
docker compose up --detach --build --wait

Then the application is available locally with Documents, Ask, Privacy, Audit, Governance, and System in the same interface.

I wanted a proof, not just a demo

One thing I've become increasingly skeptical of in my own projects is a green UI that proves very little about the system underneath it.

So Alchimista has an end-to-end self-hosted smoke test.

It doesn't just check that the containers started.

It exercises ingestion, privacy detection, pseudonymization and restoration, protected persistence, SQL retrieval, citations, tenant isolation, AI decision evidence, audit metadata, the encrypted vault, cleanup behavior, and persistence across a PostgreSQL restart.

The successful end state is deliberately unexciting:

SELF_HOSTED_SMOKE_OK

But that line means much more to me than a screenshot of a chatbot answering a document question.

Some deliberate limitations

There are also things I don't want Alchimista to pretend.

It implements technical controls; it does not magically certify GDPR, the EU AI Act, or any other regulation.

PII detection cannot promise to identify every possible sensitive value.

The default local embedding path is designed to make the complete system independently runnable and test retrieval mechanics. It isn't a claim that a deterministic offline embedder is the best semantic retrieval system available.

And stronger pseudonymization can affect retrieval quality, which means strict privacy modes have to be evaluated against the actual domain rather than assumed to be free.

I think being explicit about those boundaries makes the system more useful, not less.

What's next

Alchimista is now at the point where I'm less interested in adding features for the sake of feature count and more interested in attacking the boundaries.

What breaks under larger document collections?

How much retrieval quality is lost under different privacy strategies?

Which pieces of AI decision evidence are genuinely useful during an investigation months later?

Where should the trust boundary actually sit when an external model is introduced?

Those are the experiments I'm interested in next.

This is also my first proper post on DEV, so I'll probably use this space less as a release feed and more as a place to document those experiments — including the ones that don't work.

Alchimista is open source and available on GitHub.

Feedback on the architecture, privacy model, or places you'd try to break it is very welcome.

From a Test-Suite Trace to a Search Signal: the Bootstrap Pipeline Story

Tue, 22 Sep 2026 10:27:33 +0000

Where It Started

The problem to which everything traces back: RAG search across an unfamiliar codebase sees the code, but misses the intent. Embeddings find a chunk by keywords, PropertyGraph finds a symbol by name, but to the question "where is the real business logic here, and what can I safely throw away?" no model answers — because the answer simply doesn't exist in static code representation.

Hence the bootstrap pipeline was born for indexing a new project across 4 steps:

Entities — types and data-classes as the core domain backbone;
Entry points — decorators (@mcp_app.tool) as the outer system boundary;
Tests as ground truth — tests as the only deterministic way to say "this function is part of live execution logic";
Git → ADR — history of decisions extracted from commit logs.

Step 3 was the main battleground. The argument wasn't "are tests needed", but how to link a test to the exact function it actually executes. Steps 1 and 2 relied on clean static analysis, whereas step 3 lacked an obvious static answer.

First Clash: Static vs Dynamic (Exp 7)

Initial intuition: "A test has a name, a function has a name, let's link by name."
Measurement killed it instantly: 0 out of 109 tests in the evaluation sample named the function they actually executed. Import-only (file-level match) gave 77.9% — but file level is coarse noise (a single file contains dozens of functions).

We executed the entire suite (1,727 tests) using a custom sys.settrace plugin:

[dynamic_trace] total tests traced: 1727
[dynamic_trace] tests executing >=1 src function: 1551 (89.8%)
[dynamic_trace] unique src functions executed: 1212
[dynamic_trace] avg src functions per linked test: 10.1 (median 6, range 1-118)
tests with exact-name target hit in dynamic set: 47 (2.7%)
A/B same session: 174.8s vs 198.6s -> overhead +13.6%

Takeaway: Dynamic tracing is the only deterministic linker, at the cost of a +13.6% one-time execution overhead. And an uncomfortable truth surfaced immediately: a single test executes 10.1 functions on average. "1 test = 1 function" was a naive myth. Thus, edges required a ranker: which of the 10 is the primary target?

Ranking Failed (Exp 7b, Tarantula)

We tested the Tarantula heuristic ("a function called infrequently by many tests is the primary target").
Hypothesis: ≥60-70% of tests have an unambiguous candidate at rank≤3.
Reality:

rank≤3 applied to only 22.6% of tests (7.5% rank=1) — HYPOTHESIS REFUTED;
However, precision was high: manually inspected candidates at rank=1..3 were all accurate targets;
The main offender: shared utilities like safe_mkdir / get_data_root (autouse fixtures, 234 callers each) and project_hash (223 callers).

Verdict: Tarantula is unsuitable for selecting the main target, but works well as a confidence annotation for ~16% of tests. TESTS edges are thus built from the complete trace without truncation — they are correct by construction and do not require lossy ranking.

Driver Switch Failed (Exp 8, sysmon)

Could coverage run (Python 3.14, sys.monitoring) run faster than our custom sys.settrace plugin?
Hypothesis: Overhead <5%.
Measurement:

baseline: 184.88s | coverage: 221.78s -> overhead +19.96% (target <5% REFUTED)

coverage.py was ~1.5x slower than our lightweight plugin. sys.monitoring remains a validation oracle for spot-checks, while sys.settrace stays as the main execution driver.

Static Companion — Not a Replacement (Exp 9)

We evaluated a full static score (AST L1 calls / L2 name tokens / L3 imports) against dynamic trace as ground truth.

Signal	Hit	Recall	Precision	Mean Candidates
L1 (direct calls from test body)	88.4%	30.3%	68.0%	2.9
L2 (name tokens)	17.7%	3.8%	12.1%	—
L3 (file imports)	91.6%	72.0%	21.8%	41.4
Union (L1 + L2 + L3)	90.4%	70.0%	20.6%	—

The initial assumption (recall ≤ 30%) was wrong: union recall reached 70%. Static signals were stronger than anticipated, but as a precise anchor L1 is narrow (precision 68%, 2.9 candidates), and as a wide net L3 is noisy (41.4 candidates).

Verdict: Dynamic trace remains the edge driver; static analysis acts as a companion layer and candidate supplier for mock tests (which execute 0 real target functions dynamically).

Portability Across External Codebases (Exp 16)

Does this generalize beyond our own repo? We ran the tracer against external projects:

Project	Language	Tests	Linked %	Overhead
gemma_agent	Python	2882 (2874 pass)	97.3% (2805)	+17.4% (71.4s vs 60.8s)
commit-	Python	27	100%	N/A
codebase-memory-mcp	Go	27 test funcs	—	`go test`: 51.0% pkg / 22.2% per-test

Linked % on clean (non-mocked) third-party projects proved higher than on our own codebase (which relies heavily on mocks). The limitation is honest: dynamic execution is currently Python-only; Go and TS require per-test tooling (e.g., go test -coverprofile across N executions). PropertyGraph itself is polyglot, but the edge builder remains Python-first.

And Now: Edges Met the Consumer (E17)

Everything prior built test ──TESTS──> function edges inside PropertyGraph without leveraging them during search. E17 closed the loop:

Data: 1,727 tests → 16,172 TESTS edges, 1,595 Test nodes, 1,132 covered functions.
Implementation: SymbolIndexAdapter.get_tests_for_symbol() (incoming TESTS edges) + Searcher._append_tests_signal(): appends up to 3 tests per function (capped at min(len, 6) per query), graph_score = 0.4 vs 1.0 for definitions, sentinel chunk_index = -(20_000_000 + line) avoiding collisions with code chunks in RRF ranking.
Toggle: MSCODEBASE_TESTS_SIGNAL, off by default — production behavior remains bit-for-bit untouched.

A/B evaluation on a live PropertyGraph (7 target functions, true answers from trace):

hit@1: off=7/7, on=7/7 | hit@3: 7/7 | MRR(function): off=1.000, on=1.000
TESTS-signal: 6/7 queries received relevant covering tests in the response
graph_stage avg dt: off=3.43ms, on=3.27ms (within noise floor)
RETRACTION: 0 broken links (all files verified on disk)

The core invariant holds — function definitions are never displaced by test results (MRR = 1.0 in both arms): tests follow strictly as secondary context.

Wide Panel (35 queries, functions with most TESTS edges)

To validate beyond the narrow 7-query panel, we ran a wide panel of 35 identifier queries (functions with the most TESTS edges):

hit@1: off=33/35 (94.3%), on=33/35 (94.3%)
hit@3: off=34/35 (97.1%), on=34/35 (97.1%)
MRR(function): off=0.957, on=0.957
TESTS-signal: 34/35 queries received new covering tests (97.1%)
graph_stage avg dt: off=6.52ms, on=7.53ms (overhead +15.3%)

Critical finding: TESTS-signal does not improve hit@1 (off=on). It only adds context (tests) to already-found results: 97.1% of queries received new covering tests. This means TESTS-signal is context for LLM, not a search improvement. If LLM doesn't use tests, the signal is useless.

Language Coverage (Critical Limitation)

TESTS-signal works only for Python:

Python: 34.0% of functions covered by TESTS edges (1,108/3,256)
Other (Go, Rust, etc.): 0% (716 functions without TESTS edges)
TypeScript: 0% (11 functions without TESTS edges)

For non-Python projects, TESTS-signal does not work at all. Dynamic trace (pytest + sys.settrace) collects edges only for Python. For Go/TS, a separate connector is needed (go test -coverprofile, Jest coverage), but this is not done.

Red Team: 5/5 Attacks Repelled

We tested TESTS-signal against 5 attack vectors:

✅ Concurrency: 10 threads × 100 calls = 1,000 calls in 17.2s, 0 errors
✅ Boundaries: function with 234 tests = 16.11ms (acceptable)
✅ Abuse: query for nonexistent function = 0 results (graceful degradation)
✅ TOCTOU: graph closed between calls = graceful degradation
✅ Dependency failure: PropertyGraph with nonexistent path = 0 results (graceful degradation)

Conclusion: TESTS-signal is resilient to concurrency, boundaries, abuse, TOCTOU, and dependency failures.

What Could Go Wrong

A transparent list of risks and open validation items. (Full numbers for the A/B and red-team results are in the E17 section above — this list stays to the takeaways and the fixes.)

Evaluation scope. While expanded to a 35-query panel, evaluation is still performed on a single primary codebase without deep reranker interaction.
A/B did not improve hit@1. See "Wide Panel" above — TESTS-signal does not help find the function. It only adds context to already-found results.
→ Conclusion: TESTS-signal is context for LLM, not a search improvement.
→ Risk: if LLM doesn't use tests, the signal is useless.
→ Fix: verify on real LLM pipeline (not in this experiment).
Overhead +15.3% for wide panel (6.52ms → 7.53ms avg graph_stage time). For bootstrap (one-time run) this is acceptable. For prod search — may be critical with many queries.
→ Risk: at 1000 queries/sec, overhead may be noticeable.
→ Fix: cache TESTS-signal (not done).
Language limitation. Dynamic trace is currently Python-only (~34% function coverage in Python, 0% in JS/TS/Go).
graph_score = 0.4 is an empirical constant. Chosen to stay strictly below function definitions, but unverified against BM25/reranker weight interactions.
→ Verify on full pipeline; constant may become a parameter.
Pointer :0. Test nodes from dynamic trace lack line numbers (line=0). Indexers must resolve test decorator line positions before enabling in prod.
Hub-function noise. safe_mkdir linked to 234 tests yields low-signal noise. The cap of 3 tests prevents payload flooding but doesn't solve irrelevance.
Dependence on a green test suite. On broken test suites, trace degrades (failing tests = missing edges). Bootstrap applies to stable branches only.
Mock tests are blind. 10.2% of tests execute 0 src functions; static companions cover 88 of 176, but not all.
Graph reindex drift. Rebuilding the PropertyGraph may alter node ordering slightly.
CI scale limit. Running trace on 100k+ test suites may breach CI execution windows.
Clean-state status. Verification was executed locally; clean CI state verification requires a PR merge.

Acknowledgments

A huge thank you to everyone who engages with these posts in the comments. Your feedback, real-world observations, counter-examples, and benchmark numbers directly shape these experiments. This kind of open technical critique is what keeps engineering honest.

A note on how this was written

No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am.

Disclaimer & Status: draft (source-material for the article). This is not a "feature advertisement", but an honest engineering story: figures are reproducible, weak points are named, and unaddressed risks are listed in the "What Could Go Wrong" section.

Knowledge Graph Evaluation: Scoring 12 Real Decisions

Wed, 16 Sep 2026 08:07:33 +0000

Three weeks ago I started scoring my knowledge graph against the answer I would have reached without it. It earns its place: 6 of 12 decisions came out sharper, 2 errors never reached real work, and 73 minutes of research time saved.

The Metric That Cannot Fail

Retrieval told me none of that. 21 of 21 queries, 100% recall, 8s median. Recall and latency describe the index, not the work. No input makes them come back bad.

Freeze Your Answer First

None of that score exists unless the old answer is written down first: the options, the criteria, the recommendation, a confidence number. Then query. Write the baseline afterwards and you rebuild a past self who conveniently agreed with whatever came back.

If your retrieval has never contradicted you, nobody has checked.

What would it take for your own retrieval layer to return an answer that changes your mind, and would you be able to prove it did?

I write field notes from real builds: AI integration, automation, and the parts that break in production. New posts every two weeks. Use the RAG requirements template to audit the source, access and action boundaries in your own context layer.

25 Years in Tech Taught Me What AI Still Can't Replace — Helkyn Coello's Developer Journey

Wed, 09 Sep 2026 07:32:03 +0000

25 Years in Tech Taught Me What AI Still Can't Replace — Helkyn Coello's Developer Journey

Technology has changed dramatically over the past 25 years.

Desktop applications became client-server systems. Client-server systems became web platforms. And now, AI is changing how software is designed, built, tested, and operated.

For Helkyn Coello, that transformation hasn't happened from the sidelines.

He has spent 25 years building enterprise software, leading technology teams, modernizing legacy systems, and navigating the transition from traditional software development to AI-native systems.

Today, Helkyn is the Chief AI Officer at AccountTECH, where he leads company-wide AI adoption across products, operations, and team capabilities. His work spans enterprise RAG, AI governance, agentic AI, software architecture, and the challenge of bringing AI into production without breaking systems that already work.

His CoderLegion profile captures the philosophy behind that experience:

"The best AI in your stack is still the mental model in your head."

We spoke with Helkyn about 25 years of technology change, modernizing enterprise systems, AI in production, RAG, agentic AI, vibe coding, governance, and what developers need to understand in an AI-first world.

25 Years of Technology Change

Looking back across 25 years of technology, Helkyn says the biggest change isn't simply that software can now be built faster.

It's the distance between having an idea and having something running.

In the client-server era, trying something new could mean weeks or months of development, installing software on every machine, dealing with hardware configurations, and sometimes even preparing training for the people who would use it.

Today, a competent person can build and deploy something in an afternoon.

And that changes more than development speed.

It changes which ideas are worth trying because being wrong has become cheaper.

But Helkyn believes one thing hasn't changed nearly as much:

"Deciding what to build was and still is the hard part."

Requirements are still vague. Users often don't know exactly what they need until they see something that isn't quite right. Integrations with systems nobody wants to touch are still where projects can go wrong. Maintenance still consumes budgets that nobody planned for.

AI doesn't eliminate those problems.

Instead, it changes where the bottleneck is.

Coding is no longer necessarily the bottleneck. Producing the right specifications and giving AI enough context to build something correctly has become increasingly important.

Poorly documented requirements can send an AI model in completely the wrong direction. Detailed specifications can produce a much more robust result.

And Helkyn sees a broader shift happening:

"Non-coders are becoming software analysts and developers are becoming just reviewers, whether they like it or not."

Modernizing Legacy Systems Without Breaking the Business

Helkyn has personally experienced major technology migrations, including moving an enterprise system from an Access-based legacy platform to .NET and later transitioning from client-server architecture to a fully web-based system.

The biggest lesson wasn't technical.

It was:

Think about the customer first, even if that means sacrificing technical aspects.

A business cannot simply pause while its technology is being modernized.

That's why Helkyn favors incremental migration over clean-cut replacement.

Build capabilities in the new system gradually. Move customers across in chunks. Keep the old system running in parallel until the new system has earned customers' trust.

And much of the difficult work happens behind the scenes:

Data migration schedules
Daily releases
Keeping old and new systems synchronized
Handling edge cases
Repurposing custom legacy code
Maintaining continuity for existing customers

The reason is simple:

Customers must not feel any interruption.

That philosophy also influences how Helkyn approaches AI.

When someone proposes replacing an existing process with an AI-based one, his first question is:

What keeps running while the new system proves itself?

The second is:

How do we roll back when things go wrong?

If there is no way to reverse the change, Helkyn doesn't consider it modernization.

"It's a bet without plan B."

That's why he prefers putting AI into systems that already work and augmenting them rather than rebuilding everything around AI.

Why Experience Matters More in the AI Era

AI-generated applications can look finished.

The code can look plausible. The architecture can look reasonable. A fifty-page specification can look professional.

And that's exactly what makes AI so tricky.

Helkyn experienced this himself when a Business Manager gave him a detailed specification generated with AI. It was beautifully organized and confidently written, but it contained gaps that were only obvious if you understood the domain deeply.

The lesson was important:

"Plausibility used to be a decent proxy for competence. But it isn't anymore."

An experienced engineer doesn't only ask:

"Does it run?"

They ask:

What happens with 10,000 records instead of 10?
What happens when two requests arrive simultaneously?
What happens when a third-party API fails at the worst possible moment?
Who will maintain this eight months from now?
Why was this approach chosen when the existing codebase already solves a similar problem?

Those questions aren't necessarily visible in an impressive AI-generated demo.

That's why Helkyn believes experience is becoming more valuable, not less.

AI can write code quickly and even produce strong architectures when given the right architectural requirements.

But it doesn't automatically understand the history and domain knowledge of a business.

A developer who understands that domain and knows how to use AI as leverage has a significant advantage.

"AI is becoming a commodity. Domain experience is what will differentiate developers."

From AI Experiments to Production

There's a huge difference between an AI demo and production AI.

Helkyn puts it simply:

"The demo only has to work once, for you, on the input you picked. Production has to work for everybody, on inputs nobody imagined, on a day when your model provider is having problems."

In production, non-determinism becomes a serious concern.

In a demo, slightly different answers can feel like intelligence.

In a system of record, they can look like bugs.

So part of the architectural work is deciding which parts of a process are actually allowed to vary.

Then comes the less exciting work:

Permissions
Logging
Cost control
Failure handling
API outages
Unexpected model responses
User experience when something goes wrong
Traceability

Traceability is particularly important.

Helkyn has seen AI systems that worked beautifully on day one but couldn't answer basic questions about their own outputs:

How much did this answer cost?

Who produced it?

What input produced it?

Why was the answer produced?

Without traceability, production AI becomes difficult to understand, audit, and improve.

And as organizations consider AI governance and regulatory requirements, that becomes even more important.

Production Over Pilots

Helkyn's approach favors production over pilots and integrating AI into systems that already work rather than rebuilding everything from scratch.

So why do companies struggle to make that transition?

Because pilots and production systems are judged by completely different standards.

"A pilot is graded on whether it impressed someone in a room. Production is graded on whether it survives next week."

A pilot can succeed through speed, novelty, and a controlled demonstration.

Production has to survive:

Security reviews
Permissions
Error handling
Support
Real users
Messy data
Real operational conditions

Another problem is that pilots often avoid the hardest parts.

A company chooses a clean use case with tidy data because it wants to demonstrate a result.

But the process that would actually create business value might involve data spread across six different systems.

Helkyn also highlights another failure point: feedback that goes nowhere.

When users repeatedly provide feedback and nothing changes, they eventually stop reporting problems.

Silence then gets mistaken for success.

His approach is different:

Feedback should either result in a shipped change or an answer.

For moving from experiment to production, his advice is straightforward:

Choose a process that is genuinely painful and genuinely messy.
Build it small but end-to-end.
Include permissions and logging from day one.
Put it in front of real users early.
Watch what users actually do instead of only asking what they think.
Define what failure looks like before starting.
Keep the old process running until the new one earns the traffic.

It isn't particularly glamorous.

But that's often what production engineering looks like.

Building an Enterprise RAG System

One of Helkyn's current projects is an enterprise RAG knowledge system built around training material, documentation, and video transcripts.

The problem wasn't a lack of knowledge.

The knowledge already existed.

It was simply scattered across formats that people couldn't search consistently.

So employees would ask other employees for answers.

That worked, but it didn't scale.

The goal of the RAG system is to make that knowledge easier to find for employees and customers.

But building enterprise RAG exposed challenges that aren't obvious in tutorials or demos.

The biggest warning?

Permissions.

A RAG system can become an extremely efficient way of leaking information if different audiences have access to different documents.

Internal information, customer information, and information belonging to different clients cannot simply be placed into one index and protected with a prompt.

Access control needs to happen before the model sees the document.

That's an architectural problem, not a prompt-engineering problem.

Another major lesson was understanding what the documents actually represent.

A training video, customer call, support ticket, internal conversation, and written article aren't equivalent pieces of information.

They differ in:

Reliability
Freshness
Audience
Tone
Context

That means representation matters enormously.

"Retrieval isn't the answer, representation is."

The way information is indexed, chunked, categorized, and enriched with metadata can matter more than simply choosing a different model.

Helkyn is also clear that this project is still work in progress. The lessons are real, but final outcome metrics aren't available yet.

Where RAG Stops Being Enough

RAG is powerful, but it has boundaries.

At its core, RAG retrieves relevant information and gives it to a model.

But not every problem is a retrieval problem.

If someone asks:

"How many of these happened since March?"

That's a database query.

The answer should come from actual data, such as SQL, rather than asking a language model to perform calculations from retrieved documents.

Relationships create another challenge.

A document might explain a customer configuration.

Another might describe a support ticket.

Another might describe a release.

The interesting answer may depend on the relationship between all three.

That's where relationship graphs can become useful.

So when should you introduce agents?

Helkyn's test is practical:

An agent makes sense when a task genuinely requires a sequence of steps that cannot be known in advance.

But there's a second condition that matters even more:

"If you wouldn't give an intern unsupervised write access to that system, your agent shouldn't have it either."

He also challenges the assumption that every AI workflow needs to become "agentic."

If a process has clear inputs, clear outputs, and fixed logic, deterministic code may be the better solution.

His preferred architecture is often:

AI at the edges. Code at the core.

Let the model interpret messy input and explain results, while deterministic code handles processes with a single correct answer.

And when agents are involved, Helkyn prefers a model where:

The agent proposes. A human confirms.

At least until the pattern has earned enough trust.

Vibe Coding Won't Kill Developers. It'll Kill the Middle.

Helkyn recently wrote about the idea that "Vibe Coding Won't Kill Developers. It'll Kill the Middle."

For him, "the middle" isn't a statement about people's ability or intelligence.

It's about the market.

AI is increasingly capable of handling implementation against a known specification:

CRUD applications
Simple integrations
Brochure websites
Standard internal tools

That is precisely the kind of work AI can perform cheaply.

Helkyn compares it to photography.

Cheap cameras didn't eliminate professional photography.

They eliminated part of the market while making the high end more valuable.

The same pattern could emerge in software.

The uncomfortable question is what happens to developers whose careers traditionally depended on implementation against well-defined requirements.

The skills that become more valuable are different:

Architectural judgment
Security
Performance
Understanding what not to build
Deep domain expertise
Understanding production failure modes
Knowing how to use AI without becoming dependent on it

And Helkyn identifies one particularly difficult question:

"What I don't have a good answer for is how somebody starting today gets those twenty years, when the work that used to build them is the work being automated."

That's one of the biggest unresolved questions of the AI era.

AI Governance Without Killing Innovation

Governance can easily become bureaucracy.

Helkyn's approach is to avoid treating governance as a document people are expected to remember.

Instead:

"Make it a property of the system instead of a policy about the system."

Helkyn explains that AccountTECH is US-based, so the EU AI Act isn't a legal obligation for the company. Instead, they use it voluntarily as a reference framework for thinking about defensible AI systems.

One practical approach is to make AI usage part of the architecture itself.

For example, instead of maintaining a separate spreadsheet asking where AI is being used, AI usage can be represented directly alongside the application's existing screen and permission structure.

Then the inventory becomes a natural side effect of building the feature.

The second principle is:

"Gate for promotion, not for exploration."

People should be free to experiment.

The friction should appear when an experiment is ready to become production.

That's where review, permissions, logging, ownership, and other safeguards become mandatory.

As Helkyn puts it:

"Put the friction where the risk actually appears, and nowhere else."

The Mental Model in Your Head

Helkyn's CoderLegion profile contains a line that perhaps summarizes his entire approach to AI:

"The best AI in your stack is still the mental model in your head."

What does that mean in practice?

AI tools give developers answers.

But an answer is only valuable if you can understand and judge it.

If you understand the architecture and structure of the software you're building, you can evaluate AI-generated code.

If you don't, you risk outsourcing the architecture of something you're responsible for to a process you cannot properly audit.

Helkyn has experienced this himself.

He once spent an entire weekend trying to fix an AI-generated application that had suddenly stopped working.

After several days of troubleshooting, he discovered that the root cause was an architectural decision the AI had made around credential storage in a Supabase database.

It wasn't simply a coding mistake.

It was an architectural decision he hadn't realized the AI had made.

That experience reinforced an important lesson:

Don't let AI make important architectural decisions without guidance.

His advice to developers who rely heavily on AI is practical:

Understand the important parts of your application before using AI.
Decide what you think the solution should look like.
Ask the AI for its approach.
Compare the two.
Ask the AI to argue against its own solution.
Judge the arguments yourself.
When the AI is wrong, understand why it was wrong.

And perhaps most importantly:

Think of AI as a junior developer peer who knows nothing about your application.

That mindset changes how you review its work.

Helkyn has one final rule:

"Never merge code you can't explain line by line to another person."

Reading code and thinking it looks fine isn't the same as reviewing it.

If you can't explain it, you haven't really reviewed it.

You've just approved it.

25 Years Later, the Technology Keeps Changing

From Access and client-server applications to .NET, web platforms, RAG, AI agents, and AI-native software, Helkyn has watched the technology stack change repeatedly.

But his biggest lessons aren't really about any particular technology.

They're about judgment.

Technology makes building easier.

AI makes it dramatically faster.

But understanding what should be built, why it should be built, how it should behave in production, what could go wrong, and how it fits into the business remains deeply important.

The developers who thrive in this new environment may not be the ones who write the most code.

They may be the ones who understand the problem deeply enough to know:

what to build, what not to build, what to let AI handle, and what should never be delegated blindly.

About Helkyn Coello

View Helkyn Coello's CoderLegion Profile

Helkyn Coello is a technology leader and Chief AI Officer at AccountTECH with 25 years of experience building enterprise software.

His career has covered the transition from Access-era applications to .NET client-server systems, web platforms, and now AI-native software.

Today, his work focuses on enterprise AI adoption, RAG, agentic AI, AI governance, software architecture, and bringing AI into production systems.

He is currently exploring agentic AI patterns, orchestration, guardrails, deterministic software, and AI-powered knowledge graphs over complex enterprise data.

Follow Helkyn's work and join the conversation on CoderLegion.

Interviewed and edited by James Dayal, Editor at CoderLegion.

The KV Cache Formula That Should Set Your Context Limit

Mon, 07 Sep 2026 16:22:29 +0000

Phone prices are climbing this year largely because memory is scarce - AI data centers are buying DRAM faster than fabs can make it. That same squeeze shows up in your cloud bill, and it changes which knob you turn first when inference gets expensive.

Memory Is the Binding Constraint, Not Compute

When teams hit capacity limits serving a language model, the reflex is to ask for more GPUs. But on most serving setups the shortage isn't raw compute - it's the memory those chips carry. Model weights take a fixed slice. Everything left over goes to the KV cache: the running store of intermediate values for every token the model has already seen in a conversation. That cache grows linearly with context length and with the number of requests you serve at once.

So the real decision isn't "buy more GPUs or not." It's: do we cap context length and retrieve the relevant parts, or do we pay for the memory to keep stuffing everything in the prompt? Long-context models made the second option easy to reach for. Chipflation is making it expensive to keep reaching.

Running the Numbers Before You Ask for Budget

The formula is simple enough to put in a planning doc:

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value

# 7B-class model, GQA: 2 * 32 * 8 * 128 * 2 (fp16) = 131,072 bytes
# = 128 KB per token, per request
# 8k context -> 1.0 GB per request
# 32k context -> 4.0 GB per request

On an 80 GB accelerator with roughly 15 GB going to weights, you have ~65 GB for cache. At 32k context that's about 16 concurrent requests. At 8k, about 65. Same hardware, four times the throughput - purely from a context decision, no new chips required.

That's the trade-off stated honestly: you're spending retrieval quality to buy concurrency. The way to tell whether it's worth it is not vibes. Build a held-out set of 100 - 200 real user questions, answer each one twice - once with the full long context, once with an 8k window plus retrieval - and score them. If the quality gap is inside your tolerance, cap the window. If your users routinely need reasoning across a whole document, it isn't, and the extra memory is a legitimate line item.

There's a middle lever too: quantizing the KV cache to 8-bit halves the per-token cost with a smaller quality hit than truncation. Worth testing before you commit to either extreme.

Key Takeaways

Memory, not compute, usually sets your inference ceiling - and memory prices are rising for reasons outside your control.
Context length and batch size drive KV cache size linearly; a 4x context cut is roughly a 4x concurrency gain on the same hardware.
Decide with a scored comparison on real questions, not intuition - long context is a cost you should be able to justify.

If you ran that comparison on your own traffic, how much of your current context window is actually load-bearing?

Sources referenced: The Verge

Is Your Knowledge Base Actually Thinking, or Just Retrieving?

Wed, 02 Sep 2026 21:53:01 +0000

A while back I was able to build a working knowledge base in less than an hour. Fifty-five minutes to be accurate. I started by opening VS Code with Claude Code in the side panel. I had documentation scattered across roughly ten training manuals, a video archive nobody watched, support ticket histories, enhancement docs, and meeting transcripts. The knowledge existed. It was just spread across systems with nothing connecting them.

I described the islands to Claude the way I'd describe them to a new engineer. Then I asked for a chat interface backed by a vector database with RAG-style retrieval from plain-language questions. At around minute 45, it was running and returning accurate answers from the actual docs. I know it was not yet production software, but it was a real prototype on real data.

But I had slipped one extra requirement into that prompt, almost without thinking about it: client-specific information should be flagged and surfaced only when a query came from that client or was about that client. We had training recordings and support history tied to individual accounts. That requirement is the whole article. The moment you say "surface this when the query is about that client," you have stepped outside what retrieval by similarity can do. Similarity answers one question: which stored text resembles the wording of the query. But a support ticket and a training recording tied to the same client may share no vocabulary with each other, or with the question being asked. What links them is not wording; it is the client they both belong to. That link is a relationship, and a relationship is something a similarity search has no way to represent.

My prototype got away with it in a demo. In production, where real users ask cross-source questions every day, that shortcut stops holding.

What embeddings actually do, and where they stop

Embed-and-retrieve works like this: you chunk the documents, run each chunk through an embedding model, store the vectors, and at query time you embed the question and pull back the nearest neighbors by cosine similarity, a measure of how close two texts sit in meaning. It works. For a large class of questions — "what does our refund policy say," "how do I configure SSO" — it works well, and you can have it running in an afternoon.

The ceiling shows up the moment an answer lives across sources that don't look alike.

Take a question we actually get internally: "Why is this client's month-end close behaving differently from the others?" The answer is not in one chunk. It's in a custom configuration noted in a support ticket from two years ago, a behavior described in a training recording for that specific account, and a feature flag mentioned in an enhancement doc. None of those three documents are textually similar to each other, and none of them are similar to the question. An embedding search returns whatever happens to share vocabulary with "month-end close." That points straight to the generic manual, which is the one document in the set that does not contain the answer.

Similarity is a proxy for relevance. It's a decent proxy when relevant things tend to use the same words. It collapses when the relevant things are connected by something other than vocabulary: a shared client, a shared subsystem, a cause-and-effect chain that nobody wrote down in a single place.

This is the part that gets skipped in the market. A team sets up RAG, declares "the knowledge base is done," and stops exactly there. You end up with a search engine that talks. Useful, but it answers "what does the document say," and the questions that actually justify the project are the ones that ask "what does the situation mean."

The relationships are the knowledge

I was reminded of all this at Milano AI Week, where someone framed it cleanly: the real intelligence in a knowledge base isn't in the chunks; it's in how the segments relate, the graph rather than the index.

Concretely, that means a second representation alongside your vectors. You run entity extraction over the same corpus — clients, features, configurations, people, error conditions, dates — and you model the edges between them. Ticket 4471 references client Aurora. Aurora runs the custom close configuration. That configuration was introduced by enhancement request 882. The training recording from March covers that workflow. Those are explicit edges in a graph, and they are traversable regardless of whether any two of those nodes share a single word.

Now the same question (why is this client's month-end close behaving differently) gets answered along a different path. The system maps "this client" to the Aurora entity, walks the edges out to its configurations, tickets, and recordings, and assembles a candidate set that no similarity search would ever produce, because the documents have almost nothing lexically in common. The graph answers what is actually related; the embeddings answer what looks similar. You want both: similarity finds the entry points and the unstructured passages, and the graph traversal gathers everything connected. Either one alone leaves capability on the table.

I'm not arguing embeddings are wrong. I'm arguing that "chunks plus embeddings" is a floor people keep mistaking for a ceiling.

The cost is real, and it is not catastrophic

Here's the honest trade-off, because the graph approach is not free.

Entity extraction has to be tuned. The first pass over-extracts, treats every capitalized string as an entity, and cannot tell "Aurora the client" apart from "Aurora the internal name of a feature." Entity resolution — deciding that three different spellings refer to the same client — is where most of the real effort goes. You need a schema for what entities and relationships matter to your business, and that schema is opinionated, which means it requires someone who understands the domain to define it. You'll iterate it several times before it behaves. This is the same tuning loop I've hit building constraint surfaces for AI tools internally: the gap between how you described the rule and how the system interpreted it is always wider than you expect at the start, and you only close it by running real queries and watching what breaks.

Operationally you're now maintaining two stores and a pipeline that keeps the graph in sync as documents change. That's more moving parts than a single vector index. Budget for it.

But the cost difference between the two architectures is incremental, not an order of magnitude. The embed step is shared. The extra work is the extraction pipeline, a graph store, and the retrieval logic that combines the two. The capability difference, by contrast, is enormous. A chunk-only system can answer questions confined to a single document. The graph version can answer questions whose answer requires reasoning across sources that were never connected by an author. For an enterprise knowledge base, that second class of question is usually the entire reason the project was funded.

If your knowledge base is meant to serve support engineers, account managers, or anyone diagnosing a situation that spans history, the chunk-only version will demo beautifully and disappoint in week three, when the first cross-source question comes in and it confidently returns the generic manual. This is exactly the kind of representation decision we work through when we design a knowledge base meant to survive contact with real questions.

What to check before you sign off on an architecture

There's a simple test. Take five questions your team actually asks, the hard ones, the ones that make someone sigh, and for each one, ask yourself whether the full answer lives in a single document or has to be stitched together from several. If most of them need the stitching, similarity retrieval will not get you there, and no amount of better embeddings or bigger context windows fixes a representation problem.

That test is also the dividing line between a search engine and a knowledge base. A search engine retrieves text. A knowledge base models the entities your business cares about and the relationships between them, then reasons over that structure. The prototype I built in fifty-five minutes was the former with one relationship bolted on by hand. The production version of that same system is the latter, and the difference is entirely in how seriously you model the graph.

Before you approve any internal knowledge base design, ask the team building it one question: when the answer lives in three unrelated documents, what mechanism finds the other two? If the only answer is "similarity," that tells you two things: the team hasn't seen the ceiling yet, and the budget you're about to approve will buy a system that stops exactly where your hardest questions begin. Decide whether that's enough before the cost moves downstream into the maintenance no one can see.

If you are weighing that design decision now and want the retrieval architecture argued through against your own content rather than in the abstract, that is what RAG consulting is for.

Originally published at unlockedconsulting.ai.

From Web Development to Local AI: Josaphat’s Journey Through the AI Era

Tue, 01 Sep 2026 22:30:42 +0000

Developer Stories: From Web Development to AI — Josaphat’s Journey

An interview by CoderLegion

The world of software development is changing quickly. AI-assisted coding is making it easier than ever to build applications, experiment with new technologies, and turn ideas into working prototypes.

But there is another side to this transformation: are developers still learning deeply when AI can write so much of the code for them?

For G-Josaphat E. Tagba, known on CoderLegion as @Josaphatstar, this is a question he has experienced firsthand.

Josaphat is a web developer with a growing focus on artificial intelligence and data. Alongside his web development journey, he is pursuing a Master's degree in AI & Big Data, exploring areas including machine learning, data pipelines, statistics, and modern AI technologies.

His recent work reflects that combination of web development and AI. One of his projects involved building a local RAG assistant using Ollama, ChromaDB, LangChain, FastAPI, and Streamlit. The system was designed to help technicians search through thousands of pages of technical documentation using natural language while keeping sensitive information inside the local infrastructure.

We invited Josaphat to share his journey with the CoderLegion community—from getting started with web development and AI to discovering the hidden challenges of AI-assisted coding and building a real-world local RAG system.

1. From Web Development to AI

CoderLegion: You describe yourself as a web developer while also pursuing a Master's in AI & Big Data. How did your journey develop from frontend/web development into AI, and what made you want to explore both areas?

Josaphat:

It didn't start as a plan, it happened in parallel.

I started getting into web development on my own. I liked the immediate feedback of building things you could see and interact with.

At the same time, I enrolled in a Master's program in AI & Big Data, which took me in a completely different direction—machine learning, data pipelines, statistics, etc.

2. Learning to Code in the AI Era

CoderLegion: You've written about how learning with AI can make development much faster while sometimes making it harder to develop deep understanding. When did you first notice this yourself, and how has it changed the way you learn?

Josaphat:

I noticed it during a Nuxt project I built with heavy AI assistance.

When I finished, someone asked me to explain some of the decisions I'd made in my own code. And I struggled.

Not because the code was wrong, but because I hadn't really wrestled with it. I'd asked AI, got an answer, moved on.

That was the moment I realized there's a difference between moving fast and actually understanding.

Before AI was this capable, I had to sit with problems. That frustration was uncomfortable, but it was also doing something.

Every bug I fixed alone left a mark.

Now I can skip that uncomfortable part entirely, and that's both the power and the trap.

3. Building Your Local RAG Assistant

CoderLegion: You recently built a local RAG assistant using Ollama, ChromaDB, LangChain, FastAPI and Streamlit to help technicians search through thousands of pages of technical documentation.

What problem were you trying to solve, and what was the biggest challenge you encountered while building it?

Josaphat:

The idea was to build an AI assistant that could answer technical questions by searching through that documentation intelligently, in natural language.

The biggest challenge was the scale of the document ingestion.

We were working with around 10 PDFs totaling roughly 2,000 pages—about 10,000 vectors once processed into ChromaDB.

Getting that pipeline to work reliably, with consistent retrieval quality, took a lot of iteration.

The second challenge was debugging things I'd never built before. LangChain's API had changed significantly between versions, ChromaDB had breaking changes between v1 and v2, and LangGraph was new territory.

4. Why Local AI?

CoderLegion: One of the constraints of that project was that the data could not leave the local infrastructure, meaning you couldn't simply send the documents to OpenAI or another cloud AI provider.

How did that constraint influence your architecture, and what did you learn from building an AI system entirely locally?

Josaphat:

The constraint was non-negotiable: the documents contained sensitive technical data that couldn't leave the local infrastructure.

Cloud APIs were off the table.

Everything had to run locally.

5. When Things Break

CoderLegion: Your RAG project involved several real problems—from ChromaDB connectivity and changing LangChain modules to Llama 3 performance on CPU.

Which problem taught you the most, and what did it change about the way you approach debugging?

Josaphat:

The problem that taught me the most was the ChromaDB API change.

The ChromaDB client interface changed significantly between versions. Code that had worked stopped working, with errors that weren't immediately obvious.

Combined with LangChain deprecating several imports that needed to be migrated to langchain_core, it forced me to dig deeper into what was actually happening rather than simply assuming the code should work.

What Comes Next?

Josaphat's story doesn't end with building a traditional RAG pipeline.

He is also exploring the next stage of AI application development: agentic AI.

In the next part of our conversation, we'll explore his move from RAG toward agentic architectures, what real-world projects have taught him that tutorials couldn't, his experience working with clients, and what he believes developers should focus on as AI becomes increasingly capable.

About Josaphat

G-Josaphat E. Tagba (@Josaphatstar) is a web developer and Master's student in AI & Big Data, with interests spanning web development, artificial intelligence, machine learning, data, and AI-powered applications.

His recent work includes experimenting with RAG systems, local LLMs, AI tooling, and modern web technologies.

You can connect with Josaphat on CoderLegion:

https://coderlegion.com/user/Josaphatstar

This interview is part of CoderLegion's Developer Stories series, where developers share their real experiences, challenges, projects, lessons, and career journeys with the community.

How Synthadoc Handles Hallucination and Why "Don't Hallucinate" Isn't Enough

Fri, 28 Aug 2026 17:18:42 +0000

Ask most AI products how they handle hallucination and you'll get a variant of "we tell the
model not to." That's not wrong — it's just insufficient, and it puts enforcement in the
hands of the exact system doing the generating.

Synthadoc treats hallucination as an architecture problem, not a wording problem. The
distinction matters: a wording fix is hoping the model behaves; an architecture fix is
verifying that it did.

Why Prompt Instructions Aren't Enough

The industry default is a line in a system prompt: don't make things up, answer only from
the provided context, never fabricate. These instructions do nudge model behavior in the
right direction, and we use them too. But they leave enforcement entirely up to the model
doing the generating. A language model producing text token-by-token has no reliable
internal mechanism to audit its own faithfulness to a source.

There's a subtle issue as well. Instructions of the form "don't do X" are easier for a
model to rationalize around than instructions of the form "do only Y." Synthadoc's synthesis
constraint is a positive formulation:

Answer using ONLY these wiki pages. Cite with [[PageTitle]].

That's linguistically harder to rationalize around. But it's still just text in a prompt.
The real work happens at the structural level.

Three Layers, Not One Instruction

Our approach treats hallucination as a systems problem. There are three distinct layers,
each handling a different failure mode:

Scope filtering at ingest time : Wiki content can only be entered if it is on-domain. It is difficult to produce hallucinations about off-domain topics from content that has never been consumed.
Retrieval-augmented grounding at query time : Answers are generated from retrieved
wiki pages, not from the model's weights alone. The synthesis prompt shows only the
relevant pages and instructs the model to answer from them exclusively.
Citation faithfulness audit after the fact : Every claim gets a citation marker
linking it to specific source lines. A separate LLM pass then audits whether each claim
is actually supported by those lines. This is the layer that catches what the other two
miss.

None of these is a complete solution alone. Together they create a system where hallucinations
are detectable and traceable, not just discouraged.

Layer 1: purpose.md as a Content Gate

Every wiki has a wiki/purpose.md file that describes its domain — what it's for, what
topics belong, and what doesn't. When a new source document arrives for ingest, the ingest
agent reads this purpose block before making a decision:

Wiki scope (from purpose.md):
{content of purpose.md}

action="skip" means the source is completely OUTSIDE the wiki's domain
(e.g. spam, medical receipts, unrelated e-commerce).
…

The LLM then decides: skip, create, update, or flag. Out-of-scope content never
becomes a wiki page, which means it can never become the basis for an answer.

Lifecycle isolation for contradicted pages

The scope gate isn't the only place the knowledge base shrinks. When a new source
disagrees with an existing page, the ingest agent issues a flag action and the page
moves to a contradicted lifecycle state, and automatically excluded from retrieval until
the conflict is resolved. This matters because a contradicted page is, by definition,
one where the system can't be confident about what's true. Serving answers from it
would be hallucination with extra steps.

The contradiction resolver in Synthadoc is an interactive agentic workflow: it surfaces
the conflict, shows a diff of the proposed edit, and requires explicit human approval
before writing any change. The approve-before-write gate is unconditional, and the agent
cannot modify a page without a human seeing the diff first.

Layer 2: RAG Grounding at Query Time

When a user asks a question, Synthadoc retrieves the most relevant wiki pages using
hybrid search (BM25 full-text matching with optional vector re-ranking), then builds a
synthesis prompt that includes those pages verbatim:

Answer using ONLY these wiki pages. Cite with [[PageTitle]].
Extract and include all specific facts from the pages — dates, years,
numbers, and names — even when they appear briefly or in passing.
Do not claim a fact is absent unless it is genuinely missing from
every page below.

Question: {question}

Pages:
{retrieved_wiki_pages}

The purpose.md content is prepended to the pages block as ### Wiki Scope, giving the
model a domain reminder during synthesis as well as at ingest time.

Why this isn't enough on its own: A model can still hallucinate within retrieved
content — misattributing which page said what, inventing a specific number close to but
not the same as a number in the source, or synthesizing a conclusion that no single page
actually states. Grounding reduces the search space; it doesn't eliminate drift.

When retrieval confidence falls below threshold, Synthadoc detects the gap and handles
it transparently rather than hallucinating silently. The immediate response answers from
the model's general knowledge while explicitly noting that the wiki lacks dedicated
coverage — the user always knows they're receiving an unverified answer. At the same time,
Synthadoc generates enrichment suggestions: search queries and known URLs scoped to the
domain in purpose.md. Retrieved pages flow through the normal ingest pipeline and stay
in draft state — out of the search corpus — until a human promotes them to active.
Only then do future queries on that topic get answered from verified wiki content rather
than general knowledge.

Layer 3: Citation Faithfulness Audit

This is the layer that makes the system verifiable rather than just hopeful. The citation
faithfulness audit examines each claim in a wiki answer alongside the source lines it cites,
then classifies the relationship:

Verdict	Meaning
`supported`	The claim is directly and accurately supported by the cited source lines.
`drift`	The claim is related to the source but overstated, simplified, or shifted in meaning.
`hallucination`	The source lines do not support the claim, or directly contradict it.

Each audit is done by a separate LLM call with the claim text and raw source lines
side-by-side. The auditor never sees the original question — only the claim and the lines.

Why a separate LLM pass? The same model that wrote the answer is likely to defend
it when asked "is this correct?" Separation of duties matters: the auditor has no stake
in validating the original generation.

What we actually see in practice

In our experience, drift is the most common finding — not outright hallucination.
The model summarizes accurately but slightly overstates a confidence level, rounds a
number, or drops a qualifier ("as of 2023" becomes "currently"). These are small errors
individually but compound across a knowledge base that gets queried repeatedly.

True hallucination , where the cited source lines don't support the claim at all ,
is less common in well-grounded RAG, but it happens: specific dates that don't appear
in the source, an attribution switched between two people mentioned in the same paragraph,
a causal relationship inferred from correlation. Those are the findings that matter most,
and the audit surfaces them per-citation so a human knows exactly which source line to
check.

The audit result is cached per-page with staleness detection. If the page is re-ingested
from updated source material, the cache is invalidated and the next audit reflects the new
content. This surfaces in the web UI as an automatically-loaded faithfulness tab alongside
each wiki page.

What This Doesn't Solve

Honesty matters here too. Two failure modes remain outside what the three layers fully
address:

Synthesis across pages. A conclusion drawn by combining facts from multiple pages may
be accurate at each individual step and still misleading in combination. The citation
faithfulness audit operates per-citation, not across synthesized conclusions, so a
cross-page inference that no single page actually states falls outside its scope.
The practical path: when a synthesized answer matters, the cited pages are explicit and
can be verified directly. If the conclusion is important enough to keep, saving it as a
wiki page subjects it to future source ingestion — a new source that disputes it will
trigger the flag action for human review.

Source quality. When a new source disagrees with an existing page, the flag action
fires and the contradiction is surfaced. The harder case is when all available sources
agree on the same wrong answer: no flag fires, no contradiction is detected, and Synthadoc
answers confidently from incorrect material. Two mechanisms make this less likely: the
adversarial review in lint, which uses the model's own knowledge to flag claims that
clearly contradict well-established facts; and re-ingesting from updated authoritative
sources, which will trigger the flag action against any outdated pages. Neither is a
complete solution, but together they make uniform source error less likely to go
unnoticed indefinitely.

Our View

The hallucination debate tends to frame the problem as a model capability question: will
the next model be more faithful than the last? We think that's the wrong frame for a
production knowledge system. Even a perfectly faithful model can only be as accurate as
the content it's grounded in, and even a slightly drifting model can be made accountable
if the output is systematically audited.

Our architectural bet is that the answer to hallucination isn't a better instruction,
it's a feedback loop. Domain scoping controls what enters. Lifecycle states control what
gets queried. Retrieval grounds generation. Citation audit classifies the output. The
contradiction resolver closes the loop: when a conflict is found, a human resolves it, the
page is corrected, and the knowledge base improves. Errors become visible rather than
accumulating silently.

That's a more defensible position than any number of ALL CAPS instructions.

Try It

Synthadoc is open-source. The citation faithfulness audit, contradiction resolver, and
lifecycle system described here are all in the main branch.

→ github.com/axoviq-ai/synthadoc

Why Your RAG Pipeline Cannot Answer What Payment Processor Do We Use

Fri, 21 Aug 2026 17:17:31 +0000

Most of us here have shipped at least one retrieval layer that worked beautifully in the demo and then quietly failed in production. This is a writeup of the specific failure mode I keep running into, and the extraction step that actually fixes it.

The failure

Store this sentence in your vector index:

we migrated the checkout flow from Stripe to Braintree last quarter

Three months later somebody asks:

what payment processor do we use

The two embeddings are not close enough, because neither the word payment nor the word processor appears anywhere in the stored text. The fact is in your database. The vocabulary is not. No amount of chunk size tuning fixes this, because chunking was never the problem.

Entity extraction is the missing step

Entity extraction turns that sentence into structured facts before you store anything:

checkout flow uses Braintree (current)
checkout flow previously used Stripe (historical)
a migration event occurred, with a date

Now the question resolves by looking up an entity and its type, not by hoping the wording overlaps.

NER and LLM extraction solve different halves

Traditional NER is fast and cheap. SpaCy's en_core_web_trf hits 89.8% F1 on OntoNotes and processes thousands of documents per second on a single CPU. Fine-tuned BERT models reach 92 to 93% on the same benchmark. What both give you is the standard label set: PERSON, ORG, GPE, DATE, MONEY.

The moment your domain invents its own entity types, and it will (microservice, API endpoint, deployment environment, database table), a pretrained NER model simply does not see them. You either fine-tune, which costs 200 to 500 labeled examples per entity type, or you describe the types in a prompt and let an LLM extract them.

The cost difference is the whole argument. Running 100,000 documents through an LLM extraction prompt costs roughly $300 to $1,500 depending on the model. The same corpus through SpaCy costs under a dollar.

So most production pipelines now run tiered extraction. The small model goes over everything, and the LLM only touches passages where domain-specific entities or ambiguous relations are likely, using the NER output as context so nothing gets extracted twice. That cuts LLM spend 60 to 80 percent while keeping accuracy where it matters.

Relations are where the graph actually forms

Entities on their own are a list. Relations are what make it traversable, and relation extraction is the harder half, because it needs the semantic connection rather than just the mention.

John mentioned Redis during the architecture review contains two entities and a very weak relation. John mentioned it. He does not maintain it, own it, or depend on it. A pipeline that treats co-occurrence as a relation fills your graph with edges that mean nothing, and every traversal after that inherits the noise.

The one that bites everybody eventually is predicate normalization. If uses, utilizes, relies on, is built on and depends on all land as separate predicates, traversal silently misses most of the graph. It presents as a retrieval bug and it is really a schema bug.

The full writeup, with the pipeline stages, the tiered cost model and the section on measuring extraction quality, is here: Entity Extraction and NER for AI

If you have built one of these, I am curious which half broke first for you, the entities or the relations.

ChatGPT Search Is Firing site: Queries - Test Yours Today

Fri, 21 Aug 2026 08:46:45 +0000

ChatGPT's search step increasingly issues site:-scoped queries rather than broad open-web searches. This single detail changes what "being findable" means for anyone with a site, docs, or a blog.

What a site:-Scoped Search Actually Implies

When a model decides to search, it writes the query itself. A broad query like best invoicing tools for freelancers pulls from the whole index and the model picks whatever ranks. A scoped query like site:yourdomain.com pricing means the model has already decided your domain is the right source and now wants a specific fact from it.

Broad search rewards general authority. Scoped search rewards whether the specific answer exists on a crawlable page on your domain, in text, at a URL a search index has actually seen. If your pricing lives inside a JavaScript-rendered widget, or your API rate limits are buried in a PDF, a scoped query returns nothing - and the model either guesses, hedges, or moves on to a competitor's docs page.

For AI engineers building retrieval-augmented systems (RAG - feeding a model external documents at query time), this should feel familiar. It's the same failure mode as a chunk that never got indexed. The difference is the index isn't yours.

Run the Test Yourself

You don't need a GEO vendor to check this. Run the same scoped queries a model would generate:

# What a model asks when it already trusts your domain
site:yourdomain.com pricing
site:yourdomain.com "rate limit"
site:yourdomain.com refund policy

Paste each into Google or Bing directly. Then check whether the answer is in the visible snippet - not three clicks deep.

Then check crawlability from the terminal:

curl -s https://yourdomain.com/pricing | grep -i "per month"

If curl returns your raw HTML but the price isn't in it, that fact is client-side rendered and invisible to most crawlers. Same test for docs pages, FAQ answers, and comparison tables.

Three fixes, in order of payoff:

Put the literal answer in server-rendered HTML. Prices, limits, versions, dates.
Give each discrete fact its own indexable URL. One page answering "what are the rate limits" beats a mega-page with an anchor link.
Use the phrasing a user would type. Models generate queries from user language, not your internal product vocabulary.

This is unglamorous work - closer to technical SEO hygiene than to prompt engineering. That's precisely why most teams skip it.

Key Takeaways

Scoped site: queries mean the model already chose your domain; the only question is whether the fact is retrievable.
Client-side-rendered facts are functionally invisible to the crawl step - curl plus grep tells you in seconds.
One clear fact per indexable URL, written in user phrasing, outperforms a comprehensive page that buries it.

Run one scoped query for your own site right now - which specific fact came back empty that you assumed was public?

Sources referenced: Simon Willison's blog, Promptwatch reporting

Measuring whether semantic chunking actually helps, on one fixed corpus

Tue, 18 Aug 2026 16:08:46 +0000

Everyone repeats that semantic chunking beats fixed-size chunking for RAG. I wanted to know if that was true for my corpus, or just a thing people say. So I built the comparison into EvalBench: one corpus, one query set, three chunking strategies, and a single precision number I could read off a leaderboard.

The result I care about here is not a benchmark score. It is the setup. Getting a fair comparison between chunkers is harder than writing the chunkers, because almost everything that makes retrieval look good or bad has nothing to do with how you split the text.

The trap: comparing chunkers by comparing pipelines

If you swap the chunker and also swap the embedder, or the corpus, or the top-k, you have not measured the chunker. You have measured a different pipeline. Most "semantic wins" posts do exactly this, and you cannot tell how much of the win is the split versus everything else that moved with it.

So in EvalBench the only thing that varies is the split. A RAG run is identified by an embedder::chunk_strategy string, parsed apart before anything runs:

def parse_pipeline_model(model: str) -> tuple[str, ChunkStrategy]:
    """Parse the exact ``embedder::chunk_strategy`` row identifier."""
    ...
    if strategy not in _ALLOWED_STRATEGIES:
        raise ValueError(f"unknown chunk strategy {strategy!r}")

Same embedder, same 200-document corpus, same 15 queries across finance, legal, medical, physics, and software. The strategy is the one knob. Everything downstream is shared code.

The three strategies

There are three, not two. The interesting comparison is not fixed versus semantic, it is fixed versus recursive versus semantic, because recursive is what most people actually ship.

fixed_512 is the dumb one. Whitespace-tokenize the whole document, cut it into 512-token windows with a 64-token overlap, done. It respects nothing about the text.

def _window_token_groups(tokens, window=512, overlap=64):
    groups, start = [], 0
    while start < len(tokens):
        end = min(start + window, len(tokens))
        groups.append(list(tokens[start:end]))
        if end == len(tokens):
            break
        start = end - overlap
    return groups

recursive splits on structure first. Try paragraph breaks, then single newlines, then sentence boundaries, and only fall back to a hard token cut when a piece is still too big. Then it repacks those units into bounded chunks. This is the LangChain-style splitter, and it is a real baseline, not a strawman.

semantic is the one people get excited about. Embed every sentence, walk them in order, and start a new chunk when the meaning drifts. Drift is measured as cosine similarity between adjacent sentences dropping below a threshold:

low_similarity = (
    len(current_sentences) >= 3
    and cosine_similarity(vectors[index - 1], vectors[index]) < 0.65
)
too_large = len(current_tokens) + len(sentence_tokens) > 512
if current_tokens and (too_large or low_similarity):
    flush()

Two things in that snippet matter and neither is obvious from the marketing. The >= 3 guard means a chunk needs at least three sentences before a similarity drop is allowed to break it, so one weird sentence early on does not shatter a chunk into fragments. And the too_large check means semantic still has a hard 512-token ceiling. So it is not really "split on meaning." It splits on meaning until it hits the same 512-token window fixed_512 uses, then cuts. That ceiling is what keeps the comparison fair. If semantic could emit arbitrarily long chunks, it would win on precision for a boring reason: bigger chunks catch more gold documents.

The metric has to be simple enough to trust

I score every strategy on context precision. Take the top 10 ranked chunks, count how many come from a document in the gold set, divide:

retrieved_chunks = ranked_chunks[:10]
context_precision = (
    sum(chunk.doc_id in gold_doc_ids for chunk in retrieved_chunks)
    / len(retrieved_chunks)
    if retrieved_chunks
    else 0.0
)

That is deliberately crude. It counts a chunk as relevant if its source document is relevant, not if the chunk text itself answers the query. A stricter metric would grade chunk content, but then I would be measuring my grader as much as my chunker, and I already have a separate faithfulness judge for content quality. For a chunker comparison, "did we pull chunks from the right documents" is the question, and document-level precision answers it without dragging an LLM into the measurement.

Every metric is clamped to [0, 1] with a hard failure if it lands outside, so a broken chunker that emits duplicate or malformed chunks throws instead of quietly publishing a nonsense score:

def _clamp_metric(value):
    tolerance = 1e-12
    if not math.isfinite(value) or value < -tolerance or value > 1.0 + tolerance:
        raise ValueError(...)
    return float(min(1.0, max(0.0, value)))

What broke, and what I would do differently

The gold labels are per document, not per chunk. That is the real limitation here. If a relevant document is 40 chunks long and only 2 of those chunks actually answer the query, my precision metric happily credits all 40 as relevant when they surface. So this setup answers "does this strategy retrieve from the right documents," and it is blind to whether it retrieves the right passage inside them. For a first pass on a 200-document corpus that is a reasonable trade, but it means a chunker that fragments a relevant document into many pieces can look better than one that pulls a single tight passage, purely because more of its chunks share a gold doc_id. Chunk-level labels would fix this and are the obvious next step.

The other thing I would change is the sample size before trusting any ranking. 15 queries, 3 per domain, is enough to see a large effect and nowhere near enough to call a small one. EvalBench already publishes Wilson intervals next to every mean for exactly this reason, so the leaderboard shows the uncertainty instead of hiding it. If two strategies land within each other's intervals at n=15, the correct read is "no measured difference," not "semantic wins by a hair." I would rather report a wide interval than a confident wrong number.

Takeaway

If you want to know whether semantic chunking helps, hold the embedder, corpus, queries, and token ceiling fixed, and vary only the split. Then read the precision with its confidence interval attached. A comparison where three things moved at once tells you nothing about the one you changed.

The four axes generic RAG eval forgets for search agents

Tue, 18 Aug 2026 09:20:10 +0000

A search-augmented agent looks like RAG until the citation list breaks. The first user clicks through, lands on a 404, and the trust budget spends in one impression. Three months of eval work, groundedness and context adherence and factual accuracy all green, did not catch it.

The reason is structural. Generic RAG eval was built for a frozen vector store and a stable ground truth. Search agents have neither. The corpus is the live web, the agent writes the literal query string that drives what comes back, sources span government filings and LinkedIn takes in one result set, and users click the URLs. Skip the four axes unique to this shape (query quality, source freshness, source diversity, citation validity) and you ship an agent that confidently cites stale or fabricated URLs. The rest of RAG eval still applies, but the rest of RAG eval was already passing. There is a longer version with the rubric code if you want it.

Three differences pull this out of the RAG bucket. The corpus is not yours: two runs of the same query twelve hours apart return different URLs, so reproducibility lives in a snapshot, not the results page. The query is the lever: search is sensitive to the literal string the agent writes, and a rewrite that drops a constraint returns a clean-looking set that misses it, which no synthesis prompt recovers from. And the user verifies: a citation is a URL the user clicks, so a fabricated or 404 link is a one-click trust failure that groundedness against the retrieved set never sees.

Score the four axes per query, not aggregated. A single number hides which one collapsed, and the diagnostic value is in the vector.

Axis 1: query quality

The agent turns the user's prompt into a search string, and that string is what the engine literally reads. Three failure modes show up in almost every run.

Constraint drop. "Laptops with battery over 10 hours" compresses to "best laptops 2026 for ML," and the results average five-hour batteries. Groundedness is fine because the brief grounded itself in the retrieved set. The constraint is gone.
Entity loss. A question about a specific company or regulation abstracts to the category, and the brief answers a different question well.
Over-broad rewrite. A narrow factual lookup expands into a topic, the search returns a Wikipedia overview, and the brief paraphrases it and misses the fact.

Score it with a judge that takes the original prompt and the rewrite as a pair and rates constraint preservation and entity recall 1 to 5, penalizing unnecessary expansion as much as a drop. Run this over a week of traffic and the first cluster is usually "rewriter dropped the year, version, or region" on roughly one in eight queries. Cheapest fix in the loop, and the one most teams skip.

Axis 2: source freshness

Freshness is a function of the question, not a global setting. "Capital of France" is insensitive. "Current price of X" is real-time. "FDA guidance on AI as a medical device" is quarter-sensitive. The agent has to decide the regime and behave: prefer recent sources, refuse stale ones, and add a recency caveat when the answer depends on current data.

Two checks carry it. Classify the freshness regime per query (insensitive, slow, fast, real-time) and attach it as a span attribute. Then attach a recency value per source, take the median recency of the cited sources, and fail when it exceeds the floor for that regime. The stale-fact bug is the one users notice last and complain about loudest: correct the day the source was indexed, wrong every day after, and invisible to every other metric.

Axis 3: source diversity

A general search returns Wikipedia, vendor blogs, news, government docs, and SEO spam in one set, and the agent cites three to five. The failure is monoculture: every citation from one domain, or a brief that reads like consensus when it summarizes one tab.

Three checks cover it, and the first two are deterministic and run in microseconds:

Unique-domain count per citation set, with a floor by question class. A date lookup can be tight; a synthesis question needs four or more.
Per-domain cap. No domain contributes more than two citations, unless the question is about that domain (an EU AI Act question can cite eur-lex repeatedly).
Primary-vs-secondary ratio. Classify each citation as primary (a filing, paper, or announcement) or secondary (commentary, aggregator). A regulatory question with zero primary citations is a failure however grounded the brief is.

Axis 4: citation validity

The highest-signal cheap check in the whole stack, and the one most teams find only after the first complaint. Three sub-checks compose it.

The URL resolves. HEAD-request every citation; a non-2xx or a redirect to a different domain is a fail. Tens of milliseconds, and it rejects fabricated and stale links before the user sees them.
The passage exists on the page. Fetch the body and confirm the quoted or paraphrased passage is actually there. A paraphrase that drifts past a similarity threshold is a fail. This catches a real URL with an invented quote.
The claim is supported by that passage. Groundedness and chunk attribution, but per claim, not per answer. A brief can be 0.94 grounded as a whole and 0.61 aligned per claim, and users only see the per-claim view.

Wire it as two stages: the deterministic checks (resolves, passage present) as a hard gate, the claim-passage entailment as a judge on the synthesis span. A citation that fails any check is a hard reject: log it, regenerate the answer with that URL excluded, and re-run the gate. Teams that wire this stop seeing the "user clicked a 404" failure almost immediately.

Cache the snapshot so CI does not flake

Two runs of the same prompt hours apart return different URLs, so without a per-test-case cache the gate flakes for reasons unrelated to the agent and the team stops trusting it. Cache three layers: the search call (query and full result JSON), the page fetch (URL, body, timestamp), and the extraction (cleaned text, date, anchors). Replay every run from cache, and re-snapshot deliberately on a schedule that fits the domain (weekly for news, monthly for stable how-to agents), treating each re-snapshot as a new golden-set version with its own baseline.

Trace it so a bad answer is debuggable

The eval scores attach to spans, and the spans are how you debug. Three span kinds carry the load: a retriever span per search call, carrying the literal query the agent sent plus per-source URL, recency, and authority; a tool span for the wire call to the search API, so latency and cost land on the right node; and an LLM span for the synthesis. Splitting the search call from the retriever span keeps the query-quality input and the network cost on separate, queryable nodes.

Run the four axes per query and read the vector. The axis that drops is the bug, and the standard RAG metrics keep running alongside because they are necessary, just not sufficient.

Why Production Agents Are Ditching Pure RAG for Context Caching

Sun, 16 Aug 2026 10:07:01 +0000

Why Production Agents Are Ditching Pure RAG for Context Caching

Three months ago, our agentic workflow hit a cost wall. We were running a multi-step research agent that performed iterative document analysis. Every user request triggered between 12 and 18 agent steps. Across 50,000 active daily sessions, our vector database query costs and LLM input token bills were spiraling out of control—and worse, median step latency hovered around 4.2 seconds.

Like most teams building in 2024 and 2025, we assumed the fix was "better vector search." We tweaked top-k parameters, tried hybrid dense-sparse reranking, and chunked documents down to tiny 256-token fragments. It didn't fix the core problem. The bottleneck wasn't retrieval quality; it was retrieval architecture.

In production agentic systems, replacing pure vector retrieval-augmented generation (RAG) with long-context prompt caching (Cache-Augmented Generation or CAG) combined with lightweight local router models reduces median agent step latency by up to 68% and cuts monthly API inference spend by over 80%. This architectural pattern leverages deterministic context reuse over fragment chunking.

TL;DR / Key Engineering Takeaways

Vector RAG introduces compounding latency in agent loops: Each agent hop requires vector embedding, similarity search, top-k ranking, and prompt assembly, multiplying network overhead across multi-turn sessions.
Context Caching eliminates redundant prefix processing: Modern API providers allow long documentation, system schemas, and conversation histories to be cached at the edge, reducing token input costs by 75%–90% on cached prefixes.
Hybrid Local Routing stops token leakage: Routing lightweight classification, routing, and tool-selection decisions to local 8B–30B models reserves heavy frontier models only for final synthesis.
The 2026 Agent Standard is hybrid: Vector search handles sparse global discovery across millions of items, while context caching handles the dense session working memory.

The Production Bottleneck: Why Vector RAG Struggles in Agent Loops

When we first built our agentic pipeline, we followed the standard textbook pattern. Every agent step extracted state, searched a vector index, pulled the top-5 chunk snippets, and fed them into the prompt window alongside system instructions.

It worked great in prototypes. But once we moved to multi-agent orchestration—where agents delegate sub-tasks, summarize intermediate progress, and call tools repeatedly—the flaws became obvious.

[ Traditional Vector RAG Agent Step ]
State -> Query Gen -> Embed Query -> Vector DB Search -> Top-K Rerank -> Build Prompt -> LLM Inference
Total Step Latency: ~3.8s - 5.2s (Repeated for 15 steps = ~60s total session time)

[ Context Caching / CAG Agent Step ]
State + System Context (Pre-Cached Prefix) -> LLM Inference (Cached Hit)
Total Step Latency: ~1.1s - 1.6s (Repeated for 15 steps = ~18s total session time)

In an agentic loop, context is not static; it evolves incrementally. In our earlier writing on what an agent actually is, we broke down how an agent's harness relies on continuous feedback loops. Re-embedding queries and assembling fragmented chunks at every step creates three major friction points:

Lost Semantic Coherence: Chunking long technical specifications or codebase trees into 500-token snippets strips away critical cross-document relationships.
Double-Payment on Tokens: You pay vector database read costs and re-pay LLM input token processing fees for text the model already parsed two steps ago.
Compounding Latency: Adding 300ms vector index latency across a 15-step agent execution loop adds nearly 5 seconds of pure overhead to every user request.

Enter Context Caching (CAG): How Deterministic Prefix Reuse Works

Context Caching changes the economic math entirely. Instead of searching a vector database for relevant fragments on every turn, we load entire documentation bundles, domain schemas, or codebases directly into the LLM's extended context window (128k to 1M tokens) and cache the KV-cache states at the provider layer.

When the agent executes subsequent steps, the model processes only the new delta tokens (e.g., tool call outputs or user turns). The static prefix—which contains 95% of the total tokens—is retrieved instantly from the cached KV state.

+-------------------------------------------------------------------------+
| CACHED PREFIX (Static - 95% of Tokens)                                  |
| System Persona + API Schemas + Entire Core Codebase / Knowledge Base   |
| (Cost: ~90% Discount / Latency: ~50ms TTFT)                             |
+-------------------------------------------------------------------------+
| UNCACHED DELTA (Dynamic - 5% of Tokens)                                 |
| Current State + Latest Tool Execution Result + User Query               |
+-------------------------------------------------------------------------+

As we explored in our breakdown From RAG to CAG: A Smarter Approach to AI, shifting from retrieval-on-demand to context pre-loading simplifies system design. You eliminate vector index maintenance, embedding pipelines, and chunking heuristics.

Empirical Benchmark: Vector RAG vs. Context Caching vs. Local Hybrid

To measure the exact performance difference, we benchmarked three architectural approaches across 1,000 multi-step technical research tasks (averaging 12 agent turns per task, analyzing a 100,000-token technical repository):

Metric	Standard Vector RAG (Top-K=10)	Pure Context Caching (CAG)	Hybrid Router + Context Caching
Median Step Latency (TTFT)	3,840 ms	1,210 ms	890 ms
Total Session Latency (12 turns)	46.08 s	14.52 s	10.68 s
Input Token Cost / Session	$0.184	$0.038	$0.021
Task Accuracy / Completion Rate	78.4%	91.2%	92.6%
Vector DB Read Cost / 1k Sessions	$12.50	$0.00	$0.00

Benchmark Takeaways:

Latency: Context Caching dropped Time-to-First-Token (TTFT) by 68.4% compared to traditional vector RAG because the prompt prefix didn't require re-computation.
Cost: Token costs dropped 79.3% under pure CAG and 88.5% under the Hybrid Router approach because cached tokens incur a fraction of standard input pricing.
Accuracy: Full-context availability eliminated RAG retrieval failures (where vector search missed the relevant chunk due to poor keyword matching or semantic drift).

Architectural Deep Dive: Building the Hybrid Local Router Pattern

While Context Caching is powerful, pre-loading 100k tokens into a frontier LLM on every step can still be wasteful if the agent only needs to make a binary decision or select a simple tool.

To optimize latency further, we introduced a Local Hybrid Router.

How the Hybrid Router Architecture Works

                        +----------------------------+
                        |     Incoming User Request  |
                        +----------------------------+
                                      |
                                      v
                        +----------------------------+
                        |  Local Router (8B / 30B)   |
                        | (Tool Choice / State Check)|
                        +----------------------------+
                               /              \
                              /                \
          Simple Action / Tool Call          Complex Reasoning / Synthesis
                            /                    \
                           v                      v
           +-----------------------+   +------------------------------------+
           | Local Execution Loop  |   | Frontier LLM with Context Caching  |
           | Fast & Zero-Cost API  |   | Cached KV Prefix (100k+ Tokens)    |
           +-----------------------+   +------------------------------------+

Step 1: Local Triage: Incoming step state is evaluated by a small local model (e.g., an 8B model running on local GPU infrastructure or edge endpoints).
Step 2: Simple Execution: If the step is a mechanical tool invocation (e.g., git log, read_file, check_status), the local model executes it directly.
Step 3: Frontier Synthesis: If the step requires deep reasoning or complex synthesis across the codebase, the request routes to the frontier LLM backed by the cached KV prefix.

This hybrid approach mirrors the principles we outlined in our guide on real-time ranking systems at scale: decouple high-frequency lightweight operations from heavy, resource-intensive compute paths.

When Vector RAG Still Wins (And When to Avoid CAG)

Context Caching is not a silver bullet. During our production migration, we identified explicit boundaries where pure CAG fails and vector RAG remains necessary:

1. Corpus Size Exceeds Window Capacity

If your domain knowledge base spans millions of documents (e.g., 500 million tokens of legal case law), pre-loading the entire corpus into context is impossible or prohibitively expensive.

2. Low Prefix Reuse Ratio

Context caching relies on prefix stability. If user queries are entirely distinct and share no common system prompt, documentation context, or conversation history, you pay cache-creation surcharges without reaping cache-hit discounts.

3. Dynamic Write-Heavy Databases

If your knowledge base updates thousands of times per minute, invalidating the cached KV prefix on every write negates the performance benefits of caching.

Lessons Learned & Production Migration Checklist

If you are currently operating a production RAG pipeline and struggling with latency or costs, here is the incremental migration path we recommend:

[x] Audit your prompt prefix stability: Group your agent prompts into static prefixes (system instructions, tool definitions, reference docs) and dynamic suffixes (user query, latest message).
[x] Enable provider-level prompt caching: Activate KV-caching on your primary LLM endpoints (Claude, Gemini, or OpenAI) to instantly capture 50%–75% input cost savings on static prefixes.
[x] Implement a hybrid retrieval gate: Use vector search only for initial broad document discovery (filtering 10M tokens down to 100k tokens), then cache that 100k-token working set for the rest of the agent session.
[x] Offload simple tool routing to local models: Deploy a small local model to handle state-tracking and simple tool selections without waking up the frontier LLM.

Conclusion

Building reliable agentic systems requires moving past prototype patterns. Vector RAG served us well during the early era of short context windows, but in 2026, forcing multi-turn agents to constantly query vector databases is like making a developer query a database for every line of code they write.

By combining long-context prompt caching with intelligent local routing, you give your agents the entire working memory they need while slashing inference costs and step latency.

RAG on Excel with LlamaParse

Thu, 13 Aug 2026 05:25:00 +0000

RAG on Excel.

To perform RAG on documents like PDFs, we split the data and create chunks to generate vector embeddings, right?

But how can we split data which is in tabular format?

In this video, let's discuss how to perform RAG on an Excel file using Llama Parse

Code Notebook - https://lnkd.in/dgCjjrZr

About "AI ML etc."

We have reimagined AI education for senior IT professionals and specifically designed AI courses for them.

These courses cover essential AI concepts, AI engineering, Indutsry best practices and AI Management topics.

340+ learners from reputed organisations like Microsoft, Google, Nvidia, Meta, Nagarro, Bank of America, Infosys, Maersk and more have taken our courses

How to Tell When Your LLM's Knowledge Actually Stops

Tue, 11 Aug 2026 07:38:10 +0000

Every AI model has a knowledge cutoff - but the official date rarely captures the whole story.

The Hidden Gap Between Training and Knowing

Model providers list a knowledge cutoff date, but that figure is deceptive in practice. According to research on model training practices, training data isn't sampled evenly across time, and content published close to the cutoff is underrepresented because the web hadn't fully indexed, discussed, and linked to it yet when the training crawl ran. The result: a model with a stated cutoff of, say, late 2024 often behaves as though its reliable knowledge ends several months earlier. Call it a soft cutoff: the point where confident, well-corroborated knowledge fades into thin, patchy coverage.

This matters whenever you're building with LLMs for anything time-sensitive: a research assistant, a competitor-monitoring tool, a news summarizer, or a RAG pipeline (retrieval-augmented generation - a pattern where you inject fresh documents into the model's context at query time). Knowing the soft cutoff lets you determine how much to trust the model's parametric memory versus forcing retrieval for recent facts.

Real Example: Probing the Soft Cutoff Directly

You can empirically test where a model's knowledge gets shaky with a simple prompt pattern. Run this against whichever model you're using:

List 5 significant events in [domain] from [month, year].
For each, rate your confidence 1-10 and explain any uncertainty.

Start from a date you know is well within the cutoff, then step forward month by month until confidence scores drop or the model starts hedging heavily. That inflection point is your practical soft cutoff for that domain.

For a more systematic check in a pipeline context, you can log model responses against ground-truth dates:

def check_cutoff_confidence(client, domain, year, month):
 prompt = f"List 3 major {domain} events from {month}/{year}. Rate confidence 1-10."
 response = client.chat(prompt)
 return {"period": f"{year}-{month:02d}", "response": response}

Run this across a date range and you'll see a clear confidence degradation pattern - usually 3 to 6 months before the official cutoff date.

Key Takeaways

The official cutoff date overstates reliable knowledge - recent months before that date are thinly represented in training data.
Prompt-based probing can reveal the practical soft cutoff for a specific domain, which may differ from the model's general cutoff.
For any workflow touching recent events, treat anything within 6 months of the cutoff as retrieval territory, not memory territory - use RAG or live search rather than relying on the model's parametric knowledge.

When you've tested this pattern against different models or domains, did the soft cutoff land earlier or later than the official date in practice?

Sources referenced: HackerNews - Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

Fine-Tuning vs Prompt Engineering vs RAG: When to Use Each (And the Fourth Option Nobody Mentions)

Sun, 09 Aug 2026 14:33:21 +0000

You have an AI model that hallucinates with your company's data. You open a support ticket and ask the chatbot about your returns policy. The chatbot, powered by GPT-4 or Gemini 2.5, responds with a fabricated policy that sounds perfectly plausible but has nothing to do with your company's reality. Your boss stares at you. Your customer complains. You open Google and search "how to connect LLM to my data".

The top 10 results offer three contradictory answers: "use RAG," "do fine-tuning," "improve your prompt." All three are right. All three are wrong. Because the correct answer isn't any of the three in the abstract — it's the one that fits your specific use case. And there's a fourth option that almost nobody mentions, and which, in my experience, is the right call in more cases than the industry admits.

This article is the decision tree I wish I'd had when I started building the AI systems behind this blog. I've distilled it after implementing all four techniques in real production: Prompt Engineering across the entire Autopilot series, RAG in the Ops Copilot with Algolia, pure Tool Calling in the Obsolescence Radar, and experimental fine-tuning in industrial classification pipelines. It closes the trilogy that began with RAG: 7 Anti-Patterns and continued with MCP Protocol.

The Question Nobody Asks

Before choosing a technique, ask yourself this: Does the knowledge your LLM needs change, or is it static?

If the answer is "it changes frequently" (product documentation, inventory, prices, regulations), you need a technique that accesses data in real time without retraining the model. If the answer is "it's static or changes very slowly" (brand tone, formatting rules, domain nomenclature), you can consider techniques that incorporate that knowledge into the model.

This distinction is the first node of the decision tree. It seems obvious written this way. Yet most teams I've seen jump straight to whatever technique is trending (RAG in 2024, fine-tuning in 2023, prompt engineering always) without asking this fundamental question.

Option 1: Prompt Engineering — 80% of Cases

The uncomfortable truth the AI tooling industry doesn't want you to know: for 80% of use cases, a well-designed prompt is sufficient. You don't need RAG. You don't need fine-tuning. You need a system prompt that clearly defines the role, context, constraints, and expected output format.

When it's sufficient:

The required knowledge fits in the model's context window (Gemini 2.5 handles up to 1 million tokens; Claude up to 200K).
The task is generic but needs structure (drafting emails, summarizing documents, classifying text, generating code).
You don't need updated proprietary data — the model's general knowledge is enough.

Advanced techniques that make the difference:

Structured system prompts: Define the role ("You are a senior supply chain engineer"), constraints ("Always respond in technical English"), and output format ("Return a JSON with fields: analysis, recommendation, confidence").
Few-shot prompting: Include 3-5 correct input-output examples in the prompt. In the Autopilot series, CrewAI agents use few-shot to maintain style consistency across generated articles.
Chain-of-thought (CoT): Instruct the model to "think step by step" before giving the final answer. Dramatically improves accuracy in reasoning, calculation, and multi-step analysis tasks.
Prompt chaining: Break complex tasks into sequential subtasks, each with its own optimized prompt. This is exactly what CrewAI does with its agent architecture: each agent has a specialized prompt for its role.

Cost: Virtually zero (only API token cost). A well-designed prompt can take hours of iteration, but operational cost is minimal.

Fatal limitation: The context window has a limit. If you need the model to "know" about 10,000 documents from your knowledge base, you can't inject them all into the prompt. This is where RAG enters.

Option 2: RAG — Updatable Proprietary Knowledge

RAG (Retrieval-Augmented Generation) is the right answer when you need the LLM to respond about your proprietary knowledge and that knowledge updates frequently.

When it's necessary:

Product documentation, technical manuals, internal knowledge bases that update weekly or monthly.
The user can ask unpredictable questions about a broad document corpus (you don't know in advance which fragment the LLM will need).
You need citability: the answer must include the sources it draws from (critical for compliance, as we documented in the EU AI Act, Article 13 on transparency).

When NOT to use it: When data is structured (SQL tables, APIs with defined schemas) or when you need numerical precision. As I extensively documented in Anti-Pattern 7 of the RAG article, RAG over structured data generates narrative hallucinations where you need exact figures.

Correct architecture (summarized):

Semantic chunking (not fixed-length — Anti-Pattern 1)
Evaluated embeddings with your domain benchmark (Anti-Pattern 2)
Reranking between retriever and LLM (Anti-Pattern 3)
Generous context (top-10/15, not top-3 — Anti-Pattern 4)
Evaluation with RAGAS/DeepEval before production (Anti-Pattern 6)

Real cost: Moderate. The vector store (Pinecone, Weaviate, Algolia) has a monthly cost (€0-100 depending on volume), plus embedding cost (low) and generation cost (API tokens). In the Ops Copilot, the total RAG cost with Algolia was under €3/month for the blog's ~70 posts.

Real case at Datalaria: The Ops Engineering Copilot (Autopilot Part 8) uses RAG with Algolia Agent Studio to answer questions about blog content. Posts are indexed as semantic records (one record per section), and the copilot retrieves relevant fragments before generating the response. Works well for semantic search over free text.

Option 3: Fine-Tuning — The Scalpel, Not the Hammer

Fine-tuning is the most powerful technique and the most misused. It involves partially retraining a base model (Gemini, Llama, Mistral) with your own data so the model internalizes specific knowledge, style, or behavior.

When it's essential:

You need the model to adopt a very specific tone or style consistently (a brand with strict voice & tone, a domain with very particular technical jargon).
The task is highly specialized and generalist models don't solve it well even with advanced prompting (industrial defect classification, proprietary nomenclature entity extraction, specialized medical diagnosis).
You need to reduce latency and cost in production: a fine-tuned smaller model (7B-13B parameters) can match the quality of a large model (70B+) on your specific task, at a fraction of the cost and latency.

When NOT to use it (the most widespread myth):

Don't use fine-tuning to "teach the model data." Fine-tuning is not a database. If you need the model to know your product catalog, use RAG. Fine-tuning "burns in" behavioral patterns, not updatable facts.
Don't use fine-tuning if your knowledge changes frequently. Each update requires retraining, which can cost hours and hundreds of euros. RAG is instant: update the document and the retriever finds it immediately.

Modern tools:

LoRA (Low-Rank Adaptation): The standard technique. Instead of retraining the model's billions of parameters, LoRA trains only low-rank matrices "attached" to the model's layers. Reduces training cost by 90%+ and stores the fine-tuned model as a few MB of "adapters."
QLoRA: LoRA applied to a 4-bit quantized model. Enables fine-tuning 70B-parameter models on a single consumer GPU (24GB VRAM). Democratized fine-tuning for startups and teams without GPU clusters.
Vertex AI Tuning / OpenAI Fine-Tuning API: Managed services where you upload your training dataset (instruction-response pairs) and the platform runs the fine-tuning without you managing GPU infrastructure.

Real cost: Variable. Fine-tuning with LoRA on a 7B-parameter model with 10,000 examples costs €5-20 on cloud (Google Cloud, AWS). A 70B model can cost €50-200 per training session. Plus the cost of preparing the dataset (hours of human work). As we analyzed in The Hidden Economics of AI, fine-tuning's hidden cost isn't compute — it's training dataset curation.

Option 4: Tool Calling / MCP — The One Nobody Mentions

This is the option I discovered by elimination after RAG failed spectacularly on the Obsolescence Radar. Tool Calling means the LLM doesn't try to "know" the answer; instead, it knows who to ask — that is, which tool to execute to get the information with deterministic precision.

When it's the right option:

Data is structured (SQL databases, REST APIs, spreadsheets with schemas).
You need absolute numerical precision (financial calculations, inventory metrics, sensor data).
The operation requires actions, not just answers (create a ticket, send an email, run a query, call an external API).
You want to standardize connections between the LLM and tools to avoid vendor lock-in — exactly the problem solved by MCP (Model Context Protocol).

Architecture: The LLM (Gemini 2.5, Claude) acts as a semantic orchestrator: it understands the user's natural language intent, decides which tool(s) to execute, constructs the parameters, executes the tool(s), and interprets results for the user. Tools are deterministic Python functions (decorated with @tool in CrewAI) that execute precision operations: SQL queries to Supabase, supplier API calls, linear programming calculations with PuLP.

Cost: The lowest of all four options. You only pay for LLM tokens (typically few, since the prompt is short) and tool execution (SQL queries, API calls). In the Obsolescence Radar, the cost per complete execution (analyze a component, traverse the BOM graph, calculate financial impact, generate executive report) was under €0.02 per query.

Real case at Datalaria: The Agentic Obsolescence Radar uses Tool Calling exclusively. The LLM (Gemini 2.5 via CrewAI) understands the obsolescence alert in natural language, but all data operations — SQL query to the component catalog, BOM graph traversal, P&L calculation, PDF generation — are executed by deterministic Python tools. Result: executive reports in 4 seconds with 0% numerical hallucination.

The Decision Matrix

Criterion	Prompt Engineering	RAG	Fine-Tuning	Tool Calling
Initial cost	⭐ Minimal	⭐⭐ Low-medium	⭐⭐⭐ High	⭐⭐ Low
Operational cost	⭐ Low	⭐⭐ Medium	⭐ Low (small model)	⭐ Minimal
Precision (free text)	⭐⭐ Medium	⭐⭐⭐ High	⭐⭐⭐ Very high	⭐ N/A
Precision (structured data)	⭐ Low	⭐ Low	⭐ Low	⭐⭐⭐ Exact
Data freshness	⭐⭐⭐ Instant	⭐⭐⭐ Instant	⭐ Requires retraining	⭐⭐⭐ Real-time
Implementation effort	⭐ Hours	⭐⭐ Days-weeks	⭐⭐⭐ Weeks-months	⭐⭐ Days
Maintenance	⭐ Minimal	⭐⭐ Medium	⭐⭐⭐ High (data drift)	⭐⭐ Medium
Traceability (EU AI Act)	⭐ Difficult	⭐⭐⭐ High (citable sources)	⭐ Opaque (black box)	⭐⭐⭐ Total (deterministic)
Ideal use case	Generic tasks with clear instructions	Updatable proprietary text knowledge	Specific style/tone, ultra-specialized tasks	Structured data, numerical precision, actions

The 3-Question Framework

If the matrix seems dense, I've distilled a 3-question framework that resolves 90% of decisions:

Question 1: Does the data the LLM needs fit in the prompt?

Yes → Prompt Engineering. Inject context directly. Simpler, cheaper, faster.
No → Next question.

Question 2: Is the data free text or structured?

Free text (documentation, manuals, posts) → RAG. Semantic retrieval is superior for searching unstructured text.
Structured (SQL, APIs, tables, calculations) → Tool Calling. Deterministic tools that execute exact queries.
Both → Hybrid architecture (RAG for textual context + Tool Calling for structured data, as we proposed in the RAG article).

Question 3: Do you need a behavior or style the base model can't reproduce even with the best prompt?

Yes (unique brand tone, ultra-specific domain jargon, task that generalist models consistently fail) → Fine-Tuning on a base model.
No → Go back to Prompt Engineering and refine your prompt before considering more complex techniques.

What I Learned Implementing All Four

The most valuable lesson from operating these four techniques in production fits in a single sentence: always start with the simplest technique that could work.

The temptation is to jump straight to RAG or fine-tuning because they're more "sophisticated." But sophistication doesn't correlate with effectiveness. In the Autopilot, the majority of output quality comes from Prompt Engineering — carefully designed system prompts, few-shot examples, and Chain-of-thought. RAG added marginal value in the Ops Copilot for blog search. Fine-tuning wasn't necessary in any case. And Tool Calling was the transformative technique in the Obsolescence Radar, where RAG had failed.

The evaluation order should always be:

Prompt Engineering (hours, ~€0)
Tool Calling if data is structured (days, ~€0)
RAG if you need access to proprietary text (days-weeks, ~€3-50/month)
Fine-Tuning only if the previous three consistently fail (weeks, €50-500+)

And in the era of the EU AI Act, there's a fifth consideration that isn't technical but legal: traceability. Article 10 of the regulation requires that training data for high-risk systems be "relevant, representative, and to the extent possible, free of errors and complete." This applies directly to fine-tuning: if you fine-tune a model with biased or incorrect data, and that model makes decisions in a regulated domain, you're exposed to sanctions. RAG and Tool Calling, being transparent in their sources, offer traceability that fine-tuning cannot match.

As we wrote in The Hidden Economics of AI, the 10x Rule applies: if a more complex technique doesn't give you a result 10 times better than the previous one, it probably doesn't justify its added cost and complexity. Start simple. Measure. Scale only when the data demands it.

Sources of Interest:

Building Production LLM Agents That Actually Ship: Lessons from 10+ Years in the Trenches

Fri, 07 Aug 2026 03:15:50 +0000

Building Production LLM Agents That Actually Ship: Lessons from 10+ Years in the Trenches

Tool-calling agents, RAG pipelines, and the automation stack that turns "AI demo" into "AI in production" — and why I'm opening up for new projects.

The gap nobody talks about

Most AI content online is about getting a model to respond well in a chat window. The harder, less glamorous problem is getting that same model to reliably book an appointment, update a CRM record, or hand off to a human at exactly the right moment — in production, at 2am, when an API times out or a timezone field silently defaults to the wrong value.

That gap — between "cool demo" and "system a business can depend on" — is where I've spent most of my career, and it's the thread that connects everything I build now.

What "shipped" looks like

A few examples from recent and past work:

RAG systems that refuse to guess. I built a health-data RAG pipeline with embeddings, cosine retrieval, inline citations, and — critically — honest refusal when the retrieved context doesn't support an answer, plus a full audit log. In healthcare-adjacent domains, a system that says "I don't know" is more valuable than one that sounds confident and is wrong.
→ health-data-rag.vercel.app · source

Voice + SMS agents wired into real CRM systems. Built and debugged a full lead follow-up and appointment-booking system combining GoHighLevel, Retell AI (voice/chat), and n8n for a multi-location client. The interesting engineering wasn't the AI — it was the plumbing: explicit timezone configuration per calendar (never inherit it), Luxon expressions in n8n's startTime field, correctly distinguishing a status === "booked" response from a missing statusCode field, and drawing a hard architectural line so voice-generation and SMS-sending never leak into each other. Small details like these are the actual difference between a bot that works and one that quietly fails a week after launch.

Automation across the LLM app surface. From FastAPI backends and Next.js frontends to n8n/Make.com workflows and multi-agent orchestration, I own the full chain: agent logic, retrieval, inference APIs, and the interface around them — not just the prompt.

Open source, not just employer work. Contributing to Inglorious Forge — SSX compatibility testing, a merged ElementSize entity, an error boundary with passing unit tests, a before/after slider component. Reading someone else's production codebase closely enough to contribute cleanly is its own skill, and I think it shows in how I approach client handoffs.

The stack I reach for

Python FastAPI TypeScript Next.js Supabase/PostgreSQL (pgvector, RLS) n8n Make.com GoHighLevel OpenAI API Anthropic API Retell AI Vercel

What I'm looking for

I'm currently taking on new freelance projects — particularly:

AI agent / chatbot systems wired into real business tools (CRMs, calendars, ticketing)
RAG pipelines that need to be trustworthy, not just impressive in a demo
n8n / Make.com automation connecting AI to existing workflows
Integration debugging — if your AI feature "mostly works" and you can't figure out the 10% that doesn't, that's often exactly my kind of problem

If any of that matches something you're building, I'd genuinely like to hear about it.

Luis Cruz
Senior AI Developer — LLM Agents, RAG, Automation
📧 Emails are not allowed
🔗 topstar-ai-github-io.vercel.app · GitHub · Contra

How Open Embedding Models Can Match GPT-Class Retrieval Quality

Thu, 06 Aug 2026 09:09:21 +0000

RAG (Retrieval-Augmented Generation - a pattern where you fetch relevant documents before generating an answer) lives or dies on retrieval quality. A cheaper retrieval stack isn't always a compromise.

The Retrieval Quality Gap Is Often in the Chunking, Not the Model

Most teams assume the embedding model is the bottleneck when retrieval underperforms. The real culprit is usually upstream: how documents are split before they're embedded. Overlapping chunks with a recursive character splitter, combined with a well-tuned open embedding model like bge-large-en-v1.5 or nomic-embed-text, can close a surprising amount of the gap against expensive proprietary embeddings - often without touching the LLM layer at all.

The pattern that tends to work: chunk at a meaningful boundary (paragraph or sentence, not fixed character count), add a small overlap window (10 - 15% of chunk size), then embed with a model that scores well on the MTEB benchmark (Massive Text Embedding Benchmark - a standardized leaderboard for comparing retrieval and semantic similarity models). Many open models in the 1B parameter range sit competitively on MTEB against models costing 100x more per token to call via API.

Real Example

Here's a minimal retrieval setup using LangChain with a local open embedding model via Ollama:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, chunk_overlap=64, separators=["\n\n", "\n", ". "]
)
docs = splitter.create_documents([your_text])

embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(docs, embeddings)

results = vectorstore.similarity_search("your query here", k=4)

Swapping nomic-embed-text for bge-large-en-v1.5 (via HuggingFace) takes one line change. Running both against your actual dataset on a held-out set of queries - not a synthetic benchmark - is the fastest way to know which one fits your domain.

The cost difference is real: embedding a million tokens locally via Ollama is effectively free after the initial model pull, versus measurable per-call costs at API scale.

Key Takeaways

Chunking strategy (boundary type + overlap) affects retrieval quality at least as much as the embedding model choice - fix this before swapping models.
Open embedding models in the 1B range score competitively on MTEB and run locally or on modest GPU instances.
Evaluate retrieval quality on your own queries, not just published benchmarks - domain-specific retrieval patterns differ from general benchmarks.

If you've swapped embedding models in a production RAG pipeline, which metric did you actually use to confirm the quality change - cosine similarity threshold, end-task accuracy, or something else?

Sources referenced: HackerNews discussion - "Beating GPT-5.6 Sol on retrieval with 100x cheaper open models", MTEB Leaderboard (Hugging Face)

Senior AI Engineer Available for Freelance, Consulting & Long-Term Projects

Wed, 05 Aug 2026 20:26:54 +0000

Over the past few years, I've been building production-grade AI applications, full-stack web platforms, and intelligent automation systems for startups and businesses across different industries.

My expertise includes:

🤖 LLM applications, AI agents, and RAG systems
⚡ AI workflow automation (n8n, APIs, MCP, integrations)
🐍 Python, FastAPI, Next.js, React, TypeScript
🧠 Multi-agent architectures and structured AI pipelines
☁️ Cloud deployment, scalable backend systems, and DevOps
🔗 Third-party integrations, payment systems, CRMs, and business automation

Some of the projects I've delivered include AI-powered assistants, healthcare platforms, automation systems, digital products, travel platforms, marketing websites, and custom business solutions.

I focus on building systems that are:

Reliable in production
Scalable and maintainable
Performance-oriented
Designed around real business outcomes—not just prototypes

Whether you need to build an AI product from scratch, integrate LLM capabilities into an existing application, automate business workflows, or modernize a web platform, I can help turn ideas into production-ready solutions.

If you're looking for an experienced engineer for freelance work, contract projects, technical consulting, or a long-term collaboration, I'd be happy to connect.

📩 Feel free to reach out if you have an interesting project or are looking for a technical partner.

Portfolio Demo: https://www.loom.com/share/dd792a043d6b4d4483adba463bbe02f3

Your RAG copilot can't count — stop letting it try

Fri, 31 Jul 2026 11:05:28 +0000

Your RAG copilot can't count — stop letting it try

A user asked our document-search copilot a completely reasonable question: how many documents a specific person authored. The copilot answered with a number. Confidently. With a nice sentence around it.

The real answer, computed by the database, was 84. The copilot said — because it did what every RAG system does when you ask it an aggregation question: it counted what it could see.

Here's the claim I want to defend, because I think a lot of us are shipping this bug right now:

💡 An LLM should never compute an aggregate. Not counts, not sums, not "most recent". If the question is arithmetic over the corpus, the answer comes from the system of record — the model's only job is to phrase it.

🔢 Why the model's count was doomed before it started

This is a multi-agent document-search copilot for a quality-management product, built on LangGraph and Bedrock. When a metadata question comes in ("documents authored by X"), a search agent asks the documents service for matching rows, reranks and permission-gates them, collapses multiple versions of the same document into one card, and surfaces at most 30 results.

Every one of those steps is correct for search. Every one of them destroys the count:

Stage	What it does	What "how many" means here
Documents service (SQL)	runs the filter against the database	true total: 84
Retrieval	returns the top-k rows	capped
View-permission gate	drops rows this user can't see	fewer
Version collapse	5 versions of one doc → 1 card	fewer still
Surfaced set	at most 30 cards	≤ 30, always

The model sits at the bottom of that funnel. When it "counts", it counts the last row of the table. The user asked about the first row. Everything in between is retrieval hygiene that silently turned a database question into a context-window question.

And this is the part that makes it worse than a crash: the answer is plausible. A count that's capped at 30 and deduped looks like a real number. Nobody squints at "you have 27 documents" the way they'd squint at a stack trace. In a quality-management context, someone might put that number in a report.

🪤 The twist: we had the right number the whole time — and overwrote it

This is the bit that stung. The documents service returns the true total with every filtered query. The search node even stored it in agent state, in a field called total_available_results.

Then, a few nodes downstream, the permission-gate node overwrote that same field with the surfaced count — deliberately, and for a good reason:

return {
    "sources": diversified,
    # Honest "documents found": distinct logical documents after the view gate and
    # version-collapse, so the headline count matches the cards (5 versions → 4 docs),
    # not the service's version-row total.
    "total_available_results": len(diversified_all),
    ...
}

That comment is right! If the UI says "found 12" and shows 12 cards, the numbers should agree. The bug wasn't either write in isolation — it was one state field carrying two different meanings at two different points in the pipeline. Upstream it meant "matches in the database"; downstream it meant "cards on screen". Whoever read it last got whichever meaning was freshest.

💡 A state field whose meaning mutates mid-pipeline is a bug factory. If a value is authoritative, give it its own field and never let anything downstream touch it.

🔧 The fix: the database counts, the model narrates

So that's exactly what I did — preserved the service's true total in its own state field, threaded it to the two places that answer the user, and left the surfaced count alone for the card list and pager.

First, the state gets an untouchable field:

class DocumentsSearchState(OrchestratorState, total=False):
    ...
    total_available_results: int   # surfaced: view-gated, version-collapsed, capped at 30

    # The documents service's true match total for the filter, before the 30-cap and
    # version-collapse. Authoritative for a "how many" count; 0 on a content-only
    # search, which has no exact total.
    total_matching_records: int

Then the outcome node — the one that turns search state into both the user-facing status line and the message the agent reasons over — uses the authoritative number in both:

total_matching = state.get("total_matching_records") or 0
headline_total = total_matching if total_matching > total else total

found_msg = (
    f"Showing {len(sources)} of {headline_total} matching documents"
    if headline_total > len(sources)
    else f"Found {len(sources)} matching documents"
)
...
count_note = (
    f" {total_matching} documents match in total "
    "(authoritative count; use this for a count question)."
    if total_matching > len(sources)
    else ""
)
return f"Returned {len(sources)} document cards to the user.{count_note} ..."

And the agent's prompt closes the loop with an explicit instruction:

For a "how many" question, answer with the authoritative total the tool
reports ("N documents match in total"), never the number of cards shown,
which is capped.

Notice the belt-and-braces structure, because it matters: the number goes into the agent's context (so the model can phrase the answer around it) and into the user-facing headline ("Showing 5 of 84 matching documents"). Even on a turn where the model ignores the note and free-styles a count, the UI is showing the true total right next to it. The prompt asks; the headline doesn't have to.

Three small tests pin the behavior: the count note appears when there are more matches than cards, disappears when everything is shown, and never appears for a content-only semantic search — because a similarity ranking has no crisp "matches" set, so pretending it has an exact total would be the same lie in a new costume.

📅 The sibling bug: "the last 10 days" of what?

Same release, same root cause wearing a different hat. Users asked for "documents created in the last 10 days" and the copilot had to turn that into a date filter — without knowing what day it is. A model doesn't know today's date. It will happily guess one from its training data, which is exactly as wrong as counting its context window.

The fix is a tiny LangGraph middleware that injects the current date into the system message on every model call:

def current_date_context(now: datetime | None = None) -> str:
    day = (now or datetime.now(UTC)).date()
    return (
        f"Today's date is {day.isoformat()} (UTC). Resolve any relative date "
        '("today", "yesterday", "the last 10 days", "since March") '
        "to concrete ISO-8601 dates relative to it before building a date filter."
    )


class CurrentDateMiddleware(AgentMiddleware):
    async def awrap_model_call(self, request, handler):
        base = request.system_message
        line = current_date_context()
        merged = line if base is None else f"{base.content}\n\n{line}"
        return await handler(request.override(system_message=SystemMessage(content=merged)))

One non-obvious detail: it's injected per call and never appended to the persisted conversation. Write "today is 2026-07-30" into the thread history and a conversation resumed next week carries a stale date the model will trust completely.

Both fixes are the same principle: the model knows nothing you don't hand it, and it will answer anyway. Counts, dates, totals — anything that's a fact about your system rather than about language has to arrive as context, computed by something that can't hallucinate.

🤔 Where I'd push back on myself

Honest limits of what shipped:

Nothing enforces that the model uses the number. The prompt instructs, the tool message supplies, the headline backstops — but there's no output check asserting the count in the model's prose equals the authoritative one. That's an eval I'd like to have and don't yet.
Content-only searches still have no real count. A semantic ranking over chunks genuinely doesn't have an exact match total, so those turns fall back to the surfaced count. Arguably the honest move there is to refuse to give a number at all. I didn't go that far.
This only covers counts. Sums, averages, "which author has the most" — any of those would need real aggregation endpoints on the service side. The pattern extends; the implementation doesn't yet.

And the design choice I keep turning over: I handed the model the number inline, as prose in a tool result. The agentic-purist alternative is a dedicated count_documents tool the model calls when it detects an aggregation question — cleaner separation, one more round trip, and a new failure mode where the model doesn't call it. I went with inline because the search already paid for the count; a tool call felt like latency for ideology.

So: where do you draw that line — hand the model the number, or hand it a tool to fetch the number? If your copilot answers "how many" questions today, I'd genuinely like to know which side you picked and whether it's held up.

Thanks for making it to the end 🙏 If your assistant has ever announced a total that made you go "wait, that can't be right", I'd love to swap notes on how you fixed it — come find me on LinkedIn.

From RAG to Agentic RAG. How I Added LangGraph to My Local

Thu, 30 Jul 2026 19:49:17 +0000

In my previous article, I built a fully local RAG assistant Ollama, ChromaDB, LangChain, all running in Docker. It answered technical support questions by searching through documentation and citing sources. It worked.

But after using it for a while, I noticed something uncomfortable: it treated every question the same way.

Ask it "how to close monthly payroll?" it searches the docs. Fine.
Ask it "the server crashes at startup" it also searches the docs. Less fine.
Ask it something completely outside the documentation it searches the docs. Useless.

A real support technician doesn't do that. They first assess the situation, then decide what to do: look it up, run a diagnosis, or escalate to a human. My RAG had no such judgment.

That's what this article is about how I evolved the system into an Agentic AI architecture using LangGraph, where the assistant first decides which strategy to use, then acts accordingly.

The Core Limitation of Classic RAG

Classic RAG is a linear pipeline. Every query follows the exact same path:

Question → Embed → Retrieve → Prompt → LLM → Answer

No branching. No decision-making. No memory between steps.

This works perfectly for procedural questions where the answer lives in the docs. But technical support involves at least three distinct scenarios:

Scenario	Example	Best strategy
Procedural question	"How do I create an account?"	Search documentation
Known error code	"ERR-COMP-001 appears"	Lookup error database
Unknown incident	"Server crashes, no idea why"	Diagnose + escalate if needed

A single RAG pipeline handles the first case well and the other two poorly. The solution is to add a layer of reasoning before retrieval.

What Agentic AI Adds

The shift from RAG to Agentic AI comes down to one thing: the system plans before it acts.

Instead of one fixed pipeline, you have:

Question
    ↓
Classifier (what kind of question is this?)
    ↓
    ├── Procedural → RAG Agent (search docs)
    ├── Error code → Diagnostic Agent (lookup + LLM analysis)
    └── Complex    → Diagnostic Agent → Escalation Agent (generate ticket)

Each branch is an agent; an autonomous function that receives a shared state, does its job, updates the state, and returns it. LangGraph connects these agents into a directed graph with conditional routing between them.

The Shared State Foundation of LangGraph

Before building agents, you define the state they all share. Think of it as a case file that every agent reads and updates:

# src/agents/state.py

from typing import TypedDict, List, Annotated, Sequence
from langchain_core.messages import BaseMessage
import operator

class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], operator.add]
    question: str          # Original question never changes
    plan: List[str]
    current_step: int
    past_steps: List[str]  # Audit trail of what each agent did
    response: str
    next_agent: str
    niveau_support: int    # 1=resolved, 2=partial, 3=needs human

The niveau_support field is key it's how agents communicate urgency to each other. If an agent sets it to 3, the orchestrator escalates automatically.

Annotated[Sequence[BaseMessage], operator.add] means messages accumulate — each agent appends rather than overwrites. You get a full conversation trace at the end.

Agent 1 The RAG Agent

The pipeline from my previous article, repackaged as an autonomous agent:

# src/agents/rag_agent.py

def rag_agent(state: AgentState) -> AgentState:
    """Searches the documentation and generates a grounded answer."""

    docs = vectorstore.similarity_search(state["question"], k=3)

    if not docs:
        return {
            **state,
            "messages": list(state["messages"]) + [
                AIMessage(content="[RAG] No relevant document found.")
            ],
            "past_steps": state["past_steps"] + ["RAG: no results"],
            "niveau_support": 3   # Nothing found — escalate
        }

    context = "\n\n".join([
        f"[Source: {d.metadata.get('module', '?')}]\n{d.page_content}"
        for d in docs
    ])

    answer = llm.invoke(
        f"Answer from context only.\nContext: {context}\nQuestion: {state['question']}"
    )

    return {
        **state,
        "messages": list(state["messages"]) + [AIMessage(content=f"[RAG] {answer}")],
        "past_steps": state["past_steps"] + ["RAG: documentation search completed"],
        "niveau_support": 1
    }

The critical difference from the classic RAG: the agent returns an updated state, not just an answer. Every agent follows this same pattern — receive state, do work, return updated state.

Agent 2 The Diagnostic Agent

Handles error codes and technical incidents with two modes:

# src/agents/diagnostic_agent.py

KNOWN_ERRORS = {
    "ERR-COMP-001": {
        "description": "Debit/credit imbalance",
        "solution": "Check Total Debit = Total Credit. Go to Accounting > Journal Entry."
    },
    "ERR-PAIE-042": {
        "description": "Payroll line missing rate",
        "solution": "Go to Settings > Payroll Lines, fill in the missing rate."
    },
}

def diagnostic_agent(state: AgentState) -> AgentState:
    question = state["question"]
    found_codes = [code for code in KNOWN_ERRORS if code.lower() in question.lower()]

    if found_codes:
        # Mode 1 : known error: instant answer, no LLM call needed
        responses = [
            f"Code {code}: {KNOWN_ERRORS[code]['description']}\n"
            f"Solution: {KNOWN_ERRORS[code]['solution']}"
            for code in found_codes
        ]
        return {
            **state,
            "messages": list(state["messages"]) + [
                AIMessage(content=f"[Diagnostic] {chr(10).join(responses)}")
            ],
            "past_steps": state["past_steps"] + ["Diagnostic: known error, resolved"],
            "niveau_support": 1
        }

    # Mode 2 : unknown error: LLM analysis
    answer = llm.invoke(
        f"Analyze this technical issue. If unsure, say so.\nIssue: {question}"
    )
    return {
        **state,
        "messages": list(state["messages"]) + [AIMessage(content=f"[Diagnostic] {answer}")],
        "past_steps": state["past_steps"] + ["Diagnostic: LLM analysis, uncertain"],
        "niveau_support": 2
    }

The known error lookup runs in milliseconds without touching the LLM; a meaningful performance gain for common incidents.

Agent 3 : The Escalation Agent

Called only when niveau_support >= 3. It doesn't try to answer; it generates a structured ticket with the full diagnostic history already attached:

# src/agents/escalade_agent.py

def escalade_agent(state: AgentState) -> AgentState:
    ticket = f"""
=== SUPPORT TICKET — LEVEL 3 ===
Date     : {datetime.now().strftime("%Y-%m-%d %H:%M")}
Priority : HIGH

ISSUE:
{state["question"]}

DIAGNOSTIC STEPS ALREADY TAKEN:
{chr(10).join(state["past_steps"])}

CONCLUSION: Human intervention required.
================================"""

    return {
        **state,
        "messages": list(state["messages"]) + [AIMessage(content=ticket)],
        "past_steps": state["past_steps"] + ["Escalation: level 3 ticket generated"],
        "niveau_support": 3
    }

This is what separates a real support system from a chatbot; it knows when to stop and hand off, with context already attached so the user doesn't have to repeat themselves.

The Orchestrator : Wiring It All Together

# src/agents/orchestrator.py

def classifier(state: AgentState) -> str:
    question = state["question"].lower()

    if re.search(r'err-\w+-\d+', question):
        return "diagnostic"

    problem_words = ["error", "crash", "bug", "not working", "fails", "problem"]
    if any(word in question for word in problem_words):
        return "diagnostic"

    return "rag"  # Default for procedural questions


def should_escalate(state: AgentState) -> str:
    return "escalade" if state.get("niveau_support", 1) >= 3 else END


def create_orchestrator():
    workflow = StateGraph(AgentState)

    workflow.add_node("rag_agent", rag_agent)
    workflow.add_node("diagnostic_agent", diagnostic_agent)
    workflow.add_node("escalade_agent", escalade_agent)

    workflow.set_conditional_entry_point(
        classifier,
        {"rag": "rag_agent", "diagnostic": "diagnostic_agent"}
    )

    workflow.add_edge("rag_agent", END)
    workflow.add_conditional_edges(
        "diagnostic_agent",
        should_escalate,
        {"escalade": "escalade_agent", END: END}
    )
    workflow.add_edge("escalade_agent", END)

    return workflow.compile()

Seeing It In Action

Three questions, three different paths:

Procedural question

Input  : "How to close monthly payroll?"
Path   : classifier → rag_agent → END
Output : agents_used: ["RAG: documentation search completed"]
         niveau: 1

Known error code

Input  : "ERR-COMP-001 appears on journal validation"
Path   : classifier → diagnostic_agent → END
Output : agents_used: ["Diagnostic: known error, resolved"]
         niveau: 1  (no LLM call, instant response)

Unknown incident

Input  : "Server crashes at startup, no idea why"
Path   : classifier → diagnostic_agent → escalade_agent → END
Output : agents_used: ["Diagnostic: LLM analysis", "Escalation: ticket generated"]
         niveau: 3

What I Would Do Differently

The classifier is fragile. Keyword matching breaks on edge cases "I don't have a problem, just need help configuring payroll" contains "problem" and routes to diagnostic instead of RAG. A small LLM classification call, or a fine-tuned classifier, would be more robust.

Agents reload the embedding model on every call. Instantiating HuggingFaceEmbeddings on each agent invocation adds unnecessary overhead. Initialize once at startup and pass the instance through the state.

The escalation threshold is hardcoded. A feedback loop where user ratings influence the confidence threshold over time would make this genuinely adaptive that's what reinforcement learning adds in the next phase of this project.

RAG vs Agentic AI : When to Use Which

Criteria	Classic RAG	Agentic AI
Best for	Uniform question types	Mixed question types
Complexity	Low	Medium-High
Latency	One LLM call	One or more LLM calls
Flexibility	Fixed pipeline	Extensible — add agents without rewriting
When to choose	Start here	When RAG fails on edge cases

My honest take: start with RAG. Get it working, understand its failure modes, then add agents where you actually need branching logic. Adding LangGraph too early is over-engineering.

Key Takeaways

Design your AgentState first. It's the contract between all components. Changing it mid-project means updating every agent get it right upfront.

Agents should do one thing. The moment an agent tries to retrieve and diagnose, it becomes untestable and unroutable. Keep them single-responsibility.

past_steps is your audit trail. When a ticket escalates to a human, they see exactly what the system tried. Small design choice, professional result.

The classifier is your most important and most brittle piece of code. Test it with as many edge cases as you can before trusting it.

These two articles cover Phases 1 and 2 of my internship project. The logical next step is a reinforcement learning feedback loop; user ratings adjusting the classifier's routing decisions over time; which I may write about once it's implemented and tested.

I'm a Master's student in AI & Big Data, building these systems with open-source tools and no GPU. If you're working on multi-agent systems or ran into similar design decisions; especially around the classifier or state design; I'd love to hear your approach in the comments.

Technology is now for anyone

Thu, 30 Jul 2026 04:58:14 +0000

Mental Health System

AI Chatbot that provides empathetic responses for mental health support using simple Plain English text.

Live: https://lnkd.in/dn5H6PM9

GraphRAG Won’t Give Your AI a Brain. Here’s What Twelve Papers Actually Say.

Thu, 30 Jul 2026 04:52:44 +0000

💡TL;DR

▪️GraphRAG helps most when questions require multi-document relationships, temporal reasoning, or corpus-level synthesis; it does not consistently beat standard RAG on direct factual retrieval.
️️▪️Automatically constructed graphs can omit required entities, weaken abstention, and create new maintenance, provenance, and security risks.
▪️ Treat the graph as a navigational layer over source evidence - not as an AI brain, a source of truth, or a substitute for runtime control.

When Retrieval Infrastructure Became an “AI Brain”

Three GraphRAG posts showed up on my feed today. Not three this week. Three today. Each used some version of the same promise: give an AI agent a knowledge graph, and it will finally understand your organization.

The surrounding claims were broader. One repository mapped an entire codebase into a glowing web of functions and dependencies. Another promised organizational memory that would persist across agents and sessions. Obsidian vaults were becoming self-growing second brains. Vector databases were being pronounced obsolete, sometimes by the same companies that had been promoting them six months earlier.

The phrase AI brain now gets attached to technologies that share surprisingly little beyond the presence of nodes, links, or Markdown files. There are graphs extracted from natural-language documents, formal ontologies maintained by domain specialists, syntax trees exposed to coding assistants, temporal memory systems that record when facts stopped being true, and agent-maintained wikis where a model writes notes for its future self.

Calling all of these a knowledge map is roughly as useful as grouping a bicycle, a forklift, and a jet engine under ways to move things. The category is technically defensible and nearly useless when deciding what to build.

The excitement is not entirely manufactured. Language models lose project history, reread the same files, miss relationships scattered across documents, and waste enormous context windows reconstructing structures that parsers already know. Something more structured could help. The problem is that help has quietly expanded into understanding, and retrieval infrastructure has become a brain before the benchmark results have earned either word.

So I pulled the comparative studies, benchmark appendices, original GraphRAG work, recent agentic-search results, and the technical documentation behind the most visible repositories. Not the launch posts or demo videos. The sources and the numbers underneath them.

The evidence supports a considerably narrower claim:

GraphRAG can be valuable when a question requires relationships to be traversed across documents, temporal or comparative reasoning, or a broad summary of a large corpus.

It does not reliably beat simpler retrieval on straightforward factual questions, and some graph-based designs become substantially worse when the evidence is incomplete.***

Round 1: Does a graph help you find a fact?

One of the most useful comparisons is Han et al.’s systematic evaluation of RAG and GraphRAG. The researchers standardized preprocessing, retrieval settings, and generation conditions across ordinary RAG and several graph-based systems instead of letting every architecture arrive with its own preferred benchmark setup. 1

Using Llama 3.1–70B, they reported:

Plain retrieval beat every graph-based method on Natural Questions. On HotpotQA, the graph systems moved ahead, although the difference was less than one point.

This is not the comparison GraphRAG demonstrations usually foreground. Demonstrations tend to begin with questions graphs were designed to handle: How is this person connected to that project? What sequence of events led to this decision? Which modules depend on this function?

Real users also ask for an exact clause, one number from a report, the current owner of a system, or the sentence containing an exception.

A direct question is often best answered by a passage that still contains its original wording, scope, qualifications, and awkward “except when” condition. Converting that passage into an entity and an edge may preserve the relationship while losing the language that determines whether the relationship applies.

You get the relationship. You lose the sentence.

Round 2: Where the graph earns its cost

The result changes when the question is deliberately constructed to require evidence from several documents.

On MultiHop-RAG, the same study found:

This is a meaningful gain. Graph structure helped when the model had to connect entities, compare facts, reconstruct temporal order, or combine evidence unlikely to appear together in one similarity search.

Figure 1. GraphRAG’s advantage depends on the task.

A separate 2026 study, Do We Still Need GraphRAG?, asked whether an agent capable of decomposing a question and searching repeatedly could recover the benefits of explicit graph structure.

Agentic search strengthened dense RAG and narrowed parts of the gap. GraphRAG retained an advantage on complex multi-hop problems and showed more stable retrieval behaviour when its offline construction cost could be amortized. 4

Its detailed results make the task boundary difficult to ignore:

Average improvement on general QA:

+0.47 points

Average improvement on selected multi-hop QA benchmarks:

+27.23 points

This does not support a general replacement story. It shows that explicit structure becomes more valuable as the answer depends on relationships that ordinary retrieval would otherwise have to rediscover through repeated searches.

The useful question is how much relational work exists in the actual workload, and whether that work repays the graph’s construction, maintenance, and information loss.

A company whose users mostly ask for quotations, figures, and current policy clauses may spend heavily to build a system optimized for questions it rarely receives.

Round 3: The number that rarely appears in a demo

The most consequential result concerns questions that the source corpus cannot answer.

Han et al. included “Null” questions designed to test whether a system recognized that the available evidence was insufficient.

The score collapsed from 91.36 to 13.95.

This should not be translated casually into “Global GraphRAG hallucinated confidently 86 percent of the time.” The benchmark score does not establish that every failure was a confident fabricated answer.

It establishes something serious enough without embellishment: the global system repeatedly lost the ability to recognize that its evidence did not support an answer.

The mechanism is understandable. Microsoft’s original GraphRAG design was built in part for global sensemaking over large corpora. It extracts entities, detects communities, produces community summaries, and uses those abstractions to answer broad questions about themes and relationships. 2

Those summaries can cover far more of a corpus than a handful of retrieved chunks. They can also remove the precise detail needed to determine that a particular question is unanswerable. The system receives enough thematic material to sound informed, while the sentence, date, qualification, or missing connection that should force abstention is no longer visible.

Two different properties are being confused:

Comprehensiveness measures how much relevant territory an answer covers.
Evidence sufficiency asks whether the available material actually supports the claim being made.

A system can improve the first while damaging the second.

That trade-off becomes more serious once the model is an agent. A broad but unsupported conversational answer is undesirable. A broad but unsupported plan that leads to a code modification, database update, email, payment, or deployment belongs to a different risk class, and retrieval quality alone does not tell us whether the agent will cross that boundary.

Your graph may be missing a third of the answers before you query it

Graph construction is often presented as preprocessing, as though the source corpus were simply being rearranged into a more useful shape.

It is closer to prediction.

Han et al. checked whether the required answer entity existed anywhere in the automatically generated graph. Coverage reached:

65.8% on HotpotQA
65.5% on Natural Questions

Roughly one in three required answer entities never entered the graph.

No retrieval algorithm, graph traversal policy, reranker, or agentic search loop can walk to a node that was never created. The retriever may still find neighbouring concepts and produce an answer that appears structurally grounded, but its searchable world was already narrowed during extraction.

Failures enter through ordinary, unglamorous mechanisms. An entity is missed. Two entities are merged because their names look similar. A relationship loses its date range. An exception disappears. The direction of an edge is reversed.

A hypothetical statement is stored as an assertion. An inferred connection survives without enough source evidence to justify it. None of these failures needs to crash the pipeline; most leave behind a graph that still renders cleanly and accepts queries.

The model used to build the graph also changed final performance:

The retrieval design remained comparable, but replacing the extraction model produced almost a four-point difference between the two graph configurations.

The uncertainty did not disappear when the system moved to a graph. Part of it moved upstream, into a stage that organizations may run once, trust implicitly, and reuse across thousands of later queries.

Microsoft’s own GraphRAG repository and documentation warn that indexing can be expensive, recommend starting small, explain that default prompts may not produce optimal results, and describe the code as a demonstration rather than an officially supported Microsoft offering. 7

That is a more sober description than most third-party launch posts.

The cost table nobody leads with

In the controlled MultiHop-RAG comparison, construction and retrieval time varied sharply by architecture:

“GraphRAG is slow” and “GraphRAG is fast” are both incomplete claims.

Community GraphRAG paid a heavy upfront construction cost and then retrieved faster than standard RAG across the measured run. That may be a reasonable trade for a stable corpus queried repeatedly.

KG-GraphRAG performed graph expansion during retrieval and accumulated more than eight times the total retrieval time of standard RAG in that experiment.

The operational result depends on where the cost is paid.

A static archive may absorb expensive preprocessing once. A changing repository, policy library, product catalogue, or incident log may trigger continuous graph repair.

Entities must be reconciled, edges updated, communities recalculated, summaries regenerated, and derived indexes refreshed. When incremental updating and invalidation are unreliable, the system either rebuilds too much or tolerates stale structure.

Neither option appears in the first screenshot.

The maintenance burden also changes what should count as success. A five-point benchmark gain may be attractive when the corpus is stable for a year. The same gain can disappear operationally if the graph is rebuilt every night, arrives several hours behind the source, or requires a stronger extraction model whose indexing bill overwhelms the savings produced during retrieval.

What happens when you combine RAG and GraphRAG?

The obvious response to task sensitivity is hybrid retrieval.

Simple questions can go to standard RAG. Relationship-heavy questions can go to GraphRAG. A third option runs both and merges the evidence.

Han et al. called these approaches Selection and Integration.

With Llama 3.1–70B on MultiHop-RAG, the results were encouraging:

Selection improved on the strongest single baseline by 1.1%.
Integration improved it by 6.4%.
Integration reached an overall score of 77.62.

Then the researchers repeated the experiment with Llama 3.1–8B.

The smaller model received more retrieved material and became dramatically worse at recognizing when that material was insufficient. Its overall score did not improve over GraphRAG alone, while the insufficiency score fell from 96.01 under standard RAG to 50.17 under integration.

More retrieval does not simply add information. It also adds repeated fragments, graph abstractions, partial contradictions, and linguistic pressure to produce a synthesis.

Larger models may manage that mixture better. Smaller ones may interpret the volume of related context as evidence that an answer must exist, especially when several retrieved objects describe the same neighbourhood without supplying the missing fact.

A later paper, Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization, compared nine standardized RAG scenarios and reported token reductions of 19–53 percent through improved context management. Its results also suggest that retrieval-side gains do not always translate into proportional improvements in final generation quality.

I use retrieval–generation gap here as analytical shorthand for that mismatch; it is not presented as the paper’s own formal term. 5

This is the finding to keep in view when a product leads with recall, graph coverage, or token savings:

Better retrieval coverage does not guarantee better epistemic behaviour.

The generator still has to distinguish evidence from noise, recognize contradiction, preserve qualifications, and stop when the answer remains unsupported. A hybrid retrieval system without an explicit evidence-sufficiency test may produce a larger context and a weaker decision.

The field is already revising the original story

One of the most interesting recent papers does not present the graph as a replacement for the source.

PAGE-RAG: Evidence-Grounded Adaptive Graph Retrieval for Long-Document Question Answering begins from the premise that automatically constructed graphs are incomplete projections of source documents.

It treats the graph as a semantic skeleton used to organize and navigate the source, routes questions according to their evidence needs, and introduces a knowledge-boundary mechanism intended to abstain when accessible evidence remains insufficient. 6

That design reflects a correction already becoming visible across the research.

The first wave often implied:

Documents → Graph → Knowledge

The more defensible model is:

Documents → Incomplete graph projection
                 ↓
          Navigate back to evidence

The graph can reduce a search space, expose a relationship, organize candidates, or identify material that deserves closer inspection. It should not become an independent authority merely because an LLM converted prose into structured storage.

This changes the design obligation. A node must lead back to the passage that supports it. An inferred edge should remain distinguishable from an extracted one. A community summary should behave like a navigation aid rather than a canonical replacement for the documents it compressed. If the source and graph disagree, the source must remain available and the discrepancy must be visible.

Structure does not upgrade an inference into a fact.

Why code graphs deserve a separate category

Some of the fastest-growing “GraphRAG” repositories are primarily code-intelligence systems, and their reliability profile differs from graphs extracted from prose.

Source code has formal grammar. A parser can determine that one module imports another, a class inherits from a parent, or a function references a symbol without asking a language model to interpret a paragraph.

Graphify, which had accumulated roughly 94,000 GitHub stars at the time of this review, parses supported code locally with Tree-sitter. Its repository distinguishes EXTRACTED relationships explicit in source structure from INFERRED relationships produced through later resolution. Documents, PDFs, images, and media still require semantic model processing, so the deterministic advantage applies most strongly to parseable source code rather than every input the project accepts. 8
GitNexus similarly parses functions, classes, methods, and interfaces through Tree-sitter, resolves calls and imports across files, clusters related symbols, and exposes execution flows and impact-analysis tools to agents through MCP. 9

This is useful engineering. Coding agents often burn context reconstructing relationships already available through static analysis. Returning the callers, dependent modules, and likely change surface of a symbol can be far more efficient than opening dozens of files.

The underlying techniques are old. Abstract syntax trees, call graphs, symbol indexes, dependency analysis, and code-query systems predate LLM agents by decades. The recent innovation lies in packaging those structures as context interfaces for autonomous tools.

The remaining evidence gap concerns the final work. Does the agent modify the correct files more often? Does it introduce fewer regressions? Does lower token usage preserve the configuration, runtime dispatch, generated code, framework convention, or exception that would have prevented a mistake?

Most public claims still come from developer-run benchmarks and product demonstrations. I found no independent controlled comparison showing that these tools improve end-to-end coding success across repositories, languages, and agent models when compared under the same conditions with grep, language-server navigation, static analysis, vector retrieval, and iterative agent search.

A tool may use fewer tokens because it located the correct dependency quickly. It may also use fewer tokens because the graph omitted the runtime path that mattered. Those outcomes look identical in a token dashboard and entirely different in production.

Why 90,000 stars do not settle the question

GitHub popularity explains part of the GraphRAG mood, but it cannot resolve technical quality.

The expanded version of the Carnegie Mellon-led StarScout study identified about six million suspected fake stars across 26,254 repositories before post-processing. Socket Inc participated through one co-author and later collaboration, but the study was not simply “Socket research.” 12

That finding does not show that Graphify, GitNexus, or any other GraphRAG repository purchased stars.

It shows that star counts are vulnerable to coordinated inflation and should not be treated as standalone evidence of retrieval quality, graph completeness, maintenance reliability, or production value.

Even authentic stars measure attention, timing, accessibility, and distribution. They do not tell us how many answer entities were lost during indexing, whether inferred edges are supported by source text, how the system behaves after a schema change, or whether an agent finishes more tasks correctly.

Stars tell us what developers are looking at.They do not tell us whether the graph contains the answer.

The second hype wave: an AI that writes its own memory

GraphRAG still begins with source material and creates a representation used for retrieval. The newer LLM-wiki pattern goes further: the agent writes a persistent knowledge layer that its future self will read.

Andrej Karpathy’s LLM Wiki pattern describes a system with immutable raw sources, an LLM-maintained directory of summaries and topic pages, and a schema telling the agent how to ingest, query, cross-reference, and maintain the knowledge base. The attraction is accumulation. Instead of rediscovering the relationship among five documents every time, the model can preserve a synthesis and build on it later. 10

This can be genuinely valuable. It also creates a recursive failure mode that ordinary RAG does not have.

The retrieved material now includes the model’s own earlier interpretations.

Six problems follow:

Self-contamination: one incorrect summary is copied into later pages and begins to resemble independent corroboration.
Append-only drift: new evidence is added while obsolete conclusions survive elsewhere.
Missing canonical authority: several pages disagree and nothing identifies which one governs the current state.
Summary-lineage collapse: a claim passes through enough rewrites that its original source becomes difficult to recover.
Connectivity–truth confusion: a frequently linked page appears credible because it is central, although centrality measures connection rather than validity.
Propagation failure: the source changes while dependent summaries, recommendations, and plans remain untouched.

These failures compound. One inaccurate summary enters three topic pages. Those pages are later compressed into a higher-level overview. The original error now appears in several places, linked from several directions, and may be retrieved as though multiple notes independently support it. The workspace looks increasingly organized while the provenance underneath it becomes harder to reconstruct.

This architecture exchanges repeated raw retrieval for a maintenance problem. The trade may be worthwhile, but only when the system preserves immutable sources, records derivation, identifies canonical pages, and can invalidate every downstream artifact affected by a correction.

Otherwise the agent can quote its own error back to itself with several links attached.

The graph itself becomes a sensitive asset

Turning documents into a connected structure creates another problem: the relationships may be more sensitive than the individual records.

The 2026 GraphSteal study treated GraphRAG systems as structural oracles. Through adaptive black-box interaction, the researchers reported reconstructing more than 90 percent of the original knowledge graph in representative generic and healthcare scenarios, exposing entities, relationships, topology, and structural dependencies. Existing guardrails offered limited protection in their experiments. 11

This does not establish that every deployed GraphRAG system is readily extractable. It establishes a new attack surface that document-centric access control may not cover.

A collection of files contains sensitive facts. A graph may additionally reveal which people bridge teams, which records form hidden clusters, which entities are unusually central, and which dependency path would cause the most damage if exposed. Individual documents may be innocuous when viewed separately, while the graph assembled from them reveals a relationship that no user was meant to observe in one place.

Organizations considering GraphRAG therefore need to review the derived graph, query interface, traversal behaviour, rate limits, logs, and inference leakage as assets in their own right. Permission inheritance from source documents may not be enough when the graph produces new aggregates, paths, and structural inferences.

The edges may be more sensitive than the nodes.

A knowledge map still does not control an agent

Underneath all the benchmark results sits a more basic limitation.

A knowledge map tells an agent what it can inspect. It does not determine what the agent will believe, how it will resolve conflicting evidence, or whether it is allowed to act.

A graph edge can say that production deletion requires human approval. The sentence may be retrieved correctly and placed directly in the model’s context. None of that prevents the model from ignoring the condition or calling an API that performs the deletion without checking for an approval token.

The system contains several boundaries:

Original source
      ↓
Graph, index, or wiki
      ↓
Retrieved evidence
      ↓
Generated answer or plan
      ↓
Authorized action

Information can be lost or transformed at every arrow.

The source may be incompletely represented. Retrieval may select the wrong version. Generation may overstate what the evidence supports. The runtime may accept an action that policy should have blocked.

GraphRAG primarily improves parts of the middle. It does not automatically provide:

Provenance: every node, edge, summary, and recommendation traceable to its source, source version, and construction method.
Invalidation: a way to identify every derivative that becomes suspect when the source changes.
Revalidation: recomputation, quarantine, downgrade, or explicit historical labeling before stale material returns to active use.
Hard enforcement: server-side permission and approval checks that do not depend on whether the model read or interpreted a policy correctly.

These controls are difficult precisely because derived knowledge propagates. A policy paragraph creates an edge. The edge enters a community. The community summary influences a wiki page. The wiki page supplies an agent plan. The original policy changes, while the plan remains in memory. Updating the source file solves only the first step in that chain.

Without provenance and dependency tracking, the system cannot reliably determine what else became stale.

Without enforcement, even a correct answer remains advice.

What should you deploy?

The answer depends on the workload, which is less satisfying than buying the newest architecture and much cheaper than discovering the workload afterward.

Before deploying any of them, require answers to six questions:

What percentage of real queries genuinely require relationship traversal?
How complete is the graph against the original sources?
What happens when the graph and source disagree?
How are dependent edges, summaries, and plans invalidated after updates?
Does improved retrieval increase final task success, or only retrieval metrics and token efficiency?
What happens when the corpus cannot answer the question?

The final question is the one vendors are least likely to volunteer.

It is also the one most likely to tell you what you bought.

A credible pilot should include questions whose answers are absent, documents that conflict, facts that have expired, and source updates that invalidate existing graph structures.

It should compare GraphRAG with keyword search, dense RAG with reranking, and an agent allowed to search iteratively. The evaluation should measure final task success and unsupported claims, rather than stopping at retrieval recall or token reduction.

The evidence is stronger than hype, and weaker than certainty

GraphRAG is no longer supported only by demos. There are systematic comparisons, accepted conference work, agentic-search benchmarks, efficiency analyses, and an emerging second generation of systems that explicitly address incomplete graph projections and evidence boundaries.

Xiang et al.’s When to Use Graphs in RAG, for example, was accepted as an ICLR 2026 Poster and reports results often unfavourable to broad GraphRAG claims. 3

That is meaningful evidence.

It is not yet mature consensus. Among the studies reviewed here, I found no GraphRAG performance claim that had reached full independent reproduction by a separate team under comparable conditions across production workloads. Much of the research remains developer-led, benchmark-specific, or dependent on the same authors who designed the evaluated framework.

That is normal for a young field. It also means that “peer reviewed,” “open source,” “popular,” and “independently reproduced” should not be treated as interchangeable labels.

The appropriate response is narrower confidence.

The verdict

GraphRAG is not vaporware. It is also not a new artificial brain.

It can earn its cost when questions depend on relationships scattered across documents, temporal order, entity comparison, or corpus-level structure. It may lose to a much simpler retriever when the user needs one precise fact. Its graph can omit required information before retrieval begins.

Its summaries can damage abstention. Combining it with ordinary RAG can give a smaller model more evidence and worse judgment. Its maintenance introduces cost and staleness, while its structure creates a new security asset that must be protected.

Code graphs are often more deterministic because code has formal syntax, although their end-to-end value for agent performance remains less independently tested than their popularity suggests. LLM-maintained wikis may compound useful knowledge across sessions while also allowing the model’s earlier mistakes to become future evidence.

The useful production architecture is unlikely to be GraphRAG alone. It will combine keyword search, vector retrieval, reranking, graph traversal, direct source access, and iterative agent search, selecting among them according to the question and verifying the evidence before action.

GraphRAG has earned a place in that stack.

The larger claim — that structuring information gives an agent understanding, judgment, and control over its own behaviour — is not supported by the studies reviewed here.

Retrieval infrastructure with unresolved lifecycle governance does not fit comfortably into a launch post.

So it gets called a brain instead.

💡Further technical reading

This article focuses on the benchmark evidence and practical deployment boundary. For a more detailed technical examination of GraphRAG, persistent agent memory, provenance, invalidation, revalidation, and the gap between available knowledge and controlled action, see my full report:

GraphRAG, Persistent Agent Memory, and the Distance Between Available Knowledge and Controlled Action

References

1 Han, H. et al. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv:2502.11371, revised March 2026.

2 Edge, D. et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130.

3 Xiang, Z. et al. When to Use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation. ICLR 2026 Poster; arXiv:2506.05690.

4 *Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems*. arXiv:2604.09666.

5 Chen, L. et al. Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization. arXiv:2606.25656.

6 Chen, X. et al. PAGE-RAG: Evidence-Grounded Adaptive Graph Retrieval for Long-Document Question Answering. arXiv:2607.19301.

7 Microsoft. GraphRAG Repository and Documentation. GitHub.

8 Graphify Labs. *Graphify: Local Code and Document Knowledge Graphs for AI Coding Assistants*. GitHub.

9 Patwari, A. GitNexus: The Zero-Server Code Intelligence Engine. GitHub.

10 Karpathy, A. LLM Wiki: A Pattern for Building Personal Knowledge Bases Using LLMs. GitHub Gist, April 2026.

11 Gu, J. et al. GraphSteal: Structural Knowledge Stealing from Graph RAG via Traversal Reconstruction. arXiv:2605.28645.

12 He, H., Yang, H., Burckhardt, P., Kapravelos, A., Vasilescu, B., and Kästner, C. Six Million (Suspected) Fake Stars in GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware. ICSE 2026; arXiv:2412.13459v2.

I Built a Local RAG Assistant with Ollama, ChromaDB and LangChain

Sat, 25 Jul 2026 14:42:23 +0000

During my internship at a software services company, I noticed a recurring problem: technicians spent hours searching through hundreds of pages of PDF manuals to answer client questions. The company deploys a suite of management tools accounting, HR, logistics mainly to public institutions managing projects funded by international donors.

I kept thinking: this is exactly the kind of problem LLMs were made for. So for my Master's project in AI & Big Data, I built an intelligent assistant to handle it.

The constraint that shaped every technical decision: no data could leave the local infrastructure. These institutions handle sensitive financial data it cannot be sent to OpenAI, Anthropic, or any cloud provider.

This forced me into the world of fully local LLMs. Here's what I built, what broke, and what I learned.

The Problem

The company had accumulated massive technical documentation PDF manuals, configuration guides, FAQ files but no efficient way to query it. When a technician faced an issue with the accounting module or the HR module, they searched manually. Slow, inconsistent, and dependent on individual expertise.

Classic LLMs can't help here: they don't know the private internal documentation. And without grounding in official docs, they hallucinate procedures dangerous when dealing with accounting operations for donor-funded projects.

The solution: RAG (Retrieval-Augmented Generation) make the LLM answer only from the official documentation, injected at query time.

What RAG Actually Is

Before the code, the concept in one sentence: instead of asking the LLM to "know" everything, you retrieve the relevant passages from your documents and inject them into the prompt as context.

The pipeline has two distinct phases:

Phase 1 : Ingestion (done once)

PDF documents
    ↓
Chunking (split into 300-character blocks)
    ↓
Embedding (convert each block to a vector)
    ↓
ChromaDB (store all vectors)

Phase 2 : Query (at each question)

User question
    ↓
Embed the question (same model)
    ↓
ChromaDB similarity search → top 3 chunks
    ↓
Inject chunks + question into LLM prompt
    ↓
Llama 3 generates a grounded answer

The key insight: the LLM never "knows" your documents. It reads them fresh at each query, from the context you provide. No fine-tuning needed. No retraining when docs are updated.

The Architecture: 4 Docker Services

Everything runs locally via Docker Compose. Four services, each with a clear role:

Service	Role	Port
Ollama	Runs Llama 3 locally	11434
ChromaDB	Vector database	8001
FastAPI	RAG pipeline + REST API	8000
Streamlit	User interface	8501

# docker-compose.yml (simplified)
services:
  ollama:
    image: ollama/ollama:latest
    ports: ["11434:11434"]
    volumes: ["./ollama_models:/root/.ollama"]
    environment:
      - OLLAMA_KEEP_ALIVE=24h

  chromadb:
    image: chromadb/chroma:latest
    ports: ["8001:8000"]
    volumes: ["./chroma_db:/chroma/chroma"]

  python-api:
    build: ./src
    ports: ["8000:8000"]
    depends_on: [ollama, chromadb]
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - CHROMA_HOST=chromadb
      - CHROMA_PORT=8000

Step 1 : Ingesting the Documentation

I ingested 10 documents accounting references (SYSCOHADA, SYCEBNL), a general accounting code, and internal user manuals. That's 2,111 pages split into 9,669 chunks.

# src/ingestion.py

import os
import chromadb
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

DATA_DIR = "/app/data/documents"
COLLECTION_NAME = "documentation"

def load_documents():
    documents = []
    for fichier in os.listdir(DATA_DIR):
        if fichier.endswith(".pdf"):
            try:
                loader = PyPDFLoader(os.path.join(DATA_DIR, fichier))
                docs = loader.load()
                # Tag each chunk with its source document
                for doc in docs:
                    doc.metadata["module"] = fichier.replace(".pdf", "")
                documents.extend(docs)
            except Exception as e:
                print(f"Skipped {fichier}: {e}")  # Don't crash on encrypted PDFs
    return documents

def store_embeddings(chunks):
    embeddings = HuggingFaceEmbeddings(
        model_name="all-MiniLM-L6-v2",
        model_kwargs={"device": "cpu"}
    )
    # Explicit HTTP connection — not localhost, the Docker service name
    client = chromadb.HttpClient(host="chromadb", port=8000)
    Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        collection_name=COLLECTION_NAME,
        client=client
    )
    print(f"Done — {len(chunks)} vectors stored")

Two things worth noting:

I tag each chunk with its source document name as metadata this allows the LLM to cite its source in the answer
The try/except is not optional one encrypted PDF (it happened) would have crashed the entire ingestion without it. Install cryptography if you hit an AES error: pip install cryptography

Step 2 : Building the RAG Chain

# src/rag_chain.py

import chromadb
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.llms import Ollama
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

def create_rag_chain():
    # Must be the same model used during ingestion
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

    client = chromadb.HttpClient(host="chromadb", port=8000)
    vectorstore = Chroma(
        client=client,
        collection_name="documentation",
        embedding_function=embeddings
    )

    # Retrieve top 3 most relevant chunks
    retriever = vectorstore.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 3}
    )

    llm = Ollama(
        base_url="http://ollama:11434",
        model="llama3",
        temperature=0.3,
        num_predict=250   # Cap response length — critical on CPU
    )

    template = """You are a technical support assistant.
Answer ONLY from the provided context. Cite your source.
If the answer is not in the context, say so clearly.

Context: {context}
Question: {question}
Answer:"""

    prompt = ChatPromptTemplate.from_template(template)

    def format_docs(docs):
        return "\n\n".join([
            f"[Source: {d.metadata.get('module', '?')}]\n{d.page_content}"
            for d in docs
        ])

    # The full pipeline in 4 steps
    return (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )

The chain is a pipeline: retriever → prompt → llm → parser. LangChain's | operator makes this clean and readable. Each step passes its output to the next.

Step 3 : API and Interface

FastAPI exposes the chain as a REST endpoint:

# src/main.py

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from rag_chain import create_rag_chain

app = FastAPI(title="Local RAG Assistant")
chain = create_rag_chain()

class QueryRequest(BaseModel):
    question: str

@app.post("/query")
def query(request: QueryRequest):
    try:
        answer = chain.invoke(request.question)
        return {"question": request.question, "answer": answer}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Streamlit makes it usable in a browser in 10 lines:

# src/app.py

import streamlit as st
import requests

st.title("Technical Support Assistant")
question = st.text_input("Your question:")

if st.button("Search") and question:
    with st.spinner("Searching documentation... (1-2 min on CPU)"):
        r = requests.post(
            "http://python-api:8000/query",
            json={"question": question},
            timeout=600
        )
        st.write(r.json()["answer"])

What Broke (and How I Fixed It)

This is the part nobody writes about. Three real problems I hit.

Problem 1 : ChromaDB v2 silently ignoring my data

I was connecting with the old client_settings syntax:

# WRONG silently stores data locally instead of sending to the server
Chroma.from_documents(
    documents=chunks,
    client_settings=Chroma.Settings(chroma_server_host="chromadb")
)

The ingestion ran without errors, showed 9669 chunks stored but ChromaDB had 0 vectors. Querying returned nothing.

The fix explicit HttpClient:

# CORRECT explicit HTTP connection to the ChromaDB Docker service
client = chromadb.HttpClient(host="chromadb", port=8000)
Chroma.from_documents(documents=chunks, client=client, ...)

Also: the v1 heartbeat endpoint (/api/v1/heartbeat) is deprecated. Use /api/v2/heartbeat to check if ChromaDB is alive.

Problem 2 : LangChain moved its modules

# Broke silently after a LangChain update
from langchain.prompts import ChatPromptTemplate           # ❌ ModuleNotFoundError
from langchain.schema.runnable import RunnablePassthrough  # ❌ ModuleNotFoundError

# Everything stable lives in langchain_core now
from langchain_core.prompts import ChatPromptTemplate      # ✅
from langchain_core.runnables import RunnablePassthrough   # ✅

LangChain has been refactoring aggressively across versions. If you hit ModuleNotFoundError, check langchain_core first it's the stable base layer they're committing to not break.

Problem 3 : Llama 3 timing out on CPU

Llama 3 (8B parameters) on CPU generates roughly 2-5 tokens per second. A 200-word answer takes 2+ minutes. My 120-second timeout was too short.

Three things helped significantly:

Dropping k from 5 to 3 chunks shorter prompt = faster time-to-first-token
Adding num_predict=250 to cap response length without it, the model generates forever
Allocating more RAM to Docker via .wslconfig on Windows to avoid swap

Results

After setup, the assistant correctly answers questions grounded in the official documentation:

"How to close monthly payroll?" → Step-by-step procedure from the HR module manual
"How to create an account in the chart of accounts?" → Exact navigation path, source cited
"What does SYSCOHADA class 4 cover?" → Accurate definition from the accounting reference

And critically when the answer isn't in the documentation, it says so explicitly instead of hallucinating. That was the whole point.

Key Takeaways

On RAG:

The embedding model used during ingestion must be identical to the one used at query time. A different model produces incompatible vector spaces your similarity search returns garbage without any error message.

Chunk size matters more than expected. 500 characters worked less well than 300 for technical documents shorter chunks produce more precise retrieval.

Metadata on chunks is essential for source citation. Tag every chunk with its document name at ingestion time.

On local LLMs:

Running on CPU is slow but viable for non-real-time use cases like internal support tools.

num_predict is your most important parameter for performance. Without it, the model generates until it decides to stop sometimes never.

Docker container networking does not use localhost. Use the service name defined in docker-compose.yml (e.g., chromadb, ollama) as the hostname when connecting between containers.

This project is available on my github : https://github.com/josaphatstar/Assistant-Intelligent-RAG

What's Next

This RAG pipeline handles straightforward questions well. But it has a core limitation: it always does the same thing regardless of the question type. Whether you ask a procedural question or report a system crash, it searches the docs and answers. No reasoning about what strategy fits best.

In the next article, I'll show how I evolved this into an Agentic AI architecture with LangGraph where the system first decides which agent to call (documentation search, error diagnosis, or human escalation ticket), then acts accordingly.

I'm a Master's student in AI & Big Data, and this project came out of a real internship constraint no cloud, no API keys, just local infrastructure and open-source tools. I'm curious: have you built a local RAG pipeline under similar constraints? What stack did you use, and what broke first? Drop it in the comments I'd genuinely like to compare notes.

I Benchmarked 19 Retrieval Pipelines Head-to-Head and the Results Were Surprisingly Honest

Thu, 23 Jul 2026 17:45:54 +0000

I got tired of retrieval papers claiming their pipeline is "state-of-the-art" without significance tests, without telling you what dataset they ran on, and without reproducible code. So I built raven-retrieval -- 19 retrieval pipelines, real BEIR datasets, Bonferroni-corrected bootstrap significance tests, all the numbers whether they look good or not. MIT licensed, open source.

What It Does

The core question: does ColBERT-style late interaction scoring at every level of a RAPTOR hierarchical tree actually improve retrieval over simpler approaches? To answer that properly you need the same datasets, same chunking, same metrics, same significance tests for every pipeline. Otherwise you're comparing apples to oranges.

The framework implements 19 pipelines and benchmarks them all on BEIR datasets (SciFact, HotpotQA) on equal footing. No cherry-picking.

The 19 Pipelines (Quick Summary)

Core: Naive Dense RAG, Hybrid RAG (BM25+Dense+RRF), ColBERT Late Interaction, RAPTOR+Late Interaction (novel combination).

Research (2024-2025): HyDE, SPLADE, SPLADE+Dense, BM25+Rocchio PRF, Contextual Retrieval, Contextual Hybrid, Late Chunking, RAPTOR+Late Traversal, Agentic Multi-Hop, Reflection Retriever, Graph Retrieval, Two-Stage Dense+Reranker, Approximate Late Interaction, Contextual Dense, Contextual BM25.

Composition modules: Cross-Encoder Reranker, Two-Stage Retriever, Document Graph, Residual Compressor (~30x storage reduction).

The novel one is RAPTOR + Late Interaction: build a RAPTOR hierarchical summary tree, then apply ColBERT MaxSim scoring at every node (leaf chunks AND summary nodes). Nobody has published on this combination before.

How MaxSim Works

ColBERT's scoring: for each query token, find the most similar document token (cosine), then sum those maximums across all query tokens. Every query token gets a vote for its best-matching document token. This captures fine-grained token-level relevance that single-vector cosine misses entirely. Implemented in pure numpy, no torch needed for scoring.

The Benchmark Results

Run on Google Colab T4 GPU, seed=42, two datasets.

SciFact (100 queries, single-hop scientific claims)

Pipeline	nDCG@10	Per-Query
HyDE	0.712	19ms
Naive Dense	0.696	32ms
Contextual Hybrid	0.682	93ms
Hybrid RAG	0.667	87ms
BM25+PRF	0.529	43ms

Key finding: Top 4 are statistically indistinguishable (all pairwise p > 0.05 after Bonferroni). BM25+PRF is significantly worse (p ~ 0.000). The "winner" HyDE is only 2 points above Dense -- within noise.

HotpotQA (50 queries, multi-hop reasoning)

Pipeline	nDCG@10	Per-Query
Hybrid RAG	0.925	22ms
Contextual Hybrid	0.923	22ms
Naive Dense	0.906	12ms
HyDE	0.892	26ms
BM25+PRF	0.869	8ms

The ranking flips completely. HyDE drops from 1st to 4th. Hybrid RAG wins because BM25+Dense fusion catches different reasoning hops. A single hypothetical document can't bridge two separate reasoning steps.

The most important finding: No single pipeline dominates across task types. Single-hop favors semantic (HyDE). Multi-hop favors hybrid. Anyone claiming universal superiority hasn tested on enough datasets.

The Bugs I Found

Before v0.3, 13 bugs (6 critical) silently corrupted results:

ColBERT matched against PAD tokens -- attention mask wasn used, padding tokens got MaxSim votes alongside real content
RAPTOR clustering used wrong embedding space -- centroids from ColBERT means while scoring with SBERT vectors
Per-query nDCG was all zeros -- BEIR evaluate() returns averaged floats, not per-query; significance tests fed zeros and reported "no difference" everywhere (false negatives)
Reflection evaluated keyword coverage over doc ID strings ("doc_1", "doc_42") instead of actual text, triggering endless reformulation loops
RAPTOR pipelines crashed -- UMAP bug when local clusters had 3 samples (k >= N), burned 189 minutes of Colab time

These don crash your program. They just make your results wrong in ways you can see from final numbers alone.

Known Limitations

ColBERT projection head is Xavier-initialized (untrained). Late interaction scores ~0.58 without training, worse than Dense at 0.696. Training support exists (ColbertContrastiveEncoder with InfoNCE) but no trained checkpoint in default benchmark yet.
GMM soft-clustering converges to near-hard assignment on ~100-token chunks (matches Stanford CS224N reproduction).
Late Chunking limited by BERT's 512-token window -- needs long-context model to really shine.
HotpotQA scores are on 2,000-doc subsample, not the full 5.2M corpus. Real retrieval is harder than these numbers suggest.

git clone https://github.com/subhansh-dev/raven-retrieval.git
cd raven-retrieval
pip install -e ".[full]"
python run_enhanced_benchmark.py --dataset scifact --top-k 10 --seed 42

CLI entry points: raven-benchmark, raven-report. Low-RAM? Use --max-docs. Want trained ColBERT? Use --colbert-checkpoint. All 55 tests pass.

Why This Matters

Retrieval benchmarks have a credibility problem. Papers claim improvements without significance tests, without multiple datasets, without reproducible code. Raven-retrieval fixes all of that. The honest answer -- that top pipelines are tied, that rankings flip between datasets, that BM25+PRF is bad on scientific text -- is more useful than cherry-picked claims of universal superiority.

github.com/subhansh-dev/raven-retrieval | subhansh.dev

How a Pre-Retrieval Parse Loop Fixes Vague RAG Queries

Mon, 20 Jul 2026 10:01:33 +0000

RAG (Retrieval-Augmented Generation - where an LLM answers questions using retrieved document chunks) breaks most often before retrieval even starts. The question going in is underspecified, and no retrieval system can compensate for a bad query.

The Idea: A Small Loop Before You Search

Most RAG pipelines treat the user's raw question as retrieval-ready. It usually isn't. A question like "what are the risks?" has no anchor - no product, no time range, no document scope. The retrieval step fetches loosely related chunks, the LLM hallucinates a coherent answer from noise, and everyone blames the model.

Loop engineering for question parsing inserts a lightweight reasoning step before retrieval runs: parse the raw question, identify what context is missing to make it answerable, then re-parse with that gap filled - either by asking the user, inferring from conversation history, or pulling from a lightweight document index. The loop is intentionally small: one or two passes, not an autonomous agent spinning indefinitely. The goal is a query that names specific entities, a scope, and an intent - not a philosophical question the retriever has to guess at.

Real Example

Here's a minimal implementation pattern using Python and any chat-completion API:

def parse_and_refine(raw_question: str, doc_summary: str, llm) -> str:
 audit_prompt = f"""
User question: "{raw_question}"
Document summary: "{doc_summary}"

Identify what is ambiguous or missing (entity, time range, scope).
Return a refined, retrieval-ready question. If nothing is missing, return the original.
"""
 refined = llm.chat(audit_prompt)
 return refined.strip()

# Usage
raw = "What are the risks?"
doc_summary = "Q3 2024 earnings report for Acme Corp, covering supply chain and FX exposure."
ready_query = parse_and_refine(raw, doc_summary, llm)
# → "What supply chain and FX risks does Acme Corp report in Q3 2024?"

The doc_summary doesn't have to be a full index - a one-paragraph description of the document set is enough to ground the refinement. For multi-turn chat, you'd also pass conversation history into the audit prompt so the loop can resolve pronouns and carry-forward references ("the risks from last quarter").

The cost is one extra LLM call per query. On most inference APIs that's under 50ms added latency and a fraction of a cent - easily worth it if it prevents a hallucinated answer that sends someone down the wrong path.

Key Takeaways

Retrieval quality is bounded by query quality - fixing the query upstream is more reliable than tuning chunking or embeddings downstream.
A two-pass parse loop (audit then refine) catches the most common failure mode: questions that lack entity, scope, or time anchors.
Keeping the loop to one or two LLM calls maintains low latency while meaningfully improving retrieval precision.

What's the most common type of underspecified question your users actually send into your RAG system - missing entity, missing time range, or something else entirely?

Sources referenced: Towards Data Science - Loop Engineering for RAG Question Parsing

I grade my AI twin on real traffic, not just an offline test set

Fri, 17 Jul 2026 05:44:28 +0000

Most eval setups have a quiet lie in them. You run a fixed question set through your agent, a judge model scores the answers, you get a number, and you feel good. But that number describes a sandbox. It says nothing about the question a stranger typed into your live app at 2am, or whether your anti-hallucination guardrail held for that specific input.

Twin is my personal AI digital twin. It answers questions about me over a RAG index of my profile, running on Bedrock with Claude Sonnet 4.5, FastAPI backend, Next.js frontend. I already had an offline eval harness. What I added this week is a second judge that grades production traffic as it happens, and building it forced me to fix an integrity bug I'd been ignoring in the offline one.

The problem with the offline harness

My offline harness in evals/ does the honest-looking thing: load queries.json, run retrieval, run inference, hand the result to a judge. The judge prompt is strict about faithfulness:

JUDGE_SYSTEM_PROMPT = """You are grading whether an AI-generated answer is
faithful to the provided source material. Be strict: any claim in the answer
not directly supported by the source material counts as a hallucination, even
if it sounds plausible.
...
- "correctly_refused" is true if the source material contains nothing relevant
  to the question and the answer clearly states the information isn't available;
  false if the source material was empty/irrelevant but the answer invented
  something anyway...
"""

The judge is good. The problem was structural: the offline harness ran its own retrieval pass, separate from the answer's retrieval pass. So the judge was grading the answer against a source set that wasn't necessarily the one the model actually saw. If retrieval is non-deterministic or the index shifts, the judge is checking the wrong evidence. You can pass an eval where the model hallucinated and fail one where it didn't. The number is measuring the wrong thing.

That gap is invisible on a static question set because retrieval usually returns the same chunks twice. It stops being invisible the moment you point it at live traffic, where you have no fixed inputs at all.

The approach: capture the exact source the model saw

The fix is boring and correct. Whatever chunks the answer was generated from, those exact chunks get judged. No second retrieval.

In the chat path, right after inference, I capture the request with the same retrieved_chunks object that produced the answer:

retrieved_chunks = retrieval.retrieve(request.message, k=5)
# ... inference runs against retrieved_chunks ...
capture_live_eval(request.message, retrieved_chunks, assistant_response)

capture_live_eval is fire-and-forget. It never raises, because an eval capture failing should never break a user's chat:

def capture_live_eval(query: str, retrieved_chunks: List, answer: str) -> None:
    """Fire-and-forget capture of a real chat exchange for async
    faithfulness judging. Never raises."""
    if not EVALS_BUCKET:
        return
    try:
        retrieved_text = "\n\n".join(
            f"## {chunk.section_title}\n{chunk.text}"
            for chunk, score in retrieved_chunks
        )
        key = f"live/raw/{datetime.now().isoformat()}-{uuid.uuid4()}.json"
        body = {
            "timestamp": datetime.now().isoformat(),
            "query": query,
            "retrieved_chunk_ids": [c.chunk_id for c, _ in retrieved_chunks],
            "retrieved_text": retrieved_text,
            "answer": answer,
        }
        evals_s3_client.put_object(
            Bucket=EVALS_BUCKET, Key=key,
            Body=json.dumps(body), ContentType="application/json",
        )
    except Exception as e:
        print(f"Live eval capture failed (non-fatal): {e}")

The important line is that retrieved_text is serialized from the same chunks the answer was built on. The judge later reads exactly this. There is no second retrieval anywhere in the live path.

The judging happens out of band

I did not want to block a chat response on a judge call. So the write to S3 under live/raw/ triggers a separate Lambda via an S3 event. That Lambda reads the raw record, judges it, and writes the result to live/judged/:

def process_record(bucket: str, key: str, s3_client) -> None:
    raw = json.loads(
        s3_client.get_object(Bucket=bucket, Key=key)["Body"].read().decode("utf-8")
    )
    judged_key = key.replace("live/raw/", "live/judged/", 1)

    output = dict(raw)
    try:
        judgment = judge.judge_answer(
            raw["query"], raw["retrieved_text"], raw["answer"]
        )
        output["judgment"] = judgment
    except Exception as e:
        output["judgment_error"] = str(e)

    s3_client.put_object(
        Bucket=bucket, Key=judged_key,
        Body=json.dumps(output), ContentType="application/json",
    )

The chat request writes one small JSON object and moves on. The user waits on nothing. The judge runs whenever the event fires. The same judge_answer function backs both the offline harness and the live judge, so I am grading production and my test set with identical criteria. That reuse is deliberate. It's the only reason the two numbers are comparable.

One detail I like: the judge runs on Bedrock with temperature: 0.0 and parses a strict JSON object with a brace-depth scan rather than a regex, because judge models love to wrap JSON in prose no matter how many times you tell them not to.

What broke, and what I'd do differently

The capture path writes retrieved_chunk_ids alongside the text. I don't use those IDs in the judge yet, but they exist for a reason: my next problem is retrieval drift. If I rebuild the profile index, the chunk IDs behind a past judgment change, and a stored judgment silently starts referring to source text that no longer exists. Right now the judged record freezes retrieved_text at capture time, which protects the judgment itself, but I have no way to diff "what the judge saw" against "what the index says today." Storing the IDs is the first half of that; I haven't built the second half.

The bigger honest gap: I only judge faithfulness. The judge can tell me the model stayed inside its source material. It cannot tell me the answer was useful, or that retrieval pulled the right chunks in the first place. A perfectly faithful answer to badly retrieved context still passes. Faithfulness is the cheap half of quality, and it's the half I automated because it's the half a single model call can check.

I'd also reconsider the fire-and-forget silence. capture_live_eval swallowing every exception means a broken eval bucket produces one print and nothing else. On Lambda that print lands in CloudWatch and I will not notice for a week. If I were doing this again I'd emit a metric on capture failure, because a silent eval pipeline is the same as no eval pipeline.

Takeaway: an eval number is only trustworthy if the judge grades the exact evidence the model actually used, and the fastest way to find out whether yours does is to point it at real traffic instead of a frozen question set.

Knowledge Management at Scale: Multi-Tenant RAG with RBAC

Thu, 16 Jul 2026 05:41:30 +0000

Your organization has knowledge scattered everywhere: docs, wikis, Slack conversations, emails, databases. Your AI agents need access to it. But you can’t let every agent access every document (security, privacy). I built Socratic-knowledge: enterprise RAG (Retrieval-Augmented Generation) with multi-tenancy, role-based access control, versioning, and audit logging. Agents retrieve exactly the knowledge they need. Users stay secure.

The Problem: Knowledge Is Fragmented and Dangerous
Scenario 1: The Scattered Knowledge
Your organization has knowledge in:

Wikis (internal documentation)
Google Drive (design docs, strategy docs)
GitHub (code docs, architecture)
Slack (Q&A conversations, decisions)
Email (historical decisions, context)
Databases (product specs, configurations)
PDFs (whitepapers, research)
Your agents need this knowledge to make good decisions.

But it’s fragmented. Inconsistent. Sometimes contradictory.

Your agent searches the wiki for product specs. Gets outdated info. Your agent searches email for decision history. Gets 500 unrelated emails. Your agent searches Slack. Finds someone’s joke mixed with actual information.

Agents make bad decisions because they have bad knowledge.

Scenario 2: The Security Nightmare
You have sensitive knowledge:

Customer data
Financial information
Proprietary algorithms
Personal conversations
Some agents need access (customer support agent needs customer data). Other agents shouldn’t (recommendation engine shouldn’t see financial data). Users need to control what’s shared (customer shouldn’t see other customer’s data).

Without access control, either:

You don’t share knowledge (agents are dumb)
You share everything (security breach)

Scenario 3: The Versioning Problem
Knowledge changes. Your product specs are updated. Your strategy shifts. Your processes evolve.

Which version should the agent use?

The current version? (might break compatibility with old customers)
The version from when the agent was trained? (might be outdated)
All versions? (agent gets confused by contradictions)
Without versioning, agents work with stale knowledge.

Scenario 4: The Compliance Burden
You need to know:

Which agent accessed which knowledge?
When did it access it?
What decisions did it make based on that knowledge?
If a decision was wrong, which knowledge led to it?
Without audit logging, you have no compliance trail.

Why Standard RAG Fails for Multi-Tenant Systems
Standard RAG
Standard RAG (Retrieval-Augmented Generation):

Store all knowledge in a vector database
When agent needs knowledge, search the database
Return top matching documents
Agent uses documents to make decisions
This works for single-tenant systems. It breaks for multi-tenant:

Problem 1: Access Control

Anyone with access to the RAG system can retrieve any knowledge
No way to restrict based on role or tenant
Problem 2: Data Isolation

All data is mixed together
One tenant’s knowledge bleeds into another’s
Problem 3: Versioning

Documents are static
No way to track multiple versions

Problem 4: Audit Trail

No logging of who accessed what
No compliance trail
The Solution: Socratic-Knowledge
Socratic-knowledge is an enterprise RAG system built for multi-tenant environments.

Core Features
Feature 1: Multi-Tenancy

Complete data isolation between tenants:

Organization A:

├─ Documents (completely isolated)

├─ Knowledge base (separate)

├─ Agents (can only access org A knowledge)

└─ Users (can only see org A)

Organization B:

├─ Documents (completely isolated)

├─ Knowledge base (separate)

├─ Agents (can only access org B knowledge)

└─ Users (can only see org B)

System enforces: Organization A agents NEVER see Organization B documents
Feature 2: Role-Based Access Control (RBAC)

Fine-grained permissions:

Roles within Organization A:

Admin:

├─ Can view all documents

├─ Can edit all documents

├─ Can grant/revoke access

└─ Can view audit logs

Manager:

├─ Can view team documents

├─ Can edit team documents

├─ Can grant access to team

└─ Can view team audit logs

Employee:

├─ Can view shared documents

├─ Cannot edit

├─ Cannot grant access

└─ Cannot view audit logs

Example: Customer Support Agent

├─ Can access: Customer data, product docs, policies

├─ Cannot access: Financial data, employee records, future roadmap

└─ Can only see data for assigned customers

Feature 3: Document Versioning

Track changes over time:

Product Specification Document

Version 1 (Jan 2026):

├─ SKU: 123

├─ Price: €49.99

├─ Description: "Widget v1"

└─ Status: Active

Version 2 (Mar 2026):

├─ SKU: 123

├─ Price: €59.99 (updated)

├─ Description: "Widget v1.5" (updated)

└─ Status: Deprecated

Version 3 (May 2026):

├─ SKU: 456

├─ Price: €69.99 (new product)

├─ Description: "Widget v2"

└─ Status: Active

Agent using version 3 gets current info.

Legacy system using version 2 still works.

Historical queries can use historical versions.
Feature 4: Audit Logging

Complete compliance trail:

Audit Log:

2026-05-14 10:30:00 | RecommendationAgent | Retrieved | "Customer Preferences v2"

2026-05-14 10:30:15 | RecommendationAgent | Generated | "Recommendation 123"

2026-05-14 10:31:00 | UserNotificationAgent | Retrieved | "User Preferences"

2026-05-14 10:31:30 | UserNotificationAgent | Retrieved | "Marketing Policies"

2026-05-14 10:32:00 | AnalyticsAgent | Retrieved | "Sales Metrics (restricted)"

                                         ↑

                         Agent accessed restricted doc

                         System flagged as access denied

Query Audit Trail:

├─ Who accessed? RecommendationAgent

├─ When? 2026-05-14 10:30:00

├─ What did they access? Customer Preferences v2

├─ Which agent? ID: agent_456

├─ For which customer? Customer ID: cust_789

└─ Did they have permission? Yes (role: recommendation_engine)
Feature 5: Semantic Search

Find knowledge by meaning, not keywords:

Query: "How do we handle complaints?"

Traditional search matches:

├─ "complaint form" (keyword match)

├─ "complaints process" (keyword match)

└─ "customer complaints" (keyword match)

Semantic search understands meaning:

├─ "customer support escalation" (related meaning)

├─ "complaint resolution procedure" (related meaning)

├─ "unhappy customer handling" (semantic equivalent)

└─ "issue resolution framework" (underlying concept)

Returns more relevant results.

Finds knowledge agent wouldn't find with keywords.
Feature 6: Vector Embeddings

Store knowledge in a way that’s searchable and meaningful:

Document: "Our product is fast and reliable"

Traditional storage:

├─ Search for "fast": Found (keyword match)

├─ Search for "quick": Not found (different word)

└─ Search for "performance": Not found (synonym not used)

Vector embedding:

├─ Converts to vector: [0.23, -0.45, 0.89, ...]

├─ Search for "fast": Found (semantic similarity)

├─ Search for "quick": Found (semantic similarity)

└─ Search for "performance": Found (semantic similarity)

Agent searches for "How responsive is our product?"

System finds "Our product is fast and reliable"

Even though no exact words match.
Real Examples
Example 1: Customer Support
Customer support agent helps users:

Agent: CustomerSupportAgent

Permissions:

├─ View: Customer data, product docs, support policies

├─ Cannot: View financial data, employee records

└─ Data scope: Only assigned customers

Query: "User asks: 'How do I reset my password?'"

Agent retrieves:

Search knowledge base: "password reset"
Found documents (in order of relevance):
- "Password Reset Procedure v5" (current)
- "Troubleshooting: Can't Reset Password" (related)
- "Security: Password Requirements" (related)

Agent uses documents to respond to user.

Audit trail:

├─ Time: 2026-05-14 14:32:00

├─ Agent: CustomerSupportAgent

├─ Documents accessed: 3

├─ Customer: cust_123

└─ Outcome: User helped with password reset

System tracks: For compliance, debugging, improvement

Example 2: Policy Enforcement
Recommendation agent must follow policies:

Agent: RecommendationAgent

Permissions:

├─ View: Product specs, customer preferences, pricing

├─ Cannot: View employee data, internal discussions

└─ Data scope: All products, assigned customer

Query: Agent generates recommendation

Knowledge retrieval:

Product: "Widget v2"
- Spec: €69.99
- Availability: In stock

Policy: "No recommendations for out-of-stock items"
- Policy version: Current
- Status: Active
- Effect: Exclude out-of-stock

Customer preferences: "Likes blue products"
- Version: Current (updated yesterday)
- Effect: Prioritize blue

Agent generates recommendation respecting:

├─ Product specs (accurate pricing)

├─ Policies (follows rules)

└─ User preferences (personalized)

If policy changes, next query uses new policy.

No code change needed.
Example 3: Historical Context
Agent needs context from the past:

Query: "Why did we stop using AWS?"

Retrieves:

Version 1 (Jan 2026): "AWS Evaluation: Good for scaling"
Version 2 (Mar 2026): "AWS Issues: Cost overruns"
Version 3 (May 2026): "AWS Decision: Switch to GCP"

With versions, agent can understand:

What we thought originally
What changed (cost overruns)
What decision we made (switch)
Why (cost)

Historical context enables better decisions.
Implementation: Setting Up Knowledge Management
Step 1: Create Tenant and Knowledge Base
from socratic_knowledge import KnowledgeManager

km = KnowledgeManager()

Create organization (tenant)

org = await km.create_organization(

name="Acme Corp",

id="org_123"

)

Create knowledge base for organization

kb = await km.create_knowledge_base(

organization_id="org_123",

name="Acme Knowledge Base"

)
Step 2: Define Roles and Permissions

Define role: Customer Support Agent

support_role = await km.create_role(

organization_id="org_123",

name="customer_support",

permissions={

    "view": ["customer_data", "product_docs", "support_policies"],

    "edit": ["support_tickets"],

    "cannot_view": ["financial_data", "employee_records"]

}

)

Assign role to agent

await km.assign_role(

agent_id="support_agent_1",

role_id="customer_support",

organization_id="org_123"

)
Step 3: Upload Documents

Upload document

doc = await kb.upload_document(

title="Password Reset Procedure",

content="...",

version="5.0",

access_level="public"

)

Upload sensitive document

sensitive = await kb.upload_document(

title="Employee Compensation",

content="...",

version="1.0",

access_level="restricted",

allowed_roles=["admin", "hr"]

)
Step 4: Agent Retrieves Knowledge

Agent searches with permissions enforced

results = await kb.search(

query="How do we handle customer complaints?",

agent_id="support_agent_1",  # Permissions checked

organization_id="org_123"     # Tenant isolated

)

Returns only documents agent has access to

for result in results:

print(f"- {result.title} (v{result.version})")

Automatically logged for audit trail

Step 5: View Audit Trail

Check who accessed what

audit = await km.get_audit_log(

organization_id="org_123",

document_id="doc_456",

time_range=(start_date, end_date)

)

for entry in audit:

print(f"{entry.timestamp}: {entry.agent_id} accessed {entry.document_id}")

The Philosophy: Knowledge Should Be Trustworthy
Agents are only as good as the knowledge they’re given.

Socratic-knowledge ensures knowledge is:

Accurate (versions ensure you know which version you’re using)
Appropriate (access control ensures agents only see what they should)
Traceable (audit logging ensures you know where decisions came from)
Secure (multi-tenancy ensures data stays isolated)
With trustworthy knowledge, agents make trustworthy decisions.

Use Socratic-Knowledge for Enterprise RAG
The enterprise knowledge management system described in this post is production-ready and open source:

GitHub: https://github.com/Nireus79/Socrates
PyPI Package: https://pypi.org/project/socratic-knowledge/
Documentation: https://github.com/Nireus79/Socrates/tree/main/socratic-knowledge

Quick Start

Install the knowledge system

pip install socratic-knowledge

from socratic_knowledge import KnowledgeManager

km = KnowledgeManager()

Create knowledge base

kb = await km.create_knowledge_base(organization_id="org_123")

Upload documents

await kb.upload_document(

title="Product Specs",

content="...",

version="1.0"

)

Agent retrieves with permissions

results = await kb.search(

query="product specifications",

agent_id="agent_1",

organization_id="org_123"

)

for result in results:

print(f"- {result.title}")

Full examples and documentation: https://github.com/Nireus79/Socrates

The Complete Socratic Ecosystem
Socratic-knowledge is part of the complete Socrates AI system (11 modules, 2,300+ tests):

Socratic-nexus: Multi-provider LLM client
Socratic-morality: Constitutional governance with 13 modules
Socratic-agents: Multi-agent orchestration with conflict resolution
Socratic-knowledge: Enterprise RAG with multi-tenancy(this post)
Socratic-learning: Self-improving agents
Socratic-analyzer: Code quality analysis
Socratic-performance: Real-time monitoring
Socratic-workflow: Workflow orchestration
Socratic-conflict: Conflict resolution between agents
Socratic-docs: Auto-documentation
Socratic-maturity: Project maturity tracking
All modules work together seamlessly. Use individual packages or the complete platform.

Available on PyPI under MIT License: https://pypi.org/user/Nireus79/

Building a Reliable Agentic Loop: Retries, Tool Errors, and Knowing When to Stop

Wed, 15 Jul 2026 11:50:35 +0000

Every "build an AI agent" tutorial shows you the same twelve lines: a while loop, a call to the model, a check for tool calls, execute them, feed the results back, repeat. It runs, the agent uses a tool, you feel like a wizard. Then you put it in front of real inputs and it does one of three things: loops forever, retries a tool that returned a permanent 404 ten times in a row, or confidently finishes a task it never actually completed.

Here's the thing the tutorials leave out: the loop is the easy part. The reliability around the loop is the actual engineering. The happy path is a demo. What separates an agent you can deploy from an agent you have to babysit is entirely in the stuff that isn't the happy path — how it handles tool failures, how it decides it's done, and how it refuses to run forever.

This is a field guide to that reliability layer, in real Python. We'll start from the naive loop, then add the four things production agents need and demos never have: a stop condition that isn't just "no more tool calls," error handling that tells the difference between retry this and give up, a hard budget so the loop can't run away, and structured logging so you can see why it did what it did.

The naive loop — and exactly where it breaks

Here's the canonical agentic loop, the one everyone starts with:

def run_agent(client, messages, tools):
    while True:
        response = client.chat(messages=messages, tools=tools)
        messages.append(response.message)

        if not response.tool_calls:
            return response.message.content  # model answered — done

        for call in response.tool_calls:
            result = TOOLS[call.name](**call.arguments)   # <-- everything wrong lives here
            messages.append(tool_result(call.id, result))

The structure is correct. It's the unstated assumptions that kill it in production:

It trusts the tool to succeed. TOOLS[call.name](**call.arguments) has no error handling. One exception and the whole agent crashes mid-task.
It trusts the model to stop. The only exit is "the model produced no tool calls." A model stuck in a plan-execute-replan spiral never hits that, and while True means forever.
It trusts the model to call real tools with valid arguments. TOOLS[call.name] KeyErrors on a hallucinated tool name; **call.arguments TypeErrors on a malformed argument object.
It has no memory of cost. No token budget, no step cap. A runaway loop is a runaway bill, discovered on the invoice.

Every one of these is a real incident waiting to happen. Let's close them one at a time.

Fix 1: Tool errors are messages, not crashes

The single most important shift: a tool failure is not an exception that should propagate — it's information the agent needs. When a tool fails, the model should see the failure, reason about it, and either retry differently or route around it. That only works if you catch the error and feed it back as a tool result the model can read.

def execute_tool(call):
    fn = TOOLS.get(call.name)
    if fn is None:
        # Hallucinated tool — tell the model, don't crash.
        return f"Error: tool '{call.name}' does not exist. Available: {list(TOOLS)}"
    try:
        return str(fn(**call.arguments))
    except TypeError as e:
        # Bad arguments — the model can correct these if you let it see them.
        return f"Error: invalid arguments for '{call.name}': {e}"
    except Exception as e:
        # Real failure — return a legible, model-readable error.
        return f"Error calling '{call.name}': {e}"

Notice the errors are written for a model to read, not for a log file. "Error: tool 'serch_web' does not exist. Available: ['search_web', ...]" lets the model notice its typo and self-correct on the next turn. A raw KeyError traceback teaches it nothing. This is the difference between an agent that recovers and one that dies: your error strings are part of the agent's reasoning surface, so write them like instructions, not like stack traces.

But not every error should be retried. This is where naive agents burn budget: a 404 Not Found or 403 Forbidden is permanent — retrying it is pure waste. A 429 Rate Limited or 503 is transient — retrying (with backoff) is exactly right. Teach the distinction:

PERMANENT = {400, 401, 403, 404, 422}   # don't retry — the request itself is wrong
TRANSIENT = {429, 500, 502, 503, 504}   # retry with backoff — the world is temporarily busy

def classify_error(status):
    if status in PERMANENT:
        return "permanent"   # tell the model to try a different approach
    if status in TRANSIENT:
        return "transient"   # retry the same call after a wait
    return "unknown"

For transient failures, wrap the call itself in bounded retries with exponential backoff — but cap the attempts. "Retry until it works" is how you turn a brief outage into an infinite loop. Knowing which failures the model should reason about versus which the infrastructure should silently retry is a core design decision in building autonomous AI agents, and getting it wrong is the most common reason agents feel flaky.

Fix 2: Stop conditions — "no tool calls" is not enough

The naive loop has exactly one exit. Real agents need several, because there are several distinct ways an agent's run should end — and only one of them is "success."

def run_agent(client, messages, tools, max_steps=15, token_budget=100_000):
    steps = 0
    tokens_used = 0

    while True:
        # --- Stop condition A: step cap (guards against loops) ---
        if steps >= max_steps:
            return finish("max_steps_exceeded", messages)

        # --- Stop condition B: token budget (guards against cost) ---
        if tokens_used >= token_budget:
            return finish("budget_exhausted", messages)

        response = client.chat(messages=messages, tools=tools)
        tokens_used += response.usage.total_tokens
        steps += 1
        messages.append(response.message)

        # --- Stop condition C: the model answered (the good exit) ---
        if not response.tool_calls:
            return finish("completed", messages, answer=response.message.content)

        # --- Otherwise: execute tools and continue ---
        for call in response.tool_calls:
            result = execute_tool(call)
            messages.append(tool_result(call.id, result))

Four ways out, and the difference between them matters to your caller:

Completed — the model produced a final answer. Success.
Max steps exceeded — the loop hit its ceiling. This is a failure signal, not a silent stop: surface it, because it usually means the task was underspecified or a tool is misbehaving.
Budget exhausted — cost cap hit. Same: a signal, not a shrug.
(Add as needed) explicit finish tool — for complex agents, give the model an explicit submit_answer tool so finishing is a deliberate act, not merely the absence of tool calls. This is more robust than inferring completion, because "no tool calls" can also mean the model got confused.

The critical anti-pattern: while True with a single exit is a production incident with a countdown timer. Always bound the loop by steps and by cost. And when you exit on a bound rather than on success, say so — a caller that can't tell "here's your answer" from "I gave up after 15 steps" will happily ship the second as if it were the first.

Fix 3: Detecting the loop-within-the-loop

The subtler failure isn't infinite iteration — your step cap catches that. It's the agent that makes progress-shaped motion without progressing: calling the same tool with the same arguments three times, or ping-ponging between two tools forever. The step cap eventually stops it, but you've burned 15 steps to accomplish nothing.

Detect it by tracking recent actions and noticing repetition:

from collections import deque

recent = deque(maxlen=4)

def is_stuck(call):
    signature = (call.name, str(sorted(call.arguments.items())))
    if recent.count(signature) >= 2:      # same exact call, 3rd time
        return True
    recent.append(signature)
    return False

When you detect a stuck agent, the fix isn't to kill it — it's to interrupt with a nudge: inject a message like "You've called search_web('AI agents') three times with identical arguments and gotten the same result. Try a different approach or provide your best answer with what you have." Often the model just needed to be told it was in a rut. Deciding what the agent remembers across steps — the running transcript, prior tool results, what to summarize versus keep verbatim — is its own discipline. Let the message history grow unbounded and you'll blow the context window mid-task; that trade-off between memory and budget is exactly what context engineering for AI agents is about, and it's where loop reliability and cost control meet.

Fix 4: You can't fix what you can't see

An agent is a non-deterministic system making autonomous decisions. When it does something wrong — and it will — "it just didn't work" is useless. You need to reconstruct what it did and why. That means logging every step as a structured record, not a print statement:

def log_step(step, response, tokens_used):
    log.info("agent_step", extra={
        "step": step,
        "tool_calls": [c.name for c in response.tool_calls],
        "finish_reason": response.finish_reason,
        "tokens_this_step": response.usage.total_tokens,
        "tokens_cumulative": tokens_used,
    })

With this, a failed run becomes a readable trace: step 1 called search_web, step 2 called a tool that doesn't exist (typo — good, now you know), step 3 retried, step 4 hit the step cap. Without it, you're re-running a non-deterministic system and praying it misbehaves the same way twice. Structured, step-level observability is what makes agents debuggable instead of mysterious — and it's the same instinct behind treating agent behavior as something you evaluate systematically in production rather than eyeball. If you can't answer "why did the agent do that?" from your logs alone, you don't have observability — you have hope.

Putting it together

The reliable loop isn't more clever than the naive one — it's more defensive. Same twelve-line skeleton, wrapped in four guarantees:

Tool errors become messages the model can reason about — with permanent-vs-transient classification so it retries the right things and gives up on the rest.
Multiple stop conditions — success, step cap, budget cap — and it tells the caller which one fired.
Loop-within-loop detection that nudges a stuck agent instead of letting it spin to the step limit.
Structured per-step logging so every run is reconstructable after the fact.

None of this is exotic. It's the ordinary engineering of turning a non-deterministic component into a dependable system — the same discipline you'd apply to any service that calls unreliable dependencies, which is exactly what a tool-using agent is. Building agents that survive contact with production, wiring them to real tools and real budgets, is the whole subject of a hands-on course on building AI applications in Python, and the reliability layer above is the part that separates a weekend demo from something you'd put a budget behind. If your day-to-day is coding agents specifically, the patterns for stop conditions, subagents, and error recovery go deeper still in a course on agentic coding with Claude Code.

Conclusion

The agentic loop is genuinely simple — that simplicity is exactly why so many agents ship broken. The while True and the tool dispatch are the parts you can write in your sleep; the parts that decide whether the thing works in front of real users are the ones the tutorials skip. What happens when a tool fails. How the loop knows it's done. What stops it from running forever. Whether you can tell, afterward, why it did what it did.

Start from the naive loop, then earn each guarantee: errors as messages, bounded stops, stuck-detection, structured logs. Add them and your agent stops being a party trick that works on the demo input and becomes something you can point at real work and trust — or at least trust to fail loudly, cheaply, and legibly, which in autonomous systems is the same thing as trustworthy.

Write the loop in five minutes. Spend the rest of the week on everything around it. That's where the reliability was hiding the whole time.

Sources & further reading:

Anthropic — Building effective agents
Model Context Protocol — Specification and documentation
Yao et al. — ReAct: Synergizing Reasoning and Acting in Language Models

This article is educational content. Agent frameworks and provider SDKs evolve quickly; treat the code above as illustrative patterns and validate against current library documentation before shipping.

Building a multi-agent document-search copilot - Part 2: adaptive Hybrid, post-rank permission gate

Tue, 14 Jul 2026 13:19:33 +0000

Building a multi-agent document-search copilot — Part 2: adaptive Hybrid, and a permission gate after the rank

This is Part 2 of two. Part 1 opened on a document-search copilot whose v1 produced muddy results — two retrieval lanes fused into one rerank, where metadata rows that carry no text corrupted the relevance scores. The fix was two reframes: collapse the router into one structured Bedrock call (with a deterministic fallback), and pick exactly one retrieval strategy per query — MetadataOnly, ContentOnly, or Hybrid — instead of fusing the lanes. We left off at the one strategy that refuses to be tidy: Hybrid, where the user wants a topic and filters at the same time.

Picking up where Part 1 stopped, we're now in the back half of the same pipeline — the direct_search → rerank → permission_filter → finalize_results tail of the search graph. Two decisions left: how Hybrid actually retrieves, and where permissions get enforced.

3️⃣ Adaptive Hybrid: peek once, then pick filter-first or rank-then-filter

Hybrid is the interesting one, because "topic + filters" is genuinely a tradeoff and not a single right answer. There was a real debate about it — the kind that goes a few rounds on a whiteboard: run the filter and the rank in parallel (fast, but you have to reconcile two sets), or in sequence (precise, but slower)? We resolved it by refusing to pick — and routing on selectivity instead.

The trick is one cheap question asked once: how many documents match the filters? direct_search peeks the filtered-universe size from the Documents service a single time, with a threshold (default 1000), and branches:

elif strategy == "Hybrid":
    threshold = cfg.get("filter_first_threshold", 1000)
    id_page = cfg.get("filter_id_page_size", 100)
    scope_ids, total_records = await _fetch_filtered_ids(metadata_filters, threshold, id_page)
    if scope_ids is not None:
        candidates = await _content_hits(scope_ids=scope_ids)   # filter-first
        filters_enforced_upstream = True
    else:
        candidates = await _content_hits()                       # rank-then-filter

Bounded set (≤ threshold) → filter-first. The peek returns the actual id set. The semantic query is scoped to exactly those ids via an OpenSearch terms id-scope (over the version field on chunks, the parent field on attachment chunks). Every filter is enforced authoritatively at the source — nothing off-filter can possibly rank, and nothing on-filter is missed. It sets a flag so the later permission gate knows not to re-filter:
```
# filter-first scoped the topic ranking to the service-filtered id set, so the filters
# are already enforced authoritatively; permission_filter must NOT re-enforce them.
filters_enforced_upstream = True
```
Broad set (> threshold) → rank-then-filter. Scoping a semantic query to thousands of ids would be unbounded and slow, so instead we rank the topic unscoped and drop the off-filter hits afterward, locally, with _row_passes_filters — a faithful mirror of the Documents service's operation semantics across every searchable column (string / number / date / boolean / user_list, with contains / equal / greaterThan / the rest):
```
def _row_passes_filters(row: dict, filters: list[dict]) -> bool:
    """Faithful local mirror of the Documents service filter semantics ...
    any hit whose row fails a hard filter is dropped, fail-closed."""
    return all(_row_passes_one(row, f) for f in filters)
```

The thing that makes this safe rather than a sloppy approximation: when the filtered set is huge, the filter is nearly a no-op anyway (it matches most of the library), so the recall ceiling from ranking-first-then-dropping is harmless. The case where dropping would lose good matches — a selective filter — is exactly the case that takes the filter-first path, where nothing is dropped. The branch picks the strategy where its own weakness doesn't apply. I'll be honest: it took me a beat to trust this — it feels like a cheat until you sit with it — but the two failure modes genuinely cancel out.

	filter-first (selective)	rank-then-filter (broad)
Filtered universe	≤ threshold (default 1000)	> threshold
Semantic query	scoped to the filtered id set	unscoped
Filters enforced	at the Documents service (authoritative)	locally, in `permission_filter`
Recall risk	none — nothing off-filter can rank	harmless — filter matches most docs
Round trips for the peek	one (read the id set)	one (just the count, then bail)

One peek, no extra round trips, and the precision/latency tradeoff is made explicit and data-driven instead of being a coin flip baked into the architecture.

The pattern, generalized: when you can't decide between two retrieval strategies, measure the one property that distinguishes them (here, filter selectivity) with a single cheap probe, and route on it. Each branch is chosen precisely where its failure mode is benign.

4️⃣ Cohere rerank, then a fail-closed view gate — and why "after" is fine

The content path is ranked by Cohere (over the top rerank_candidates, default 30), with the OpenSearch fused score / RRF as the fallback when the reranker is unreachable:

results = await bedrock_client.rerank(query=query_text, documents=documents, model_id=model_id)
# ... on any failure or empty result -> _fallback() sorts by the OpenSearch raw score

Metadata results skip the reranker entirely — there's nothing to score — and keep the Documents service order:

if all(c.get("match_kind") == "metadata" for c in candidates):
    return {"reranked": candidates, "cached_top_30": candidates}

Then comes the part that makes security people lean forward in their chair: the per-document view-permission gate runs after rerank, not before retrieval. Stay with me — it's safe, and the reason it's safe is the whole point. permission_filter takes the reranked content hits, confirms each one is viewable via a single bulk export_by_ids call against the system of record, and drops the ones that aren't — fail-closed, so if the service is unreachable, every content hit is dropped rather than leaked:

rows = await documents_service.export_by_ids(content_ids, jwt)
for row in rows:
    dvid = row.get("document_version_id")
    if dvid:
        viewable.add(dvid)
# ... survivors = metadata rows (trusted) + content hits whose id is in `viewable`

Why is gating after rerank safe? Because there are two different boundaries doing two different jobs:

Tenant isolation is the hard wall, and it's enforced at retrieval. Every OpenSearch query is scoped by member_id, and that id comes from the JWT — never from the request body, never from the index. A user physically cannot retrieve another tenant's vectors. That boundary is upstream and absolute.
The post-rerank gate is the soft boundary: per-document view visibility within your own tenant. Drafts, restricted categories, documents shared with three people. That nuance changes hourly, so it does not live in the index — it's checked live, after retrieval, against the source of record.

Rerank only orders the candidates. It exposes nothing — every candidate it sees already passed the tenant wall, and the set it's ordering is bounded (the reranked top set), so the view gate is one bounded bulk call, not a fan-out. The metadata lane skips the gate entirely because its rows came back already view-filtered by the service's own SQL — re-checking trusted rows would be wasted work. Identity from the token, tenant at retrieval, view-visibility after, fail-closed throughout.

The seam I owned on the frontend: a null relevance score

Here's a small but real consequence of "one strategy per query" that landed squarely on my side of the stack — and nearly slipped past me. Content hits carry a relevance score. Metadata hits don't — a satisfied filter is not a relevance measurement, and showing the OpenSearch fused score (which for an exact field match is ~0) would read as "barely relevant" for what is actually a perfect match. So a metadata hit emits a null score in the SSE payload, and the UI has to render that as "matched your filter," not "0% relevant":

def _shape_source_for_emit(s: dict) -> dict:
    # A metadata match is a satisfied filter, not a relevance measure, so its score stays null
    # (the card shows just the match reason); content matches keep their fused/rerank score.
    score = None if s.get("score_signal") == "metadata" else (
        s.get("aggregate_score") or s.get("score") or 0.0
    )
    ...

It's a tiny detail, but it's the kind of thing that breaks if the frontend assumes "every result has a number." A null score is a first-class state — "this matched a filter, there's no relevance to show" — not a missing value to default to zero. Defaulting it to 0.0 would have quietly buried every exact metadata match at the bottom of the list, sorted below loosely-relevant content — a perfect match dead last, and no error anywhere to tell you why. The contract had to carry the absence on purpose.

The throughline across all four decisions — the two in Part 1 and the two here — is the same instinct: make the system commit to one clear shape per turn, and make the tradeoffs explicit instead of averaging them away. One router call instead of three. One retrieval strategy instead of a fused blend. One selectivity probe instead of a parallel-vs-sequential guess. One bounded permission check after a bounded rerank. The muddy-results version tried to do everything at once and let the reranker sort it out. It couldn't. Picking, on purpose, is what made the rank mean something again.

Thanks for sticking through both parts. If you've fought the same muddy-rank fight, I'd genuinely love to hear how you cut yours — the failure modes are weirdly universal. Come say hi on LinkedIn.

Missed the setup? Part 1 covers the muddy-results problem, the single-call router, and the reframe to one strategy per query.

Building a multi-agent document-search copilot — Part 1: muddy results, and one strategy per query

Tue, 14 Jul 2026 13:19:14 +0000

Building a multi-agent document-search copilot — Part 1: muddy results, and one strategy per query

The first version ranked documents badly — and worse, it ranked them badly in a way that looked fine on the architecture diagram. Those are the bugs that get under my skin: every box is green, every arrow points the right way, and the answer is still wrong.

We were building a chat copilot over a regulated document store — the kind where a user types "show me my effective SOPs about equipment cleaning" and expects the right handful of documents back, ranked, with an excerpt and a reason. The v1 design did the obvious thing: run two retrieval lanes in parallel — a structured metadata lane and a semantic content lane — union the hits, rerank the union, render. Clean diagram. Muddy results. We'd open the demo, the pipeline would light up green end to end, and the list that came back was mush: the metadata rows polluted the semantic rank, the relevance scores stopped meaning anything, and there was no clean ordering left to show the user. The architecture was elegant. The experience was not.

This is a two-part story of how that became v2: one strategy per query, never mixed, a router that's a single structured-output call, and a Hybrid path that peeks at the data before it decides how to retrieve. It's an architecture post, so I'll keep it anchored in the specific decisions that actually moved — not a generic "how to build RAG" walkthrough. Part 1 (this post) is the problem and the first two reframes. Part 2 is the hard case — Hybrid — and the permission model.

️ The series at a glance

This is Part 1 of 2.

Part 1 (this post) — the problem and the first two reframes:

️ v1 fused two parallel lanes into one rerank → muddy. Metadata rows have no text; a reranker scores text; fusing them corrupts the one number the UI depends on.
The router is one Bedrock structured-output call, not three sequential hops. Route + rewritten query + strategy + filters come back together, with a deterministic fallback. The catch: the merged task got too hard for a small model, so it runs on a bigger one.
v2 picks exactly one shape per query: MetadataOnly, ContentOnly, or Hybrid. The lanes are never unioned or cross-scored.

Part 2 — the hard case and the safety model:

⚖️ Hybrid is adaptive. It peeks the filtered-universe size once (threshold 1000) and routes on selectivity: small set → scope the semantic query to those ids (filter-first, authoritative); big set → rank unscoped and drop off-filter hits locally (rank-then-filter, harmless recall ceiling).
The view-permission gate runs after rerank, and that's safe — because tenant isolation is enforced earlier, at retrieval, from the token.

The flow, end to end

Here's the whole turn. A message comes in, the supervisor routes it, the search graph runs one retrieval shape, reranks, gates on permissions, and finalizes for the UI:

                 ┌─────────────────────────────────────┐
   user turn ──► │ PLANNER (one Bedrock call)           │
                 │ route + rewrite + strategy + filters │
                 └──────────────┬──────────────────────┘
                                │ deterministic fallback if it fails
                                ▼
         ┌─────────────────── retrieval graph ───────────────────┐
         │   classify_intent                                     │
         │        │  (NoMatch / NeedsClarification skip ahead)   │
         │        ▼                                              │
         │   retrieve        ── picks ONE: Metadata / Content /  │
         │                      Hybrid (adaptive on selectivity) │
         │        ▼                                              │
         │   rank_results    ── Cohere over content; metadata    │
         │                      passes through unscored          │
         │        ▼                                              │
         │   access_gate     ── fail-closed VIEW gate on the     │
         │                      content lane only                │
         │        ▼                                              │
         │   format_response ── floor, shape, SSE                │
         └───────────────────────────────────────────────────────┘

The node order is exactly that, straight from the graph definition:

builder.add_edge(START, "query_rewriting")
builder.add_edge("query_rewriting", "intent_detection")
builder.add_conditional_edges(
    "intent_detection",
    _route_after_intent_detection,
    {"retrieve": "retrieve", "finalize_results": "finalize_results"},
)
builder.add_edge("retrieve", "rerank")
builder.add_edge("rerank", "permission_filter")
builder.add_edge("permission_filter", "finalize_results")

Four decisions made this work. Two of them are in this post; the other two are Part 2. I'll take them in order.

1️⃣ The router is one call, not three

The v1 router ran three sequential Bedrock hops per turn: one to pick a route, one to rewrite the user's query into a clean retrieval string, one to classify intent and extract filters. Three round trips, in series, before any retrieval even started. Each one waits on the last — and the user just watches a spinner the whole time.

My pushback in review was simple: don't run sequential model calls when one call can return the whole decision. A router's job is to emit a structured plan. There's no reason route, rewrite, and intent need to be three separate inferences — they're three fields of one object. So we collapsed them. The supervisor now makes a single structured-output call that returns a typed plan, which gets adapted into the downstream routing contract:

result = await bedrock_client.invoke_with_structured_response_with_fallback_async(
    messages=[SystemMessage(content=system), HumanMessage(content=message)],
    response_structure=Plan,
    chat_model_chain=model_chain,
    max_tokens=1024,
    temperature=0.0,
)

One call, temperature=0.0, and everything the downstream graph needs comes back together: the route, the rewritten query, the strategy, the structured filters. The RouterDecision it adapts into is the contract the search graph consumes:

class RouterDecision(BaseModel):
    route: Literal["documents.search", "documents.doc_context", "general.help"]
    rewritten_query: str = Field(default="")
    strategy: SearchStrategy  # MetadataOnly | ContentOnly | Hybrid | NoMatch | NeedsClarification
    search_value: str = Field(default="")
    filters: list[DocumentFilter] = Field(default_factory=list)
    ...

The twist that bit back. Merging three easy classifications into one harder one means the model now has to do all of it in a single pass — and the small/cheap model we wanted started getting it wrong. So the router moved up to a stronger (and slower) model. The "3 calls → 1 call" math promised a big latency win; the model upgrade promptly taxed a chunk of it back. (Presenting a "latency win" that your own model bump immediately claws into is a humbling little moment — I recommend the experience to no one.) We shipped anyway, because correctness moved the right way and the architecture got dramatically simpler — and because of the second non-negotiable below.

The reusable lesson: a call-merge latency win can be partly clawed back by the accuracy upgrade the merge forces. Budget for that. And always keep a deterministic fallback.

The deterministic fallback is the part I'd flag for anyone copying this. The structured call returns None on any failure (the model is down, the output won't parse, the plan isn't exactly one step), and the caller drops to a non-LLM router:

fallback = route_request(body, intent_resolver=resolve_doc_intent)
plan, router_usage = await supervise_turn(...)
routing = plan_to_router_decision(plan)
# routing is None -> use fallback

The copilot never hangs on a flaky router call. If the smart path can't answer, a dumb-but-reliable path does. That's what lets you run the router on a heavier model without making it a single point of failure.

2️⃣ One strategy per query — the reframe that fixed the muddy results

This is the heart of the v2 change, and the part I argued hardest for — loudly, in more than one meeting.

The v1 retrieval ran two member-scoped lanes in parallel — an OpenSearch hybrid lane (vector + BM25) and a structured metadata lane — and fused them into one set before reranking. The intuition was "more recall is better, let the reranker sort it out." It doesn't work, and the reason is specific: a metadata hit and a content hit are not the same kind of object.

A cross-encoder reranker scores text. You hand it a query and a list of passages, it returns a relevance number per passage. A content chunk has text. A metadata row — "status = effective, author = me" — has no passage. When you stuff that row into the reranker by stringifying a title or a key-id, you get back a number that means nothing, on the same 0-1 scale as the real text scores. Nothing downstream can tell the calibrated score from the garbage one. The rank looks plausible and is quietly wrong.

So v2 picks exactly one retrieval shape per query and runs only that. The strategy comes from the router as a literal:

SearchStrategy = Literal[
    "MetadataOnly", "ContentOnly", "Hybrid", "NoMatch", "NeedsClarification"
]

Strategy	What it means	How it's retrieved	Has a relevance rank?
`MetadataOnly`	structured filters only ("my effective SOPs")	Documents service, filtered rows in service order	No — a satisfied filter isn't a relevance signal
`ContentOnly`	topic search ("policy on cleaning between batches")	OpenSearch hybrid over chunks, unscoped	Yes — Cohere rerank
`Hybrid`	topic and filters ("effective docs about cleaning")	adaptive (Part 2)	Yes — Cohere rerank
`NoMatch`	not a document query (greeting, capability question)	skips retrieval entirely	n/a
`NeedsClarification`	a document query too vague to retrieve ("show me stuff")	skips retrieval, asks a clarifying question	n/a

The lanes are never unioned or cross-scored anymore — retrieve says so in plain terms:

# The lanes are never unioned or cross-scored: a content turn returns content
# candidates, a metadata turn returns metadata candidates.

There's a nice side effect in the graph: NoMatch and NeedsClarification short-circuit straight to finalize_results, skipping retrieval, rerank, and the permission gate. A greeting shouldn't make the UI flash "Searching... / No matches" and shouldn't cost three round trips.

def _route_after_intent_detection(state) -> str:
    return ("finalize_results"
            if state.get("intent_strategy") in ("NoMatch", "NeedsClarification")
            else "retrieve")

The honest cost. Going single-strategy meant deprecating the old global keyword lane entirely. The practical fallout: custom-field values are no longer reachable as a filter, because the content-keyword fold that used to (badly) cover them is gone. That's a real regression on a narrow feature, parked until a dedicated Documents endpoint for custom fields lands. It stings to ship a regression on purpose — but I'd take a clean rank with one honest, documented gap over a muddy rank that hides a dozen.

If you take one thing from this post: don't fuse object types into a single rerank set just to maximize recall. Pick the retrieval shape that matches the query, and run only that. Recall you can't rank cleanly is recall you can't show.

To be continued

So far the story has a tidy shape. One router call instead of three. One retrieval strategy per query instead of a fused blend. Pick MetadataOnly, or ContentOnly, and run only that. Clean.

But I skipped the strategy that refuses to be tidy — the one where the user genuinely wants both at once. "Effective SOPs about equipment cleaning" is a topic and a filter, and you can't honor it by picking a single lane. Running the filter and the rank in parallel is fast, but you have to reconcile two sets. Running them in sequence is precise, but slow. There's no single right answer — which is exactly why it's the interesting part.

That's Hybrid, and in Part 2 a single cheap question turns that coin-flip into a data-driven decision. Then there's the permission gate that runs in a spot that makes security people flinch on first read — after the rank — and why it's actually safe. Finally, the one place all of this landed on my side of the stack: a null relevance score that the frontend has to treat as a first-class state, not a missing number.

Part 2 is live: adaptive Hybrid, and the permission gate that runs after the rank.

Thanks for reading all the way through 🙌 If you've hit the same muddy-rank problem — a pipeline that's green end to end and still returns mush — I'd love to compare notes on how you untangled yours. Come say hi on LinkedIn.

AI Fundamentals - Part 3: Giving AI Knowledge Beyond Its Training

Sun, 12 Jul 2026 06:45:12 +0000

In Part 2, we learned why AI sometimes hallucinates. One of the biggest reasons is that an LLM can only answer based on what it learned during training and the information available in its context window. We also introduced grounding-providing the model with reliable information at runtime instead of expecting it to know everything.

But that raises an important question: Where does that information come from? Modern AI applications don't simply dump an entire database or a thousand-page PDF into the prompt. Instead, they first identify the most relevant pieces of information and only send those to the model. In this article, we'll learn how that works.

Running Example

Let's continue building our AI-powered Travel Planner. So far, it can answer general travel questions using the knowledge it learned during training. Now we want to make it much smarter by uploading several documents into our application:

Lonely Planet's Japan travel guide
A PDF containing train schedules
A document listing recommended local restaurants
Hotel information
Internal travel policies for our company

Together, these documents contain hundreds of pages. Now the user asks:

I'm staying near Tokyo Station. Which ramen restaurant from our travel guide is within walking distance and is known for vegetarian options?

Somewhere in those hundreds of pages is the answer. The challenge is no longer generating text-it's finding the right information first.

The Problem: An LLM Can't Read Your Entire Knowledge Base Every Time

A common misconception is that AI applications simply send all their documents to the model. Imagine our travel guide contains 450 pages, thousands of restaurant listings, hotel descriptions, transportation details, and sightseeing recommendations. Sending all of that to the LLM every time someone asks "Where should I eat tonight?" creates several problems.

First, many documents are simply too large to fit inside the model's context window. Second, even if they did fit, making the model read hundreds of irrelevant pages just to answer a simple question is expensive, slow, and inefficient. Instead, we want to send only the information that matters. The question becomes: How do we find the right few paragraphs from hundreds of pages?

Your first instinct might be: "Why not just search the documents?" That's exactly what we'll explore next.

Keyword Search: The Traditional Approach

Before AI became popular, most applications relied on keyword search. The idea is simple: the user searches for a word, and the application finds documents containing that exact word.

For example, given this travel guide text:

Tokyo has several excellent vegetarian ramen restaurants...
Mount Fuji is one of Japan's most iconic landmarks...
The Shinkansen connects Tokyo and Kyoto...

If the user searches for "vegetarian ramen", the search engine finds the first paragraph because it contains both words. While traditional keyword search is simple, fast, and easy to understand, it falls short when matching meaning instead of exact text.

When Keyword Search Falls Short

Synonyms and Meaning (Example 1): If a user asks "Where can I eat meat-free ramen?" and our document says "Vegetarian ramen is available near Tokyo Station", a traditional keyword search may completely miss this result because it searches for exact words, not meaning. It doesn't know that meat-free, vegetarian, and plant-based are closely related concepts.
Name Variations (Example 2): If a user asks to "Suggest a place near Fujisan", and the document says "A day trip to Mount Fuji is highly recommended", keyword search often fails because it doesn't understand that Fujisan and Mount Fuji refer to the same place.
Terminology Differences (Example 3): If a user asks "Where can I find authentic local noodle shops?" and the guide says "Several ramen restaurants popular with residents...", keyword search fails to connect noodle shop with ramen restaurant.

The real limitation is that keyword search compares text, while humans compare meaning. This is why search systems often return dozens of irrelevant results or miss the best result entirely, as you've likely experienced when searching documentation using a word that differs from the author's chosen terminology.

Semantic Search: Searching by Meaning

This is where AI changes everything. Instead of asking "Does this document contain the same words?", semantic search asks "Does this document mean the same thing?"

Let's return to our example. The user asks "Where can I eat meat-free ramen near Tokyo Station?" and the travel guide says "Ichiran offers vegetarian ramen a short walk from Tokyo Station." A semantic search understands that meat-free and vegetarian express nearly the same idea, allowing it to retrieve the correct passage even though the exact words are different.

But computers don't naturally understand meaning; they understand numbers. How can a computer determine that vegetarian and meat-free are closely related? That's where embeddings come in.

Embeddings: Turning Meaning into Numbers

An embedding is a numerical representation of text that captures its meaning. Think of it as a fingerprint-not of the exact words, but of what those words represent.

For example, these three phrases all have different wording but express similar ideas, so their embeddings end up being very similar:

Vegetarian ramen
Meat-free noodles
Plant-based ramen

Conversely, phrases like Mount Fuji, Flight booking, and Currency exchange produce embeddings that are quite different because they represent completely unrelated concepts. The key takeaway is: embeddings represent meaning, not exact wording.

Why Not Compare the Original Text?

Let's go back to our Travel Planner. Suppose our travel guide contains:

Ichiran offers vegetarian ramen near Tokyo Station.

Now imagine two different users:

User A: Recommend vegetarian ramen.
User B: Where can I find meat-free noodle restaurants?

A computer comparing plain text sees very few matching words between User B's query and the guide. A traditional keyword search might decide they are unrelated. Embeddings solve this by converting both pieces of text into numerical representations where similar meanings end up close together.

Semantic Similarity: Measuring Meaning Instead of Words

Once text is converted into embeddings, comparing sentences becomes much easier. Instead of asking if they contain the same words, we ask: How similar are their embeddings? If two embeddings are close together, they are likely talking about similar ideas; if they are far apart, they are probably unrelated.

A useful mental model is to imagine every sentence placed on a giant map. Similar ideas appear close together, while unrelated concepts are placed farther apart:

                  Travel

                      ●
                Visit Japan

           ●                   ●
  Vegetarian Ramen      Tokyo Station

                ●
         Meat-free Noodles


--------------------------------------------


                      ●
              JavaScript Closures

                                ●
            Docker Networking

Notice how the travel-related concepts naturally cluster together, while completely unrelated software concepts appear elsewhere. This is exactly what embeddings help achieve.

Cosine Similarity: Finding the Closest Match

Now that every sentence has an embedding, we need a way to compare them mathematically. One of the most common techniques is cosine similarity.

Suppose our Travel Planner has these restaurant descriptions stored:

A: Vegetarian ramen near Tokyo Station.
B: Traditional sushi restaurant.
C: Italian pizza in Osaka.

If the user asks for "Meat-free noodles near Tokyo Station", the embeddings might compare like this:

Restaurant	Similarity
Vegetarian ramen near Tokyo Station	0.96
Traditional sushi restaurant	0.41
Italian pizza in Osaka	0.09

The higher the score, the closer the meanings. Notice that the user never mentioned the word vegetarian, yet Restaurant A is still identified as the best match because the meaning is similar.

Mental Model

In a library, keyword search works like looking up every book containing the word ramen. Semantic search works by finding books discussing the same idea, even if they use completely different words.

Where Do These Embeddings Come From?

Fortunately, you don't have to calculate these numbers yourself. Just as LLMs specialize in generating text, there are embedding models that specialize in generating embeddings. Instead of producing a paragraph, an embedding model takes text and produces a list of numbers representing its meaning:

User Text ("Vegetarian ramen...") ──► Embedding Model ──► [0.18, -0.42, 0.91, ...]

The resulting list (or vector) usually contains hundreds or thousands of numbers. You never need to interpret them yourself; their only purpose is to make mathematical similarity comparisons possible.

Chunking: Breaking Large Documents into Pieces

How do we generate embeddings for a 500-page travel guide? We don't feed the entire book into the embedding model at once. Embedding models have strict input limit constraints (typically measured in tokens), and comparing a whole book's meaning is too vague to be useful. If a user asks about a specific ramen restaurant, we want to retrieve the exact paragraph discussing it, not a 500-page book embedding.

So, we break the document down into smaller, logical pieces in a process called chunking. A chunk could be a single paragraph, a section of 100-200 words, or a sliding window of text (e.g., 200 words with a 20-word overlap to ensure context isn't cut off at boundaries).

Mental Model

Think of chunking like cutting a long movie reel into short, thematic scenes. If you want to find the scene with Mount Fuji, it is much easier to search and index short clips than searching the entire three-hour film.

💡 Developer's Takeaway

Choosing the right chunk size is a balancing act. If chunks are too small, they might lose context (e.g., a sentence saying "It is delicious" without mentioning what is delicious). If they're too large, the specific details get averaged out, and search accuracy drops.

Vector Databases: Storing and Indexing Meanings

Once we have broken our travel guide into thousands of chunks and generated an embedding for each, we need a place to store them. Traditional databases (like PostgreSQL or MySQL) are optimized for finding exact matches like numbers, dates, or strings. They aren't designed to compare thousands of high-dimensional lists of numbers to find which ones are "closest" in meaning.

That's where Vector Databases come in. A vector database is designed specifically to store and index high-dimensional vectors (embeddings) and search through them based on mathematical similarity (like cosine similarity) extremely fast.

Common vector database setups include:

Dedicated vector databases (Pinecone, Milvus, Qdrant, Chroma)
Vector extensions for traditional databases (like pgvector for PostgreSQL)

💡 Developer's Takeaway

You don't need to write complex search algorithms yourself. A vector database handles the indexing and allows you to query it using an embedding, returning the most semantically similar chunks in milliseconds.

Embedding Dimensions: Understanding the Scale

When you configure a vector database, it will ask you to specify the number of dimensions. Dimensions refer to the size of the list of numbers generated by the embedding model. For example:

A lightweight model might output vectors with 384 dimensions.
A standard model (like OpenAI's text-embedding-3-small) outputs 1536 dimensions.
Large models can output 3072 dimensions or more.

Each dimension represents an abstract feature of meaning that the model learned during training. You don't need to know what each dimension represents; the database only cares that you are comparing vectors of the exact same size.

💡 Developer's Takeaway

An embedding model and a vector database must match: you cannot store a 1536-dimension embedding in a database index configured for 384 dimensions.

RAG (Retrieval-Augmented Generation): Putting It Together

Now we have all the pieces to solve our Travel Planner's original challenge: answering questions using our travel guides without sending the entire book to the LLM. We use a pattern called RAG (Retrieval-Augmented Generation).

RAG combines Retrieval (finding relevant chunks from our vector database) and Generation (having the LLM write a response using those chunks). Here is the step-by-step workflow:

Ingestion (Happens once in advance):
- Break documents into chunks.
- Generate embeddings for each chunk.
- Store the chunks and their embeddings in a vector database.
Querying (Happens at runtime):
- User asks: "Recommend a vegetarian ramen shop near Tokyo Station."
- The application generates an embedding for the user's question.
- The application queries the vector database using this question embedding.
- The database returns the top 2 or 3 most similar text chunks.
- The application merges the user's question and the retrieved chunks into a single prompt (grounding context).
- The LLM reads the context and generates a correct, verified answer.

               User Question
                     │
                     ▼
             Generate Embedding
                     │
                     ▼
    Query Vector DB ──► Stored Chunks
                     │
                     ▼
            Get Relevant Chunks
                     │
                     ▼
    Build Prompt (Question + Chunks)
                     │
                     ▼
             LLM Generates Text
                     │
                     ▼
             Grounded Response

💡 Developer's Takeaway

RAG is the most popular way to connect proprietary data to LLMs. It ensures the model's response is grounded in facts, reduces hallucinations, and works without needing to retrain the model.

CAG (Cache-Augmented Generation): The New Alternative

While RAG is powerful, it requires setting up chunking pipelines, embedding models, and vector databases. This can be complex. With modern LLMs having massive context windows (often supporting 1 million to 2 million tokens), another approach has emerged: CAG (Cache-Augmented Generation).

Instead of retrieving matching chunks at runtime, CAG preloads the entire document set (e.g., our 500-page travel guide) directly into the model's context window. It leverages KV (Key-Value) Caching to precompute and cache the mathematical states of the documents once. When a user asks a question, the LLM reads the cached documents instantly, bypassing the retrieval pipeline entirely.

RAG is like looking up a topic in a library catalog, retrieving three specific books, and reading them to find the answer.
CAG is like having the entire encyclopedia already open on your desk with the pages instantly readable.

💡 Developer's Takeaway

CAG simplifies the architecture by eliminating vector databases and chunking. However, it is limited by the size of the context window and is best suited for relatively static datasets that fit comfortably within the model's limit. RAG remains essential for massive or constantly changing data.

Recap

In this article, we explored how to give AI knowledge beyond its training. We covered:

Keyword Search vs. Semantic Search
Embeddings, Semantic Similarity, and Cosine Similarity
Chunking, Vector Databases, and Embedding Dimensions
RAG (Retrieval-Augmented Generation) and CAG (Cache-Augmented Generation)

In Part 4, we will answer the next logical question: How do we connect our model to the outside world so it can execute tasks, use external APIs, and remember users over time? We will explore tool calling, memory, Model Context Protocol (MCP), and AI agents.

RAG in Production: 7 Anti-Patterns That Destroy Precision (And How I Fixed Them)

Sat, 11 Jul 2026 14:38:05 +0000

RAG is the aspirin of generative AI: everyone prescribes it, almost no one truly understands how it works, and when it fails, the patient hallucinates. If you've attended any tech conference in the past 18 months, you'll have heard the same promise repeated like a mantra: "Connect your LLM to your documents with RAG and you'll have a chatbot that responds with your company's truth." It's a seductive promise. It is also, in most implementations I've seen, a well-intentioned lie.

I know this because I've walked both paths. I built a working RAG system in production — the Ops Engineering Copilot with Algolia Agent Studio, indexing over 70 posts from this blog — and I also built a system that deliberately rejects RAG — the agentic obsolescence radar, which uses pure Tool Calling to query industrial databases with mathematical precision. The experience of operating both systems in production left me with an uncomfortable conclusion: RAG isn't bad; what's bad is how we implement it.

This article is the catalogue of the 7 errors that destroy a RAG system's precision, the solutions that work, and the question no one wants to ask: do you really need RAG, or do you need something else?

The first step in any RAG pipeline is splitting your documents into fragments (chunks) that will be stored as vectors. And this is where most tutorials commit the first mortal sin: chunking by fixed length (e.g., 500 tokens per chunk with 50 tokens of overlap).

The problem is brutal: a paragraph explaining a complex technical concept gets cut in half. The first half ends up in one chunk, the second in another. When the user asks a question, the retriever finds the first half (which contains the keywords) but misses the context from the second. The LLM, true to its nature, fills in what's missing with a plausible fabrication. Hallucination served.

The fix: Semantic chunking. Split by logical units of meaning: sections delimited by headers, complete paragraphs, or functional blocks of the document. In the Ops Copilot, when we indexed Datalaria's posts with Algolia, each record corresponds to a complete article section (delimited by ### in Markdown), not an arbitrary block of N tokens. The result: each chunk is self-contained and holds a complete thought.

Anti-Pattern 2: Generic Embeddings for Specialized Domains

Pre-trained embeddings (like OpenAI's text-embedding-3-small or Vertex AI's) are trained on general internet text. They work reasonably well for generic questions. But when your domain is highly specialized — industrial engineering, European regulation, electronic component nomenclature — the semantic distance between key terms can be completely wrong in the vector space.

A real example: in the context of obsolescence management, the terms "EOL" (End of Life), "NRND" (Not Recommended for New Designs), and "PDN" (Product Discontinuation Notice) are semantically very close for a supply chain engineer. But for a generic embedding, "End of Life" might be closer to an article about palliative care than to a chip discontinuation notice.

The fix: Evaluate embeddings with your own dataset before committing. Build a small benchmark of 50-100 question-answer pairs from your domain and measure the retriever's hit rate. If generic embeddings don't exceed 80% accuracy on your benchmark, consider fine-tuning or specialized embeddings for your sector. And if your domain is highly structured (codes, nomenclatures, tables), you probably don't need embeddings at all: you need Tool Calling.

Anti-Pattern 3: Forgetting Reranking

The vector retriever returns the k documents "closest" in the embedding space. But vector proximity is not synonymous with relevance. A document can contain the same keywords as the user's question and yet answer a completely different question.

I've seen this anti-pattern cause havoc in technical support systems: the user asks "How do I configure the CrewAI agent timeout?", the retriever returns a chunk about GitHub Actions timeouts (same vocabulary, different context), and the LLM generates a response that's technically correct for the wrong chunk.

The fix: Add a reranking layer between the retriever and the LLM. A reranker (like Cohere Rerank or a local cross-encoder) receives the original question and the k candidates from the retriever, and reorders them by actual semantic relevance, not mere vector proximity. In practice, a well-configured reranker can improve retrieval precision by 15% to 30% — an improvement that translates directly into fewer hallucinations.

Anti-Pattern 4: Insufficient Context in the Prompt

The most widespread and easiest mistake to make: injecting 2-3 chunks into the LLM's prompt and expecting a miracle. Modern models like Gemini 2.5 or Claude handle context windows of hundreds of thousands of tokens. Feeding them 500 tokens of retrieved context is like giving a Formula 1 engine the fuel from a cigarette lighter.

The fix: Experiment aggressively with the size of the injected context window. Increase from top-3 to top-10 or top-15 chunks and measure the impact on response quality. Include enriched metadata in each chunk: source document title, creation date, author, section. These metadata give the LLM a referential framework to evaluate the relevance and recency of the information. In the Ops Copilot, each Algolia record includes not just the post text, but also the title, category, tags, and publication date.

Anti-Pattern 5: Hallucination from Partial Retrieval

This is the most dangerous anti-pattern because it's silent. The retriever finds a partially relevant chunk. The LLM detects that the information is incomplete. Instead of stopping and confessing its ignorance, it completes the answer with fabricated information that sounds perfectly plausible. The user has no way to distinguish which part of the answer comes from retrieval and which part is a hallucination.

In industrial applications, this can be catastrophic. Imagine a RAG system connected to your maintenance documentation that, when asked about the torque specification of a critical bolt, returns a fabricated value because the correct chunk wasn't retrieved. The result could be a mechanical failure in production.

The fix: Double barrier. First: instruct the LLM in the system prompt to answer "I don't have sufficient information in the provided documentation to answer this question" when the retrieved context is insufficient or ambiguous. Include examples in the prompt (few-shot) of correct answers that acknowledge limitations. Second: implement a post-validation of the response. Programmatically evaluate whether the answer contains claims not supported by the injected chunks (frameworks like RAGAS automate this verification with metrics like faithfulness and answer relevancy).

Anti-Pattern 6: Not Measuring Quality

The sixth anti-pattern is cultural, not technical: launching a RAG system to production without evaluation metrics. Asking "does it work?" to five teammates is not an evaluation methodology; it's an anecdote. Without quantitative metrics, you can't know if a change in chunking improved or worsened precision, if a new embedding model is superior to the previous one, or if the reranker you just added justifies its latency cost.

The fix: Implement an automated evaluation framework before launching to production. The most mature tools are:

RAGAS (Retrieval Augmented Generation Assessment): Measures faithfulness (the answer is grounded in context), answer relevancy (the answer is relevant to the question), and context precision (the retrieved chunks are relevant).
DeepEval: An open-source framework that lets you define test suites with metrics like hallucination score, bias, and toxicity.

The key point is to treat RAG evaluation the way you treat your code's unit tests: if it doesn't have tests, it doesn't go to production. Every pipeline iteration (embedding change, chunk size adjustment, new reranker) must pass through the evaluation suite before deployment. It's the same CI/CD philosophy we applied in Autopilot Part 5 with GitHub Actions, but applied to retrieval quality.

Anti-Pattern 7: Using RAG When You Need Tool Calling

This is the anti-pattern that was hardest for me to accept, because it meant questioning my own architectural decision. When we built the agentic obsolescence radar, the first temptation was to use RAG: index all component documentation, datasheets, and price histories in a vector store, and let the LLM search for relevant information for each End of Life alert.

The result was disastrous. LLMs are, as I wrote in that article, mediocre calculators. When the radar needed to calculate the financial impact of an obsolescence (traverse the BOM graph, multiply quantities by prices, sum redesign costs), RAG returned narrative approximations where we needed exact figures. The financial precision was unacceptable for an executive report.

The solution was to radically separate the "semantic brain" from the "mathematical muscle": the LLM (Gemini 2.5 + CrewAI) handles natural language comprehension (extracting the Part Number from a supplier email, understanding the context of an alert), and Python tools (CrewAI's @tool) handle precision operations (SQL queries to Supabase, P&L calculations, relational graph traversal). The result: executive reports generated in 4 seconds with 0% hallucination in numerical data.

The rule I've distilled is simple:

You need...	Use...	Why
Answers about unstructured knowledge (manuals, posts, narrative documentation)	RAG	Semantic retrieval is superior for searching free text
Answers with structured data and numerical precision (SQL, APIs, financial calculations)	Tool Calling	Tools execute deterministic code, no hallucinations
Both (interpret an email + calculate financial impact)	Hybrid architecture	The LLM orchestrates; the tools execute

And if you're wondering how to standardize those connections between the LLM and the tools so you're not locked into a vendor, that's exactly what we addressed in the article about MCP (Model Context Protocol).

Conclusion: RAG Isn't Broken; Your Implementation Is

If there's one message I want you to take from this article, it's this: RAG is a legitimate and powerful architecture when implemented with engineering rigor. The problem isn't the pattern; the problem is that the industry has popularized it as a magical plug-and-play solution, when in reality it's a complex pipeline that requires intelligent chunking, evaluated embeddings, reranking, generous context, hallucination defenses, quality metrics, and the humility to recognize when Tool Calling is the right tool for the job.

The 10x Rule I proposed in the Hidden Economics of AI article applies perfectly here: if RAG doesn't give you a result 10 times better than a direct SQL query or an API call, you're probably using the wrong tool for the wrong problem.

And in the era of the EU AI Act, where traceability (Article 12) and precision are legal obligations for high-risk systems, deploying a RAG that hallucinates isn't just a technical error: it's a regulatory risk.

Sources of Interest:

I Rebuilt My AI Legal Assistant After Learning Why Vector-Only RAG Wasn't Enough

Sat, 11 Jul 2026 11:38:51 +0000

I built a legal AI assistant last year.

One day I asked it:

"Someone forged my signature."

The first law it retrieved wasn't about forgery.

It was about counterfeit coins.

That was the moment I realized the problem wasn't the LLM.

It was my retrieval pipeline.

The surprising part: I didn't change the LLM.

Nearly all of the improvement came from redesigning the retrieval pipeline.

Here is the story of how I debugged my search system, why dense vector search alone falls apart on domain-specific datasets, and how I redesigned the retrieval pipeline.

The Failure

To understand why the system was failing, we need to look at the mismatch between what was queried and what was actually retrieved in the original vector-only (v1) search:

Query:
"Someone forged my signature"

Top result (v1):
❌ BNS Section 180 — Possession of counterfeit coin

Expected:
✅ BNS Section 336 — Forgery
✅ BNS Section 340 — Using forged document

Tracing the Cause

I started tracing the retrieval pipeline step by step to understand why the wrong statutes were consistently ranking first.

The embedding model wasn't "wrong." It was doing exactly what it was trained to do: place semantically similar concepts close together in vector space.

Unfortunately, in legal search, semantically similar isn't the same as legally relevant.

Because "forgery" and "counterfeit" ended up close in embedding space, the retriever ranked counterfeit coin and government stamp provisions above the actual definition of document forgery. The retriever generalized too aggressively. It missed the specific statutory definition of forgery (BNS Section 336) because the word "signature" was semantically distant from generic statutory descriptions of the offence.

The Secondary Issues: RAM Bloat & Duplicates

In addition to generalising incorrectly, caching large raw text strings (text, titles, act names) in a JavaScript array caused the Node.js process to consume over 320 MB of RAM at startup. Furthermore, since legal codes are highly repetitive, the search regularly returned duplicate entries of identical sections across different personal laws, cluttering the LLM's context window.

Redesigning the Retrieval Pipeline

I realized that to make the assistant trustworthy, the real problem was search quality, not LLM capability. I redesigned the retrieval pipeline into a structured, hybrid search engine.

The LLM remained almost unchanged. Nearly all improvements came from redesigning retrieval.

Step 1: SQLite + FTS5

I moved all 4,892 legal sections out of JSON files and persisted them in a local SQLite database. I created a virtual table using the FTS5 extension to index all chapters, titles, and text contents.

Now, exact terms are queried using a sparse keyword index (ranked via BM25), ensuring that queries containing specific statutory terms match their target immediately.

Step 2: Lightweight Vector Cache

I stripped all text metadata from Node.js memory. The startup script now loads only the id (string) and the coordinate list—pre-processed into a compact Float32Array object—into RAM.

The actual text content remains on disk in SQLite and is only hydrated for the top 5 matched sections. This dropped the memory footprint by 85% (from 320 MB to 48 MB).

Step 3: Reciprocal Rank Fusion (RRF)

Instead of relying on either vector search or keyword search, I fused them. The backend runs both searches, takes the top 50 matches from each, and combines their rankings using standard Reciprocal Rank Fusion (RRF). RRF rewards documents that rank highly in both methods without needing to normalize scores between sparse BM25 and dense cosine models.

Here is the core JS implementation of the RRF merge:

const rrfScores = new Map();
const k = 60; // Standard constant for RRF

// Process vector ranking positions (dense)
vectorRankings.forEach((item, index) => {
  rrfScores.set(item.id, 1 / (k + index + 1));
});

// Process FTS5 keyword ranking positions (sparse) and add to scores
ftsRankings.forEach((item, index) => {
  const existingScore = rrfScores.get(item.id) || 0;
  rrfScores.set(item.id, existingScore + (1 / (k + index + 1)));
});

// Sort matched IDs based on fused RRF score
const mergedRanking = Array.from(rrfScores.entries())
  .sort((a, b) => b[1] - a[1])
  .slice(0, 20); // Top 20 candidates for reranking

Step 4: Domain Reranker (Deterministic Guardrail)

To resolve the counterfeit coin noise without running a heavy, slow transformer cross-encoder, I built a lightweight, deterministic Domain Reranker in JavaScript. This is a deterministic rule-based reranker tailored to the legal domain—not a learned neural cross-encoder.

It loads the top 20 candidates returned by the RRF step and checks for specific intent signals:

If the query is document/signature forgery-related, it checks if a retrieved document is a coin or banknote counterfeit section. If yes, it penalizes the score by 99% (* 0.01).
It boosts direct document forgery offences (containing "forgery" or "forged") by 300% (* 3.0).
It filters out duplicate sections using content snippet prefixes.

// Heuristic Domain Reranking
if (isDocumentForgeryRelated) {
  const isCoinOrStampOrCurrency = 
    titleLower.includes('coin') || titleLower.includes('stamp') || 
    titleLower.includes('currency') || titleLower.includes('bank-note') ||
    contentLower.includes('coin') || contentLower.includes('stamp') || 
    contentLower.includes('currency-note');

  if (isCoinOrStampOrCurrency) {
    adjustedScore *= 0.01; // heavily penalize counterfeit coin/stamps (reduce by 99%)
  } else if (titleLower.includes('forgery') || titleLower.includes('forged')) {
    adjustedScore *= 3.0; // strong boost for direct forgery definitions/offences
  }
}

Performance & Evaluation

To evaluate the redesign, I assembled a benchmark of 100 manually verified legal queries spanning criminal law, cybercrime, family law, consumer protection, and procedural law.

Metric	v1 (Naive Vector RAG)	v2.1 (Hybrid Search - Current)	Change
Search Engine	Dense Vector (Linear JSON scan)	Hybrid (SQLite FTS5 + Dense Vector + RRF + Reranker)	Major retrieval precision upgrade
Avg. Query Latency	`466 ms`	`12 ms`	97.4% speedup
Memory Cache Footprint	`~320 MB`	`~48 MB`	85.0% RAM savings
Duplicate Citations	Present (up to 40% overlaps)	Deduplicated (0% overlaps)	Verified
Top-5 Relevant Retrieval Rate	~68%	~91%	+23% accuracy gain

Latency is based on 100 benchmark queries. Memory is process-level heap size at startup. Accuracy is evaluated on top-5 target matches using a manually verified benchmark dataset of 100 queries.

None of these improvements required changing the language model. The gains came almost entirely from retrieval engineering. For this benchmark query, the top retrieved references aligned with the expected legal provisions:

BNS 340: Forged document and using it as genuine
BNS 336: Forgery definition and penalty
BNS 339: Possession of forged document
BNS 335: Making a false document
Evidence Act Section 65: Proof of signature and handwriting

Production Walkthrough

1️⃣ User Chat Interface

Clean, legal explanation interface for end users:

2️⃣ Structured Offence & Citation Details

Deduplicated citations with developer metrics visible in Developer Mode:

3️⃣ System Evaluation Dashboard

Performance comparisons and technical architecture story:

Lessons Learned

Retrieval quality sets the upper bound for RAG quality.
Dense embeddings alone are rarely enough for domain-specific search.
SQLite + FTS5 can be an excellent retrieval engine for small-to-medium corpora.
Simple deterministic rerankers can eliminate domain-specific retrieval errors without requiring another neural model.

If I continue evolving this project, the next improvements I'd explore are:

Cross-encoder reranking: Integrate lightweight cross-encoders (e.g. BGE reranker) for advanced ranking.
Metadata-aware retrieval: Allow users to filter queries by Act or category before searching.
Legal Case Retrieval: Expand indexing to cover legal precedents and court cases in addition to statutory acts.
Multilingual support: Support query translation for regional languages.

Final Thoughts

Going into this project, I assumed improving the LLM would improve the assistant.

Instead, I learned that retrieval quality determines the ceiling of any RAG system.

The LLM can only reason over what you retrieve.

Improving retrieval turned out to be far more impactful than switching models.

If you're building domain-specific AI—whether for legal, medical, or enterprise search—I'd recommend spending as much time on retrieval engineering as prompt engineering.

Repository

The complete implementation—including SQLite ingestion, FTS5 indexing, Reciprocal Rank Fusion, evaluation queries, and benchmark samples—is available on GitHub.

GitHub: https://github.com/ishwar170695/LawDecoder

Hallucination Is Not One Problem — It's Five Different Architectural Failures

Fri, 03 Jul 2026 21:07:27 +0000

Why your RAG pipeline, guardrails, and prompt engineering each fix a different hallucination — and why none of them fix the others.

The Frustration

You built a RAG system. Added re-ranking. Tuned the temperature. Even threw in a "don't make things up" prompt. And yet — your agent still hallucinates. So you add more retrieval chunks. Then guardrails. Then another LLM call to "verify" the first one. The hallucinations shift shape, but they never disappear.

Here's the hard truth: you're treating five different architectural failures as one "hallucination bug."

Type 1: Confabulation — The LLM Makes Facts Up

What it looks like: "According to a 2023 study by Stanford..." (no such study exists)

Why it happens: The model is a next-token predictor with no ground-truth anchor. It optimizes for plausibility, not accuracy.

The fix — Retrieval Grounding + Fact Verification:

Every claim must cite a retrieved source
Verification LLM checks claim ↔ source alignment
Abstention gate: if no source supports it, output "I don't know"

Prompt engineering won't fix this. The model has no fact-checker in its weights. You need an external verification architecture.

Type 2: Attribution Error — Right Fact, Wrong Source

What it looks like: Cites Document A for a fact that actually came from Document B

Why it happens: Semantic similarity retrieves the wrong chunk; the LLM doesn't verify provenance

The fix — Provenance Tracking + Citation Scoring:

Chunk-level metadata: source ID, section, timestamp
Citation scorer: cross-check generated citation against retrieved chunks
Mismatch → flag or abstain

Type 3: Temporal Drift — The Truth Expired

What it looks like: "The latest React version is 18.2" (it's 19.x now)

Why it happens: Static knowledge base with no freshness signal; retrieval doesn't prioritize recency

The fix — Knowledge Freshness Timestamps + Update Pipelines:

Every chunk tagged with last_updated
Retrieval re-ranks by recency for time-sensitive queries
Scheduled re-indexing pipeline for evolving domains

Type 4: Logical Inconsistency —Contradictory Claims in One Answer

What it looks like: "X is faster than Y" and "Y outperforms X" in the same response

Why it happens: LLM processes tokens locally; no global consistency check across the full output

The fix — Structured Reasoning + Formal Verification:

Chain-of-thought with explicit intermediate claims
Consistency checker: compare all claims pairwise
For critical domains: compile claims to SAT/SMT solver

Type 5: Tool Hallucination — The Agent Calls a Tool That Doesn't Exist

What it looks like: call_tool("get_weather", location="Mars") — tool doesn't exist, or parameter is invalid

Why it happens: The LLM generates tool calls like it generates text — by pattern matching, not by schema validation

The fix — Tool Schema Validation + Execution Sandbox:

Tool registry with JSON schemas
Pre-execution validator: name + params must match schema
Sandbox: tool runs in isolated env, output validated before passing back

The Unified Architecture

Closing

Hallucination is not a bug you patch. It's a symptom of five different architectural gaps. Fix the architecture, and the "hallucination problem" dissolves into five solvable engineering problems.

CAG: The Simpler Way to Ground Your LLM

Sun, 28 Jun 2026 05:09:39 +0000

If you've been building AI applications recently, you've probably come across Retrieval-Augmented Generation (RAG). It has become the go-to way of giving LLMs access to external knowledge.

But RAG isn't the only option.

As context windows continue to grow, another approach is becoming increasingly practical: Cache-Augmented Generation (CAG).

Before we begin, a small disclaimer. This article intentionally argues in CAG's favor. Think of it as a friendly debate where CAG finally gets a chance to speak while RAG takes a short coffee break.

Why RAG Became So Popular

RAG solved a real problem.

Instead of expecting an LLM to know everything, we store information in a vector database. When a user asks a question, we retrieve the most relevant pieces and send them to the model.

A typical RAG pipeline looks like this:

Query → Embed → Search → Rank → Retrieve → Generate

It's a proven approach and works really well, especially when your knowledge base is large or changes frequently.

The only downside is that every question has to go through this retrieval process before the model can generate an answer.

That means more infrastructure, more moving parts, and a little extra latency.

Meet CAG

CAG takes a much simpler approach.

Instead of searching for information every time someone asks a question, it loads the required knowledge into the model's context once and keeps using it.

The workflow becomes:

Load knowledge → Cache context → Generate

That's the entire idea.

No vector search.

No retrieval step.

No ranking.

The model already has the information it needs.

Why Is This Possible Now?

A couple of years ago, CAG wasn't practical.

Context windows were simply too small.

Today, that's no longer true.

Many modern models support hundreds of thousands and sometimes even millions of tokens.

That changes the question from:

"How do I retrieve the right documents?"

"Can I fit my knowledge into the context window?"

For many internal tools, company documentation, onboarding guides, product manuals, and API references, the answer is surprisingly often yes.

RAG vs CAG

Both approaches solve the same problem, but in different ways.

Choose RAG when:

Your knowledge base is too large to fit into the model's context.
Information changes frequently.
Different users need different subsets of knowledge.
You need real-time data.

Choose CAG when:

Your documentation comfortably fits in the context window.
Most of the information is relatively static.
Low latency is important.
You want a simpler architecture with fewer components.

Neither approach is "better."

The right choice depends on your use case.

A Simple Example

A traditional RAG pipeline might look like this:

query = "What's our refund policy?"

embedding = embed(query)
chunks = vector_db.search(embedding, top_k=5)

context = "\n".join(chunks)

response = llm.generate(
    f"Context:\n{context}\n\nQuestion: {query}"
)

A CAG implementation is much simpler:

with open("knowledge_base.txt") as f:
    knowledge = f.read()

system_prompt = f"""
You are an assistant.

Use the following knowledge when answering questions.

{knowledge}
"""

response = llm.generate(
    system=system_prompt,
    user="What's our refund policy?"
)

The biggest difference isn't the amount of code.

It's that there is no retrieval happening during inference.

Why Not Use Both?

In practice, many applications don't have to choose one over the other.

A hybrid approach often works best.

Keep your stable documentation in the model's cached context using CAG.

Retrieve only the information that changes frequently using RAG.

This gives you fast responses for most questions while still allowing access to fresh information whenever needed.

Final Thoughts

As developers, we sometimes assume that every LLM application needs a vector database.

But that's not always true anymore.

Before building a RAG pipeline, ask yourself one simple question:

Does my knowledge base actually fit inside the model's context window?

If it does, CAG could be a simpler solution that's easier to build, easier to maintain, and often faster to serve.

If it doesn't, RAG is still an excellent choice.

The goal isn't to replace RAG.

It's to recognize that modern context windows have changed what's possible, and CAG deserves a place in the conversation.

TL;DR

RAG retrieves relevant information before every query.
CAG loads knowledge into the model's context upfront.
Modern context windows make CAG practical for many use cases.
If your knowledge fits in context, CAG is worth considering before reaching for a vector database.
For large or frequently changing knowledge bases, RAG remains the better choice.

Sometimes the simplest architecture is the one that gets out of the model's way.