Coder Legion - Recent posts tagged knowledgebase

How Synthadoc Handles Hallucination and Why "Don't Hallucinate" Isn't Enough

Fri, 28 Aug 2026 17:18:42 +0000

Ask most AI products how they handle hallucination and you'll get a variant of "we tell the
model not to." That's not wrong — it's just insufficient, and it puts enforcement in the
hands of the exact system doing the generating.

Synthadoc treats hallucination as an architecture problem, not a wording problem. The
distinction matters: a wording fix is hoping the model behaves; an architecture fix is
verifying that it did.

Why Prompt Instructions Aren't Enough

The industry default is a line in a system prompt: don't make things up, answer only from
the provided context, never fabricate. These instructions do nudge model behavior in the
right direction, and we use them too. But they leave enforcement entirely up to the model
doing the generating. A language model producing text token-by-token has no reliable
internal mechanism to audit its own faithfulness to a source.

There's a subtle issue as well. Instructions of the form "don't do X" are easier for a
model to rationalize around than instructions of the form "do only Y." Synthadoc's synthesis
constraint is a positive formulation:

Answer using ONLY these wiki pages. Cite with [[PageTitle]].

That's linguistically harder to rationalize around. But it's still just text in a prompt.
The real work happens at the structural level.

Three Layers, Not One Instruction

Our approach treats hallucination as a systems problem. There are three distinct layers,
each handling a different failure mode:

Scope filtering at ingest time : Wiki content can only be entered if it is on-domain. It is difficult to produce hallucinations about off-domain topics from content that has never been consumed.
Retrieval-augmented grounding at query time : Answers are generated from retrieved
wiki pages, not from the model's weights alone. The synthesis prompt shows only the
relevant pages and instructs the model to answer from them exclusively.
Citation faithfulness audit after the fact : Every claim gets a citation marker
linking it to specific source lines. A separate LLM pass then audits whether each claim
is actually supported by those lines. This is the layer that catches what the other two
miss.

None of these is a complete solution alone. Together they create a system where hallucinations
are detectable and traceable, not just discouraged.

Layer 1: purpose.md as a Content Gate

Every wiki has a wiki/purpose.md file that describes its domain — what it's for, what
topics belong, and what doesn't. When a new source document arrives for ingest, the ingest
agent reads this purpose block before making a decision:

Wiki scope (from purpose.md):
{content of purpose.md}

action="skip" means the source is completely OUTSIDE the wiki's domain
(e.g. spam, medical receipts, unrelated e-commerce).
…

The LLM then decides: skip, create, update, or flag. Out-of-scope content never
becomes a wiki page, which means it can never become the basis for an answer.

Lifecycle isolation for contradicted pages

The scope gate isn't the only place the knowledge base shrinks. When a new source
disagrees with an existing page, the ingest agent issues a flag action and the page
moves to a contradicted lifecycle state, and automatically excluded from retrieval until
the conflict is resolved. This matters because a contradicted page is, by definition,
one where the system can't be confident about what's true. Serving answers from it
would be hallucination with extra steps.

The contradiction resolver in Synthadoc is an interactive agentic workflow: it surfaces
the conflict, shows a diff of the proposed edit, and requires explicit human approval
before writing any change. The approve-before-write gate is unconditional, and the agent
cannot modify a page without a human seeing the diff first.

Layer 2: RAG Grounding at Query Time

When a user asks a question, Synthadoc retrieves the most relevant wiki pages using
hybrid search (BM25 full-text matching with optional vector re-ranking), then builds a
synthesis prompt that includes those pages verbatim:

Answer using ONLY these wiki pages. Cite with [[PageTitle]].
Extract and include all specific facts from the pages — dates, years,
numbers, and names — even when they appear briefly or in passing.
Do not claim a fact is absent unless it is genuinely missing from
every page below.

Question: {question}

Pages:
{retrieved_wiki_pages}

The purpose.md content is prepended to the pages block as ### Wiki Scope, giving the
model a domain reminder during synthesis as well as at ingest time.

Why this isn't enough on its own: A model can still hallucinate within retrieved
content — misattributing which page said what, inventing a specific number close to but
not the same as a number in the source, or synthesizing a conclusion that no single page
actually states. Grounding reduces the search space; it doesn't eliminate drift.

When retrieval confidence falls below threshold, Synthadoc detects the gap and handles
it transparently rather than hallucinating silently. The immediate response answers from
the model's general knowledge while explicitly noting that the wiki lacks dedicated
coverage — the user always knows they're receiving an unverified answer. At the same time,
Synthadoc generates enrichment suggestions: search queries and known URLs scoped to the
domain in purpose.md. Retrieved pages flow through the normal ingest pipeline and stay
in draft state — out of the search corpus — until a human promotes them to active.
Only then do future queries on that topic get answered from verified wiki content rather
than general knowledge.

Layer 3: Citation Faithfulness Audit

This is the layer that makes the system verifiable rather than just hopeful. The citation
faithfulness audit examines each claim in a wiki answer alongside the source lines it cites,
then classifies the relationship:

Verdict	Meaning
`supported`	The claim is directly and accurately supported by the cited source lines.
`drift`	The claim is related to the source but overstated, simplified, or shifted in meaning.
`hallucination`	The source lines do not support the claim, or directly contradict it.

Each audit is done by a separate LLM call with the claim text and raw source lines
side-by-side. The auditor never sees the original question — only the claim and the lines.

Why a separate LLM pass? The same model that wrote the answer is likely to defend
it when asked "is this correct?" Separation of duties matters: the auditor has no stake
in validating the original generation.

What we actually see in practice

In our experience, drift is the most common finding — not outright hallucination.
The model summarizes accurately but slightly overstates a confidence level, rounds a
number, or drops a qualifier ("as of 2023" becomes "currently"). These are small errors
individually but compound across a knowledge base that gets queried repeatedly.

True hallucination , where the cited source lines don't support the claim at all ,
is less common in well-grounded RAG, but it happens: specific dates that don't appear
in the source, an attribution switched between two people mentioned in the same paragraph,
a causal relationship inferred from correlation. Those are the findings that matter most,
and the audit surfaces them per-citation so a human knows exactly which source line to
check.

The audit result is cached per-page with staleness detection. If the page is re-ingested
from updated source material, the cache is invalidated and the next audit reflects the new
content. This surfaces in the web UI as an automatically-loaded faithfulness tab alongside
each wiki page.

What This Doesn't Solve

Honesty matters here too. Two failure modes remain outside what the three layers fully
address:

Synthesis across pages. A conclusion drawn by combining facts from multiple pages may
be accurate at each individual step and still misleading in combination. The citation
faithfulness audit operates per-citation, not across synthesized conclusions, so a
cross-page inference that no single page actually states falls outside its scope.
The practical path: when a synthesized answer matters, the cited pages are explicit and
can be verified directly. If the conclusion is important enough to keep, saving it as a
wiki page subjects it to future source ingestion — a new source that disputes it will
trigger the flag action for human review.

Source quality. When a new source disagrees with an existing page, the flag action
fires and the contradiction is surfaced. The harder case is when all available sources
agree on the same wrong answer: no flag fires, no contradiction is detected, and Synthadoc
answers confidently from incorrect material. Two mechanisms make this less likely: the
adversarial review in lint, which uses the model's own knowledge to flag claims that
clearly contradict well-established facts; and re-ingesting from updated authoritative
sources, which will trigger the flag action against any outdated pages. Neither is a
complete solution, but together they make uniform source error less likely to go
unnoticed indefinitely.

Our View

The hallucination debate tends to frame the problem as a model capability question: will
the next model be more faithful than the last? We think that's the wrong frame for a
production knowledge system. Even a perfectly faithful model can only be as accurate as
the content it's grounded in, and even a slightly drifting model can be made accountable
if the output is systematically audited.

Our architectural bet is that the answer to hallucination isn't a better instruction,
it's a feedback loop. Domain scoping controls what enters. Lifecycle states control what
gets queried. Retrieval grounds generation. Citation audit classifies the output. The
contradiction resolver closes the loop: when a conflict is found, a human resolves it, the
page is corrected, and the knowledge base improves. Errors become visible rather than
accumulating silently.

That's a more defensible position than any number of ALL CAPS instructions.

Try It

Synthadoc is open-source. The citation faithfulness audit, contradiction resolver, and
lifecycle system described here are all in the main branch.

→ github.com/axoviq-ai/synthadoc

Snapshots, Diffs, and Rollback: The Audit Trail Every Living Wiki Needs

Sun, 09 Aug 2026 03:47:33 +0000

There's a class of bug that only shows up in knowledge management systems: the invisible overwrite. You open a wiki page in your editor, make a small tweak to clarify a sentence, and save. Three weeks later a query that used to return the right answer comes back empty. The page is active. The source file hasn't changed. Everything looks fine from the outside. But somewhere in that "small tweak," you deleted a paragraph that the search index depended on.

You have no diff. No snapshot. No way back except your memory of what it used to say, which is already suspect, because if it were reliable you would have caught the problem sooner.

This is the problem we built Content Snapshots to solve in Synthadoc v1.2.0.

Why a Lifecycle System Isn't Enough on Its Own

Synthadoc already had a detailed audit trail before v1.2.0. Every lifecycle transition — draft to active, active to stale, active to archived — was recorded in the lifecycle_events table with a timestamp, a trigger source, a reason string, and the before/after states. You could reconstruct when every state change happened and what caused it.

What the audit log couldn't answer was: what did the page say at that moment?

The distinction matters more than it sounds. A wiki page doesn't just have a state; it has content. And the content can change independently of the state. You can promote a page from draft to active on Monday, edit it in Obsidian on Tuesday, and have the lint agent mark it stale on Wednesday, but if the Tuesday edit introduced a subtle error, the Wednesday stale flag won't tell you anything about what went wrong. You need a record of what the page contained at each of those moments.

The audit infrastructure was already there — the lifecycle_events table, the TriggerSource enum, the log queries. What we were missing was a content_snapshot column to store the page body alongside each event. That was the architectural gap.

Architecture: Three Triggers, One Table

The design decision we landed on was to capture snapshots at the points in the lifecycle where content is most likely to have meaning: immediately before or at the moment of a consequential event. Not on every save, not on a timer, but at the moments that already have semantic weight in the audit trail.

Three distinct triggers produce snapshots:

Manual lifecycle transitions. Whenever the user calls lifecycle activate, lifecycle archive, or lifecycle restore — through the CLI, the Obsidian Lifecycle modal, the REST API, or the MCP tool — the current page body is captured into the event record. This gives you a snapshot of exactly what the page contained when you decided it was ready to go live, or when you chose to take it offline.

Lint-driven state changes. When the lint agent transitions a page automatically — promoting a draft to active, marking an active page stale because its source hash changed, auto-resolving a contradiction — the snapshot fires there too. For the auto-resolve case in particular, this captures the post-resolution body that was just written, which is the version you'd want to roll back to if the resolution turns out to be wrong.

Direct edits in Obsidian. When you edit a wiki page directly in the Obsidian vault, the plugin's file watcher fires after a two-second debounce. This trigger is the only one with deduplication: snapshot_if_changed() compares the incoming content to the last stored snapshot, and only writes a new row when the content actually differs. Since the file watcher fires on every keystroke-batch, without dedup you'd generate hundreds of identical rows for a single editing session.

The state flow looks like this:

For the Obsidian edit trigger, from_state and to_state are both set to the current page state, since no lifecycle transition is occurring. The snapshot is a pure content checkpoint.

Storage is roughly 3 KB per snapshot at typical wiki page sizes. The existing purge_lifecycle_events() endpoint handles cleanup, the --before and --keep-latest options remove event rows along with their embedded snapshots.

What Was Harder Than Expected

The three-trigger model looks clean in the design doc. Getting the semantics right in the implementation was less clean.

The rollback problem. Rollback sounds simple: take the page body from snapshot N and write it to disk. But what if you roll back to a snapshot that was itself wrong, and need to undo the rollback? We made rollback undoable by always recording the current body as a new snapshot before writing the target. That means index 1 is always the most recent snapshot, and rolling back to index 2 creates a new index 1 that captures what you just had. The operation never changes page_states or calls set_page_state , it's a content operation, not a lifecycle operation.

The dedup boundary. The two-second debounce on the Obsidian file watcher prevents the snapshot log from filling up with per-keystroke noise. But two seconds isn't always enough if the user pauses mid-edit. We added a content equality check on top of the debounce — snapshot_if_changed() — so even if multiple debounced events fire for the same editing session, only one row gets written if the content hasn't changed between fires. The check is a simple string equality comparison; at typical page sizes this is negligible overhead.

The ingest carve-out. One decision that caused more discussion than expected: ingest-agent transitions do not produce snapshots. This is intentional. The ingest agent writes the page body from scratch based on the source document; capturing a snapshot at that point would just duplicate the ingested content. The meaningful history is what humans wrote or edited, not what the ingest agent synthesised. Lint-driven transitions — where the agent may overwrite human edits to resolve a contradiction — do snapshot, because that's the case where you might want to go back.

Use Case Walkthrough

Here's the scenario this was designed for. You have an active wiki page on a topic that your team references frequently. A colleague opens the page in Obsidian, adds a paragraph, and saves. The edit looks fine. Two days later the lint agent runs, sees that the source file hash has changed, and marks the page stale.

Before v1.2.0, you'd have an audit trail entry saying "stale, reason: source hash mismatch" but no way to see what the page contained when the colleague edited it versus what it contained when it went stale. You'd have to diff the current file against the source document by hand.

Now the audit history for that page shows three entries: the original activation (with the body as it was promoted), the colleague's Obsidian edit (captured by snapshot_if_changed(), a content-only checkpoint), and the lint-driven stale transition (captured with the post-lint body). You can open a diff between any two of them in the Obsidian Content Snapshots tab and immediately see what changed.

If the colleague's edit inadvertently removed something important, you can roll back to the activation snapshot. The rollback records the current body first, so if you change your mind, rolling back to index 1 gives you back the stale version.

CLI Commands

Content Snapshots are accessible from the command line without the Obsidian plugin.

List all snapshots for a page, with index, timestamp, transition, and content size:

synthadoc lifecycle history alan-turing

View the content of a specific snapshot (index 1 is always the newest):

synthadoc lifecycle history alan-turing --index 2 --show-content

Roll back to a previous snapshot. The current body is saved as a new snapshot first, so the rollback itself is undoable:

synthadoc lifecycle rollback alan-turing --index 2 --reason "reverting bad edit"

Remove old snapshot data while keeping the most recent N events per page:

synthadoc audit lifecycle purge --keep-latest 5

Diff between snapshots is available in the Obsidian Content Snapshots tab, select any two entries in the history list to compare them side by side. The CLI surfaces the raw snapshot content via --show-content; piping two snapshots through your preferred diff tool achieves the same result.

Why This Matters

The content snapshot architecture is, at its core, a discipline about what you need to know when things go wrong. Most knowledge management systems are optimised for retrieval — for answering "what does this page say now?" Content Snapshots shifts some of that engineering toward answering "what did it say before?" and "which change caused the problem?"

The practical consequence is that a wiki built with Synthadoc behaves more like a version-controlled document than a database record. You get the full lifecycle audit — who promoted what and when — plus a point-in-time content record at every moment that mattered. You can reconstruct the editorial history of any page from activation to the present, view any snapshot's content, and roll back to any of them.

What we found in building this is that the rollback undoability was the right call even though it slightly complicates the storage model. A rollback that can't itself be undone is a footgun. The cost is an extra row per rollback operation; the benefit is that the snapshot log is always accurate and reversible.

Synthadoc v1.2.0 is available at github.com/axoviq-ai/synthadoc. The Content Snapshots feature works across all Synthadoc interfaces — CLI, Obsidian plugin, REST API, and MCP.

For a step-by-step walkthrough — including the Obsidian diff view and a live rollback — see the Snapshots and content recovery section of the Quick Start Guide.