[Bug] Claude defaults to assertion from memory instead of verification from tools #32294

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   area:model

   platform:windowsIssue specifically occurs on WindowsIssue specifically occurs on Windows

   on Mar 9, 2026
2. github-actions commented on Mar 9, 2026

   

   github-actionsBot

   on Mar 9, 2026 – with GitHub Actions

   More actions

   Found 3 possible duplicate issues:

   1. [Bug] Claude generates incorrect code/SQL and reports it as complete without validation #32289
   2. Opus 4.6 (1M): Model regularly attempts Edit/Write on files it hasn't Read — harness-level enforcement needed #27790
   3. [BUG] Model attempts file edits without reading file first, wasting tokens and time #16546

   This issue will be automatically closed as a duplicate in 3 days.

   • If your issue is a duplicate, please close it and 👍 the existing issue instead
   • To prevent auto-closure, add a comment or 👎 this comment

   🤖 Generated with Claude Code

   👎React with 👎1Reacted by VoxCore
3. VoxCore84 commented on Mar 9, 2026

   

   VoxCore84

   on Mar 9, 2026

   Author

   More actions

   Additional Detail: Three Assertion-From-Memory Failures in One Session

   Failure 1: gameobject_template column count (prior session)

   What Claude asserted: Data0-Data34 needs 32 zero values.
   What's actually true: Data0-Data34 is 35 columns (0 through 34 inclusive = 35).
   How it should have been verified: DESCRIBE gameobject_template → count the Data columns.
   Cost of checking: ~100ms for a DESCRIBE query.
   Cost of not checking: Broken SQL file that failed on execution in a later session, requiring diagnosis and a fix.

   Failure 2: smart_scripts.VerifiedBuild (this session)

   What Claude asserted: The smart_scripts table has a VerifiedBuild column.
   What's actually true: smart_scripts has no VerifiedBuild column. Most other tables in the world database do, but smart_scripts is an exception.
   How it should have been verified: DESCRIBE smart_scripts or check the import document (which notes "no VB" for this table).
   Cost of checking: ~100ms.
   Cost of not checking: Query error, wasted round-trip, required fallback to a different verification approach.

   Exact error:

   Error: Unknown column 'VerifiedBuild' in 'where clause'
   

   Failure 3: Schema match assumption across databases (this session)

   What Claude assumed: Tables with the same name in world and lorewalker_world have identical schemas.
   What's actually true: Several tables differ:

   • world.creature has an extra size column (default -1)
   • world.gameobject has extra size and visibility columns
   • world.npc_vendor has an extra OverrideGoldCost column
   • lorewalker_world.scene_template has an extra RTComment column
   • world.gameobject_template has an extra StringId column

   The import document (written by a human after running DESCRIBE on both sides) accounts for all of these with explicit column lists. Claude never independently verified these schema differences — it relied entirely on the pre-written SQL being correct.

   The underlying pattern

   Claude's training data contains millions of examples of database schemas. It develops strong priors about what columns a table "should" have. These priors are often correct (which reinforces the behavior), but when wrong:

   1. The error is silent — no warning that the fact is from memory
   2. The confidence is identical to verified facts
   3. The user can't distinguish "I checked" from "I assumed" in Claude's output
   4. The cost of the assumption only surfaces when the code is executed

   What "default to verification" looks like

   Before referencing any column in generated SQL:

   1. Run DESCRIBE on the target table (or read it from a recent tool output in context)
   2. Confirm the column exists and note its type
   3. If the DESCRIBE output isn't in the current context, run it — don't rely on memory
   

   This adds ~100ms per table and eliminates an entire class of bugs.
4. VoxCore84 commented on Mar 9, 2026

   

   VoxCore84

   on Mar 9, 2026

   Author

   More actions

   This Is Not a Duplicate

   vs #32289 (our own issue — "Generates incorrect code, reports success")

   The bot flagged our own issue as a duplicate of our other issue. These describe different failure modes in the same pipeline:

   • [Bug] Claude generates incorrect code/SQL and reports it as complete without validation #32289 = the output is wrong (SQL with 32 values for 35 columns). Focuses on the generated artifact.
   • [Bug] Claude defaults to assertion from memory instead of verification from tools #32294 (this issue) = the reasoning process is wrong (stating schema facts from memory instead of checking). Focuses on the decision not to verify before generating.

   #32289 asks: "Why is the code broken?"
   #32294 asks: "Why did Claude think it knew the schema without checking?"

   Fixing #32289 (better code generation) doesn't fix #32294 (defaulting to assertion over verification). And fixing #32294 (always verify before asserting) would prevent #32289 from occurring. #32294 is closer to the root cause.

   vs #27790 ("Model regularly attempts Edit/Write on files it hasn't Read") — OPEN

   #27790 is a narrow, specific instance: Claude tries to edit files without reading them first. That's one manifestation of the broader pattern described in #32294.

   #32294 covers assertion-over-verification across ALL domains — not just file edits:

   1. Schema assertions (assuming columns exist without DESCRIBE)
   2. Configuration assertions (assuming table structures match across databases)
   3. State assertions (claiming "zero errors" without checking logs)

   Our detailed comment documents 3 specific assertion-from-memory failures from a single session, each with the exact assertion made, the actual truth, how it should have been verified, and the cost of both checking (~100ms) and not checking (broken SQL, failed queries, wasted debugging time). #27790 doesn't provide this breadth.

   vs #16546 ("Model attempts file edits without reading file first") — OPEN

   Same as #27790 — a narrow instance (file edits) of the broader pattern (#32294 covers schema, configuration, and state assertions). Our issue has a different scope and different proposed fix ("default to verification" across all domains, not just "read before edit").
5. VoxCore84 commented on Mar 12, 2026

   

   VoxCore84

   on Mar 12, 2026

   Author

   More actions

   Community Validation Update — Asserts Facts from Memory Instead of Checking

   This failure mode has no direct community reports under this name, but it is the root cause behind many other reported failures.

   Indirect Evidence

   • [SAFETY] Claude Code autonomously published fabricated technical claims to 8+ platforms over 72 hours #27430 (SAFETY): The most severe community report — Claude's "sustained confabulation-to-publication pipeline" was driven by asserting from memory. Session N wrote an unverified claim to persistent storage; Session N+1 read it back and treated it as verified fact, building on it and publishing it to 8+ platforms. The memory-to-assertion pipeline is this exact issue at scale.

   • Hacker News threads: "Confident-but-wrong" is the most common description across HN threads about Claude Code quality. Users describe Claude stating database schema facts, API behavior, and file contents from memory that turn out to be wrong when actually checked. Thread 47035289 and 45809090 both describe this pattern.

   • Blog posts: "When Claude Forgets How to Code" (hyperdev.matsuoka.com) describes a research agent that asserted negative search results from memory — claiming information didn't exist rather than admitting it couldn't find it. The assertion was stated with the same confidence as verified results.

   Why This Is Foundational

   This is arguably the most fundamental issue in the taxonomy because:

   1. Every other failure mode is made worse when Claude asserts from memory instead of checking
   2. CLAUDE.md instructions say "verify X" → Claude remembers a prior result for X and skips verification ([Bug] Claude reads files but ignores actionable instructions contained in them #32290 + this issue)
   3. Phantom execution ([Bug] Claude Code reports task completion without actually executing operations #32281) is often memory-assertion with extra steps: Claude "remembers" running a command it didn't run

   Mitigation

   The only reliable mitigation is tool-enforced verification: if a fact needs to be true, a tool call must confirm it in the current session. Memory should be treated as a hint for where to look, never as a source of truth. This is the philosophy behind our edit-verifier hook and the broader PostToolUse verification pattern.

   Part of the completion-integrity taxonomy tracked in #32650.

   👍React with 👍2Reacted by schugazi and Alex Barkan
6. VoxCore84 commented on Mar 12, 2026

   

   VoxCore84

   on Mar 12, 2026

   Author

   More actions

   Pass 5 — Final Evidence Update (Asserts Facts from Memory)

   Root Cause Evidence

   • #29230 — KV cache stale context identified as technical root cause. When the KV cache retains stale state, the model's "memory" of schema facts, file contents, and system state becomes actively wrong — not just unverified, but outdated.

   Subagent Phantom Execution (NEW — 5 reports)

   Pass 5 identified a new variant: Task tool subagents fabricate Bash/MCP outputs. The parent agent receives fabricated results and asserts them as verified facts. This is memory-assertion at one layer of indirection — the subagent asserts from memory, the parent treats it as tool-verified.

   #27430 (SAFETY) — Compounding Across Sessions

   The most severe instance: Session N writes an unverified assertion to persistent storage → Session N+1 reads it back as "verified fact" → builds on it → publishes to 8+ platforms. Memory assertion + persistent storage = automated confabulation pipeline.

   HN Threads

   • "Confident-but-wrong" is the Create SECURITY.md #1 description across 47035289, 45809090, and 4+ other threads
   • Users describe Claude stating database schema facts, API behavior, and file contents from memory that are wrong when checked

   Why This Is Foundational

   Every other failure mode is made worse when Claude asserts from memory:

   • Ignores CLAUDE.md ([Bug] Claude reads files but ignores actionable instructions contained in them #32290) + asserts compliance from memory = silently non-compliant
   • Phantom execution ([Bug] Claude Code reports task completion without actually executing operations #32281) = asserting "I ran that command" from memory
   • Blind edits ([Bug] Claude does not verify file edits applied correctly — blind edit-and-proceed #32658) = asserting "the edit applied" from memory
   • Tautological QA ([Bug] Verification queries are tautological — cannot return failure results #32291) = asserting "this check proves success" from memory

   The fix is tool-enforced verification: if a fact needs to be true, a tool call must confirm it in the current session.

   Part of the completion-integrity taxonomy tracked in #32650.
7. github-actions

   added

   staleIssue is inactiveIssue is inactive

   on Mar 26, 2026
8. VoxCore84 commented on Mar 27, 2026

   

   VoxCore84

   on Mar 27, 2026

   Author

   More actions

   Still reproducing. Claude will state schema column names from memory rather than running DESCRIBE, leading to incorrect SQL that it then reports as "applied cleanly." See #32650.
9. github-actions

   removed

   staleIssue is inactiveIssue is inactive

   on Mar 27, 2026
10. raye-deng commented on Mar 31, 2026

    

    raye-deng

    on Mar 31, 2026

    More actions

    This is the inverse of the validation problem — instead of generating wrong output and reporting it as correct, Claude generates confident assertions about system state without checking.

    The pattern is particularly dangerous because it compounds: a wrong assertion in session N becomes cached context for session N+1, and the error propagates through the agent's conversation history. By the time someone notices, the original incorrect assumption may be buried deep enough that it looks like a legitimate fact.

    This is related to what research on AI code review calls "correlated failures" — when the generating agent and any reviewing agent share the same training distribution, they tend to make the same mistakes and can't catch each other's errors. The only reliable fix is an external reference point: actual tool output, live schema queries, or executable tests.

    A practical pattern that helps:

    1. Force tool-before-assert — Configure the agent so that any claim about system state (file contents, schema, config) must be immediately preceded by a tool call that reads the actual state. No tool call = invalid claim.
    2. Stale context markers — Track the age of cached information and require re-verification for anything older than N minutes or after any write operation.
    3. Schema drift detection — Compare the agent's internal model of a schema/contract against the live version before generating code. If there's a mismatch, surface it explicitly rather than silently using the stale version.

    For catching these kinds of assertion-without-evidence patterns in generated code, tools like npx @opencodereview/cli scan . --sla L1 can flag when code relies on assumptions that should have been verified.

    The fix really needs to be at the agent framework level — make verification a hard requirement, not a soft suggestion.
11. VoxCore84 commented on Apr 1, 2026

    

    VoxCore84

    on Apr 1, 2026

    Author

    More actions

    @raye-deng — good framing on the 5 defect categories, especially "correlated failures" (AI makes the same wrong assumption in both the code and the test). That's exactly what we see with SQL: Claude generates a 32-value INSERT for a 35-column table, then writes a "verification" query that also assumes 32 columns — the check can't catch what it shares the same blind spot about.

    The "force tool-before-assert" mitigation aligns directly with what this taxonomy proposes — a mandatory gate where the agent must produce tool output before any factual claim about schema, file contents, or execution results. The cost is trivially low (a DESCRIBE query is ~100ms) but the model consistently skips it in favor of asserting from memory.
12. kironkeyz commented on Apr 10, 2026

    

    kironkeyz

    on Apr 10, 2026

    More actions

    I need to escalate a serious reliability issue with Claude Code.

    This is not about incorrect answers — it’s about the system presenting incomplete workflows as fully verified.

    Claude repeatedly:

    • claims a full pipeline ran successfully
    • produces confident outputs
    • then later admits critical data layers were missing or not verified
    • and changes its conclusions after basic re-checking

    This creates a major trust issue because there is no way to distinguish between:

    • verified results
    • partial results
    • or narrative guesses

    This behavior has already impacted real decision-making and caused financial consequences.

    The core problem:
    Claude does not enforce verification before completion and does not fail when required data is missing.

    I’ve already submitted detailed GitHub issues, but I want to make sure this is properly escalated internally because this is a system-level trust failure, not a normal bug.

    Can you confirm this is being escalated to the product/engineering team?
13. github-actions

    added

    staleIssue is inactiveIssue is inactive

    on May 8, 2026
14. github-actions commented on May 24, 2026

    

    github-actionsBot

    on May 24, 2026 – with GitHub Actions

    More actions

    Closing for now — inactive for too long. Please open a new issue if this is still relevant.
15. github-actions

    closed this as not plannedon May 24, 2026
16. github-actions commented on Jul 15, 2026

    

    github-actionsBot

    on Jul 15, 2026 – with GitHub Actions

    More actions

    This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.
17. github-actions

    locked as resolved and limited conversation to collaborators on Jul 15, 2026