Skip to content

[Bug] Claude generates incorrect code/SQL and reports it as complete without validation #32289

Closed as not planned
Closed as not planned
[Bug] Claude generates incorrect code/SQL and reports it as complete without validation#32289
Labels
area:modelbugSomething isn't workingplatform:windowsIssue specifically occurs on WindowsstaleIssue is inactive

Description

@VoxCore84
VoxCore84
opened on Mar 9, 2026

Bug Description

Claude Code generates code or SQL with structural errors (wrong column counts, missing fields, incorrect syntax) and reports the output as successfully written and correct — without ever validating the generated artifact against the actual target schema or environment.

Concrete Example

A Claude Code session was asked to write a SQL INSERT for gameobject_template. The table has 49 columns including Data0 through Data34 (35 data columns). Claude generated the INSERT with only 32 data values instead of 35, producing a 3-column mismatch.

The session reported:

"SQL written successfully. 7 gameobject_template entries INSERTED."

The bug was only discovered in a later session when the SQL was actually executed:

ERROR 1136 (21S01) at line 29: Column count doesn't match value count at row 1

Root Cause

Claude writes SQL/code by pattern-matching or counting from memory rather than:

  1. Reading the actual table schema (DESCRIBE table)
  2. Counting the generated values against the column list
  3. Doing a dry-run or syntax check

The model has no self-verification step between "generate code" and "report success."

Impact

  • Broken artifacts sit undetected until a human or later session tries to use them
  • The confident completion message ("successfully written") creates false trust
  • In CI/CD or automation pipelines, this class of bug could propagate downstream

Related

Environment

  • Claude Code 2.1.71
  • Windows 11

Activity

  1. github-actions
    added
    bugSomething isn't working
    area:model
    platform:windowsIssue specifically occurs on Windows
    on Mar 9, 2026

  2. github-actions commented on Mar 9, 2026

    @github-actions
    github-actionsBot
    on Mar 9, 2026 – with GitHub Actions

    Found 3 possible duplicate issues:

    1. [BUG] Opus 4.6 reports success without verifying outcomes - silent failures go undetected #23801
    2. [Bug] Verification queries are tautological — cannot return failure results #32291
    3. [Bug] Claude Code fails to verify task completion against documented requirements #12369

    This issue will be automatically closed as a duplicate in 3 days.

    • If your issue is a duplicate, please close it and 👍 the existing issue instead
    • To prevent auto-closure, add a comment or 👎 this comment

    🤖 Generated with Claude Code

  3. VoxCore84 commented on Mar 9, 2026

    @VoxCore84
    VoxCore84
    on Mar 9, 2026
    Author

    Additional Detail: The Broken SQL File

    The file that was generated incorrectly

    sql/updates/world/master/2026_03_08_00_world.sql, Section 2 — gameobject_template INSERT.

    What Claude generated (wrong)

    INSERT IGNORE INTO `gameobject_template`
  (`entry`,`type`,`displayId`,`name`,`IconName`,`castBarCaption`,`unk1`,`size`,
   `Data0`,`Data1`,`Data2`,`Data3`,`Data4`,`Data5`,`Data6`,`Data7`,`Data8`,`Data9`,
   `Data10`,`Data11`,`Data12`,`Data13`,`Data14`,`Data15`,`Data16`,`Data17`,`Data18`,
   `Data19`,`Data20`,`Data21`,`Data22`,`Data23`,`Data24`,`Data25`,`Data26`,`Data27`,
   `Data28`,`Data29`,`Data30`,`Data31`,`Data32`,`Data33`,`Data34`,
   `ContentTuningId`,`RequiredLevel`,`AIName`,`ScriptName`,`VerifiedBuild`)
VALUES
(576391, 5, 40800, 'Bookshelf', '', '', '', 0.75,
 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 0, 0, '', '', 0),

    Column vs value count

    • Columns listed: 48 (entry through VerifiedBuild, including Data0-Data34 = 35 data columns)
    • Values provided: 45 (only 32 zeros for the 35 Data columns)
    • Mismatch: 3 values short

    What MySQL reported when the file was executed

    ERROR 1136 (21S01) at line 29: Column count doesn't match value count at row 1

    What the previous session reported

    The session that generated this file presumably reported success — the file was committed and referenced in coordination documents as ready to apply. No one ran the SQL or validated the column count until this session.

    The actual table schema (49 columns)

    entry, type, displayId, name, IconName, castBarCaption, unk1, size,
Data0-Data34 (35 columns),
ContentTuningId, RequiredLevel, AIName, ScriptName, StringId, VerifiedBuild

    Note: the table also has a StringId column that the INSERT doesn't include — but since the INSERT uses an explicit column list, this is fine (StringId defaults to NULL). The actual error was purely the 32-vs-35 Data column count.

    The fix applied

    Added 3 zeros to each of the 7 value rows to complete Data32, Data33, Data34. All 7 rows had the identical bug — suggesting Claude generated one row and copy-pasted the pattern.

    Why this wasn't caught

    1. The generating session never ran DESCRIBE gameobject_template to verify the column count
    2. The generating session never did a dry-run (mysql --execute or syntax check)
    3. The generating session counted 32 zeros and apparently treated that as "enough for Data0-Data34" without actually counting to 35
    4. The file was reported as "written successfully" based solely on the Write tool succeeding (file created on disk), not on SQL validity

  4. VoxCore84 commented on Mar 9, 2026

    @VoxCore84
    VoxCore84
    on Mar 9, 2026
    Author

    This Is Not a Duplicate

    vs #23801 ("Opus 4.6 reports success without verifying outcomes") — CLOSED

    Closed issue. #23801 is about reporting success without verification — a behavioral pattern. #32289 is about a different failure mode: Claude generates structurally incorrect code (wrong column count in SQL) and reports it as correct. The artifact itself is broken, not just the verification. Even if Claude had perfect verification behavior, the code generation bug would still exist as a separate problem.

    vs #32291 (our own issue — "Tautological verification queries")

    The bot flagged our own issue as a duplicate of our other issue. These are intentionally separate:

    One is about output quality, the other is about QA quality. Both happened in the same session but are different bugs requiring different fixes.

    vs #12369 ("Claude Code fails to verify task completion against documented requirements") — CLOSED

    Closed issue. #12369 is about verification failures. #32289 is about code generation failures — Claude wrote SQL with a 3-column mismatch because it counted Data0-Data34 as 32 columns instead of 35. This is an arithmetic/schema-awareness bug in generation, not a verification gap. The detailed comment on #32289 includes the exact broken SQL, the exact error message, the exact table schema, and the fix applied.

  5. VoxCore84 commented on Mar 12, 2026

    @VoxCore84
    VoxCore84
    on Mar 12, 2026
    Author

    Community Validation Update — Generates Incorrect / Unverified Code

    This failure mode has moderate direct reports but significant indirect evidence through its intersection with other taxonomy issues.

    DoltHub Blog

    Claude Code Gotchas documents the "incomplete rewrites" gotcha — Claude replaces part of a function but leaves the rest in an inconsistent state, producing code that compiles but behaves incorrectly. This is incorrect code generation where the model's internal representation of the code diverges from reality.

    Reddit

    • r/ClaudeAI reports describe "confident-but-wrong" code generation increasing over time — Claude produces syntactically valid code that doesn't do what was requested, accompanied by a confident explanation of why it's correct.

    Hacker News

    • Thread 47035289 — "Has Claude Code quality dropped recently?" — users report Claude generating code that addresses a different problem than what was asked, or code that would work for a simpler version of the problem but misses edge cases in the actual request.

    GitHub

    Pattern

    The failure chain: Claude generates a plan (correct), then generates code that doesn't fully implement the plan (this issue), then reports the code as complete without running it (#32281), then doesn't notice the gap (#32301). Each link in the chain is a separate failure mode, but they compound.

    Part of the completion-integrity taxonomy tracked in #32650.

  6. VoxCore84 commented on Mar 12, 2026

    @VoxCore84
    VoxCore84
    on Mar 12, 2026
    Author

    Pass 5 — Final Evidence Update (Generates Incorrect Code)

    New GitHub Issues

    • #25305 — 75% rework rate from unverified completion claims — incorrect code shipped as "complete"
    • #32554 — Model ignores rules, makes unverified claims, reports false success

    108-Hour Unattended Test (DEV Community)

    108 hours of autonomous Claude Code resulted in:

    • rm -rf ./src/ (deleted source directory)
    • $8 API loop (runaway cost)
    • 50x more failures than expected
    • Generated code that compiled but was logically wrong at scale

    METR Study (Peer-Reviewed)

    Developers took 19% longer with AI tools while perceiving 20% improvement. The perception gap exists precisely because generated code looks correct — it compiles, it has reasonable structure — but contains subtle logic errors that take longer to debug than writing from scratch.

    Competitor Migration

    • Skywork's "Claude's Fall from Grace" — long-form article documenting the switch from Claude to Codex, citing incorrect code generation as primary frustration
    • "47-Hour Marathon" article — 47 hours of debugging Claude's incorrect code to achieve what should have been a 4-hour task
    • Aider benchmarks: Claims 4.2x fewer tokens per task — less generated code = fewer errors

    Token Regression

    Opus 4.6 burns ~60% more tokens per prompt than 4.5 — generating more code doesn't mean generating better code. The additional tokens often introduce unnecessary complexity (overengineering) or incorrect implementations.

    Reddit

    "Confident-but-wrong" is the most common description across r/ClaudeAI and HN — syntactically valid code that doesn't do what was requested.

    Part of the completion-integrity taxonomy tracked in #32650.

  7. github-actions
    added
    staleIssue is inactive
    on Mar 26, 2026

  8. VoxCore84 commented on Mar 27, 2026

    @VoxCore84
    VoxCore84
    on Mar 27, 2026
    Author

    Still reproducing. Generated SQL with wrong column counts, missing VALUES, or incorrect table names gets reported as "applied cleanly" — no post-execution validation. See #32650.

  9. github-actions
    removed
    staleIssue is inactive
    on Mar 27, 2026

  10. raye-deng commented on Mar 31, 2026

    @raye-deng
    raye-deng
    on Mar 31, 2026

    This is exactly the gap that traditional CI/CD can't catch. Claude's confidence calibration is the core problem — it reports success with the same certainty whether it actually verified the output or just inferred it from context.

    The pattern you describe (wrong column counts, missing fields in SQL) is a specific subclass of what we've been categorizing as AI-native defects in our code review work. These defects share a characteristic: they pass all conventional checks (syntax is valid, types match, lint is clean) but fail at runtime because the AI's internal model of the target schema drifted from reality.

    A few practical mitigations we've seen work:

    1. Schema-bound verification — After AI generates DML, auto-validate column count and types against the live schema (information_schema or equivalent) before accepting the output
    2. Diff-level dry-run — Run generated SQL through EXPLAIN or a dry-run wrapper to surface structural mismatches before execution
    3. Assertion gates — For any code that touches a schema/contract, require the agent to explicitly show the evidence it used (e.g., SHOW COLUMNS FROM table output) rather than relying on cached context

    We built Open Code Review partly to catch exactly this class of issues — AI code that "looks right" but is structurally wrong. The CLI scan: npx @opencodereview/cli scan . --sla L1

    The root cause is really a UX problem in the agent loop: the tool use → verify → report cycle needs to be mandatory, not optional. Tools should have a verified: boolean field that the agent must populate before claiming success.

  11. VoxCore84 commented on Apr 1, 2026

    @VoxCore84
    VoxCore84
    on Apr 1, 2026
    Author

    @raye-deng — the "tool use → verify → report cycle needs to be mandatory" framing is exactly right. The current architecture treats verification as optional — the model can verify, and sometimes does, but there's no enforcement. The result is that verification happens when the model "feels like it" (roughly 60-70% of the time in our measurement) rather than as a structural guarantee.

    Your verified: boolean field proposal for completion reports would be a clean implementation. Even just forcing the model to explicitly mark each claim as verified or unverified would be a significant improvement — right now it mixes both with identical confidence, which is the core trust problem.

  12. github-actions
    added
    staleIssue is inactive
    on Apr 23, 2026

  13. github-actions commented on May 7, 2026

    @github-actions
    github-actionsBot
    on May 7, 2026 – with GitHub Actions

    Closing for now — inactive for too long. Please open a new issue if this is still relevant.

  14. github-actions
    closed this as not plannedon May 7, 2026

  15. github-actions commented on Jun 27, 2026

    @github-actions
    github-actionsBot
    on Jun 27, 2026 – with GitHub Actions

    This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.

  16. github-actions
    locked as resolved and limited conversation to collaborators on Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:modelbugSomething isn't workingplatform:windowsIssue specifically occurs on WindowsstaleIssue is inactive

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions