Four Ways a Test Can Lie to You

Green and worthless: what verifying one small application taught me, when an AI agent wrote most of the code.

The Sanity DEV Challenge wrapped up last week, and The Cellar won Path Two. The judges called the build writeup “a study in making verification hard to fake,” which is a generous reading of what was mostly me discovering that my verification was broken.

My submission covered what I built. This is about the part of the build I keep thinking about: how often my checks were wrong.

I spent two weeks building a wine cellar application that can be read at any date in its history. One developer, one agent, about 530 tests by the end. Three of the four real bugs in the project were invisible to every check designed for the feature they lived in.

That was uncomfortable, so I went looking. What I found was worse than missing tests: checks that were green, or red, and meant nothing either way. There turned out to be four distinct ways that happens, and I hit all four in a project where I was being unusually careful.

Lab scientists have had vocabulary for this for a long time. An experiment needs a positive control, proof that your method can detect the effect when it's present, and a negative control, proof that it stays quiet when nothing is there. Each of the four lies below is a missing control, or a control pointed at the wrong thing.

First: the test that cannot fail

No positive control.

The application resolves a wine's drinking window from dated claims. Claims can be proposed by an agent and must be accepted by a person, and a proposed claim is supposed to change nothing until someone accepts it.

That is the load-bearing rule of the whole review workflow, so it got a test. Build the cellar, inject a proposed claim, assert that all 542 bottle states are byte-identical. Green.

It was measuring the fixture, not the rule. Resolution picks the highest authority tier first, then the most recent claim within that tier:

Both claims are the same tier, so the one under test loses on recency, and it loses whether it is proposed or accepted. Delete the entire review-state filter from the source and the test still passes.

What caught it was a companion assertion written for a different reason: accepting must change at least one state. That one failed, which is how I learned the first one was vacuous.

The rule falls straight out of it. Every test asserting that X has no effect needs a paired assertion showing the effect exists when X is enabled. Without the pair, you haven't tested that X is inert. You've tested that your fixture was inert.

An audit of the rest of the suite found two more of the same shape. One claimed to check that a bottle opened during a period doesn't count as a missed opportunity, but two separate mechanisms excluded that bottle, so the clause under test could be deleted without anything turning red.

I assumed that clause couldn't be isolated and told the agent to fall back to a weaker assertion. It ran the experiment instead, and found a fixture by realizing that "now" in this system is a parameter rather than the clock. As of June 2024, what had I missed that year? is an ordinary question to ask a time machine, and it made the case constructible. The fix was better than the instruction.

All three fixes were verified by mutation: delete the clause the test claims to cover, and watch it finally fail. The original fixture stayed in the suite, renamed to say what it actually tests, because a record of the mistake is more useful than a quiet deletion.

Second: the check that always fails

No negative control.

A week later, building a public static bundle, I added a grep of the built output for leaked API credentials. Five seconds to run, and the failure it catches is unrecoverable, so it seemed like an obvious win.

It fired on every clean build. The pattern was sk[A-Za-z0-9]{20,}, and it matched skipCrossDatasetReferenceValidation, a legitimate export name inside @sanity/client.

This is the mirror image of the first lie, and in practice it's worse. A test that never fails is invisible: nobody notices it isn't doing anything. A check that always fails is noticed immediately, and then it is dismissed, every time, by the person who knows it's a false positive. After a week, the real failure it exists to catch would scroll past with the rest.

A check that always fails certifies exactly as little as one that never can. It just looks more diligent while doing it.

The fix was small. The bound went from twenty characters to forty, which real credentials clear easily and that export name doesn't. Then the scan moved into CI, where it runs on every push instead of whenever someone remembers. A check that always fails is worthless, and a check nobody runs is worth the same.

In control terms, the original grep had no negative control: nothing showing it stays quiet on a build known to be clean. One run against a clean build was all it took to expose it, which is how it was found, by running it rather than shipping it. The other half, proving the tightened pattern still fires on a planted fake credential, is the test I'd add next.

Third: the correct test of a wrong contract

Both controls present, measuring the wrong thing.

This one is the hardest to spot, because there is nothing structurally wrong with the test at all.

The application has a date control spanning 1996 to 2042. Typing a date into it worked for month and day, and the moment you started typing the year, the field snapped back to 1 January 1996. You had to finish with the picker.

The cause is a genuinely surprising piece of browser behavior. A native date input never reports partial text. It reports a complete, well-formed date with the year zero-padded to four digits. Typing 2023 into a field already holding a month and day emits this:

0002-12-31   ->  clamped to the 1996 floor
0020-12-31   ->  clamped to the 1996 floor
0202-12-31   ->  clamped to the 1996 floor
2023-12-31   ->  accepted, but the field was overwritten three keystrokes ago

Every intermediate value is a real, valid date. Every one is out of range. Every one got clamped, and the controlled input wrote the floor back over the edit.

There was a test covering this. It asserted that parseDateInput('1901-01-01') returns the 1996 floor. It passed, and it would have failed if clamping broke. Positive control, negative control, meaningful assertion, all present.

The contract it encoded was wrong. Clamping an out-of-range date is correct when the user has finished entering it and wrong while they're still typing, and the test had no way to know the difference because the specification didn't either.

That's the dangerous category, because the first two look like mistakes once you see them and this one doesn't. An assertion about clamping out-of-range input reads as diligence. It would survive code review. It survived mine. I covered this failure mode in more depth in The Contract Discovery Bottleneck.

Fourth: the right test against the wrong system

Controls present, pointed at something else.

To test the review workflow safely, I set up a staging copy of the dataset. The plan: point the authoring tool and the application at staging, propose a claim, accept it, and watch the right bottles change state, all without touching production.

The authoring tool honored the environment variable that selects a dataset. The application, built by the same command with the same variable set, silently ignored it. Its built bundle was byte-identical to a build with the variable unset, falling back to production. No warning, no error. It built and ran perfectly.

So the authoring tool would have written a proposal into staging while the application reported production. Every number on the screen would have been internally consistent, and about a different cellar.

No amount of rigor in the test helps here. The assertions can be perfect and still certify the wrong thing, because the question that came first, which system am I actually looking at?, was never asked.

The fix had two parts. The application's dataset became a hand-edited literal, so it can't be silently wrong: any change to it shows up in the diff. And the application now displays the dataset name in its header whenever it isn't production. The reasoning matters more than the fix. The question was never how to point it at staging. It was whether I was sure which dataset I had just written to.

The examples are the paths you already walk

Separate from all four, and the one I think about most.

The same date control had an arithmetic bug: the span between 31 January and 1 March computed wrong, because the implementation borrowed days from February and silently dropped the remainder.

I had five verification dates for that feature. They were chosen carefully, from real data, before the code existed. They're the dates in the demo video and the screenshots.

All five passed under the buggy version. The application worked. The screenshots were right. Every date I planned to show anyone was fine.

What found it was a property test across all 17,167 positions the slider can occupy, asserting an identity that has to hold for any pair of dates. The bug lived in a region nobody would drive to on purpose.

Demo-driven verification proves the paths you already walk. That's not nothing, and it's not what you think you're getting.

What actually worked

None of it was "write more tests."

Oracles generated before the implementation, with an explicit instruction not to repair them. Expected outputs computed independently and committed before the code existed. The prompt said to treat them as an oracle rather than as output to be fixed, and to stop and report on any disagreement. When a test fails, the cheapest available move is adjusting the fixture, and that instinct needs to be foreclosed in advance rather than resisted in the moment.

Mutation, but only on the tests that matter. Delete the clause the test claims to cover. If it stays green, the test is decorative. This found two of the three vacuous tests, and the agent started doing it unprompted after the first.

Measurement instead of reasoning, on anything with a number in it. A performance bug hid in four lines of correct-looking memoization that had already passed review. Rebuilding an index cost 6 ms; recomputing the thing the index served cost 0.07 ms. The code memoized the wrong one. Nothing about reading it revealed that. Benchmarking both took ten minutes.

A gate that named what would not count as proof. For checks that needed eyes on a rendered screen, I wrote out the full path being tested and then listed the ways it could be faked: don't satisfy this by calling the module from a script, querying the data another way, or pointing at the test suite. Then: you cannot see a browser, so do not report this as passed. Give me the command, the URL, and the expected numbers, and I'll confirm.

That last one matters more in an AI-assisted build than it used to. A model asked to confirm something it cannot observe is under real pressure to confirm it anyway. The fix isn't trusting it less. It's giving it an honest third answer between passed and failed: not verified, here's how you check. A result nobody could observe isn't a pass, and a harness that can't say so will round it up to one.

Agents are good. They aren't exempt.

The agent wrote nearly all of the code in this project, tests included. So it's worth being plain about who wrote most of the lies above: the same tool that wrote the code they were checking.

That isn't an indictment. Over two weeks the agent was better than I am at a lot of this. Before every stage, it read my specification against the platform and found errors in it; by the eighth session, that pass was turning up ten conflicts at a time, two of them outright mistakes in my spec. When I proposed testing a Function against production with a trigger scoped to a document type that didn't exist, it declined, correctly: scoping a trigger is safe, but the only way to invoke one is to write a matching document, and my plan would have left two permanent mutations in production's history to avoid creating a scratch dataset. It found the fixture I told it was impossible. It measured the cost of a feature I hadn't asked it to measure.

It was also wrong in ways a careful human would recognize. It confidently reported that an API couldn't do something the documentation demonstrated, then fixed it in one line once asked to check rather than assert. It reached the right conclusion about date comparison for the wrong reason, which is right only until the case changes. Asked for a list of the polish work that remained, it drafted the list from the build plan instead of the code, and three of the items were already done. A plan describes work that was scheduled, not work that is left.

And it broke rules. Twice, after being told not to run git commands that change the repository, it ran git checkout to undo its own uncommitted edits. Earlier, it had modified an oracle file it was explicitly told not to touch, then restored it. Every time, it reported the violation unprompted, before I found it. "The agent broke a rule and told me" is more useful data than either cleaner story, "it never broke a rule" or "it broke a rule and I caught it." The rule now says the file tools are the only way to revert anything, including its own edits.

The pattern underneath is the part worth keeping. An agent writes tests the way it writes code: quickly, fluently, plausibly, and from the same understanding of the problem. When one tool writes an implementation and its tests from the same prompt, they share one set of assumptions, and a wrong assumption gets encoded twice and then agrees with itself. That's the third lie, and agents make it cheaper to commit at scale. Speed doesn't remove the verification problem. It moves the bottleneck there, which I've written about before.

What changed the outcome wasn't trusting the agent less. It was making verification structurally hard to fake: oracles committed before the code, controls on every check, gates that named what wouldn't count, and final confirmation in the hands of the one participant with eyes. Under those conditions, the agent didn't just pass the checks. It started improving them.

It happened again

A month later I built a small experiment harness for something unrelated, comparing small language models against plain code on a Raspberry Pi. I knew this taxonomy by then. I found the same lies anyway.

A fixture that couldn't distinguish the behaviors it was meant to compare. A metric that rewarded a model for always giving the safe answer, which is a check that always passes wearing a lab coat. And a missing directory that ran an entire round of questions against an empty reference corpus, producing a detailed, confident, wrong write-up: the right test against the wrong system, again.

Knowing the categories didn't prevent them. It made them faster to recognize, which is the most I'd claim for any checklist.

The honest limit

The oracles catch implementation bugs. They can't catch specification bugs, because I generated them from the same specification.

One of them was stronger by accident. The missed-opportunity oracle used brute-force evaluation of every day in a period, while the implementation used a boundary scan that only evaluates the dates where state can change. Two genuinely different algorithms agreeing is better evidence than one algorithm agreeing with itself, for the same reason a winemaker checks a refractometer against a hydrometer: instruments that fail differently are unlikely to fail together. I wrote about that idea recently, too. But even that oracle shares the specification's assumptions about what a missed opportunity is.

Tests tell you the implementation disagrees with your expectations. They can't tell you whether your expectations deserve to win.

What I'd keep

Four habits, one per lie:

Give every inertness test a positive control. If it asserts that something has no effect, prove the effect exists when it should.

Give every alarm a negative control. If it can't be shown passing on known-clean input, it isn't a check. It's noise with a red icon.

Test properties across the domain, not just the examples. Your examples are the paths you already know work, and the contract is only as good as the cases you thought to ask about.

Verify the target before the result. Make the system under test visible on the screen and in the diff, so "which one am I looking at?" is never a guess.

None of those habits is about AI, and that's the point. The agent made every one of these lies cheaper to commit and faster to find. It didn't change which checks were honest.

The rest is the usual advice, and it wasn't what saved me.

This came out of building The Cellar, an event-sourced wine collection for the Sanity DEV Challenge. The full build is documented in the friction logs in that repository, including every check above and the audits that found them.