Coder Legion - Recent posts tagged benchmarking

Benchmarking the FSM: Pure Strings, Already Fast — and About to Get Faster

Sun, 06 Sep 2026 19:41:30 +0000

Benchmarking the FSM: Pure Strings, Already Fast — and About to Get Faster

There's a particular reaction you tend to get when you tell an experienced systems programmer that your architecture is string-backed:

"Don't use strings."

It's almost reflexive.

Use enums.

Use integers.

Use precomputed IDs.

Avoid dictionary lookups.

Watch the garbage collector.

And, above all, don't put strings anywhere near a hot loop.

There's good engineering history behind those instincts. Strings can be expensive. Poorly designed string-heavy systems can generate allocations, create unnecessary comparisons, and turn perfectly reasonable abstractions into performance problems.

But there's a problem with rules of thumb:

Eventually, you have to measure.

So that's what I did.

The current FSM_API is purely string-backed.

Not string-backed with an integer cache hiding underneath.

Not enums converted to strings at the API boundary.

Not generated IDs.

Strings.

And I wanted to know what that actually costs.

The Baseline

Using BenchmarkDotNet, I measured the cost of updating active process groups as the workload increased from one group to ten and then to fifty.

The results:

Method	Active Process Groups	Mean	Error	StdDev	Gen 0	Allocated
`UpdateMultipleGroups`	1	305.1 ns	1.89 ns	1.76 ns	0.0572	360 B
`UpdateMultipleGroups`	10	3,115.7 ns	20.14 ns	18.84 ns	0.5722	3,600 B
`UpdateMultipleGroups`	50	15,736.6 ns	121.61 ns	107.81 ns	2.8687	18,000 B

The first number is the one that gets your attention.

305 nanoseconds.

That's the measured cost of updating one active process group.

A 60 FPS application has approximately 16.67 milliseconds available for an entire frame.

That's 16,670,000 nanoseconds.

The measured FSM operation therefore consumes approximately 0.0018% of that frame budget.

However, the more interesting result isn't the 305 ns.

It's what happens when I increase the workload.

Fifty Groups. Fifteen Microseconds.

At ten active process groups, the benchmark reports:

3.12 µs

At fifty:

15.74 µs

That's approximately 0.0157 milliseconds.

Against a 16.67 ms frame budget, fifty active process groups consume approximately:

0.094% of one frame.

That's not a theoretical "eventually it should scale" statement.

That's measured behavior.

And the scaling is almost perfectly linear.

One group:

305 ns

Ten groups:

3.12 µs

Fifty groups:

15.74 µs

The architecture isn't encountering some mysterious performance cliff as the workload grows.

More groups mean more work.

Approximately proportionally more work.

That's exactly what we want from this kind of system.

Why Fifty?

Fifty isn't a magic number.

It isn't a claim that fifty process groups represents the maximum this FSM_API can handle.

It doesn't.

It is simply a useful workload for demonstrating what happens when we stop talking about one isolated state machine and start talking about multiple active systems being serviced together.

That's an important distinction.

A single-state benchmark can make almost anything look fast.

Real software rarely gets the luxury of doing only one thing.

A game might have character state, animation state, weapon state, UI state, navigation state, interaction state, environmental state, and networking state all operating simultaneously.

An application outside of games has its own collection of concurrent concerns.

The useful question therefore becomes:

What happens when we start stacking these things together?

In this benchmark, the answer is refreshingly boring.

The cost grows with the workload.

It doesn't suddenly explode.

What This Benchmark Does — and Doesn't — Prove

I believe it is worth being precise here.

This benchmark is measuring the cost of the FSM's process-group update machinery.

It is not claiming that the actual work performed by your states is free.

If your state executes an expensive pathfinding operation, performs a large physics calculation, processes network data, or does something computationally expensive, the FSM doesn't magically make that work disappear.

That's not what we're measuring.

We're measuring the machinery responsible for organizing and updating those processes.

And that machinery is currently taking approximately fifteen microseconds for fifty active groups.

That distinction is important because abstractions are often blamed for the work they organize.

A state machine can be blamed for a slow AI routine simply because the AI routine happens to execute inside a state.

This benchmark lets us separate those concerns.

The abstraction has a cost.

I measured it.

And at this scale, that cost is fairly small.

But There's Something Important Hiding in That Table

The benchmark also reports allocation.

One group:

360 bytes

Ten:

3,600 bytes

Fifty:

18,000 bytes

That is approximately 360 bytes per active process group per update under this benchmark.

At 60 FPS, if fifty groups were updated every frame, that corresponds to:

approximately 1.08 MB of managed allocation per second.

That isn't catastrophic.

But it isn't zero either.

And this is where the benchmark becomes particularly useful.

Because I'm not finished...

This Is the Performance Baseline — Not the Destination

The FSM_API architecture is currently fully string-backed.

That's important because the numbers above aren't the result of some hidden integer representation doing the real work underneath.

The strings are actually part of the machinery being measured.

And that's intentional.

One of the architectural goals of FSM_API has always been to make state management expressive and discoverable.

States can have meaningful names.

Process groups can be identified without forcing the developer to maintain a parallel universe of integer constants.

The API remains readable.

The architecture remains flexible.

And now we have measured what that flexibility costs.

The answer is:

Not very much.

But there is another conclusion hiding in that result.

If this is what the system can do while paying the cost of its string-backed implementation, then we now have a very interesting optimization target.

The Next Version Removes the String Cost

I'm currently refactoring the underlying implementation so that the FSM can retain its string-facing architecture and developer experience while moving the internal backing representation toward integer indexing.

That's an important distinction.

The goal isn't to tell developers:

"Sorry, strings are slow. Go back and rewrite your application using integers."

The goal is the opposite.

Keep the expressive interface. Optimize what happens underneath it.

The developer can continue thinking in terms of:

"Idle"

"Running"

"Paused"

"Processing"

while the machinery underneath increasingly has the opportunity to operate on compact integer representations.

That's the architecture we're moving toward.

And the benchmark above gives us something extremely valuable:

a control group.

We already know what the purely string-backed implementation costs.

When the integer-backed implementation is complete, we can run the same benchmark again.

Same workload.

Same benchmark harness.

Same measurement methodology.

Then we can put the two implementations beside each other and see exactly what the architectural change bought us.

We Can Already See the Shape of the Opportunity

We shouldn't manufacture a performance number before we have measured it.

That's one of the reasons I like this experiment.

We have a baseline.

We have a known workload.

We have known execution times.

We have known allocations.

Now we have something to beat.

The current 50-group benchmark takes:

15,736.6 ns

and allocates:

18,000 bytes.

Suppose, purely as a thought experiment, the integer-backed implementation reduced execution time by 25%.

That would take the measured operation from approximately:

15.7 µs → 11.8 µs

A 50% reduction would produce:

15.7 µs → 7.9 µs

A 75% reduction would produce:

15.7 µs → 3.9 µs

Those aren't predictions.

They're not marketing numbers.

They're simply showing the size of the optimization space we're about to measure.

And the allocation story could be even more interesting.

If the integer-backed representation eliminates allocations currently associated with string construction or other string-oriented work in the measured path, the memory profile could change substantially even if the raw execution-time improvement turns out to be modest.

That matters for systems that run continuously.

A few microseconds once isn't particularly interesting.

A few microseconds and a small allocation repeated every frame, across hundreds or thousands of objects, is a different conversation.

This is where optimization becomes architecture.

The Goal Isn't "Integer Good, String Bad"

This is probably the most important point in the entire experiment.

I'm not refactoring FSM_API because the string-backed implementation failed.

It didn't.

It's fast.

It scales predictably.

It's readable.

It's flexible.

And it has now survived measurement.

The integer-backed refactor is about taking an architecture that is already useful and asking:

Can we preserve everything developers like about it while making the machinery underneath even cheaper?

That's a much more interesting engineering problem.

We're not sacrificing usability to chase a benchmark score.

We're trying to make the implementation disappear even further beneath the abstraction.

The developer shouldn't have to care whether the machine underneath is using strings, integers, tables, caches, or some future representation we haven't invented yet.

They should be able to describe their state.

The machine should do the rest.

And This Is Where "Don't Use Strings" Gets Interesting

The benchmark doesn't prove that strings are universally fast.

They aren't.

It proves something much more useful:

A carefully engineered string-backed FSM does not automatically become a performance problem.

Our current implementation can update fifty active process groups in approximately fifteen microseconds.

That's fast enough that the abstraction itself occupies less than one tenth of one percent of a 60 FPS frame in this benchmark.

And that's before the integer-backed optimization is finished.

So perhaps the better engineering rule isn't:

Don't use strings.

Maybe it's:

Don't assume. Measure.

If the strings are costing us something measurable, optimize them.

If they aren't costing us enough to matter, spend our engineering effort somewhere more valuable.

And if we can preserve the developer-facing benefits of strings while removing much of their runtime cost?

Then we've got the best of both worlds.

The Benchmark Becomes Part of the Architecture

This is ultimately why I like this benchmark more than a simple performance chart.

It isn't a victory lap.

It's a timestamp.

This is what the FSM costs today.

Pure strings.

305 ns for one group.

3.12 µs for ten.

15.74 µs for fifty.

360 bytes per group under the measured workload.

Linear scaling.

And now we're changing the internal representation.

So we'll measure it again.

The interesting question isn't whether the next implementation is faster.

It almost certainly should be.

The interesting question is:

How much faster?

And more importantly:

How much allocation can we eliminate without sacrificing the API and architectural flexibility that made the string-backed design worthwhile in the first place?

That's something worth measuring.

And when those numbers are available, we'll have something better than a benchmark claim.

We'll have an A/B measurement of the architecture itself.

The string-backed FSM already flies.

Now we're taking the strings out of the engine and finding out just how much faster it can fly.

You Don't Have to Take My Word for It

And here's the part where the benchmark stops being an article and becomes an invitation.

The FSM_API is available now.

You don't have to agree with me about strings.

You don't have to take the 305 ns number on faith.

You don't even have to like the architecture.

Install it.

Build something with it.

Put it under your own workload.

Benchmark it against whatever you're currently using.

Then tell me what happens.

That's one of the reasons I publish these experiments in the first place.

The goal isn't to tell developers what they should believe.

The goal is to build the tooling, expose the architecture, measure the results, and let other engineers kick the tires.

And if you find a workload where it falls over?

I want to know about that too.

Because that's useful data.

Resources & Code:

The FSM Package (Unity Asset Store):
https://assetstore.unity.com/packages/slug/332450

NuGet Package (Non-Unity Core):
https://www.nuget.org/packages/TheSingularityWorkshop.FSM_API

GitHub Repository:
https://github.com/TrentBest/FSM_API

Support Our Work:

Patreon Page:
https://www.patreon.com/c/TheSingularityWorkshop

Support Us (PayPal Donation):
https://www.paypal.com/donate/?hosted_button_id=3Z7263LCQMV9J

Please Like this post, Love the code, and Share your experience in the comments.

The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.

Sat, 05 Sep 2026 07:33:09 +0000

IBM's Benchpress pins the seeds that build its circuits and passes no seed to the code that compiles them. On one circuit, over 400 seeds per version, the real change between two Qiskit releases is +5.37%. The suite's own three-run protocol reports it as a ≥10% regression 24.4% of the time. Twenty runs per version — about forty hours of compute — still leaves 3.7%.

A benchmark suite is an instrument. You point it at two versions of a compiler and it tells you whether the new one got worse. Somebody then merges or reverts a pull request on the strength of that answer.

So the question I care about is not "is this compiler slower". It is how often does the instrument give an answer that disagrees with its own long-run behaviour. I call that finite-sample decision risk: the probability that a verdict computed from k runs per version disagrees with the verdict implied by the mean of the same measurement.

Where the randomness comes from

Qiskit's transpiler is stochastic. Its routing pass uses randomness, and generate_preset_pass_manager takes a seed_transpiler argument to pin it.

At the pinned revision I measured, Benchpress's Qiskit gym does not pass it. Circuit construction seeds are fixed — seed=12345, fourteen occurrences. The compilation seed is not. So every gate count the suite reports is one draw from a distribution nobody measured.

One detail matters for honesty here, and I got it wrong in an earlier draft: this is not universal across the suite. The BQSKit gym in the same repository does seed its compiler, with seed=0. seed_transpiler is a Qiskit-specific API the other gyms cannot call. The claim is about the Qiskit gym, and only about it.

What one circuit does

bv_n140 is a Bernstein–Vazirani circuit that Qiskit issue #14402 names directly. Mapped to a heavy-hex lattice, going from Qiskit 1.4.3 to 2.0.0, measured at 400 seeds per version across 21 operating-system processes:

quantity	value	95% interval
long-run mean change	+5.37%	+4.27% to +6.50%
called a ≥10% regression, 3 runs/version	24.4%	19.5% to 31.1%

The true change is nowhere near the threshold. The verdict crosses it a quarter of the time anyway.

I did not trust that number when I first got it, because all 200 of the original seeds came from one contiguous block inside one process — so any per-process state, hash randomisation, allocator layout, a cached RNG, was held constant. I drew a disjoint set of 200 seeds spanning 5.9 million to 1.08 billion, ran them across ten fresh processes with differing PYTHONHASHSEED, and got 22.6% [16.4, 30.6] against 26.3% [18.5, 34.2] from the contiguous set. Each point falls inside the other's interval.

Running it more does not fix it

The obvious response is "then run it more times". I measured exactly how much that buys, pooling all 400 seeds:

runs per version	1	3	5	8	10	20
false-positive rate	34.58%	24.38%	18.56%	12.94%	10.36%	3.74%

Twenty runs per version is on the order of forty hours of compute, at the issue's own stated "about two hours" per suite run. It still leaves 3.74%. One seed_transpiler argument removes the sampling variance at a single run.

Taking the minimum of three runs instead of the mean — a common instinct — makes it worse, not better. All eight circuits I tested still err, and every rate roughly doubles: bv_n280 goes from 17.3% to 28.4%.

The number in the issue

Issue #14402 reports +46.1% for that circuit. A single three-run comparison of the same circuit, under the same protocol, returns anywhere from −10.5% to +100.0%, with a 95% range of [+9.3%, +57.2%].

+46.1% sits at the 88th percentile of that distribution. The same range extends below the +10% threshold. The reported figure is not wrong — it is one draw, reported as though it were a measurement.

The part I pre-registered

One circuit is an anecdote. So before I looked at any result, I wrote down the selection rule, the analysis, the endpoint and the labels, and committed them. Then I collected the data. The commit ordering is checkable:

a36a34a  2026-09-03 23:51:08  pre-registration
fbc573d  2026-09-03 23:52:38  analysis code
30224a0  2026-09-04 09:13:07  raw data

The rule selected every qasmbench-large circuit whose twelve-seed heavy-hex runtime was under ten seconds — 39 circuits, 200 seeds per version per arm.

The pre-registered endpoint: 12 of 26 eligible circuits have a decision-error rate whose interval excludes zero. That is 46.2%, Wilson 95% [28.8%, 64.5%]. Seven of 26 sit at 5% or worse, four at 10% or worse.

What I withdrew

That 46.2% is the number I trust least in this work, and I want to say why rather than let someone find it.

The 26 circuits are not 26 independent observations. They come from eleven algorithm families, and within a family the results are nearly all-hit or all-miss. A cluster bootstrap over families gives [17.4%, 81.0%] — against the Wilson interval's [28.8%, 64.5%]. The effective sample size is far below 26, and no interval I can compute is informative about a population of circuits.

So the suite-level rate is withdrawn as a headline. The 12/26 stands as a descriptive count of these 26 circuits and nothing more.

Three other claims went the same way, and all four withdrawals are still in the repository with the evidence that defeated them. One of them was mine: I had reported a correlation of ρ = +0.876 between "compilation is stochastic" and risk, presented as a discovered mechanism. Circuits whose two arms are constant have zero risk by arithmetic, so a correlation computed over a sample containing them is measuring a definition, not a finding. Simulated data containing no compiler at all reproduces the statistic.

What went wrong in my own instrument

The paper's whole argument is that measurement apparatus should be checked rather than trusted, so it would be poor form not to turn that on myself.

After three rounds of adversarial review and a sixteen-phase red team — all of which attacked the claims — I had an independent audit read the ~4,600 lines of Python that nobody had ever read. It found six defects. None changed a published number. The worst one was this:

My numeric inventory recomputed every reported figure and compared it to a recorded value. But the recorded value was itself a function call evaluated when the registry was built, and the recomputation called the same function on the same file. It compared f() with f(). It could not fail. I proved it by falsifying every value in a summary file — tripling them and adding 40 percentage points — and the checker still reported "41/41 reproduce".

The values are frozen literals now, twenty-nine of forty-one re-derive from raw measurements rather than from a summary, and re-running that same falsification produces twelve failures. A checker that cannot fail is worse than no checker, because it buys confidence it has not earned.

What this does and does not say

It does not say Qiskit's transpiler got worse, or that Benchpress is a bad suite. It measures one property of one benchmarking protocol: that an unseeded stochastic compiler, sampled three times, produces regression verdicts that disagree with its own long-run behaviour at rates that are not small.

Limits, stated plainly: one SDK, one version pair, one machine. The +10% threshold is mine — Benchpress defines none and the issue states no formal cut. The estimate of the true change is a plug-in from 200 seeds, not an external criterion. And seed_transpiler is not a universal remedy: it removes false positives on the circuits I tested, but was worse on three of the four circuits showing false negatives.

Everything is in the record

The paper, the complete analysis code, and all 41,790 raw per-seed measurements are archived with a DOI: 10.5281/zenodo.22310060. verify.py re-runs the toolchain pin check, the test suite, the numeric inventory, the replication artifact and a proof of one withdrawn claim, in under a minute.

The full write-up is on my site: The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.

If you maintain a benchmark that drives accept/reject decisions on a stochastic system, the cheap version of this check is: run the same comparison twenty times without changing anything, and look at the spread of verdicts rather than the spread of values. If the verdict moves, the number of runs is part of your instrument, and it belongs in the write-up.

Fewer LLM turns, more commands: a practical way to verify research-heavy agent work

Tue, 04 Aug 2026 21:13:10 +0000

What if an agent spent less time going back to the model, but did much more useful work inside each turn?

That is the idea I have been testing: let an LLM turn make a decision, then let one batch of commands search, inspect, compare, and verify the result. The goal is not “fewer turns” by itself. The goal is fewer expensive decision loops and more observable work between them.

I used a deliberately messy task: build a 15-slide guide to 10 East Asian squid recipes, with illustrations, recipe sources, and cooking videos. This kind of job is easy to make look finished while leaving behind broken links, duplicated videos, missing assets, or a layout that only works on one screen.

Across four runs, the two CLI-style runs averaged 13.5 LLM turns and 10.5 tool calls. The two direct-execution runs averaged 10 LLM turns and 82 tool calls. So, in this small matrix, the direct approach used 25.9% fewer LLM turns while issuing about 7.8× more commands.

The interesting part was not the headline number. In one early turn, the agent launched roughly 20 discovery commands for recipes and matching videos. Later checks confirmed 10 sources, 10 unique video destinations, 15 local assets with none missing, navigation behavior, and rendering at 1440, 768, and 390 pixels. One bad recipe destination was corrected, and the remaining video IDs were validated.

This pattern is useful anywhere the work is “collect, produce, then prove it is usable”: research briefs, slide decks, market maps, buyer guides, travel plans, content migrations, link audits, and QA-heavy reports.

The mental model is simple:

Use the LLM turn to decide what needs to happen.
Fan that decision out into many independent commands.
Gather structured results.
Use the next turn only when judgment is actually needed.

This is only a four-run case study, not proof that one setup always wins. Runtime was mixed, and token accounting came from different harness sources. But it does show why “turn count” and “command count” should not be treated as the same thing.

The full interactive task and the raw run artifacts are public:

Disclosure: I maintain Tura and this benchmark.

I built a repeatable way to check whether token-saving tools actually work

Tue, 28 Jul 2026 17:24:03 +0000

Hi, I’m Yu. Earlier this month I released Tura and wrote about why agent harnesses need long-horizon benchmarks, not just one-off demos.

Over the past two weeks, I’ve seen more people publish tests of token-saving tools such as RTK and Ponytail. That’s great, but the results are hard to compare when everyone uses different tasks, logs, and output formats.

So I open-sourced the benchmark framework I use:

https://github.com/Tura-AI/benchmark/tree/main

It can run the benchmark workflow, collect logs and artifacts into one result schema, and let CI index any result added to the repository. The website then turns those results into comparable graphs and detailed run pages.

If you have a tool, harness, or test case you want included, tell me. You can also run the benchmark locally, reproduce the evaluation, and submit the result folder as a pull request. CI handles the indexing automatically.

I’d especially like feedback on what should be tested next and where the framework may introduce bias.

Disclosure: I maintain Tura and this benchmark framework.

The Divergence Problem: Why Your Proxy Ages Faster Than You Think

Mon, 29 Jun 2026 02:13:16 +0000

You know that moment when your test suite goes green but the feature is clearly broken? Multiply that by an entire field, stretch it over decades, and you've got what dendroclimatologists call the divergence problem.

The benchmark that quietly expired

HumanEval — 164 hand-written Python problems from 2021 — became the benchmark for LLM coding ability. For three years, pass@1 scores tracked real capability improvements. Then frontier models started hitting 90%+ and the benchmark was quietly dropped from comparisons.

How much of that 90% was real?

Bradbury and More (2024) built HumanEval-T — same problems, different enough to prevent memorization. Every model dropped 5 to 14 percentage points. Qwen-2.5-Coder now explicitly decontaminates against HumanEval using 10-gram collision detection. A major lab admitting in their training pipeline that the benchmark was compromised.

The proxy expired. Nobody printed the warranty.

This is actually a thousand-year-old problem

For a thousand years, tree rings at far-northern latitudes tracked temperature faithfully. Wider rings meant warmer summers. Dendroclimatologists built entire climate reconstructions on this relationship — centuries of temperature data, cited by the IPCC.

Then around 1960, the trees stopped matching the thermometers. Ring widths diverged downward while temperatures went up. Nobody noticed for 35 years.

The field has six plausible explanations. None definitively rules out the others. D'Arrigo's 2008 review laid them all out: drought stress, global dimming, UV-B damage, snowmelt timing shifts, survivorship bias in sample selection, and statistical artifacts. The forensics are permanently messy — because when a proxy fails, multiple causal pathways break at once.

The pattern shows up in AI evaluation too

GPT-4 passed theory-of-mind tests (the Sally-Anne false-belief task) at 75%. Looked like emergent reasoning.

Then Ullman (2023) changed one thing: made the container transparent instead of opaque. GPT-3.5 dropped to 6%. SCALPEL (2024) found GPT-4 at 20.35% on the variant.

Here's the kicker: add one explicit line saying the character "recognizes" the contents, and GPT-4 jumped back to 89.64%.

The capability existed. The proxy just lost the ability to measure it.

The five-phase warranty

Every proxy failure follows the same arc:

Calibration — proxy tracks reality
Reliance — you build systems on it
Silent divergence — proxy quietly decouples
Hindsight discovery — a second instrument reveals the gap
Overdetermined forensics — multiple explanations, none clean

What you can actually do

Rotate your benchmarks. LiveCodeBench (ICLR 2025) uses rolling monthly updates so models can't train on the test set. If your internal benchmark hasn't changed in two years, treat that as a yellow flag.

Treat stability as suspicious. A metric that hasn't moved could be saturated, gamed, or decoupled. Stable ≠ reliable.

Don't scale your way out. Pan et al. (2022) showed larger models get higher proxy rewards but lower true rewards. More optimization against a broken proxy makes the problem worse.

Every metric you rely on — test coverage, sprint velocity, interview rubrics, code review approval rates — was calibrated under conditions that will not hold forever. The trees kept faith for a thousand years. When they stopped, nobody heard it happen.

Sources: D'Arrigo et al. (2008); Bradbury & More (2024), arXiv:2412.01526; Pi et al. (2024), arXiv:2406.14737; Kosinski (2024), PNAS; Ullman (2023); Pan et al. (2022); LiveCodeBench (ICLR 2025); "Dead rats, dopamine, performance metrics, and peacock tails," BBS (2023).

The essay's prescription: build a second instrument. Chain of Consciousness applies this to agent systems — every action anchored to a verifiable external record, so your audit trail doesn't depend on the agent's own self-report. pip install chain-of-consciousness

The 1,000-Message Test: A Benchmark for AI Memory That Most Apps Fail

Thu, 30 Apr 2026 20:28:06 +0000

Most apps that claim "memory" don't have it.

I spent 200 days testing AI companion apps. 15 platforms, every subscription paid out of pocket. What I found, consistently, is that "memory" in marketing copy usually means one of three things: a context window, a summarizer, or nothing at all.

None of those are memory.

Real memory means the app can recall specific things you said days, weeks, or months ago, reliably, across sessions. That's a much higher bar than most apps clear. So I built a benchmark to test it.

What follows is the full methodology. It works for any AI product that claims persistent memory: companions, chatbots, agents, therapy apps, coding assistants. If it claims to remember you, this test tells you whether it actually does.

Why this test exists

The problem with evaluating AI memory is that most apps feel like they remember, as long as you stay within one conversation. Open a session, chat for 30 minutes, reference something you said 20 messages ago. Works fine. Looks like memory.

It's not. It's a context window.

The real test is what happens when:

The conversation gets longer than the window
You close the app and come back later
You reference something specific from weeks ago

This is where apps diverge. Some hold up. Most don't.

The methodology

The test has four phases. Total time to run: about 3-4 hours of real usage, spread across multiple days.

Phase 1: The planting

Over roughly 1,000 messages, you plant specific, memorable facts at known checkpoints. The facts need two properties:

Specific enough to verify later. Not "I like coffee." Something like "My cat's name is Mortimer and she's a tabby with one white paw."
Varied in type. A name, a number, a preference, a relationship, an event, an opinion.

I use four checkpoints in the 1,000-message conversation:

Checkpoint	Message #	Fact type
Early	50	A specific name (pet, friend, coworker)
Medium	200	A numerical detail (age, date, price)
Late	500	A preference paired with a reason
Very late	900	A story or event with multiple details

The varied types matter. Apps that summarize conversations often retain categories of information (names, preferences) but lose specifics within those categories.

Phase 2: The break

Between planting and retrieval, you need to force the app out of its comfort zone:

Exceed the context window. Most apps advertise 8K, 16K, or 32K token windows. Push past whatever they claim.
Close the session. Fully exit the app. Don't just background it.
Wait at least 24 hours. This catches apps that hold state in RAM or short-lived caches.
Open a new session. Fresh conversation, no context carried over.

This sequence is deliberate. Each step exposes a different class of failure.

Phase 3: The retrieval

In the new session, you query each planted fact. The key is how you ask.

Don't lead. "Remember Mortimer?" is leading. The app will often confabulate a plausible response even if it doesn't actually remember.

Do ask open-ended questions. "Tell me about my pet" or "What was that thing I mentioned about my cat?"

Ask both specific and general. Test whether the app can surface the right memory without being handed the answer.

Vary the phrasing. Ask about the same fact two or three different ways across a few minutes. Apps that retrieve purely on keyword match will fail on paraphrase.

Phase 4: The scoring

For each planted fact, the response falls into one of four buckets:

Score	What it means
Pass	App recalls the specific fact accurately without leading
Partial	App recalls the category but misses the specific (remembers "cat" but not "Mortimer")
Hallucination	App confidently invents details you never shared
Fail	App says it doesn't know, or the response ignores the fact entirely

Hallucinations are the most dangerous failure mode, worse than an honest "I don't know." An app that makes things up is creating false memories of a relationship that doesn't exist.

What the results actually show

When you run this test across a catalog of apps, a pattern emerges fast. Out of the 15 platforms I tested, only a small minority passed the full benchmark. Most failed in predictable ways, and the way they failed said more about their architecture than any marketing copy could.

Here are the buckets I consistently found:

Context window pretenders. Apps that advertise 16K or 32K context windows and then hard-truncate when the window fills. These fail at message 20-50, well before any of the planted facts even enter long-term consideration.

Session-only memory. Apps that feel great inside a single session but reset to zero on reopen. These pass Phase 1 easily, then fail every single retrieval in Phase 3.

Preference summarizers. Apps that remember what kind of thing you like but not the specific thing. These produce the most Partial scores: "You mentioned having a pet" instead of "Your cat Mortimer."

Confident hallucinators. Apps that confidently invent details rather than admit they don't know. These are the scariest failure mode, because a user would have to already know the right answer to catch the mistake.

Actual memory. Apps that pass cleanly across all four checkpoints with accurate recall. In my testing, this bucket was the smallest. These apps tend to have architecture built specifically for this: a retrieval layer, embeddings on past messages, a distinct fact store separate from the conversation buffer. I wrote about the three most common failure modes in more detail here.

Why apps fail this test (and how to pass it)

Same failure modes, over and over:

Failure mode: context-only storage. The app treats the prompt as the memory. When the prompt fills up, the oldest messages get discarded. There is no second layer.

Failure mode: session-scoped state. The app persists within a session but not across sessions. Often this is a database design choice: the conversation is stored per-session with no cross-session retrieval.

Failure mode: summary without retrieval. The app summarizes old conversations into a rolling memory document but discards the raw text. If the summary loses a detail, it's gone forever.

Failure mode: retrieval without salience. The app stores everything and retrieves based on keyword or embedding similarity, but has no sense of importance. A passing mention of "Mortimer" six weeks ago gets outranked by a recent mention of "cat food brands."

Apps that pass the test typically have all of the following:

Persistent storage of raw messages (never deleted)
Compressed knowledge layer (facts + summary, updated async)
Semantic retrieval (embeddings + similarity search)
Salience scoring (emotional weight, personal facts, recency decay)

If you're building, I wrote up the full architecture I'd build for this in a separate post.

How to use this benchmark

If you're a user trying to decide whether to pay for an app, run the test before you subscribe. Most apps have a free tier that's plenty for the 1,000-message run.

If you're a developer building a memory-dependent product, run the test on your own system before shipping. If you can't pass it, don't claim "long-term memory" in your marketing. Users notice, usually after they've paid for three months.

If you're a reviewer or journalist covering AI products, this benchmark gives you something to cite. "The app failed standard memory benchmarks" is a much stronger claim than "it seemed to forget things."

The quiet thing about memory

Most product teams discover the memory problem the hard way: users churn, and exit interviews say some version of "it didn't feel like it really knew me." The team then scrambles to retrofit memory onto a system that wasn't designed for it, which is much harder than designing for it from day one.

The 1,000-message test is a forcing function. Run it early, run it honestly, and you'll know whether your product has what it claims.

Most don't.

I test AI companion apps and write about what I find at AI Companion Picker. If you're building in this space, I'm always down to compare notes.

We Measured the N+1 Query Problem. The Numbers Are Worse Than You Think.

Thu, 12 Feb 2026 00:58:11 +0000

I kept seeing the same advice in every ORM tutorial: "Watch out for N+1 queries." Every code review, every blog post, every conference talk. Always the same warning, never the actual numbers.

So I decided to measure it.

I built a benchmark suite, seeded a PostgreSQL database, wrote the "bad" code that everyone warns about, wrote the "good" code that everyone recommends, and ran them side by side. Four different scenarios, each one timed and query-counted down to the individual SQL statement.

The short version? On a tiny dataset — just 100 users — the worst pattern was 22x slower and fired 200 unnecessary queries. And that's the conservative estimate, because I ran everything on localhost where network latency is basically zero.

Here's what I found.

What's the N+1 problem, actually?

You probably already know this, but it's worth being precise about it because the devil is in the details.

Say you've got 100 users and you want each user's posts. The naive approach does this:

One query to grab all users
Then, inside a loop, one query per user to get their posts

That's 101 database round-trips. The "1" to get the users, plus "N" (100) to get each user's posts. Hence, N+1.

The thing is, each individual query is fast — sub-millisecond in most cases. But every single one of those queries carries overhead. The app has to send the request to PostgreSQL, PostgreSQL has to parse it, plan it, execute it, serialise the results, and send them back. Do that 101 times and the overhead stacks up fast.

The fix is usually straightforward: tell your ORM to grab the related data upfront. In Prisma, that's the include keyword. Instead of 101 queries, you get 2. Same result, fraction of the time.

But how much of a fraction? That's what I wanted to pin down.

How I set this up

I wanted the benchmarks to be fair and reproducible, so here's what I did.

The stack: Prisma 6.3.1, PostgreSQL 17, TypeScript, Node.js 22 — all running locally on the same machine. Running the database locally means there's no network hop between the app and the database. That makes the numbers more conservative than what you'd see in production, where the database is usually on a separate server (or across the internet, if you're using a managed service).

The data: 100 users, each with 3 posts, 2 comments per post, and 2 orders. All generated with Faker so the string content is realistic — not just "test" repeated a thousand times. Serialisation costs are real costs, and I didn't want to cheat there.

The measurement: Every query that Prisma sends to PostgreSQL gets intercepted and counted. Not estimated, not sampled — counted. Each benchmark runs 5 times (after a warmup run) using performance.now(), and I report the mean, median, P95, and P99.

The test cases:

#	What it tests	Relationship
TC1	Users with their posts	One-to-many
TC2	Users → posts → comments	Three-level nesting
TC3	Orders with their user	Many-to-one (reverse)
TC4	Active orders, conditionally load user	Conditional access

Each test case has a "bad" version (the N+1 pattern) and a "good" version (the recommended fix). Both return the exact same data.

Source code for everything: github.com/liangk/empirical-study

TC1: Users with posts — the textbook case

This is the one everyone uses as an example. Grab all users, loop through them, fetch posts for each one.

The bad version fires 101 queries. The good version uses include: { posts: true } and fires 2.

Metric	Bad (N+1)	Good (Eager)	Improvement
Queries	101	2	98.0% fewer
Avg time	39.31 ms	4.28 ms	9.2x faster
Median	38.96 ms	4.21 ms	9.3x faster
P95	41.49 ms	4.53 ms	9.2x faster

Here's what caught my attention: the variance in the good version is tiny. The standard deviation is under 0.25ms. That matters if you're building an API where you care about tail latency — and you should care about tail latency.
The bad version stays surprisingly steady too, with a relatively small variance. It feels fine at this scale. 39ms isn't going to trigger any alarms. But it's doing 50x more work than it needs to, and the cost grows linearly with your user count.

TC2: Nested relationships — where it gets ugly

This one goes deeper. Users → posts → comments. Three levels of entities.

The bad code has two nested loops, each firing queries. With 100 users and 3 posts each, that's 1 query for users + 100 for posts + 300 for comments = 401 queries.

The good version? Nested include directives. 3 queries total. One per entity level, no matter how many records.

Metric	Bad (N+1)	Good (Eager)	Improvement
Queries	401	3	99.3% fewer
Avg time	153.59 ms	10.50 ms	14.6x faster
Median	143.21 ms	10.07 ms	14.2x faster
P95	185.53 ms	11.82 ms	15.7x faster

401 queries versus 3. That's not a percentage I need to calculate to know it's bad.

But the timing tells a more interesting story. The bad version's P95 is 185ms — that's 29% higher than its median. Some of those runs are hitting extra overhead, probably connection pool pressure or plan cache misses from hammering PostgreSQL with 400 queries in rapid succession. The good version's P95 is only 17% above its median. Fewer queries means less contention, which means more predictable performance.

And this is the part that should worry you: 153ms with 100 users. The N+1 cost here isn't linear — it's multiplicative. Each level of nesting multiplies the query count. The general formula looks something like:

Queries = 1 + N + (N × posts_per_user) + (N × posts_per_user × comments_per_post) + ...

With eager loading, it's just the number of entity levels + 1. Always. Doesn't matter if you've got 100 users or 100,000.

If you're thinking "well, my app doesn't have three levels of nesting" — check again. User → projects → tasks. Store → products → reviews. Organization → teams → members. It's more common than you'd expect.

TC3: Orders → User — the biggest speedup

This one flips the relationship. Instead of fetching children for each parent, you're fetching the parent for each child. Classic scenario: you've got an orders list and you want to show the customer name next to each order.

The bad version grabs all orders (200 of them — 2 per user), then calls findUnique for each order's user. That's 201 queries.

The good version adds include: { user: true }. Two queries.

Metric	Bad (N+1)	Good (Eager)	Improvement
Queries	201	2	99.0% fewer
Avg time	82.54 ms	3.72 ms	22.2x faster
Median	86.10 ms	3.66 ms	23.5x faster
P95	88.71 ms	4.02 ms	22.1x faster

22.2x. That's the largest speedup across all four test cases.

Why so much bigger than TC1? Two reasons. First, there are more records involved — 200 orders versus 100 users. More queries means more wasted overhead. Second, many of those findUnique calls are fetching the same user — each user has 2 orders, so each user gets fetched twice. Prisma doesn't deduplicate those calls. It doesn't have an identity map or a request-level cache. So you're not just making too many queries — you're making redundant ones too.

The eager loading version sidesteps all of that. Prisma resolves the relationships from the batch result set in memory.

TC4: Conditional loading — the tricky one

Here's where things get interesting from a design perspective. You've got active orders, but you only need user data for orders where requiresUserData is true. Not all of them.

You can't just slap include: { user: true } on this one — that would over-fetch user data for orders that don't need it. So the fix is different: collect the user IDs you actually need, fetch them all in one batch, build a lookup map, and assign the results.

The bad version loops through active orders and calls findUnique for each one that needs user data. About 70% of them do, which gave us 56 queries.

The good version collects the needed IDs into a Set, does one findMany with WHERE id IN (...), builds a Map, and does the assignment in a second pass. Two queries total.

Metric	Bad (N+1)	Good (Batch)	Improvement
Queries	56	2	96.4% fewer
Avg time	22.75 ms	2.16 ms	10.5x faster
Median	23.05 ms	2.11 ms	10.9x faster
P95	23.49 ms	2.57 ms	9.1x faster

Even though the conditional logic cuts the query count from a potential 200+ down to 56, you're still paying a 10.5x penalty. Partial N+1 is still N+1.

This batch-query pattern is worth keeping in your back pocket. It works everywhere — not just in Prisma, not just in JavaScript. Collect the IDs, batch fetch, build a map, assign. You trade a bit of memory for the Map and an extra loop, but you eliminate dozens (or hundreds or thousands) of database round-trips. That trade-off pays for itself every time.

The full picture

Here's everything in one table:

Test Case	Bad Queries	Good Queries	Reduction	Bad Time	Good Time	Speedup
TC1: One-to-many	101	2	98.0%	39.31 ms	4.28 ms	9.2x
TC2: Nested (3-level)	401	3	99.3%	153.59 ms	10.50 ms	14.6x
TC3: Many-to-one	201	2	99.0%	82.54 ms	3.72 ms	22.2x
TC4: Conditional	56	2	96.4%	22.75 ms	2.16 ms	10.5x

A few things jump out.

The optimised versions never exceed 3 queries. Doesn't matter which pattern, doesn't matter how many records. That's the whole point — you go from O(N) queries to O(1). And the benefit scales with your data. At 1,000 users, the bad TC2 version would fire over 4,000 queries. The good version would still fire 3.

The speedup correlates with query count. TC3 had the most queries (201) and the biggest speedup (22.2x). TC4 had the fewest (56) and the smallest speedup (10.5x). Per-query overhead is roughly constant, so more queries = more waste = more gain from fixing it.

These numbers are the floor, not the ceiling. I ran everything on localhost. In production, add network latency between app and database — maybe 1-5ms per round-trip. Now multiply that by 401 queries. That's where a 150ms endpoint becomes a 2-second one.

What happens at 1,000 users?

I ran the same benchmarks on a 10x larger dataset. 1,000 users, 3,000 posts, 6,000 comments, 2,000 orders.

Here's the medium dataset alongside the small one:

Test Case	100 Users		1,000 Users		Change
TC1: One-to-many	39ms → 4ms	9.2x	364ms → 35ms	10.4x	Slightly worse
TC2: Nested	154ms → 11ms	14.6x	1,271ms → 88ms	14.5x	About the same
TC3: Many-to-one	83ms → 4ms	22.2x	739ms → 28ms	26.5x	Noticeably worse
TC4: Conditional	23ms → 2ms	10.5x	169ms → 8ms	21.9x	Way worse

A few things I didn't expect.

TC2 blew past 1 second. 1,271ms for a single data fetch. With 4,001 queries. On localhost. In production with a network hop, you're looking at multiple seconds — from one API call. That's not a performance issue. That's a broken endpoint.

TC4's speedup doubled. It went from 10.5x to 21.9x. That surprised me. But the reason makes sense once you look at the numbers: the batch query version barely grew (2ms → 8ms, only 3.6x) while the loop-with-findUnique version grew 7.4x. The batch approach has near-constant overhead regardless of how many IDs you're fetching — PostgreSQL is really good at WHERE id IN (...) queries. The loop approach pays full overhead for every single query.

TC3 got worse too. 22.2x → 26.5x. Same story — the good version scales better than the bad version because batch fetching is fundamentally more efficient than doing it one at a time.

The bad code scales roughly linearly. 10x more data, roughly 8-9x more time across the board. Which makes sense — 10x more users means ~10x more queries in the loop, and per-query overhead is approximately constant.

The good code sometimes scales better than linearly. TC4's good version grew only 3.6x for 10x more data. The batch query with deduplication is doing less redundant work at scale.

The bottom line: the N+1 penalty doesn't shrink as you scale. It gets worse. Every test case either maintained or increased its speedup factor at 1,000 users compared to 100. If you've got N+1 patterns in your code and you're thinking "I'll fix it when we scale" — the data says you're already paying for it.

How to spot this in your own code

The pattern is always the same: a database call inside a loop. If you see await prisma.something.findMany() or findUnique() inside a for, map, forEach, or any kind of iteration — that's your red flag.

Three ways to catch it:

Turn on query logging. Set log: [{ emit: "stdout", level: "query" }] in your Prisma client config. If you see the same SELECT statement repeated 50 times in a row, differing only in the WHERE clause value, you've got an N+1.

Use static analysis. Tools like Code Evolution Lab walk your AST (Abstract Syntax Tree) to find database calls nested inside loops. They can catch these before your code even runs — during CI, code review, or in your IDE. They'll also suggest the right fix for your specific ORM.

Watch your APM dashboards. If an endpoint's query count scales linearly with the number of records it returns, something is wrong. A healthy endpoint has a roughly constant query count.

Two fixes that cover almost everything

Across all four test cases, two patterns handled every situation:

Eager loading for cases where you know you'll need the related data. In Prisma, that's include. It works for one-to-many, many-to-one, and nested relationships. You can stack include directives as deep as your schema goes, and Prisma always resolves it in a constant number of queries (one per entity level).

Batch pre-fetching for cases where you need related data conditionally. Collect the IDs, do one findMany with WHERE IN, build a Map for O(1) lookup, and do the assignment in memory. It's a few extra lines of code, but it turns 56 queries into 2.

Both approaches do the same fundamental thing: they take the number of database round-trips from "proportional to your data" down to "constant, no matter how much data." That's the whole game.

What I'd push back on

There's a common belief that N+1 queries are a "scaling problem" — something you can ignore early and fix when your dataset gets big enough to matter.

The data doesn't support that.

At 100 users, we're already seeing 9x-22x slowdowns. At 1,000 users, it's 10x-27x. That's not a rounding error. If your API has a 200ms budget per endpoint (pretty standard), the bad TC2 version is already eating 75% of that with just 100 records and zero network latency. In production, you're probably blown past it.

The other thing I'd push back on: the idea that findUnique is somehow safe because it's a primary key lookup. It's fast per query, sure. But when you call it 200 times in a loop (TC3), the fixed overhead of 200 round-trips completely dominates. A fast query executed 200 times is slower than a slightly heavier query executed once.

What about the honesty section?

A few things that could affect how you read these numbers.

Localhost flatters the good version. With no network latency, the per-query overhead is as small as it can be. In production, the penalty for each extra query would be higher, which means the speedup numbers would be larger than what I measured. So think of these as conservative estimates.

I only tested 100 users. The scaling behaviour at 1K, 10K, and 100K records is something I want to measure next. I expect the gap to widen, not narrow, because connection pool contention and buffer pressure get worse with more queries.

This is Prisma-specific. Sequelize, TypeORM, and other ORMs have different internals. The per-query overhead might be higher or lower. The fix patterns are similar though — eager loading and batching work everywhere.

Warm caches. I did a warmup run before measuring, which means PostgreSQL's query plan cache and shared buffers are warm. Both versions benefit equally from this, so it shouldn't bias the comparison. But cold-start numbers might look different.

Try it yourself

Everything's open source:

git clone https://github.com/liangk/empirical-study.git
cd empirical-study
npm install
npx prisma db push
npm run seed -- small
npm run bench:all

Swap small for medium (1,000 users), large (10,000), or xlarge (100,000) to see how the numbers scale.

What's coming next

This is the first piece in a series I'm working on. The questions I still want to answer:

How do the numbers scale beyond 1K? The 1,000-user results are in (see above) and the penalties got worse, not better. Next up: 10K and 100K, where I expect connection pool contention to make things even uglier.
Can automated tools fix this? I'm going to feed the bad code into Code Evolution Lab and see if the auto-generated solutions perform as well as the hand-written ones.
Is Prisma better or worse than other ORMs? Same benchmarks, Sequelize and TypeORM. The per-query overhead profile might be very different.
What about concurrent load? All my benchmarks are single-threaded. Under real traffic, connection pool contention makes the N+1 penalty even worse.

The bottom line: the N+1 problem isn't a theoretical concern you can defer. It's a measurable, order-of-magnitude performance hit that shows up at 100 records and gets worse at 1,000. The nested pattern crossed 1.2 seconds. The conditional pattern's penalty doubled. And fixing it is rarely more than a one-line change.

The benchmark suite and raw data are on GitHub

This article was originally published on Stack Insight. For the best reading experience and future updates, visit the original post.