Coder Legion - Recent posts tagged testing

Your Tests Are Green and Prove Nothing: 4 Signs You're Testing the Mock

Sat, 31 Oct 2026 03:38:56 +0000

Coverage says 94%. Every test is green. You deploy on Friday afternoon, and by Monday a customer has found a bug in the exact function you "tested".

When you open the test, the reason is obvious in hindsight. The test never ran your code. It ran a fake version of everything around it and then asserted that the fake behaved like a fake.

Here are four ways this happens, and a check you can do in a minute to see how many of your own tests are guilty.

1. You Mocked the Thing You Are Testing

// pricing.ts
export function applyDiscount(total: number, code: string) {
  return code === 'SAVE10' ? total * 0.9 : total;
}

// checkout.test.ts
import * as pricing from './pricing';

vi.spyOn(pricing, 'applyDiscount').mockReturnValue(90);

test('applies discount at checkout', () => {
  expect(checkout(100, 'SAVE10')).toBe(90);
});

This passes if applyDiscount is correct. It also passes if applyDiscount returns NaN, because the test replaced it with a hard-coded 90. You are testing that checkout can return the number you handed it.

Fix: mock what is outside your code, such as network, clock and payment providers. Run your own logic for real.

2. You Assert the Calls, Not the Outcome

test('saves the user', async () => {
  await createUser({ email: '*Emails are not allowed*' });
  expect(db.insert).toHaveBeenCalledWith('users', { email: '*Emails are not allowed*' });
});

This test proves the function called insert with those arguments. It does not prove a user exists. If someone renames the table, adds a required field or changes the query, this test keeps passing against the mock while real users stop being saved.

It also fails for the wrong reason. Refactor the internals without changing behavior, and it turns red, so people learn that failing tests are noise.

Fix: assert what a user or caller would observe.

test('saves the user', async () => {
  await createUser({ email: '*Emails are not allowed*' });
  const user = await findUserByEmail('*Emails are not allowed*');
  expect(user?.email).toBe('*Emails are not allowed*');
});

3. The Fake Database Only Says Yes

A mocked database returns whatever you tell it to. A real one has rules:

A unique index rejects duplicate emails.
A required field rejects a missing value.
A foreign key rejects an ID that does not exist.
A query with a typo in a field name returns nothing.

None of these exist in a mock, so none of these bugs can appear in your tests. They show up in production, on the first real duplicate signup.

Fix: for anything that touches storage, test against a real database, either a throwaway local instance or an in-memory one with the same engine, reset between tests. It is slower than a mock, but it is the only thing that can fail the way production fails.

test('rejects a duplicate email', async () => {
  await createUser({ email: '*Emails are not allowed*' });
  await expect(createUser({ email: '*Emails are not allowed*' })).rejects.toThrow();
});

You cannot write that test against a mock that always resolves.

4. Snapshots Approved Without Reading

expect(renderInvoice(order)).toMatchSnapshot();

A snapshot records whatever the output is today and fails when it changes. It does not know whether today's output is correct. When it fails, the usual reaction is to press "update" and move on, so the test stores bugs as the expected answer.

Fix: use snapshots for small, stable things, and assert the fields that matter for anything involving money, permissions or dates:

const invoice = renderInvoice(order);
expect(invoice.total).toBe(118.0);
expect(invoice.lines).toHaveLength(2);

The 60-Second Sabotage Check

Here is how to find out whether a test protects you. Take a function you care about, break it on purpose, and run its tests.

export function applyDiscount(total: number, code: string) {
  return total; // sabotage: discount never applies
}

Then ask one question: did a test fail?

If yes, that test is doing its job.
If no, you have found a test that would have stayed green while the feature was broken in production.

Revert the change, and fix or delete the test that did not notice. Try this on three important functions and you will learn more about your suite than coverage ever told you. The formal version of this idea is called mutation testing, and tools exist that automate it, but doing it by hand on your most critical code is a good start.

What to Mock, and What Not To

Mock it	Run it for real
Network calls to third parties	Your own business logic
The clock and random numbers	Your database queries
Email and SMS sending	Validation and permission rules
Payment providers	Anything that can fail on real data

The rule is that you mock the edges, the places where your code talks to someone else, and keep everything in the middle real.

The Rule

A test that cannot fail when the code is wrong is not a test. Mock the world outside your app, assert outcomes instead of calls, and use sabotage to check whether your tests notice.

Quick audit: pick the three functions that would hurt most if they broke, such as pricing, permissions and sign-up. Break each one on purpose. Count how many tests turn red. If the answer is zero for any of them, that is your next ticket.

If you want a Next.js starting point to build and test on, you can browse my templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

What is the most misleading green test you have ever found? Share it in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

I Solved the Same Gmail Problem Two Ways Without Noticing

Thu, 29 Oct 2026 13:52:12 +0000

This is the third post from my n8n intake classifier, a practice project for a fictional automation agency that I built with an AI collaborator. The first was about the classification prompt. The second was about ambiguous, invalid and system_error. This one is about the wiring after classification: where logging goes, what the Gmail node does to your data, and a bug I only found by opening real inboxes. The workflow is in the GitHub repository.

A quick orientation: n8n is a workflow automation tool, and a workflow is built from nodes, each one a single step. The ones that matter here are Switch (sends an item down one of several paths), Set (reshapes the data) and Gmail (sends an email).

Where logging was supposed to go

After classification, a Switch node sends each submission down one of five paths, and each path sends emails. Every submission also gets logged to Google Sheets. For this phase I planned to work the way I had in the earlier ones: Sheets first, then emails, one branch at a time.

Before building anything, I caught a conflict. Logging is the same logic on all five paths, and only the data changes. Putting it first inside each branch meant five copies of one node, or converging after Sheets and then branching again to send category-specific emails. Neither made sense. So I flipped the order: build the emails first, since those differ per category, and let all five paths converge on one shared Sheets node at the end.

Gmail replaces your data

The first branch I built showed a second problem. When the Gmail node sends an email, the item it passes on is Gmail's own send confirmation, not the submission. Name, Email and description were gone by the time anything downstream needed them. The LLM Chain node had done the same thing to me earlier.

In the ambiguous/system_error branch, I added a Set node right after the internal notification to rebuild one clean object. The IF node and both client emails read from that, instead of each reaching back to whichever earlier node had the data. In simple_automation, I added a Set node after the client email to rebuild the object before it reached the shared Sheets node.

Noticing I'd solved it twice

While wiring the final Sheets node, I looked at the ambiguous/system_error branch and saw it fed Sheets from two points off the same rebuild node, one path through the client emails and one straight to Sheets. That wasn't a bug. The email path never reconnected to Sheets, so nothing was logged twice. But it showed me I had solved the Gmail problem two different ways. Everywhere else I rebuilt the data after the email. In this branch I branched before the email, because Sheets never needed anything the email step produced.

Moving logging upstream

That comparison pointed at something bigger. Every branch converged on one Sheets node, so there was no reason logging should depend on any email succeeding. A Gmail send can fail for reasons that have nothing to do with the submission.

I moved the Sheets connection to the two points right after classification: one for a successful classification and one for the failure handler. Both now feed the Switch and Sheets in parallel. Two side effects followed. has_partial_intent only had to be added in two places instead of four, and I could delete the end-of-branch Set nodes that existed only to feed the old Sheets wiring.

Before deleting any of them, I checked each one. Some still fed a client email's own data, and removing one of those would have broken a client-facing email without touching Sheets. I kept those.

The final build still contains both patterns. The ambiguous/system_error branch has its rebuild node, and the other client emails reach back to earlier nodes by name:

={{ $('Route: By Classification Category').item.json.Email }}

I haven't compared whether one pattern is more robust than the other. What changed is that logging no longer depends on either.

Checking real inboxes

After the rewiring, I retested every branch individually and checked real inboxes, not just node output. That caught a bug in the ai_workflow branch: the internal notification and the client email had their destinations swapped. The internal-only message went to the client, and the client's confirmation went to the team.

Both emails sent without errors, so nothing had failed that would make me suspect a problem. Only who received what showed it. I then confirmed the simple_automation branch had no swap, since I'd used the same inbox check there.

My notes don't record how the swap got there, or whether the rewiring caused it, so I won't guess. The current build sends each internal email to the team address and each client email to the submitter.

A test case I wrote wrong

While testing ambiguous, my first case was a deliberately vague description. It came back as invalid, and I took that as a bug. Reading the reasoning showed the classifier was right. Vague wording alone doesn't give any category a specific signal to rest on, which is a different situation from two categories each being concretely supported. I wrote a sharper case with explicit signals for both ai_workflow and custom_integration, and that one returned ambiguous.

What this doesn't show

Every test used descriptions I wrote. A forced classification failure does produce a row in the sheet, with the has_partial_intent cell empty, because the model never produced a value on that path. I did not test what happens if the Sheets step itself fails.

What I take from it is narrow: logging shouldn't wait on anything that can fail for unrelated reasons, and a branch that sends emails isn't tested until I've read the emails.

Cancel or send: one winner. An order flow on Prisma Postgres and Compute

Wed, 28 Oct 2026 21:30:34 +0000

Werbung / Disclosure: Prisma gave me a coupon for two months of a paid plan for free and said it will add free credits, so I could try Prisma Postgres and Compute for this project. All results below come from my own runs.

Summary

I built an order flow with a 10-minute grace period. Cancellation and dispatch must have exactly one winner in the database.
The winner is decided in Postgres, with a guarded update, a row lock and SKIP LOCKED. No clock and no timer decides it.
I broke the code on purpose, one line at a time, to check that my tests really fail. One of my race tests passed even without the lock.
On Compute, an idle app sleeps. My poller is a timer inside the server, so it only runs when someone calls the app. This changes the timing, not the correctness.

The problem

A customer orders a custom T-shirt. I promise a grace period of 10 minutes. During that time the customer can cancel. After that, the printer gets an email and the customer cannot cancel any more.

Two things can happen at almost the same moment. The customer presses cancel, and my poller wants to send. Exactly one of them must win. A cancelled order must never reach the printer, because an email cannot be recalled.

Design decisions

Claim first, then send

The poller claims an order by changing its status from PENDING to SENT before it sends the email. If the customer cancels first, the claim finds nothing. If the claim wins, the cancel is rejected with a 409.

Here, SENT means the order has been claimed for dispatch, not that the email has been delivered. If the server crashes after the claim, the printer may get no email. I set printerNotifiedAt only after Resend accepts it. A stuck-order query finds SENT orders without that timestamp, and a resend route lets me recover them.Sending the email before claiming the order could allow it to be sent even if the cancellation wins the database race.

A guarded transition

Every status change goes through one function. It checks that the transition is allowed for the actor. Then it writes with the old status in the where clause:

const { count } = await prisma.order.updateMany({
  where: { id, status: order.status },
  data: { status: to },
});
if (count === 0) throw new ConflictException('Order changed, please retry');

If someone changed the row between my read and my write, count is 0 and the caller gets a 409. This is the sixth time I used this read-then-write pattern across my projects.

A Postgres poller, not a queue

The hosted version has no Redis, so there is no BullMQ. The order and its deadline are written in one transaction. A timer loop claims due orders with one statement:

UPDATE "Order" SET status = 'SENT' -- plus updatedAt
WHERE id IN (
  SELECT id FROM "Order"
  WHERE status = 'PENDING' AND "sendAt" <= (now() AT TIME ZONE 'UTC')
  ORDER BY "sendAt" LIMIT 10
  FOR UPDATE SKIP LOCKED
)
RETURNING id

There is no enqueue gap, because the deadline is data. The costs are a polling delay and retry logic that I write myself. Today there is no automatic retry. The safety net is the stuck-order query and the resend route.

The server owns sendAt. A client-supplied value could skip the grace period, so the request schema rejects unknown keys.

A row lock on the design

An order points at a design. If the design changes after the order exists, the printer prints something the customer never ordered, and nobody sees an error. Creating an order, updating a design and deleting a design all take SELECT ... FOR UPDATE on the design row first. A design counts as locked when it has any order that is not cancelled.

Tests, and a test that proved nothing

I wrote integration tests against a real Postgres database. Then I broke the code on purpose, one line at a time, and checked that a test turned red.

I removed	What went red
the status in the `updateMany` guard	the cancel-versus-poller test, and all 30 printer/admin confirm iterations
`status = 'PENDING'` in the claim	a cancelled order was emailed, and two pollers sent two emails
`SKIP LOCKED`	the locked-row test (the pollers waited)
`FOR UPDATE` in order creation	only the deterministic lock test
`FOR UPDATE` in design update	only the deterministic lock test (65 passed, 1 failed)
`FOR UPDATE` in design removal	the remove-versus-create test: a foreign key error instead of a 409

The lock rows are the lesson. First I tested the lock with 30 free-running iterations of update versus create. They all passed with the lock removed. The broken interleaving ends in the same final database state as the harmless one, so no assertion on the final state can tell them apart. Only a test that pauses order creation after it takes the lock, and then checks whether the update can finish, sees the bug.

A test also found a timezone bug. I ran the integration session in America/Los_Angeles. Comparing sendAt with a bare now() failed because sendAt is a timestamp without time zone. The comparison now uses now() AT TIME ZONE 'UTC', assuming sendAt is stored as a UTC timestamp. This fixes the timezone interpretation, but not clock skew between the application and database; I discuss that remaining limitation below.

Prisma Postgres and Compute

Prisma ORM 8 (I write Prisma 8 for short) is a release candidate and Composer is in Early Access. Compute has been generally available since August 2026. Everything in this section was true on my machine in October 2026. Versions: Prisma CLI 8.0.0-rc.27, @prisma/orm-postgres 8.0.0-rc.20, Composer 0.34.0.

Why Prisma ORM 8 is in this project

The Console told me that my repository used prisma ^7.7.0, that deploys run on Prisma 8, and that I should upgrade first. So I installed Prisma 8 next to Prisma 7, inferred a contract from my database, and planned a baseline migration. The plan had 15 operations from an empty database: four enums, four tables, the (status, sendAt) index and four foreign keys.

Later, Martin from Prisma told me that a Prisma 7 app runs fine on Compute and that Prisma 8 is not required. I did not test a Prisma 7 only deploy myself, so I cannot confirm it, and I reported the Console banner to Prisma.

I kept Prisma 8 for one job: the deploy applies my baseline migration to the new database. My queries still run on Prisma 7 with the pg adapter. The docs also describe a plainer option, rawPostgres(), which hands your own client a db.url. With it, you migrate the database yourself. I did not use it.

I kept the queries on Prisma 7 for another reason. The release status page lists what Prisma 8 does not have yet, for example transaction isolation levels, atomic increment, most nested writes and P2002-style error codes. My locks use raw SQL and a guarded updateMany, and I did not try them on Prisma 8.

Two more facts for Prisma 7 users. First, npx prisma now runs the Prisma 8 CLI, which has different data commands (contract, db and migration). Pin prisma to ^7 and use npx prisma@7, or install an alias package like I did. Second, Prisma 8 is a release candidate, and the release status page expects general availability in October 2026.

Getting a NestJS 11 app onto Compute

Compute runs a single bundled file. My notes:

The deploy tool refused Node 22.23.4 and asked for Node 24.11.1 or newer. The Compute docs say Node 22.18 or newer on the 22 line is enough, so the docs and the tool disagree.
bcrypt is a native addon. Bundled into one file, it crashed at startup with __dirname is not defined. I switched to bcryptjs. Login and the tests still passed.
prisma dev is not supported on Windows yet, so I tested the bundle with plain node.
Composer needs type: module. I put the Composer files in a deploy folder with its own package.json, so nest build stays CommonJS. A small launcher reads the database URL that the platform provides and then starts the compiled app.
The Console's setup pull request failed at npm install with ERESOLVE. I could reproduce a conflict on my machine by adding alchemy next to Jest 30 (a Babel peer dependency clash). I could not see the Console's log, so I do not know if it is the same cause.

The first deploy created ten resources, including the database and the migration step, and returned a public URL. The docs say Compute runs deployed apps on Bun, and I built my bundle for Node. The app worked when I tried it. The poller is switched off on that deployment, so orders stay PENDING there.

The surprise: scale to zero versus my poller

My poller is a timer inside the server. I deployed a small test app with a counter that ticks every 5 seconds and shows its history. Then I left it alone overnight.

The startedAt value was the same after 15 hours, so the app was never restarted.
Only 2 ticks were recorded. The first gap was 54,940 seconds.
New ticks appeared at the moments I opened the page. Then they stopped again.

The Compute docs describe this: idle instances sleep with a memory snapshot and resume in milliseconds. A timer inside the process only runs when someone calls the app.

What still holds: one database claim wins, and a cancelled order cannot be claimed for dispatch through this flow. What breaks is the timing. The email goes out after the next request, not at minute 10. Exactly-once email delivery is not guaranteed; a crash during recovery can result in a duplicate.

Open question: where should the poller run?

Compute has no built-in cron scheduler. A Prisma engineer told me that the recommended patterns are options 2 and 3 below, and that keep-awake is meant for short, bounded work. The Compute limitations page and the Composer cron building block describe both. I have not chosen between them yet, and I would like to discuss it. These are the options I found:

Keep the instance awake. Prisma says keep-awake is meant for short, bounded work, not for an always-on loop. The docs add that it does not guarantee continuity through restarts or deployments. I have not measured what a permanently awake instance costs.
An external trigger (recommended by Prisma). A scheduler outside Compute calls a protected route. The route wakes the app, runs the same claim logic and returns. My tests show that extra triggers are safe: two pollers claim an order once. The timing precision depends on the scheduler, and the request must start responding within 60 seconds.
The Composer cron building block (recommended by Prisma). Its scheduler is a service that never scales to zero, so it costs one warm instance. Composer is in Early Access, and I did not find an example of mounting the cron runner in a NestJS app.

Questions that are still open:

Does the always-warm cron scheduler count against the Free plan, and how much memory does it use?
Has anyone mounted the cron runner inside a NestJS app?

If you solved this, I would like to hear how.

Raw output

These are trimmed outputs from my own runs.

The deploy:

t-shirt-backend-test
├─ database   postgres-database db_...
└─ app        compute-service cps_...
Stack deployed (10/10)
Deployed t-shirt-backend-test to production in 3m 1s.

The bundle with the native bcrypt addon, at startup:

ReferenceError: __dirname is not defined in ES module scope
    at ... require_node_gyp_build()(path.resolve(__dirname))

The test app's ticks, first call (15 hours after the deploy):

{"startedAt":"2026-10-25T16:36:23.113Z","now":"2026-10-26T07:59:53.581Z","count":2,
 "gaps":[{"from":"2026-10-25T16:42:26.838Z","to":"2026-10-26T07:58:06.462Z","seconds":54940}]}

The second call, about 10 minutes later (the first gap is shortened):

{"startedAt":"2026-10-25T16:36:23.113Z","now":"2026-10-26T08:10:05.860Z","count":4,
 "gaps":[{"seconds":54940},{"from":"2026-10-26T07:58:06.462Z","to":"2026-10-26T07:59:53.591Z","seconds":107}]}

Two failures from the mutation runs:

status guard removed, confirm race:   Expected length: 1   Received length: 2
design lock removed, lock-held test:  Expected: "rejected"  Received: "fulfilled"

What I did not verify

I did not run the poller on Compute under load, and I did not measure how late the emails are.
I have not fixed clock skew between the app server and the database. sendAt is written from the app clock and compared with the database clock, so skew changes the grace period silently.
There is no automatic retry for failed emails.
I did not port locks and transactions to Prisma 8. They stay on Prisma 7.

Takeaways

Let the database decide the winner. Do not let a timer or a clock decide it.
Break your code on purpose. A test that also passes without the guard proves nothing.
A free-running race loop can pass for the wrong reason. Force the interleaving.
Scale to zero changes what a timer inside your server means. Check it before you design around it.

I'm Mhamd, a backend engineer working with NestJS and TypeScript. Portfolio: mhamd021.github.io · GitHub: Mhamd021

A Stopwatch Needs an Isolated Lane

Wed, 28 Oct 2026 10:49:36 +0000

Timing tests look wonderfully objective. Start a stopwatch, call the component, stop the watch, and compare the result with a budget.

The number is precise. The experiment may not be.

That distinction matters in continuous integration, where many tests share CPU time, thread pools, network stacks, containers, and database capacity. A wall-clock assertion can fail because the product became slower. It can also fail because the test runner became busier. If we cannot distinguish those two explanations, the test is producing urgency without producing useful evidence.

The contract was reasonable

Consider a service with separate liveness and readiness checks. Liveness says the process is still running. Readiness says the instance can safely accept work.

One integration test deliberately points the service at an unreachable dependency. The expected behavior is not success. The service should return an unavailable response, and it should do so within a fixed upper bound. That is a meaningful reliability contract: an orchestrator needs a quick, honest answer rather than a request that hangs behind retry policies intended for normal user operations.

The test boots the real host, makes the request, verifies the unavailable result, and checks elapsed time.

So far, so good.

The failure blamed the wrong interval

The timing assertion began failing repeatedly in CI. Raising the limit would have made the build green. Adding an automatic retry might also have hidden the symptom.

Neither action would have explained the result.

The endpoint implementation had not changed during the relevant window. More importantly, the surrounding timing evidence showed that host startup in the same test took roughly as long as the delay attributed to the readiness request. The test also ran beside many parallel integration classes doing heavier work.

The stopwatch was attached to one request, but that request still depended on threads scheduled in a shared process. Under contention, continuation work can wait. Startup can wait. Connection failure can be observed later than expected. The test reports wall-clock time, not pure execution time.

This does not prove the service is fast. It proves the original experiment was unable to tell product latency from runner contention.

Keep the budget before changing it

When a timing test turns red, I prefer to keep the bound intact while investigating. The bound represents a real operational expectation. Relaxing it before understanding the measurement destroys evidence that may still be valuable.

A useful first pass is to separate three intervals:

Setup — host boot, fixture creation, migrations, container startup, and test data.
Contention — time spent waiting for CPU, threads, sockets, database slots, or another shared resource.
Behavior — the operation whose latency the assertion is meant to protect.

Instrumenting those boundaries does not require a production profiler. Even a few carefully placed timestamps can reveal that the supposedly slow operation accounts for only a small part of the elapsed test.

Isolate the claim, not the whole suite

The narrow repair was to move only the timing proof into a non-parallel test collection.

The response status did not change. The time budget did not change. The endpoint code did not change. Functional readiness tests stayed in the normal parallel suite. Only the test making a wall-clock claim received an isolated measurement lane.

That scope matters. Serializing an entire integration project can turn a useful diagnosis into an expensive habit. Parallel tests are valuable because they shorten feedback cycles and expose accidental shared state. The aim is not to remove parallelism. It is to stop using an uncontrolled parallel environment as a precision instrument.

I like to keep separate tests for separate questions:

Does the check return the correct status when the dependency is unavailable?
Does the response avoid leaking infrastructure details?
Does the host remain alive while readiness is false?
Under controlled contention, does the readiness answer stay within its operational budget?

Only the last question necessarily needs timing isolation.

The trade-off is real

An isolated collection adds a serial tail to the build. It also creates another test boundary that future maintainers must understand. If many tests demand exclusive execution, the suite can become slow and the isolation can conceal real capacity problems.

That is why this technique should be applied to the measurement contract, not used as a blanket cure for flakiness.

In return, failures become more actionable. A red timing check is less likely to mean “the runner happened to be crowded” and more likely to mean “the protected latency contract changed.” The team spends less time rerunning the same experiment and less time normalizing noise by widening budgets.

A practical review checklist

Before changing a failed CI time limit, ask:

Did the measured implementation change?
Is setup included in the timed region?
Which resources are shared with parallel tests?
Does the test need fixtures it is currently competing for?
Can functional correctness remain parallel while the timing proof runs alone?
Would the same isolation exist in the production condition we care about?

The last question prevents overfitting. An isolated test should remove irrelevant test-runner noise, not erase meaningful load. If the production contract is specifically about behavior under concurrency, create a controlled load test with explicit pressure instead of relying on incidental CI contention.

Precision is a property of the experiment

A stopwatch does not make a test scientific by itself. The environment, boundaries, and competing work determine what the number means.

Keep meaningful budgets. Investigate before retrying. Isolate the smallest timing claim that needs a controlled lane, and leave the rest of the suite parallel.

The result is not merely a greener pipeline. It is a test whose failure tells you something you can act on.

MT5 Tick Size vs Point: A Test Harness for Price Distance

Wed, 28 Oct 2026 06:37:05 +0000

The smallest quoted point and the minimum price change used as a trading tick can differ. If a program divides an exit distance by SYMBOL_POINT but multiplies by the monetary value of a trading tick, the resulting loss estimate may use incompatible units. Test the units before testing position sizing.

Educational and simulation content only. This is not financial advice, intermediation or an investment recommendation. Trading can result in loss of capital.

A fictional contract

Imagine a demo symbol with a point of 0.01 price units, a tick size of 0.05, and a loss-side tick value of 2 account-currency units per lot. A 0.50 price move spans 10 trading ticks, not 50. For 0.10 lots, a simplified linear estimate is 10 × 2 × 0.10 = 2 account-currency units. Substituting point for tick size would multiply that estimate by five. These inputs are fictional and do not describe any particular broker or instrument.

The official MQL5 symbol reference defines SYMBOL_POINT, SYMBOL_TRADE_TICK_SIZE, and SYMBOL_TRADE_TICK_VALUE_LOSS separately. Read actual symbol properties from the platform; do not assume that every symbol, account currency, or loss direction shares the same values.

An executable unit test

This example checks arithmetic under a deliberately narrow assumption: constant loss-side tick value and linear scaling with volume. It does not submit orders or model commissions, gaps or currency conversion changes.

from decimal import Decimal

def linear_estimate(entry, exit_price, tick_size, tick_value, lots):
    entry, exit_price, tick_size, tick_value, lots = map(
        lambda x: Decimal(str(x)),
        (entry, exit_price, tick_size, tick_value, lots),
    )
    if tick_size <= 0 or tick_value <= 0 or lots <= 0:
        raise ValueError("Tick size, tick value and lots must be positive")
    distance = abs(entry - exit_price)
    return distance / tick_size * tick_value * lots

assert linear_estimate(100, "99.50", "0.05", 2, "0.10") == Decimal("2.00")
assert linear_estimate(100, "100.50", "0.05", 2, "0.10") == Decimal("2.00")
assert linear_estimate(100, 100, "0.05", 2, "0.10") == 0

The absolute distance checks units only. It does not prove that the exit lies on the loss side: that depends on whether the simulated position is long or short. Check direction and valid prices separately. Prices and tick values can change; contract rules may make a simplified linear calculation unsuitable.

Compare with the platform

OrderCalcProfit estimates the result in the account currency for an order type, symbol, volume and specified open and close prices. In a demo environment:

Record symbol, account currency, direction, open price, exit price, volume, point, tick size and loss-side tick value.
Check the return value of OrderCalcProfit. If it fails, record the platform error and reject the estimate; never interpret failure as zero.
Compare the signed result with the simplified calculation, using the same prices and volume. Investigate discrepancies rather than silently taking the smaller number.
Repeat for long and short positions, multiple symbols and the broker's allowed volume increments. Keep fixtures and platform version with the results.
Account separately for known costs, execution gaps and contract-specific rules. An estimate at the requested exit price is not a guaranteed loss limit.

MT5 also specifies minimum, maximum and step sizes for volumes. Those constraints are a separate test; see our earlier position-sizing article for the minimum-volume edge case. The Gueta Quant position sizer provides an educational comparison, not a guarantee of execution.

My quality gate wasn't strict. It was dead.

Tue, 27 Oct 2026 21:03:40 +0000

There is a job in my CI called judge. It reads the monthly issue my content pipeline writes, scores it against a rubric with five criteria, and blocks the merge if anything comes back at 2 out of 5 or lower. An LLM grading an LLM, with a written policy about what counts as broken.

It went red on a content PR this week. My first thought was that the piece was weak. My second thought, about four minutes later, was worse: this gate had been red on every content PR for a while, and I had been reading that as a high bar.

It wasn't a high bar. The job never reached the rubric at all.

Error: anthropic messages -> 400: output_config.format.schema:
For 'integer' type, properties maximum, minimum are not supported

The judge asks the model for structured output, and the schema it sends had this in it:

voto: { type: "integer", minimum: 1, maximum: 5 },

Which is correct JSON Schema, and which the structured output layer rejects. The request never made it to the model. The job exited 1 before scoring a single word.

A gate that cannot go green is not enforcing anything. It has the same effect on your pipeline as continue-on-error, except it also makes you feel virtuous while it does nothing.

The error message named half the problem

The obvious fix is to delete minimum and maximum and move on. I nearly did. What stopped me was noticing that the schema had a second kind of constraint in it:

motivo: { type: "string", maxLength: 300 },

Length constraints are rejected too. maxLength, minLength, pattern, minItems, all of it. My schema had three of them, and the 400 mentioned none, because validation stops at the first failure. If I had fixed what the message named, I would have pushed, waited for CI, and collected the next 400. Then the one after that.

This is a general property of error messages, not a quirk of one API: they report the first thing that broke, not the set of things that are broken. The fix that only addresses what the message says is a fix sized to the message, not to the defect.

The other thing worth knowing here: the Python and TypeScript SDKs strip unsupported constraints for you before the request goes out. My pipeline is zero-dependency and calls the API with native fetch, so nobody was stripping anything. If your SDK has been quietly saving you, you will find out the day you drop it.

So the test is about the contract

I could have written a test asserting no minimum on integers. It would have passed, and it would have been useless three weeks later when someone adds a pattern to a string.

Instead the accepted and rejected keywords are written down once, and every schema I send to the API gets checked against them:

const RIFIUTATE = new Map([
  ["minimum", "numeric constraint"],
  ["maximum", "numeric constraint"],
  ["maxLength", "length constraint"],
  ["pattern", "string constraint"],
  // ...
]);

export function keywordRifiutate(schema, path = "$") { /* walks the tree */ }

It walks nested schemas, and it knows that inside properties the keys are my field names rather than schema keywords, so a field genuinely called pattern doesn't trip it.

On first run it found sixteen violations across the judge schema. Not two. My generation schema, checked at the same time, was already clean, which I would not have bet on.

There is a second check next to it that lists keywords the contract doesn't mention. It fails rather than allowing them, which sounds aggressive for something the API might accept fine. The reason is that I want the moment where somebody adds an unfamiliar keyword to be the moment somebody reads the docs, instead of the moment CI goes red for a reason nobody connects to the schema.

Where the constraint went

The rubric still runs on a 1 to 5 scale. That range now lives in the two places the API can't reject it.

The prompt describes it, with each score defined. And the code treats an out-of-range score the way it already treats a missing one:

const fuoriScala = (v) => !Number.isInteger(v) || v < 1 || v > 5;

Fail closed. If the model returns a 7, the rubric is unreadable, and an unreadable rubric doesn't promote anything. That was already the policy for a criterion the model forgot to fill in. A score outside the scale is the same class of problem, so it gets the same answer.

Moving a constraint out of a schema usually means giving it up. Here it just meant enforcing it somewhere with a worse reputation and better reach.

What I'd check on your own pipeline

Grep your CI history for a job that has never been green. Not "was red today", never green at all, or not since some commit that had nothing to do with it. That job is not protecting you, and it is currently costing you the specific kind of comfort that comes from thinking it is.

Mine ran for weeks. It was the most reassuring line in the workflow file.

Four times my system looked correct and wasn't.

Tue, 27 Oct 2026 14:27:47 +0000

Most of the bugs that taught me something weren't crashes. The system worked, the checks were green, and the property I cared about was still broken. Here are four of them, from three different projects.

1. The prompt-injection test that passed while the attack worked

In LLM-Council, three models answer a question, rank each other's answers blind, and a chairman model writes the final synthesis. Output from one stage becomes input to the next, so I had to fence untrusted model output off from the instructions around it.

My first version used static delimiters. I wrote tests that checked where the markers were placed and how the string was built. They passed.

The problem: those tests checked that the Python string was assembled correctly. An attacker doesn't care about that. A hostile model output can simply reproduce the static closing delimiter, and the next model reads the untrusted block as finished.

The fix was a fresh random nonce in the delimiters for every prompt, so untrusted output can't predict the real closing marker. I found a second boundary that needed the same treatment: the rankings going into the chairman stage were untrusted too.

Then I broke the fix on purpose. I replaced the random nonce with a static one and checked that the tests went red. I removed the ranking fences and checked that those tests failed as well.

One limit worth stating: the nonce protects the boundary against delimiter forgery. It doesn't make the content inside trustworthy, and it doesn't solve prompt injection in general.

Full write-up: I wrote a test for prompt injection. It passed while the attack worked.

2. Retrieval that returned valid results for the wrong decision

MonferrinoAI is a RAG pipeline (Firecrawl, Voyage AI embeddings, Supabase with pgvector) that prepares weekly content briefings for a local business website. A human still writes and publishes.

I tested retrieval against the real database and found that a keyword-only query could rank the wrong page above the one that was actually relevant. Nothing was technically broken: every result was a valid row. Adding context to the query, the keyword cluster and the target page, fixed the ranking.

The same project has a security side. Web pages are untrusted input and can contain text that looks like an instruction. So the page bodies never reach the final briefing: the handoff only carries URLs, headings and similarity distances.

3. Telemetry that carried more than I expected

Agentic-OS collects Claude Code telemetry through an OpenTelemetry Collector, with Prometheus and Grafana for metrics and a private Loki pipeline for logs. The point is to see what an agent actually did, not what it says it did.

Testing against the real client showed that it can send identity attributes, including an email address. My monitoring pipeline was a privacy boundary and I hadn't treated it as one.

Now attributes go through an allow-list, and the tests run against the real telemetry shape. Anything unknown is dropped instead of stored.

4. A strict CSP that the build quietly broke

My website uses a strict hash-based Content Security Policy without unsafe-inline. Syntax highlighting added inline styles that the policy would block.

The quick fix was to loosen the CSP. I removed the incompatible behavior instead, and added CI tests that run against the built HTML, the files that actually ship, checking inline scripts, hashes and style attributes. Some of those problems only exist after the build, so testing the source wouldn't see them.

The question I ask now

In all four cases a check existed and looked reasonable. It just wasn't looking at the property I cared about.

So before I trust a control, I ask: what should turn red if this property stops being true? Then I break the property and see if it does.

This post grew out of my Developer Story on CoderLegion, written by Mehadi Hasan, which covers the projects and the path behind them in more detail.

The Guardian redacted my secret. The Read tool walked around it.

Tue, 27 Oct 2026 14:08:26 +0000

I put a fake AWS key in a file and asked Claude Code to read it twice in the same session: once with its Read tool, once with cat. An OWASP Guardian sat between the agent and its tools.

With cat, the model got this:

AWS_ACCESS_KEY_ID=[REDACTED]

With Read, it got this:

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE

That's the example key from the AWS documentation, not a real one. Only the tool changed, and I got the same result in all four runs.

What I was testing

The Agent Control Standard (ACS) is a wire specification from the OWASP GenAI Security Project. A separate Guardian Agent sees what an AI agent is about to do, and can permit, deny or modify it before it happens. The project ships a reference implementation: a Guardian that runs Microsoft's Agent Governance Toolkit (AGT) as its policy engine, plus hook shims for Claude Code and OpenCode.

Issue #92 asks people to run it against a real harness and report where behavior and specification disagree. The Claude Code runs I found there used a synthetic executor, drove the hook by hand, or ran without a Guardian. I wanted Claude Code with its own model deciding which tool to call, going through the shipped hook into the reference Guardian.

The setup: Claude Code 2.1.308, ACS at commit fd6ff383, AGT at the commit the repository pins, all inside a Docker container on a VPS, so nothing touched the host.

Claude Code had two roles in this project. It was the system under test, and it was my pair for the work: it wrote the protocols and the scripts, ran the automated arm and drafted the comparisons. That is the situation a blind is for.

Each protocol went like this:

The protocol and the scripts were frozen first, and I got their sha256 before anything ran.
The non-interactive arm ran three times with claude -p. Its output went straight into a tar archive, and the terminal showed only the run conditions and the archive's hash.
I ran the same prompts by hand in an interactive session. My archive was sealed before anyone opened the automated one.
Only then were both archives opened and compared against the protocol.

I ran three protocols with four runs each. The pair that ran the automated arm couldn't steer my manual run, and nobody could write the comparison to fit a result they had already seen.

The first protocol had a confound: my Claude account loaded its own plugins, with their hooks and 55 MCP connectors, in both arms. The second protocol started from a clean environment, and nothing changed on the wire for Bash.

What worked as documented

The README's own demo is to ask Claude Code to run echo rm -rf /. AGT's stock rule denies it, and ls -la in the same session runs. That's what happened in every run. Once I removed UUIDs, timestamps, sequence numbers, the text the model writes and the ls listing, the envelopes on the wire were identical across all eight runs of the first two protocols.

Interactive and non-interactive sessions gave the same decisions, the same executions and the same content reaching the model, prompt by prompt.

After a deny, the model reported the block, called it a false positive and did not try another tool. That's the model's judgment, not mine: the README picks this command because it stays harmless if the hook ever fails. In all three interactive runs, and in none of the automated ones, it also offered a way out: type ! echo rm -rf / yourself. Claude Code's ! prefix runs a shell command outside the tool loop, so no hook sees it. So the model stayed inside the control and told me how to step outside it.

With the Guardian stopped, the command ran, and the audit log recorded posture: proceed with a transport failure. That is the documented default, and other reports in #92 had already measured it.

Read never reaches the Guardian

The shipped hook settings match Bash and WebFetch before a tool runs, and only Bash after it. The README says so: "Both matchers are anchored, so no other tool is intercepted."

So the gap is documented. What the README doesn't spell out is the consequence. Redaction happens at the result gate, after the tool runs and before its output reaches the model. For cat, the Guardian answered modify with redaction_applied, and the model got [REDACTED]. For Read there was no envelope at all, so there was nothing to inspect and nothing to redact.

The specification asks for more than the adapter does. It says frameworks MUST fire toolCallRequest for every action that escapes the agent's reasoning context, and that this "includes built-in operations such as filesystem reads/writes". Claude Code can hook Read: its hooks documentation lists Read among the tools PreToolUse matches. The gap sits in the adapter's matcher. It's the same gap that #212 tracks for the Cursor adapter.

I filed it as #330.

MCP tools: three states

The README says wrapped MCP isn't implemented yet. I measured what happens today, with a small local MCP server that exposes one tool, run, taking a command argument:

Configuration	What happened
Shipped hook	No envelope. The command runs.
Matcher widened to `mcp__.*`	The call reaches the Guardian and is denied as `runtime_error:tool_unknown`, whatever the command
Tool also registered in the policy manifest	Judged like Bash: `echo ciao` runs, `echo rm -rf /` is denied by the same rule

The middle row can mislead. The deny looks like protection, but when I drove the hook by hand, echo ciao got the same deny. AGT refuses a tool name it doesn't know before any rule reads the content.

The third row took seven lines in the manifest, in the same shape as the Bash entry. It works only because my tool's argument is called command, which is the argument the mapping hands to the policy by default. The output of an MCP tool still never reaches the result gate.

The Guardian's log keeps the secret

In the cat run the model saw [REDACTED], but the Guardian's envelope log on disk holds the key in plaintext, inside the result request the hook sent. The Inspector's log has it too. The README warns that the envelope log holds raw tool arguments, and my runs show it holds raw tool output as well. There is already an issue for this (#71), and I added my measurements there.

What I didn't measure

I didn't test Edit, Write, Grep or Glob, a Guardian restart in the middle of a session, WebFetch with the egress allowlist, real authenticated MCP servers, or any MCP tool whose argument isn't called command. Each protocol had one manual run, by one person.

The question I ask now

Every control I tested did what it said on the path it watched. The secret took the path next to it.

So before I trust a control on an agent, I list every tool that can touch the same data and check that each one goes through the control. If one doesn't, the control protects a path, not the data.

The report is #330, and the full write-up is my comment on #92. The protocols and logs are available if you want to rerun them.

Quotes from the ACS README and specification are © The OWASP GenAI Security Project and the ACS contributors, licensed CC BY-SA 4.0.

A Query Window Is Not a Calendar Day

Mon, 26 Oct 2026 20:12:29 +0000

Software often uses broad time windows for good reasons. A dashboard may need recent history, work that is actionable now, and the next scheduled item. Fetching all of that in one bounded query can be efficient and easy to reason about.

The trouble begins when the retrieval window quietly becomes the business definition of “today.”

This is a small distinction with a very visible failure mode: shortly after midnight, yesterday’s missed item appears under today’s heading. The query is returning exactly what it was asked to return. The selection rule is wrong.

Retrieval and meaning are different jobs

Imagine a “Today” screen that loads items from several hours in the past through the next day. The look-back allows the screen to include work that recently closed or may still be open. The look-ahead allows it to show what comes next.

That range is a retrieval concern. It answers:

Which records could be relevant enough to inspect?

It does not answer:

Which record belongs to the user’s current calendar day?

If selection simply chooses the first item whose state is due or missed, a previous day’s missed item can win after midnight. If display text assumes every future item is later today, tomorrow morning can be described as though it were later tonight.

Both errors come from using temporal proximity as a substitute for calendar meaning.

Define “today” in the relevant time zone

Server time is rarely the correct authority for a human-facing day. UTC is excellent for storing instants and comparing elapsed time. It is not a user’s calendar.

The safer sequence is:

Read the current instant from an injected clock.
Convert it once into the time zone that owns the workflow.
Derive a local date from that value.
Use that date when classifying calendar-bound states.

Computing the value once matters. It keeps every decision in one request anchored to the same instant, including the unlikely case where execution crosses midnight.

The relevant time zone also needs to be explicit. It may belong to a person, site, account, or workflow. The correct choice is a domain decision, not an infrastructure default.

Preserve the exception that actually matters

A tempting fix is to discard every record whose local date is not today. That is neat and wrong when an actionable window crosses midnight.

Suppose a task opened before midnight and remains valid until shortly afterwards. A strict date filter would hide it at the exact moment it still needs attention.

This is where explicit domain rules beat blanket filtering:

missed and completed items count as current only when their local date is today;
a still-open item remains current while it is actionable, regardless of its start date;
a scheduled item on another day can still appear as next, but its wording names that day.

The rule is slightly longer because the product meaning is slightly richer. That extra branch is not accidental complexity. It records why two superficially similar records are treated differently.

Make wording part of correctness

Time bugs are not confined to sorting and filtering. Labels can encode false claims too.

“Later today” is only correct if the next item is on the same local day. “Between 08:00 and 09:00” may be ambiguous when viewed the night before. Adding “tomorrow” or a concrete day makes the model visible to the user.

That wording also gives tests a useful public outcome. A test can verify that the screen says tomorrow without reaching into a private helper or asserting a particular LINQ expression.

Test the boundary as a sequence

A single test at noon will not find this class of defect. The useful test moves a controllable clock through the state transition.

One focused integration scenario can prove three behaviours:

On the item’s own evening, the missed item is still classified as belonging to today.
The next day’s scheduled item is labelled explicitly as tomorrow.
After midnight, the old missed item no longer becomes the current card.

Using the real persistence provider is valuable when stored dates, generated occurrences, or query translation participate in the behaviour. The clock is the external boundary worth controlling; internal services do not need to be mocked if the public result proves the rule.

Also test a window that genuinely spans midnight. Without that case, a cleanup that filters everything by date can look correct while introducing a different regression.

The engineering trade-off

Explicit local-date classification costs a few things:

more branches than a simple state check;
a time-zone conversion at the application boundary;
boundary-focused tests and more deliberate copy.

In return, the system becomes predictable at the moment when implicit assumptions fail most often. Retrieval can remain broad for performance and completeness. Domain selection becomes precise. User-facing language reflects the same model.

The broader leadership lesson is to ask two questions whenever a time range appears in code:

Is this range finding candidates, or defining business membership?
Whose calendar gives words such as today, tomorrow, and missed their meaning?

Keeping those answers separate prevents an implementation convenience from becoming an accidental policy.

Practical checklist

When building a day-oriented view:

store and compare instants consistently;
identify the domain-owned time zone;
derive the local date once per operation;
keep retrieval windows separate from classification rules;
preserve intentionally actionable cross-midnight windows;
make off-day wording explicit;
test before midnight, after midnight, and a spanning window.

The code may still query eighteen or thirty hours. The user should experience a calendar day.

The Testing Trophy, and Where TWD Fits

Mon, 26 Oct 2026 19:17:07 +0000

Most frontend teams I talk to have plenty of tests and still find out about broken screens from users. The unit tests are green, the refactor broke every one of them anyway, and nobody is sure the checkout flow works until someone clicks through it.

That tension is exactly what the Testing Trophy is about.

The trophy

Kent C. Dodds drew testing as a trophy instead of a pyramid. Four layers, and the size of each one is roughly how much of your effort it deserves:

Static: typos and type errors caught as you write the code. TypeScript, ESLint.
Unit: one function, hook or component checked in isolation. Vitest, Jest.
Integration: several units working together. The biggest part of the trophy.
End to End: a robot clicking through the deployed app, real backend included. Playwright, Cypress.

The shape comes from one line, written by Guillermo Rauch and popularized by Kent:

Write tests. Not too many. Mostly integration.

Each layer trades speed for confidence. Unit tests are cheap, but they can pass while the screen is broken. End-to-end tests give the most confidence, but they need the whole system running and are the slowest to write and run. Integration tests sit in the middle: they use the app the way a user does, and they stay fast and deterministic because the only thing faked is what you don't own.

Kent's guiding principle for them:

The more your tests resemble the way your software is used, the more confidence they can give you.

What integration means on the frontend

For a frontend app, an integration test renders the real thing (routes, providers, state, components) and mocks only the network. Then it does what a user would: visit a page, type, click, and check what is on the screen.

The classic way to do this is Testing Library with jsdom, in Vitest or Jest. It is a good setup and it is where most of us learned this style. It runs in a simulated DOM in your terminal, though, separate from the app you have open while you build.

Where TWD fits

TWD is built for that middle layer, and it runs the tests in the browser you develop in:

Next to your app. A sidebar in your dev server runs the tests against the real app. You watch each one drive the page.
Mock what you don't own. API calls are answered by a service worker, and contract testing checks every mock against your OpenAPI spec, so the mocks don't drift from the real API.
Tests that read like the user. Testing Library queries find elements by role, label and text.
The same tests everywhere. twd-cli runs them headlessly in CI, with coverage and a video per test if you want one, and your AI agent can write, run and fix them.

In practice this layer can carry a lot. On two production apps we run 872 tests on every pull request, mostly user flows, in one headless Chrome on a 2-vCPU GitHub runner, in about 5 minutes per app.

What it doesn't replace

TWD doesn't cover the whole trophy, and it shouldn't.

Keep a few end-to-end tests with Playwright or Cypress for the critical paths, the ones that need the real backend, the payment provider or the third-party SDK. Use unit tests for logic that is easier to check on its own. Keep TypeScript and ESLint at the base (and twd-lint for your test files). Then let integration tests carry most of the weight.

A map, not a ritual

TWD doesn't ask you to pick a layer before you write a test. You automate the check you were already doing by hand while you build. The trophy just shows where those checks end up: mostly in the middle.

The full page, with the trophy and the tools per layer, is at twd.dev/testing-trophy. For the strategy itself, go to the source: Kent's TestingJavaScript.com.

Why I Test the Proxy Path Before I Trust a WAF Block

Sun, 25 Oct 2026 02:27:04 +0000

Installing a console and proving that visitor traffic is inspected are different tests. SafeLine can filter a request only after that request enters its reverse proxy. If the public name still points at the origin, or if you call the upstream address directly, a malicious query can succeed while the attack list stays empty. When I need to know how to test if your WAF is working, I ask three questions in order: did this request reach SafeLine, did a documented malicious query get blocked, and does the console detail belong to that same attempt?

SafeLine is a self-hosted web application firewall. The product introduction says it protects web apps by filtering and monitoring HTTP traffic, including SQL injection, XSS, code injection, OS command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, brute force, HTTP flood, and bot abuse. It works as a reverse proxy: clients pass through it before they reach the server, malicious HTTP traffic is blocked, and unauthorized data is prevented from leaving the app. The application guide adds the operational detail: SafeLine is developed on nginx, receives traffic for the original website, cleans malicious attack traffic, and forwards safe traffic to the original application. Those statements are in https://docs.waf.chaitin.com/en/home and https://docs.waf.chaitin.com/en/GetStarted/AddApplication.

Prove the proxy path first

I do not start with an exploit string. I log in to the Web Admin Console, open Applications, and check the application record. Domain is the name, hostname, or address clients use. Port is the port SafeLine listens on, such as 80 or 443, and an HTTPS app needs the SSL option checked. Upstream is the real address of the original app. After those fields are saved, the domain has to resolve to the IP of the server where SafeLine runs. Until that is true, a browser request is not a test of this WAF.

A negative check matters too. If I request the upstream address and skip the published domain, I am not exercising SafeLine. A normal page from the origin only shows that the origin is up. It says nothing about detection. The home page also includes an effect evaluation on a fixed sample set. I do not treat that table as proof that my own site is protected. A live test still has to travel through my domain, my port, and my upstream.

Use the cases the guide already published

The application guide includes a short set of cases for this check. Replace the example host with the protected domain. SQL injection is /?id=1+and+1=2+union+select+1. XSS is /?id=<img+src=x+onerror=alert()>. Path traversal is /?id=../../../../etc/passwd. Code injection is /?id=phpinfo();system('id'). XXE places <?xml version="1.0"?><!DOCTYPE foo SYSTEM ""> in the id query value. Request those URLs on the domain and port configured for the application. Do not send them to the upstream IP.

The guide says that if you see the blocking page, SafeLine defended the attack. That page is the first signal. It is not the whole proof. Open the web console, find the attack list, and click detail. I treat the test as passed only when the blocking page and that detail both line up with the probe I just sent. A block page without a matching console event can come from another device, a cached response, or a different hostname. A console row without a block on the public URL can mean I tested a name visitors do not use.

Do not stretch one sample into full coverage

Those five cases are enough to show that common web-attack filtering is active on that path. They do not exercise every capability listed in the introduction. Rate limiting is described as throttling traffic that exceeds defined limits, including DoS attacks, brute-force attempts, and traffic surges. CAPTCHA challenge is described as a way to allow human users while blocking crawlers and bots. Authentication challenge, when turned on, requires a password or the visitor is blocked. Dynamic protection, when turned on, dynamically encrypts HTML and JS on each visit. None of the published one-line probes is a test of those controls. If the site depends on one of them, I record that the web-attack check did not cover it, instead of assuming that control is active.

The management console is a separate path as well. The deploy guide shows how to open https://<safeline-ip>:9443/ and how to recover the initial administrator account with docker exec safeline-mgt resetadmin. A successful login proves that the console answered. It does not prove that traffic on the application port was inspected. Install checks, including Docker version, architecture, memory, disk, and the CPU ssse3 note, belong to setup. Protection checks belong to the application domain. The deploy steps are documented at https://docs.waf.chaitin.com/en/GetStarted/Deploy.

I run these probes only against an application I control. They are examples from the product documentation, not a reason to scan someone else's site. After a block is confirmed, I load an ordinary page through the same domain. Safe traffic is supposed to be forwarded upstream. If that normal page fails, the problem is proxy or upstream configuration, and I do not call the security test successful. I write down the time, the exact URL, the application name, and whether detail opened. The next DNS or upstream change can then be checked the same way, instead of by memory.

How do I know the request reached SafeLine?

Resolve the application domain to the SafeLine server, use the port set in Applications, and send the probe to that domain. A request aimed at the upstream address skips the reverse proxy, so it cannot prove detection.

What result counts as a successful block?

The application guide says the blocking page means the attack was defended. Then open the attack list and click detail, and keep the result only if that event matches the probe you just sent.

Does one blocked query prove every control?

No. The published samples cover SQL injection, XSS, path traversal, code injection, and XXE. Rate limiting, CAPTCHA challenge, authentication challenge, and dynamic HTML and JS protection are separate documented capabilities.

The management port shows that the admin UI is reachable. Visitor protection depends on domain, listen port, SSL setting, and upstream. Test that path, not only the login screen.

What if the malicious URL loads normally?

Check DNS, port, and SSL, and confirm you did not call the origin directly. If the path is correct and a documented sample is still not blocked, treat the test as failed and review the application settings before you assume the site is covered.

SafeLine has surpassed 700,000 installations worldwide as of September 30, 2026. SafeLine Pro annual promotional pricing is $650-$880, compared with the regular $1,000 per year.

SafeLine website: https://ly.cyberserval.tech/MqhYdmI
Demo: https://demo.waf.chaitin.com:9443/statistics
Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Pro offer: https://cyberserval.tech/landing/lto

How to Test If Your WAF Is Working After Cutover

Sun, 25 Oct 2026 02:26:33 +0000

A console login only proves that the management service answered. It does not prove that public clients are being filtered. SafeLine is a self-hosted web application firewall placed in front of a web application so that it can filter and monitor HTTP traffic and block malicious HTTP and HTTPS requests before they reach the app, following policies that separate malicious traffic from safe traffic (https://docs.waf.chaitin.com/en/home). The application guide describes the same idea in proxy terms: SafeLine is developed based on nginx, receives traffic for the original website, cleans malicious attack traffic, and forwards the remaining traffic to the original application (https://docs.waf.chaitin.com/en/GetStarted/AddApplication). A working test has to show that your hostname is on that path, that a documented attack is blocked, and that the console recorded the same attempt.

Confirm the application path first.

Protection is configured in the web admin console on the Applications page. Choose Add Application, then enter Domain, Port, and Upstream. Domain is the domain name, hostname, or IP of the original app. Port is what SafeLine will listen on, such as 80 or 443, and HTTPS apps need the SSL option checked. Upstream is the real address SafeLine uses to forward traffic (https://docs.waf.chaitin.com/en/GetStarted/AddApplication). The guide then tells you to resolve that domain to the IP address of the server where SafeLine is located and to access the protected application through that domain.

Use that sequence as the first half of the test:

The hostname in your test URL must be the Domain you configured.
Public DNS for that hostname must point at the SafeLine server, not at the upstream host.
The port in the test must be the listener you set. If clients use HTTPS, the SSL option has to be enabled or you are testing a different service than production.
Upstream must be the origin you intend to protect. A successful page load after cutover only shows that forwarding works. It does not show that filtering works.

Keep the admin URL out of this path check. After installation, the web console is opened at https://<safeline-ip>:9443/ (https://docs.waf.chaitin.com/en/GetStarted/Deploy). Reaching that page shows that the management plane is up. It says nothing about whether a visitor to your public site ever touches the proxy.

Send the published samples, then match them to the attack list.

The application guide gives five cases and says to replace the example host with your website domain (https://docs.waf.chaitin.com/en/GetStarted/AddApplication). Run them only against the application you added:

SQL injection: /?id=1+and+1=2+union+select+1
XSS: /?id=<img+src=x+onerror=alert()>
Path traversal: /?id=../../../../etc/passwd
Code injection: /?id=phpinfo();system('id')
XXE: /?id=<?xml+version="1.0"?><!DOCTYPE+foo+SYSTEM+"">

The documented success signal is the blocking page. After each request, open the SafeLine web console, find the attack list, and click detail. Count the test as passed only when both signals refer to the same attempt: the browser was blocked, and the detail view exists for that request. One signal without the other is incomplete. A normal origin response means the request was not cleaned on that path, so go back to Domain, Port, SSL, Upstream, and DNS before you change anything else.

Do not stretch these five URLs into a claim about every control.

The introduction lists many attack types that a WAF of this kind is typically used to address, including SQL injection, XSS, code injection, OS command injection, CRLF injection, LDAP injection, XPath injection, RCE, XXE, SSRF, path traversal, backdoor, bruteforce, HTTP flood, and bot abused (https://docs.waf.chaitin.com/en/home). The same page lists core capabilities: defenses for web attacks, proactive bot abused defense, HTML and JS code encryption, IP-based rate limiting, and a web access control list. The feature list adds rate limiting against DoS attacks, bruteforce attempts, traffic surges, and other abuse; CAPTCHA challenges that allow human users and block crawlers and bots; an authentication challenge that blocks visitors who do not enter the password; and dynamic protection that encrypts HTML and JS on each visit (https://docs.waf.chaitin.com/en/home). Those are product facts, not extra test steps. The only click-by-click attack samples in the supplied application guide are the five URLs above. A pass on those URLs supports the web-attack blocking check. It does not prove that rate limiting, CAPTCHA, authentication challenge, dynamic protection, or an access list is enabled. Mark those items untested unless you have a documented procedure for them.

A sample-set score is a different question from a cutover test. The home documentation publishes an effect evaluation on 33669 samples. In that table, SafeLine Balance is listed at 71.65 percent detection, 0.07 percent false positive, and 99.45 percent accuracy, while SafeLine Strict is listed at 76.17 percent detection, 0.22 percent false positive, and 99.38 percent accuracy (https://docs.waf.chaitin.com/en/home). Use those figures only as published evaluation results. They do not inspect your DNS, your listener, or your upstream. The supplied pages also do not provide a procedure for switching between Balance and Strict, so do not treat the table as a tuning guide.

When you record the result, store the hostname, port, SSL choice, the five URLs you requested, whether each one returned the blocking page, and whether attack-list detail opened for each one. If you compare by requesting the upstream directly, do that from an admin network you control and treat the response as an origin-only observation. Close that direct path to the public Internet afterward. The documented design is that clients pass through the WAF before reaching the server (https://docs.waf.chaitin.com/en/home). A WAF you can bypass is not a WAF you have tested.

Does opening the console mean the WAF is working?

No. The console address https://<safeline-ip>:9443/ is the management page described in the install guide (https://docs.waf.chaitin.com/en/GetStarted/Deploy). Filtering starts only after an application is added and clients reach SafeLine on the configured domain and port (https://docs.waf.chaitin.com/en/GetStarted/AddApplication).

Which requests are the documented way to test blocking?

Use the five samples in the application guide, pointed at your own protected domain: SQL injection, XSS, path traversal, code injection, and XXE. The guide treats the blocking page as evidence that the attack was defended, and it tells you to inspect the attack list and open detail (https://docs.waf.chaitin.com/en/GetStarted/AddApplication).

What should I check if the payload is not blocked?

Check the path before you assume a detection miss. The application must use the hostname you requested, listen on that port, have SSL selected for HTTPS, forward to the intended upstream, and have DNS pointed at the SafeLine server (https://docs.waf.chaitin.com/en/GetStarted/AddApplication). A normal page from the origin means that request did not pass through the cleaning proxy.

Do the sample URLs prove rate limits and bot checks?

No. Rate limiting, CAPTCHA challenge, authentication challenge, and dynamic HTML and JS protection are documented separately (https://docs.waf.chaitin.com/en/home). The published attack exercise does not include steps for those controls, so a blocked XSS URL is not evidence that they are on.

Can the published accuracy table replace a live test?

No. The 33669-sample evaluation reports SafeLine Balance and SafeLine Strict metrics (https://docs.waf.chaitin.com/en/home). It is not a test of your hostname. Keep the live blocking page and the matching attack-list detail as the cutover evidence.

SafeLine has surpassed 700,000 installations worldwide as of September 30, 2026. SafeLine Pro annual promotional pricing is $650-$880, compared with the regular $1,000 per year.

SafeLine website: https://ly.cyberserval.tech/MqhYdmI
Demo: https://demo.waf.chaitin.com:9443/statistics
Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Pro offer: https://cyberserval.tech/landing/lto

Let the Router Own the Next Paint

Sat, 24 Oct 2026 07:06:52 +0000

Navigation bugs are often judged by their final state. Did the correct page appear? Did the authorization check pass? Did the URL settle on the expected address?

Those questions matter, but they can miss the frame that users actually notice.

In a recent component-based web fix, the destination page was correct. The address was correct. The authorization result was correct. Yet, between the address change and the destination render, the departing page painted once more. Because the framework also applied its scroll reset during that interval, users briefly saw stale content at the top of the viewport before the new page arrived.

The repair produced a useful general lesson: after an address change, the router should own the next visible paint.

A correct destination can still have a wrong journey

An end-state test sees a simple sequence:

start on page A;
navigate to page B;
assert that page B is visible.

The browser may have seen something richer:

page A is visible at the user’s current scroll position;
the address changes to B;
page A renders again under B’s address;
a framework scroll reset moves that stale page to the top;
page B finally renders.

Steps three and four may last only one frame. They are still part of the product.

This is why navigation correctness is not only about eventual state. It also includes the ordered render sequence between the user action and the destination becoming authoritative.

The hidden effect of an event receiver

The source of the extra paint was subtle. An asynchronous navigation callback was registered with the component that hosted the route tree as its receiver.

In many component frameworks, completing an event callback schedules a render of the receiving component. That behaviour is normally helpful: event handlers change state, so the framework redraws the owner automatically.

Navigation is different. The callback prepared the transition, but the router—not the departing component—owned the destination. When the receiver-bound callback completed, the framework rendered the old route tree once more. The router rendered the new page immediately afterward, but it was no longer responsible for the first batch after the address changed.

The fix did not add a delay, loading overlay, or animation. It removed the accidental receiver. The callback still ran, but its completion no longer implied that the departing route component needed to render. The router’s destination batch became the first visible paint after the address change.

Measure the interval, not just the answer

The strongest part of the repair was the test shape.

Instead of sleeping and hoping to catch a flicker, the test marked the instant the address changed. It then counted every component render until the destination page entered the render tree.

Before the change, that interval contained twelve render notifications in the exercised path. After the change, it contained zero.

The specific number is less important than the invariant:

Nothing else paints after the address changes and before the destination appears.

That assertion is deterministic. It does not depend on monitor refresh rate, screenshot timing, or a human noticing a flash. It also fails for the causal behaviour rather than a visual approximation of the symptom.

Keeping the old page has a boundary

There is a reasonable usability goal behind retaining the current page: an established screen is often a better placeholder than a blank shell while the destination prepares.

But retained content needs a strict boundary. The old page belongs to the source route and its source authorization decision. It must not start making decisions for the new address, consume state prepared for the destination, or imply that access to the destination has been granted.

Security invalidation is the important exception. If the user signs out or loses a role, the retained page must stop being authoritative immediately. Smoothness must never extend stale permission.

That gives us two complementary rules:

retain a certified source view only until its replacement is ready;
invalidate it immediately when the identity or authorization basis changes.

The result is continuity without authority leakage.

Test the negative space

A good transition test suite should cover more than the successful move between two ordinary pages.

Useful cases include:

an allowed destination;
a refused destination;
a layout or shell change;
sign-out during preparation;
role loss during preparation;
an older asynchronous result completing after a newer transition;
a callback that finishes on a different scheduling context.

These cases reveal who truly owns the transition. If several components can repaint or decide access for the same interval, the design is probably carrying more than one authority.

A practical review method

When reviewing client-side navigation, draw a short timeline: user action, address change, preparation, first destination render, and interaction-ready state.

For each gap, ask:

Which component may render?
Which route does that render belong to?
Which authorization decision supports it?
Can an event callback schedule an incidental redraw?
What happens if identity changes or an older task completes late?

Then write at least one sequence assertion. Final-state tests remain valuable, but they should be paired with a guard over the transition when a single wrong frame matters.

The leadership lesson

Small visual defects often expose unclear ownership rather than missing polish. A loading spinner might hide the flash, but it would not explain why the departing component could repaint under the destination address.

The more durable repair is to assign one owner to the critical interval and test that ownership directly.

Users experience every frame, not only the last one. Our tests should be willing to do the same.

Validate Image Enlargement at the Size Users Will Actually See

Fri, 23 Oct 2026 16:37:50 +0000

Disclosure: I work on upscaleimg.org. These are practical quality checks, not measured performance claims about the service.

An enlarged image can look impressive in a small preview and still fail when placed on a product page. More pixels are not the same thing as more reliable detail. A useful review checks both the output file and its intended display context.

Calculate the required dimensions first

For a web image, start from the largest rendered width and the display density you want to support. A 600 CSS-pixel slot at 2x density calls for roughly 1200 image pixels across. That is a delivery target, not a guarantee of perceived sharpness.

Choose an upscale image workflow only after identifying the source dimensions and crop. Upscaling a wide photo does not solve the problem of needing a portrait composition.

Use paired comparisons

Keep the original and enlarged file. Compare them at the same display size, then inspect output pixels around text, faces, thin lines, and repeated patterns. Look for halos, invented texture, altered letters, and softened edges.

Do not compare a zoomed original with a fit-to-window output and call the difference an accuracy improvement. Record the viewing scale and note which details changed.

Check delivery separately

Record output width, height, format, byte size, and the intended slot. Generate appropriately sized delivery variants after approval rather than serving the largest download everywhere. Preserve the approved master so later compression changes can be reviewed independently.

Upscale Image provides browser-based enlargement and enhancement. The final review is still important when the image contains factual details such as product labels or diagrams. AI-created detail should not be treated as recovered evidence.

For print, calculate pixels per inch using the final crop and physical size, then inspect a proof when practical. A resolution calculation alone cannot certify print quality.

Website: https://upscaleimg.org/

Version Your Image Prompts Before Comparing Generated Results

Fri, 23 Oct 2026 12:41:45 +0000

Disclosure: I work on flux-3.org. This article describes an evaluation process and does not claim official model-vendor affiliation, benchmark results, or access to private model internals.

When an image prompt changes in several places at once, comparing the outputs becomes guesswork. The fix is not a longer prompt. It is a small amount of version control around the creative request.

Keep inputs separate from observations

Store the prompt, reference filenames, selected output format, and interface settings in one record. Store reviewer notes in another. Never edit the old prompt after seeing the result; create a new revision instead.

A minimal naming scheme is campaign-subject-revision-attempt. The revision identifies the input changes; the attempt identifies another run with the same inputs. This distinction matters because a repeated prompt need not produce identical pixels.

Compare one decision at a time

In a browser image workflow such as flux 3, first settle the subject and composition. Then compare lighting instructions. Finally compare style. Keep any available settings constant during each comparison and record settings the interface does not expose as unknown.

Review each candidate against the same checklist: subject count, required objects, camera framing, readable text, edge quality, and unintended additions. Do not collapse these into “best image” until you know which requirements are mandatory.

Make selection reproducible, not generation deterministic

A useful review record explains why an image was chosen. For example: “Revision 3 keeps the full product in frame; revision 2 crops the handle.” That remains useful even if the model cannot regenerate the exact winning file.

Flux 3's browser workspace is one place to try this process. Keep original downloads and use an explicit approval step before publishing. A good-looking preview is not evidence of accurate packaging, text, or rights clearance.

Website: https://flux-3.org/

Separate Subject, Scene, and Style When Testing Image Remix Prompts

Fri, 23 Oct 2026 12:02:58 +0000

Disclosure: I work on whisk-ai.co, an independent image-remix website. This is not an announcement from Google or a claim of affiliation with its products.

Reference-based image generation is easier to debug when each reference has a clear job. A product photo can specify the subject, a room can specify the setting, and an illustration can suggest a visual style. Mixing these roles makes it harder to understand why a result changed.

Define a reference manifest

For each experiment, record the source filename, its role, permission to use it, and the features you want to carry over. A subject reference might contribute shape and color without contributing its original background.

Whisk AI exposes subject, scene, and style reference inputs. Those labels help organize a request; they are not guarantees that every output element will follow the assigned role perfectly.

Run an ablation sequence

Start with the subject alone. Add the scene while holding the subject and text instruction fixed. Then add style. At each step, check whether the main object remains recognizable and whether the camera angle and lighting are coherent.

If the third step introduces an unwanted material change, remove or replace the style reference before rewriting everything. This does not prove how the model works internally. It simply narrows the input changes associated with a visible failure.

Review conflicts explicitly

A flat illustration and a reflective studio photograph can imply conflicting materials. A close-up subject and a wide room reference can imply conflicting scale. Write down which requirement should win before generating another version.

The whisk ai remix workflow can be useful for exploring concepts, but do not treat a reference image as a license to reproduce a third party's work. Review ownership, recognizable people, text, and brand details before sharing outputs.

Keep the manifest beside your selected image so another person can understand the creative decision, not merely repeat an unexplained prompt.

Website: https://whisk-ai.co/

A Regression Checklist for Prompt-Based Image Editing

Fri, 23 Oct 2026 11:10:20 +0000

Disclosure: I work on nanobanana-ai.org. The checklist below is a suggested evaluation method, not a published benchmark or a description of private model internals.

Prompt edits are difficult to test because the expected output is not a single exact file. A byte-for-byte snapshot comparison will reject harmless variations while saying little about whether the important content survived.

Build a small fixture set

Choose images you have permission to use: a product with readable text, a portrait, an object with fine edges, and a scene with repeating texture. Keep the input files unchanged between evaluations. Record the selected model or mode, prompt, aspect ratio, and date when available.

For a browser editor such as nano banana, start with one narrow edit per fixture. Changing background, color, framing, and lighting in the same request makes a failure hard to diagnose.

Test invariants instead of identical pixels

For each fixture, define a short acceptance checklist. A product image might require the same package shape, legible original label, and no invented attachments. A portrait needs identity and expression review. A patterned surface needs checks for repeated seams and distorted perspective.

Separate required checks from preferences. A more attractive result can still fail if it changes a product's identifying details. Log the failed check and keep the output that demonstrates it.

Compare changes fairly

When refining a prompt, keep the source and output format fixed. Repeat a small number of attempts if your budget permits, but report the number of runs and failures instead of presenting one selected image as typical performance.

Nano Banana AI provides image upload and text instructions in the browser. This article does not claim deterministic generation, guaranteed identity preservation, or an official relationship with a model vendor.

A practical release rule is simple: generation creates a candidate; a separate review approves an asset.

Website: https://nanobanana-ai.org/

Treat AI Photo Edits Like Changesets: A Practical Review Workflow

Fri, 23 Oct 2026 11:10:04 +0000

Disclosure: I work on Photo Tea. This is a proposed review workflow, not a claim that the product implements the code or automatically preserves every detail.

An image edit has two requirements: make the requested change and leave unrelated content alone. Checking only the first is how a clean background can hide a changed product label.

Write a small change contract

Before opening an ai photo editor, record three fields: change, preserve, and reject. For a product shot, change might be the background; preserve might include packaging geometry and label text; reject might include invented accessories or new shadows crossing the label.

Keep the original file. Give each attempt its own name rather than overwriting the last result. A useful record is:

{"asset":"bottle-01","revision":2,"change":"light gray background","preserve":["label","cap shape"],"decision":"needs-review"}

This is an example record for your own process, not a Photo Tea API payload.

Inspect the change and the invariants separately

First compare the full composition at the intended display size. Then inspect the label, silhouette, reflections, and contact shadow at full resolution. A successful background replacement does not compensate for an incorrect ingredient label.

Record failures by region, not just as a score. “Cap became wider” gives the next prompt a concrete constraint; “looks bad” does not. Change one instruction per attempt so you can understand which change helped.

Keep approval outside generation

Photo Tea offers prompt-based photo editing. The final publishing decision should still be explicit: approved, rejected, or needs manual repair. File existence and image dimensions are useful automated checks, but neither proves product accuracy.

Use only images you own or may edit, and check privacy requirements before uploading customer photos. For sensitive details, a conventional editor may be the better final step.

Website: https://phototea.art/

Test Video Background Removal Across Motion, Not Just One Frame

Fri, 23 Oct 2026 09:47:09 +0000

Disclosure: I work on videobgremover.org. This is a suggested QA procedure, not a published segmentation benchmark or a description of our private processing implementation.

A background-removal result can have a convincing still frame and distracting flicker during playback. Treating video like a collection of unrelated screenshots misses the temporal defects people notice first.

Choose a representative test segment

Use footage you own or have permission to process. Include a few seconds of stillness, normal movement, and a moment when the subject crosses a contrasting background. Fine hair, fingers, transparent objects, and motion blur deserve separate attention.

Before using a video background remover, keep the original file and record its duration, frame dimensions, frame rate, and audio requirements. These form the acceptance checklist for the export.

Review three views

First watch the result at normal speed. Look for edge shimmer and parts of the subject appearing or disappearing. Then inspect difficult moments frame by frame. Finally composite the result over both light and dark backgrounds to reveal halos that one background may hide.

Record timestamps for failures. “Hair flickers at 00:04” is more actionable than an overall quality score. A checkerboard preview by itself does not prove the downloaded file preserves transparency.

Validate the actual delivery file

Import the export into the editor or player that will consume it. Confirm the expected duration, frame dimensions, audio behavior, and whether the chosen format supports the transparency your workflow needs. A preview and a downloaded file can travel through different conversion paths.

Video Background Remover can be evaluated with this checklist without assuming that every input or export format behaves identically. Reject clips with missing subject regions even when most frames look clean.

Keep a short approved fixture set for future comparisons, with consent and source provenance stored alongside it.

Website: https://videobgremover.org/

Track Reference Images as Dependencies in AI Image Experiments

Fri, 23 Oct 2026 09:47:08 +0000

Disclosure: I work on nano-banana-3.org. This is a workflow recommendation, not an official model-vendor statement or a claim that the website exposes a dependency-tracking API.

When a generated image changes unexpectedly, the prompt is not always the only changed input. Someone may have replaced a reference image, reordered the uploads, or used a different crop. Treat reference files as dependencies so those changes remain visible.

Create an input manifest

For each experiment, list the exact source filenames, upload order, intended role, and permission to use each file. Keep the source files unchanged. For a team workflow, a file hash can help detect accidental replacement; it does not establish ownership or permission.

In nano banana 3, image references and text instructions can be used to explore an image concept. Record the settings visible in your session rather than assuming a particular model version or hidden seed.

Reduce conflicting references

Start with the smallest set that communicates the goal. Two images showing incompatible perspectives can make the desired geometry unclear. If one reference supplies a subject and another supplies a style, say so in the instruction and review whether those roles stayed separate.

When a result fails, replace one input at a time. Keep both the failed output and the corresponding manifest. Otherwise the next person may reproduce the failure without knowing which file introduced it.

Review outputs independently of provenance

A complete manifest explains where inputs came from; it does not prove an output is correct. Check identity, object count, text, geometry, and unintended additions before approval. Review rights and privacy separately from visual quality.

Nano Banana 3 is the browser tool associated with this example. The broader practice works with any reference-driven editor: preserve your inputs, log changes, and make selection decisions explicit.

Website: https://nano-banana-3.org/

A Practical QA Checklist for Browser-Based Song Search

Fri, 23 Oct 2026 07:38:41 +0000

Disclosure: I work on Hum to Search. This is a practical input-quality and review checklist, not an accuracy benchmark or a description of our private recognition backend.

When a user remembers a melody but not a title, a search box is not always enough. A browser-based song search needs a different starting point: determine what kind of evidence the user actually has.

Our song finder presents separate "Hum or sing" and "Identify a recording" modes. That distinction is useful when designing a test plan. A person humming a chorus and a recording of the original song are different inputs; a good interface should not make the user guess which path to choose.

Start with an input checklist

Before searching, answer three questions:

Is this a hummed melody, singing, or a recording of music playing elsewhere?
Is the useful section audible, or is most of the clip silence or conversation?
Can the person record another attempt without replaying the same weak sample?

For humming, the current input screen suggests 10-20 seconds in a quiet place and trying the chorus without background music. Treat that as capture guidance, not a promise of a match. A clear, continuous phrase is a better test input than several disconnected fragments.

For a recording, keep a short section where the music itself is audible. Do not upload unrelated conversations or other people's private audio just to identify background music.

Separate capture problems from recognition problems

A failed result can mean several things. The microphone may not have captured sound, the recording may be unclear, or the supplied evidence may not produce a useful candidate. These should be different states in a product test report.

Here is a small QA matrix a developer can adapt:

Test input	What to inspect	Useful next action
Microphone access declined	Is the reason visible?	Explain how to retry or choose upload
Silent recording	Is there usable audio?	Record again before another search
Hummed chorus	Correct input mode and continuous melody	Compare a different clear phrase
Music with speech over it	Whether music is actually audible	Choose a cleaner section
Ambiguous candidate	Title, artist and familiar melody	Listen and verify rather than accept blindly

This is a proposed test matrix. It does not claim that every browser or every case has passed testing on our site.

Keep a lightweight test log

For a reproducible evaluation, write down the conditions rather than just saying "it worked" or "it failed". A simple CSV is enough:

case_id,input_mode,clip_seconds,environment,expected_song,result,review_note
hum-01,hum,15,quiet,known_test_track,pending,compare chorus
record-01,recording,12,background_speech,known_test_track,pending,try cleaner section

These are example rows, not measured results. Use audio you own or have permission to test. Keep identifying information out of shared logs, and set your own retention policy before keeping recordings.

Change one variable at a time. If the first attempt is noisy, improve the recording before changing the melody, device and duration together. Otherwise you will not know which change helped.

A candidate still needs a human check

A returned title is a candidate to investigate, not a reason to stop thinking. Listen to a legitimate preview and compare the part you remember. Covers, live recordings and similar melodies can make a result feel familiar without being the exact track you wanted.

For product design, "no useful match" deserves its own helpful recovery path. Suggest a clearer phrase or a different input mode instead of repeating a generic error. If the service uses credits, show the current cost before the user submits; avoid promising unlimited retries unless the actual plan supports them.

Try the workflow

Hum to Search is our browser-based starting point for this workflow. Choose the appropriate input mode, provide a clear sample, and review the candidate instead of assuming recognition is certain. Check the live interface and pricing page for current usage conditions.

Website: https://humtosearch.app/

For developers building an audio interface, the practical takeaway is simple: make input type explicit, make failures distinguishable, and make the next attempt easier to understand.

Tolerant Decoding, Strict Release Gates

Fri, 23 Oct 2026 06:38:59 +0000

A release verifier has two jobs that are easy to blur together. It must understand the representation produced by the build chain, and it must enforce the policy that decides whether an artifact may leave that chain.

Those jobs need different attitudes.

Decoding can be narrowly tolerant of documented, meaning-preserving variation. Release policy should remain exact and fail closed.

That distinction surfaced in a recent mobile build repair. A generated JSON settings file could begin with the optional UTF-8 byte-order marker. The JSON content was valid, but the verifier decoded it as plain UTF-8 and failed before it could evaluate the values that actually governed release safety.

The fix was not to “clean up” the input generally. It was to accept one specific encoding variation, then run the same strict semantic checks as before.

Representation is not policy

A byte-order marker sits at the representation boundary. In this case, accepting UTF-8 with or without that marker did not change the JSON values.

An endpoint or feature-handoff setting is different. It changes what the package will do after installation. That is policy-bearing data.

Treating both categories as raw bytes creates a brittle gate: a harmless encoding detail can block a valid package. Treating both categories as broadly normalisable creates a permissive gate: a meaningful difference may be trimmed, defaulted, coerced, or discarded before verification sees it.

The useful middle ground is narrow tolerance:

Decode only the representation variants the producer is documented to emit.
Parse into an explicit data shape.
Compare policy-bearing values exactly.
Publish nothing until every check has passed.

This is a small design choice, but it keeps compatibility and authority in their proper layers.

Why broad normalisation is risky

“Normalise before comparing” sounds sensible. Sometimes it is. The danger is that normalisation is often written without naming which differences are harmless.

Should surrounding whitespace be trimmed? Should an empty string become null? Should a missing value receive a default? Should a hostname lose its trailing separator? Should key casing be ignored?

Each transformation might be correct for one contract and wrong for another. Once a release gate performs all-purpose cleanup, reviewers can no longer see which deviations the gate intentionally accepts.

A better question is: who owns this variation?

The decoder owns permitted encoding markers. The JSON parser owns JSON syntax. The release policy owns the approved endpoint and capability settings. Package signing owns artifact identity. Device acceptance owns behaviour on the target platform.

Let each boundary make only the decision it has enough information to make.

Test tolerance and strictness together

A positive test for the newly accepted variant is necessary, but it is not enough. It proves compatibility; it does not prove that policy remained intact.

The stronger test set combines the representation variant with both approved and disallowed semantics:

a marker-bearing file with approved values passes;
the verifier emits a clean, parseable receipt;
a marker-bearing file with an unapproved endpoint fails;
a marker-bearing file with an unapproved handoff configuration fails;
either failure produces no distributable package and no success receipt.

Those negative cases are the important counterweight. They show that the change expanded the accepted representation, not the accepted deployment policy.

The no-output assertions also matter. A verifier that reports failure after copying the package has already lost the race. Publication must happen after verification, not alongside it. The absence of a receipt is useful too: downstream automation must not mistake a partially completed run for approval.

Make the artifact boundary transactional

File-based pipelines rarely have a database transaction around the whole release. We can still design them with transactional thinking.

Build into a private staging location. Verify the package, its embedded settings, its signature, and any required metadata there. Only after the full decision succeeds should the pipeline copy or promote the artifact and write the corresponding receipt.

On failure, the public output location should remain unchanged.

This ordering turns the output directory into a simple commit boundary. Before promotion, the candidate is merely work in progress. After promotion, the package and its evidence agree.

A practical review checklist

When one tool generates a file and another tool gates a release from it, review four things:

1. Enumerate representation variants

Record the encodings, markers, line endings, ordering guarantees, and optional fields the producer may legitimately emit. Avoid accepting variants merely because a parser can.

2. Identify policy-bearing fields

List the values that change destination, authority, enabled capabilities, signing identity, or runtime behaviour. Compare them deliberately rather than through a generic object cleanup step.

3. Add crossed tests

Test each permitted representation with approved values and with at least one disallowed value. This catches the accidental move from narrow compatibility to broad permissiveness.

4. Prove fail-closed publication

Assert not only that verification returns failure, but also that no package, receipt, or other success signal appears in the publish location.

The leadership lesson

Strictness is not the same as rejecting every variation. Good gates are precise about what they tolerate and equally precise about what they refuse.

That precision makes a pipeline easier to operate. Engineers can fix harmless toolchain differences without weakening release policy. Reviewers can see the exact compatibility allowance. Operators can trust that a failed gate did not leave an ambiguous artifact behind.

The goal is not a gate that never bends. It is a gate that bends only where meaning is unchanged—and stays immovable where deployment meaning begins.

Nothing Reported It

Thu, 22 Oct 2026 13:59:00 +0000

Five ways my own test harness lied to me, and the one that nearly got published

The number that gave it away was sitting in a table I had already read three times.

prompt size    97 tokens median, 112 max

That is the grounded question-answering workflow. It sends four reference documents with every question. The documents are 447 words.

Four hundred and forty-seven words cannot be ninety-seven tokens.

I had written that number into a results document as good news about context headroom. "Prompt sizes never approached the 4096-token context window." It was not good news about headroom. It was the harness telling me, in the only field capable of telling me, that the reference material was not in the prompt. The directory holding it had never been committed to the repository. Every one of those fifteen questions had been asked against an empty reference block.

Both models had behaved impeccably. One had been instructed to answer NOT IN THE MATERIAL when the material did not contain the answer, and the material contained nothing, so it said NOT IN THE MATERIAL thirteen times out of fifteen. The other had no such instruction, so it answered from whatever it had memorized during training and invented a wine.

Every call returned 200. The scorer scored it. The scores were internally consistent, had plausible and different failure modes on each side, and I wrote two thousand seven hundred words about what they revealed regarding the capabilities of small language models.

They revealed nothing about small language models. I had measured what happens when you ask a question with no reference material, which is a question I was not asking and which has an uninteresting answer.

This is the post about that, and about the four other things I found in the same weekend, because the missing directory was not the first one and it was not even the most instructive.

The experiment, briefly

A Raspberry Pi 4, two small models through Ollama, and three workflows: pull an identifier out of a message, route a message to one of six queues, answer a question from four documents. Ninety-five test cases. A deterministic baseline for the first workflow. Predictions committed before anything was installed.

The findings are a separate post. This one is about the code I wrote to produce them, which turned out to be where the actual bugs were.

Five flaws. Every single one was found by reading output, not by reading code. I never once found one of these by looking at the scorer.

Kind one: a metric that rewards the safe failure

This one I hit twice, in two different workflows, and did not recognize the second instance as the same bug until I had already fixed the first.

The classification scorer. Six labels, one of which is escalate. The instruction said to escalate when a message needs a person, including when you cannot tell what it is about. So I built a metric called confident wrong: wrong, and not an escalation. The reasoning is sound. A model that escalates when unsure is usable. A model that routes a trademark notice to billing is not.

Then the 360M model answered escalate to 43 of 50 messages.

It scored 3 on confident wrong. The 1.5B model, which was genuinely attempting the task and getting 46% of it right, scored 26.

My metric had just certified the useless model as roughly six times safer than the mediocre one. Because escalations are exempt, a model that escalates always cannot register a confident error. The exemption I added to reward caution had instead rewarded paralysis, and it did so in the direction that looks best.

The grounded QA scorer. Four of the fifteen questions have no answer in the corpus. Correctly saying so is the behavior that matters most, because a system that invents a plausible answer with no evidence is worse than one that fails loudly: it is indistinguishable from one that worked. So the headline number was correct abstentions out of four.

Then the 1.5B model said NOT IN THE MATERIAL to 13 of 15 questions.

It scored 4 out of 4. A perfect score, on the metric I cared most about, for a model that had answered almost nothing.

Same bug. Different workflow. Two hours apart. I fixed the classification one, wrote a comment in the code explaining the principle, and then failed to apply that principle to the file I opened next.

The shape, stated once: a metric that rewards the safe behavior will certify any model that only ever does the safe thing. If you exempt a behavior because it is cautious, you have to pair that exemption with evidence that the model is making distinctions at all. Otherwise, the exemption is the score.

Both fixes are the same three lines: count how concentrated the answers are, and when one answer dominates, print a warning that the headline number is meaningless and say why.

Kind two: a metric that measures nothing at all

The vacuous fixture. I caught this one before the run, which is the only reason it is not a bigger part of this post. A test case that passed regardless of what the model did, because the fixture was inert either way. It contributed a tick to a column and no information.

The keyword check, which is worse, because it survived longer and looked more useful.

The eleven answerable QA questions each carry a list of words the correct answer should contain. The scorer checks whether any of them appear anywhere in the reply. I labeled it "indicative only" and moved on.

On the invalid run it scored 1 of 11 for both models. The single hit was the word "reservation", appearing inside this answer:

You owe a tasting fee of $100 for the weekend. You also need to book a reservation for the weekend.

The correct fee is forty dollars. The hundred is invented. That one hit was the entire positive signal in my automated QA scoring and it was noise.

Here is the part that makes it a better story. After I fixed the corpus and reran, the keyword check scored 11 of 11 for the 1.5B model. Vindicated, apparently. I read all eleven answers by hand anyway, and nine of them are right.

So the check was wrong both times. It was wrong in the pessimistic direction when the answers were bad and wrong in the optimistic direction when they were good, and on the rerun one of its eleven hits is an answer that gets the arithmetic wrong and hits on the word "reservation" anyway.

A check that agrees with the truth because the truth happened to be agreeable has not been validated. It has been lucky.

I wrote a decay check because of a thermal problem. A Pi 4 with heat sinks and no fan hit its soft temperature limit during a thirty-call run and started capping the clock, and nothing anywhere said so. So the scorer now compares generation speed in the first third of a run to the last third and flags a slowdown.

It then reported this, as steady:

across the run 4.3 -> 7.6 tok/s (176% of starting speed, steady)

The hardware got seventy-six percent faster partway through a run. My detector, which was looking for numbers below a threshold, saw a number above it and reported no problem.

Hardware does not speed up. That reading is either contention at the start of the run or, as it turned out, generations too short to time. Either way it is a signal that the measurement is unreliable, and I had built a detector that would only ever notice half of the ways that can happen.

Anything you threshold in one direction, threshold in both. An implausibly good number is as informative as an implausibly bad one and nobody ever writes the check for it.

Kind four: errors that get dropped instead of counted

Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty. The traceback was useless because urllib raises HTTPError and discards the response body, which is where Ollama puts its reason. Reading e.read() gave it up immediately:

{"error":"prediction aborted, token repeat limit reached"}

The 360M model, asked to find an order number in "Thanks for a lovely afternoon yesterday," where the correct answer is one word, had fallen into a repetition loop and been killed after twenty-one seconds. Reproducible at temperature zero. The input causes it.

My first fix was to record failures and keep going, which was right. My second was to exclude all failures from the scores as infrastructure noise, which was badly wrong.

That call was not infrastructure noise. The model ran. It received the prompt, generated tokens, and produced nothing usable. That is a result, and excluding it would have dropped from the denominator the single case this model demonstrably could not handle, scoring it only on the cases it survived.

So the runner now tags every failure as one of two kinds:

Model failure. The call reached a working model and the model failed. Stays in the denominator. Counts against the model. Reported on its own line as "no usable answer."
Transport failure. The call never reached a working model. Ollama was down, the tag was wrong, the request timed out. Nothing was measured, so it is excluded and the run is reported as incomplete.

The distinction is a five-line change and it is the difference between measuring a model and measuring a model on the subset of inputs it happens to tolerate.

Kind five: inputs that go missing without raising

This is the missing corpus from the top of the post, and it is a different and worse category than everything above it.

The function was four lines:

def corpus_text():
    parts = []
    for p in sorted((FIX / "corpus").glob("*.md")):
        parts.append(p.read_text(encoding="utf-8").strip())
    return "\n\n".join(parts)

A glob that matches nothing yields nothing. An empty list joins to the empty string. No exception, no warning, no exit code. It returns a perfectly valid value that happens to be the absence of the thing it was asked for.

And on the same evening, a sibling. The offline test's entire product is a text transcript. I copied it into the run directory, committed, pushed, and it was not there. The standard Python .gitignore template, which I had pulled in wholesale, carries *.log on line 165, and a pattern without a leading slash matches at any depth.

The copy succeeded. git status came back clean. The push reported success. The one artifact the test exists to produce was gone, and everything in the chain said fine.

Absence does not raise. A missing directory, a missing file and an inert fixture all present as an ordinary successful run. That is narrower and more useful than "test your tests," and it generalizes: every input a harness depends on needs a loader that refuses to proceed without it, and every artifact it produces needs to be verified as present rather than inferred from an exit status.

The loader now looks like this, and the comment is longer than the code because the comment is the part I will need next year:

paths = sorted(corpus_dir.glob("*.md"))
if not paths:
    raise SystemExit(
        f"No corpus documents found in {corpus_dir}\n"
        "Grounded QA cannot run without reference material, and running\n"
        "it against an empty reference block produces numbers that look\n"
        "like a result."
    )

It also refuses a corpus implausibly smaller than the real one, the run prints the document count and token estimate before its first call, and the scorer independently flags any QA run whose median prompt is under 400 tokens. Three checks for one bug, in the input, in the run, and in the output, because this is the bug that cost a day.

The detector that fired, and was right, and told me the wrong thing

Here is the part I find hardest to write and most worth keeping.

The abstention-collapse check from kind one did fire on the invalid run. It printed exactly this:

!! ABSTAINED on 13/15 questions overall. It is refusing, not judging.
   The abstention score below is not evidence of discrimination:
   a model that never answers gets it right for free.

It was correct. Something was badly wrong with that run.

Its diagnosis was completely false. The model was not refusing. It was obeying, precisely, an instruction to report when the material lacks the answer, in a situation where the material lacked everything. The fault was not in the model's judgment, and it was not in the metric. It was in a directory that did not exist, two layers upstream.

And I acted on the diagnosis. I wrote it up: the model has a stuck needle, its perfect abstention score is an artifact of refusal, here are the implications for anyone building on a 1.5B model. Confident, specific, sourced to real output, and wrong in every particular.

A flag that tells you something is off is not a flag that tells you what. The check could only see the output. The fault was in the input. It fired for the right reason and pointed in the wrong direction, and because it fired I felt more confident in a conclusion that was false, not less.

That is the failure mode I had no defense against and still only partly do. A monitoring system that is right about the existence of a problem and wrong about its cause is more dangerous than no monitoring, because it terminates the search.

What I actually changed

For anyone who wants the diffs rather than the moral:

Collapse checks on both classification and QA: when one answer exceeds half the set, print the concentration and declare the headline metric meaningless for that model.
The decay check flags implausible speedups as well as slowdowns, and notes when generations are too short to time.
Failures split into model and transport, with model failures staying in the denominator.
corpus_text() raises on an empty or implausibly small corpus.
The QA run prints its corpus size before the first call.
The scorer flags a QA run whose prompts are too small to contain the corpus.
The keyword check is relabelled as a weak lower bound, in the output, in the README, and in the results.
.gitignore no longer lets *.log swallow run transcripts.
vcgencmd get_throttled before and after every run, recorded in the run document rather than checked and forgotten. The reading reports bits set since boot, so it is perishable: power-cycle the board and the evidence is gone.

The uncomfortable part

Every one of these was found by looking at data. Not one was found by reading code, reviewing the scorer, or writing a unit test for the harness.

The vacuous fixture turned up while hand-checking a passing case. The confident-wrong flaw turned up because a score of 3 against 26 looked backwards. The abstention flaw turned up because 4 out of 4 looked too good. The keyword flaw turned up because 1 of 11 looked too bad, and I read the one. The missing corpus turned up because I was double-checking a latency figure and noticed a prompt that was too small to be possible.

Which means the only reason I found them is that I kept pulling threads after I had numbers that looked publishable. The version of this project that shipped on schedule shipped on Tuesday, with a clean verdict table, a confident conclusion about small models and grounded question answering, and a repository anyone could clone to reproduce the exact same wrong result.

Every number in it would have been reproducible. That is the thing I keep turning over. Reproducibility is table stakes and it would not have saved me, because the harness reproduced its own bug faithfully every single time.

I do not have a process that catches this reliably. What I have is a weaker and more honest claim: the checks I added are each a direct response to one specific thing that actually happened, which is the only kind of check I trust, and the number of them that I thought up in advance is zero.

So the two rows I am adding to my prediction template, to be filled in before anything runs:

What would have to be true for these numbers to be meaningless?
Would anything in the harness tell me if it were?

Twenty-two predictions about what the models would do. Not one about what the measuring code would do. Four of the five flaws were in the measuring code.

The harness, the fixtures, the committed predictions and the full run record, including the invalid run and what it produced, are at
github.com/kenwalger/right-size-the-model.
The findings the harness eventually produced are in the companion post.

Ambiguous, Invalid, or System Error?

Thu, 22 Oct 2026 11:37:02 +0000

In my previous post, I wrote about the problems I found while defining the categories for an AI intake classifier. I left ambiguous and invalid for another post.

Those two states turned out to be only part of the problem. While wiring the workflow, I also needed to distinguish a request the model could not classify from a request the model never managed to evaluate.

This is the same practice project for a fictional automation agency, built in n8n with an AI collaborator. It is not a production client system. The workflow and project notes are in the GitHub repository.

Three different reasons for not choosing a project category

The classifier had three normal project categories:

simple_automation
custom_integration
ai_workflow

But not every submission should be forced into one of them.

ambiguous means the request is usable, but the text supports more than one category. The workflow should not pretend that a guess is a confident classification. In my sharper overlap test, I deliberately included concrete signals for both ai_workflow and custom_integration. The classifier returned ambiguous.

invalid means the input does not provide enough usable project intent to assign a normal category. This does not necessarily mean the person has no genuine request. A vague but relevant message can still contain partial intent. On the other hand, an out-of-scope message may contain nothing useful to route.

system_error means classification did not complete. The workflow did not reach a decision about the request, so treating it as invalid would report something the system did not know.

The difference matters because each state calls for a different response. A real request with overlapping categories may need a person to decide. An out-of-scope submission may not need a client email. A system failure needs troubleshooting or a retry, not a judgment about the request itself.

`invalid` needed one more field

My first definition of invalid treated missing information as the main signal. That overlapped with ambiguous, because both could describe a submission that could not be assigned cleanly. I changed the distinction: ambiguous is for multiple categories supported by the text; invalid is for input that does not support a responsible category assignment.

Then I noticed that one invalid label was still hiding two different situations:

The person has a real project in mind, but the request is too vague to classify yet.
The submission is unrelated or out of scope, with no useful project intent to follow up on.

I could have tried routing by keywords in the model's reasoning text, but that would make the workflow depend on the exact wording of an explanation. I also considered a conditional schema. Because the result passed through n8n's Structured Output Parser, I did not want to assume that conditional-schema behavior would be preserved through its translation layer.

I chose a simpler, always-present boolean: has_partial_intent.

{
  "category": "invalid",
  "has_partial_intent": true
}

That keeps invalid as the category and makes the next decision explicit. It is easier for a downstream node to route on a boolean than to interpret free-text reasoning.

I tested it with three small examples:

help automating my workflow returned has_partial_intent: true. The request was vague, but it did point to a real automation need.
fix my car returned false. It was a request, but not one this intake workflow was built to handle.
I love football also returned false as unrelated input.

These examples helped check the distinction I had defined. They are not a representative dataset, and they do not tell me how often the classifier would get this right with real submissions.

A model failure is not an invalid submission

I also needed system_error for cases where classification itself failed. A network interruption or a broken model credential should not make the workflow label a valid request as invalid. The system has not made a decision about the content at all.

I tested the failure path by temporarily breaking the Gemini credentials. The workflow retried classification three times at five-second intervals, then reached the fallback route for human review. I also checked that the original submission data remained available on that path.

The ambiguous and system_error routes shared some downstream handling, but I kept the category values separate. The reason for sending something to human review is important: in one case the model made a decision that needs human judgment; in the other, it did not complete the decision.

That distinction affected the client-facing message too. For a system failure, it would have been misleading to send the same confirmation as if a normal classification had succeeded. The workflow needed to acknowledge the issue without claiming the request was invalid.

Why I used explicit routing

I had considered using an AI Agent to choose how these outcomes should be handled. I did not need another model-driven decision, though. The possible states were already known, and the workflow's job was to route each one consistently.

I used explicit Switch and IF branches instead. That made the expected paths visible in the workflow and easier to test. For the invalid case with has_partial_intent: false, I confirmed that no email was sent while the submission was still logged to Google Sheets. Logging did not depend on sending a client message.

What I can claim from these tests

I exercised a deliberately ambiguous request, vague and out-of-scope inputs, and a forced model failure. I also checked the fallback path and verified that the invalid/no-partial-intent case was logged without triggering emails.

That is evidence that these particular paths were exercised in this build. It is not proof of general classifier accuracy or production reliability. The examples were written by me, and a larger, independently labelled set of realistic submissions would be needed to measure that properly.

The useful lesson for me was that a fallback is not just a place to send everything that does not match a normal category. The workflow needs to preserve why it did not choose a category: competing interpretations, insufficient or irrelevant intent, or a failure to classify at all.

An Empty Field Is Still Signed Data

Thu, 22 Oct 2026 06:11:15 +0000

An empty form field looks unimportant. In ordinary application logic, treating an empty value like a missing value is often reasonable. In signature verification, that assumption can reject a legitimate message—or, in a different protocol, verify a representation the sender never signed.

The important lesson is simple: canonicalization is not generic data cleanup. It is part of the cryptographic protocol.

The tidy abstraction that hid a real difference

Consider an integration with two flows:

The application builds an outbound form and signs it.
A provider later sends an inbound form notification that the application verifies.

Both flows need URL encoding, ordered key=value pairs, a digest, and a comparison. A shared helper therefore seems sensible.

But suppose the protocol defines different empty-field rules. The outbound form omits empty values. The inbound notification includes every posted field, even an empty one represented as description=. Reusing the outbound rule during verification silently removes part of the inbound canonical representation.

Nothing is wrong with the network message. Nothing is wrong with the secret. The application simply hashes different bytes from the sender.

That is the dangerous kind of bug: the code is deterministic, the crypto primitive behaves correctly, and the abstraction looks clean. The failure lives in the representation contract.

Empty and absent are different inputs

Application code tends to reason semantically:

note= means there is no useful note.
no note field also means there is no useful note.

A signing algorithm reasons syntactically. It may see:

reference=abc&note=&status=complete

as a different message from:

reference=abc&status=complete

The same applies to field order, repeated keys, character encoding, spaces, escaping, decimal formatting, and line endings. Two representations can produce the same business meaning while producing different signatures.

That is why “normalize before verifying” is unsafe advice unless the protocol defines the normalization precisely. The verifier’s job is not to improve the input. Its job is to reconstruct exactly what the other side signed.

The repair pattern I prefer keeps the truly common pieces together while making direction-specific rules visible.

Shared mechanics may include:

the exact form-encoding routine;
digest calculation;
secret or passphrase handling;
fixed-time signature comparison;
consistent fail-closed behaviour.

The canonicalization policies remain separate:

outbound signing can omit empty fields when the outbound specification says so;
inbound verification can retain empty fields when the inbound specification says so;
both can exclude the signature field itself;
both can preserve their required ordering.

This introduces a small amount of repetition. That is a healthy trade when the alternative is a boolean option buried inside a generic helper whose callers must remember invisible protocol rules. If a shared primitive is retained, give the public operations names and documentation that make the distinction impossible to miss.

Test the representation, not your implementation

Signature tests often contain a subtle circularity: the test calls the production signing helper to create the expected signature, then asks the same helper to verify it. The test proves internal consistency, not compatibility with the protocol.

A stronger test builds the canonical input independently. For the inbound case, start from a literal ordered field sequence containing several key= entries. Calculate the expected digest with a small test-only oracle or a published fixture. Then submit the body to the real verification path.

Useful negative cases include:

remove one empty field and expect rejection;
change field order and expect rejection when order is significant;
replace an empty field with an absent field;
vary spaces and reserved characters;
include a blank expected signature and fail closed;
change only the signature field and confirm it is excluded from canonical input.

These tests describe the protocol boundary directly. They are harder for a refactor to accidentally “update” into agreeing with the bug.

Exercise the framework pipeline too

A pure canonicalization test is necessary, but it is not sufficient for a web endpoint.

Framework model binding and form parsing can consume a request stream, decode characters, merge values, or present fields through an API with different ordering behaviour. Middleware can also affect buffering or body access. A verifier that works against a handcrafted dictionary may still fail when the real HTTP pipeline handles the request first.

Add one focused integration test that:

creates the literal encoded body;
independently calculates the signature over that exact representation;
sends an actual form POST through the application’s HTTP stack;
verifies that the original body remains available where required;
proves the request reaches the normal accepted outcome.

This is not an argument for turning every edge case into a slow end-to-end test. Keep the combinatorial matrix in fast unit tests. Use the HTTP-level case as a boundary witness that the framework and your canonicalization contract still agree.

Verification is necessary, not complete trust

Correct signature verification answers a narrow question: does this message match the shared signing contract and secret?

It does not automatically prove that the event is fresh, unique, authorized for the referenced resource, or safe to apply twice. Production handlers may still need replay protection, timestamp or nonce validation, amount and ownership checks, idempotency, audit evidence, and careful failure handling.

Keeping that claim bounded matters. Strong security engineering is usually a chain of explicit checks, not one impressive cryptographic step.

A practical review checklist

When reviewing a signed integration, ask:

What exact bytes or characters are signed in each direction?
Are empty and missing values distinct?
Is field order defined and preserved?
When does decoding or encoding happen?
Are repeated fields possible?
Does the comparison avoid timing leaks?
Do invalid inputs fail closed?
Is at least one expected signature calculated independently?
Does a real HTTP test cover framework parsing?

The broader engineering principle is worth carrying elsewhere: an abstraction should share sameness, not erase difference. At a cryptographic boundary, the awkward representational details are often the contract itself.

Flip One Field: A Fairness Test That Beats Aggregate Parity

Wed, 21 Oct 2026 21:04:43 +0000

Most teams check an LLM-assisted decision system for fairness by comparing approval rates across groups. That check can pass cleanly while the model is still changing its mind on individual cases for reasons nobody would defend out loud. Recent fairness-stress work on LLM decision-making points at a better default test - and it's a decision you can make before writing any code.

Aggregate Parity Nets Out; Per-Case Instability Doesn't

The choice is between two measurements. Outcome parity asks: across 1,000 applications, did group A get approved at roughly the same rate as group B? Counterfactual stress asks something narrower and harder: for this one application, if I change only a decision-irrelevant field - the name, the phrasing of the same request, the order of the bullet points - does the answer move?

A model that favors one group on early-career cases and another on senior cases can show perfect parity overall. Parity is cheap and it's what regulators often ask for, but it averages. Per-case stress testing catches that, because it never aggregates across people - it compares a case to itself.

The trade-off is real. Stress testing costs N× the inference calls, and it requires someone to declare which fields are decision-irrelevant. That list is a policy judgment, not an engineering one, which is exactly why a PM should define it.

Running the Test on One Application

base = {"years_exp": 6, "skills": "python, sql"}
names = ["Alex Chen", "Aisha Khan", "John Smith", "Priya Patel"]

votes = [decide(tmpl, {**base, "name": n})
 for n in names
 for tmpl in PROMPT_TEMPLATES] # 3 paraphrases, same instruction

flip_rate = 1 - max(votes.count(v) for v in set(votes)) / len(votes)

Twelve variants of one application that a human reviewer would call identical. Any flip_rate above zero means the decision moved on something that shouldn't matter.

The number alone isn't the point - the pattern is. Read the flips two ways:

Clustered - all rejections land on the same name group across all three paraphrases. That's bias. Fix it in the prompt, the retrieval data, or the model choice.
Scattered - flips appear randomly across names and templates. That's instability. Lower the temperature, constrain the output format, or stop auto-deciding borderline cases.

Both fail the same test and need different fixes, which is why one aggregate percentage can't tell you what to do next. Report worst-case flip rate per application, not the mean - the mean hides the cases that would end up in a complaint.

Key Takeaways

Group-level parity can look clean while individual decisions flip on irrelevant details; counterfactual stress testing compares each case against itself and catches what averages hide.
Separate clustered flips (bias - fix prompt, data, or model) from scattered flips (instability - fix sampling, output constraints, or add human review).
The hard part isn't the code, it's agreeing on which input fields are decision-irrelevant. That list belongs to product and policy, not just engineering.

Which fields in your decision prompt would your team actually agree are irrelevant - and how long would that meeting take?

Sources referenced: FairStress: A deeper evaluation of fairness in LLM decision-making, Hugging Face Blog

Build a Python agent harness you can extend and test without calling a model

Wed, 21 Oct 2026 12:50:26 +0000

A team ships a small Python agent with two tools. A month later someone renames an argument on one tool function and forgets the hand-written JSON schema sitting a few lines above it. The model keeps sending the old argument name, the function raises on every call, and nobody finds out until a real task hits that tool, because the only way anyone ever tested the loop was by running it against a live model.

The layout caused this one. When tool contracts are written by hand and the loop only runs end to end, every change to a tool risks the whole agent, and every test costs an API call. A harness you can live with keeps three things apart: the loop, the tool contracts, and the tests that prove the loop behaves.

The loop has three parts, and only one of them should change when you add a tool

A minimal agent harness in Python has a loop over the model's stop_reason, a dispatcher that maps a tool name to a Python function, and a messages list that grows every turn. The loop keeps running while the model asks for tools (stop_reason == "tool_use") and returns the text answer when it sends end_turn.

Of those three, only the set of tools should grow as the agent gains abilities. The loop itself should not change when you add a tool, and the message list should follow the same rules on every turn. A production harness wraps more around this core, such as sandboxing, permission checks, context compaction and session state. Those layers are easier to add later when the core stays this small.

Generate the schema from the function's input type

Every tool needs a name, a description and an input_schema that tells the model which arguments it accepts. Writing that JSON Schema by hand is how the opening bug happens. The schema and the function live in two places, and only one of them gets edited.

Describe each tool's inputs with a Pydantic model and generate the schema from it:

One model per tool. A ReadFile model with a single path: str field describes the input of read_file.
Let the library write the schema. ReadFile.model_json_schema() returns a plain dict with path as a required string, which is the input_schema shape the Anthropic API validates against.
Edit the type and the schema follows. Add a field or mark one optional and the generated schema updates itself. This method ships with Pydantic v2.

The Pydantic model becomes the single source of truth for a tool's inputs, and the function's arguments have to match its fields. The dispatch fixture in the testing section below fails the moment the two drift apart, long before a real task finds it.

Dispatch by name so the loop never names a tool

Keep a dict called HANDLERS that maps each tool name to its function, and dispatch with one line: HANDLERS[block.name](**block.input). The loop reads the tool name from the model's tool_use block and never refers to a specific tool.

That one decision is what makes the harness extendable. Adding a tool touches three places, and none of them is the loop:

Define the input model. A WriteFile model with path and content fields.
Add one entry to the tool list. Name, description, and WriteFile.model_json_schema() as the schema.
Register the function. HANDLERS["write_file"] = write_file.

The agent can now write files as well as read them, and run_agent did not change by a single character. If adding a tool ever forces an edit inside the loop, a tool name has leaked into it.

Treat what comes back as part of the contract

The input side is only half of a tool's contract. The output has rules too. The content of a tool_result must be a string or a list of content blocks. A read_file that returns a string passes straight through, while a list_files that returns a Python list has to be serialized first, for example with json.dumps.

Put that rule in the loop once, so no single tool has to remember it. The same goes for pairing: copy the id from each tool_use block into the result's tool_use_id, and append the assistant message before the user message that carries the results. These are loop rules, and they belong next to the dispatch line, where every tool gets them for free.

Test the loop without calling a model

Testing a harness by running it against a live model and reading the output costs money, takes seconds per run, and only covers the path the model happened to take that time.

The loop is ordinary Python, so test it like ordinary Python. Mock the messages.create call to return fixtures, and assert on what the loop does with them:

A tool-use fixture drives one turn. Return a response with stop_reason set to tool_use and a fake tool_use block, then check that the matching handler ran with the arguments in the block.
The append order holds. After that turn, the assistant message should sit before the user message that carries the result, and the result's tool_use_id should match the block's id.
Parallel calls arrive together. Return a fixture with two tool_use blocks and assert that both results go back in a single user message before the next call.
Non-string results get serialized. Point a fake tool at a list return and check that the content the loop sends back is a string.
The loop exits on end_turn. Follow the tool turn with an end_turn fixture and check that run_agent returns its text.
The ceiling fires. Feed tool_use fixtures forever and check that the loop raises once it reaches max_turns, instead of spinning.

One detail matters when you read the mock's recorded calls: run_agent keeps appending to the same messages list, so copy the list inside the mock if you want to assert on what each call saw. These tests run in milliseconds and never reach the API, and they cover the rules that break a harness without anyone noticing. Run them on every change, and keep the live-model run for checking behavior that only a model can show.

Know what a framework would take over

The OpenAI Agents SDK runs the same loop for you through Runner.run. You describe tools as Python functions with type annotations, and the SDK raises MaxTurnsExceeded when a run passes its max_turns, which is the same safety valve as the manual counter.

Reach for the SDK when you need multi-agent handoffs, built-in guardrails or tracing. Keep the manual loop when you want zero framework dependency or need to see exactly what happens on every turn. OpenAI Swarm is an experimental, stateless project, and the Agents SDK has superseded it for anything you plan to run in production.

Either way, the design rule carries over. Tool definitions stay separate from loop logic, and switching to a framework later holds no surprises in the control flow.

Pitfalls that keep showing up

Hand-written schema dicts. They drift from the function the first time someone edits a signature.
Tool names inside the loop. Every new tool turns into a change to the control flow.
Raw Python objects in tool_result. The API expects a string or content blocks, so a returned list fails validation.
A missing max_tokens. It is required on every Anthropic request and has no default, so the SDK raises before the model ever runs.
Live-model runs as the only test. They are slow, they cost money, and they never force the edge cases.

The full program, with the tool definitions, the loop and the stop conditions in one file you can run, is in the build guide with the full code.

A loop that never names a tool, schemas generated from types and tests that need no model call together keep a harness small. Every new tool after that is a few lines next to the others.

Test the Owner, Not the Teardown

Tue, 20 Oct 2026 17:41:25 +0000

Flaky lifecycle tests often produce a tempting diagnosis: the cleanup code must be unreliable. Sometimes that is true. Sometimes the test is asking a framework harness to prove a narrower ownership contract, and the harness is free to finish work on a different schedule.

That distinction matters in interactive server-side UI systems. A component may subscribe to live updates, start a timer, register an observer, or attach a callback. The resource needs a clear owner and a reliable release path. But the test for that contract should not depend entirely on whether a whole rendered tree has completed teardown at one particular instant.

The shape of the failure

Consider a header component that subscribes to a process-local notification service. A child pop-up also owns a separate subscription. The component test renders the tree, publishes an update, observes the visual change, disposes the test context, and immediately expects the global subscription count to be zero.

That sounds reasonable. It is also broader than the claim we really want to prove.

Under a full test-suite load, the notification can overlap a render. The harness teardown call may return while the renderer still has a queued batch. At that instant the header might not yet have received its disposal callback. The non-zero count is therefore ambiguous: either the header leaked, or the harness has not disposed it yet.

A test with two possible explanations is a weak regression guard.

Waiting is not the same as proving

The first instinct is usually to add a retry window: wait for the subscription count to reach zero, perhaps for a few seconds.

That is appropriate when the product contract is genuinely eventual. It is not a universal cure for lifecycle tests. If the harness has left the component alive, waiting merely spends more time observing the wrong boundary. The test remains dependent on incidental scheduling and can still fail without a cleanup defect.

The key question is not, “How long should we wait?” It is, “Which object owns the resource, and what exact behaviour are we trying to prove?”

Put the assertion at the ownership boundary

The stronger test calls the owning component's disposal contract directly, on the correct renderer context, and checks only the resource that component owns.

If the header owns one subscription and a still-rendered child owns another, disposing the header should reduce the count by one. Expecting zero would incorrectly make the header test responsible for the child's lifetime as well.

That narrower assertion is not weaker. It is more precise:

the owner releases its resource;
an independently owned resource is not silently claimed by the parent test;
the result does not depend on whole-tree teardown timing; and
removing the owner's cleanup call makes the test fail consistently.

That last point is especially valuable. A small mutation check—temporarily disabling the cleanup call—shows whether the test detects the regression it claims to detect. If the test still passes, it is probably measuring something else.

Production code still needs lifecycle discipline

A focused test does not excuse fragile implementation. Cleanup should normally be idempotent because framework callbacks, navigation, error handling, and explicit shutdown can converge on the same release path.

The object should also refuse to acquire new resources after disposal. An asynchronous load may resume after the component has gone away. If its continuation subscribes again, a perfectly correct earlier disposal becomes irrelevant because nothing will release the new subscription.

The same reasoning applies beyond subscriptions. Failed in-flight loads should close or abandon temporary buffers. Retained arrivals should have a sensible bound. Work dispatched to another recipient should not accidentally carry ambient request state from the publisher. These are all lifecycle questions: who owns the state, when does ownership end, and what context is allowed to cross the boundary?

Keep two layers of evidence

Direct ownership tests and integration tests answer different questions.

An owner-level test asks: when this component is disposed, does it release what it owns, and can it remain released?

An integration test asks: does navigation, circuit shutdown, or framework teardown actually invoke the right lifecycle path across the rendered tree?

Both matter. The mistake is using only the second, broad test as proof of the first, narrow contract. When the broad test fails, you cannot tell whether ownership or orchestration is broken.

Separating the layers improves diagnosis. A direct test failure points toward the owner. An integration-only failure points toward wiring, scheduling, or framework interaction. The test suite becomes a map of contracts instead of a collection of timing guesses.

A practical checklist

For each subscription, timer, observer, or callback:

Name the owner explicitly.
Make release safe to call more than once.
Prevent acquisition after the owner is disposed.
Test the owner's release contract directly.
Give child-owned resources separate coverage.
Keep one integration test for the framework path.
Prove the test's sensitivity by breaking cleanup locally and seeing it fail.

The trade-off is a few more focused tests and slightly more explicit lifecycle code. In return, failures become actionable. You know whether the owner leaked, the child still legitimately owns something, or the framework has not completed teardown.

Reliable tests do not merely wait longer. They make ownership visible.

One Bash Script to Audit a Polyglot Codebase (Python, TS, Java, Go, C, HTML)

Mon, 19 Oct 2026 11:43:05 +0000

If your project mixes languages, one terminal command can run each language's own tests and security checks and fail the build when anything is red.

The script (1) counts files per language, (2) runs build and type checks, (3) runs tests and SAST, (4) scans dependencies and secrets, (5) runs a quick concurrency check, and (6) writes a checksummed report.

I tested it on a small demo project: python compile passed, missing scanners showed as SKIP, an invalid HTML file produced a real FAIL, and the run exited with code 1.

Full guide with per-language settings, a troubleshooting table and a CI example: https://www.seosiri.com/2026/10/polyglot-code-audit-bash-testing-pipeline.html#conclusion

Which scanner would you add first?

Can a Keyboard-Only User Finish Your App’s Main Task?

Mon, 19 Oct 2026 07:41:47 +0000

Pick the main task in your web app: submitting a form, uploading a file, booking an appointment, or creating a project.

Now put the mouse aside and try to finish it using the keyboard.

Where does the journey first become confusing or impossible?

Some useful places to check:

A custom dropdown that opens but cannot be operated
A dialog that opens without a clear focus position
A validation error that appears somewhere the user does not notice
A drag-and-drop interaction with no usable alternative
A focus indicator that disappears against the background
A component you can enter but cannot leave

W3C’s keyboard accessibility guidance explains why functionality needs a keyboard-operable route, with a limited exception for inherently path-dependent input. Its no-keyboard-trap guidance addresses being able to leave a component again.

A keyboard walkthrough is a useful first check. It does not establish full accessibility or replace screen-reader and other accessibility testing.

What is one keyboard-accessibility bug your team found late, and what changed in your development or review process afterward?

If you are building something new, which critical flow will you test first?

Waqas Ahmad | Software Engineer | Content Creator

Is your coding agent reading the memory you wrote?

Sun, 18 Oct 2026 21:21:32 +0000

I maintain Selvedge. This article was prepared with AI assistance and checked against the released Selvedge v0.3.17 package from PyPI (tag v0.3.17, commit 3b67b8f) and the repository's main branch at commit 66c51d3.

I saved one decision, read it back six ways, and got nothing from two of them. The record never moved. The reads that missed it had opened a different, empty database.

This is a reproduction, not an incident report. I installed Selvedge v0.3.17 from PyPI in a fresh virtual environment, pointed HOME at a throwaway directory, and logged one invented decision from a project root:

selvedge init
selvedge log src/billing/charge.py::retry_charge reject \
  --reasoning "Rejected automatic retries inside retry_charge; went with a single attempt plus an idempotency key, because a retry after a timeout can charge a customer twice." \
  --constraint "A payment request must never be submitted twice for one order." \
  --stale-when "the payment provider offers server-side idempotent retries" \
  --agent synthetic-demo

(--agent only sets an attribution label.) Then I asked for that record from different places.

One record, six reads

From src/billing, a subdirectory of the project, selvedge prior-attempts src/billing/charge.py::retry_charge --json returned the record.

From an unrelated directory, the same lookup (without --json) printed "No prior attempts found" and exited 0. By the end of the run, the throwaway home directory held a new per-user database with zero events.

From that same unrelated directory with SELVEDGE_DB=/path/to/demo-app/.selvedge/selvedge.db set, the command returned the record with the same ID.

With the same variable set, the Python package returned the same ID:

from selvedge import SelvedgeStorage, get_db_path

store = SelvedgeStorage(get_db_path(read_only=True))
blame = store.get_blame("src/billing/charge.py::retry_charge")
print(blame["id"] if blame else "not found in this store")

Finally, a small test script started selvedge-server over stdio and called its prior_attempts tool. With SELVEDGE_DB set, it returned the same ID. Without it, from the unrelated directory, it returned an empty list.

Every read that found the database returned one ID. The two misses weren't missing data; they were the wrong file.

Why the empty answer happened

Selvedge stores decisions in local SQLite. In v0.3.17, the CLI and the MCP server both find that database with get_db_path, which the package also exports. It checks three places, in order:

the SELVEDGE_DB environment variable;
an existing .selvedge/selvedge.db file, searched for upward from the current directory;
the per-user fallback ~/.selvedge/selvedge.db.

The subdirectory read found the project file by searching upward. The unrelated directory had no project file above it, so the read fell through to the per-user path.

The tool didn't hide this. A normal lookup that falls back writes a warning to stderr, once per process, unless SELVEDGE_QUIET is set. Both the CLI miss and the MCP miss printed it. On its own, get_db_path(read_only=True) stays silent and creates nothing. But opening that path with SelvedgeStorage, as the snippet above does, still creates the per-user directory and an empty database, with no warning. In a fresh home directory without SELVEDGE_DB, the snippet printed "not found in this store". selvedge doctor reports the resolved path and which step matched: "via walked up from CWD to project .selvedge/" from the project root, and "via global fallback (~/.selvedge/)" with status WARN from the unrelated directory. That doctor run still exited 0, so a script that checks only exit codes won't notice.

An MCP server makes this easier to miss. A client starts the server with whatever working directory and environment the client chooses, and whether you see the server's stderr depends on the client. The README says to set SELVEDGE_DB in the server configuration when the client starts outside your project.

Checking your own setup, with or without Selvedge

None of this is specific to one tool. Any file-based memory, whether a decisions folder, a JSON file or a SQLite database, is located by each process for itself. So, before trusting an empty answer, find out how your tool picks its file. Usually it's some mix of explicit path, project discovery and a per-user default.

Then ask the question from where the reader actually runs. A check in your terminal describes your terminal, not an agent client or a CI job with its own directory and environment. For processes you don't launch yourself, set the path explicitly rather than relying on discovery.

When a lookup comes back empty, read it as "nothing in this store," not "nothing was ever decided." And once, after any change to clients or configuration, write a test record through one route and read it through the others, comparing IDs rather than text. Matching text could be a copy; a matching ID means the same record.

What's on main, and not released

PR #76, merged into main as commit 66c51d3 on October 17, 2026 (PT), reworks Selvedge's architecture plan. It describes MCP as "one route to the core, not a prerequisite for using it," and plans a small local core shared by the library, CLI, MCP and optional integrations. The PR also adds a regression check that saves and retrieves a record across isolated Python processes, and rewords the bundled prompt to present CLI and MCP as routes to the same store.

None of this is in v0.3.17, and the architecture document says the phases add no release dates. It notes that the current package still installs MCP dependencies, and that making them optional needs a separate compatibility assessment. It lists compact review presentation, native GitHub validation and an editor integration as future work.

What this run doesn't show

It's synthetic: one invented record, one machine, Python 3.13 and a throwaway home directory. The CLI's PyPI update check was turned off, and telemetry was never enabled.
No agent client was tested. The MCP route used a test script built on the mcp Python client, so check your own client's configuration, launch directory and activation separately.
Selvedge stores what a person or agent explicitly saves. It doesn't verify that the reasoning is true, and it doesn't guarantee that an agent will consult it.
Pinning the path makes routes agree on a file. It doesn't create records nobody saved.

Would an independent final-email rule check earn a place beside your existing tests?

Sun, 18 Oct 2026 11:12:47 +0000

I’m cun he, maker of InlinerAPI. If you maintain business receipts, reset messages or invitations, I’d appreciate one current-work comparison: https://inlinerapi.netlify.app/requirements

Use synthetic final rendered HTML and an independently expected copy, displayed amount or CTA destination. The browser preview needs no signup. It checks the supplied content; it does not refresh application data, run your templates, validate tokens, or test inbox appearance or delivery.

What rule did you attempt, what result did you observe, and how did setup and maintenance compare with your existing tests? If it helps recurring work, what would your team need to justify paying for these current checks?

What evidence should an AI coding assistant provide before saying a bug is fixed?

Sun, 18 Oct 2026 10:06:10 +0000

An AI coding assistant can produce a plausible patch and a confident explanation in the same response. What evidence should accompany that explanation before a team treats the issue as resolved?

A useful baseline might include:

A reproduction that fails before the change and passes afterward
The exact tests or commands run, with their outcomes
An explanation connecting the change to the reported failure
A clear statement of anything that could not be checked

The difficult part is deciding how much evidence is enough. A typo in a static label and a change to payment processing deserve different levels of verification. Passing tests also leave an important question: do those tests cover the behavior the user actually reported?

For a timing bug, a happy-path test seems especially weak. A stronger check would deliberately reverse the order of events, delay a response, or force a failure at the point where the bug appears.

Would you require a regression test for every AI-generated bug fix, or allow a documented manual check for smaller changes? And should assistants distinguish between “patch prepared,” “automated checks passed,” and “verified in the target environment” instead of compressing all three into “fixed”?

Curious where you would set the minimum bar, especially when the assistant cannot access the production environment.

Selvedge 0.3.17: test your agent memory before you rely on it

Sat, 17 Oct 2026 16:10:57 +0000

I maintain Selvedge. This article was prepared with AI assistance and checked against the released v0.3.17 documentation.

A coding agent can tell you it saved a decision. The useful test comes later: close the session, start again, and retrieve the same record. Can you recover the reason you rejected an approach, or just a convincing explanation of the code that remains?

Selvedge gives compatible AI agents persistent decision memory through a local MCP server or the CLI. You save a choice, its reason, and the alternatives you ruled out against a named part of your project: a function, database column, API route, or dependency. Those records live in SQLite under .selvedge/, so another session can query the same history.

Version 0.3.17 makes two parts of that workflow easier to check: whether the local MCP server can retrieve a decision after restarting, and whether the Markdown digest you bring to review still matches the stored history.

Test the local MCP server

With uv installed, you can try the first check without connecting an agent or touching a project's decisions:

uv tool install --upgrade selvedge==0.3.17
selvedge demo --mcp

The command starts a real local stdio MCP server, saves a synthetic rejected approach through it, shuts that server down, and starts another. It then checks that retrieval returns the same record ID and content. Add --json to get the checked IDs and version information in machine-readable form.

The data is disposable. The demo uses a temporary database, skips background update and telemetry checks, and has a deadline followed by subprocess cleanup. That makes it a useful first step when evaluating the package: a small, concrete persistence check with no invented project history left behind.

It also has a precise boundary. A passing result checks the Python installation running Selvedge and that local MCP exchange. Your editor may launch a different executable, point at a different database, or require you to enable tools. The demo cannot establish that your agent's configuration or lifecycle hooks are active.

Try one real decision across two sessions

The next test belongs in your own workflow. Follow the quickstart, connect your agent, and ask it to save one real decision with the reason and any rejected alternative. Keep the entity path and record ID. Then open a fresh session in the same project and ask it to retrieve that decision. Compare the result with what you saved.

That is also where to test usefulness. Suppose you kept a dependency because its replacement lacked a feature you needed. Retrieving that constraint gives the next session something specific to check before attempting the replacement again. Only explicitly saved decisions can be recovered, and recorded reasoning still needs judgment: Selvedge stores the account supplied by you or your agent; it cannot establish that the account is correct or complete.

Check the digest before review

The second addition addresses what happens when those records reach a review. Selvedge already exports a deterministic Markdown digest. Once you have project history, generate it with:

selvedge export --format markdown -o DECISIONS.md

Version 0.3.17 adds a check you can run before review or in CI wherever that same history is available:

selvedge export --format markdown -o DECISIONS.md --check

If the file matches, the command exits successfully without output. If it is stale or missing, it exits with status 1 and a unified diff. The check preserves both the digest and the stored records; it does not initialize or migrate the database. Read failures produce guidance, and invalid arguments return status 2.

This gives a reviewer a straightforward consistency check. A digest can look reasonable while omitting a recently recorded decision. The check makes that drift visible. It does not validate the decisions themselves or fill gaps in capture. Review the exported reasoning before sharing or committing it.

There are smaller fixes with the same aim of making the record easier to interpret. Human-readable output now distinguishes an approach considered and rejected from a change tried and reverted. Literal bracket characters in logged text no longer cause the affected CLI rendering failures. The release adds no runtime dependencies, migrations, or MCP tools.

If you use a coding agent on a project you regularly return to, try one decision across two sessions. A local stdio MCP connection works; an agent with shell access can use the CLI. No hosted account is required, and core storage and retrieval make no LLM calls. The repository has the source and release notes. Feedback about where saving or retrieving that first real decision breaks is especially useful—share it in the existing trial discussion.

Three audio checks worth running before you ship any upload feature

Fri, 16 Oct 2026 13:08:03 +0000

Most "the audio is broken" reports are not DSP bugs. They are format and metadata bugs that surface weeks later, after the feature has already shipped. These three checks are cheap and catch the majority of them.

1. Read the header, not the extension

A .wav can be 8-bit PCM, 24-bit float, or something a recorder invented. If you branch on the file extension you will eventually hand a decoder something it silently mangles. Parse the actual header first, then decide what to do with it.

2. Sample rate mismatches are silent

Drop a 44.1 kHz source into a 48 kHz graph and nothing throws. It just plays back at the wrong speed and pitch. Normalise on ingest, then assert the rate at every stage boundary. An assertion costs nothing; a support ticket does not.

3. Peak level is not loudness

A track can peak at 0 dBFS and still sound quiet next to its neighbours. If your UI says "loud" or "quiet", you are measuring loudness (LUFS), not peaks. Pick one metric, name it in the UI, and stop guessing which one you meant.

None of this needs a DSP background. It needs one explicit place in the pipeline where the invariant is written down and checked.

These three came straight out of debugging the upload path on Creatune AI Music Generator, which I work on — so treat the recommendation with the usual scepticism.

Queued Work Is a Snapshot, Not the Truth

Fri, 16 Oct 2026 05:43:57 +0000

Background work is usually introduced to make a system more reliable. We persist an intent, let the request finish, and deliver the external side effect later. That protects the user-facing path from temporary provider failures and gives us room for retries.

It also creates a time gap.

Inside that gap, the world can change. A task can be completed. A new task can appear. A recipient can lose access. A preference can change. Another worker can advance the same operation. The queued record may still be durable and validly serialized while the message it would produce is already wrong.

The lesson is simple: queued intent is evidence about an earlier snapshot. It is not timeless authority.

Store the promise, not the presentation

Suppose a service groups several outstanding items into one reminder. The tempting design is to serialize the titles, amounts, names, and finished message into an outbox row. Delivery is then easy: deserialize and send.

That convenience turns the row into a stale cache of user-facing truth. It may also retain more sensitive data than the queue needs.

A safer durable record can be much smaller. Keep the routing identity, recipient identity, sequence, source, and a deterministic fingerprint of the complete set the message was created for. The fingerprint represents the promise: “this message describes exactly this observed set.”

The display payload is rebuilt later from authoritative state. The durable row says what snapshot was intended without pretending that its old presentation is still current.

Rebuild from the complete current set

At execution time, re-read what the authoritative source currently exposes to the recipient. Then run the same grouping rules again and compare the resulting fingerprint with the stored fingerprint.

Why the complete set? Because stale work changes in two directions.

The obvious case is subtraction. An item was completed or removed after the reminder was queued. The old message now overstates what is outstanding.

The less obvious case is addition. A new item appeared after the reminder was queued. If the worker checks only the item keys remembered by the row, every remembered item may still be valid. The worker can still send a message that understates the current request.

Reconstructing the whole set catches both. If the current fingerprint differs, the old intent is superseded. Stop it. Let the current state produce a new intent under a new identity.

Keep authorization in the authoritative read

The background worker should not invent a second, simplified version of the authorization and status rules used by the foreground feature. That duplicate will drift.

Prefer a shared underlying read path with an explicit recipient parameter. The interactive request and the delayed worker should agree on what the recipient is currently allowed to see and how current status is mapped.

This also makes an important distinction possible. “Nothing remains” is not the same as “the recipient is no longer authorized.” Both may yield an empty visible set, but they mean different things operationally and in an audit trail.

Completion can finish the old intent. Revocation should stop disclosure for a security reason. A missing source integration is different again: it is usually a retryable deployment fault, not evidence that the work is complete.

Model more than send or fail

A robust recheck tends to produce several outcomes:

Send: the complete current snapshot still matches, preferences allow delivery, and the operation sequence is current.
Stop as completed: the intended work no longer exists.
Stop as changed: the set still exists but is no longer the set the queued message promised to describe.
Stop as revoked: the recipient no longer has authority.
Stop as superseded: a later sequence has already advanced the operation.
Defer: authoritative evaluation is temporarily unavailable.

These outcomes can feel verbose compared with a Boolean. That verbosity is useful. It prevents a temporary infrastructure gap from being filed as success, and it prevents a security revocation from being misreported as ordinary completion.

Test addition as carefully as subtraction

The most valuable regression test in this pattern is often the one teams forget: add something after the row is written.

A focused suite should cover an unchanged set, a removed item, an added item, an empty set, revoked authority, changed preferences, a superseded sequence, and an unavailable source. It should also prove that each evaluation performs a fresh authoritative read rather than accidentally reusing stored data.

These tests do more than exercise branches. They define the temporal contract of the workflow: which facts may age, which facts must be refreshed, and which outcome owns each kind of change.

The trade-off is real

This design adds a database read at the most sensitive point, deterministic reconstruction, fingerprint maintenance, and more explicit state transitions. At high volume, that cost deserves measurement. Poorly indexed rechecks can simply move a reliability problem into the database.

The benefit is not that fingerprints make distributed systems magically consistent. They do not. There is still a race between the final read and the external side effect, and systems with stricter requirements may need provider idempotency, transactional boundaries, or domain-specific reservations.

The narrower claim is valuable enough: a delayed worker should not knowingly act on an obsolete snapshot when it can cheaply revalidate current truth.

A practical review checklist

When reviewing delayed email, push, webhook, export, billing, or approval work, ask:

What exact snapshot did this row promise to act on?
Does the queue store minimal identity or stale presentation data?
Can the worker re-read the full authoritative set?
Are additions detected as well as removals?
Are completion, revocation, supersession, and temporary unavailability distinct?
Is the external side effect itself idempotent or otherwise bounded?

Durability keeps work from disappearing. Revalidation keeps durable work from becoming confidently wrong.

Isolation Is Sometimes a Feature, Not a Smell

Thu, 15 Oct 2026 06:01:03 +0000

Two projects expose similar helpers. One has no project references. It is missing from the solution most developers build every day. The obvious reaction is understandable: share the code, add the reference, and bring the lonely project back into the fold.

That cleanup can be exactly wrong.

A recent architecture correction I reviewed was a useful reminder that repository shape is not the whole system. Build agents, installed workloads, runtime origins, signing steps, and release pipelines are part of the architecture too. An isolated .NET MAUI head looked neglected when viewed only from the solution file. Viewed from its release path, the isolation was load-bearing.

Start With the Delivery Graph

Before deduplicating across project boundaries, ask how each project is actually built and shipped.

In the case I reviewed, a server-focused solution deliberately excluded the mobile head. That kept routine server builds from requiring mobile workloads on every developer machine and CI runner. The mobile app was still real software: a separate pipeline restored, built, signed, and released it independently.

Adding a shared project reference would therefore do more than remove a few repeated lines. It would expand the mobile restore graph and couple its release path to a project the mobile pipeline had never needed. Adding the mobile project to the server solution would impose a different cost in the other direction: machines building the server product would now need tooling they did not otherwise use.

Neither consequence is visible in a tidy dependency diagram. Both matter in production delivery.

Similar Code Can Encode Different Assumptions

The same review contained two link resolvers that appeared ripe for consolidation. The browser implementation derived an external base address from its hosting origin. That was valid because the browser app was served by the web host.

The mobile app ran inside an embedded web view. Its internal origin did not identify the external service it needed to reach. Reusing the browser resolver would not create elegant sharing; it would import an invalid environment assumption.

This is a useful test for any proposed abstraction: are the implementations merely similar in syntax, or are they equivalent in authority and context?

Imagine a browser client and a desktop client that both produce a download URL. The browser can safely resolve a relative path against the page origin. The desktop client may need an explicitly configured service endpoint. A common interface can still be useful, but a common implementation may be actively misleading.

Duplication is not automatically good. It is sometimes the smallest honest representation of two different worlds.

Guard the Constraint—and Its Reason

Documentation corrected the misleading story, but the stronger improvement was an architecture test suite.

The tests did not merely assert that the mobile head had no project references and stayed outside the server solution. They also checked that the separate mobile build path still existed. That second half changes the character of the rule.

Without it, the test says: “keep this isolated forever.”

With it, the test says: “keep this isolated while this independent delivery path remains true.”

That is a healthier architectural guard. It protects a decision without pretending the decision is timeless. If the separate pipeline disappears, the guard fails and asks maintainers to reconsider the boundary. The test becomes executable rationale rather than fossilized preference.

This pattern generalizes well. If a component must stay dependency-free because it is loaded by a restricted host, test both the dependency rule and the existence of that host. If a migration assembly must remain separate because tooling discovers it independently, guard both the assembly boundary and the discovery path.

The Trade-Off Is Still Real

Deliberate isolation has costs.

Separate helpers can drift. A visual shell copied for a second head may receive fixes later. Architecture tests add maintenance, and source-shape checks can become brittle if they assert incidental text instead of meaningful structure.

The answer is not to ignore those costs. It is to compare them with the coupling being avoided.

Prefer the smallest duplication that preserves the boundary. Keep behavioural contracts aligned where the behaviour truly is shared. Document the assumptions that must differ. Review the boundary when release topology changes. And avoid broad claims such as “this project shares nothing” when the important truth is more precise: it ships independently and intentionally owns a narrow set of platform-specific choices.

A Practical Review Checklist

Before merging apparently duplicate code across application heads, ask:

Which pipeline builds and releases each head?
What workloads and SDKs does each build environment guarantee?
Will a new reference expand the restore or signing boundary?
Do the implementations rely on different origins, storage, identity, or lifecycle rules?
Can we test the constraint and the continuing fact that justifies it?
What event should trigger reconsideration of the decision?

These questions turn “DRY versus duplication” into a more useful discussion about operational ownership.

Architecture Includes the Way Software Leaves the Repository

The most valuable lesson for me was a leadership one: do not label unusual structure before tracing its history and delivery path.

An isolated project may be forgotten. It may also be a carefully bounded product whose release assumptions are invisible from the main solution. We should investigate before simplifying, explain the decision in language that cannot be mistaken for neglect, and encode the rationale where future changes will encounter it.

Clean architecture is not the fewest projects or the fewest repeated lines. It is a system whose dependencies match the realities of how it runs and ships—and whose exceptions are specific, reviewable, and allowed to expire.

A Passing Test Can Still Prove Nothing

Wed, 14 Oct 2026 06:12:33 +0000

Some of the most misleading tests are not flaky, slow, or obviously incomplete. They are fast, deterministic, and green.

They also pass for the wrong reason.

That is especially dangerous in fault-tolerant code. A test may claim that a failed dependency is contained, a fallback runs, or an established workflow continues. Yet its assertion may observe an outcome that can occur both before and after the recovery boundary. The suite reports confidence while the behaviour named by the test remains unproved.

The false-positive shape

Imagine a dispatcher with an optional policy source in front of an established workflow. The intended behaviour is straightforward:

Ask the policy source whether the old workflow should stand down.
If that optional source fails, contain the fault.
Continue into the established downstream path.

The unit test configures a policy source that throws. It also uses a strict downstream test double that throws if reached, because the rest of that workflow is outside the test's concern.

Then it asserts only that dispatch throws an exception.

At first glance, this can feel reasonable. An exception from the strict downstream dependency appears to prove that execution continued. But the same broad assertion also accepts the earlier exception from the policy source. If the recovery logic is deleted, the first exception simply escapes and the test remains green.

The assertion recognizes the shape of the final outcome, not the path that produced it.

Test the claim, not the punctuation

The claim is not “an exception occurs.” The claim is “a fault at this boundary does not stop progress into the established path.”

That distinction changes what the test must observe.

A useful test needs a stable witness on the far side of the boundary. Depending on the design, that witness could be:

a recorded call to the downstream collaborator;
a domain result that only the fallback path can produce;
a distinct exception identity from a deliberately strict dependency;
a state transition committed after the failed operation; or
an emitted event that occurs only after recovery has completed.

The specific technique matters less than the causal guarantee. The witness must be impossible—or at least deliberately difficult—to observe if execution stops at the failure being contained.

In the example above, the corrected test distinguishes the downstream failure from the policy-source failure. It therefore passes only when execution crosses the recovery boundary. A recording test double would also work and may be clearer when the downstream call itself is the contract.

Why mutation testing helped

Code coverage could not expose this weakness. Both the recovery code and the test might execute during the normal run. Coverage tells us which lines ran; it does not tell us whether an assertion depended on their behaviour.

Mutation testing asks a sharper question: if a meaningful piece of behaviour is changed or removed, does a test fail?

Removing the recovery behaviour created a surviving mutant. That was valuable evidence. It did not necessarily mean the production design was wrong. It meant the suite could not distinguish the intended implementation from a broken one.

This is one of mutation testing's best uses: finding assertions that observe something adjacent to the claim rather than the claim itself.

You do not need mutation tooling to apply the same reasoning in review. Mentally invert the behaviour:

If the catch disappeared, would the test fail?
If the fallback were skipped, would the test fail?
If execution stopped one step early, would the test fail?
Could a different branch produce the same asserted result?

If the answer is uncomfortable, look for a witness beyond the boundary.

The trade-off: path awareness without brittleness

Path-sensitive assertions carry a real cost. A test that verifies every internal call order can become coupled to implementation detail and obstruct harmless refactoring. Replacing one collaborator with another should not require rewriting an entire suite if the observable contract is unchanged.

The answer is not to avoid path awareness altogether. It is to choose the smallest stable witness that represents the behaviour promised by the test.

Prefer domain-visible evidence when it exists: a returned decision, a committed state, a durable message, or a recorded recovery status. Use collaborator verification when crossing that collaborator boundary is itself the contract. Use exception type or identity when a strict test double intentionally marks progress, but make that intent explicit so a future reader understands why the exception matters.

This produces a useful hierarchy:

Assert the externally meaningful result when it uniquely proves recovery.
Assert a durable checkpoint when the final result is produced asynchronously.
Assert a boundary interaction when progress across that boundary is the behaviour under test.
Use a distinct failure identity as a narrow witness when a strict dependency deliberately terminates the test.

The closer the witness is to the real reliability contract, the less brittle the test will be.

A practical review habit

When reading a test name, underline the verb: continues, retries, falls back, persists, publishes, or rejects. Then find the assertion that proves that verb happened.

If you can only find evidence that something threw, returned, or logged, ask whether the same evidence could appear before the named behaviour occurred.

Green is not the standard. Discrimination is. A useful test must distinguish the system we intended from a nearby broken version.

For fault-tolerance tests in particular, do not merely observe the aftermath of a failure. Prove that the system crossed the boundary it promised to survive.

MCP connected, hooks inactive: lessons from five coding-agent adapters

Wed, 14 Oct 2026 00:48:27 +0000

Originally published on DEV on September 24, 2026. This cross-post preserves that article’s version-specific implementation and test results.

An agent can successfully call an MCP server while its lifecycle hooks never run. The server connection and the hook process have separate configuration, execution environments, and trust checks.

That distinction mattered while extending Selvedge's native hooks beyond Claude Code in version 0.3.15. The release adds adapters for Codex, Cursor, Gemini CLI, VS Code Copilot's Local harness, and Windsurf/Cascade. The useful engineering lesson is how to separate shared behavior from each client's protocol—and how to prove the integration actually ran.

Disclosure: I maintain Selvedge. This article was drafted with AI assistance and checked against the released implementation and recorded test evidence.

Keep the decision logic separate from the protocol

The feature in this example checks whether an agent has consulted recorded decisions before a supported edit to a watched schema or migration path. A saved rejection might explain that a column cannot be removed because an importer still needs it.

The adapter has three jobs:

native client payload
  -> normalize project, session, tool and affected paths
  -> evaluate the shared decision check
  -> encode the result in the client's response format

The shared check shouldn't need to know whether a client calls its shell tool Bash, Shell, or run_shell_command. Conversely, the response encoder must know which fields—or exit status—the client recognizes.

The released adapters make these mappings explicit. For example, the Codex apply_patch payload supplies patch text in tool_input.command; the VS Code Local apply_patch payload uses tool_input.input. Treating those tools as interchangeable would lose the files the check needs to inspect.

Similar event names don't imply identical configuration

These are the project files and pre-edit events used by the five new adapters:

Client target	Project hook configuration	Event used for the edit check
Codex	.codex/hooks.json	PreToolUse
Cursor	.cursor/hooks.json	preToolUse
VS Code Copilot, Local	.github/hooks/selvedge.json	PreToolUse
Gemini CLI	.gemini/settings.json	BeforeTool
Windsurf/Cascade	.windsurf/hooks.json, or existing .devin/hooks.json	pre_write_code, pre_run_command

The table describes this implementation's supported targets, not every harness sold under those product names. In particular, VS Code Local support doesn't establish support for Copilot CLI, cloud sessions, or Agent Host.

Configuration shape also differs: copying the same event name into each JSON file is insufficient. Installers need to validate existing structures, preserve unrelated hooks, back up modified files, and surface conflicts rather than silently replacing customized commands.

Project identity deserves the same care. If a multi-root workspace payload doesn't identify an unambiguous current directory, checking another repository's decisions would be worse than reporting that the check couldn't be applied.

A notification isn't saved context

All five new adapters implement the watched-edit check. Four also provide a startup digest and a user notification before compaction. The Windsurf adapter provides neither startup context nor a compaction notification.

Those are different capabilities:

A startup digest supplies selected saved context at a supported startup event.
A pre-edit check can delay a supported tool call until the lookup requirement is satisfied.
A compaction notification tells the user something. It doesn't make the model save a decision, and it isn't model-context injection.

Calling all three “memory preservation” hides the distinction users need when debugging. Explicitly recorded rationale is also different from a model's hidden reasoning; these adapters don't inspect hidden reasoning or read chat transcripts.

Test installation, invocation, and behavior separately

A generated hook file proves configuration was written. A successful direct invocation proves a script accepts a particular payload. Neither proves the client will launch it.

For a real-client test, use a disposable project and inspect this sequence:

Seed one synthetic rejection against a watched file.
Review and enable the hooks through the client's normal trust controls.
Attempt the relevant edit before retrieving history; inspect the actual denial.
Retrieve the saved decision, retry, and inspect the final file.
Test an ordinary source edit and a read operation independently.

The recorded native test used Codex CLI 0.154.0-alpha.6.2. It observed an apply_patch denial, a real prior-attempts CLI lookup, then an allowed retry with the resulting file checked. It used normal hook trust and an absolute executable path with a local payload recorder. Earlier isolated runs that ignored user configuration never invoked the project hooks; they were excluded from the successful result.

The other four new clients have automated payload, configuration, SQLite, and subprocess coverage, but haven't been verified end to end in those clients. Startup and compaction behavior also remains contract-tested. That's the difference between implementing an adapter and observing it work in a particular client version.

State the limits of the check

This check covers known mutating tool schemas and configured watched paths. Shell write detection is heuristic. Unknown payloads, missing project databases, and internal evaluation errors fail open, so it is workflow assistance rather than a security boundary.

The existing lookup window can also be satisfied by a recent query from another session. An allowed edit therefore doesn't prove this particular agent understood the decision—or that the old constraint still applies. Retrieval, correct application, and a successful code change need separate evidence.

The next diagnostic improvement is to report configuration and observed hook execution separately. A single “connected” status cannot establish both.

Protocol references: Codex, Cursor, VS Code Local, Gemini CLI, and Cascade. The contracts cited by this implementation were checked September 24, 2026; client versions can change them.

Do You Actually Need an LLM?

Tue, 13 Oct 2026 14:17:00 +0000

I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control.

The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took an afternoon to write. The better of the two models scored twenty-eight, took five seconds a call, and the two it missed are both mistakes a regex is structurally incapable of making.

That is the kind of result I was looking for, and the question behind it is one I keep avoiding: what is the smallest amount of computation that reliably does this job? Not the cheapest model. Not the best small model. The smallest thing that works, which means the ladder has four rungs and the bottom one is not a model at all.

Deterministic code. A regex, a lookup table, an if statement.
A small local model. Runs on hardware you already own.
A larger local model. Needs a GPU.
A remote frontier model. Someone else's hardware, metered.

Most projects start at rung four and never ask about the others. This is a weekend spent asking.

The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a hundred and seventy dollars most of which I already owned. The models were smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are ninety-five questions about a fictional small winery, so nothing can be looked up and every answer can be judged by reading. Sixteen of the fifty classification messages are deliberately ambiguous, because real inboxes are mostly ambiguous.

Four verdicts, assigned per workflow, to force a result you can act on instead of a number:

Unnecessary: deterministic code already does this.
Sufficient: a small local model does it well enough to run unattended.
Marginal: usable with a person checking the output.
Insufficient: the error profile or the latency is not acceptable.

And one rule, which is the part I would insist on if you copy nothing else: I wrote predictions for every workflow and committed them before I installed Ollama. Twenty-two of them, each marked confident, leaning, or coin flip. Once there is a timestamped expectation there is a defined moment when you are done, and no room to decide afterward that you knew all along. I will come back to how they scored, because it is not flattering.

Extraction: pulling an order number out of a message

Find ORD- followed by six digits, or report that there isn't one. Thirty cases: twenty-two with an identifier, five without, and three decoys. The decoys are a membership number, a UPS tracking number, and a phone number, all of which look like identifiers and none of which are order numbers.

	regex	qwen2.5:1.5b	smollm2:360m
Overall	30/30	28/30	8/30
Declined correctly when absent (8)	8	8	0
Decoys ignored (3)	3	3	0
Invented an answer	0	0	7

The regex won. Thirty out of thirty, in zero milliseconds, at zero watts above idle.

That result is boring until you read the two the best model missed.

On ORD-999000: wrong address, please hold. it answered 999000, dropping the prefix. Annoying, recoverable, the kind of thing you fix with a post-processing step you should not need.

On ORD-111222 please refund it answered NONE.

That is the shortest and least decorated input in the whole set. Almost the entire message is the order number. And the model returned a confident, correctly formatted assertion that there is no order number present. In a pipeline, that is a refund request for an order that exists arriving at a system that has decided it does not.

A regex cannot make that mistake. Not "rarely does" — cannot. The two errors separating 28 from 30 are both of a class that the deterministic baseline is structurally incapable of producing, which is a different and much stronger statement than a two-point accuracy gap.

The 360M result needs two numbers, not one

By the contract, smollm2:360m scored 8 of 22 on the cases that contained an identifier. That reads like a model that cannot find things.

It is not. All fourteen of its wrong answers are the correct six digits with ORD- stripped. ORD-556120 came back as 556120, fourteen times out of fourteen.

It located the right identifier in 22 of 22 and formatted it correctly in 8. Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)

The decoys follow from the same gap. It handed back MEM-443322, 1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no model of the format at all. It returns whatever in the message looks most like an identifier.

Which suggests a use for it that is not extraction

A model with perfect recall and no precision is not a bad extractor. It is a candidate generator that has been asked to do the wrong job.

Twenty-two of twenty-two located, eight of twenty-two formatted. Put that output through the regex as a validator rather than trusting it directly, and you get the model's recall with the pattern's precision. The model proposes anything that looks like an identifier, including format variants the pattern was never written for. The pattern rejects anything malformed.

Crucially, neither half of that pipeline can produce the error the 1.5B model produced alone. The model cannot cause a false negative because it does not decide, and the regex cannot invent because it only matches.

I have not tested that composition and it is not in these numbers. But it is the one place in the extraction result where a small model has a job worth giving it, and it is a job where being sloppy is free.

Verdict for extraction itself: Unnecessary. Write the regex.

Classification: routing a message to one of six queues

Appointment, billing, availability, complaint, question, escalate. Fifty messages. The instruction said to use escalate when the message needs a person, including when you cannot tell what it is about.

	qwen2.5:1.5b	smollm2:360m
Overall	23/50 (46%)	15/50 (30%)
Labels used	6 of 6	3 of 6
Most common label	complaint, 38%	escalate, 86%
Wrong and not an escalation	26	3 (meaningless)

The 360M model answered escalate to 43 of 50 messages, which is not classification, and there is nothing further to say about it here. The rest of this section is the 1.5B model.

It is classifying, and it is not good at it. Forty-six percent, with twenty-six messages routed confidently to a wrong queue that was not the safe one.

Read those twenty-six and the verdict writes itself:

Someone slipped on the steps by the barrel room and hurt their wrist →
complaint
A formal notice regarding a trademark → complaint
"I have a severe sulfite allergy, what is actually in the wine" →
complaint
A reporter asking about the 2024 harvest → question
A distributor wanting to carry the wine in forty stores → question

An injury report, a legal notice and an allergy question, all routed to the complaints queue, confidently, with no escalation. All five billing messages about money problems also went to complaints.

Forty-six percent sounds like something you could work with. Twenty-six confident misroutes including an injury and a legal notice is not something you can work with.

And then I broke the score down by class

This is the part I nearly missed, and it is the most useful thing in the experiment.

Label	Correct	Of
complaint	6	6
question	9	10
availability	3	6
appointment	4	9
billing	1	7
escalate	0	12

Two classes near perfect. Four poor. One at zero.

The split is not difficulty, and it is not class size. It is what kind of label each one is.

complaint and question are topics. You can tell what a message is from the words in it. Someone complaining sounds like someone complaining.

escalate is a routing decision. Nothing in the content of a message makes it an escalation. It is an escalation because of what should happen next. "I have a severe sulfite allergy, what is actually in the wine" and "what time do you close" are both questions about the business, with the same grammar and the same tone, and one of them is a liability event.

The model did topics at 90 to 100 percent and routing decisions at zero. And billing sat in the middle for exactly the same reason: a message about being charged twice is a complaint, as text. The label is about which department owns the problem, which is not a property of the sentence.

Small models answer "what is this about." They do not answer "what should we do about it."

That reframes the five examples above. They are not five unlucky misroutings. All five are escalations, and escalate scored zero of twelve. The model was never going to find them, because finding them requires knowing a consequence that the text does not contain.

Two caveats on how hard to lean on this. The classes hold between six and twelve cases each, so "six of six" is six cases and the individual numbers are thin. And this is one model. The pattern across all six classes is consistent enough that I believe the explanation, but I am offering it as the best account of the data rather than as a measurement.

Verdict: Insufficient. And the actionable version: if one of your labels means "a human needs to see this," a small model will not find it. Build that one as a rule.

Grounded QA: answering a question from four documents

Fifteen questions against a 650-token corpus: club policy, tasting room, production notes, shipping. Eight are answerable from one passage, three require combining two, and four have no answer in the material at all.

I read all thirty answers by hand against the corpus. That is the only way to score this, and I will say why in a moment.

	qwen2.5:1.5b	smollm2:360m
Single passage (8)	8/8	7/8
Two passage (3)	1/3	0/3
Answerable total (11)	9/11	7/11
Correctly said "not in the material" (4)	2/4	0/4

Three boundaries separate cleanly here, and they are the useful result.

Retrieval works

The 1.5B model answered all eight single-passage questions correctly. Six bottles in the Vineyard tier. Twenty dollars per person. September 9th. Thirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five dollar restocking fee. Ten people on the barrel room tour.

Asked what happens if you join the club and immediately want out, it returned the two sentences that answer the question and stopped:

Cancellation requires one completed shipment. Members who cancel before receiving a shipment are charged for that shipment.

That is competent lookup across four documents, on a $170 computer, with the wifi switched off. I did not expect it and it is the most encouraging thing in the weekend.

Combination does not

One club member brings five guests on a Saturday. Club policy waives the tasting fee for the member plus three guests. The tasting room charges twenty dollars a head. Six people, four waived, two paying: forty dollars.

Both models answered twenty.

Neither could hold "plus three guests" against a party of six and subtract. The 1.5B model did get the second half right, correctly working out that a party of six is not a group larger than six and therefore needs no reservation, which is the only piece of multi-document reasoning either model performed all weekend.

The second two-passage question is the one I keep thinking about. Asked why someone would not find a 2024 rose and what they would be offered instead, the 1.5B model quoted the corpus exactly, and then:

...the reference material states that "We did not bottle a rose in 2024. The fruit that would normally go to rose was needed to make volume on the estate Pinot Noir." Therefore, the 2024 rose would be replaced with the fruit that would typically go into the estate Pinot Noir production.

The quotation is verbatim and right. The next sentence reverses the only relationship in it. The fruit went into the Pinot Noir, so what you would be offered is the Pinot Noir. You cannot be offered fruit.

Correct retrieval and inverted inference, in consecutive sentences, with the evidence on screen between them.

The 360M model's multi-document failures are worse than wrong. They are contaminated. It took the fourteen-days-notice rule, which governs skipping a club shipment, and applied it to booking a tasting room reservation. It reached into the shipping document to answer a cancellation question. Given more documents to search, it finds more wrong ones.

Knowing the answer is absent is the weakest link

Four questions have no answer in the corpus, chosen to be things a reader would reasonably expect a winery to have written down: a wine's alcohol percentage, who the winemaker is, what a club tier costs, what yeast was used.

The 1.5B model abstained on two and fabricated two. The 360M model fabricated all four.

And the fabrications are a more dangerous species than invention. Asked for the alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model answered:

The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.

Every component of that is true. Right wine, right vintage, right numbers, right document. Brix is sugar at harvest and has nothing to do with alcohol in the bottle. It reached for the nearest real fact instead of making one up, and produced something that reads as expertise while answering a question nobody asked. If you do not know what Brix is, that answer is indistinguishable from a correct one.

Both models made the same mistake on what the Cellar tier costs: both answered "twelve bottles per quarter." Grounding gave them the right tier and they handed back the quantity in place of the price.

The one answer the evidence did not change

The 360M model, asked whether a dog can come inside the tasting room:

Yes, you can bring a dog inside the tasting room.

The corpus says the patio is dog friendly and the indoor room is not, service animals excepted. I have the same answer from a run where the model had no corpus at all, and it is identical byte for byte. Having the right answer directly in the prompt changed nothing.

In a category where being wrong means a family drives out to a winery and gets turned around at the door.

Verdict: Marginal. Usable with a person reading the output. Not usable without one.

What it costs

Workflow	Model	tok/s	Prompt	Answer	Median call
extraction	qwen2.5:1.5b	4.3	92	9	5.0 s
classification	qwen2.5:1.5b	7.5	104	2	2.9 s
QA	qwen2.5:1.5b	3.7	708	15	6.9 s

Two things in here surprised me.

The compute tax does not track parameter count. The 1.5B model is slower than the 360M on extraction, 4.3 against 5.9 tokens per second, and faster on classification, 7.5 against 6.2, because on classification it answers in two tokens where the smaller model takes six. Four times the parameters, and which one wins depends entirely on how much each one decides to say.

A bigger prompt is cheap. A longer answer is what you pay for. I have a clean comparison here by accident: an earlier QA run went out with 97-token prompts and the real one with 708. Seven times the prompt added fifteen percent to the wait before output started, because prompt evaluation parallelizes. The latency increase came from answers three times longer, plus a twenty-one percent drop in per-token generation speed from attending over a longer context. Median latency went from 3.6 to 6.9 seconds and the slowest single call took ninety-eight.

If you are sizing a local system, that is the inversion to internalize. Stuffing more context in is affordable. Letting the model be chatty is not.

And check your thermals, which is not optional advice. My first attempt at this ran with heat sinks and no fan. Thirty calls to a 360M model was enough to hit the Pi 4's soft temperature limit and start capping the clock. vcgencmd get_throttled returned 0xe0000. I threw those latency numbers away.

Nothing in Ollama's output, my scorer, or the result files said a word about it. Every number on screen was internally consistent and quietly describing a machine that was slowing down. The only way to find out was to ask the firmware a question I had to already suspect the answer to. Buy the fan.

And it does run with the network off. I switched the radio off, proved there was no route out, and ran one call from each workflow. All three worked. The claim that supports is narrow and worth stating carefully: this
configuration, on this afternoon, ran with no route to the internet. It is not a claim that local models are private or that this setup is secure, and a Pi can be misconfigured as thoroughly as anything in a data center. But for the specific question of whether the inference needs the network, it does not.

So: do you actually need an LLM?

Workload	Verdict
Pull a known-format identifier out of text	Unnecessary
Route a message to one of six queues	Insufficient
Answer a question from your own documents	Marginal

Three workflows that all sound like language tasks, three different answers, and not one of them is "yes, use a model."

And the same line runs through all three. Extraction asks what this string is. Topic labels ask what this message is about. Single-passage lookup asks what the document says. Those went well, or well enough. Escalate asks what we should do, the fee question asks what the customer owes, and the unanswerable questions ask whether we know. Zero of twelve, both models wrong, and two of four.

A small model will tell you what something is. It will not tell you what to do about it. Every result here falls on one side or the other of that sentence, and I did not have it written down when I started.

Which gives four properties worth checking before you reach for one:

The answer is already present in the input. Not inferable from it. Present in it.
There is exactly one of it. The moment two facts have to be combined, this stops working, and the arithmetic in that tasting fee question is the proof.
The output is short. Longer answers are where the latency lives on this hardware, and where the fabrication lives everywhere.
Something other than the model decides whether the output is acceptable. A regex, a schema, a person, an index that either returns rows or does not.

Take away any one of those four and the experiment says do not do it. The 46 percent and the zero of twelve are what it looks like when you ask a small model to be the thing that decides.

And if you have all four, you may not need the model either. That is worth checking first, because it was true for one of my three workflows and I did not expect it to be.

Where I was wrong

Twenty-two predictions, committed before I installed anything. Ten right, eleven wrong, one unresolvable.

Two of my seven confident calls broke. I was sure the 360M model would get most of the easy extraction cases, and it got 8 of 22 by failing at format rather than at finding. I was sure it would produce invalid output often, and it did so once in fifty, then emitted the same valid word forty-three times.

But the number that bothers me is not in the confident column.

Confidence	Right	Wrong
Confident	5	2
Leaning	2	7
Coin flip	3	2

My hunches did worse than my coin flips. Two of ten on the calls where I had a feeling but not much behind it, three of five on the ones I explicitly labeled as guesses.

That ordering is backwards and it is the most useful thing on the sheet. On the coin flips I knew I was guessing, so I guessed cleanly. On the leaning calls I was pattern-matching against intuitions about model behavior that turned out not to describe these models at all, and the feeling of having a reason was the problem. If I had marked all twelve of those as coin flips I would have scored better.

I will take that over a clean result. The honest version of this exercise is not "here are the numbers," it is "here is what I believed, here is what happened, and here is the gap."

The gap, this time, was mostly in the hunches.

The harness that produced these numbers had five measurement flaws in it, none of which manufactured an entire finding out of a missing directory and nearly got published. That is a separate post.

Everything here is reproducible: fixtures, runner, scorer, the committed predictions and the run record are at
github.com/kenwalger/right-size-the-model.
Fixture set v1. Read LIMITATIONS.md before citing any of these numbers:
two models, one board, one quantization, ninety-five cases. This is an illustration of a method, not a benchmark.

Pytest in Practice

Tue, 13 Oct 2026 12:45:48 +0000

I've been learning Python by building a video clipping CLI application, and along the way I've ended up learning a lot more than I expected.

Some of it was straightforward. Some of it made me stop and think, especially when I first started writing tests.

Things like:

How does pytest even know which functions are tests?
Why does everyone keep putting things in conftest.py?
When should I use a fixture?
What's the difference between patch and monkeypatch?
Why did my mock not work when I was sure I patched the right thing?
When should I mock something, and when should I just run the real thing?
What's the point of parametrization when I could just use a loop?
How do I debug one failing test without running the entire suite every time?

These are the kinds of things that can feel like arbitrary rules when you're first encountering them.

Once I understood the reasoning behind them, though, pytest started making a lot more sense.

So this isn't meant to be a complete pytest reference or an explanation of every feature pytest has. It's more like a collection of the things that finally "clicked" for me while testing a real Python project.

The examples are based on my video clipping CLI, so you'll see things like FFmpeg, video metadata, audio extraction, transcription, temporary files, external dependencies, and slow tests.

Hopefully, if you're also learning Python by building something rather than just following tutorials, some of these lessons save you a few of the "why does it work like that?" moments I had.

Installing pytest

If you're using uv, installing pytest and coverage support is straightforward:

uv add --dev pytest pytest-cov
uv run pytest --version

You now have everything needed to start writing and running tests.

Your First Test

A pytest test can be nothing more than a function:

# tests/core/test_video_inspector.py

from clipper.core.video_inspector import get_video_metadata


def test_get_video_metadata_returns_none_on_missing_file():
    result = get_video_metadata("does_not_exist.mp4")
    assert result is None

Run it with:

uv run pytest

That's it.

There is no base class, no registration, and no decorator required.

pytest finds the test because:

the file starts with test_
the function starts with test_

That naming convention is the mechanism.

How Does pytest Find Your Tests?

pytest automatically walks the current directory and its subdirectories looking for test files.

By default, it looks for files matching patterns such as:

test_*.py
*_test.py

Inside those files, it looks for functions matching:

test_*

and, inside classes whose names start with Test*, methods that also match test_*.

So this:

tests/
└── core/
    └── test_video_inspector.py

is enough for pytest to discover the file automatically. You don't need to register the test anywhere.

This is also why commands such as:

uv run pytest --cov=src

work without explicitly listing every test file. pytest's discovery mechanism does the work.

Assertions: Just Use `assert`

One of the nicest things about pytest is that you use Python's built-in assert.

def test_fps_math():
    assert round(29.970029970029969, 2) == 29.97

You don't need a collection of methods like:

assertEqual(...)
assertTrue(...)
assertFalse(...)
assertIn(...)

Instead, pytest rewrites assert statements when running tests so that failures contain useful information about what each side evaluated to. That makes a failed assertion much more informative than simply:

Assertion failed

Comparing Floats with `pytest.approx`

Don't compare floating-point calculations using exact equality.

Instead:

import pytest


def test_duration_seconds():
    assert 125.5 == pytest.approx(float("125.500001"))

Floating-point calculations rarely land exactly where you expect. This matters especially when dealing with things like frame rates, durations, timestamps, or calculations involving Fraction.

Without pytest.approx, you might end up writing this everywhere:

assert abs(actual - expected) < 0.001

pytest.approx gives you the tolerance without repeating that boilerplate.

`pytest.approx` Works with Nested Structures

pytest.approx isn't limited to individual numbers. You can use it when comparing nested structures:

def test_transcribe_matches_golden_output(
    sample_speech_path,
    sample_transcript,
):
    result = transcribe(str(sample_speech_path))

    assert result == pytest.approx(sample_transcript)

This is particularly useful for regression or "golden file" tests. Imagine a transcript like:

{
    "segments": [
        {
            "start": 0.123456,
            "end": 1.987654,
            "words": [
                {"word": "hello", "start": 0.123},
                {"word": "world", "start": 0.456},
            ],
        }
    ]
}

You don't want to write a loop comparing every timestamp manually. pytest.approx can recursively compare numeric values with tolerance while still using normal equality for strings and other values — so it's practical to compare an entire expected transcript while allowing for harmless floating-point differences.

Testing That an Error Is Actually Raised

When the expected behavior is for a function to raise an exception, use pytest.raises.

import pytest


def test_extract_audio_raises_on_bad_codec():
    with pytest.raises(ffmpeg.Error):
        extract_audio(
            "in.mp4",
            "out.wav",
            codec="not_a_real_codec",
        )

This test verifies two things:

The code raises an exception.
The exception is the expected type.

The test fails if the function doesn't raise anything. It also fails if it raises a different exception type. Both outcomes are useful information.

`pytest.raises` Only Works When the Exception Escapes

There's an important distinction here. Suppose your function catches its own exception:

def extract_audio(...):
    try:
        ...
    except ffmpeg.Error:
        logger.error(...)
        return None

The exception never reaches the caller. Therefore, this won't work:

with pytest.raises(ffmpeg.Error):
    extract_audio(...)

There's nothing left for pytest.raises to catch. Instead, test the function's actual contract:

result = extract_audio(...)

assert result is None

And, if logging is part of the behavior you care about, use caplog to verify that the expected error was logged.

A useful rule of thumb:

If the function returns None from its except block, test the return value rather than expecting an exception.

Parametrize: One Test, Many Inputs

Suppose you need to test several timestamp formats. You could write three tests:

def test_timestamp_one():
    ...


def test_timestamp_two():
    ...


def test_timestamp_three():
    ...

But the test logic is identical. That's exactly what pytest.mark.parametrize is for:

@pytest.mark.parametrize(
    "timestamp,expected_seconds",
    [
        ("00:00:01", 1.0),
        ("00:01:00", 60.0),
        ("01:00:00", 3600.0),
    ],
)
def test_validate_timestamp_parses_valid_formats(
    timestamp,
    expected_seconds,
):
    assert validate_timestamp(timestamp) == pytest.approx(expected_seconds)

Think of it like a form letter: the first argument names the blanks to fill in (timestamp, expected_seconds), and the second argument is a stack of filled-in versions. pytest prints one out and runs the test for each one:

test_validate_timestamp_parses_valid_formats[00:00:01-1.0]
test_validate_timestamp_parses_valid_formats[00:01:00-60.0]
test_validate_timestamp_parses_valid_formats[01:00:00-3600.0]

Each case can pass or fail independently.

Why independence matters

You could achieve something similar with a loop:

def test_timestamps():
    for timestamp, expected in cases:
        assert validate_timestamp(timestamp) == expected

But there's a major difference: a normal loop stops at the first failure. With parametrization, pytest runs every case. So one test run tells you exactly which inputs are broken instead of merely telling you:

test_timestamps failed

When should you use it?

The signal is simple:

If you're about to write several nearly identical test functions that differ only in their input values, consider parametrize.

It works just as well for valid inputs as it does for malformed or invalid ones.

Grouping Tests in a Class

pytest supports test classes:

class TestGetVideoMetadata:
    def test_returns_none_for_missing_file(self):
        assert get_video_metadata("missing.mp4") is None

    def test_has_audio_true_when_audio_stream_present(self):
        ...

But classes aren't mandatory. Use them when they provide something useful, such as:

shared setup
fixtures scoped to the group
applying a marker to multiple tests

Don't use a class purely because it feels more organized. A well-named test file already provides plenty of organization.

An important detail about test classes

Each test method gets a fresh instance of the class. So this:

class TestSomething:
    def test_one(self):
        self.value = 1

    def test_two(self):
        assert self.value == 1

is not valid. test_two doesn't receive the instance used by test_one. That's intentional — tests should be isolated from one another. Depending on mutable state from another test creates order-dependent and difficult-to-debug failures.

Fixtures

Fixtures are one of pytest's most useful features. A fixture is a function that provides something a test needs — think of it like an ingredient a recipe asks for by name. You don't prep it inside the recipe itself; you just say "I need sample_video_path" and pytest hands it to you already made.

# tests/conftest.py

import pytest
from pathlib import Path


@pytest.fixture
def sample_video_path() -> Path:
    return Path(__file__).parent / "fixtures" / "sample.mp4"

A test can then request it simply by putting the fixture name in its parameters:

# tests/core/test_video_inspector.py

def test_metadata_against_real_file(sample_video_path):
    result = get_video_metadata(str(sample_video_path))

    assert result is not None

pytest sees sample_video_path in the test signature, finds the fixture with that name, runs it, and passes the result to the test. No import is required.

Why `conftest.py` Matters

conftest.py is automatically discovered by pytest. Fixtures defined there are available to test files in the same directory and its subdirectories — it works like a shared pantry that any test file in that part of the tree can pull from without asking for a delivery.

For example:

tests/
├── conftest.py
├── core/
│   └── test_video_inspector.py
└── api/
    └── test_transcription.py

A fixture defined in tests/conftest.py is available to both tests/core/ and tests/api/.

You can also have a more specific conftest.py deeper in the tree:

tests/
├── conftest.py
└── core/
    ├── conftest.py
    └── test_video_inspector.py

The deeper fixture configuration can then be used specifically by that subtree. A useful convention is: root conftest.py for shared fixtures, nested conftest.py files for fixtures that are genuinely local to a particular test area.

Fixture Scope

Fixtures don't necessarily have to run for every test. You can control their lifetime:

@pytest.fixture(scope="function")  # default
@pytest.fixture(scope="module")    # once per test file
@pytest.fixture(scope="session")   # once per entire test run

The default is scope="function", which means a fresh fixture is created for every test. Prefer the default unless there is a good reason to reuse the fixture.

A broader scope can make sense when setup is genuinely expensive and safe to share. For example, generating a synthetic sample video once per session may be much cheaper than generating it once for every test.

Built-in Fixtures Worth Knowing

pytest provides several useful fixtures out of the box.

`tmp_path`

Provides a unique temporary directory for each test. Great for anything that needs to write files.

`caplog`

Captures logging output. Useful when you want to verify that something logged an error or warning.

`capsys`

Captures stdout and stderr. Useful for testing CLI output.

`monkeypatch`

Temporarily changes attributes, environment variables, dictionary entries, and other values. pytest automatically restores the changes after the test.

You can inspect available fixtures with:

uv run pytest --fixtures

`tmp_path` in Practice

Suppose you want to verify that audio extraction creates an actual file:

def test_extract_audio_writes_real_file(
    sample_video_path,
    tmp_path,
):
    output_path = tmp_path / "output.wav"

    result = extract_audio(
        str(sample_video_path),
        str(output_path),
    )

    assert result == str(output_path)
    assert output_path.exists()

This is an integration-style test. It actually runs FFmpeg and checks that a real file was created. But because tmp_path is temporary, the test doesn't leave generated files scattered around your project — pytest creates the directory and handles its cleanup.

Mocking: Two Tools, Different Jobs

Python gives you unittest.mock. pytest gives you monkeypatch. They overlap, but each one shines in a different situation.

`unittest.mock.patch`

Use patch when you want a more capable mock, particularly when you need to inspect how something was called.

from unittest.mock import patch


def test_metadata_no_audio():
    fake_probe = {
        "format": {
            "filename": "silent.mp4",
            "duration": "10.0",
        },
        "streams": [
            {
                "codec_type": "video",
                "r_frame_rate": "25/1",
            },
        ],
    }

    with patch(
        "clipper.core.video_inspector.ffmpeg.probe",
        return_value=fake_probe,
    ):
        result = get_video_metadata("silent.mp4")

    assert result["has_audio"] is False

patch temporarily replaces the target with a mock. It is particularly useful when you want to assert things like:

mock_input.assert_called_once_with("in.mp4")

`monkeypatch`

monkeypatch is pytest's own fixture. It's particularly convenient for simple temporary substitutions.

def test_uses_default_bitrate_env_var(monkeypatch):
    monkeypatch.setenv(
        "CLIPPER_DEFAULT_BITRATE",
        "256k",
    )

    assert get_default_bitrate() == "256k"

There's no context manager and no manual cleanup — pytest automatically restores the environment variable after the test. You can also replace attributes with monkeypatch.setattr(...) or dictionary entries with monkeypatch.setitem(...).

`patch` vs `monkeypatch`: A Practical Mental Model

Imagine your code is a toy robot and you want to test what happens when one of its components behaves differently. You don't want to actually break the component — you just want to temporarily pretend that it behaves differently.

monkeypatch is like putting a temporary sticky note over the component: "For this test, pretend this value is different." pytest removes the note when the test finishes.

patch is another version of the same idea, but with a much more powerful mock object behind it. patch can easily track call count, arguments, return values, exceptions, and chained calls.

Both clean themselves up reliably when used correctly. With patch, the common form in this project is:

with patch(...):
    ...

The mock is restored automatically when the block ends, even if the test fails.

Which One Should You Use?

A practical rule:

Need	Use
Assert how something was called	`patch`
Check call count or exact arguments	`patch`
Mock a chained/fluent API	`patch`
Replace an environment variable	`monkeypatch`
Temporarily change one attribute	`monkeypatch.setattr(...)`
Temporarily change a dictionary entry	`monkeypatch.setitem(...)`
Replace an external dependency and inspect the mock	`patch`

The important thing isn't to memorize a strict rule. It's to recognize the distinction:

If you need to inspect how the dependency was used, patch is usually the natural choice. If you just need to temporarily replace a value or attribute, monkeypatch is often simpler.

`side_effect`: Make a Mock Fail

Mocks aren't only useful for returning fake successful results. They can also simulate failures.

return_value means "always return this." side_effect means "raise this exception instead."

with patch(
    "clipper.core.video_inspector.ffmpeg.probe",
    side_effect=ffmpeg.Error(
        cmd="ffprobe",
        stdout=b"",
        stderr=b"bad file",
    ),
):
    result = get_video_metadata("bad.mp4")

This lets you exercise failure paths without needing to create a genuinely broken video file.

Mock the Boundary, Not Your Own Logic

This is probably the most important mocking rule in the entire guide:

Mock the boundary, not the logic you're trying to test.

Suppose your code does this:

probe = ffmpeg.probe(path)

The interesting logic is what your application does with the result. So mock ffmpeg.probe and provide realistic fake data — don't mock the parsing logic itself. A test that mocks everything, including the thing you're trying to test, can end up proving nothing more than: "my mocks returned what I told them to return."

Put another way — you're testing your recipe, not reinventing the oven. External boundaries are good mocking candidates:

FFmpeg
HTTP APIs
databases
filesystem operations
cloud services
external processes

Your own business logic is usually what you want to execute for real.

The Most Common Mocking Mistake: Patching the Wrong Path

This catches a lot of people. Suppose your module contains:

# video_inspector.py

import ffmpeg


def get_video_metadata(path):
    return ffmpeg.probe(path)

You might think you should patch ffmpeg.probe. But that's not necessarily the correct target — the code under test looks up ffmpeg in its own module namespace. Therefore, patch where the name is looked up:

patch(
    "clipper.core.video_inspector.ffmpeg.probe"
)

The general rule is:

Patch where the object is looked up, not where it was originally defined.

If you patch the wrong path, the real dependency may still run. That can make a test unexpectedly hit FFmpeg, the network, a database, or some other external system.

Mocking a Chained Call

Some APIs aren't a single function call. For example, ffmpeg-python can use a fluent chain:

ffmpeg.input(...).output(...).overwrite_output().run(...)

Mocking that requires walking the chain. MagicMock automatically creates mocks for attributes you access, so mock_input.return_value represents the result of ffmpeg.input(...), .output.return_value represents the next link in the chain, and so on until you reach .run — the actual operation you want to make fail.

def test_extract_audio_sad_path(caplog):
    fake_error = ffmpeg.Error(
        cmd="ffmpeg",
        stdout=b"",
        stderr=b"encoding failed",
    )

    with patch(
        "clipper.core.video_inspector.ffmpeg.input"
    ) as mock_input:
        mock_run = (
            mock_input
            .return_value
            .output
            .return_value
            .overwrite_output
            .return_value
            .run
        )
        mock_run.side_effect = fake_error

        result = extract_audio(
            "in.mp4",
            "out.wav",
        )

    assert result is None
    assert "error" in caplog.text.lower()

Walking the chain into its own variable (mock_run) before setting side_effect keeps the assignment itself simple and unambiguous — trying to set an attribute in the middle of a long parenthesized chain is a common source of syntax mistakes.

Organizing Your Test Files

A simple and predictable test layout goes a long way. If your source looks like:

src/
└── clipper/
    └── core/
        └── video_inspector.py

mirror it under tests/:

tests/
└── core/
    └── test_video_inspector.py

This makes it immediately obvious where the tests for a source module live.

Keep `tests/` Outside `src/`

Use:

project/
├── src/
├── tests/
└── pyproject.toml

rather than:

src/
├── clipper/
└── tests/

Keeping tests outside the package helps avoid accidentally shipping them with your built distribution and keeps the distinction between application code and test code clear.

You Usually Don't Need `init.py`

pytest doesn't require your test directories to be Python packages. So this is perfectly fine:

tests/
├── core/
│   └── test_video_inspector.py
└── api/
    └── test_transcription.py

No __init__.py required. There can be edge cases where you intentionally want package semantics, but don't add __init__.py to test directories just because you think pytest requires it. It doesn't.

Keep Fixtures Separate from Test Code

tests/
├── conftest.py
├── fixtures/
│   └── sample.mp4
└── core/
    └── test_video_inspector.py

This makes the distinction obvious: test_*.py contains test logic, fixtures/ contains test data.

Mark Slow and External Tests

Not every test costs the same amount to run. A test that calls real FFmpeg is different from a test that checks a pure Python function.

You can mark integration tests:

@pytest.mark.integration
def test_extract_audio_against_real_file(...):
    ...

Then register the marker in pyproject.toml:

[tool.pytest.ini_options]
markers = [
    "integration: calls real ffmpeg/ffprobe against a fixture file",
]

Now you can run the fast suite without integration tests:

pytest -m "not integration"

And run everything with:

pytest

This gives you a useful distinction between fast unit tests and slower integration tests.

Debugging pytest Without Sprinkling `print()` Everywhere

pytest captures output by default. That can be surprising when you're debugging — a print(result) that produces nothing on screen. Here are the options you'll reach for most often.

`-s`: Show stdout/stderr

uv run pytest -s

This disables output capturing, so your print() statements become visible. It's useful, although there are usually better debugging tools than adding prints everywhere.

`-k`: Run Tests by Name

If you're working on audio extraction, there's no reason to run 200 unrelated tests.

uv run pytest -k "extract_audio"

pytest runs tests whose names match the expression. This is one of the most useful options during active development.

`-x`: Stop at the First Failure

uv run pytest -x

Instead of getting a wall of failures, pytest stops after the first one. This is particularly useful when fixing a cascading failure.

`--pdb`: Drop Into the Debugger

uv run pytest --pdb

When a test fails, pytest drops you into an interactive debugger at the point of failure. Inside the debugger, you can inspect variables and control execution — c continues, q quits. This can be much faster than trying to predict where a bug is before running the test.

`-vv`: More Verbose Output

uv run pytest -vv

Useful when assertions involve large dictionaries, long strings, or parametrized tests and you want more detail in the output.

`--lf`: Run Only the Last Failures

uv run pytest --lf

--lf means "last failed." It reruns tests that failed during the previous run instead of running the entire suite. This is particularly useful when you're fixing several failures one by one.

A Practical Debugging Command

These options can be combined:

uv run pytest -k "extract_audio" -x --pdb

This means: run only tests matching extract_audio, stop at the first failure, and open the debugger at the failure. That's a very practical development loop.

Running a Specific Test

pytest doesn't natively use line numbers to select a test. This isn't standard pytest test selection:

pytest test_file.py:29

Instead, select by node ID:

pytest tests/core/test_video_inspector.py::test_metadata_no_audio

Or use -k:

pytest -k "metadata_no_audio"

Editors such as VS Code and PyCharm can provide "run this test" buttons. They translate your selection into pytest's name-based test selection behind the scenes. There are plugins that provide literal line-number selection, but you generally don't need one.

Quickly Inspecting a Return Value

Sometimes you just want to see what a function actually returned. One quick technique is to deliberately make an assertion fail:

def test_scratch(sample_video_path):
    result = get_video_metadata(str(sample_video_path))

    assert result == {}

If {} isn't the actual result, pytest's failure output will show you the value. Delete the temporary test afterward.

Another option is:

uv run pytest --showlocals

or the shorthand -l, which shows local variables when a test fails.

Dealing with Slow Tests

Don't guess which tests are slow. Measure them.

uv run pytest --durations=10

This reports the ten slowest tests after the run. You may discover that one or two tests account for most of the delay. Optimizing everything before measuring can waste the very time you're trying to save.

Mark Slow Tests

Tests that load a real model or perform real inference are good candidates for a slow marker:

@pytest.mark.slow
def test_transcribe_real_audio_produces_valid_shape(
    sample_speech_path,
):
    ...

[tool.pytest.ini_options]
markers = [
    "integration: calls real ffmpeg/ffprobe against a fixture file",
    "slow: tests that load a real model or run real transcription",
]

Then your normal development loop can skip them:

pytest -m "not slow"

And your complete suite can still run with pytest.

Caching Can Already Solve Some Performance Problems

Suppose your application loads a model once at module level, and subsequent calls reuse it. Then only the first test that needs the model may pay the expensive loading cost — the others can reuse the already-loaded instance.

You can verify whether this is actually happening with uv run pytest --durations=10. If the first model-related test is slow and subsequent tests are much faster, your existing caching may already be doing the job. If every test is slow, something may be reloading the model when it shouldn't. Measure first.

Parallelizing the Suite

If the entire suite is slow, rather than just a handful of expensive tests, parallel execution may help.

uv add --dev pytest-xdist
uv run pytest -n auto

-n auto lets xdist determine a useful number of workers. But don't reach for parallelization automatically — for many projects, simply excluding slow tests during development (pytest -m "not slow") is enough.

Coverage Is a Flashlight, Not a Scoreboard

Coverage is useful. It tells you which lines of code weren't exercised by your tests.

uv run pytest --cov=src

This can help you find code paths you haven't thought about. But coverage doesn't tell you whether your tests are good. You can achieve 100% coverage with tests that assert almost nothing meaningful:

def test_everything():
    my_function()

The code executed. Coverage increases. But the test may tell you almost nothing about whether the behavior is correct. So treat coverage as a flashlight — use it to illuminate parts of the code you haven't tested. Don't treat it as a scoreboard you have to maximize.

A Simple Checklist for Every New Test

When adding tests for a new function or feature, ask:

1. What's the happy path?

What should happen when everything is valid?

2. What's at least one edge case?

Consider: empty input, missing fields, boundary values, unexpected but valid combinations.

3. What's the failure mode?

What should happen when input is invalid, an external dependency fails, a file doesn't exist, or an API returns an unexpected result?

4. What should be mocked?

Usually mock things that are external, slow, non-deterministic, or side-effecting. But let your own logic run for real.

5. Is the test asking one question?

If the test name contains the word "and," that's sometimes a sign that you're testing two behaviors at once. A test should ideally answer one clear question.

The Mental Model I Ended Up With

pytest can look like a large collection of features when you're first learning it. But after working with it for a while, I found that most of what I actually needed came down to a few ideas.

Tests are ordinary Python functions. pytest discovers them through naming conventions.

Assertions are ordinary Python assert statements. pytest makes their failures informative.

Fixtures provide reusable things that tests need. conftest.py makes shared fixtures available without imports.

Parametrization lets you run the same test logic against multiple inputs.

Mocks let you replace boundaries you don't want to exercise for real. Things like FFmpeg, APIs, databases, and other external systems are often good candidates — while the logic you're actually testing should usually run normally.

Markers let you separate tests by purpose or cost. And the debugging tools make it much easier to figure out why something failed without resorting to print() everywhere.

Most importantly, I stopped thinking of testing as "how do I get my coverage number up?" and started thinking more in terms of "what question am I trying to answer with this test?" For me, that shift made pytest much easier to understand.

The syntax is the easy part. The useful part is learning what behavior is worth testing, what should be isolated, what should run for real, and how to make a failure tell you something useful.

A Route Is Not a Feature Until Someone Can Reach It

Tue, 13 Oct 2026 07:01:11 +0000

Modern applications are often assembled from modules. That is good for ownership, deployment boundaries, and keeping large codebases understandable. It also creates a subtle user-experience failure mode: a page can compile, carry a valid route, pass its local tests, and still be effectively absent because no one can discover it from a real entry point.

This is not primarily a visual-design problem. It is an architecture-contract problem.

Routable is not reachable

Imagine an administrative application with pages spread across two modules. The team already has a test that scans routed pages and checks that each one appears in a navigation catalogue. The test is green, so the navigation surface looks covered.

Later, someone discovers that the scan only walks one module. The second route tree was outside its boundary. Pages there could be added indefinitely without the test knowing they existed.

That is the first important lesson: a completeness check is only as complete as its discovery mechanism.

When the scan is expanded, it finds an entry screen that has documentation and valid follow-on steps, but no link from the application itself. The workflow exists technically. Operationally, it begins with “type this URL.” That is not a usable feature contract.

Define a genuine entry point

The tempting fix is to accept any reference to the route. Perhaps another page links to it. Perhaps it appears on a tab rail. Perhaps a help article mentions it.

Those references are not equivalent.

A help article explains a feature; it is not application navigation. A link on a neighbouring screen can be useful, but it may make discoverability depend on an accidental journey. A tab rail is particularly deceptive: it helps someone move within an area only after they have reached one of its pages. It cannot prove that the area itself has an entrance.

The architectural rule should therefore name the user-facing property you actually want. For example:

Every static administrative workflow page must be represented by a discoverable entry in the primary hub available to its entitled audience.

That sentence is stricter than “the route is mentioned somewhere,” and that is precisely why it is useful.

Test pages, not strings

Route graphs have legitimate shapes that a simple string search will misclassify.

A parameterised detail page is usually reached from a list or search result. Giving every detail route a top-level card would be noisy and misleading. The test can classify routes containing parameters as downstream pages rather than entry pages.

The reverse case also matters. One page may answer on two route aliases for compatibility. Requiring two navigation cards would duplicate the same destination. The unit of the contract is the page, with at least one discoverable canonical route—not every literal route declaration.

These distinctions keep the test strict without making it simplistic.

Make the scan fail honestly

Source-scanning tests have a dangerous failure mode: they can pass after discovering nothing. A changed directory, glob, parser, or build layout can turn the result set into an empty collection, after which every “all pages are covered” assertion becomes vacuously true.

Add a coarse lower bound to the discovered page count. It does not need to encode the exact inventory. It only needs to make a broken scan conspicuous. Pair that with a useful failure message that lists uncovered entry pages and explains the intended remedy.

Exceptions deserve the same discipline. Sometimes an exception is legitimate, but an exemption list easily becomes a hiding place. Store a reason beside each exception, keep the list narrow, and assert that every exception still corresponds to a real uncovered case. Once navigation is fixed, the stale exception should fail the next test run.

That turns exceptions into temporary, reviewable decisions rather than permanent silence.

Align visibility with authorization

Navigation and authorization answer related but different questions.

Navigation visibility asks, “Should this person be offered this destination?” Authorization asks, “May this request proceed?” A hidden card is not a security boundary. Someone can still follow a saved link or construct a request directly.

The useful contract is alignment: the navigation entry should use the same entitlement expressed by the destination policy, while the page or endpoint continues to enforce that policy independently.

This check often exposes ambiguity. If a destination has only broad authentication while its navigation card is restricted to a privileged audience, do not quietly declare the card to be protection. Flag the mismatch for an explicit security decision. Navigation work should reveal authorization uncertainty, not accidentally resolve it.

The trade-off is real

A strict hub contract has costs. The catalogue grows. New pages require deliberate placement and wording. A hub can become crowded if every capability is presented at the same level.

The alternative, however, is not free. A loose rule produces accidental journeys, invisible capabilities, and tests that certify the presence of links without certifying usable discovery.

The practical balance is to keep the entry-point contract strict while allowing the interface to group, search, filter, and progressively disclose destinations. The architecture test should protect reachability; the visual design can protect clarity.

A small checklist

For a modular routed application:

Inventory every route tree, not only the first one the test was written for.
Classify static entry pages, parameterised downstream pages, and route aliases.
Identify the navigation roots users can actually reach.
Require every entry page to appear from an appropriate root.
Add a minimum discovery count to catch broken scans.
Make exemptions explicit, reasoned, narrow, and self-cleaning.
Align link visibility with destination authorization, while enforcing security at the destination.

The deeper lesson is that architecture includes the path a person can take through the system. If a feature has no honest entrance, its route is an implementation detail—not a delivered capability.

Isolate to Elevate: Why Component Testing in Isolation Saves Your Codebase

Mon, 12 Oct 2026 13:46:09 +0000

Isolate to Elevate: Why Component Testing in Isolation Saves Your Codebase

Component testing in isolation is a software verification strategy where an individual piece of an application—such as a button, user profile card, or navigation bar—is evaluated entirely on its own. Instead of connecting the component to real databases, live servers, or neighboring UI elements, developers replace external dependencies with controlled, artificial placeholders called mocks or stubs. This guarantees that if a test fails, the issue is guaranteed to exist inside that specific component rather than in an external network or backend service.

A Real-Life Analogy: The Bench-Tested Alternator

Imagine a mechanic inspecting a newly manufactured car alternator. Instead of installing the alternator inside a fully assembled luxury vehicle, turning the key in the ignition, and hoping the dashboard lights illuminate, the mechanic places the alternator onto a specialized workbench.

On this test bench, artificial wires power the alternator, and a standalone machine spins its rotor while measuring electrical voltage. If the meter shows an improper charge, the mechanic instantly knows the alternator itself is defective. There is zero ambiguity—the problem cannot be blamed on a dead car battery, a frayed ignition wire, or a blown fuse elsewhere in the vehicle. Component testing in isolation applies this exact bench-test philosophy to user interface elements.

Why Component Testing in Isolation Matters Daily

In modern web development, frontend components frequently rely on complex API (Application Programming Interface) calls—the background communication channels that fetch data from databases over the internet. If you attempt to test a component while connected to live production or staging servers, your automated tests become fragile and unpredictable. A minor network hiccup or scheduled database maintenance will cause UI tests to fail, triggering false alarms and wasting valuable engineering hours.

Testing components in isolation eliminates this instability. It yields significant daily benefits:

Lightning-Fast Execution: Tests run in a local memory environment without waiting for HTTP network responses, executing hundreds of checks in milliseconds.
Deterministic Feedback: Tests yield identical, repeatable results regardless of backend availability or database state.
Effortless Edge-Case Coverage: Developers can effortlessly simulate tricky scenarios, such as slow network connections, server crash codes (like 500 errors), or extremely long text strings, without needing to manipulate a real server.

Code Example: Testing a React Component in Isolation

Below is a practical example using React (a popular JavaScript library for building user interfaces) and React Testing Library.

First, here is our component, UserProfile:

// UserProfile.jsx
import React from 'react';

export function UserProfile({ user, isLoading, error }) {
  if (isLoading) {
    return <div data-testid="loading">Loading user details...</div>;
  }

  if (error) {
    return <div data-testid="error">Error loading user profile.</div>;
  }

  return (
    <div className="user-card">
      <h2>{user.name}</h2>
      <p>Email: {user.email}</p>
    </div>
  );
}

Now, we test UserProfile in total isolation. Notice how we pass fake JavaScript object data directly through props (inputs passed to React components) without making real network requests:

// UserProfile.test.jsx
import React from 'react';
import { render, screen } from '@testing-library/react';
import { UserProfile } from './UserProfile';

test('renders user details correctly when data is provided', () => {
  // 1. Arrange: Create mock user data
  const mockUser = { name: 'Sarah Connor', email: 'sarah@skynet.com' };

  // 2. Act: Render component in isolation
  render(<UserProfile user={mockUser} isLoading={false} error={false} />);

  // 3. Assert: Verify the rendered output
  expect(screen.getByText('Sarah Connor')).toBeInTheDocument();
  expect(screen.getByText('Email: sarah@skynet.com')).toBeInTheDocument();
});

test('renders error message during failure state', () => {
  render(<UserProfile user={null} isLoading={false} error={true} />);

  expect(screen.getByTestId('error')).toHaveTextContent('Error loading user profile.');
});

In these tests, the render function mounts UserProfile inside a simulated virtual browser. We control every input state directly, ensuring complete certainty over what we are testing.

Summary Takeaway

Building resilient digital experiences requires modular confidence. By isolating individual UI components during automated testing, engineering teams replace slow, flaky test pipelines with deterministic feedback loops that pinpoint bugs in milliseconds. Mastering this technique ensures that every piece of your user interface is proven battle-ready before it ever integrates into a larger software ecosystem.

Originally published on my blog. You can read the alternative breakdown here.

Don’t Turn Delegated Access Into an Identity Oracle

Mon, 12 Oct 2026 06:01:17 +0000

Delegated access often starts with an innocent mismatch.

A person knows an email address. The authorization system knows an account identifier. The obvious bridge is a search box: enter the address, find the account, then grant access.

That bridge can become a privacy leak.

If the screen behaves differently for a known address and an unknown one, every attempt answers a sensitive question: “Does this person have an account here?” An authenticated caller can repeat that question across many addresses. The feature may be correctly authorized for its intended task while still exposing more identity information than the task requires.

The deeper lesson is simple: account discovery is not a necessary prerequisite for delegated access. Identity can be resolved later, at the moment the recipient proves ownership and gives consent.

The convenient design carries a hidden query

Consider a generic workflow in which one person wants to share a limited capability with another.

The direct version usually looks like this:

Enter the recipient’s address.
Search for a matching account.
Show whether a match exists.
Create access against that account.

Authorization to share the resource does not make the lookup harmless. Even subtle differences disclose the answer: different text, a disabled button, a profile preview, an HTTP status, or noticeably different timing.

This is a useful design-review question: what does the caller need to accomplish, and what does the interface reveal merely because it is convenient?

Move identity resolution to acceptance

An invitation workflow changes the order of proof.

The initiator provides an address and the intended scope. The system records an invitation and returns the same public outcome whether the address currently belongs to an account or not. Nothing attempts to answer the account-existence question.

Most importantly, the invitation itself grants no access.

When the recipient follows the invitation, they authenticate. The system compares the account’s confirmed address with the invited address. Only a match may inspect the offer and accept it. The access record is then created against the identity that actually accepted.

That ordering gives the workflow better semantics:

the initiator names a person in a form they genuinely know;
the system does not reveal whether an account already exists;
possession of a link is insufficient on its own;
the recipient proves control of the invited address;
access records consent rather than an assumption made by the initiator.

A token is necessary, but not sufficient

Invitation links are bearer secrets, so their handling deserves the same care as password-reset links.

Persist a one-way hash rather than the raw token. Submit the token in a request body when practical so routine URL logging does not repeat it across server, proxy, and browser records.

Bind the token to the intended identity as well. A forwarded or screenshotted link should not be enough to inspect the offer. Requiring a signed-in account with the matching confirmed address contains that failure mode.

Resending needs a clear contract: mint a new token and invalidate the old one. Otherwise each resend increases the live secrets while the interface still shows one invitation.

Finally, build the link from trusted server configuration. Request-controlled host information should not decide where a live token is sent.

Retry safety belongs in the workflow

Mobile networks fail at awkward boundaries. The server may commit an invitation and send the message while the response never reaches the device.

If pressing the button again creates a fresh invitation, one uncertain response can produce two live tokens. The interface may show only one of them.

A stable command identity lets the server recognize a replay and return the original result without sending again. That identity must describe the request as well as the button press: reusing it with different arguments should be rejected, not silently mapped to an unrelated result.

Delivery outcomes need honest language. A definite rejection differs from an unknown result after handoff. “Unknown” is safer than encouraging a duplicate.

Acceptance changes two facts: the invitation is consumed, and access is created.

Persist those facts together.

If access is written but the invitation remains pending, the token may be accepted again. That can create a second grant that nobody sees. Revoking the visible grant would leave the duplicate active.

One transaction makes the invariant explicit: there is no state in which acceptance succeeded while the invitation is still redeemable.

Re-check the initiator’s authority inside this transition too. Authority can change between invitation and acceptance. An invitation should not outlive the permission that allowed it to be created.

Test the negative space

The most valuable tests are not only happy-path checks. They prove what the workflow refuses to reveal or create:

known and unknown addresses produce indistinguishable initiator results;
an invitation alone authorizes nothing;
a different or unconfirmed account cannot inspect or accept it;
retrying does not send again;
resending invalidates the previous token;
stale initiator authority blocks new access;
repeated acceptance does not create duplicate grants;
consumption and access creation succeed or fail together.

The trade-off is deliberate complexity

This design adds states, expiry, delivery handling, retry receipts, token rotation, concurrency, and tests. It is unquestionably more work than a directory lookup followed by an insert.

The added complexity corresponds to uncertainty the simpler design hides. The invitation makes privacy, consent, identity proof, and atomicity explicit.

The practical takeaway is to inspect every workflow that translates a human-readable identifier into an account. Ask whether the caller truly needs account discovery, or whether identity can be resolved by the recipient at acceptance.

Often, the safer flow is also the more truthful one.

When the Tool Cannot See the Whole Model

Sun, 11 Oct 2026 06:55:17 +0000

Modular architecture creates an uncomfortable class of failure: the application can understand its complete model while design-time tooling sees only a legitimate subset.

That difference is easy to misread as drift. It is also dangerous to dismiss. The useful engineering move is to identify exactly what the tool cannot observe, then relocate each safety check to a boundary with enough context to answer the question correctly.

The warning was real, but its conclusion was wrong

Consider a shared EF Core context that can be extended by optional runtime modules. Each module contributes mappings without forcing the lower-level persistence package to reference every higher-level feature package. That dependency direction is valuable: the shared module remains reusable, and optional features remain optional.

At runtime, the host loads the contributing modules and the complete model is assembled. At design time, the tooling process starts from a narrower package. Loading one contributor directly would create a dependency cycle, so that contributor is absent.

The migration snapshot still represents the complete model. The design-time model does not. A comparison therefore reports a difference and may propose removing an object that is not obsolete at all; it is merely invisible from that process.

The warning accurately says, “these two views differ.” It cannot establish why they differ.

Separate database update from migration authoring

The first mistake would be treating every operation as though it asks the same question.

A database update applies migrations that have already passed review and validation. If a known model subset causes a framework warning to block that operation, a narrowly scoped suppression can be reasonable. The important word is narrowly. The exception belongs in the design-time path that has the structural blind spot, with an explanation of why the complete model cannot be assembled there.

Migration authoring is different. A scaffolder turns its limited view into proposed schema changes. If it cannot see an extension-owned table, a removal operation may look perfectly logical. Suppressing a warning does nothing to make that proposal safe.

So the second boundary needs a stronger invariant: inspect the generated migration operations and fail if they remove, rename, or destructively alter protected extension data. That converts a warning in prose into an executable gate.

This distinction matters. “Allow the known update path” and “trust every future generated migration” are not the same decision.

Keep a complete-model drift check

Once a noisy comparison is suppressed, the obvious risk is hiding real drift. The answer is not to pretend the partial view became authoritative. Keep a separate test that runs in a host where all contributors are loaded, builds the complete runtime model, and compares that model with the snapshot.

Now each check has one clear responsibility:

the design-time update path tolerates the known subset;
the full-model test detects genuine model-to-snapshot drift;
the migration-operation test blocks destructive scaffolding.

This is more precise than relying on one framework signal for three different questions.

Prove the guard is not vacuously green

Negative tests deserve a positive control.

Imagine a test that scans every migration and asserts that none removes a protected table. It passes. Good news, perhaps. But it also passes if migration discovery silently returns an empty collection.

Add a companion assertion that the scan finds the migration that originally created the protected object. If discovery breaks, the positive control fails. The pair proves both that the dangerous operation is absent and that the evidence set is present.

This pattern generalizes beyond migrations. Any test that says “no forbidden item exists” should make you ask whether it would still pass when the scanner found nothing.

Shared process state is another hidden boundary

The same change exposed a smaller reliability lesson. Tests for design-time factories temporarily set a process-wide configuration value. Separate test classes ran concurrently, and one could restore the value while another was still using it. Each class passed alone; together they failed intermittently.

The repair was not another retry. The tests were placed in one non-parallel collection because they share mutable process state.

That is the same ownership principle in miniature: the scope of the resource determines the scope of coordination. A process-wide variable cannot safely be treated as test-local state.

The trade-off is explicit complexity

This approach is not free. There is now a documented exception, a complete-model drift test, a migration-operation guard, a positive control, and serialized tests around shared environment state. Future maintainers must understand why those pieces exist.

But the alternative is false simplicity: a generic warning that blocks legitimate work, generated operations that can target valid data, and flaky tests that disguise the real contract.

The practical rule I keep is this:

When tooling has a structural blind spot, narrow the exception and move safety to executable checks that can observe the complete truth.

Do not silence evidence broadly. Do not force an architectural dependency merely to satisfy a tool. Give each check the context it needs, and make dangerous assumptions fail before they become database changes.

42 of our 71 regression guards stayed green when we broke the code on purpose

Sat, 10 Oct 2026 19:24:40 +0000

We build an AI virtual staging tool: a user uploads a photo of a room, an image model restyles it, and a set of checks compares the result with the original before anyone sees it. We wrote about those checks in a previous post. This one is about the layer under that: the scripts that make sure the checks, the billing rules and the consent logic stay the way we fixed them.

We call them guards. Each is a standalone Node script, .mjs or .mts, that asserts one behaviour and exits non-zero if it is gone. A fix ships with a guard. By late August we had about seventy, run together with npm run qa.

Then we asked each one a simple question: if we break the thing you protect, do you go red? For 42 of 71, the answer was no.

How we got there

The first problem was not quality but running them at all. Until 22 August there was no runner; the list lived in a Markdown file that named 33 of the 78 scripts on disk. A runner that compares its list with the files actually present found one guard that was red (15 of 18 checks) with nobody noticing, 27 live guards that had not been run for months, and nine that were on disk but not in git.

Now everything ran, and everything was green. That felt like proof. It was not.

Two days later an independent review tried something small: in the error-tracker configuration it renamed the environment variable that carries the DSN, the kind of typo you make during a rename. All 55 checks of the guard watching that configuration stayed green. In production that typo means the error tracker is silently a no-op, the exact failure the guard existed to prevent.

That was reason enough to do it to every guard.

The audit

Method, by hand, one guard at a time: edit the production file so the guarded behaviour is really broken, run the guard, restore the file, write down the numbers. A hole counted only if the guard stayed green on a real break.

Result: 42 holes across 71 guards, and 42 guards that stayed green on at least one real break and had to be rewritten (35 in the first batch, 7 of the last eight). These are two separate counts that happen to match: one guard can hide more than one hole, and a few weak guards only showed up while we were fixing the others.

Group	Guards	Holes
Error tracking, observability, infrastructure	16	7
Image engine and render pipeline	19	5
Billing, consent, privacy, SEO	28	21
The last eight (older zones, referrals, video, pins)	8	9 in 7 files
Total	71	42

Some individual findings:

One guard had no checks at all. It printed a green tick and exited 0 whatever the code looked like.
One check compared the position of two strings in a route file to assert that storage blobs are deleted before database rows (rows first would orphan a customer's files for good). The first string had been removed by a refactor more than three weeks earlier. indexOf returned -1, -1 is less than anything, and the check had passed every run since.
One guard tested a hand-written copy of the SQL instead of the production webhook.

The pattern behind most of the 42: the guards were written against text, not behaviour. We read the route file as a string and looked for names. Such a guard fails both ways: an honest refactor turns it red for nothing, and a real break slides past because the names are still there.

One guard, before and after

The consent route does three things in order: write the user's choice to the auth provider, append to the audit log, unpublish their renders from the public showcase. The guard checked that order with indexOf on the source. Nine checks, all green. The mutation that beat it, and the rewrite:

// production: awaited; a failure returns 500 to the user
await client.users.updateUserMetadata(userId, { ... });
// mutation: same call, same position in the file - all nine checks still green
void client.users.updateUserMetadata(userId, { ... }).catch(() => {});

// rewritten guard: cut out the statement itself, not a name near it
const i = ROUTE.indexOf("client.users.updateUserMetadata");
const stmt = ROUTE.slice(ROUTE.lastIndexOf("\n", i) + 1, ROUTE.indexOf(";", i) + 1).trim();
ok("write is awaited, not fired in the background", /^await client\.users\.updateUserMetadata\(/.test(stmt));
ok("a failed write is not swallowed", !/\.catch\(/.test(stmt));
ok("and not silenced with void", !/^void\b/.test(stmt));

Under the mutation the order of the three calls did not move, so the old checks passed. The meaning was gone: the write is fire-and-forget, the failure is swallowed, the audit log records "revoked", the user believes they opted out, and the showcase keeps publishing because the provider still says "allowed".

The rewrite is still text, but the text of the statement itself, and the mutation that defeated the old version is now a permanent check. Where the code can be imported, we go further: the guard loads the real function through a loader and runs it over a table of inputs.

The runner

Every one of those experiments was a throwaway script, fifty lines each, and each hit the same traps: CRLF on the Windows working copy versus LF in git, a snippet that matched twice, a "mutation" that deleted a line and could not be reverted by an empty pattern (one stayed in the code and was caught on a second pass).

On 28 August it became one tool. A job is JSON: the guard command plus named substitutions.

{
  "guard": "node tmp/qa/geometry-escalation-proof.mjs",
  "mutations": [
    {
      "file": "src/lib/draft-rank.ts",
      "name": "a judge that errored no longer ranks last - an empty verdict beats a checked image",
      "from": "  if (input.errored) return Number.MAX_SAFE_INTEGER - 1;\n",
      "to":   "  void input.errored;\n"
    },
    {
      "file": "src/app/api/generate/route.ts",
      "name": "escalation became unconditional - furniture-placement failures go to the expensive engine too",
      "from": "if (!economyMode && lastGeometryBroken && attempt < MAX_ATTEMPTS) {",
      "to":   "if (!economyMode && attempt < MAX_ATTEMPTS) {"
    }
  ]
}

The loop is short; the rules around it are what we paid for:

if (!runGuard()) fail("guard is already red; it proves nothing");          // 1. green before

for (const m of mutations) {
  if (!m.to.trim()) { bad(m, "empty replacement is a deletion, not a mutation"); continue; } // 2
  const src = original(m.file);
  const hits = src.match(rx(m.from, "g"));                      // rx() tolerates \r?\n  // 3
  if (!hits || hits.length !== 1) { bad(m, "snippet must occur exactly once"); continue; } // 4
  fs.writeFileSync(m.file, src.replace(rx(m.from), m.to));
  const stillGreen = runGuard();
  fs.writeFileSync(m.file, src);
  stillGreen ? bad(m, "GUARD STAYED GREEN") : good(m);
}
assertEveryFileRestoredByteForByte();                                         // 5

Mutations are named and semantic on purpose. Classic mutation testing flips operators automatically; we write each mutation as a sentence about how a customer would get hurt, because that sentence also documents why the guard exists. The trade-off is coverage: we test only the breaks we thought of.

What the mutations found in production code

The day the runner was born, a run on a new guard did something we did not expect. The function that decides whether a payment-return URL is stale had an explicit "payment time is in the future" branch. Six mutations turned the guard red. The seventh, removing that branch, changed nothing: a future timestamp gives a negative age, and a negative age is never above the threshold. The branch was a crutch the guard could not tell from working code. We removed it and wrote the reason above the function.

In the same session, 35 mutations across seven fixes found three guards checking the wrong thing: a parser that stopped at the first closing brace and broke on a nested object (four of six mutations passed on obviously broken code); a "the catch block logs" check that stayed green when the log left one branch because a neighbouring branch still had one; and a scanner for prototype-key guards that cleared a whole file if the guard appeared anywhere in it.

Where it stands

93 mutation jobs, 867 mutations, in the repo next to the guards.
171 guards in the runner's list, 12 named skips (they cost money, run on a calendar or need a fresh build); the list is compared with the disk on every commit.
Written rule: a new guard without a mutation run counts as not written.
The pre-commit hook checks only the registry, in 0.08 seconds. The full run takes minutes, needs the network and a database, and can go red from someone else's outage; a hook like that gets --no-verify on the first urgent fix, and a bypassed hook is worse than none. The full run stays manual and mandatory before a merge.

What still fails

Hand-written mutations come from the same person who wrote the guard, with the same blind spots. On 5 October a mutation named "only seven articles left in the links block" removed 5 of 17 links and the guard stayed green. The mutation was too weak, not the guard; it now removes every link. Nothing finds weak mutations except somebody noticing.

Many guards still read source text. Executed guards are better, but much of our logic sits in route handlers tied to auth and a database, and extracting it just to test it is a refactor we have not earned yet. Text guards are honest only as long as their mutation set is.

There is no CI. The guards run on a laptop before a merge.

And the first finding of all: 42 of 71 is not a number about carelessness. Every guard was written in good faith after a real bug. A green check is a claim, and until something tries to make it red, it is an unverified one.

We build this at AI Flip Room.

Correction, 11 October: an earlier version said 44 of 70. The table adds up to 71 guards and 42 holes, and 42 guards had to be rewritten; the headline mixed two counts. Thanks to a reader on DEV for doing the sums.

The Async Result Belongs to the Snapshot That Started It

Sat, 10 Oct 2026 06:01:28 +0000

Modern client components are long-lived, reactive, and full of asynchronous work. That combination creates a subtle class of defect: a response can be factually correct for the request that produced it, yet wrong for the screen that exists when it arrives.

This is especially important when a component mixes route parameters with identity-scoped state. The user can navigate while a request is in flight. Authentication can resolve, refresh, or change. The component instance may survive both events. Unless the component tracks which context owns each result, an older task can update a newer view.

The core rule is simple:

An awaited result belongs to the input snapshot that started it.

That snapshot is more than a component reference. It can include the current route value, the current authentication-state operation, and a monotonically increasing generation number.

How the race forms

Imagine a detail page that loads an item, then asks whether the current viewer has reacted to it. The page shows a highlighted action when the answer is yes.

The component begins loading item A for identity A. Before the viewer-specific request returns, the user signs out and another identity becomes current—or the same component navigates to item B. The old request then completes with a truthful answer: identity A did react to item A.

If the continuation writes directly to a shared isActive field, that truth is now displayed in the wrong context. Item B may appear active, or identity B may see state derived from identity A.

This is not merely a cosmetic race. Identity-derived state can influence which actions are enabled, what ownership badges appear, and which command a user can submit next.

Make ownership explicit

A practical pattern is to treat every meaningful input change as a new generation.

When a route or authentication input changes:

Increment the generation.
Capture the route and authentication input in local variables.
Clear state derived from the previous identity.
Disable identity-scoped actions while the new identity is unresolved.
After each await, confirm that the generation and captured inputs are still current.
Apply results only when those checks pass.

The checks matter after every awaited boundary, not only at the end. A component may load the main item, related content, entitlement state, reaction state, and cart state in sequence. Any await is a point where navigation or identity can change.

Cancellation tokens can reduce wasted work, and they are worth using when the underlying operation supports cancellation. But cancellation alone is not the ownership proof. Cancellation can race with completion, and some dependencies cannot cancel promptly. The continuation still needs to ask, “Does this result belong to the current snapshot?”

Fail closed while identity is unknown

There is another important state between old identity and new identity: unresolved identity.

Keeping the old viewer-specific UI visible during that interval is convenient, but unsafe. The component should clear ownership badges, active reactions, and similar derived state before awaiting the new authentication result. Any action that depends on identity should be disabled until the new snapshot is ready.

That brief disabled state is honest. The application does not yet know which identity owns the next command.

This principle also prevents an action from being sent using newly refreshed credentials while the visible UI still describes the previous viewer. Read state and write authority move together.

Test the interleaving, not only the outcome

Ordinary happy-path tests rarely expose this defect because requests finish in the same order they started.

A stronger component test controls the schedule:

Start a viewer-specific read and hold its response.
Change the route or authentication identity.
Wait for the new screen to settle.
Release the old response.
Assert that the current screen does not adopt the old result.

A separate test should hold authentication unresolved and verify that identity-scoped actions remain disabled. This proves that the UI fails closed during the transition.

There is also a useful counter-test: rerender with exactly the same inputs while a valid request is pending. The component should not discard or duplicate work merely because the framework rendered again. The boundary is a meaningful input change, not every render cycle.

These tests are valuable because they verify time and ownership, not just a final value. They turn a race that might appear once in a thousand interactions into a deterministic scenario.

The trade-off

Generation markers, captured snapshots, and repeated currentness checks add code. Temporarily disabling an action can introduce a small pause. Overusing the pattern can also make a component difficult to read.

The answer is not to attach a counter to every asynchronous line. Use the pattern where results are scoped to changing inputs: routes, selected records, authentication identity, workspace, locale, or another boundary that can change while the component remains mounted. Extract a small helper when the same ownership rule appears repeatedly.

The payoff is larger than visual correctness. Explicit async ownership reduces stale-route updates, prevents state from crossing identity boundaries, and stops commands from being enabled under an unresolved context.

The humble takeaway is this: asynchronous correctness is not only about receiving the right answer. It is also about proving that the answer still belongs here.

Why tests should verify runtime contracts, not inferred types

Fri, 09 Oct 2026 19:59:18 +0000

If your Node tests are green but callers still need casts, you’re mixing two different guarantees. A Node test proves what the function did when executed: the values returned, whether the input was mutated, and which inputs throw. It does not and cannot prove how the TypeScript compiler will narrow that result at a later use site.

Implementation summary

Keep the runtime contract explicit: the helper should validate inputs (e.g., reject non-arrays) and return a new array that removes only null and undefined (use a nullish check rather than truthiness). The server-inserted example shows a small function that performs exactly that behavior and does not mutate the input.

Expected behavior to test

Mixed-array case: the output contains the original non-nullish items in order and preserves falsy-but-valid values like 0, '', and false.
Empty input: returns a new empty array (not the same reference).
All-nullish input: returns an empty array.
Invalid input: throws a TypeError for non-array arguments.

Where static checking belongs

If you need the helper to provide compiler-level narrowing (so consumers can assign without casts), verify that with the TypeScript compiler or editor diagnostics. A minimal compile-time check is to write a small use-site that only accepts the narrowed type and run tsc to ensure it typechecks.

Limitations and pitfalls

Compiler inference changes across TypeScript versions (e.g., TypeScript 5.5 adds inferred predicates in some filter callbacks). Relying on runtime tests to encode compiler behavior makes tests brittle or misleading.
Truthiness-based filters (e.g., !!x) can silently drop legitimate falsy values — runtime tests must check exact outputs, not just lengths.

Practical rule

Tests: prove the runtime contract you actually depend on. Compiler: prove the static narrowing your consumers rely on. Keep them separate so failures point to the right fix: change JS behavior, adjust types, or update consumers.

The article includes a repaired example and concrete test cases that illustrate this split and avoid overclaiming what a JavaScript test can prove. The server will append the full code example and the article URL.

Working example

export const note = `This module demonstrates runtime removal of null and undefined from arrays.
Runtime tests (node:test) validate behavior at execution time but do NOT prove any TypeScript compile-time narrowing. To check static narrowing you must run the TypeScript compiler (tsc) or use editor/IDE type checks.`;

/**
 * Remove null and undefined from an array while preserving order and other falsy
 * values such as 0, '', and false.
 *
 * Runtime contract: returns a new array containing only elements that are not
 * null and not undefined. Throws a TypeError if the argument is not an Array.
 *
 * Note: This demonstrates runtime behavior only. TypeScript's ability to
 * narrow types (e.g. from (T | null | undefined)[] to T[]) is a compile-time
 * property checked by tsc / the language service and is not proven by these
 * runtime tests.
 */
export function filterNonNullish(arr) {
  if (!Array.isArray(arr)) {
    throw new TypeError('filterNonNullish expects an Array');
  }

  return arr.filter((x) => x != null);
}

Read the full article: https://chriseugenerodriguez.com/blog/tests-should-verify-runtime-contract-not-inferred-types

How we check every AI-generated room render before a user sees it

Fri, 09 Oct 2026 18:55:25 +0000

Image models are very good at restyling a room and quietly bad at leaving the room alone. Ask for a Japandi living room and you get beautiful furniture, and sometimes also:

a window that moved, shrank, or appeared over a kitchen sink where the photo shows a solid wall;
a closet door that came back as a curtain, a sliding screen, or plain wall;
a sofa parked across the only archway;
a new doorway at the frame edge, or a niche carved into a flat wall;
a camera that slid sideways;
or the opposite: a picture that is almost exactly the photo you uploaded.

Our users put these images in property listings, where a staged photo may change the furniture but not the windows, walls or doors (why that line exists). So every render is checked automatically before anyone sees it. Here is how, and what building it taught us.

The pipeline at a glance

For an indoor room, each render goes through three gates in parallel:

A vision-model judge compares the original photo with the render and answers a fixed list of yes/no questions.
A mechanical camera check measures whether the viewpoint moved, with no language model involved.
A crop-level presence check looks at each door, window and staircase found in the original, at full resolution.

If all three pass, the render is served. If not, we retry. If nothing passes, the user gets the best attempt, clearly marked as a draft.

Gate 1: a judge that only answers booleans

The judge gets the original and the render, both downscaled, and must return a fixed JSON object. We use structured output with a schema, so it cannot reply in prose. An earlier version asked for "only JSON" and scraped it out with a regex; when the output got long or cut off, the parse failed and the gate silently did nothing.

Indoors there are 16 checks, outdoors 9. Indoors they cover:

Openings: every door is still a door you could open; every passage has clear floor to walk through.
Windows: same count, position, size and shape; none invented; not hidden behind a tall wardrobe; the same view outside.
The room shell: ceiling, fireplace, no carved niches, features on the same walls, solid frame edges, no new rooms or corridors.
Finish consistency: no half-repainted rooms.
Content rules: no people, and no photo or realistic portrait of a person in the wall art.
Camera: same viewpoint, same room depth.

Outdoors: buildings, fence line and gates, terrain (no new pool), mature trees, background, and a usable driveway.

A simplified, illustrative verdict:

{
  "doors_preserved": true,
  "openings_clear": false,
  "windows_ok": true,
  "window_unblocked": true,
  "walls_uniform": true,
  "frame_edges_solid": true,
  "no_faces_in_art": true,
  "fail_reasons": ["sofa end stands on the archway floor"]
}

A render passes only if every counted boolean is true; the reasons feed logs and the retry prompt. Most of the work went into the wording:

Say what must pass. Restyling is the product, so the instructions list what is expected (new floors, paint, wallpaper, fixtures) and say "unsure means pass". The exception is a short list where unsure means fail, led by a window that may not have been in the original. An invented window is the worst defect a listing photo can have.

One question per field. While the frame-edge rule was a sub-clause of a broader "no phantom architecture" check, new doorways at the frame edge got through. Its own boolean made the model actually look.

Naming things primes the judge. An early niche check listed which shelves were allowed, and the judge started flagging every shelf. The current wording is purely geometric: is this flat wall still one plane?

Allow "nothing to compare". The "same view through the window" check kept firing on originals with blinds drawn or glass blown out to white. That case is now an explicit pass.

Wording is measurable. The old wall-art wording wrongly failed 10 of 16 good frames (stylised line drawings counted as faces). The rewrite failed 0 of 16 and still caught all 5 real portraits.

Gate 2: measuring the camera instead of asking about it

Early on, the vision model was close to blind to viewpoint changes: renders shot from a visibly different spot passed every neural check. Later, a stronger judge had the opposite problem: on 43 renders where the camera had not moved, it said it had 13 times. So indoors the judge's camera answers no longer decide pass or fail. A measurement does, with two channels:

Ceiling line. We trace the wall/ceiling boundary as a curve (dynamic programming over the vertical brightness gradient) and compare the original's curve with the render's.
Depth profile. A monocular depth model estimates both images. Per column we take how far the far wall reads, then compare the profiles with a robust fit, a rank correlation and a search for a sideways shift.

Each channel is fooled by something different. Paint fools the ceiling line (dark beams, art on a mantel). Glass fools depth (windows and mirrors are estimated differently each time). Paint is flat in depth, and glass does not move the ceiling line, so we flag a shift only when both agree: the minimum of the two normalised scores.

On 24 hand-labelled pairs from two rooms, this caught 5 of 5 real shifts with 0 false flags. The margin was thin, so a grey zone around the threshold is logged, not failed. Outdoors there is no ceiling to trace, so the mechanical check is off and the judge's camera question stays in charge.

Gate 3: crops, because a door is 80 pixels wide

At the judge's resolution a front door is a thin strip. So doors, windows and stairs are detected at upload, and after each render a model is asked about each crop at full resolution: is it still there? A mechanical channel also measures whether a window drifted or widened. On 87 labelled pairs, the crop judge caught 7 of 10 broken renders with no false alarms on the 75 clean ones; the mechanical channel caught the other 3.

When a render fails

Retries depend on the kind of defect.

Placement defects (a blocked passage, a face in the art, half-painted walls) are retried on the same image model with the judge's reasons added to the prompt.
Geometry defects go to a second, pricier image model. We measured a same-model retry fixing broken geometry about one time in five. In an earlier bake-off the two models were clean on 18 and 17 of 20 renders, and 19 of 20 together: their failures barely overlap.

The geometry retry gets a fixed, count-based instruction ("same set of openings, not one more, not one less"), never the judge's description of the defect. When a retry prompt named an arched opening at the left edge, the retry model painted exactly that.

If nothing passes, the user still gets the best attempt, under a "rough draft" banner. On paid plans the first four rejected drafts in a rolling week don't use a render; the free plan gets one. The cap is deliberate: a draft is a real, downloadable image, so unlimited forgiveness would be a free image farm.

The checks fail open. If the judge itself errors or times out, the user is not blocked: the image is served flagged, and an unverified render is not charged as a draft.

The near-copy trap

At first, "best attempt" meant the one with the fewest failed checks. The attempt that breaks the least is often the one where the model barely did anything. In one real case the first attempt was a convincing industrial restyle that failed on geometry, and the user was served the second: their old room, old wallpaper and all, plus a pair of lamps. The ranking was paying the model for doing nothing.

The fix is a change meter: shrink both images to a tiny thumbnail and count how much of the frame visibly changed. No model call, and we calibrated it on 110 real renders and 10 synthetic near-copies (recompressed, blurred, brightened, shifted, a couple of lamp-sized patches). The ranking now:

// simplified
function draftRank(a: Attempt): number {
  if (a.checkerErrored) return LAST;
  let rank = a.failedChecks.length;
  if (a.hasPeopleOrRealFacesInArt) rank += HARD_BAN;
  if (a.isNearCopy && a.intensity !== "light") rank += NEAR_COPY;
  return rank; // lowest is served
}

A light restage is exempt, because there the user asked us to keep the room.

Picking the judge model with a calibration set

The judge runs on every attempt, so a cheaper model is tempting. We keep a calibration set of 31 hand-labelled frames and run every candidate on it twice. A cheaper model at under half the price per call missed 6 defects per run against our current judge's 3. On 46 live renders with 8 defects marked by eye, it caught 0 and 1 across two runs; the current judge caught 6. A newer model wasn't clearly better either: fewer misses (2 vs 3), twice the false alarms (4 vs 2). We kept the current judge.

Takeaways

Ask the judge narrow yes/no questions, not one "is this good?".
Where geometry can be measured, measure it, with channels whose failure modes don't overlap.
Calibrate every wording change and model swap on a labelled set, twice.
Design the failure path as carefully as the check: ranking, draft label, who pays.
"Least broken" can mean "least changed". Measure change directly.

We build this at AI Flip Room.

Mi Definition of Done: 5 gates que convierten "creo que funciona" en "puedo demostrarlo"

Thu, 08 Oct 2026 07:28:13 +0000

Mi Definition of Done: 5 gates que convierten "creo que funciona" en "puedo demostrarlo"

Nota honesta: esto no es un producto, es una mini-app de uso personal en mi Linux Mint. Funciona y la uso a diario; su adaptación para terceros la planifico para los próximos días.
Read time: ~8 min · Etiquetas: python security testing opensource

Un commit. Una hora. Y no es un juguete

El proyecto del que hablo hoy tiene un solo commit y tardó una hora en funcionar de punta a punta. Lo uso cada mañana. Y no es un juguete.

Lo digo porque la pregunta interesante no fue cómo lo programé rápido, sino cómo supe que estaba terminado. Ese es el punto: "terminado" no es una sensación, es una decisión que puedes defender. Cuando esa decisión tiene una definición clara, programar una hora deja de ser un atajo y pasa a ser consecuencia.

Este post es esa definición. La llamo Definition of Done y son cinco gates.

Qué hace, sin tecnicismos

Imagina un llavero. Guarda tus claves API y credenciales cifradas dentro de tu propio ordenador y se las sirve a tus aplicaciones durante toda la sesión después de un único desbloqueo al iniciar sesión. Nada sube a la nube. No hay telemetría. No hay cuenta que crear.

En la práctica, mis scripts y programas ya no llevan claves escritas dentro ni dependen de un .env en claro: consultan el llavero, que ya está abierto por mí, y arrancan sin volver a pedirme nada. Si me voy, el llavero se cierra; las claves desaparecen de la memoria.

Se llama pass-encrypt-env. Corre en mi Linux Mint y, hoy, es para mí.

Lo que es hoy (y lo que todavía no)

Ser claro con el estado real es parte del método. Un proyecto "terminado" también declara sus límites.

Es: una herramienta personal, funcional, que uso a diario en mi equipo.
No es: un producto instalable por cualquiera sin fricción, ni algo auditado por terceros.
Viene: en los próximos días planifico la adaptación —empaquetado, arranque en otras distribuciones y documentación para quien no sea yo— como un esfuerzo aparte y planificado.

Y hay una razón de fondo para separar esa adaptación del desarrollo: no quiero declarar "lista" una app porque pase en mi máquina. La declaro lista cuando pasa los gates en cualquier máquina. Es exactamente la misma lógica que un buen test: no vale que funcione una vez, tiene que fallar cuando corresponda.

Por qué exijo evidencia y no sensaciones

Soy relativamente nuevo en el mundo de la IA agéntica, aunque llevo años gestionando sistemas bajo protocolo: control de accesos, mínima exposición, registros. Y una disciplina que aprendí ahí —y que el prompting me recordó— es esta:

No das algo por resuelto porque parezca resuelto; lo das por resuelto cuando puedes demostrarlo.

El hábito de protocolo es la mínima exposición: asumes que algo fallará y diseñas para que, cuando falle, no arrastre contigo todo lo demás. Aplicado al software, eso se traduce en una Definition of Done: una lista corta de condiciones que convierten "yo creo que funciona" en "aquí está la prueba".

En este proyecto, esa lista lleva cinco puertas.

Los 5 gates

Cada gate existe porque previene un fallo concreto, no por completitud.

1) Tests que pueden fallar

No me sirve "tengo tests"; me sirve que fallen cuando rompo algo. En pass-encrypt-env hay 19 tests y uno de los que más valor tiene es este, porque vigila una promesa sobre el disco:

def test_no_plaintext_on_disk(tmp_path):
    path = tmp_path / "vault.enc"
    v, unlocked, _ = _new()
    vault.set_secret(unlocked.payload, "TOKEN", "super-secret-value")
    vault.reseal(v, unlocked.dek, unlocked.payload)
    vault.save_vault(v, path)

    blob = path.read_text()
    assert "super-secret-value" not in blob          # el secreto no está en claro
    assert (path.stat().st_mode & 0o777) == 0o600     # y el fichero no es legible por terceros

Ese test no comprueba que "va bien": comprueba que la promesa del producto se cumple. El fallo que previene es el peor de todos: creerte seguro y no serlo.

2) Lint limpio

ruff check .

No es estética. Un lint en verde elimina de antemano la clase de error más tonta —imports sin usar, variables fantasma, comparaciones sospechosas— antes de que la ejecutes. Es ruido que no llega a producción porque ni siquiera llega al commit.

3) Escaneo de seguridad

bandit -q -c bandit.yaml -r src

bandit busca patrones inseguros en el código: criptografía mal usada, subprocess peligrosos, aleatoriedad no criptográfica. En una app cuyo único trabajo es guardar secretos, este gate no es opcional. El fallo que previene es el uso accidental de una primitiva parecida a la correcta.

4) CI reproducible

Los tres gates anteriores corren solos en GitHub Actions, sobre Python 3.11 y 3.12:

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.11", "3.12"]
steps:
  # ...
  - name: Gate (ruff + bandit + pytest)
    run: ./scripts/ci.sh python

Y aquí está la parte importante: ese ./scripts/ci.sh es el mismo comando que ejecuto en local. No hay una verdad para mi máquina y otra para el servidor. El fallo que previene este gate es "en mi máquina funciona", que no es un bug: es una mentira cómoda.

5) Smoke end-to-end

Los unit tests verdes no garantizan que el sistema entero arranque. Para eso hay un smoke test headless que levanta el agente, habla con él por su canal interno y comprueba el comportamiento observable:

python3 -m pass_encrypt_env.cli env --if-unlocked >/dev/null 2>&1 || rc=$?
echo "env --if-unlocked (bloqueada) -> exit $rc (esperado 3)"
[ "$rc" -eq 3 ]

Aun con piezas perfectas, el conjunto puede no montarse. El smoke test es el gate que mira el sistema, no las partes.

La meta-regla

Todos los gates caben en un comando:

./scripts/ci.sh    # esto corre en local y esto corre en CI. La misma verdad.

Qué protege y qué no

Un proyecto terminado con honestidad publica su techo, no lo esconde. Este es el mío:

Amenaza	¿Mitigado?
Robo offline de `vault.enc`	Sí — sin master + TOTP no hay datos
Manipulación del fichero	Sí — AEAD detecta la alteración
Alguien sin tu TOTP	Sí — segundo factor obligatorio
Pérdida de master o TOTP	Sí — clave de recuperación
Proceso del mismo UID con la bóveda abierta	No — techo del diseño cero-prompt
`root` / malware ya dentro	No — fuera de alcance

Que la última fila diga "No" no es una debilidad: es una promesa cumplible. Prefiero un alcance pequeño y cierto que uno grande y falso.

Detalle técnico (para quien quiera el fondo)

El "cómo", para quien lea esto con la ingeniería delante:

Cifrado en reposo: vault.enc con Argon2id como función de derivación (64 MiB, 3 pasadas, 4 lanes) y AES-256-GCM como cifrado autenticado (confidencialidad e integridad).
Doble envoltura de la clave de datos (DEK): K1 = Argon2id(master) · K2 = HKDF(K1 ‖ totp_secret) abre la DEK · K3 = HKDF(recovery_key) guarda una copia para recuperación. El secreto TOTP viaja cifrado: sin un TOTP válido no se obtiene la DEK.
Agente de sesión: escucha en un socket Unix en $XDG_RUNTIME_DIR/passenv/, permisos 0600 y validación de UID del par (SO_PEERCRED). Se endurece con PR_SET_DUMPABLE=0 (anti-ptrace), sin volcados de memoria y mlockall best-effort.
CLI passenv: init, set/get/list/rm, export, exec, change-master, backup, restore.
Modelo de amenazas completo en docs/THREAT_MODEL.md, con el techo del diseño cero-prompt declarado a propósito.

Lo que esto cambia (Takeaway)

El gate no es un freno; es lo que te deja pisar el acelerador sin miedo. Un proyecto de una hora y un commit no llega "terminado de verdad" por suerte: llega porque las condiciones para llamarlo terminado ya estaban escritas antes de empezar a escribir código.

"Creo que funciona" es una sensación. "Aquí está la prueba" es un proyecto.

La checklist

- [ ] Tests que fallan cuando rompo algo (no que existan)
- [ ] Lint limpio
- [ ] Escaneo de seguridad sin hallazgos
- [ ] CI reproducible (mismo comando en local que en remoto)
- [ ] Smoke end-to-end en verde

Y tú

¿Qué requisitos le exiges tú a un proyecto para darlo por terminado? ¿Cuál añadirías a mis cinco gates? Y, ya que el mío está en estado de uso personal: ¿tú publicarías una herramienta así o la dejarías privada hasta "adaptarla"? Cuéntamelo en los comentarios; leo y respondo a todo.

_{English version below · El original está en español. If you're an English speaker, scroll down to the translation and tell me what your own Definition of Done looks like. I reply to everyone.}

My Definition of Done: 5 gates that turn "I think it works" into "here's the proof"

Honest note: this isn't a product, it's a personal-use mini-app on my Linux Mint. It works and I use it daily; adapting it for other people is a planned effort for the coming days.
Read time: ~8 min · Tags: python security testing opensource

One commit. One hour. And it's not a toy

The project I'm talking about has a single commit and took one hour to work end to end. I use it every morning. And it isn't a toy.

The interesting question wasn't how I built it fast, but how I knew it was done. That's the point: "done" isn't a feeling, it's a decision you can defend. Once that decision has a clear definition, writing code for an hour stops being a shortcut and becomes a consequence.

This post is that definition. I call it a Definition of Done, and it's five gates.

What it does, without jargon

Picture a keyring. It keeps your API keys and credentials encrypted inside your own computer and serves them to your applications for the whole session after a single unlock at login. Nothing goes to the cloud. There's no telemetry. There's no account to create.

In practice, my scripts and apps no longer carry keys inside them or depend on a plaintext .env: they ask the keyring, already unlocked on my behalf, and start without asking me again. If I leave, the keyring closes; the keys disappear from memory.

It's called pass-encrypt-env. It runs on my Linux Mint and, today, it's for me.

What it is today (and what it isn't yet)

Being clear about the real state is part of the method. A "finished" project also declares its limits.

It is: a personal, working tool that I use daily on my machine.
It isn't: something anyone can install without friction, nor something audited by third parties.
It's coming: in the coming days I'm planning the adaptation — packaging, booting on other distributions and documentation for someone who isn't me — as a separate, planned effort.

There's a deeper reason to separate that adaptation from development: I don't want to call an app "ready" because it passes on my machine. I call it ready when it passes the gates on any machine. It's the same logic as a good test: it doesn't count if it works once, it has to fail when it should.

Why I demand evidence, not feelings

I'm relatively new to the world of agentic AI, though I've spent years managing systems under protocol: access control, minimal exposure, logs. And one discipline I learned there — which prompting reminded me of — is this:

You don't call something solved because it looks solved; you call it solved when you can prove it.

The protocol habit is minimal exposure: you assume something will fail and you design so that, when it fails, it doesn't drag everything else down with it. Applied to software, that becomes a Definition of Done: a short list of conditions that turn "I think it works" into "here's the proof".

In this project, that list has five doors.

The 5 gates

Each gate exists because it prevents a concrete failure, not for completeness.

1) Tests that can fail

I don't care about "I have tests"; I care that they fail when I break something. pass-encrypt-env has 19 tests, and one of the most valuable guards a promise about the disk:

def test_no_plaintext_on_disk(tmp_path):
    path = tmp_path / "vault.enc"
    v, unlocked, _ = _new()
    vault.set_secret(unlocked.payload, "TOKEN", "super-secret-value")
    vault.reseal(v, unlocked.dek, unlocked.payload)
    vault.save_vault(v, path)

    blob = path.read_text()
    assert "super-secret-value" not in blob          # the secret isn't in cleartext
    assert (path.stat().st_mode & 0o777) == 0o600     # and the file isn't readable by others

That test doesn't check that "it works": it checks that the product's promise holds. The failure it prevents is the worst of all: believing you're safe when you aren't.

2) Clean lint

ruff check .

It isn't aesthetics. A green lint removes the dumbest class of error —unused imports, ghost variables, suspicious comparisons— before you even run it. It's noise that never reaches production because it never reaches the commit.

3) Security scan

bandit -q -c bandit.yaml -r src

bandit looks for insecure patterns in the code: misused cryptography, dangerous subprocess, non-cryptographic randomness. In an app whose only job is to store secrets, this gate isn't optional. The failure it prevents is the accidental use of a primitive that looks like the right one.

4) Reproducible CI

The previous three gates run on their own in GitHub Actions, across Python 3.11 and 3.12:

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.11", "3.12"]
steps:
  # ...
  - name: Gate (ruff + bandit + pytest)
    run: ./scripts/ci.sh python

And here's the important part: that ./scripts/ci.sh is the same command I run locally. There's no one truth for my machine and another for the server. The failure this gate prevents is "it works on my machine", which isn't a bug: it's a comfortable lie.

5) End-to-end smoke test

Green unit tests don't guarantee the whole system boots. For that there's a headless smoke test that starts the agent, talks to it over its internal channel and checks the observable behavior:

python3 -m pass_encrypt_env.cli env --if-unlocked >/dev/null 2>&1 || rc=$?
echo "env --if-unlocked (locked) -> exit $rc (expected 3)"
[ "$rc" -eq 3 ]

Even with perfect pieces, the whole may fail to assemble. The smoke test is the gate that looks at the system, not the parts.

The meta-rule

Every gate fits in one command:

./scripts/ci.sh    # this runs locally and this runs in CI. The same truth.

What it protects and what it doesn't

An honestly finished project publishes its ceiling, it doesn't hide it. This is mine:

Threat	Mitigated?
Offline theft of `vault.enc`	Yes — without master + TOTP there's no data
File tampering	Yes — AEAD detects the alteration
Someone without your TOTP	Yes — second factor required
Loss of master or TOTP	Yes — recovery key
A process of the same UID with the vault open	No — ceiling of the prompt-less design
`root` / malware already inside	No — out of scope

That the last row says "No" isn't a weakness: it's a promise you can keep. I'd rather have a small and certain scope than a large and false one.

Technical detail (for those who want the depth)

The "how", for whoever reads this with the engineering in front of them:

Encryption at rest: vault.enc with Argon2id as the key-derivation function (64 MiB, 3 passes, 4 lanes) and AES-256-GCM as authenticated encryption (confidentiality and integrity).
Two-layer wrapping of the data key (DEK): K1 = Argon2id(master) · K2 = HKDF(K1 ‖ totp_secret) unwraps the DEK · K3 = HKDF(recovery_key) keeps a recovery copy. The TOTP secret travels encrypted: without a valid TOTP you don't get the DEK.
Session agent: it listens on a Unix socket at $XDG_RUNTIME_DIR/passenv/, mode 0600 and UID validation of the peer (SO_PEERCRED). It's hardened with PR_SET_DUMPABLE=0 (anti-ptrace), no memory dumps and best-effort mlockall.
passenv CLI: init, set/get/list/rm, export, exec, change-master, backup, restore.
Full threat model in docs/THREAT_MODEL.md, with the prompt-less design's ceiling declared on purpose.

What this changes (Takeaway)

The gate isn't a brake; it's what lets you hit the accelerator without fear. A one-hour, one-commit project doesn't end up "truly finished" by luck: it does because the conditions to call it finished were written before the code.

"I think it works" is a feeling. "Here's the proof" is a project.

The checklist

- [ ] Tests that fail when I break something (not that they exist)
- [ ] Clean lint
- [ ] Security scan with no findings
- [ ] Reproducible CI (same command locally and remotely)
- [ ] End-to-end smoke test green

Over to you

What requirements do you demand from a project before calling it done? Which one would you add to my five gates? And since mine is in a personal-use state: would you publish a tool like this or keep it private until you "adapt" it? Tell me in the comments; I read and reply to everyone.

_{Project license: MIT · No secrets or keys in this post. · Project: github.com/yosoyignicion/pass-encrypt-env · Profile: github.com/yosoyignicion}

Coder Legion - Recent posts tagged testing

Your Tests Are Green and Prove Nothing: 4 Signs You're Testing the Mock

1. You Mocked the Thing You Are Testing

2. You Assert the Calls, Not the Outcome

3. The Fake Database Only Says Yes

4. Snapshots Approved Without Reading

The 60-Second Sabotage Check

What to Mock, and What Not To

The Rule

I Solved the Same Gmail Problem Two Ways Without Noticing

Where logging was supposed to go

Gmail replaces your data

Noticing I'd solved it twice

Moving logging upstream

Checking real inboxes

A test case I wrote wrong

What this doesn't show

Cancel or send: one winner. An order flow on Prisma Postgres and Compute

Summary

The problem

Design decisions

Claim first, then send

A guarded transition

A Postgres poller, not a queue

A row lock on the design

Tests, and a test that proved nothing

Prisma Postgres and Compute

Why Prisma ORM 8 is in this project

Getting a NestJS 11 app onto Compute

The surprise: scale to zero versus my poller

Open question: where should the poller run?

Raw output

What I did not verify

Takeaways

A Stopwatch Needs an Isolated Lane

The contract was reasonable

The failure blamed the wrong interval

Keep the budget before changing it

Isolate the claim, not the whole suite

The trade-off is real

A practical review checklist

Precision is a property of the experiment

MT5 Tick Size vs Point: A Test Harness for Price Distance

A fictional contract

An executable unit test

Compare with the platform

My quality gate wasn't strict. It was dead.

The error message named half the problem

So the test is about the contract

Where the constraint went

What I'd check on your own pipeline

Four times my system looked correct and wasn't.

1. The prompt-injection test that passed while the attack worked

2. Retrieval that returned valid results for the wrong decision

3. Telemetry that carried more than I expected

4. A strict CSP that the build quietly broke

The question I ask now

The Guardian redacted my secret. The Read tool walked around it.

What I was testing

A reduced double blind

What worked as documented

Read never reaches the Guardian

MCP tools: three states

The Guardian's log keeps the secret

What I didn't measure

The question I ask now

A Query Window Is Not a Calendar Day

Retrieval and meaning are different jobs

Define “today” in the relevant time zone

Preserve the exception that actually matters

Make wording part of correctness

Test the boundary as a sequence

The engineering trade-off

Practical checklist

The Testing Trophy, and Where TWD Fits

The trophy

What integration means on the frontend

Where TWD fits

What it doesn't replace

A map, not a ritual