Coder Legion - Recent posts tagged software-quality

Why One AI Review Pass Is Not Enough

Thu, 20 Aug 2026 03:54:55 +0000

Most of us here are already shipping AI-assisted code, and the review side of that has not caught up. This is a write-up of what changes when you stop treating AI review as a single yes-or-no gate and start treating it as a pipeline.

The Volume Problem Nobody Planned For

GitHub's Octoverse 2025 numbers put roughly 41% of new code as AI-assisted, and that share keeps climbing. The awkward part is what comes with it: AI-assisted pull requests carry about 1.7 times more issues per PR than human-written ones, logic errors show up roughly 75% more often, and security vulnerabilities appear 1.5 to 2 times more frequently.

Meanwhile review capacity is flat. Senior engineers already spend 20 to 30 percent of their week reading other people's code. Accelerating the writing phase does not create reviewers, it just moves the bottleneck downstream and makes it worse. PRs pile up, reviewers skim, and defects reach production.

Why A Second And Third Pass Pay For Themselves

A single review pass is like one read of a dense paper. You catch the surface errors and the obvious structural problems, and you miss the things that only appear on closer inspection.

The measured curve is consistent across teams running both approaches. The first pass catches roughly 60% of the issues the system can detect at all. The second pass adds another 25%, largely because some findings only resolve once cross-file context is available. Three passes reach about 95%, which makes four the practical ceiling for most codebases.

A workable pipeline has four stages. Planning estimates the scope of the change and groups related files, so a database migration gets different scrutiny than a front-end component. The initial pass runs broad and fast on obvious bugs, style and basic security patterns. The deep pass takes those findings, adds cross-file context, and traces data flow across function boundaries, checks error handling consistency along a call chain, validates transaction scoping and looks for race conditions. It costs 3 to 5 times more compute than the first pass and it catches the bugs that cause incidents.

The fourth stage is the one people leave out: fix verification. After developers address the earlier findings, re-review only the modified areas to confirm the fixes are correct and did not introduce anything new. That is what prevents the classic case of a fix creating a second bug.

A Model Reviewing Itself Retraces Its Own Reasoning

This is the part with the biggest payoff per unit of effort. When a model reviews code it wrote, it tends to walk back through its original reasoning and confirm its original conclusions. Prompting it to be critical does not fix that, because the problem is structural rather than motivational.

Models from the same family share training distributions, architectures and optimization objectives, and those shared foundations produce correlated blind spots. Where one model misses an integer overflow, another flags it immediately because its training made it more sensitive to numeric boundaries. Teams running cross-model review report 40 to 60 percent better defect detection than same-model review, with the largest gains on logic errors and edge case handling.

Running the review in a separate session matters as much as changing the model. The reviewer should see the code and the requirements, not the reasoning that produced the code. That structural separation is what creates real independence.

There is a fuller writeup of the pipeline stages and the cross-model architecture in our guide to multi-pass AI code review if you want the details.

Where It Still Falls Short

Worth being honest about the ceiling. AI review is strong on exhaustive path tracing, so off-by-one errors, null dereferences, resource leaks and concurrency races get caught reliably. It is strong on vulnerability patterns and on consistency enforcement, which humans apply unevenly under deadline pressure.

It is weak on whether the code should exist. It will confirm a function implements its logic correctly and say nothing about whether that logic fits the business problem. Code that is technically correct and architecturally wrong passes without comment. It is also weak on algorithmic complexity, where a cleanly written quadratic solution often gets approved, and on novel vulnerability classes that are thin in training data.

The Takeaway

If you already run AI review, the cheapest upgrade available is not a better model. It is a second pass, a different model family doing the reviewing, and a fix verification stage at the end.

Reserve the expensive multi-pass configuration for the code that deserves it: authentication, payments, data access and infrastructure config. Let UI and docs changes ride on a single pass. Cost then scales with risk instead of with repository size.

QA is not clicking buttons: it is the quality gatekeeper for the whole SDLC

Mon, 20 Apr 2026 21:06:37 +0000

QA gets stereotyped into a clicker-of-buttons role: open the app, walk the flow, file a Jira ticket when something looks wrong. That caricature is what junior QA looks like at companies that have not yet figured out what QA is actually for. The role at senior is something else entirely — the single person who owns quality across the full software development lifecycle, from the requirements meeting through post-deployment monitoring. The button-clicker is a subset of the role, not the role.

I run this posture in claude-code-mcp-qa-automation as the concrete operator surface: 16 Claude Code skills, a Python implementation that pulls Jira, aggregates sprint trends into a 7-table SQLite store, emits deterministic HTML reports, and posts Slack digests. The tool exists because senior QA work is pipeline work, and pipeline work wants reproducible artifacts, not manual checklists.

Six SDLC stages, one gate per stage. Each gate has the button-clicker version (the failure mode) and the systems version (the senior posture).

Stage 1 — Requirements review
Button-clicker version. QA shows up after requirements are finalized, reads the spec, writes test cases against it.

Systems version. QA is in the requirements meeting and asks problem-first questions before the spec is written. Who is the user? What outcome are they after? Does a non-software workaround already solve 70% of this? What is the observable failure condition? This is Eugene Yan's "start with the problem, not the technology" applied at the gate where technology gets chosen.

The reason QA has to be here is that half the bugs that ship are not implementation bugs — they are requirements bugs. A feature that was built correctly against a vague spec is still a failure to the user. Catching vagueness at Stage 1 is exponentially cheaper than catching it in Stage 6 when the user is asking why the thing does not do what they expected.

Stage 2 — Design review
Button-clicker version. QA waits until the code is ready to be tested, then asks "how do I test this?"

Systems version. QA reads the architecture doc or attends the design review. Asks which services are touched, which data flows cross a trust boundary, which new external dependencies are introduced. Names the error budget. Names the rollback plan.

The question the senior QA is trying to answer at Stage 2 is: "What is the smallest change downstream that breaks this design?" That question is what separates a testable design from one that will collapse the first time production traffic hits a pattern the designers did not consider.

Design-review QA does not require a separate title. It requires the literacy to read an architecture doc and the authority to flag a risk before implementation starts.

Stage 3 — Implementation
Button-clicker version. QA is a ticket consumer. Developer writes code, opens a PR, QA picks up the ticket when it hits the test-ready column.

Systems version. QA has peer-review authority on PRs touching user-facing flows or critical paths. Not final sign-off on code — sign-off on the test coverage, the observability hooks, the feature flag plan, the rollback path. A PR without a flag wrapping a risky change does not pass Stage 3. A PR that adds an endpoint without a structured-logging statement on the failure path does not pass Stage 3.

This is the stage where "works on my machine" gets caught. The senior QA is reading the PR not for correctness of the logic — that is the developer's job — but for the surface area the logic creates. What can break? How would we know? How would we undo it?

Stage 4 — Pre-merge
Button-clicker version. CI runs the test suite. If it passes, merge. QA's role is "make sure there is a test."

Systems version. QA owns the test pyramid. Unit tests for logic. Integration tests for service boundaries. Contract tests for APIs that other services consume. End-to-end tests only for the three or four flows that literally cannot regress. The shape of the pyramid is the senior QA's artifact, not a byproduct.

Pre-merge QA also owns the flake-free contract. A flaky test in the suite is a tax on every future PR — developers start ignoring the red, which is how a real failure slips through undetected. Flake investigation is senior QA work; "re-run the failing job" is not.

Stage 5 — Pre-deploy
Button-clicker version. QA smoke-tests the staging environment by clicking the happy path.

Systems version. QA owns the staging canary. Before a release ships, the senior QA has run the same load profile staging sees in the first 60 minutes of prod. Has checked that feature flags default to off, or default to on for a small cohort. Has verified the migration rolls back cleanly on a cloned schema.

This is also the stage where environment teardown and recreation matter. A staging environment that drifted out of date from prod weeks ago is not a pre-deploy gate; it is a rubber stamp. The senior QA rebuilds staging from scratch on a defined cadence, or asks for the pipeline authority to do it themselves. Which connects to the next stage.

Stage 6 — Post-deploy
Button-clicker version. QA files tickets when users complain.

Systems version. QA reads CloudWatch (or the equivalent) first, before triaging user reports. Correlation IDs on every request. Structured-logging queries that surface error-rate deltas. X-Ray traces or OpenTelemetry spans that localize the blame span. Alarms written by QA, tuned by QA, routed to the right team by QA.

Triage theater is the 40-minute meeting where QA says "the user reports the upload is broken," devs say "we do not see anything in the logs," and nobody has opened CloudWatch. A senior QA who opened CloudWatch first routes the bug directly to the developer who owns the failing span — with the correlation ID, the timestamp, and the stack trace already attached. The "is this a real issue?" step and the "who owns it?" step are already done.

Why this shape matches the Claude-Code operator pattern
The six-stage gate shape is structurally identical to the agentic-engineering discipline I run on the code side: spec-first work at Stage 1, eval loops at Stage 2, review authority at Stage 3, test pyramid at Stage 4, canary + rollback at Stage 5, traced observability at Stage 6. Same operator pattern, different surface.

That is not an accident. The QA-automation repo exists because the same "spec-first, sub-agent orchestration, eval on output" pattern that ships reviewable Claude-Code artifacts is what makes sprint-reporting and alarm-triaging reproducible instead of manual. The role-specific parts of QA (the Jira literacy, the CloudWatch reading, the flag hygiene) plug into a pipeline shape that does not care whether the artifact is a sprint digest, a staging deploy, or a Claude-Code skill file.

The cultural objection
The objection senior QAs hit when they try to operate this way is always some version of "QA should not have that authority." Not prod write, not design-meeting presence, not rollback triggering, not peer-review sign-off on PRs.

The answer is not that QA should have prod write. The answer is that the role requires the authority to do the job it is measured against. If QA is accountable for quality across the SDLC but only has the authority to file tickets, QA is being set up to fail publicly while having no lever to succeed privately. The six gates above are what the authority looks like concretely.

What this means for a QA career
The stereotype of QA as a dead-end "tester" role is true for the button-clicker version and false for the senior version. The ceiling on button-clicker QA is flat. The ceiling on SDLC-wide quality gatekeeper work is the same as any other senior engineering role that owns a cross-cutting concern.

Two capabilities decide which version a QA becomes: CS fundamentals literacy (how the stack works, so you can diagnose instead of triage) and pipeline freedom (the authority to create environments, trigger deploys, and write alarms without filing a ticket to somebody else). Both are skills. Neither is inherent to the role. Both compound.

The next two posts in this series go deeper on each: why CS fundamentals are a QA superpower, and why pipeline freedom is what separates a senior QA from a blocked one.

Aman Bhandari. Operator of an AI-engineering research lab running Claude Opus as the coaching partner, plus a QA-automation surface shipping against a real sprint workload. Public artifacts: claude-code-agent-skills-framework and claude-code-mcp-qa-automation. github.com/aman-bhandari.

Surface Tension of Software: why systems hold together

Sun, 07 Dec 2025 14:07:00 +0000

If you press your finger against water, it pushes back. That invisible resistance, surface tension, keeps the liquid whole even when disturbed.

Good software has something like it. Some systems hold together when you change them; others leak at the slightest touch. The difference lies in integrity — the way a system manages its side effects without losing its shape.

I've seen codebases that felt strangely calm, where every possible state meant something real and nothing arbitrary could slip in. Others allowed nonsense to exist, and from there, entropy spread quietly like cracks beneath paint.

Type systems, invariants, and boundaries exist to make meaning explicit. They define where things start and stop — what’s allowed, and what isn’t. Without that structure, logic turns soft; assumptions spread, and the system eventually folds under its own ambiguity.

Systems stay whole when their structure insists on coherence: clear boundaries, honest interfaces, consistent language.
Each adds its own gravity, and together they make a world that holds. Stability isn’t declared; it emerges from the sum of small, consistent forces.

Constraint-driven design makes that gravity visible. In software, these laws have names: purity, immutability, idempotence, transparency, composability. They’re the physics that keep a system in orbit.

Pure functions return the same output for the same input, with no hidden effects. Immutable data can't be changed after creation, only transformed. Idempotent operations produce the same result no matter how many times you apply them. These aren't academic exercises — they're physics that prevent the impossible.

But let one careless change skip a step and the world tears.

Consider a UI that fetches user data. Without tension, it leaks:

struct UserProfile {
    loading: bool,
    error: Option<String>,
    data: Option<User>,
}

What does it mean when loading is false, error is Some, and data is also Some? The type allows nonsense. You write defensive checks everywhere. Every render must guess which combination is real.

Now give it shape:

enum UserProfile {
    Loading,
    Failed(String),
    Loaded(User),
}

The impossible states vanish. You can't be loading and failed. Pattern matching forces you to handle every valid case, and only those. The type system becomes the membrane — it holds the shape.

In well-shaped systems, nonsense simply cannot exist because the universe of the program doesn't contain it. You don't defend against the impossible. You design a world where the impossible has no syntax.

When those laws are clear, surface tension appears on its own — you feel it when refactors stop rippling outward, when a change bends without breaking, when boundaries push back just enough to preserve meaning.

Good patterns and abstractions behave like membranes. They don't restrain motion; they guide it. They contain side effects the way water's surface holds its ripples — movement without spillage, energy without collapse. Parsing, typing, composition: these are the laws of motion that let a system stay whole when disturbed.

There's an old engineering instinct that says, "We'll handle it later if it happens." But in coherent systems, it simply can't happen. You can only move through valid states, and that constraint is what makes motion possible at all.

But tension has its limits. Too much and water hardens into ice — flawless, unmoving, lifeless. Software can freeze the same way, becoming so rigid it forgets to flow. Balance lives somewhere between order and change, between holding on and letting go.

The best systems live there, in that delicate balance where structure meets freedom. And perhaps that's where the worlds of hackers and painters quietly meet, each shaping their medium until form and motion become one.

That precise balance is the true art of code.

Further reading:
Rich Hickey’s Simple Made Easy — a classic talk on why
simplicity comes from separation, not convenience.