Coder Legion - Recent posts tagged playwright

Why the LLM in my automation agent never clicks anything

Tue, 27 Oct 2026 16:57:45 +0000

I'm a computer engineering student in Manaus, Brazil, and for my research project I've been building ANCHOR, a web automation agent: you describe a task in plain language ("Register Maria Silva, IT department, contractor, accept the terms and save"), and it runs the task in the browser. Everything runs locally, with small models (Qwen 3.5, 4B and 9B, through Ollama) on a GTX 1050 Ti with 4 GB.

This post is about the one design decision that made the biggest difference, and a few things I learned by measuring everything.

The problem

Classic RPA depends on fixed selectors written by a developer:

page.locator('[data-test="add-to-cart"]').click()

It works until the page changes, and every automation needs someone who codes, far from the person who actually knows the task.

The trendy answer is to put an LLM in control: it looks at the page and decides where to click. With small models, that has a serious problem, which I measured (below): the model says it's done when it isn't.

The decision: the model only plans

In ANCHOR, the model never clicks anything. It gets the request and a summary of the page (the elements, numbered), and returns a plan: goals and steps, using the elements' names ("fill Full name", "check Contractor", "click Save registration").

A deterministic engine, with no AI, carries it out:

A heuristic finds each element from the step's description, combining text, label, role and context (synonyms, nearby text), and refuses when it isn't sure.
When it refuses, the user decides: the candidates are numbered on the page, and you pick one (or click the right element).
The choice is remembered, so the next run doesn't ask.
Each action's effect is checked (did the page change? did the field keep the value? did an error appear?), and the end of the run is checked against the request.

In the GIF, the site changed after the automation was saved ("Add to cart" became "Add to bag"). Instead of guessing, ANCHOR asks which button to use, remembers the answer, and the next run goes on its own.

Learn once, replay without the model

A task that worked can be saved. The first run plans with the model (about 70 seconds on my GPU); the next ones replay the approved plan without calling the model, in about 3 seconds, even once per row of a spreadsheet. When the site changes and a saved step breaks, the automation recovers and records what it changed, and that record can be reviewed and undone.

What the measurements showed

I built a resilience benchmark: 30 tasks on pages altered in 5 levels (renamed IDs, synonyms, restructured layouts, cookie banners and hidden sections, lookalike decoys with injected instructions), 150 runs per executor and model, with no user to help:

Executor	Tasks done	False successes
Fixed-selector script (classic RPA)	53%	0
LLM in control (4B / 9B)	68% / 72%	33 / 41
ANCHOR (4B / 9B)	82% / 84%	2 / 1

The column that surprised me most was false successes: the LLM-in-control agent claimed it had done the task, without doing it, in about one run in four. For whoever relies on the automation, that's worse than failing, because nobody finds out.

ANCHOR's weak spot is pages with part of the content hidden behind a "Show more" button (47%, vs 57–60% for the LLM in control). And an honest caveat: these are development numbers, where I tuned the system looking at the tasks. For 1.0, there's a closed set of 80 requests written by colleagues who never saw the project, which will run only once, at the end.

Three lessons

1. Re-measure everything after every change. The first full benchmark run revealed four bugs no test had caught (for example, the barrier against destructive actions let a "Delete all" through when the request was "delete Bruno Lima"). And two of my first fixes created new problems, which only showed up because I re-ran the other evaluations too.

2. A prompt change affects tasks that have nothing to do with it. When I added table extraction, I put two lines in the prompt explaining the new actions. The 4B model started stopping a search after filling the box, and the 9B one left the contract type out of a registration. The fix was to show those lines only when the request is about reading data; for every other request, the prompt is identical, byte for byte, to the previous version.

3. Never put your password in a request to an AI. Testing on LinkedIn, I wrote my e-mail and password in the request, and it typed them. The password never left my computer, but it went through the model, which shouldn't happen. That became version 0.3.1: ANCHOR never types a password, refuses a request with a password before it reaches the model, and when a task needs a login, it stops and waits for you to log in by hand (the session is kept for the next runs).

Try it

The project is on GitHub, with the setup steps in the README:

https://github.com/LucasDantas2701/ANCHOR

Testers and opinions are very welcome, especially failure cases on real sites: that's the most useful feedback. The next version (0.4) adds file downloads and uploads, new tabs and iframes.

Disclosure: the code was written with help from an AI assistant (Claude), which also helped me write this post; the project, the decisions and the evaluation are mine. The code is source-available (PolyForm Strict): you can read and use it, but not modify or redistribute it.

STOP reproducing bugs twice. Record once, get both a Markdown Bug Report and a Playwright Regression

Tue, 13 Oct 2026 08:53:10 +0000

📝Every QA engineer and developer knows the drill:
You find a bug. You manually write down steps to reproduce, take screenshots, and format a ticket for Jira or GitHub. Then, you open your IDE and write the exact same steps all over again in code for your automated regression suite.

Why do the same work twice?

I built QA Bug → Regression Test to bridge the gap between manual reporting and automated regression testing. 🎯

💡 How It Works:
Record the Issue: Click Start Bug Session in your browser and reproduce the bug naturally. Clicks, inputs, navigation, and timing are recorded cleanly.
Capture Inline Assertions: Highlight the visible error message, banner, or missing element right on the page to insert an immediate assertion.

Instant Dual Output:
🐞 Markdown Bug Report: Clean, formatted steps to reproduce with environment details and evidence, ready for GitHub Issues, Jira, or Linear.

⚡ Playwright Test (.spec.ts): Production-ready code using resilient locators (getByRole, getByLabel, getByText) that drops straight into your test repository.

🔒 Built for Privacy & Real-World QA:
Zero-Leak Redaction: Passwords, API keys, and session tokens are automatically detected and replaced with REPLACE_WITH_TEST_VALUE in code and [REDACTED] in reports.
100% Local & Offline-First: Manifest V3 compliant. Zero external tracking or servers—everything runs entirely inside your browser.

🔗 Chrome Web Store: https://chromewebstore.google.com/detail/qa-bug-→-regression-test/khoelciicbkdeieibchdndccdncpelbg?authuser=1&hl=tr
🎬 Quick Demo (YouTube): https://www.youtube.com/watch?v=GrtotYTfhbg

I’d love to hear your feedback on how this fits into your daily QA/developer workflow! Try it out and let me know your thoughts below! 👇

Testing on production without polluting your data (and the day my tests voted 8 times)

Tue, 06 Oct 2026 13:36:20 +0000

Every release of Cá Viên Chiên ends with the same step: walk every changed flow on production, in Chromium and WebKit, at phone and desktop sizes, in both languages. Co-op rooms run on the real room server, share codes come from the real API, rewards are written to the real database.

That's a lot of fake players. Early on they all showed up in Google Analytics and in my own player count, so I wrote down one rule:

Test traffic never goes to Google Analytics. Our own API and database are used for real, with a test tag, and the tagged data is deleted when the tests are done.

Here is how that works on Cloudflare Pages + D1, and where it broke.

1. Nothing reaches Google Analytics

Every Playwright context goes through one helper before any page opens:

export async function guardContext(context, { presence = 'tag' } = {}) {
  // GA4's own opt-out flag, so gtag sends nothing even where its script loads
  await context.addInitScript({ content: `window['ga-disable-${GA_ID}'] = true;` });
  // Silence: the game's Web Audio graph runs into a gain of 0
  await context.addInitScript({ content: MUTE_INIT });
  // Drop analytics and ad hosts before the request leaves the browser
  for (const pattern of BLOCKED) await context.route(pattern, (r) => r.abort('blockedbyclient'));
  // Our own player count goes through, tagged as a test browser
  if (presence === 'tag') {
    await context.route('**/api/presence**', (r) =>
      r.continue({ headers: { ...r.request().headers(), 'x-cvc-test': '1' } }));
  }
}

BLOCKED covers Tag Manager, google-analytics.com, analytics.google.com, DoubleClick and AdSense. A local astro build loads GA4 too, so the guard applies to localhost as well.

There is a test for the guard itself: it scans every script in scripts/ that opens the site and fails if one doesn't import guardContext.

Chrome DevTools tabs driven over MCP can't block requests, so they get the same thing as an init script: the GA opt-out flag, the mute, and a fetch wrapper that adds the header.

2. The real API, with a tag

I don't fake our own endpoints. A faked /api/presence means no play record, which means no share code, which means the referral reward flow can't be tested at all. I found that out the hard way, by skipping the reward test on production once.

So the server stores test browsers like anyone else, with a flag:

// functions/api/presence.ts
const test = request.headers.get('X-CVC-Test') === '1' ? 1 : null;
await db.prepare(
  'INSERT OR IGNORE INTO installs (id, …, test) VALUES (?, …, ?)'
).bind(id, …, test).run();

The statistics page and pnpm stats skip installs.test = 1. Everything that hangs off an install (play days, first-session steps, replays, share codes, referral rewards, poster events) can be traced back to a tagged record.

3. Delete it when you're done

After every run, local or production:

node scripts/clean-test-data.mjs --remote        # delete
node scripts/clean-test-data.mjs --remote --dry  # count only: must be 0 everywhere

The script selects by the tag only, deletes dependents first, then the records, all in one call, then counts again and exits non-zero if anything is left:

const T = 'SELECT id FROM installs WHERE test = 1';
const TABLES = [
  ['referral_rewards', `newcomer_install_id IN (${T}) OR code IN (SELECT code FROM share_codes WHERE install_id IN (${T}))`],
  ['share_codes',   `install_id IN (${T})`],
  ['replays',       `install_id IN (${T})`],
  ['install_steps', `install_id IN (${T})`],
  ['poster_events', `install_id IN (${T})`],
  ['install_days',  `install_id IN (${T})`],
  ['installs',      'test = 1'],
];
run(TABLES.map(([t, w]) => `DELETE FROM ${t} WHERE ${w};`).join(' '));

The release checklist has test-rules before the deploy and test-cleanup after it. The deploy script refuses to run until the before items are answered, and the after items must pass before the release record is committed.

After the 1.37.0 release walk it deleted 129 test installs and everything attached to them. A real player's row is never selected.

4. The table that had no tag

Then I tested the new rating stars.

The game page has a "rate this game" card. My check opened it, hovered, and voted 4 stars, at two widths, in two languages, in two engines. Eight votes. Locally that was fine. On production, the log told the story on its own:

4 stars saved | 4.42/5 · 57 player ratings → 4.41/5 · 58 player ratings
…
4 stars saved | 4.38/5 · 64 player ratings → 4.37/5 · 65 player ratings

The ratings table has a voter id, a score, a hashed IP and timestamps, but no install id and no test flag. My cleanup script had no way to find these rows, and every vote moved the public average shown on every page and in the structured data.

Cleaning it up carefully:

List the newest rows, read-only. Eight rows from the same IP hash, alternating vi / en, all 4 stars, inside a three-minute window, matching the four Chromium and four WebKit runs exactly. The vote just before them came from a different address and stayed.

Count before deleting, with exactly the conditions the delete would use:

SELECT COUNT(*) FROM ratings
WHERE ip_hash LIKE 'a2c680bee7%' AND score = 4
  AND created_at BETWEEN strftime('%s','2026-10-03 13:50:00')*1000
                     AND strftime('%s','2026-10-03 13:53:30')*1000;
-- 8

Run the delete with the same WHERE: changes: 8.
Check the result: back to 57 votes and 4.42. The public summary is cached at the edge for 5 minutes, so the API caught up a few minutes later.

Then I made sure it can't happen the same way twice:

The rating check now stubs the POST when it runs on production and reads the live summary for its assertions. The real vote is tested only on the local stack.
The testing rules say it in one line: a test never votes on production, because a rating can't be tagged or found afterwards.
The script was run once more against production with the stub: 24 / 24 checks, still 57 votes.

What I'd tell myself a month ago

Block analytics in the browser, not in a dashboard filter. GA4's internal-traffic filters only apply from the day you turn them on.
Tag at the source. One request header, one column, and every derived row can be traced back to it.
Make cleanup a script with a count, not a habit. Exit non-zero if anything is left.
List every table a test can write to. Any table that can't be traced back to a tagged record needs its writes stubbed on production.
Count before you delete on production, with exactly the WHERE you're going to use.

The game is free in the browser at cavienchien.net.

AI-Driven Playwright E2E Tests From One Reference Doc

Sun, 30 Aug 2026 08:34:31 +0000

Writing E2E Tests With One Doc Instead of New Tooling

We had a growing monorepo with two frontend apps and zero end-to-end coverage. The blocker was never Playwright itself. It was that every new feature meant babysitting an AI agent through the tests, click by click, and the output still needed rewriting by hand.

Here is what actually fixed it, and the part I got wrong on the first try.

Why a dedicated package, not a shared test util

The first instinct was to drop Playwright into the existing shared testing package. That does not work, for boring but real reasons:

The unit-test runner and Playwright's runner cannot cohabit in one package.
E2E runs must not be build-cached, while unit tests are cacheable. One package can't be both.
Browser binaries and report output have no business leaking into a shared utility package.

So the e2e suite is its own package, built from scratch. That separation is what let the rest of the decisions stay clean.

The structure that made tests cheap to write

Three rules, enforced by the setup rather than by review comments:

Selectors live only in Page Object Models. Spec files never contain a raw locator.
Page objects are injected as fixtures. No manual instantiation in a spec, ever.
Auth runs once. A setup project logs in, saves the session state to disk, and every test loads it. No login flow inside a test.

The result is that a spec reads like the user story and nothing else:

test('user can open the create form', async ({ createItemPage }) => {
  await createItemPage.goto();
  await expect(createItemPage.heading).toBeVisible();
});

Already authenticated, page object already wired in, no selector in sight.

The part I got wrong first

With the structure in place, I expected the AI-assisted authoring to be the easy win. It was the opposite. Pointing Codegen and the agent at a page produced tests that looked plausible and were quietly wrong. The agent guessed expected values it had no way to know, and every generated spec came back to my desk for a manual rewrite. It was slower than just writing them myself.

The missing piece was context. The agent knew how to click a button. It did not know what the feature was supposed to do — the state machine, the validation limits, which behaviors were correct and which were known bugs.

So I wrote a skill for one feature that spelled all of that out: the flows, the rules, the routes, the confirmed bugs. The next tests it wrote were good. Not "good for AI" good. Better than what I was writing by hand, because it now had the right direction and never lost the thread halfway through a complex flow.

Then I generalized it.

One doc per feature, not a skill per feature

A skill per feature does not scale. The insight was that the skill was generic; only the facts were feature-specific. So I split them.

The authoring engine became one reusable skill. It reads a per-feature reference doc, discovers the routes, reuses or extends the page objects, drives the real app, captures the actual outcomes as assertions, then runs and self-heals the spec. On top of that sits a thin layer: the official Playwright browser-driving skill for the low-level interaction, and one dedicated skill for the single most complex domain that earned extra shortcuts.

What a new feature needs now is one document: entities, state machine, validation rules, and a numbered list of known bugs. That is it. The reference doc is the contract. If it does not exist, the agent is told to stop and ask for it instead of inventing rules from the UI, which is exactly the failure mode that cost me the first week.

Before: walk the agent through every flow, then rewrite the output.
After: write one ground-truth doc, point the generic skill at it, review real tests.

The reusable idea

The lesson had nothing to do with Playwright. Agents do not fail e2e work because they can't drive a browser. They fail because they guess at behavior no one wrote down. Encode the business truth in a document the agent is forced to read, keep the authoring engine generic, and adding coverage for the next feature costs one doc instead of a new pile of tooling.

The Waiting Problem Behind Most Flaky Suites

Sat, 29 Aug 2026 21:40:45 +0000

Most of us have had a test suite that passed locally, failed once in CI, then passed on the rerun until nobody trusted it as evidence of anything. This is about why that happens, and why one framework's default behavior removes most of it.

Flake Is Usually Not A Bug

A failing browser test is rarely finding something real. It is acting on a page that has not finished arriving. The markup is there, the element is not interactive yet, and the click lands on nothing.

The usual patch is a fixed sleep. Long enough to be safe and the suite crawls, short enough to be fast and it is flaky again the first time CI has a slow day. Retries hide the problem rather than fix it, which is worse, because now a real failure looks like noise.

Auto-Waiting As A Default

Playwright makes the wait part of the action instead of a line you remembered to write. Before it clicks, it checks that the element is attached to the DOM, visible, stable and enabled. Suites converted from older tools normally lose their sleep statements outright.

It is also why the same library gets used well outside testing, for scraping, PDF generation and screenshot pipelines, where the timing problem is identical and nobody writes assertions.

No Translator In The Middle

WebDriver based tools put a separate process between your script and the browser. Playwright speaks the browser's own debugging protocol, the Chrome DevTools Protocol on Chromium, with Firefox and WebKit implementations maintained by the same team.

The commonly quoted two to five times speedup over an equivalent Selenium suite comes mostly out of that removed hop, and it widens as the suite grows. The more interesting consequence is that network interception, request mocking and multi-page control behave like first class features rather than bolt-ons.

Isolation Removes The Second Kind Of Flake

Every test runs in its own browser context, effectively a fresh incognito profile with its own cookies and storage, but cheap enough to create per test. Login state stops leaking between tests and execution order stops mattering, which kills a whole class of failure that has nothing to do with timing.

The Takeaway

Benchmark the thing that actually costs you time. Not how fast a green suite runs, but how many reruns you needed last month before you believed a result. A fuller breakdown, selectors, the Python, Java and C# bindings and the honest comparisons against Selenium and Puppeteer, is in this Playwright guide.

How Browser Agents Read A Page, And When A Script Is Still Better

Sat, 29 Aug 2026 17:44:46 +0000

Most of us here have written a scraper or a browser test suite that worked beautifully until someone shipped a redesign and every selector went stale overnight. This is a look at what actually changes when an agent reads the page instead of replaying a recorded path, and where that tradeoff stops paying for itself.

Why Recorded Steps Break

Traditional web automation targets elements by a fixed path. A class name, an xpath, a nth-child chain. That works right up until the markup moves, and then the script is not wrong in an obvious way, it is confidently clicking nothing. Anyone who has maintained a suite across a design refresh knows the failure is rarely a crash. It is a green run that tested an empty page.

An agent driving the same browser reasons about the page as it exists at that moment and finds the element by its purpose rather than its address. The login button is whatever currently behaves like the login button. That is the entire difference, and it is why the same agent can complete a task on a site it has never seen.

Three Ways An Agent Perceives A Page

Under the surface there are only a few representations in play, and most tool differences come down to how they mix them.

The DOM gives precise targeting and full text, and it drowns you in presentational noise while hiding the meaning that only layout carries. The accessibility tree, the semantic view browsers already build for screen readers, strips the clutter and labels each element by role and purpose, which usually makes it the cleanest input a model can get. And screenshots with numbered overlays on the interactive elements let a vision model say element 7 instead of describing a position in prose.

The setups that hold up in production combine them. Structure for exact targeting, the visual pass for understanding and disambiguation. A deeper breakdown of how agents control and perceive a browser is in this guide to AI browser automation, including where the frameworks and protocols underneath fit.

The Part That Still Bites

Perception does not save you from timing. Most of the modern web arrives nearly empty and fills itself in after running scripts, so an agent that reads too early acts on a page that does not exist yet. It sees a skeleton, decides confidently, and clicks a placeholder. This is the same wait-for-content problem the testing world has fought for a decade, and adding a language model does not remove it. It just makes the wrong answer more fluent.

When A Plain Script Still Wins

The honest limit is cost and determinism. An agent run costs model tokens and takes longer, and it can make a different choice on two identical pages. If a clean API exists, call it. If the target is one stable page you control, a plain script is cheaper, faster and easier to debug at 3am.

Browser agents earn their keep on the messy end: many sites with different layouts, no programmatic interface, and a task description that survives a redesign better than a selector does. That is a real category of work, and it is worth reaching for exactly there rather than everywhere.

Adding Visual Regression Tests To A Playwright Suite

Tue, 18 Aug 2026 20:18:09 +0000

Most of us here already have a Playwright or Cypress suite running green in CI. This post is about the layer that suite does not cover, and how to bolt it on without drowning the team in false diffs.

Functional assertions confirm that an element exists and behaves. They say nothing about where it sits, what it overlaps, or whether a font that failed to load left the heading unreadable. That gap is what a visual testing layer fills, and if you are already on Playwright most of the machinery is sitting in your repo unused.

Where Screenshot Assertions Belong In An Existing Suite

The cheapest place to start is inside tests you already wrote. A checkout test that asserts the confirmation message appears can also capture the confirmation page:

await expect(page).toHaveScreenshot('checkout-confirmation.png');

One line per test, no new tool, no new service. Playwright captures on the first run, stores the baseline next to the spec, and compares on every run after that. The --update-snapshots flag re-baselines when a change was intentional.

The other placement is component level. If you maintain a Storybook, every story is already a visual test case with no test code to write, including the states nobody exercises in an end to end flow: empty, error, loading, and text overflow. Story level diffs are smaller and faster to review than full page diffs, so regressions get caught earlier and cheaper. Running both is the strongest setup, stories for component regressions and end to end shots for composition problems.

Making The Rendering Environment Reproducible

This is the step teams skip, and it is why most visual suites get abandoned in month two. A page rendered on your Mac and the same page rendered on a Linux CI runner produce different images: different font stacks, different anti-aliasing, different sub-pixel hinting. Compare across those two environments and every screenshot fails.

The fix is to make one environment authoritative. Playwright ships official Docker images for exactly this. Capture baselines inside the container, run comparisons inside the same container, and commit only container-produced images. Local runs become a preview and CI becomes the source of truth. Once that holds, the diff percentage on an unchanged page drops to a real zero instead of hovering around 0.3 percent for reasons nobody can explain.

Baselines Belong In Version Control

Store the approved images next to the code that produces them. When a developer intentionally changes a component, the same commit carries the code change and the new baseline, and the reviewer sees both together in one pull request.

The alternative, approving visual changes in a dashboard detached from the diff, is how a team ends up with baselines nobody can trace back to a decision. Version control also answers the question "when did this page start looking like that", which is otherwise a very unpleasant afternoon.

Threshold Tuning And The Noise Worth Fixing First

Start with a tight threshold, around a tenth of a percent of pixels, and treat every failure as real until proven otherwise. Loosening the threshold to make a flaky test pass hides the actual defect, which is an unstable rendering pipeline.

Fix the noise at its source instead. Inject CSS that disables animations and transitions. Mask timestamps, usernames, avatars and anything else that changes per run. Await document.fonts.ready and image load before capturing. Blur the focused element so a blinking cursor never lands in the shot. Use fixed seed data or mocked API responses so the layout is not data dependent. Each of these takes minutes and removes a whole category of false positives permanently.

A visual suite is either trusted or ignored, and the difference is almost entirely noise discipline. Get five high value pages to a clean zero-diff baseline, hold them there for a couple of weeks, and expand only after the team has stopped reflexively clicking approve.

Playwright vs TWD: A Frontend Developer's Honest Comparison

Mon, 15 Jun 2026 16:52:22 +0000

I had both Playwright and TWD pointed at the same small app for a working day. Same backend, same UI, the same bugs to chase. The point wasn't to declare a winner. It was to notice what each one feels like to live with, especially when an AI agent is in the loop too.

Two different rhythms. Different things they make easy. Different things they make slower. One of them I ended up reaching for much more often.

The first thing you notice: where the tests live

Playwright spawns its own browser. A separate window, a separate context, a separate world. You write tests in Node, run them, watch them happen in a window you can look at but not really use.

TWD lives in the tab you're already developing in. The sidebar sits on top of your app while you work. Click run on a test, the test drives the same page you were just clicking around on, then hands it back to you. You can keep using the app.

That single difference shapes most of what comes after. When you can keep using the app while tests run, you reach for them more often. When tests take you out of the app to another window, you reach less.

The feedback loop

Inside TWD, the cycle is short: write a line, save, click run, watch. If something fails, the failure shows up next to the code that produced it. You're never out of the dev tab.

Playwright's loop is longer in feel, even when the run itself is fast. There's a separate browser, a terminal, a report, sometimes a trace viewer. Each piece is fine on its own. Together they add up to context-switching cost.

It isn't that one runs in CI and the other doesn't. Both have a headless CLI runner. Both are fine in CI. The difference is which side of the day each one is designed around. TWD is shaped for the inside of a dev session and treats CI as the export target. Playwright is shaped for the run-the-whole-thing pass and treats local dev as a smaller slice of that.

The debugging story (and why it matters more now)

Both runners are good at telling you why something failed. They tell you in different shapes.

When a TWD test fails, the error is usually about something specific. "Couldn't find this label." "The request payload didn't match." "This element wasn't visible." It's the vocabulary you used to write the test, applied to what just happened. The diagnosis is the test reading itself back to you, in a sentence or two.

When a Playwright test fails, you get richer artifacts. Screenshots, traces, a structured dump of the accessibility tree. The diagnosis is forensic. It's good at telling you what the browser looked like just before things went wrong.

That forensic detail has a cost that didn't matter much when humans were the only audience. Now that the audience is often an AI agent reading test output as part of a session, the cost shows up. A single Playwright failure can drop kilobytes of locator HTML, class lists, and page snapshots into the context window. The agent reads all of it. The token bill scales accordingly. On a long debugging loop where the agent runs the suite, reads the failure, edits a file, runs the suite again, that footprint stacks up fast.

TWD's failures are one-liners. The label name. The payload diff. The selector that didn't match. Cheap to read, cheap for an agent to act on, almost always enough to point at the line that broke.

If your debugging loop involves an LLM in any way, the per-error footprint stops being a stylistic preference and starts being a budget line.

What you get without having to build it

This is where the gap shows up the most.

TWD ships with coverage. Run the headless runner and you get a report. It also ships with contract testing: every mock you write gets validated against your OpenAPI spec on every run. You don't wire it up. It just runs.

Playwright's job is testing. Coverage and contract testing aren't part of that job by design. You can add them. There are packages and recipes for both. None of it is hard, but none of it is free either. You write fixtures, capture data, post-process the output, integrate with reporters. The work isn't huge, but it's the kind of work that quietly gets deprioritized.

This isn't a knock on Playwright. It's a question of which tool defines its scope to include the surrounding tooling, and which one stays narrow on purpose.

How they actually compose

The clean answer for choosing Playwright is real cross-browser. If your app has to work identically across Chromium, Firefox, and WebKit, Playwright gives you that natively. TWD runs in whichever browser you're developing in, one engine at a time. That's the lane. It's a real one. Most apps don't actually live in it, though.

But the framing that earned the most ground for me during the session wasn't "pick one." It was layering them.

TWD on the inside. Every day. The tab you're already developing in. Component mocks, network mocks, fast feedback. Coverage and contract testing carried into CI as part of the same stack.
Playwright at the gate. A small set of true black-box smokes that run in real Chromium, Firefox, and WebKit before a deploy. Login flow, checkout, anything that has to behave identically across engines. Half a dozen tests at most.

Most teams don't need 200 Playwright tests. They need 200 TWD tests and 6 Playwright tests. The math gets cheaper, the dev loop stays fast, and the cross-browser worry stays answered.

That's the stack worth running, if you're going to run both.

What the session left me with

Both tools earned their place by the end. Just not in the same place.

TWD was the right hand for everything I wrote and rewrote during the session: tests in the dev tab, instant feedback, errors short enough to read at a glance. The headless mode brought coverage and contract checks into CI without a separate setup. That's a lot of testing surface from one config.

Playwright was the right hand for the cross-engine question. Not for every spec. For the small set that has to behave identically across browsers.

Two tools, two scopes, one healthy boundary between them. That's the shape of it.

The TWD runner is at twd.dev. The repo is at BRIKEV/twd-docs-tutorial.

From mock-only-works to real-world-works: 48 hours of reCAPTCHA debugging

Mon, 25 May 2026 02:50:17 +0000

Honest framing first: mk-qa-master is an open-source MCP server for QA engineers. The reCAPTCHA solver in it is a Tier 3 fallback for testing your own apps when Tier 1 (Google's official test keys) and Tier 2 (feature flags / IP allowlist) aren't available. It is not a "beat captcha" tool. It refuses to run on Google / Apple / Microsoft / Discord login pages regardless of consent flag. With that out of the way…

This is a diary about the 48 hours it took to go from "shipped a reCAPTCHA solver, all unit tests green" to "it actually works against the real Google demo." Four versions (v0.7.0 → v0.7.4), three broken intermediate ones, and a bunch of lessons that I want to write down before I forget.

The setup

The idea behind the solver is simple. Two atomic MCP tools:

inspect_visual_challenge — finds the captcha iframe on the current page, screenshots it, returns the tile grid coordinates + a screenshot.
solve_visual_challenge — accepts the AI client's tile selection (which tiles contain buses, which contain crosswalks, etc.), clicks them, presses Verify, returns the token.

The AI client (Claude Code, Cursor, etc.) sees the screenshot, decides which tiles match the prompt, and calls solve. The server is the eyes and hands; the AI is the brain. Multimodal models like Claude 4.7 are surprisingly good at this — they were trained on the open web, which has a lot of bus pictures.

So far so good in theory.

Day 1 — v0.7.0 ships

The first version landed Monday. It detected the reCAPTCHA iframe[src*="bframe"], screenshotted it, computed tile coordinates by dividing the iframe's bounding box into a 3×3 or 4×4 grid, and clicked at the center of each selected tile.

Unit tests passed. The bundled mock fixture (a self-contained HTML page that mimics reCAPTCHA's structure) round-tripped end-to-end. I wrote a PRD, shipped a release, posted a Dev.to walkthrough. Felt great.

The mock fixture's structure was:

<table class="rc-imageselect-table">
  <tr><td>...</td><td>...</td><td>...</td></tr>
  ...
</table>

Selectors in the fingerprint:

"tile_table_selector": ".rc-imageselect-table",
"tile_cell_selector": "td",

What could go wrong?

Day 2 — v0.7.1 adds hCaptcha

The next day I extended the fingerprint table to support hCaptcha. Same architecture — different selectors. No new MCP tools. Tests stayed green. I felt good about the design: when a vendor changes, you add a row to the fingerprint table, you're done.

I didn't run a real-world dogfood for hCaptcha either. (We'll come back to this.)

Day 3 — v0.7.2: the first "fix"

I wrote a tiny dogfood script — open Chromium, navigate to https://www.google.com/recaptcha/api2/demo, click the anchor to trigger an image challenge, call inspect_visual_challenge, save the screenshot, ask the AI for tile indices, call solve_visual_challenge, see if a token comes back.

The first run came back with status failed. I asked the user (in this case: me) what they saw in the browser. The answer was unsettling: "I told it to click 2, 5, and 8 — only 5 and 8 actually got highlighted."

I dug into the coordinate math. The iframe-divide approach split the full iframe into rows × cols cells. But the iframe contains a header banner (the prompt text) above the grid and a footer (the Verify button) below. So:

For a 3×3 grid in a 400×580 iframe with header ~130px and footer ~130px:
- The actual grid is 320px tall, ~106px per row.
- Naive iframe-divide gives 193px per row.
- Row 0's computed center lands in the header banner.
- Row 2's computed center lands in the footer.
- Only row 1 happens to be roughly correct.

I wrote v0.7.2 to fix this. Instead of dividing the iframe, I'd read each cell's real bounding_box() from the DOM via Playwright:

for index in range(tile_count):
    bb = cells.nth(index).bounding_box()
    if not is_real_dict(bb):
        # fall back to iframe-divide for mock fixtures
        break
    candidate.append({"viewport_x": bb["x"], ...})

The unit test (against the mock fixture) immediately confirmed the fix. I bumped to v0.7.2, opened a PR, merged, released, published to PyPI. Done.

Day 4 morning — wait, it's still broken

Next morning, ran the dogfood again. Console output for inspect:

"tiles": [
  {"index": 0, "viewport_x": 85, "viewport_y": 92,  "w": 133, "h": 193},
  {"index": 1, "viewport_x": 218, "viewport_y": 92, "w": 133, "h": 193},
  ...
]

133 × 193 rectangles. The exact dimensions you'd get from dividing a 400×580 iframe by 3×3. Which meant the per-cell bounding_box() path was returning None on every cell in real reCAPTCHA, silently falling back to the same broken iframe-divide math.

Looked at the code path: it had a try / except swallowing the error. I added a debug field _coord_method so the inspect response would show which path actually fired:

"_coord_method": "iframe_divide"  // ← v0.7.2's "fix" never ran

So v0.7.2 fixed the mock fixture and shipped to PyPI. In production, against real Google reCAPTCHA, it behaved identically to v0.7.0. The unit test was green because the mock fixture's <td> elements had real CSS dimensions; in real reCAPTCHA the tiles aren't <td>. I just didn't know that yet.

Day 4 afternoon — going DOM-spelunking

Wrote a one-off debug script that opened the real reCAPTCHA bframe and ran arbitrary JavaScript inside it. The first query was: "does .rc-imageselect-table even exist?"

{
  "tableExists": false,
  "altSelectors": {
    "table[class*=\"rc-imageselect\"]": true,
    ".rc-imageselect-target": true,
    ".rc-image-tile-wrapper": true
  }
}

false. The class I'd been targeting since v0.7.0 doesn't exist in production.

The real DOM looks like this:

Element	Mock fixture	Real Google reCAPTCHA
Table class	`rc-imageselect-table`	`rc-imageselect-table-33` (or `-44`)
Tile element	`<td>` with real CSS	`<div class="rc-image-tile-wrapper">` (the `<td>` is 0×0 because tiles are absolutely-positioned)
Challenge text	`.rc-imageselect-desc`	`.rc-imageselect-desc-no-canonical` (in dynamic-replace mode)

The whole fingerprint table had been wrong all along. Unit tests passed because I wrote the mock fixture to match the selectors I'd hardcoded. Tautology. The mock fixture lied because the person who wrote it was the same person who wrote the selectors.

Day 4 evening — v0.7.3 actually fixes it

I rewrote the fingerprint to chain both real and mock selectors via the CSS comma operator (which means "or"):

"challenge_text_selector": (
    ".rc-imageselect-desc-no-canonical, .rc-imageselect-desc"
),
"tile_table_selector": (
    'table[class*="rc-imageselect-table"], '
    '.rc-imageselect-target, .rc-imageselect-table'
),
"tile_cell_selector": ".rc-image-tile-wrapper, .rc-imageselect-table td",

Now the same fingerprint matches both production reCAPTCHA AND the mock fixture. The per-cell bounding_box() path finally runs against real DOM, returning real 95×95 squares instead of distorted 133×193 rectangles. Tile 0 sits at y=211 (just below the 200px header), not y=92 (inside the header banner).

I also fixed a different UX problem in the same release. The MCP server was returning the screenshot as a base64 string embedded in a JSON TextContent. Multimodal AI clients can't "see" base64 — they see a giant string of iVBORw0KG.... The fix: return the screenshot as a native MCP ImageContent:

return [
    ImageContent(type="image", data=b64, mimeType="image/png"),
    TextContent(type="text", text=json.dumps(metadata)),
]

Now Claude Code receives the screenshot as if you'd dragged it into the chat. No manual screenshot juggling.

Day 4 night — v0.7.4 closes the multi-round gap

One more dogfood run, this time the challenge text was different: "Select all images with buses. Click verify once there are none left."

This is reCAPTCHA's dynamic-replace mode. Click a matching tile, the tile gets replaced with a new image. You have to keep selecting until no buses remain, then click Verify. v0.7.3 always clicked Verify after the first round, so it always failed against this mode even with perfect tile judgment.

v0.7.4 added a new return status: "continue". When solve detects dynamic mode (the prompt contains "none left" / equivalent), it does the clicks, waits for the replace animation, re-screenshots the iframe, and returns status: "continue" with a fresh screenshot + new tile geometry. The AI client looks at the new grid, finds any remaining matches, calls solve again. When the AI sees no more matches, it passes an empty selected_tile_indices: [] to signal "click Verify now."

// Round 1 response
{
  "status": "continue",
  "rounds_used": 1,
  "screenshot_base64": "...new grid...",
  "tiles": [...],
  "hint": "Dynamic-replace round 1/5. Look at the new screenshot and call solve again."
}

// Round 2 AI sees no more buses
{ "selected_tile_indices": [], "confirm": true }

// Round 2 response
{ "status": "passed", "token": "03AGdBq25..." }

Hard cap of 5 rounds prevents infinite loops on pathological challenges. Static mode (no marker phrase) is unchanged — legacy flow runs verbatim.

And — the lesson from this entire saga — I added a weekly GitHub Action that runs the dogfood script against the real Google reCAPTCHA demo and asserts _coord_method != "iframe_divide". If Google ships a DOM change next week that breaks the fingerprint again, I'll get a CI failure email within seven days instead of finding out from a user issue six months later.

on:
  schedule:
    - cron: "0 2 * * 0"  # Sunday 02:00 UTC

What works now

✅ reCAPTCHA v2 image-grid (3×3 + 4×4) — verified against the real Google demo
✅ hCaptcha image-select — same fingerprint infrastructure, fixture verified, real-vendor TBD
✅ Multi-round dynamic-replace — unit-test verified, end-to-end real-vendor TBD
✅ MCP ImageContent — multimodal clients see screenshots natively
✅ Consent gate, domain allowlist, hard-stop blacklist (Google / Apple / Microsoft / Discord login pages)
✅ Weekly real-world CI guard

What doesn't (yet)

❌ Mobile WebView — v0.8.0 mini-PRD drafted, ~6 working days of implementation ahead
❌ reCAPTCHA v3 — pure behavior scoring, no visible challenge, out of scope by design
❌ Cloudflare Turnstile — same reason
❌ Audio captcha fallback — accessibility tier, low usage in QA context
❌ The dynamic-replace loop on real Google reCAPTCHA with AI in the loop — that's my next dogfood session

Lessons I want to remember

Mock fixtures can lie. When the same person writes both the production selectors and the mock that tests them, the mock matches by construction. There's no signal. The fix is dogfood against the real thing — and if you can't dogfood, at minimum run a recorded HAR of the real DOM and assert against that.
Silent fallbacks are the worst kind of bug. v0.7.2's try / except swallowed the failure of every per-cell bounding_box() and quietly fell back to broken math. A _coord_method debug field that surfaces which path actually fired would have caught this in minutes. I now add a debug field every time I have more than one code path for the same output.
Multi-round is UX, not a bug. reCAPTCHA's "Click verify once there are none left" isn't an edge case — it's the dominant mode on hard challenges. I built the static-only solver, said "ship it," and was surprised when most real-world challenges fell into the dynamic-replace bucket I hadn't designed for.
Weekly CI catches what unit tests can't. The dogfood workflow runs once a week against a third-party demo. It's noisy, it depends on a vendor's continued cooperation, and it'd be wrong to depend on it for blocking merges. But as a background signal that catches selector drift, it's exactly the right level of investment.

Try it

pip install mk-qa-master==0.7.4

# In your MCP host (Claude Code config, Cursor, etc.)
{
  "mcpServers": {
    "qa-master": {
      "command": "python",
      "args": ["-m", "mk_qa_master.server"],
      "env": {
        "QA_VISUAL_CHALLENGE_CONSENT": "true",
        "QA_VISUAL_CHALLENGE_AUTHORIZED_DOMAINS": "your-staging.example.com"
      }
    }
  }
}

Then ask Claude: "Test the signup flow on staging. If you hit a captcha, solve it." The MCP tools take it from there.

Repo + walkthrough: https://github.com/kao273183/mk-qa-master.

What's next

v0.8.0 — mobile WebView captcha via Maestro CLI (same fingerprint table, new driver). PRD is up in the repo. Probably another diary entry when that one ships.

If you find a bug, the dogfood script lives at scripts/dogfood-inspect-only.py — run it against the page that broke and the inspect output will tell you exactly which coordinate path fired. Beats debugging blind.

A Playwright Reporter That Sends Structured, Interactive Slack Messages

Thu, 30 Apr 2026 07:30:00 +0000

What this is

@playwright-labs/reporter-slack is a Playwright Reporter implementation that sends formatted Slack messages when a test run ends. It handles structuring results by status, surfacing error details inline, masking sensitive environment variables, and posting via either Incoming Webhooks or the Slack Web API.

This post covers the full API.

Installation

pnpm add -D @playwright-labs/reporter-slack @playwright-labs/slack-buildkit

slack-buildkit is a peer dependency used internally and also available for writing custom templates. If you only use built-in templates, you still need it installed.

Basic configuration

// playwright.config.ts
import { defineConfig } from "@playwright/test";
import { WithOptionsTemplate } from "@playwright-labs/reporter-slack/templates";

export default defineConfig({
  reporter: [
    ["@playwright-labs/reporter-slack", {
      send: { webhook: process.env.SLACK_WEBHOOK_URL },
      template: WithOptionsTemplate,
    }],
  ],
});

The reporter:

Implements onTestEnd(test, result) — accumulates [TestCase, TestResult][]
Implements onEnd(result) — resolves template(result, testCases) → SlackBlock[] → posts to Slack

The template option accepts a function, a SlackBlock[] array, a SlackMessage object, or an async function returning any of the above.

Three built-in templates

`BaseTemplate`

Minimal. Header, total/duration summary, list of failed tests (max 10 by default), optional report button.

import { BaseTemplate } from "@playwright-labs/reporter-slack/templates";

template: (result, testCases) =>
  BaseTemplate(result, testCases, {
    projectName: "My App",
    reportUrl: process.env.REPORT_URL,
  }),

Config:

type BaseTemplateOptions = {
  projectName?: string;  // default: "Playwright"
  reportUrl?: string;
  maxFailures?: number;  // default: 10
};

`WithOptionsTemplate`

Status-grouped with an interactive static_select element. Groups are ordered: failed → timedOut → interrupted → skipped → passed. Filter options are built dynamically — if no tests timed out, no "Timed out" option appears.

import { WithOptionsTemplate } from "@playwright-labs/reporter-slack/templates";

template: (result, testCases) =>
  WithOptionsTemplate(result, testCases, {
    projectName:   "My App",
    reportUrl:     process.env.REPORT_URL,
    maxPerStatus:  5,
    showTestNames: true,
    show: {
      passed:   false,  // omit passed group from the message
    },
  }),

Config:

type WithOptionsTemplateConfig = {
  projectName?: string;       // default: "Playwright"
  reportUrl?: string;
  maxPerStatus?: number;      // default: 10
  showTestNames?: boolean;    // default: true
  show?: {
    failed?: boolean;         // default: true
    passed?: boolean;         // default: true
    skipped?: boolean;        // default: true
    timedOut?: boolean;       // default: true
    interrupted?: boolean;    // default: true
  };
};

Overflow behavior: when testCases.length > maxPerStatus, the section ends with _… and N more_.

Error display: for failed and timedOut tests, the first line of result.errors[0].message is rendered below the test name in italics.

`WithTableTemplate`

Same test summary as WithOptionsTemplate, but prepends a GFM table of environment variables (rendered via Slack's markdown block type).

import { WithTableTemplate } from "@playwright-labs/reporter-slack/templates";

template: (result, testCases) =>
  WithTableTemplate(result, testCases, {
    projectName:    "My App",
    reportUrl:      process.env.REPORT_URL,
    tableTitle:     "Build Environment",
    env: {
      CI:           process.env.CI,
      BRANCH:       process.env.BRANCH,
      BUILD_ID:     process.env.BUILD_ID,
      COMMIT_SHA:   process.env.COMMIT_SHA,
      DEPLOY_ENV:   process.env.DEPLOY_ENV,
      DB_HOST:      process.env.DB_HOST,
      DB_PASSWORD:  process.env.DB_PASSWORD,  // auto-masked
      API_KEY:      process.env.API_KEY,       // auto-masked
      OPTIONAL_VAR: process.env.OPTIONAL_VAR, // omitted if undefined
    },
    showRunSummary: true,
    rowsPerChunk:   30,
  }),

Config:

type WithTableTemplateConfig = {
  projectName?: string;                        // default: "Playwright"
  reportUrl?: string;
  env: Record<string, string | undefined>;     // required; undefined values omitted
  tableTitle?: string;                         // default: "Environment"
  mask?: boolean | string[];                   // default: true
  showRunSummary?: boolean;                    // default: true
  rowsPerChunk?: number;                       // default: 30
};

rowsPerChunk controls how many rows per markdown block before the table is split. Slack's text limit per block is ~3000 characters; 30 rows is a safe conservative default.

Auto-masking

When mask: true (default), values are replaced with •••••••• for any key matching:

/token|secret|password|pass(?:word)?|credential|auth|api[_-]?key/i

Examples of keys that match: DB_PASSWORD, AUTH_TOKEN, API_KEY, SECRET_KEY, STRIPE_CREDENTIAL, OAUTH_CLIENT_SECRET.

Masking happens in the template before any data is serialized to JSON. The raw value never appears in the Slack payload.

Override options:

mask: ["CUSTOM_KEY", "INTERNAL_TOKEN"]  // explicit list — only these keys
mask: false                              // no masking

Transport configuration

Incoming Webhook

send: { webhook: "https://hooks.slack.com/services/..." }

POSTs { blocks: SlackBlock[] } to the webhook URL. No OAuth, no scopes. Channel is fixed by the webhook app configuration.

Web API (`chat.postMessage`)

send: {
  token:   "xoxb-...",    // bot token with chat:write scope
  channel: "#ci-reports",
}

POSTs to https://slack.com/api/chat.postMessage with Authorization: Bearer <token>. Supports all Slack API features including thread replies and channel selection at post time.

Writing a custom template

A template is a function:

type SlackTemplate = (
  result: FullResult,
  testCases: [TestCase, TestResult][],
) => SlackBlock[] | SlackMessage | Promise<SlackBlock[] | SlackMessage>;

You can return raw SlackBlock[] JSON, or use the JSX primitives from @playwright-labs/slack-buildkit/react:

/** @jsxImportSource @playwright-labs/slack-buildkit/react */
import {
  Actions, Blocks, Button, Context, Divider,
  Header, Section, Table, Td, Th, Tr,
} from "@playwright-labs/slack-buildkit/react";
import type { FullResult, TestCase, TestResult } from "@playwright/test/reporter";

export function MyTemplate(
  result: FullResult,
  testCases: [TestCase, TestResult][],
) {
  const failed = testCases.filter(([, r]) => r.status === "failed");
  const passed = testCases.filter(([, r]) => r.status === "passed");
  const emoji = result.status === "passed" ? ":white_check_mark:" : ":x:";

  return (
    <Blocks>
      <Header>{`${emoji} My App — ${result.status === "passed" ? "All clear" : "Tests failed"}`}</Header>
      <Section>{`*Total:* ${testCases.length}  •  ✅ ${passed.length}  •  ❌ ${failed.length}`}</Section>

      {failed.length > 0 && (
        <>
          <Divider />
          <Table>
            <Tr><Th>Test</Th><Th>First error</Th></Tr>
            {failed.map(([test, r]) => (
              <Tr>
                <Td>{`\`${test.titlePath().slice(1).join(" › ")}\``}</Td>
                <Td>{r.errors[0]?.message?.split("\n")[0] ?? "—"}</Td>
              </Tr>
            ))}
          </Table>
        </>
      )}

      <Divider />
      <Actions>
        <Button url={process.env.REPORT_URL!} style="primary" action_id="view_report">
          View Full Report
        </Button>
      </Actions>
      <Context>{`My App • ${new Date().toUTCString()}`}</Context>
    </Blocks>
  );
}

JSX runtime notes:

Activated via /** @jsxImportSource @playwright-labs/slack-buildkit/react */ or tsconfig.json
No React, no DOM, no reconciler — purely synchronous function calls
<Table>/<Tr>/<Th>/<Td> converts to a { type: "markdown" } block with GFM table syntax
<Blocks> flattens all children into a flat SlackBlock[]

template: MyTemplate,

Testing templates

Templates return plain JSON arrays, so unit tests are straightforward:

import { test, expect } from "@playwright/test";
import type { FullResult } from "@playwright/test/reporter";
import { WithTableTemplate } from "@playwright-labs/reporter-slack/templates";

const failedRun: FullResult = {
  status: "failed",
  duration: 20000,
  startTime: new Date("2025-04-28T10:00:00Z"),
};

test("sensitive values are masked", () => {
  const blocks = WithTableTemplate(failedRun, [], {
    env: { DB_PASSWORD: "supersecret", BRANCH: "main" },
  });
  const text = blocks
    .filter((b) => b.type === "markdown")
    .map((b) => (b as any).text)
    .join("\n");
  expect(text).not.toContain("supersecret");
  expect(text).toContain("••••••••");
});

test("undefined env values are omitted from table", () => {
  const blocks = WithTableTemplate(failedRun, [], {
    env: { PRESENT: "yes", ABSENT: undefined },
  });
  const text = blocks
    .filter((b) => b.type === "markdown")
    .map((b) => (b as any).text)
    .join("\n");
  expect(text).toContain("`PRESENT`");
  expect(text).not.toContain("`ABSENT`");
});

test("actions block absent when reportUrl is not set", () => {
  const blocks = WithTableTemplate(failedRun, [], { env: {} });
  expect(blocks.find((b) => b.type === "actions")).toBeUndefined();
});

test("actions block present when reportUrl is set", () => {
  const blocks = WithTableTemplate(failedRun, [], {
    env: {},
    reportUrl: "https://ci.example.com",
  });
  expect(blocks.find((b) => b.type === "actions")).toBeDefined();
});

The test suite uses Playwright's runner — no additional test framework needed.

Source

github.com/vitaliharadkou/playwright-labs

5 Things This Playwright SQL Fixture Does So You Don't Have To

Mon, 13 Apr 2026 07:53:00 +0000

5 Things This Playwright SQL Fixture Does So You Don't Have To

If you write Playwright tests that touch a database, you know the boilerplate: connect, seed, clean up, close. Repeat for every file. Hope nothing leaks.

@playwright-labs/fixture-sql handles that boilerplate. Here is exactly what it takes off your plate.

1. Opens and closes connections automatically

Without the fixture, every test file that needs a database connection looks like this:

let client: SqlClient;
beforeEach(async () => { client = await createConnection(); });
afterEach(async () => { await client.close(); });

With the fixture, it is one line:

test.use({ sqlAdapter: sqliteAdapter(':memory:') });

test('my test', async ({ sql: db }) => {
  // db is ready — no setup needed
  // db.close() is called by the fixture after this function, even if it throws
});

The connection lifecycle is entirely handled by Playwright's fixture system.

2. Prevents connection leaks when tests fail

When a test throws an error, code in afterEach might not run correctly depending on how the error propagates. Fixtures are different — Playwright guarantees their teardown phase runs regardless of test outcome.

Every connection opened through sql or useSql is tracked and closed in teardown. You cannot leak a connection through a failing test.

3. Catches SQL errors at compile time

The fixture integrates with @playwright-labs/sql-core's compile-time SQL validator. Write your queries with sql("…") instead of a raw string:

// ❌ runtime error
await db.query('SELECT * WHERE id = ?', [1]);

// ✅ compile error — TypeScript catches missing FROM before tests run
await db.query(sql("SELECT * WHERE id = ?"), [1]);
//             ~~~~~~~~~~~~~~~~~~~~~~~~~~~ never

The type system models SQL grammar as a finite-state machine. Invalid structure → never. Wrong number of params → compile error.

const q = sql("SELECT name FROM users WHERE id = ?");
// typeof q → SqlStatement<[unknown]>

await db.query(q, [1]);        // ✅
await db.query(q);             // ❌ params required
await db.query(q, [1, 2, 3]); // ❌ too many params

4. Handles multiple connections with one call

Need to test replication, or talk to two databases in one test? useSql opens additional connections and registers them all for teardown:

test('primary → replica', async ({ useSql }) => {
  const primary = await useSql(pgAdapter(process.env.PRIMARY_URL!));
  const replica  = await useSql(pgAdapter(process.env.REPLICA_URL!));

  await primary.execute(sql("INSERT INTO events (type) VALUES ($1)"), ['login']);

  const { rows } = await replica.query(sql("SELECT * FROM events WHERE type = $1"), ['login']);
  expect(rows).toHaveLength(1);
});
// both connections closed automatically

No manual cleanup. No try/finally.

5. Works with PostgreSQL, MySQL, and SQLite out of the box

Same API, three drivers:

// SQLite — in-memory, zero infrastructure
import { sqliteAdapter } from '@playwright-labs/fixture-sql/sqlite';
test.use({ sqlAdapter: sqliteAdapter(':memory:') });

// PostgreSQL
import { pgAdapter } from '@playwright-labs/fixture-sql/pg';
test.use({ sqlAdapter: pgAdapter(process.env.DATABASE_URL!) });

// MySQL / MariaDB
import { mysqlAdapter } from '@playwright-labs/fixture-sql/mysql';
test.use({ sqlAdapter: mysqlAdapter(process.env.MYSQL_URL!) });

Drivers are optional peer dependencies — install only the one you use.

Bonus: editor autocomplete

Pair with @playwright-labs/ts-plugin-sql and add a schema file to get table/column autocomplete and diagnostics inside sql templates directly in your editor.

# generate schema types from your live DB
pnpm playwright-labs-sql-core pull --adapter pg --url postgresql://localhost/mydb --out ./src/db-types.ts

// tsconfig.json
{
  "compilerOptions": {
    "plugins": [{ "name": "@playwright-labs/ts-plugin-sql", "schemaFile": "./src/db-types.ts" }]
  }
}

Get started

pnpm add -D @playwright-labs/fixture-sql
pnpm add -D better-sqlite3   # or pg / mysql2

Source: github.com/vitalics/playwright-labs — MIT license.

Angular-Aware E2E Testing: Query Components by @Input and Signals in Playwright

Thu, 02 Apr 2026 12:12:00 +0000

The Gap Between Angular and Its Tests

Angular applications are composed of typed components with declared @Input() properties, signal() state, and @Output() events. Yet most E2E tests interact with these apps through raw CSS selectors — completely ignoring the component model.

This leads to tests that know too much about the DOM and too little about the Angular contract. @playwright-labs/selectors-angular fills that gap.

Installation

npm install -D @playwright-labs/selectors-angular

Quick Setup

// playwright.config.ts
import { selectors, defineConfig } from "@playwright/test";
import { AngularEngine } from "@playwright-labs/selectors-angular";

selectors.register("angular", AngularEngine);

export default defineConfig({
  use: { baseURL: "http://localhost:4200" },
});

Or use the all-in-one import that registers the engine automatically:

import { test, expect } from "@playwright-labs/selectors-angular";

The `angular=` Selector

Query components by their @Input() values instead of DOM attributes:

// By exact @Input value
page.locator('angular=app-button[label="Submit"]')

// By nested property
page.locator('angular=app-user-card[user.role="admin"]')

// By substring
page.locator('angular=app-button[type*="primary"]')

// Multiple conditions (AND)
page.locator('angular=app-button[type="danger"][disabled]')

// Regex
page.locator('angular=app-button[label=/^Sub/i]')

// Case-insensitive
page.locator('angular=app-button[label="submit" i]')

Supported operators mirror CSS attribute selectors:

Syntax	Meaning
`[prop]`	truthy
`[prop="val"]`	exact match
`[prop*="val"]`	contains
`[prop^="val"]`	starts with
`[prop$="val"]`	ends with
`[prop~="val"]`	word in list
`[prop\\|="val"]`	exact or hyphen-prefix

The `$ng` Fixture

Access component state directly from test code:

test("reads component properties", async ({ page, $ng }) => {
  await page.goto("/");

  // @Input value
  const label = await $ng("app-button").first().input<string>("label");
  expect(label).toBe("Submit");

  // WritableSignal value
  const count = await $ng("app-counter").first().signal<number>("count");
  expect(count).toBe(0);

  // List inputs
  const inputs = await $ng("app-button").first().inputs();
  expect(inputs).toContain("disabled");

  // Check directives
  const dirs = await $ng("router-outlet").directives();
  expect(dirs.some((d) => d.includes("RouterOutlet"))).toBe(true);
});

Custom Matchers

import { expect } from "@playwright-labs/selectors-angular";

test("component matchers", async ({ page }) => {
  const btn = page.locator("app-button").first();
  const counter = page.locator("app-counter").first();

  await expect(btn).toBeNgComponent();
  await expect(btn).toHaveNgInput("label");
  await expect(btn).toBeNgInput("label", "Submit");
  await expect(btn).toHaveNgOutput("clicked");

  await expect(counter).toHaveNgSignal("count");
  await expect(counter).toBeNgSignal("count", 0);
});

A Before/After Comparison

Before:

// Depends on template class names — breaks on refactor
const deleteBtn = page.locator("button.danger:disabled");
await expect(deleteBtn).toHaveCount(1);

After:

// Queries the Angular component contract directly
const deleteBtn = page.locator('angular=app-button[type="danger"][disabled]');
await expect(deleteBtn).toHaveCount(1);

The second version continues to work even if the template is rewritten to use different class names, a different element, or a different disabled mechanism — because it queries the component, not the DOM.

Signal-Based Inputs (Angular 17+)

The engine transparently handles input() signal-based inputs alongside traditional @Input():

@Component({ selector: "app-badge" })
export class BadgeComponent {
  count = input<number>(0);   // signal-based input
  @Input() label = "";         // traditional input
}

// Both work identically in selectors:
page.locator('angular=app-badge[count=5]')
page.locator('angular=app-badge[label="New"]')

Requirements

Angular 14+ in development mode (ng serve or --configuration=development)
Playwright 1.40+

Conclusion

By teaching Playwright to speak Angular's language, @playwright-labs/selectors-angular makes E2E tests that are more resilient, more readable, and more aligned with how the application is actually built.

Source: github.com/vitalics/playwright-labs

4 new Playwright email templates with shadcn/ui — and how to make shadcn work in email

Mon, 30 Mar 2026 13:28:00 +0000

What is this?

@playwright-labs/reporter-email is a Playwright reporter that emails your team the test results after each CI run. This release adds 4 new templates built on shadcn/ui.

The core challenge

shadcn/ui is not email-compatible out of the box. Three things break:

Radix UI — shadcn uses Radix for interactive behaviour (Select, Dialog, etc.). Radix calls browser APIs (window, document) at import time. @react-email/render runs in Node.js with none of these.
CSS variables — shadcn default config generates hsl(var(--foreground)). Email clients don't support CSS variables.
JSX transform conflict — in a Playwright project, esbuild replaces React's JSX with Playwright's own internal representation. That breaks @react-email/render.

Fixes

Radix: Rewrote all primitives as plain HTML. SelectItem is a <div> with a conditional ✓ span. Button is an <a> tag. No Radix import anywhere.

CSS variables: Set "cssVariables": false in components.json. shadcn then generates bg-slate-900 (concrete value) instead of bg-background (CSS variable).

JSX transform: Added /* @jsxImportSource react */ pragma to every .tsx component file. esbuild respects per-file pragma overrides, so React's JSX runtime is used instead of Playwright's.

New templates

`base-chart` — pass-rate bar

import { PlaywrightReportShadcnChartEmail }
  from "@playwright-labs/reporter-email/templates/shadcn/base-chart";

<PlaywrightReportShadcnChartEmail result={result} testCases={testCases} >

Adds a stacked horizontal bar showing the percentage of passed / failed / skipped tests. Pure CSS, no images.

`base-button` — CTA links

import { PlaywrightReportShadcnButtonEmail }
  from "@playwright-labs/reporter-email/templates/shadcn/base-button";

<PlaywrightReportShadcnButtonEmail
  result={result}
  testCases={testCases}
  reportUrl="https://ci.example.com/report/42"
>

A footer with "View Full Report" and optionally "View N Failure(s)". Buttons are <a> tags styled with shadcn button variants.

`base-themes` — 6 color themes

import { PlaywrightReportShadcnThemesEmail }
  from "@playwright-labs/reporter-email/templates/shadcn/base-themes";

<PlaywrightReportShadcnThemesEmail result={result} testCases={testCases} theme="green" >

Import only what you need

Each template is a separate bundle under dist/templates/shadcn/. Individual imports:

// Only loads base-select bundle, nothing else
import { PlaywrightReportShadcnSelectEmail }
  from "@playwright-labs/reporter-email/templates/shadcn/base-select";

Or use the barrel for multiple templates:

import {
  PlaywrightReportShadcnChartEmail,
  PlaywrightReportShadcnThemesEmail,
} from "@playwright-labs/reporter-email/templates/shadcn";

Requires "moduleResolution": "bundler" in tsconfig.json to resolve the exports subpaths.

Run the preview locally

git clone https://github.com/vitalics/playwright-labs
cd playwright-labs && pnpm install
pnpm --filter @playwright-labs/reporter-email build
cd packages/reporter-email/examples && pnpm email:preview

Opens at http://localhost:3000. Edit any file in emails/ and the server hot-reloads.

GitHub: https://github.com/vitalics/playwright-labs

npm: @playwright-labs/reporter-email

How We Eliminated Container Boilerplate from Playwright Integration Tests

Thu, 26 Mar 2026 19:20:06 +0000

Integration tests should be honest. When a service talks to Postgres, the test should talk to Postgres too — not a mock that pretends to be Postgres but lacks constraints, triggers, and the dozen behaviors that actually matter in production.

The blocker has never been philosophy. It has been plumbing. Starting a container, waiting for it to be ready, passing the connection details to the test, stopping it after — and doing all of that reliably when a test throws — is tedious boilerplate that sits between you and the test you actually want to write.

We solved this with @playwright-labs/fixture-testcontainers.

What was wrong with the existing approach

The Testcontainers Node.js library is excellent. The missing piece was ergonomic integration with Playwright's test lifecycle. The typical pattern before this package looked like:

import { GenericContainer, StartedTestContainer } from "testcontainers";
import { test } from "@playwright/test";

let pg: StartedTestContainer;

test.beforeAll(async () => {
  pg = await new GenericContainer("postgres:16")
    .withEnvironment({ POSTGRES_PASSWORD: "secret" })
    .withExposedPorts(5432)
    .start();
});

test.afterAll(async () => {
  await pg?.stop();
});

test("insert and query", async () => {
  // finally, the actual test
});

This pattern has several failure modes:

Shared state across tests. Multiple tests modify the same database instance, producing order-dependent failures that are nearly impossible to reproduce in isolation.
Leaking containers. If beforeAll throws after partially initializing, afterAll may receive undefined for pg and fail to clean up.
File-level coupling. Adding a second container means another let, another beforeAll call, another afterAll call. The infrastructure setup grows in proportion to your test file, not to your test logic.

The fixture model fixes all three

Playwright's fixture system already manages this lifecycle for browsers, pages, and any other shared state. Fixtures are set up on demand, scoped to the test (or file, or worker), and torn down after the test ends — even if the test throws.

@playwright-labs/fixture-testcontainers brings that same model to Docker containers:

import { test } from "@playwright-labs/fixture-testcontainers";
import { Wait } from "testcontainers";

test("insert and query", async ({ useContainer }) => {
  const pg = await useContainer("postgres:16", {
    ports: 5432,
    environment: { POSTGRES_PASSWORD: "secret" },
    waitStrategy: Wait.forLogMessage("ready to accept connections"),
  });

  // this container is scoped to this test
  // it stops automatically when the test ends
  const port = pg.getMappedPort(5432);
});

Each test that needs a database gets its own container. No shared state, no cleanup code, no coupling between tests.

Realistic usage: building domain fixtures

The most powerful use of this package is not the raw useContainer fixture — it is using useContainer as a building block for your own fixtures.

// test-helpers/fixtures.ts
import { test as base } from "@playwright-labs/fixture-testcontainers";
import { Wait } from "testcontainers";
import { Pool } from "pg";

type Fixtures = {
  db: Pool;
  redisUrl: string;
};

export const test = base.extend<Fixtures>({
  db: async ({ useContainer }, use) => {
    const container = await useContainer("postgres:16", {
      ports: 5432,
      environment: { POSTGRES_PASSWORD: "secret" },
      waitStrategy: Wait.forLogMessage("ready to accept connections"),
    });
    const pool = new Pool({
      host: container.getHost(),
      port: container.getMappedPort(5432),
      password: "secret",
      database: "postgres",
      user: "postgres",
    });
    await use(pool);
    await pool.end();
  },

  redisUrl: async ({ useContainer }, use) => {
    const container = await useContainer("redis:8", { ports: 6379 });
    await use(`redis://${container.getHost()}:${container.getMappedPort(6379)}`);
  },
});

Tests import from fixtures.ts and receive ready-to-use clients:

import { test } from "./test-helpers/fixtures";
import { expect } from "@playwright-labs/fixture-testcontainers";

test("user record persists", async ({ db }) => {
  await db.query(`INSERT INTO users (name) VALUES ($1)`, ["Alice"]);
  const { rows } = await db.query(`SELECT name FROM users WHERE name = $1`, ["Alice"]);
  expect(rows[0].name).toBe("Alice");
});

The test has no idea containers are involved. It receives a Pool. The infrastructure is an implementation detail of the fixture.

All the options, none of the extra API

ContainerOpts maps one-to-one with GenericContainer.with* methods. There is nothing new to learn — if you know Testcontainers, you know the options:

Option	Testcontainers method
`ports`	`withExposedPorts`
`environment`	`withEnvironment`
`waitStrategy`	`withWaitStrategy`
`healthCheck`	`withHealthCheck`
`network`	`withNetwork`
`bindMounts`	`withBindMounts`
`copyFiles`	`withCopyFilesToContainer`
`pullPolicy`	`withPullPolicy`
`resourcesQuota`	`withResourcesQuota`
... and 20 more

For cases where you need the full builder, pass a pre-configured GenericContainer:

import { GenericContainer } from "testcontainers";

const container = await useContainer(
  new GenericContainer("postgres:16")
    .withEnvironment({ POSTGRES_PASSWORD: "secret" })
    .withExposedPorts(5432)
    .withReuse(),
);

Container assertions

The package exports an extended expect with matchers specific to StartedTestContainer:

import { expect } from "@playwright-labs/fixture-testcontainers";

// state
await expect(container).toBeContainerRunning();
await expect(container).toBeContainerHealthy();
await expect(container).not.toBeContainerStopped();

// logs
await expect(container).toMatchContainerLogMessage("ready to accept connections");
await expect(container).not.toMatchContainerLogMessage("FATAL");

// ports
expect(container).toBeContainerPort(5432);
expect(container).toMatchContainerPortInRange(5432, { min: 1024, max: 65535 });

// metadata
expect(container).toHaveContainerLabel("env", "test");
expect(container).toHaveContainerNetwork("app-net");
await expect(container).toHaveContainerUser("postgres");

These matchers turn container state inspection from raw Docker API calls into readable assertions that belong in test output.

Getting started

npm install -D @playwright-labs/fixture-testcontainers testcontainers

Requirements: @playwright/test >= 1.57.0, testcontainers >= 10.0.0, Docker available in your environment.

The package is part of playwright-labs, an open-source collection of Playwright utilities.

If you maintain a Playwright test suite and have been using mocks for anything that should really be a database or a cache, this is a practical path to replacing them without rewriting your entire test setup.

AI Watch Tester (AWT)

Wed, 25 Feb 2026 12:50:30 +0000

Every new project, same story: write login tests, write form validation tests, write navigation tests. Copy-paste from the last project, tweak selectors, pray nothing breaks.

After 25 years in IT, I decided to automate the boring part. I built AWT (AI Watch Tester) — an open-source tool where you enter a URL, and AI writes the tests for you.

How It Works

Enter a URL — that's your only input
AI scans the page — analyzes DOM structure + takes screenshots
Generates test scenarios — login flows, form validation, navigation checks
Runs them with Playwright — real browser, real clicks, real screenshots

No selectors to write. No test scripts to maintain. AI handles the planning, Playwright handles the execution.

"Can't Claude/GPT Just Do This with Computer Use?"

Fair question. I get it a lot.

Computer Use is a general-purpose GUI agent — it can click buttons and type text. But for E2E testing, you'd still need:

Docker environment setup
Screenshot pipeline management
Result parsing and storage
CI/CD integration
Scenario tracking across runs

And each test costs $0.50–2.00 because the AI processes every screenshot.

AWT uses AI only for test generation (analyzing what to test), then runs tests with Playwright — no per-screenshot AI cost. A typical scan costs $0.002–0.03. That's 10–100x cheaper.

Think of it this way: Computer Use is the hammer. AWT is the furniture store.

What Makes It Different

	AWT	Playwright/Cypress	testRigor/Applitools
Test writing	AI writes them	You write them	AI assists
Cost	Free (MIT) + BYOK	Free	$800+/mo
AI provider	Your choice (OpenAI, Anthropic, Ollama*)	N/A	Locked in
Local mode	Yes (Ollama, experimental)	Yes	No

*Ollama adapter is included but experimental — works best with larger models (70B+). Results may vary with smaller models.

The Honest Limitations

This is v1.0 by a solo developer. Let me be upfront:

✅ Works well on simple login/form pages (SauceDemo, standard auth flows)
⚠️ Complex SPAs with heavy dynamic content — still improving
⚠️ No cancel button for long scans yet
⚠️ Free plan is limited (5 pages per scan)

Tech Stack

Backend: Python, FastAPI, Playwright
Frontend: Next.js, TypeScript
Database: PostgreSQL (Supabase)
AI: OpenAI / Anthropic / Ollama adapters
License: MIT

Try It

Cloud: https://ai-watch-tester.vercel.app
GitHub: https://github.com/ksgisang/AI-Watch-Tester

Ollama adapter is also included for local execution, though it's still experimental — best results with larger models.

What I Learned Building This

AI is great at generating test plans, bad at executing them. That's why I separated generation (AI) from execution (Playwright). Trying to do both with AI is expensive and fragile.
Language detection matters. My first users got Korean test scenarios on English sites. Lesson: always detect the target site's language before generating.
Assert validation is critical. AI sometimes generates structurally invalid assertions. A post-processing validator that auto-corrects the schema saved me from shipping broken tests.

Bug reports, feedback, and PRs are all welcome. What edge cases should I try next?

Graceful Test Cancellation: Why Your Playwright Tests Need AbortSignal

Mon, 16 Feb 2026 19:41:08 +0000

There is a class of bugs that only shows up at scale. Not in your code. Not in your infrastructure. In the gap between them -- where your test suite meets your staging environment.

I am talking about resource leaks from timed-out tests.

The Scenario You Have Definitely Seen

A developer pushes a change. CI kicks off 300 Playwright tests across 8 workers. Fifteen of those tests hit a slow endpoint -- maybe the service is under load, maybe there is a network hiccup. They time out after 30 seconds each.

Playwright marks them as failed and moves on. Good. That is what it should do.

But here is what does not happen: the HTTP requests those tests initiated do not get cancelled. They are still in flight. The server is still processing them. The TCP connections are still open.

Fifteen tests, each making two or three API calls. That is 30 to 45 orphaned requests still hammering your staging environment. They will finish eventually -- or time out on their own after another 30 seconds. In the meantime, they consume server threads, database connections, and memory.

Now imagine this happens every CI run. Multiple times a day. Across multiple teams sharing the same staging environment.

This is not hypothetical. This is Tuesday.

Why Playwright Does Not Handle This

This is not a Playwright bug. It is a fundamental constraint of how JavaScript async operations work.

When Playwright times out a test, it stops waiting for the test function to resolve. But fetch, axios, WebSocket connections, and other async operations do not know the test has ended. They were started, they are running, and nothing has told them to stop.

JavaScript has no built-in mechanism to kill arbitrary async operations from outside. You cannot reach into a running fetch and terminate it. You cannot walk the promise chain and cancel everything.

What you can do is provide a cancellation token upfront -- a signal that says "when I tell you to stop, stop." This is what AbortController and AbortSignal are for.

AbortSignal: A Quick Refresher

AbortController is part of the web platform and is also available in Node.js. It provides a simple cancellation mechanism:

const controller = new AbortController();

// Start an operation with the signal
fetch('/api/data', { signal: controller.signal });

// Cancel it from outside
controller.abort();
// The fetch promise rejects with an AbortError

The key insight is that you decide at the start of the operation that it should be cancellable. You pass the signal in. Later, when you need to cancel, you call abort() on the controller.

This is the missing piece for Playwright tests.

Wiring AbortSignal Into the Test Lifecycle

The @playwright-labs/fixture-abort package provides this wiring. It extends Playwright's test with fixtures:

abortController -- a fresh AbortController for the current test
signal -- the associated AbortSignal
useAbortController(options?) -- returns the controller with optional abort callback
useSignalWithTimeout(ms) -- returns a signal that auto-aborts after a duration

The controller is automatically aborted when the test times out. Every operation that received the signal cancels immediately.

import { test, expect } from '@playwright-labs/fixture-abort';

test('should fetch data with cancellation support', async ({ signal }) => {
  const response = await fetch('https://api.example.com/data', {
    signal
  });
  const data = await response.json();
  expect(data).toBeDefined();
});

If this test times out, signal fires. The fetch call is cancelled. The server stops processing. The connection is freed. No cleanup code required.

Where This Pattern Really Shines

Polling Loops

Tests that poll an endpoint are the most common source of zombie requests. Without an abort signal, a polling loop can run indefinitely after a timeout:

import { test, expect } from '@playwright-labs/fixture-abort';

test('should handle long polling', async ({ signal }) => {
  while (!signal.aborted) {
    const response = await fetch('/api/poll', {
      signal
    });

    if (response.status === 200) {
      const data = await response.json();
      expect(data.ready).toBe(true);
      return;
    }

    await new Promise(resolve => setTimeout(resolve, 1000));
  }
});

Two layers of protection here. The while (!signal.aborted) check prevents the loop from continuing after cancellation. The signal on the fetch call cancels any in-flight request. If the test times out mid-request, the fetch is cancelled. If it times out between requests, the loop condition catches it.

Parallel Request Batches

When tests fire multiple requests simultaneously, all of them need to be cancellable:

import { test, expect } from '@playwright-labs/fixture-abort';

test('should aggregate data from multiple services', async ({ signal }) => {
  const [users, products, analytics] = await Promise.all([
    fetch('/api/users', { signal }),
    fetch('/api/products', { signal }),
    fetch('/api/analytics', { signal })
  ]);

  // Process results...
});

One signal, shared across all requests. When it fires, everything stops.

Retry Logic

Tests with retry mechanisms are particularly dangerous without abort signals. A retry loop can keep hammering a failing endpoint long after the test has timed out:

import { test, expect } from '@playwright-labs/fixture-abort';

test('should retry on failure', async ({ signal }) => {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < 5 && !signal.aborted; attempt++) {
    try {
      const response = await fetch('/api/flaky-endpoint', {
        signal
      });
      if (response.ok) {
        const data = await response.json();
        expect(data).toBeDefined();
        return;
      }
    } catch (err) {
      lastError = err as Error;
      if (signal.aborted) break;
      await new Promise(resolve => setTimeout(resolve, 1000 * (attempt + 1)));
    }
  }
});

The signal check in the loop condition, the signal on the fetch call, and the signal check in the catch block create a robust cancellation pattern that exits quickly no matter where in the retry cycle the timeout occurs.

Timeout Signals

Need a signal that auto-aborts after a specific duration, independent of the test timeout? Use useSignalWithTimeout:

import { test, expect } from '@playwright-labs/fixture-abort';

test('should complete API call within 5 seconds', async ({ useSignalWithTimeout }) => {
  const timeoutSignal = useSignalWithTimeout(5000);

  const response = await fetch('https://api.example.com/slow', {
    signal: timeoutSignal
  });
  const data = await response.json();
  expect(data).toBeDefined();
});

Custom Expect Matchers

The package includes matchers for asserting abort states:

import { test, expect } from '@playwright-labs/fixture-abort';

test('should verify signal states', async ({ signal, abortController }) => {
  expect(signal).toBeActive();
  expect(abortController).toHaveActiveSignal();

  abortController.abort('done');

  expect(signal).toBeAborted();
  expect(signal).toBeAbortedWithReason('done');
  expect(abortController).toHaveAbortedSignal();
});

The Bigger Picture: Tests as Good Citizens

There is a broader principle here. Your tests are not running in isolation. They share infrastructure with other tests, other teams, and often with real users (if staging is a shared environment).

A test that leaks resources is like a function that leaks memory. It might not matter once. It matters a lot when it happens thousands of times a day.

AbortSignal integration is not just about preventing flaky tests. It is about making your test suite a responsible consumer of shared resources. It is about ensuring that when a test ends -- whether it passes, fails, or times out -- it cleans up after itself completely.

The Cost of Not Doing This

Without abort signal integration, you are relying on:

Server-side timeouts to eventually clean up orphaned requests (adding load until they fire).
Connection pool limits to cap the damage (causing legitimate requests to queue).
Manual investigation when staging environments degrade (burning engineering time).

None of these are solutions. They are damage control.

Getting Started

npm install @playwright-labs/fixture-abort

The package provides raw Playwright fixtures. Import test and expect from @playwright-labs/fixture-abort instead of @playwright/test, use signal in your async operations, and your tests clean up after themselves automatically.

Full source code and documentation: github.com/vitalics/playwright-labs

The change is small. The impact on your infrastructure reliability is not.

Coder Legion - Recent posts tagged playwright

Why the LLM in my automation agent never clicks anything

The problem

The decision: the model only plans

Learn once, replay without the model

What the measurements showed

Three lessons

Try it

STOP reproducing bugs twice. Record once, get both a Markdown Bug Report and a Playwright Regression

Testing on production without polluting your data (and the day my tests voted 8 times)

1. Nothing reaches Google Analytics

2. The real API, with a tag

3. Delete it when you're done

4. The table that had no tag

What I'd tell myself a month ago

AI-Driven Playwright E2E Tests From One Reference Doc

Writing E2E Tests With One Doc Instead of New Tooling

Why a dedicated package, not a shared test util

The structure that made tests cheap to write

The part I got wrong first

One doc per feature, not a skill per feature

The reusable idea

The Waiting Problem Behind Most Flaky Suites

Flake Is Usually Not A Bug

Auto-Waiting As A Default

No Translator In The Middle

Isolation Removes The Second Kind Of Flake

The Takeaway

How Browser Agents Read A Page, And When A Script Is Still Better

Why Recorded Steps Break

Three Ways An Agent Perceives A Page

The Part That Still Bites

When A Plain Script Still Wins

Adding Visual Regression Tests To A Playwright Suite

Where Screenshot Assertions Belong In An Existing Suite

Making The Rendering Environment Reproducible

Baselines Belong In Version Control

Threshold Tuning And The Noise Worth Fixing First

Playwright vs TWD: A Frontend Developer's Honest Comparison

The first thing you notice: where the tests live

The feedback loop

The debugging story (and why it matters more now)

What you get without having to build it

How they actually compose

What the session left me with

From mock-only-works to real-world-works: 48 hours of reCAPTCHA debugging

The setup

Day 1 — v0.7.0 ships

Day 2 — v0.7.1 adds hCaptcha

Day 3 — v0.7.2: the first "fix"

Day 4 morning — wait, it's still broken

Day 4 afternoon — going DOM-spelunking

Day 4 evening — v0.7.3 actually fixes it

Day 4 night — v0.7.4 closes the multi-round gap

What works now

What doesn't (yet)

Lessons I want to remember

Try it

What's next

A Playwright Reporter That Sends Structured, Interactive Slack Messages

What this is

Installation

Basic configuration

Three built-in templates

BaseTemplate

WithOptionsTemplate

WithTableTemplate

Auto-masking

Transport configuration

Incoming Webhook

Web API (chat.postMessage)

Writing a custom template

Testing templates

Source

5 Things This Playwright SQL Fixture Does So You Don't Have To

5 Things This Playwright SQL Fixture Does So You Don't Have To

1. Opens and closes connections automatically

2. Prevents connection leaks when tests fail

3. Catches SQL errors at compile time

4. Handles multiple connections with one call

`BaseTemplate`

`WithOptionsTemplate`

`WithTableTemplate`

Web API (`chat.postMessage`)

The `angular=` Selector

The `$ng` Fixture

`base-chart` — pass-rate bar

`base-button` — CTA links

`base-themes` — 6 color themes