Coder Legion - Recent posts tagged puppeteer

Making a screenshot PDF searchable — no OCR, because we rendered the page

Thu, 20 Aug 2026 13:13:18 +0000

If you generate PDFs from web pages by screenshotting them, you've probably hit this: the PDF looks perfect and is completely useless the moment you want to use the text. Ctrl+F finds nothing. You can't copy a sentence. A screen reader opens the document and sees one big image on an otherwise empty page.

We archive whole pages this way — a full-height screenshot dropped onto a PDF page — so we had exactly this problem. Here's how we gave those PDFs a real, invisible, searchable text layer with pdf-lib and @pdf-lib/fontkit, and the one part that turned out to be genuinely hard.

The trick is the same one a "searchable scan" uses: draw the real text invisibly over the image, at the exact coordinates where each word appears. The difference is that a scanner needs OCR to guess the text — we rendered the page ourselves, so we already have the ground truth. No OCR, no guessing.

The shape of it

While the page is still open in the headless browser, ask the DOM where every word is.
Assemble the PDF: embed the screenshot as the page background.
For each word, drawText it at its coordinates with opacity: 0.

Steps 1 and 3 are easy. The trap is in which words you're allowed to draw.

Step 1 — ask the browser where the words are

Running inside the page (Puppeteer's page.evaluate), walk every text node and measure each word with a Range:

const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);
// ...for each word in each text node:
const range = document.createRange();
range.setStart(node, start);
range.setEnd(node, end);
const rects = range.getClientRects();
if (!rects.length) continue;           // display:none or empty line box
const b = rects[0];                    // first rect = where the word starts
out.push({
  t: word,
  x: b.left + window.scrollX,          // document coordinates, not viewport
  y: b.top  + window.scrollY,
  w: b.width, h: b.height,
  fs: parseFloat(getComputedStyle(el).fontSize) || 12,
});

getClientRects() returns viewport coordinates, so we add scrollX/scrollY to get document coordinates — the ones that line up with a full-page screenshot. A word wrapped across two lines returns several rects; the first is where a search should land, so that's the one we keep.

Step 2 — the hidden-text trap (the hard part)

The naive version above shipped a bug immediately. On a real page, search started highlighting words that weren't in the picture — items inside a collapsed dropdown, off-canvas nav, text-indent: -9999px screen-reader text. The image showed them collapsed; the text layer had them anyway.

The tempting assumption is "hidden text has no client rects." It's false. Only display: none produces no rects. Every other way of hiding something still reports perfectly good coordinates:

visibility: hidden
opacity: 0
max-height: 0 + overflow: hidden (the collapsed accordion / submenu)
parked offscreen with text-indent: -9999px or a negative position

So each word has to be checked against its whole ancestor chain, not just its own box. Memoise one walk up the tree per element (a deep DOM would otherwise re-read computed styles thousands of times):

function chainInfo(el) {
  if (cache.has(el)) return cache.get(el);
  const above = el.parentElement ? chainInfo(el.parentElement)
                                 : { visible: true, opacity: 1, clips: [] };
  let info;
  if (!above.visible) {
    info = { visible: false, opacity: 0, clips: above.clips };
  } else {
    const cs      = getComputedStyle(el);
    const opacity = above.opacity * (parseFloat(cs.opacity) || 0);   // multiplied down the chain
    const visible = cs.visibility === 'visible' && opacity > 0.05;
    const clips   = /\b(hidden|clip)\b/.test(cs.overflow)
      ? above.clips.concat([el.getBoundingClientRect()])
      : above.clips;
    info = { visible, opacity, clips };
  }
  cache.set(el, info);
  return info;
}

A word survives only if its chain is visible, its multiplied opacity is above a floor, it intersects every clipping ancestor (a zero-height collapsed container clips its contents to nothing), and it isn't parked entirely outside the document box. Clipping is judged on zero intersection, not partial — a word half-cut by a container is still half in the picture, and dropping it would lose real text.

That single ancestor-walk is the difference between "searchable PDF" and "PDF that lies about what's in it."

Step 3 — flip the Y axis

The DOM measures from the top-left; PDF space is bottom-left. So when we draw, Y inverts, plus a small nudge so the selection highlight sits on the glyphs rather than below them:

const y = imageH - b.y - b.h + b.h * 0.2;

Step 4 — the font problem

pdf-lib's built-in fonts are WinAnsi. They can't encode Cyrillic, Greek or CJK — and they even choke on curly typographic quotes, of which there are four on our own English blog. Draw an unencodable glyph and the call throws.

So embed a real TrueType font, subset to only the glyphs actually used, via fontkit:

const fontkit = require('@pdf-lib/fontkit');
doc.registerFontkit(fontkit);
const font = await doc.embedFont(fs.readFileSync(fontPath), { subset: true });

subset: true is what keeps the cost near zero — only the handful of glyphs on the page get embedded, not the whole 300 KB face. We probe a small candidate list and take the first that exists on the box (DejaVu Sans on our Linux servers covers Latin + Cyrillic + Greek; Arial/Segoe locally). If none is found, we ship the image without a text layer rather than failing the capture — losing search is not losing the archive.

Step 5 — draw it invisibly, and don't let one glyph kill the archive

for (const b of boxes) {
  const y = imageH - b.y - b.h + b.h * 0.2;
  try {
    pdfPage.drawText(b.t, { x: b.x, y, size: b.fs, font, opacity: 0 });
    drawn++;
  } catch (e) {
    skipped++;   // a glyph even this font can't encode — skip the word, keep the PDF
  }
}

opacity: 0 is the whole point: the image is what you see, the text is what you search. And the per-word try/catch matters — a single exotic glyph should cost you one unsearchable word, not the entire multi-page document.

Did it actually work?

Measured on a Cyrillic page, before/after:

626 of 626 visible words got coordinates and a text entry
file grew 7086 KB → 7147 KB — +0.9%
/ToUnicode is present in the output, so the text is genuinely searchable and copyable, not just present

That +0.9% buys you Ctrl+F, copy-paste, and a document a screen reader can actually read — on what is still, pixel for pixel, an exact image of the page.

How We Built a Free SEO Audit Tool with Puppeteer and Chromium

Sat, 23 May 2026 01:30:15 +0000

Real browser rendering, 60+ modular checks, and the engineering lessons we learned building a headless Chromium audit pipeline

When we started building our SEO auditing pipeline, the first architectural decision we made was also the most important one:

We would never parse raw HTML.

Most SEO audit tools still rely on that approach because it’s lightweight and efficient for static websites. But frontend frameworks like React, Next.js, and Vue changed the landscape completely.

We kept running into the same issue:

Traditional parsers were auditing code that users and Google never actually saw.

So we took a different route:
real browser rendering with Puppeteer and headless Chromium.

Here’s how we built the system and what we learned along the way.

The Core Problem With HTML Parsers

Most SEO auditors work something like this:

const response = await fetch(url);
const html = await response.text();

// parse html and inspect tags
For static sites, that works fine.
But JavaScript applications often return almost empty HTML responses initially:
<html>
  <body>
    <div id="root"></div>
    <script src="/static/js/main.chunk.js"></script>
  </body>
</html>

The actual SEO relevant content, including:

H1 tags
Meta descriptions
Structured data
Canonical tags
Images
Internal links

doesn’t exist yet.

It gets generated after JavaScript executes in the browser.

An HTML parser never sees that content.

Googlebot renders JavaScript. Your audit tool should, too.

One of the most surprising things we discovered during testing was how little meaningful content some frameworks returned before hydration. In several React-based applications, the initial response contained almost none of the content users eventually saw in the browser.

In our internal testing, more than half of the React-based sites we audited returned incomplete metadata before rendering. Some pages were missing titles, canonical tags, structured data, or even visible heading content entirely until JavaScript finished executing.

That gap became impossible to ignore.

Why We Chose Puppeteer

We evaluated several options before deciding on our stack:

Playwright: excellent tooling, but heavier than we needed for a Chromium-only workflow
Selenium: powerful, but designed more for browser testing than rendering audits
Cheerio + axios: extremely fast, but limited to static HTML parsing
Puppeteer: lightweight Chromium automation with a straightforward API and strong ecosystem support

Puppeteer ultimately made the most sense for our use case.

We didn’t need multi-browser automation.

We needed rendering accuracy.

That narrowed the field quickly.

The Rendering Pipeline

Here’s a simplified version of the core audit flow:

const puppeteer = require('puppeteer');

async function auditPage(url) {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });

  const page = await browser.newPage();

  await page.setUserAgent(
    'Mozilla/5.0 (compatible; DeepAuditBot/1.0; +https://axiondeepdigital.com)'
  );

  const resources = [];
  page.on('request', (req) => resources.push(req));

  await page.goto(url, {
    waitUntil: 'networkidle2',
    timeout: 30000,
  });

  await autoScroll(page);

  const dom = await page.evaluate(() => document.documentElement.outerHTML);

  await browser.close();

  return { dom, resources };
}

The key detail here is:

waitUntil: 'networkidle2'

This tells Puppeteer to wait until there are no more than two active network requests for at least 500ms.

Without this step, audits frequently captured incomplete pages before JavaScript finished rendering critical content.

This became especially important for:

Hydration heavy React apps
Lazy-loaded images
Dynamically injected metadata
Client-side routing frameworks

Waiting for the network to stabilize before auditing eliminated many of the incomplete renders we encountered early in development.

Handling Lazy-Loaded Content

Another challenge we encountered was lazy loading.

Many sites only load images and components once the user scrolls down the page. A simple page load misses large portions of the content entirely.

To solve this, we implemented an incremental scrolling helper:

async function autoScroll(page) {
  await page.evaluate(async () => {
    await new Promise((resolve) => {
      let totalHeight = 0;
      const distance = 200;

      const timer = setInterval(() => {
        window.scrollBy(0, distance);
        totalHeight += distance;

        if (totalHeight >= document.body.scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, 100);
    });
  });
}

This scrolling behavior triggers:

Intersection observers
Lazy load listeners
Deferred image requests

in much the same way as real user interaction would.

Without scrolling, some audits completely missed below-the-fold image sections, deferred components, and dynamically injected content.

Building the Audit Engine

Once we had a fully rendered page, we built the audit engine itself as a collection of independent modules.

The structure looked roughly like this:

checks/
  meta/
    title.js
    description.js
    og-tags.js
    canonical.js
  headings/
    h1-presence.js
    heading-hierarchy.js
  images/
    alt-text.js
    lazy-load-detection.js
    oversized-images.js
  performance/
    render-blocking.js
    resource-hints.js
    font-loading.js
  structured-data/
    json-ld-validation.js
    schema-types.js
  links/
    internal-links.js
    broken-links.js
    anchor-text.js

Each check receives:

The rendered DOM
Network resource data
Page metrics

and returns a standardized result object:

{
  check: 'h1-presence',
  status: 'pass',
  message: 'H1 tag found: "Your Page Title"',
  impact: 'high'
}

Some checks were intentionally simple:

if (!document.querySelector('h1')) {
  return fail('Missing H1 tag');
}

Others required additional context, especially performance analysis and structured data validation.

This modular structure ended up saving us repeatedly as the platform expanded. Once the number of checks started growing, isolating each audit into independent modules made debugging, maintenance, and feature development far easier.

It also allowed us to:

Disable problematic checks quickly
Add new audit rules independently
Prioritize issues by impact
Generate cleaner reporting output

As the project evolved, modularity became one of the best architectural decisions we made.

Challenges We Didn’t Anticipate

1. Timeout Handling
Some pages are genuinely slow.

Large JavaScript bundles, third-party scripts, tracking pixels, and API delays can dramatically increase render time.

Originally, slow pages caused full audit failures.

We eventually redesigned the pipeline so incomplete audits could still return partial results instead of failing entirely.

That change made the platform far more resilient in production environments.

2. Bot Detection
Some sites actively detect headless browsers and serve different content.

In a few cases, pages rendered perfectly in a normal browser but returned stripped-down responses when rendered in headless Chromium.

We mitigated part of the problem using:

Realistic user agents
Browser fingerprint adjustments
Standard viewport sizes

but avoiding bot detection remains an ongoing challenge across the industry.

3. Single Page App Routing

Single-page applications introduced another issue:
Deep routes sometimes triggered unexpected re rendering behavior during navigation.

We initially experimented with broader crawling behavior, but dynamic client-side routing made the process unreliable very quickly.

In several cases, navigating between routes caused the application state to reset entirely, producing inconsistent audit results between runs.

We eventually simplified the pipeline and audited only the exact URL requested.

That decision made results far more predictable and reduced unnecessary complexity.

4. Memory Management

Chromium gets expensive fast under concurrency.

Early versions of the system launched a fresh browser instance for every audit request. During our first large-scale load tests, memory usage escalated far faster than we expected.

The rendering itself worked well.

The infrastructure did not.

Under concurrency, even small memory leaks became amplified rapidly because every Chromium instance carried the overhead of a full browser environment.

At one point, a single improperly terminated Chromium process accumulated enough memory to destabilize an entire worker node under concurrent load.

That was the moment we realized browser lifecycle management mattered just as much as the audit logic itself.

We eventually learned that even one failed browser cleanup could quietly accumulate memory until the worker became unstable under load.

What We’d Do Differently

If we were starting over, we would implement a reusable browser pool from day one.

Launching a fresh Chromium instance for every audit works initially, but it becomes inefficient very quickly at scale.

Reusing browser instances with isolated pages is far more resource-efficient and improves throughput under concurrency.

In later testing, browser reuse reduced memory overhead noticeably compared to isolated browser launches per request.

We also would have invested earlier in DOM snapshot caching.

Rendering is by far the most expensive part of the pipeline, especially for repeat audits against the same URL.

Caching rendered snapshots would have reduced both rendering overhead and infrastructure costs.

Final Thoughts

Building a browser-rendered SEO auditing system proved far more demanding than parsing static HTML, but it also exposed how incomplete traditional auditing approaches had become for JavaScript-driven applications.

As we expanded the platform, we found ourselves solving problems that had less to do with SEO itself and more to do with rendering stability, browser orchestration, memory management, and infrastructure scaling.

What started as a rendering experiment eventually forced us to rethink nearly every assumption traditional SEO tools make about how websites should be analyzed in modern frontend environments.

The result became the foundation for our public SEO auditing platform, allowing developers and site owners to analyze pages using the same Chromium-based rendering pipeline we built internally.

axiondeepdigital.com/free-seo-audit

Crystal A. Gutierrez is Chairperson & Infrastructure Lead at Axion Deep Digital, a web development and SEO agency based in Las Cruces, NM.