Skip to content

Keep the runtime base64 regexes linear so composed parse paths cannot overflow - #6534

Merged
colinhacks merged 2 commits into
mainfrom
fix-base64-pattern-overflow
Aug 31, 2026
Merged

colinhacks merged 2 commits into
mainfrom
fix-base64-pattern-overflow

Conversation

@colinhacks

@colinhacks colinhacks commented Aug 31, 2026 •

Copy link
Copy Markdown
Owner

Fixes the finding from the #6528 review: z.templateLiteral(["id-", z.base64url()]).safeParse threw RangeError: Maximum call stack size exceeded on multi-megabyte input. The quantified block forms ((?:[...]{4})*) recurse per block in V8's regex engine, so ~8M chars overflows the stack. z.base64() and z.base64url() were already mitigated internally, but template literals compose member def.patterns and test them on every parse.

The split, reworked per review: the public z.regexes.base64 and z.regexes.base64url keep their strict shipped semantics and are untouched. The lax linear forms (/^[0-9a-zA-Z+/]*={0,2}$/, /^[A-Za-z0-9_-]*$/) live module-local in schemas.ts and become def.pattern for the two formats — parse validation is unchanged because isValidBase64/isValidBase64URL enforce the length rules. The JSON Schema emitter swaps the lax patterns back to the strict exports by reference, at both consumer sites: stringProcessor's pattern/allOf and recordProcessor's patternProperties (the review caught that z.looseRecord(z.base64url(), …) would otherwise emit the lax key — now pinned by a test). toJSONSchema(z.base64url()) still emits the length-aware pattern from #6527; its snapshot and agreement tests pass unchanged, and base64 gains the symmetric agreement test. templateLiteralProcessor is deliberately left lax: its emitted pattern then agrees with the (length-lax) composed parse.

This also fixes a second bug the overflow investigation surfaced: regexes.base64's leading ^$| alternation leaked through template-literal composition, so on main z.templateLiteral(["id-", z.base64()]) accepts "AAAA" with the prefix dropped and rejects the valid "id-AAAA". Both directions are now pinned by a test. Composed base64url segments are length-lax again ("id-A" passes), as before #6527 — block length can't be meaningfully enforced mid-composition.

Verified with a controlled repro: on main the template-literal paths throw on 10MB input; on this branch they parse cleanly, and the big-input test now exercises them. Direct use of the strict z.regexes.* exports on multi-MB strings can still overflow — that's inherent to their quantified shape and out of scope here.

Axes: parse-path validator code is unchanged; template-literal parses get a simpler regex. Memory unchanged. Bundle, measured against origin/main with esbuild + gzip -9 on the treeshake fixtures: classic +47 B (classic retains the JSON Schema processors unconditionally via processJSONSchema), mini −2 B (mini shakes them until toJSONSchema is used).

…annot overflow

The block-structured quantified forms of regexes.base64 and regexes.base64url throw RangeError: Maximum call stack size exceeded on multi-MB input. z.base64()/z.base64url() themselves were mitigated, but z.templateLiteral composes member patterns and tests them on every parse, and z.regexes exports the raw regexes — both throw out of safeParse. Swap the runtime regexes to linear charset forms (length rules stay enforced by isValidBase64/isValidBase64URL) and emit the exact block-structured patterns only in toJSONSchema, which zod never executes. The base64url JSON Schema pattern still rejects length % 4 === 1, which was the point of the pattern change that exposed this.
Copilot AI lite review requested due to automatic review settings August 31, 2026 19:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The identity-keyed pattern swap is applied in stringProcessor but not in recordProcessor, so z.looseRecord(z.base64(), …) now emits a patternProperties key that disagrees with parse — the same class of bug #6527 fixed. Details inline.

Reviewed changes — full diff at c199d32c, plus the surrounding regexes.* / bag.patterns consumers, the JSON Schema processors, and a measured before/after of parse behavior, emitted patterns and bundle size.

  • Runtime regexes go linear — regexes.base64 becomes /^[0-9a-zA-Z+/]*={0,2}$/ and regexes.base64url becomes /^[A-Za-z0-9_-]*$/, removing the quantified block forms that overflow V8's regex stack at ~8M chars.
  • Exact patterns move to the JSON Schema emitter — a module-level Map<RegExp, RegExp> keyed by object identity, swapped in at stringProcessor's patternList.
  • isValidBase64URL de-duplicates — the private base64urlCharset added by #6527 is deleted in favor of regexes.base64url, whose source is now byte-identical. No behavior change.
  • Big-input test extended — the exported regexes and template-literal composition are now exercised on 10MB input; those four lines throw RangeError on main, so the test genuinely fails without the fix.

Confirmed the core claim: z.base64() / z.base64url() parse is untouched (def.pattern is never tested for these formats, and PATTERN_IS_COMPLETE excludes them so AOT hoists the validator too), z.hash(alg, { enc: "base64" }) resolves its own md5_base64-style regexes and is unaffected, and the four relevant test files pass. Worth noting the change also fixes a pre-existing bug it doesn't mention: on main the composed template-literal source for ["id-", z.base64()] was ^id-$|^(?:[0-9a-zA-Z+/]{4})*…$, whose top-level alternation leaked past its own segment, so .parse("AAAA") matched with the id- prefix dropped entirely.

⚠️ z.regexes.base64 silently stops validating base64 structure for external consumers

z.regexes is documented public API (api.mdx shows z.string().regex(z.regexes.X)), and regexes.base64 has been strict in every released version. After this change z.regexes.base64.test flips false → true for "A", "AA", "AAA", "AAAAA", "=", "==", "A=" and "AAAA==" — anyone using it directly loses length and padding validation entirely, with no deprecation or release note. (regexes.base64url is not affected by this: it was already /^[A-Za-z0-9_-]*$/ before the unreleased #6527, so that half is a revert, not a regression.)

The bundle argument doesn't force the choice, either — see the section below.

Technical details
# `z.regexes.base64` loses its structural guarantees

## Affected sites
- `packages/zod/src/v4/core/regexes.ts:83` — public export weakened from the block-structured form to a charset-only form.

## Required outcome
- Either `z.regexes.base64` keeps the semantics it has shipped with, or the weakening is stated explicitly in the PR body and release notes as a breaking change to a public export.

## Suggested approach (optional)
- Invert the indirection instead of weakening the export: keep `regexes.base64` strict and public, add a module-local lax pattern used only for `def.pattern`, and key `exactPatterns` the other way (lax → `regexes.base64`).
- This should shake identically on `zod/mini`, since the strict export would then be referenced only from `json-schema-processors.ts`, which mini drops unless `toJSONSchema` is called. Worth measuring before committing to it — I did not measure this variant.

## Open questions for the human
- Is `z.regexes.base64` considered stable enough that a silent widening needs a release note, or is `z.base64()` the only contract that matters here?

ℹ️ The pattern map is not actually behind toJSONSchema in classic

The PR body says "the pattern map lives behind toJSONSchema, so shaking bundlers drop it for users who never call it". That holds for zod/mini, but not for classic: classic/schemas.ts imports * as processors at the top and all 42 constructors unconditionally assign inst._zod.processJSONSchema = (…) => processors.XProcessor(…), so json-schema-processors.ts is reachable from every classic bundle whether or not toJSONSchema is used.

Measured with esbuild (bundle, minify, --conditions=@zod/source) + gzip -9: classic pays +67 raw / +42–46 gz unconditionally, and a classic fixture that only uses z.base64() now also carries the base64url exact pattern it never references. Mini is genuinely unaffected (0, or −53 raw on a z.base64() fixture) unless toJSONSchema is called, in which case it is +140 raw / +75 gz rather than +2. The quoted numbers are roughly reproducible; it's the "opt-in" framing that isn't.

Technical details
# Bundle cost of `exactPatterns` is unconditional on classic

## Affected sites
- `packages/zod/src/v4/core/json-schema-processors.ts:32-36` — new module-level `Map`.
- `packages/zod/src/v4/classic/schemas.ts:4` and the 42 `processors.*Processor` closures — the reason the module is never shaken from classic.

## Required outcome
- The PR body's bundle rationale should match what classic actually pays, so the tradeoff against the public-export weakening above is judged on real numbers.

## Suggested approach (optional)
- No code change strictly required — ~45 B gz on classic is a defensible price. But since classic pays for both regex forms regardless, the bundle axis is not a reason to prefer weakening `z.regexes.base64` over keeping it strict.

ℹ️ Nitpicks

  • packages/zod/src/v4/core/json-schema-processors.ts:32 and packages/zod/src/v4/classic/tests/string.test.ts:296 are long capitalized multi-clause sentences. CLAUDE.md asks for "one lowercase sentence fragment, one clause, no trailing period" — the deleted // charset only — the quantified regexes.base64url overflows… was closer to the house register.
  • There's a "base64url pattern agrees with parse" test but no symmetric one for base64. Now that the emitted pattern for both formats reaches toJSONSchema through an indirection, the base64 half is pinned only by an inline snapshot of the string, not by agreement with safeParse.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using Claude Opus | 𝕏

if (contentEncoding) json.contentEncoding = contentEncoding;
if (patterns && patterns.size > 0) {
const patternList = [...patterns];
const patternList = [...patterns].map((p) => exactPatterns.get(p) ?? p);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stringProcessor isn't the only consumer of bag.patterns — recordProcessor's loose branch writes pattern.source straight into patternProperties with no exactPatterns lookup, so the swap misses it.

Measured on this branch: z.looseRecord(z.base64(), z.number()) emits patternProperties keyed on ^[0-9a-zA-Z+/]*={0,2}$, which matches "A". Zod's loose record accepts { A: "not-a-number" } (key isn't valid base64, so it's skipped), while the emitted schema requires that value to be a number — an external validator rejects input zod accepts. On main the emitted key was the exact pattern and didn't match "A", so this is the pattern/parse disagreement #6527 removed, reintroduced through a different processor.

Technical details
# `exactPatterns` swap misses `recordProcessor`

## Affected sites
- `packages/zod/src/v4/core/json-schema-processors.ts:583` — `assignProp(json.patternProperties, pattern.source, valueSchema)` uses the raw (now lax) runtime regex.
- `packages/zod/src/v4/core/json-schema-processors.ts:57` — the only site currently applying the swap.

## Required outcome
- Every emitted JSON Schema construct derived from `bag.patterns` uses the exact block-structured form, not just `stringProcessor`'s `pattern` / `allOf`.
- A test pins it, so a future consumer of `bag.patterns` doesn't silently regress. `z.looseRecord(z.base64(), z.number())` currently has no coverage — `to-json-schema.test.ts`'s `patternProperties` tests all use `.regex(...)` keys.

## Suggested approach (optional)
- Hoist a one-liner next to the map and use it at both sites:

```ts
const exactPattern = (p: RegExp): RegExp => exactPatterns.get(p) ?? p;
```

- `stringProcessor`: `[...patterns].map(exactPattern)`; `recordProcessor`: `assignProp(json.patternProperties, exactPattern(pattern).source, valueSchema)`.

## Open questions for the human
- `templateLiteralProcessor` also reads a composed `_zod.pattern` and is deliberately left lax, which is correct — its emitted pattern then agrees with the (now length-lax) template-literal parse. Worth a line in the PR body so the asymmetry with `recordProcessor` is intentional on the record and not just on the template literal.

Comment thread packages/zod/src/v4/core/regexes.ts Outdated
// https://stackoverflow.com/questions/7860392/determine-if-string-is-in-base64-using-javascript
export const base64: RegExp = /^$|^(?:[0-9a-zA-Z+/]{4})*(?:(?:[0-9a-zA-Z+/]{2}==)|(?:[0-9a-zA-Z+/]{3}=))?$/;
export const base64url: RegExp = /^(?:[A-Za-z0-9_-]{4})*(?:[A-Za-z0-9_-]{2,3})?$/;
// Linear-time on purpose: the block-structured quantified forms overflow the regex stack on multi-MB input, and these regexes run on parse paths — template literals compose them, and `z.regexes` exports them. Length rules are enforced by `isValidBase64`/`isValidBase64URL`; `toJSONSchema` emits the exact block-structured patterns instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, house style: this is three capitalized sentences on one line. Trimming to the constraint that actually explains the code keeps it in register — the JSON Schema half is already documented next to exactPatterns, and the isValidBase64 half is documented at the validator.

Suggested change
// Linear-time on purpose: the block-structured quantified forms overflow the regex stack on multi-MB input, and these regexes run on parse paths — template literals compose them, and `z.regexes` exports them. Length rules are enforced by `isValidBase64`/`isValidBase64URL`; `toJSONSchema` emits the exact block-structured patterns instead.
// linear on purpose: the quantified block forms overflow the regex stack on multi-MB input, and these run on parse paths

…site

Per review: the public exports keep their shipped semantics, and the lax linear patterns move module-local into schemas.ts as def.pattern for the two formats. The exact-pattern swap now also covers recordProcessor's patternProperties, which otherwise emitted the lax form for z.looseRecord(z.base64url(), ...) — the same pattern/parse disagreement class this is meant to prevent. Adds the base64 pattern/parse agreement test, a looseRecord pin, and a test for the composition leak the lax pattern fixes: the ^$| alternation in regexes.base64 leaked through template-literal composition, accepting "AAAA" prefix-free and rejecting "id-AAAA".
@pullfrog

pullfrog Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Your Claude subscription has hit its usage limit. It resets at 10pm (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using Claude Opus | 𝕏

@colinhacks
colinhacks merged commit 07c43e2 into main Aug 31, 2026
7 of 8 checks passed
@colinhacks
colinhacks deleted the fix-base64-pattern-overflow branch August 31, 2026 20:01

This branch was successfully deployed

1 active deployment
Preview – zod-v4 — 13bb1a6e Deployed Aug 31, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants