Skip to content

perf(ext/web): apply the base64url op design to standard base64 - #36422

Merged
bartlomieju merged 6 commits into
denoland:mainfrom
tomas-zijdemans:perf/base64-op-unification
Aug 10, 2026
Merged

perf(ext/web): apply the base64url op design to standard base64#36422
bartlomieju merged 6 commits into
denoland:mainfrom
tomas-zijdemans:perf/base64-op-unification

Conversation

@tomas-zijdemans

@tomas-zijdemans tomas-zijdemans commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #36398, delivering the two items promised there: the stale <= 4096 op split in base64Slice and the transfer of the sentinel/no-split findings to the standard base64 ops.

Same-tree A/B, release-lite builds, macOS aarch64, medians of 5, ns/op:

benchmark main this PR speedup Node 26.5
buf.toString("base64"), 32 B 49.5 35.9 1.4x 39.6
buf.toString("base64"), 256 B 67.2 49.3 1.4x 56.9
buf.toString("base64"), 1 KiB 115.2 96.9 1.2x 132.2
buf.toString("base64"), 4 KiB 427 381 1.1x 449
hash.digest("base64"), sha256 554 510 1.09x 298
Buffer.from(s, "base64"), junk chars, 346 ch 7,141 1,089 6.6x 150
Buffer.from(s, "base64"), embedded =, 345 ch 6,626 570 11.6x 88
buf.write(s, "base64"), junk chars, 346 ch 6,646 1,030 6.5x 130
btoa, 96 B 84.1 58.4 1.4x 29.7
btoa, 4 KiB 488 460 1.06x 447
btoa, 16 KiB (external-string path) 1,611 1,439 1.12x 1,484

Guard rows, all within noise (Âą5% across repeated runs): clean padded/unpadded decode (344 ch and 87,384 ch), write with clean input, sub-range and 64 KiB encode, atob, and a full re-run of the #36398 base64url table (every row unchanged).

Encode at 32 B to 4 KiB now beats Node. The dirty-input rows stay behind Node because the cleaning semantics themselves remain in JS (base64clean), as on the base64url side.

What changed

Four commits, reviewable independently:

  1. Unify the helper families (refactor, no behavior change). simdutf_base64url_* collapses into the standard helpers, parameterized by simdutf::Base64Options. Two hardening items from the perf(ext/web): implement base64url encode/decode as simdutf ops #36398 review ride along. The output-capacity assert! now covers every decode-into caller (it guards memory safety: simdutf may write up to the maximal decoded length before detecting an error, and the assert previously existed only on the url side). And the debug_assert_eq! in the standard encode path moves above set_len, the same placement fix the url side got in review.

  2. -1 sentinel for op_base64_decode_into. Same contract as op_base64url_decode_into: invalid input returns -1 instead of materializing an exception across the op boundary (~4.4 Âĩs, measured in perf(ext/web): implement base64url encode/decode as simdutf ops #36398). The strict pre-pass stays, because clean padded input is the common case for the standard alphabet, unlike base64url (base64_std_decode_into_strict_and_loose pins this). One structural find: base64Write's fallback previously went through base64ToBytes, whose eager decode attempt would always throw after a sentinel. The op and the wrapper share the same loose decode, so an input that produced -1 can never pass it. The fallback now goes straight to the cleaning path (base64CleanToBytes), which is where the 6.5-11.6x on the dirty rows comes from. The orphaned forgivingBase64DecodeInto infra export is deleted (no consumers).

  3. Drop the 4096 encode split, delete op_base64_encode. Re-measured during the perf(ext/web): implement base64url encode/decode as simdutf ops #36398 review: op_base64_encode_from_buffer (direct one-byte V8 string) wins at every size, so the split predates that encode path. With the split gone, op_base64_encode had no caller that isn't better served by the from-buffer op, so it is deleted rather than kept for the case it loses (the same reasoning as perf(ext/web): implement base64url encode/decode as simdutf ops #36398's digest fix). forgivingBase64Encode and the jupyter image display rewire to the from-buffer op. hash.digest("base64") rides along.

  4. btoa builds its result as a V8 one-byte string directly via the now-shared helper, skipping one full copy of the output through the op glue. Error semantics untouched: the InvalidCharacterError mapping lives in the JS wrapper's ByteString-conversion catch. This commit is independent and droppable if it raises any concern.

Behavior

None intended. Verified:

  • A 45-string / 14-buffer corpus (padding variants, whitespace in every position, both alphabets mixed, junk, len % 4 == 1 residues, high-Unicode strings, sizes straddling the old 4096 split and the 8192 stack buffer) run through Buffer.from, buf.write at offsets, buf.toString sub-ranges, Buffer.byteLength, atob, and btoa is byte-identical to main and to Node 26.5.0.
  • Property test, 100k random buffers (0-4096 B): toString("base64") matches the btoa reference, decode round-trips, atob round-trips. The perf(ext/web): implement base64url encode/decode as simdutf ops #36398 base64url corpus and property test also re-run unchanged against this branch.

Testing

  • New Rust unit tests: std-alphabet encode/decode pins for the parameterization, strict-vs-loose contract, sentinel on all three base64_decode_into_slice branches for the standard alphabet, and a should_panic pin on the capacity assert.
  • New TS tests in buffer_test.ts: std dirty-input matrix (junk, embedded =, url alphabet, >U+00FF chars through the catch path), truncating writes through both the op and the cleaning fallback, and base64 on Buffer views with non-zero byteOffset (strict and loose write paths, sub-range toString, surrounding bytes checked). That last one is the test class the perf(ext/web): implement base64url encode/decode as simdutf ops #36398 review asked for.
  • WPT html/webappapis/atob (760 assertions) and FileAPI/reading-data-section/filereader_readAsDataURL green locally. unit_node::buffer_test, unit_node::crypto::crypto_hash_test, unit::text_encoding_test, unit::filereader_test, unit::websocket_test green. tools/lint.js and tools/format.js clean.

Left for follow-ups

  • The JS cleaning fallback (base64clean) stays in JS. It only runs on invalid input.
  • The atob/decode counterpart of commit 4 (a from-buffer shape for atob) does not apply: atob must return a binary string and already reuses the input allocation on the small path.

I used Claude Code to help investigate and write this change.

tomas-zijdemans and others added 5 commits August 5, 2026 10:06
Collapse the parallel simdutf_base64url_* helper family into the standard
helpers, parameterized by simdutf Base64Options. The loose/strict decode-into
pair merges into one function that also carries the output-capacity assert
(previously only on the url side), and the unified encode-to-V8-string keeps
the url side's debug_assert placement (before set_len, not after).

No behavior change.
op_base64_decode_into now reports invalid input as a -1 sentinel instead of
throwing, mirroring op_base64url_decode_into: materializing an exception
across the op boundary is expensive and the caller treats invalid input as a
fallback signal, not an error. Both decode-into ops now share
base64_decode_into_slice; the standard op keeps its strict pre-pass for
clean padded input.

base64Write's fallback goes straight to the cleaning path
(base64CleanToBytes) instead of through base64ToBytes, whose eager decode
attempt would always throw after a sentinel: the op and the wrapper share
the same loose decode, so an input that produced -1 can never pass it.

Deletes the orphaned forgivingBase64DecodeInto infra export (no consumers
remained).
â€Ķcode

base64Slice split at 4096 between op_base64_encode (#[string] return) and
op_base64_encode_from_buffer. Re-measured on main during the denoland#36398 review,
the from-buffer op (direct one-byte V8 string) wins at every size, so the
split predates that encode path and is now pure overhead. base64Slice uses
the from-buffer op unconditionally, matching base64urlSlice.

That leaves op_base64_encode with no caller that isn't better served by
op_base64_encode_from_buffer, so it is deleted: forgivingBase64Encode and
the jupyter image display rewire to the from-buffer op. hash.digest("base64")
rides along via forgivingBase64Encode.
op_base64_btoa returns via the shared base64_encode_to_v8_string helper
instead of a Rust String, skipping one full copy of the output through the
op glue. Error semantics are unchanged: the InvalidCharacterError mapping
lives in the JS wrapper's ByteString-conversion catch, which this does not
touch.

@bartlomieju bartlomieju left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good follow-up to #36398. The collapse of the two helper families into one parameterized set is a clear net win — the base64/base64url pair had already drifted (the capacity assert! and the debug_assert_eq! placement only existed on the url side), and unifying them fixes that drift rather than just deduplicating it.

Things I verified rather than assumed:

  • The capacity assert can't fire. simdutf_base64_decode_into now asserts output.len() >= maximal_binary_length_from_base64(input) on every path. All four call sites satisfy it: the strict pre-pass in op_base64_decode_into and the fast path in base64_decode_into_slice both guard on target.len() >= max_len, and the two stack-buffer sites guard on max_len <= STACK_BUF_SIZE with a full-size STACK_BUF_SIZE slice.
  • The base64Write fallback rewiring is sound. The old fallback went through base64ToBytes, whose first attempt is op_base64_decode → simdutf_base64_decode_to_vec(Default, Loose) — exactly the decode the op already ran to produce -1. It could only ever throw. Going straight to base64CleanToBytes is the same observable behavior. The >U+00FF case also lands identically: the op2 onebyte-string conversion throws, written stays -1, and cleaning strips the invalid chars — same as the old path's inner catch.
  • op_base64_encode has no remaining callers. forgivingBase64Encode and both jupyter sites are rewired; every forgivingBase64Encode caller (websocket userinfo via core.encode, filereader DataUrl, hash.digest) passes a real TypedArray, so TypedArrayPrototypeGetByteLength is safe there. forgiving_base64_encode keeps its one out-of-crate consumer in cli/ops/jupyter.rs, so it's correct to leave it.
  • forgivingBase64DecodeInto really was orphaned — no consumers in tree.
  • op_base64_btoa returning a v8::Local<v8::String> doesn't move error semantics. InvalidCharacterError comes from the ByteString conversion on the input, which is unchanged.

Pushed one trivial thing: ext/web/README.md still listed op_base64_encode in "Provided ops". (That list is also missing the four base64url ops added in #36398 — separate pre-existing gap, not yours to fix here.)

Benchmark table and the corpus/property-test verification are more than enough. CI green. LGTM.

@petamoriken

Copy link
Copy Markdown
Contributor

Native support for base64 conversion is now available using Uint8Array, so it might be a good idea to compare it with using this method.
https://github.com/tc39/proposal-arraybuffer-base64

@tomas-zijdemans

Copy link
Copy Markdown
Contributor Author

Ran the numbers: for base64 these ops beat the native methods, so nothing changes here. But the native methods beat our hex paths 3-12x, so I'll follow up with a PR switching Buffer's hex over to them. As soon as this merges ðŸĪž

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants