[CNSeniorious000]

• <- Keep this line and put an x between the brackts.

Description

ShikiStreamRenderer (React) renders its tokens as one flat tokens.map(...) list, so every chunk that arrives re-reconciles the entire code block. Per-chunk work grows with the length of the block rather than with the size of the chunk, which is the wrong shape for the case this package targets — a long block streaming out of an LLM.

This groups tokens into lines and wraps each finished line in memo, so React skips everything above the line currently being appended to.

The rendered DOM does not change. Lines are grouped with a Fragment, which emits no element, so the output is still one flat list of <span>s directly under <code>. In the benchmark below this PR and main produce byte-identical DOM mutation and node-creation counts.

Measured with the reproduction from #1310 — a ~640-line TypeScript block streamed 32 bytes per animation frame, 783 frames, 5 interleaved rounds (each round rotates which arm runs first so a drifting system load hits all of them equally), median ± IQR:

renderer	DOM mutations	nodes created	wall clock
no highlighting at all (floor)	782	1	15.49s ±0.17
ShikiStreamRenderer on main	18516	12556	29.21s ±7.83
ShikiStreamRenderer with this PR	18516	12556	15.52s ±0.41

DOM writes are unchanged and wall clock drops by ~47%, which puts the win entirely in reconciliation. With this change, streaming a long highlighted block costs the same as not highlighting it at all.

I also tried switching the span key from objectId(token) to an index, which lets React patch recalled tokens in place instead of replacing the node. It only moved wall clock from 15.52s to 15.27s — inside the noise — and it would change the "a replaced token is a brand-new node" behaviour that svelte-renderer.test.ts asserts on purpose, so I left the key alone and kept this PR to the grouping change.

recall handling: the tokenizer sets recall to tokensUnstable.length, i.e. only ever the tokens of the line currently being streamed, so recall() just trims the tail.

Linked Issues

fixes #1310

Additional context

New test file packages/stream/test/react-renderer.test.ts, modelled on the existing Svelte renderer test. Two of its three cases fail on main and pass with this change — a token in a completed line gets its style read 22 times as 20 further chunks arrive, and 2 times with this PR:

AssertionError: expected 22 to be 2  // does not re-render completed lines when later chunks arrive
AssertionError: expected 5 to be 2   // drops recalled tokens and keeps completed lines intact

The test observes re-render scope through a token whose htmlStyle getter counts reads, since reconciliation depth isn't visible in the DOM.

react-dom and @types/react-dom are added as devDependencies (both already in the workspace catalog) — the package had react but no renderer, so there was no way to mount the React component in a test.

This PR was written with the help of an AI agent; I've read through the diff and run the benchmark and tests myself.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[netlify]

❌ Deploy Preview for shiki-next failed.

Name	Link
🔨 Latest commit	7e192b7
🔍 Latest deploy log	https://app.netlify.com/projects/shiki-next/deploys/6a90fc147e6f8e00087794fc

[netlify]

❌ Deploy Preview for shiki-matsu failed.

Name	Link
🔨 Latest commit	7e192b7
🔍 Latest deploy log	https://app.netlify.com/projects/shiki-matsu/deploys/6a90fc14184e2f0008878013

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.75%. Comparing base (48cd2cc) to head (7e192b7).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1311   +/-   ##
=======================================
  Coverage   89.75%   89.75%           
=======================================
  Files          79       79           
  Lines        3552     3552           
  Branches     1008     1006    -2     
=======================================
  Hits         3188     3188           
  Misses        327      327           
  Partials       37       37           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[CNSeniorious000]

The 5 failing Netlify checks are not caused by this PR — the docs deploy is currently broken on main.

They fail identically on #1308 and #1309, and I reproduced it locally on an unmodified main (48cd2cc) by running the netlify.toml build command:

pnpm run build && pnpm run docs:build

pnpm run build succeeds (23/23 tasks); docs:build then fails during page rendering:

build error:
__VUE_PROD_DEVTOOLS__ is not defined
ReferenceError: __VUE_PROD_DEVTOOLS__ is not defined
    at createPinia (node_modules/.pnpm/pinia@4.0.2_.../pinia/dist/pinia.js:847:48)
    at Object.enhanceApp (docs/.vitepress/.temp/app.js:9408:11)

pinia@4.0.2 reads __VUE_PROD_DEVTOOLS__ at createPinia() time, and the flag isn't defined in vitepress 2.0.0-alpha.19's SSR build. Happy to open a separate issue for it if that's useful.

All GitHub Actions checks on this PR pass (lint, typecheck, and test on 22.x/24.x × ubuntu/macos/windows).

[CNSeniorious000]

Thanks for merging these, @antfu!