Skip to content

fix(transformers): handle split # and -- comments in rose-pine theme - #1317

Merged
antfu merged 2 commits into
shikijs:mainfrom
anandghegde:fix/transformers-rose-pine-hash-comments
Oct 1, 2026
Merged

antfu merged 2 commits into
shikijs:mainfrom
anandghegde:fix/transformers-rose-pine-hash-comments

Conversation

@anandghegde

Copy link
Copy Markdown
Contributor
  • <- Keep this line and put an x between the brackts.

Description

The rose-pine themes tokenize line comments into two tokens: # and [!code ++]. #1118 added a lookback for this split, but it only accepted a previous token containing //. That means [!code ...] notations did nothing in yaml, bash, python, sql, lua and other languages that use # or -- comments. The notation text showed up in the rendered output.

This PR changes the lookback check. The previous token can now be any line comment prefix the existing matcher already supports: //, #, ", ', ;, % or --. The combined text still has to pass matchToken, so nothing else changes.

Checked with rose-pine-dawn against dracula, which keeps each comment as one token:

lang tokens (rose-pine-dawn) before after
yaml foo # [!code ++] not applied applied
bash / python … # [!code ++] not applied applied
sql / lua … -- [!code ++] not applied applied
javascript … // [!code ++] applied applied

Tests added to notation-diff-rose-pine.test.ts:

  • yaml, bash and sql inline notations;
  • a # comment-only line with [!code highlight:2], checking that the line is removed and the next 2 lines get highlighted.

Both fail without the change. pnpm build && pnpm test passes (631 tests) and pnpm lint is clean.

I used Claude Code to implement this. I reviewed the diff and ran the tests myself.

Linked Issues

fixes #1272

Additional context

Block comments are still split into three tokens in rose-pine (<!--, [!code ++], --> for html, and /* … */ for css), so they are still not handled. The v3 algorithm only looks at the last token on the line, so supporting them would take a bigger change. I left that out to keep this PR small. I can follow up if you'd like.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MA8JGs7pCoXFL2W6jEJ8sD

Rose-pine tokenizes `# [!code ++]` into `#` and ` [!code ++]`. The
multi-token lookback from shikijs#1118 only accepted a `//` prefix, so notation
transformers silently did nothing for yaml, bash, python, sql, lua, etc.
Accept any line comment prefix the matcher already supports.
@netlify

netlify Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for shiki-matsu ready!

Name Link
🔨 Latest commit a485e3a
🔍 Latest deploy log https://app.netlify.com/projects/shiki-matsu/deploys/6aa974e264675b0008fb15cf
😎 Deploy Preview https://deploy-preview-1317--shiki-matsu.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@netlify

netlify Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for shiki-next ready!

Name Link
🔨 Latest commit a485e3a
🔍 Latest deploy log https://app.netlify.com/projects/shiki-next/deploys/6aa974e23642570008f5da4b
😎 Deploy Preview https://deploy-preview-1317--shiki-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.85%. Comparing base (ac8e1c4) to head (a485e3a).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1317      +/-   ##
==========================================
+ Coverage   89.82%   89.85%   +0.03%     
==========================================
  Files          81       81              
  Lines        3587     3588       +1     
  Branches     1028     1026       -2     
==========================================
+ Hits         3222     3224       +2     
+ Misses        328      327       -1     
  Partials       37       37              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@anandghegde
anandghegde force-pushed the fix/transformers-rose-pine-hash-comments branch from a4439c2 to a485e3a Compare September 15, 2026 16:40
@anandghegde

Copy link
Copy Markdown
Contributor Author

Heads up on the red check: typecheck is failing on bench/bundle-test/bundle.bench.ts:27 and bench/engines/engines.bench.ts:45 with TS2339: Property 'compare' does not exist on type 'TestAPI'.

That is pre-existing on main — the same two errors fail the typecheck job on ac8e1c4 (push to main, 11 Sep), before this branch existed. This PR only touches packages/transformers/src/shared/parse-comments.ts and its test; nothing under bench/. lint, codecov, and all eight test matrix jobs are green here.

Looks like fallout from a Vitest major, where bench.compare went away. Happy to send that as a separate PR if it is not already being handled.

@antfu
antfu merged commit 63e6da8 into shikijs:main Oct 1, 2026
14 of 15 checks passed
antfu pushed a commit to SHUBHI-KUMARI/shiki that referenced this pull request Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transformers not work with rose-pine theme and some langs like bash, yaml...

2 participants