Fix async parser decoding of short ASCII values split across input feeds - #770
Merged
cowtowncoder merged 2 commits intoSep 5, 2026
Merged
Conversation
`NonBlockingByteArrayParser._finishToken()` compared `_minorState` against `MINOR_FIELD_NAME_SHORT_ASCII` inside the `MINOR_VALUE_STRING_SHORT_ASCII` / `MINOR_VALUE_STRING_SHORT_UNICODE` case. Those constants can never be equal, so the check was always false and short ASCII *values* that span a feed boundary were always decoded via `_decodeShortUnicodeText()`. Result: the same document could decode to different Strings depending on how the caller chunked the input. The equivalent check for property names (line 233) is correct and is left as-is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbhNqemoB4csempvUS9n5z
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbhNqemoB4csempvUS9n5z
cowtowncoder
deleted the
tatu-claude/2.21/async-smile-short-ascii-chunk
branch
September 5, 2026 00:43
cowtowncoder
added a commit
that referenced
this pull request
Sep 5, 2026
Also add 2.22.3 release notes entry for #770.
cowtowncoder
added a commit
that referenced
this pull request
Sep 5, 2026
Also add 3.2.3 release notes entry for #770.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
NonBlockingByteArrayParser._finishToken(), theMINOR_VALUE_STRING_SHORT_ASCII/MINOR_VALUE_STRING_SHORT_UNICODEcase picks its decoder with:MINOR_FIELD_NAME_SHORT_ASCIIis5; the two case labels are16and17. Thecondition can therefore never be true, and short ASCII values that span a feed
boundary are always decoded as short Unicode.
For well-formed ASCII content both decoders agree, so this is invisible in normal use.
For content that is not valid ASCII, the same document decodes to different Strings
depending on how the caller chunks the input â a
0xC5byte inside a short-ASCII tokenyields
aïŋ―cdwhen fed contiguously butaÅĢdwhen fed one byte at a time.The equivalent check for property names (line 233) tests the right constant and is
left unchanged. These are the only two such comparisons in the async parser; the third
(
MINOR_HEADER_INLINE, line 396) is reachable and correct. CBOR has no async parser on2.x, so Smile is the only affected backend.
Fix
Compare against
MINOR_VALUE_STRING_SHORT_ASCII.Test
SimpleStringArrayTest.testShortAsciiValueChunkIndependencewrites a short ASCII value,corrupts one content byte so the two decoders disagree, then asserts that feed sizes of
length+1, 3 and 1 all produce the same String. It fails on the unfixed code withexpected: <aïŋ―cd> but was: <aÅĢd>.Full smile module suite: 270 tests, 0 failures. CI green on JDK 8/17/21.
Note for merge-forward
3.x already carries this same one-token fix via #767, so the merge-forward will conflict
on that line at 3.x (either side is fine to take). 3.1 and 3.2 still have the bug and do
need it. The test will also need the 3.x API names (
SmileReadFeature, noIOException)when it reaches the 3.x side.
ðĪ Generated with Claude Code
https://claude.ai/code/session_01BbhNqemoB4csempvUS9n5z