Repository navigation
Conversation
nanjiangwill
marked this pull request as ready for review
August 21, 2026 05:46
nanjiangwill
requested review from
BBuf,
FlamingoPg,
HaiShaw,
ispobock,
merrymercy and
yizhang2077
as code owners
August 21, 2026 05:46
nanjiangwill
force-pushed
the
fix/speculative-zero-probability-boundary
branch
from
August 21, 2026 05:55
44f8d10 to
ba34758
Compare
Collaborator
|
/rerun-test python/sglang/kernels/aot/tests/speculative/test_speculative_sampling.py |
Contributor
|
Results for ⛔ |
hnyls2002
requested review from
DarkSharpness,
Fridge003,
HydraQYH,
Kangyan-Zhou,
Qiaolin-Yu,
Ying1123,
bingxche,
celve,
hnyls2002,
kpham-sgl and
yuan-luo
as code owners
September 17, 2026 21:56
hnyls2002
changed the title
github-actions
Bot
added
the
dependencies
Use half-open sampling intervals and require positive target mass before accepting a draft token in the shared target-only sampler.\n\nAdd native regression coverage for zero-mass drafts, exact CDF boundaries, and endpoint coins.\n\nFixes sgl-project#35771
hnyls2002
force-pushed
the
fix/speculative-zero-probability-boundary
branch
from
September 18, 2026 00:34
7e5c943 to
705cb5d
Compare
hnyls2002
changed the title
TreeSpeculativeSamplingTargetOnly
Collaborator
|
/rerun-test test_speculative_sampling.py |
Contributor
|
Results for 🚀 |
nanjiangwill
added a commit
to modal-projects/sglang
that referenced
this pull request
Sep 19, 2026
…g CDF boundaries Backport the corrected target-only sampler through the JIT kernel path available on main. Reject zero target mass and use half-open CDF intervals for DFlash and EAGLE. Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
nanjiangwill
added a commit
to modal-projects/sglang
that referenced
this pull request
Sep 19, 2026
…gl-project#35798) Reject zero-probability candidates and use half-open CDF intervals in the v0.5.20 AOT sampler.
nanjiangwill
added a commit
to modal-projects/sglang
that referenced
this pull request
Sep 20, 2026
…gl-project#35798) Reject zero-probability candidates and use half-open CDF intervals in the v0.5.20 AOT sampler.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 24, 2026
…SpeculativeSamplingTargetOnly Upstream 740f57a (sgl-project#35798, nur main). Upstream liegt der Kernel im JIT- Baum (kernels/jit/include/sgl_kernel/speculative/sampling.cuh), im Fork im AOT-Baum sgl-kernel/csrc/speculative/speculative_sampling.cuh:80. Portiert (nur Quelle, NICHT gebaut): Annahme nur mit Zielmasse (target_prob_single > 0) und halboffene CDF-Grenze (coin < acc statt <=). Vorher akzeptierte der Kernel bei coin == 0 einen Draft-Token, dem das Ziel Null Masse gibt, und nahm an einer inneren Intervallgrenze das falsche Intervall. Tests: upstream-Erwartungen fuer den threshold-0/0-Fall angepasst und test_target_only_sampling_cdf_boundaries (5 Faelle) in sgl-kernel/tests/speculative/test_speculative_sampling.py (CUDA + neu gebautes sgl_kernel noetig, hier nicht lauffaehig). Anwendbarkeit 27B: NICHT auf dem Standardpfad. D mit DFlash2 akzeptiert T>0-Runden ueber den Selector (_selector_sampling_accept -> dspark accept_sampling -> reject_sampling.chain_speculative_sampling_triton, dort "coin * q < p" = strikt, Nullmasse wird nie angenommen). Der CUDA- Kernel laeuft nur im DFLASH-Rueckfall ohne Selector (compute_dflash_sampling_correct_drafts_and_bonus, dflash_utils.py:763; z.B. SGLANG_DFLASH_DISABLE_SELECTOR=1 oder ein DFLASH-Draft ohne Selector) und bei EAGLE/NEXTN mit Verify und T>0 (P ist draft-kv-only). Pruefung ohne GPU: Python-Emulation des Annahme-Praedikats (float32) ueber die 5 upstream-Grenzfaelle: neu = erwartete Tokens/Anzahl, alt nimmt in 3 Faellen falsch an (Nullmasse bei coin 0, innere Grenze). Kein Kernel-Lauf. Build (NICHT ausgefuehrt): sgl-kernel-Wheel aus diesem Baum mit SGL_KERNEL_LIMIT_CUDA_ARCHS='86;120a' (CMake-Cache, CMakeLists.txt:250) bauen, in /spinning/htsglang-gpu/.venv ersetzen (derzeit sglang-kernel 0.4.4 aus /spinning/wt-398-wheel, 03.08.) -- ausserhalb eines Boots, danach pytest sgl-kernel/tests/speculative/test_speculative_sampling.py auf einer Karte (sm86 und sm120).
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
TreeSpeculativeSamplingTargetOnlydraws coins from[0, 1)but selects a target CDF bucket withcoin <= cumulative_probability. At the lower endpoint (coin == 0) that accepts a draft token whose target probability is zero, and exact interior CDF boundaries land in the preceding bucket instead of the next positive-mass one.Fixes #35771.
Modifications
coin < prob_acc / threshold_accand requiretarget_prob_single > 0before either acceptance path can take a draft, in the shared device headersgl_kernel/speculative/sampling.cuh.Single-site change: the shared header backs the CUDA/ROCm JIT launchers and the MUSA AOT build, so all three pick the fix up. Rebased onto #40033, which moved these kernels to JIT — CUDA compiles them at runtime, so the fix ships without an
sgl-kernelrelease.Accuracy Tests
test/registered/kernels/ops/speculative/test_speculative_sampling.py:7 passed.4 failed, 3 passed—zero-mass-zero-threshold,zero-mass-default-threshold,interior-boundary, and the zero-threshold tree case all regress, confirming the new coverage is what guards the fix.Checklist