Skip to content

sdpa_vector_2pass_1_gqa kernel batch offset for K/V in gqa decode - #4431

Merged
zcbenz merged 1 commit into
ml-explore:mainfrom
dudududukim:du-gqa-batch-fix
Sep 2, 2026
Merged

zcbenz merged 1 commit into
ml-explore:mainfrom
dudududukim:du-gqa-batch-fix

Conversation

@dudududukim

Copy link
Copy Markdown
Contributor
  • ✅ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: Claude was used as coding agent and i've cross-checked the written codes.

Fix batching idx for sdpa_vector_2pass_1_gqa kernel's K/V pointer

Following my recent PRs(#4077, #4380) introduced sdqa_vector_2pass_1_gqa for eliminating redundancy reading for each KV head, there was bug when using batched decode without mask.

So the batch idx is added for pointer update and properly works for batch inputs.

kernel batch offset status
sdpa_vector (single-pass) folded into tid.x → kv_head_idx = q_batch_head_idx / gqa correct
sdpa_vector_2pass_1 (non-GQA) explicit kv_batch_head_idx = batch_idx*num_kv_heads+kv_head_idx correct
sdpa_vector_2pass_1_gqa batch term missing THIS FIX

Test for before fix and after fix (following CONTRIBUTION.md)

  • New test case (B, L) = (2, 8192) added to test_sdpa_vector_gqa_long: FAILS on upstream main (AssertionError), PASSES with the fix.
  • No regression: B == 1 is byte-identical (batch_idx == 0 → kv_batch_head_idx == kv_head_idx), so all existing (B==1) benchmark shapes are unchanged by construction. bench_sdpa_gqa.py two-build compare confirms within measurement noise (the gqa-4 control, identical code on both builds, drifts as much as the gated kernels — that drift is the noise floor).
  • pre-commit (clang-format / black / isort) passes on the 2 files.

Formatting

uvx pre-commit run --all-files ✅ PASSED

@nastya236
nastya236 requested a review from jagrit06 August 30, 2026 19:17

@zcbenz zcbenz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants