Skip to content

[BUG] Kernel dispatch uses incorrect block_dim due to shared mutable state #1254

Closed
Bug
Closed
[BUG] Kernel dispatch uses incorrect block_dim due to shared mutable state#1254
Bug
Assignees
c0d1f1ed
Labels
bugSomething isn't working
Milestone
1.12.1

Description

@c0d1f1ed
c0d1f1ed
opened on Feb 24, 2026

Bug Description

Kernel dispatch can use the wrong block_dim value when the same module is loaded on different devices (CPU vs CUDA), causing out-of-bounds shared memory access and runtime failures.

Module.options["block_dim"] is shared mutable state that gets permanently modified by each load() call. Several code paths don't explicitly pass block_dim to Module.load(), causing them to use stale values from previous launches:

  1. Launch.__init__() (line 7060) - Used for command recording
  2. get_module_identifier() (line 2648) - Used for module hash calculation

This creates a race condition where a module compiled for one block dimension can be loaded for execution with a different dimension.

System Information

No response

Activity

  1. c0d1f1ed
    added this to the 1.12.1 milestone on Feb 24, 2026
  2. c0d1f1ed
    self-assigned this
    on Feb 24, 2026
  3. c0d1f1ed
    added
    bugSomething isn't working
    on Feb 24, 2026
  4. c0d1f1ed
    added the
    Bug
    issue type on Feb 24, 2026

  5. c0d1f1ed commented on Feb 24, 2026

    @c0d1f1ed
    c0d1f1ed
    on Feb 24, 2026
    ContributorAuthor

    The immediate bug was addressed in 9435922. But there's probably room for improvement. Copying the discussion on the code review here:

    I'm not sure if I have the best solution here. Currently kernel execution on the CPU only supports block_dim=1 , and we enforce that in launch() and launch_tiled() but then have to pass it to other parts correctly, which is what this MR fixes.

    I'll see if there's a more elegant solution by doing the override at a later point, like at load(). I also question whether we should really be altering the whole module's block_dim (which I believe should act as a default if not provided). It currently defines WP_TILE_BLOCK_DIM at the .cu level, but perhaps we should use blockDim instead?

    Yeah I remember this being a messy issue, but we didn't have a better solution at the time. As you mention it defines WP_TILE_BLOCK_DIM, but that is used in several compile-time definitions so I don't think switching to blockDim would be feasible. There might be a better solution on the Python side.

  6. pull
    added a commit that references this issue on Feb 24, 2026

    Merge branch 'ncapens/fix-block-dim-dispatch' into 'main'

    9435922
  7. shi-eric
    closed this as completedon Feb 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions