Activity

1. LauraGPT commented on Aug 11, 2026

   

   LauraGPT

   on Aug 11, 2026

   Last edited by LauraGPT

   Collaborator

   More actions

   Thanks for the detailed report. I traced the runtime pin and merged #3482 as 500956b.

   What we verified:

   • The January AMD coopmat fix cited in this issue (1051ecd2) was already present in runtime v0.1.9, so that alone cannot explain the remaining crash.
   • The old FunASR pin predates newer Vulkan timeout work: configurable submission batches (51eae8cf), reduced thresholds for small AMD GPUs (0bbc87b1), and the corrected batching/DeviceLost diagnostics (803b7fca).
   • The SenseVoiceSmall graph does not create a flash-attention operation, so adding --no-flash-attn would not affect this path.
   • With the refreshed pin, all three CPU binaries build locally and the SenseVoice Vulkan target builds and links successfully on Linux.
   • Both complete nine-platform workflows passed, including Windows x64 Vulkan and Windows x64 CUDA.
   • The final public Release contains exactly nine archives. Every archive was downloaded again, passed tar/ZIP integrity checks, and matched its GitHub SHA-256 digest.
   • The public Windows Vulkan ZIP SHA-256 is 90b45240c6ccc9177c25490a11848de60a406e129391c8736b14521c0c28cdcb. Its SenseVoice executable SHA-256 is 699fa877fecbaa98719b39e9aca72f61cbfdfeeda6641a41399e24142583a91b and embeds both diagnostic environment variables.

   Stable release:
   https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.2.0

   Direct Windows Vulkan download:
   https://github.com/modelscope/FunASR/releases/download/runtime-llamacpp-v0.2.0/funasr-llamacpp-windows-x64-vulkan.zip

   Please test that ZIP on the RX 7600M XT / 780M machine. The updated docs expose GGML_VK_MAX_NODES_PER_SUBMIT and GGML_VK_SERIALIZE_SUBMISSIONS so a remaining driver failure can produce actionable diagnostics. This release improves batching and diagnostics but does not claim the hardware-specific driver crash is fixed until this test succeeds. CPU remains the reliable fallback.

   I am keeping this issue open until the Windows AMD hardware result is known.
2. LauraGPT

   added

   bugSomething isn't workingSomething isn't working

   needs feedbackWaiting for reporter feedback or retest resultsWaiting for reporter feedback or retest results

   on Aug 11, 2026
3. LauraGPT

   added a commit that references this issue on Aug 11, 2026

   fix(vulkan): refresh ggml for AMD submission handling (#3482)

   ...

   500956b
4. chengligen commented on Aug 13, 2026

   

   chengligen

   on Aug 13, 2026

   Last edited by chengligen

   Contributor

   More actions

   This is consistent with the known ggml Vulkan backend instability on AMD Windows drivers (RDNA3). 0xc0000005 is an access violation — typically an invalid memory access inside a shader (often the flash-attention coopmat path) surfacing as a driver-triggered crash.

   Since you've already tried coopmat / integer dot product / validation / driver rollbacks, the remaining isolation steps:

   1. Flash attention: if your build exposes a flash-attn toggle, disable it. Vulkan FA uses coopmat on RDNA3 and can still crash even when coopmat is globally disabled.
   2. Force FP32 path: set GGML_VK_DISABLE_F16=1 — some AMD shader bugs only manifest on the FP16 dispatch path.
   3. Disable SSBO: set GGML_VK_DISABLE_SSBO=1 — changes descriptor binding, bypasses driver indexing bugs.
   4. Pinpoint the failing op: set LLAMA_VK_RUN_TESTS=1 and run the exe; it runs ggml's Vulkan op tests and reports the first failing op — this tells us whether it's the matmul path or something narrower.
   5. Check the crash module: Event Viewer → Windows Logs → Application, find the crash record. If the faulting module is amdvlk64.dll / amdihk64.dll / aticfx64.dll, it's a driver fault; if it's the exe itself, it's a ggml host-side bug.

   If it still crashes after all of the above, the pragmatic conclusion is that the Vulkan backend isn't usable on AMD+Windows in the current build — worth documenting as a known limitation (CPU/OpenCL as fallback) and reporting a minimal repro upstream to ggml-org/llama.cpp.

   A minidump from %LocalAppData%\CrashDumps plus the first failing op from step 4 would give the fastest path to a real fix.
5. LauraGPT commented on Aug 13, 2026

   

   LauraGPT

   on Aug 13, 2026

   Collaborator

   More actions

   Maintainer clarification after checking the exact source shipped in runtime-llamacpp-v0.2.0:

   • GGML_VK_DISABLE_F16, GGML_VK_DISABLE_SSBO, and LLAMA_VK_RUN_TESTS are not implemented by this pinned runtime, so setting them cannot change or diagnose this executable.
   • The SenseVoice graph builds attention from mul_mat -> scale -> soft_max -> mul_mat; it does not create a flash-attention operation. A flash-attention toggle therefore would not isolate this crash.
   • The supported AMD Windows diagnostics in v0.2.0 are GGML_VK_MAX_NODES_PER_SUBMIT and GGML_VK_SERIALIZE_SUBMISSIONS.

   For the original reporter, please test the v0.2.0 Windows Vulkan ZIP first, then retry from PowerShell with:

   $env:GGML_VK_MAX_NODES_PER_SUBMIT = "16"
   $env:GGML_VK_SERIALIZE_SUBMISSIONS = "1"
   .\llama-funasr-sensevoice.exe `
     -m models\sensevoice-small-f16.gguf --vad models\fsmn-vad.gguf `
     -a audio\examples\zh.mp3 --backend vulkan

   If 16 still crashes, retry with 8, then 1. Please attach the complete stderr, GPU selected by the runtime, driver version, and the Event Viewer faulting module. A minidump is also useful if available.

   We should not conclude that all AMD Windows Vulkan devices are unsupported until the refreshed runtime is tested on this reported hardware. The issue remains open with needs feedback for that result.
6. rjffrwy16-ux commented on Aug 21, 2026

   

   rjffrwy16-ux

   on Aug 21, 2026

   More actions

   Hi, I can also reproduce a very similar Vulkan crash on an AMD Radeon RX 9070 XT under Windows 11.

   I'm new to GitHub and this is my first time posting here, so please let me know if I should provide anything differently or if additional diagnostic information would be useful.

   Environment

   • OS: Windows 11
   • GPU: AMD Radeon RX 9070 XT
   • FunASR runtime: runtime-llamacpp-v0.1.9
   • Model: SenseVoiceSmall-GGUF q8
   • Model revision: 90c1c61912018b70ada0fcc024ea24aca62f2e63
   • Backend: Vulkan

   Vulkan device enumeration succeeds and the RX 9070 XT is detected, but the native process crashes before successful model inference with:

   Exception code: 0xC0000005
   Operation: read
   Address: 0x0000000000000060
   

   I reproduced the same crash after:

   1. Restricting execution to a single detected GPU
   2. Disabling cooperative matrix support

   Both resulted in the same access violation.

   I intentionally did not test CPU fallback because my goal was specifically to validate the Windows Vulkan path.

   I could not reach successful model inference, transcript output, or a normal process exit. I also do not currently have a symbol-enabled dump, so the exact crashing function is unknown.

   I noticed the maintainer comments above about the refreshed runtime-llamacpp-v0.2.0 and the supported diagnostics:

   • GGML_VK_MAX_NODES_PER_SUBMIT
   • GGML_VK_SERIALIZE_SUBMISSIONS

   I have not tested v0.2.0 yet, so this comment only confirms an additional RX 9070 XT reproduction on v0.1.9.

   If useful, I can provide:

   • the crash screenshot
   • the exact command line
   • Vulkan device information
   • runtime/model SHA-256 hashes
   • additional diagnostic output

   Thanks!
7. rjffrwy16-ux commented on Aug 21, 2026

   

   rjffrwy16-ux

   on Aug 21, 2026

   More actions

   runtime-llamacpp-v0.2.0 follow-up on Radeon RX 9070 XT

   Following up on my previous RX 9070 XT report: I tested the official runtime-llamacpp-v0.2.0 Windows Vulkan build.

   Unfortunately, the crash is still reproducible.

   Environment

   • OS: Windows 11
   • GPU: AMD Radeon RX 9070 XT
   • AMD Software: 26.7.1
   • Windows display driver: 32.0.31035.1003
   • Runtime: runtime-llamacpp-v0.2.0
   • Backend: Vulkan

   Tests performed

   I tested all of the following:

   1. Default Vulkan configuration
   2. GGML_VK_SERIALIZE_SUBMISSIONS=1 + GGML_VK_MAX_NODES_PER_SUBMIT=16
   3. Same with GGML_VK_MAX_NODES_PER_SUBMIT=8
   4. Same with GGML_VK_MAX_NODES_PER_SUBMIT=1

   All four paths terminated with the same process return code:

   0xC0000005
   

   They also produced identical stderr:

   • Size: 399 bytes
   • SHA-256: 6186F15212F693ED5F76553B94175180C942BD0758A555B987C30CF32CA76D46

   ggml_vulkan: Found 2 Vulkan devices:
   ggml_vulkan: 0 = AMD Radeon RX 9070 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
   ggml_vulkan: 1 = AMD Radeon(TM) Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
   

   The process stops immediately after Vulkan device enumeration.

   It does not reach:

   • model load
   • inference
   • transcript output
   • CPU fallback

   Therefore I did not run the formal ASR benchmark after this preflight failure.

   GPU selection caveat

   My test summary records the RX 9070 XT as the intended/selected GPU.

   However, the preserved raw stderr ends immediately after device enumeration and does not contain an explicit selected device line.

   So, to be precise, the raw stderr itself directly proves that the RX 9070 XT was enumerated as Vulkan device 0, but it does not contain a later explicit device-selection confirmation.

   Windows Event Viewer

   I also checked the relevant execution window in Windows Event Viewer.

   No matching Application Error / Event ID 1000 was emitted for these v0.2.0 runs.

   Therefore I cannot provide:

   • faulting module name
   • exception code from Event Viewer
   • fault offset

   The 0xC0000005 value above comes from the independently captured process return code, not from Event Viewer.

   So on this RX 9070 XT system, reducing submission size from 16 → 8 → 1 and serializing submissions did not change the failure behavior.

   If there are any additional diagnostics supported by the v0.2.0 build that would help isolate this further, I'd be happy to test them.

   Thanks!
8. terrytotalk commented on Aug 21, 2026

   

   terrytotalk

   on Aug 21, 2026

   More actions

   I tested the v0.2.0 Windows Vulkan ZIP (SHA-256 verified: 90b45240c6ccc9177c25490a11848de60a406e129391c8736b14521c0c28cdcb) on AMD Radeon 780M (Windows 11). My issue is different from the original crash
   report:

   The program does not crash. Instead it reports:

   ggml_vulkan: Found 1 Vulkan devices:
   ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
   Vulkan backend requested, but no Vulkan GPU backend is available; build with -DGGML_VULKAN=ON and install a Vulkan driver/ICD

   The Vulkan device is detected, but the ggml backend registry does not contain a Vulkan backend entry.

   Environment:

   • OS: Windows 11
   • GPU: AMD Radeon 780M (RDNA3, integrated only)
   • AMD Driver: 26.7.1 (LLPC), Vulkan API 1.4.349
   • System vulkan-1.dll: 1.4.341
   • Vulkan SDK 1.4.357.0 installed (glslc available)
   • No GGML_DISABLE_VULKAN or other GGML env vars set

   Tested with:

   $env:GGML_VK_MAX_NODES_PER_SUBMIT = "16"
   $env:GGML_VK_SERIALIZE_SUBMISSIONS = "1"
   .\llama-funasr-sensevoice.exe -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a test.mp3 --backend vulkan

   Same result — device enumerated but backend not available.

   vulkaninfo --summary correctly shows the 780M. CPU mode (--backend cpu or no flag) works fine.
9. LauraGPT commented on Aug 25, 2026

   

   LauraGPT

   on Aug 25, 2026

   Collaborator

   More actions

   Thanks @terrytotalk — I traced the 780M result against current main (8c60d278) and the pinned llama.cpp revision (803b7fca). This is a separate, now-confirmed selector bug rather than a missing Vulkan driver or registry entry:

   • Vulkan enumerates the Radeon 780M, and the pinned ggml Vulkan backend classifies integrated devices as GGML_BACKEND_DEVICE_TYPE_IGPU.
   • find_gpu_backend_device in runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cpp currently rejects every device whose type is not GGML_BACKEND_DEVICE_TYPE_GPU before matching the requested backend name.
   • As a result, an enumerated Vulkan iGPU is deterministically skipped and FunASR emits the misleading “no Vulkan GPU backend is available” message.

   The bounded correction is to prefer a matching dedicated GPU, then fall back to a matching integrated GPU, while still rejecting CPU or unrelated backends when --backend vulkan is requested. No further driver/ICD diagnostics are needed to establish this 780M selector failure.

   The RX 9070 XT 0xC0000005 report remains distinct: it gets past enumeration and fails during or immediately around backend initialization. The iGPU selector correction must not be represented as fixing that discrete-GPU crash, so this issue should remain open while the RX path is investigated.
10. LauraGPT commented on Aug 27, 2026

    

    LauraGPT

    on Aug 27, 2026

    Collaborator

    More actions

    Thanks @terrytotalk. The bounded iGPU selector fix described above has now shipped.

    • Fix PR: fix(runtime): allow integrated GPU backends #3531
    • Exact release commit: c69e69a0b78fb9becc8b2be91db8ae57d2a41f3c
    • Release: https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.2.1
    • Windows Vulkan ZIP: https://github.com/modelscope/FunASR/releases/download/runtime-llamacpp-v0.2.1/funasr-llamacpp-windows-x64-vulkan.zip
    • ZIP SHA-256: 5fd179c6aefc18477eb79fa6235c70d131a96eba621318bb84e901ae85035578

    The release workflow built all nine platform archives successfully from the exact commit above, including Windows x64 Vulkan. The selector now prefers a matching dedicated GPU and falls back to a matching integrated GPU instead of rejecting every GGML_BACKEND_DEVICE_TYPE_IGPU device.

    Could you please retest v0.2.1 on the Radeon 780M with the same command and report the complete stderr plus whether inference reaches a transcript?

    .\llama-funasr-sensevoice.exe `
      -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf `
      -a test.mp3 --backend vulkan

    This release does not claim to fix the separate RX 9070 XT 0xC0000005 path. A dedicated GPU was already accepted by the old selector, so that backend-initialization crash remains open and is documented separately from the 780M selector result. I am keeping this issue open for both the 780M hardware confirmation and further RX 9070 XT diagnosis.
11. terrytotalk commented on Aug 28, 2026

    

    terrytotalk

    on Aug 28, 2026

    More actions

    Thanks @LauraGPT . I have thoroughly tested runtime-llamacpp-v0.2.1 on Windows 11 with the AMD Radeon 780M iGPU.

    Tested with:
    .\llama-funasr-sensevoice.exe -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a "test.mp3" --backend vulkan --srt

    The program reported:
    ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat

    Then I tried:
    echo $LASTEXITCODE

    Reported:
    -1073741819

    Observation:
    • The iGPU selector fix in PR #3531 works: the 780M is now recognized and no longer throws the "no Vulkan GPU backend is available" error.
    • However, inference does not reach a transcript. Immediately after printing the device info, the process silently terminates/crashes with exit code -1073741819 (0xC0000005).
    • Setting $env:GGML_VK_MAX_NODES_PER_SUBMIT = "16" (or "1") and $env:GGML_VK_SERIALIZE_SUBMISSIONS = "1" does not change the behavior, because the crash occurs before graph execution/submission.
12. LauraGPT commented on Aug 28, 2026

    

    LauraGPT

    on Aug 28, 2026

    Collaborator

    More actions

    Thanks @terrytotalk and @rjffrwy16-ux. I have shipped a diagnostic follow-up built from the current fix:

    • Exact commit: 05be4863fc94e3df37924a849b9790484ac74a29
    • Release: https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.2.2
    • Windows Vulkan ZIP: https://github.com/modelscope/FunASR/releases/download/runtime-llamacpp-v0.2.2/funasr-llamacpp-windows-x64-vulkan.zip
    • ZIP SHA-256: 86a7d5ca7c134ae2fd3c9c1b356fcff041e29baf569c1145a75c73bb5bc5ea90
    • Build evidence: https://github.com/modelscope/FunASR/actions/runs/33182316846

    All nine platform archives completed successfully. I downloaded every public asset again, verified all tar/ZIP structures, and matched all nine actual SHA-256 values to GitHub's asset digests.

    v0.2.2 makes two bounded changes to CUDA/Vulkan initialization:

    1. a null return from ggml_backend_dev_init() is now checked before the default buffer type is queried;
    2. stderr is flushed at three boundaries: before backend initialization, after initialization, and after buffer-type resolution.

    Please retest the ZIP above with the same command and paste the complete stderr, including the final line printed before termination:

    .\llama-funasr-sensevoice.exe `
      -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf `
      -a test.mp3 --backend vulkan --srt
    echo $LASTEXITCODE

    The final boundary tells us where control was lost:

    • no initializing vulkan backend ...: enumeration/selection or an earlier path;
    • initializing ... but no initialized ...: inside ggml_backend_dev_init() or the Vulkan driver below it;
    • initialized ...; resolving buffer type but no vulkan backend ready ...: default buffer-type resolution;
    • vulkan backend ready ...: initialization succeeded and the failure is later in model loading or graph execution.

    This release does not claim that the AMD Windows 0xC0000005 crash is fixed. The null guard can report a normal initialization failure, but an access violation inside the Vulkan driver/backend initializer may terminate the process before FunASR regains control. I am keeping this issue open until the new boundary logs from both the Radeon 780M and RX 9070 XT paths are available.
13. terrytotalk commented on Aug 28, 2026

    

    terrytotalk

    on Aug 28, 2026

    More actions

    Thanks @LauraGPT , just tested v0.2.2

    Tested with:
    .\llama-funasr-sensevoice.exe -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a "test.mp3" --backend vulkan --srt

    The program reported:
    ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
    initializing vulkan backend on Vulkan0 (AMD Radeon 780M Graphics)
    initialized vulkan backend on Vulkan0; resolving buffer type
    vulkan backend ready on Vulkan0

    Then I tried:
    echo $LASTEXITCODE

    Reported:
    -1073741819

    Setting $env:GGML_VK_MAX_NODES_PER_SUBMIT = "16" (or "1") and $env:GGML_VK_SERIALIZE_SUBMISSIONS = "1" does not change the behavior.
14. LauraGPT commented on Aug 28, 2026

    

    LauraGPT

    on Aug 28, 2026

    Collaborator

    More actions

    Thanks @terrytotalk. The v0.2.2 boundary result is decisive: vulkan backend ready on Vulkan0 confirms that device selection, ggml_backend_dev_init(), and default buffer-type resolution all completed. The remaining 0xC0000005 occurs later, but the current executable has no stderr boundary between backend readiness and completion.

    I opened #3546 to add flushed boundaries around model loading, audio/VAD preparation, graph construction/allocation, and the first backend compute. This is diagnostic instrumentation only; it does not claim to fix the AMD Windows crash.

    • PR: fix(runtime): trace SenseVoice backend stages #3546
    • Exact head: 7429dbd787bb24fdff7a0dac5f48b59e4e3472b6
    • Nine-platform validation: https://github.com/modelscope/FunASR/actions/runs/33193912133

    Local CPU Release builds and complete inference paths with and without FSMN-VAD pass, including the new boundaries. The exact-head workflow is currently validating all nine packages, especially Windows x64 Vulkan and CUDA. I will not ask for another hardware retest or publish a diagnostic release until that matrix is green and the PR is merged.
15. LauraGPT commented on Aug 28, 2026

    

    LauraGPT

    on Aug 28, 2026

    Collaborator

    More actions

    The promised post-backend diagnostic release is now available:

    • Release: https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.2.3
    • Windows Vulkan ZIP: https://github.com/modelscope/FunASR/releases/download/runtime-llamacpp-v0.2.3/funasr-llamacpp-windows-x64-vulkan.zip
    • ZIP SHA-256: 1c90daf2292fcf9041c14a8017be65a05acc8e42443ee0bbc8a8dfddb78cbdc7
    • Exact commit: 820f1c6
    • Nine-platform build evidence: https://github.com/modelscope/FunASR/actions/runs/33197306623

    v0.2.3 adds flushed boundaries after backend readiness: model ready, graph allocated, and compute starting. Please retest with the same command and paste complete stderr plus $LASTEXITCODE. The final new boundary narrows the remaining 0xC0000005 stage:

    • backend ready without model ready: model loading;
    • model ready without graph allocated: graph allocation;
    • graph allocated without compute starting: preparation immediately before graph execution;
    • compute starting: graph execution or a lower backend/driver layer.

    All nine public assets were independently downloaded and matched byte-for-byte to their workflow artifacts and GitHub digests. This remains diagnostic instrumentation only and does not claim the AMD Windows crash is fixed.
16. terrytotalk commented on Aug 29, 2026

    

    terrytotalk

    on Aug 29, 2026

    More actions

    Thanks @LauraGPT . I have tested it with the following command:

    Command:
    .\llama-funasr-sensevoice.exe -m gguf\sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a "test.mp3" --backend vulkan

    The program reported:
    ggml_vulkan: Found 1 Vulkan devices:
    ggml_vulkan: 0 = AMD Radeon 780M Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
    initializing vulkan backend on Vulkan0 (AMD Radeon 780M Graphics)
    initialized vulkan backend on Vulkan0; resolving buffer type
    vulkan backend ready on Vulkan0
    [sensevoice] loading model metadata
    [sensevoice] model ready: 919 tensors
    [sensevoice] loading audio
    [sensevoice] audio ready: 1835840 samples
    [sensevoice] running VAD
    [sensevoice] VAD ready: 23 segments
    [sensevoice] building graph: 504 frames
    [sensevoice] graph built
    [sensevoice] allocating graph
    [sensevoice] graph allocated
    [sensevoice] compute starting

    Then I tried:
    echo $LASTEXITCODE

    Reported:
    -1073741819

    Setting $env:GGML_VK_MAX_NODES_PER_SUBMIT = "16" (or "1") and $env:GGML_VK_SERIALIZE_SUBMISSIONS = "1" does not change the behavior.
17. LauraGPT commented on Aug 29, 2026

    

    LauraGPT

    on Aug 29, 2026

    Collaborator

    More actions

    Thanks @terrytotalk. This v0.2.3 result narrows the failure substantially. The process completed Vulkan device selection and backend initialization, loaded the model and audio, completed VAD, built the 504-frame graph, and allocated it. Because the final flushed boundary is [sensevoice] compute starting, the access violation occurs in the first graph-compute call or below it in the Vulkan backend/AMD driver. It is no longer consistent with the former iGPU selector problem or a model-loading/allocation failure.

    The submission-size and serialization settings producing the same result also means the currently exposed submission knobs do not isolate this pre-return compute failure.

    runtime-llamacpp-v0.2.4 is now available, but it fixes an unrelated F16 GGUF query-embedding type bug. Your reproducer uses the Q8 model, so I am not asking you to repeat the same test on v0.2.4; it would not be evidence for this crash.

    The next useful artifact is a Windows crash dump or symbolized stack from this exact v0.2.3 executable. If you have Microsoft Sysinternals ProcDump, please launch the same command under it:

    New-Item -ItemType Directory -Force .\dumps | Out-Null
    procdump64.exe -accepteula -ma -e 1 -x .\dumps `
      .\llama-funasr-sensevoice.exe `
      -m gguf\sensevoice-small-q8.gguf --vad fsmn-vad.gguf `
      -a test.mp3 --backend vulkan

    Please provide the generated dump, or at minimum WinDbg !analyze -v and k output, together with the exact executable SHA-256. The module and stack frames will distinguish a ggml host-side fault from a fault inside the AMD Vulkan driver. A dump can contain local paths and process memory, so redact or share it privately if needed.

    I am keeping the issue open. The current reliable fallback remains the CPU package; neither v0.2.3 nor v0.2.4 claims that AMD Windows Vulkan 0xC0000005 is fixed.
18. terrytotalk commented on Aug 29, 2026

    

    terrytotalk

    on Aug 29, 2026

    More actions

    Thanks @LauraGPT . I tesed v0.2.3 again, and mailed the debug file to your mail box. Check it please.
19. LauraGPT commented on Aug 29, 2026

    

    LauraGPT

    on Aug 29, 2026

    Collaborator

    More actions

    Thanks for sending it. I do not yet see the attachment in the operational inbox available to this maintenance session, and the issue thread does not identify the recipient address. Please confirm which address you sent it to.

    To keep analysis moving without exposing the dump, you can also paste the sanitized text output from WinDbg for this exact v0.2.3 crash by running: .symfix, .reload, !analyze -v, .ecxr, k, and lm.

    The most useful fields are MODULE_NAME, IMAGE_NAME, FAILURE_BUCKET_ID, EXCEPTION_RECORD, the exception-context stack, and the loaded AMD Vulkan/ggml module versions. Please redact usernames and local paths. Do not attach the raw dump publicly if it may contain process memory; once the recipient is confirmed, I will correlate the private artifact with the exact executable SHA-256.
20. terrytotalk commented on Aug 29, 2026

    

    terrytotalk

    on Aug 29, 2026

    More actions

    @LauraGPT Please check your alibaba-inc.com mailbox. And ok, let me paste the debug data here.

    SHA256: c09c575f407eb96870556ac5023ceb69936e376e2ecacfd3c580f9cd09bc1739

    COMMENT:
    *** "C:\funasr-llamacpp-vulkan\procdump\procdump64.exe" -accepteula -ma -e 1 -x .\dumps .\llama-funasr-sensevoice.exe -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a test.mp3 --backend vulkan
    *** First chance exception: C0000005.ACCESS_VIOLATION

    NTGLOBALFLAG: 70

    APPLICATION_VERIFIER_FLAGS: 0

    CONTEXT: (.ecxr)
    rax=0000000000000000 rbx=0000000000000000 rcx=00000235601dc8e0
    rdx=0000023519cf0000 rsi=000000a3017aed30 rdi=0000000000000000
    rip=00007ff71ebffde3 rsp=000000a3017aeab0 rbp=0000000000000000
    r8=7ffffffffffffffc r9=0000000000000000 r10=0000000000000000
    r11=0000000000113140 r12=00000235601dc8e0 r13=0000023519f03ae0
    r14=00000000ffffffff r15=0000023572b409a0
    iopl=0 nv up ei pl zr na pe nc
    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
    llama_funasr_sensevoice+0x2dfde3:
    00007ff71ebffde3 488b4860 mov rcx,qword ptr [rax+60h] ds:0000000000000060=????????????????
    Resetting default scope

    EXCEPTION_RECORD: (.exr -1)
    ExceptionAddress: 00007ff71ebffde3 (llama_funasr_sensevoice+0x00000000002dfde3)
    ExceptionCode: c0000005 (Access violation)
    ExceptionFlags: 00000000
    NumberParameters: 2
    Parameter[0]: 0000000000000000
    Parameter[1]: 0000000000000060
    Attempt to read from address 0000000000000060

    PROCESS_NAME: llama-funasr-sensevoice.exe

    READ_ADDRESS: 0000000000000060

    ERROR_CODE: (NTSTATUS) 0xc0000005 - 0x%p 0x%p %s

    EXCEPTION_CODE_STR: c0000005

    EXCEPTION_PARAMETER1: 0000000000000000

    EXCEPTION_PARAMETER2: 0000000000000060

    STACK_TEXT:
    000000a3017aeab0 00007ff71ea253d5 : 0000023519f03b00 00000235601dc890 00000235637b3fe8 0000000000000001 : llama_funasr_sensevoice+0x2dfde3
    000000a3017aeb30 00007ff71ea61f94 : 0000000000000001 0000000000000000 000002357465aa40 000000a3017aef60 : llama_funasr_sensevoice+0x1053d5
    000000a3017aee60 00007ff71ea0fd15 : 0000000000000001 000002357465aa40 000000a300000001 0000000000000006 : llama_funasr_sensevoice+0x141f94
    000000a3017af080 00007ff71ec0f816 : 000000a3017af538 00000235637af490 00000235601fc1a0 00000235744110a0 : llama_funasr_sensevoice+0xefd15
    000000a3017af500 00007ff71e92a36a : 000002357465a8d0 0000023574107060 0000000000113a00 0000023574107060 : llama_funasr_sensevoice+0x2ef816
    000000a3017af530 00007ff71e96e873 : 000000a3017af990 000000a3017af890 000000a3017af7b8 0000023500000bb6 : llama_funasr_sensevoice+0xa36a
    000000a3017af780 00007ff71ec4c14f : 0000000000000000 00007ff71ec4c1c9 0000000000000000 0000000000000000 : llama_funasr_sensevoice+0x4e873
    000000a3017afac0 00007ffece21ccb7 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : llama_funasr_sensevoice+0x32c14f
    000000a3017afb00 00007ffecfb4caec : 0000000000000000 0000000000000000 000004f0fffffb30 000004d0fffffb30 : kernel32!BaseThreadInitThunk+0x17
    000000a3017afb30 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x2c

    STACK_COMMAND: ~0s; .ecxr ; kb

    IP_IN_PAGED_CODE:
    llama_funasr_sensevoice+2dfde3
    00007ff7`1ebffde3 488b4860 mov rcx,qword ptr [rax+60h]

    SYMBOL_NAME: llama_funasr_sensevoice+2dfde3

    MODULE_NAME: llama_funasr_sensevoice

    IMAGE_NAME: llama-funasr-sensevoice.exe

    FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_llama-funasr-sensevoice.exe!Unknown

    OS_VERSION: 10.0.26100.1

    BUILDLAB_STR: ge_release

    OSPLATFORM_TYPE: x64

    OSNAME: Windows 10

    FAILURE_ID_HASH: {ae515eb5-6b75-0015-fb30-e87b0d39281d}
21. LauraGPT commented on Aug 29, 2026

    

    LauraGPT

    on Aug 29, 2026

    Collaborator

    More actions

    Thanks @terrytotalk. The sanitized stack and exact executable SHA-256 were enough to move this from stage-level diagnosis to a source-level root cause.

    I downloaded the official runtime-llamacpp-v0.2.3 Windows Vulkan ZIP and verified that its SenseVoice executable SHA-256 is exactly c09c575f407eb96870556ac5023ceb69936e376e2ecacfd3c580f9cd09bc1739, matching your report.

    I then produced a PDB/MAP build from the exact v0.2.3 source and symbolized the reported stack:

    • llama_funasr_sensevoice+0x2dfde3 is ggml_vk_tensor_subbuffer
    • the faulting source line dereferences tensor->buffer->context
    • the caller is resolving the second source tensor for a Vulkan graph operation
    • tensor->buffer is null before command submission or driver execution

    A graph audit confirmed the source: SenseVoice loaded GGUF weights into an ordinary host context, while graph intermediates used the selected Vulkan buffer type. CPU accepts those raw host pointers; Vulkan requires registered host storage or a real backend buffer. In the reported path, 2,386 graph source references pointed at model tensors with no backend buffer.

    I opened #3555 with the bounded fix:

    • exact signed+DCO head: 949ccf46f32c6b432f355e31f6d69f8c3bf7c438
    • load GGUF metadata without allocating raw host weights
    • allocate all 919 model tensors in the selected backend's weight buffer
    • upload each tensor from its exact GGUF offset
    • read query embeddings through the backend API

    Local evidence on the pinned llama.cpp revision 803b7fca is green: 23 runtime tests, CPU Q8/F16 inference, and actual Vulkan Q8/F16 inference all completed with compute status=0 and the same transcript.

    Nine-platform exact-head validation is now running at https://github.com/modelscope/FunASR/actions/runs/33255112265. I will not ask you to retest or claim the AMD Windows crash fixed until that matrix is green and an official Windows Vulkan asset is published. This new evidence also corrects my earlier broad wording: the symbolized v0.2.3 failure is a FunASR/ggml host-side null-buffer path, not a demonstrated AMD driver fault.
22. LauraGPT commented on Aug 29, 2026

    

    LauraGPT

    on Aug 29, 2026

    Collaborator

    More actions

    runtime-llamacpp-v0.2.5 is now published from exact commit f371370d4c5e4c61d13d4eb9c55cda2f4dd95e4f with all nine platform jobs passing.

    For this AMD Windows report, please retest with:

    • asset: funasr-llamacpp-windows-x64-vulkan.zip
    • SHA256: d51e34afed66c0898dba3d944cb17d2abc92c5fb7758fa4ee376561d5ce35a4c

    The fix in v0.2.5 copies host weights into the Vulkan backend buffer before graph execution. We validated Q8 and F16 execution locally with Linux Vulkan llvmpipe, but that does not prove the AMD Windows hardware crash is resolved.

    Please paste the new initialization-boundary logs plus your GPU, driver version, model/quantization, command, and complete output. I am keeping this issue open until we have your hardware result.
23. rjffrwy16-ux commented on Aug 29, 2026

    

    rjffrwy16-ux

    on Aug 29, 2026

    More actions

    Following up on the v0.2.5 retest request above:

    runtime-llamacpp-v0.2.5 retest on Radeon RX 9070 XT

    I retested the official runtime-llamacpp-v0.2.5 Windows Vulkan build on the same Radeon RX 9070 XT system used for my previous v0.2.0 report.

    Environment

    • OS: Windows Pro 25H2, build 26200.9168
      (the legacy registry ProductName value reports Windows 10 Pro)
    • GPU: AMD Radeon RX 9070 XT
    • AMD Software: 26.8.1
    • Windows display driver: 32.0.31041.1004
    • Vulkan instance: 1.4.341
    • Runtime: runtime-llamacpp-v0.2.5
    • Release commit: f371370d4c5e4c61d13d4eb9c55cda2f4dd95e4f
    • Backend: vulkan
    • Model / quantization: sensevoice-small-q8.gguf / Q8
    • Model revision: 90c1c61912018b70ada0fcc024ea24aca62f2e63
    • ZIP SHA-256: d51e34afed66c0898dba3d944cb17d2abc92c5fb7758fa4ee376561d5ce35a4c
    • Executable SHA-256: 5387a97e458b251643da808293866dca5b4b72c6ab02c6d82d97e4636e681aeb
    • Model SHA-256: 4ae45c94422de949b387e2e0fb10d7e14e4c42c69db30c3444ecc7d4b844b7c5

    I used the same Q8 model lane and saved WAV fixture as in my previous test, with explicit --backend vulkan.

    The previous exact command did not use --vad, so I did not add it for this retest. No Vulkan diagnostic environment variables were set.

    v0.2.5 hardware retest result

    • Vulkan device enumeration: 2 devices
    • RX 9070 XT enumerated as device 0
    • Backend initialization: explicitly selected Vulkan0 (AMD Radeon RX 9070 XT)
    • Runtime reported: vulkan backend ready on Vulkan0
    • Model load: reached (model ready: 919 tensors)
    • Graph allocation: reached
    • Compute: compute complete: status=0
    • Transcript: 今日の予定を教えてください。
    • Exit code: 0x00000000
    • Repeated runs: 3/3 successful
    • CPU fallback: none
    • Residual process: none

    Compared with v0.2.0, the previous 0xC0000005 crash did not reproduce and execution crossed the previous blocking boundary through successful Vulkan compute and transcript output.

    One important caveat: the AMD stack has changed since my v0.2.0 evidence:

    • previous: AMD Software 26.7.1 / display driver 32.0.31035.1003
    • current: AMD Software 26.8.1 / display driver 32.0.31041.1004

    Therefore, this confirms the successful behavior of the current combined runtime/driver configuration, but does not isolate the v0.2.5 runtime fix as the sole cause.

    ---

    Additional local ASR benchmark

    Since the runtime blocker was gone, I was also able to run the same fixed 9-fixture local STT evaluation that I use for comparing my local ASR candidates.

    This is a local, hardware-specific benchmark and is not intended as a general model-quality claim.

    Runtime / performance

    • Radeon RX 9070 XT
    • Windows Pro 25H2
    • runtime-llamacpp-v0.2.5
    • Vulkan
    • SenseVoiceSmall Q8
    • provider-direct, no --vad
    • packaged CLI automatic language behavior
    • 57/57 processes completed successfully
    • measured transcript runs: 48
    • crashes: 0
    • non-zero exits: 0
    • compute failures: 0
    • CPU fallback: 0
    • residual process: 0

    Single-shot new-process measurements:

    • p50: 1.0354 s
    • p95: 1.0752 s
    • RTF p50: 0.2110
    • RTF p95: 0.3440

    I am not reporting an in-process warm latency because this packaged CLI accepts one audio per process and exposes no persistent multi-audio mode. I did not want to label repeated model reloads as a canonical warm measurement.

    Japanese fixture observations

    Stable outputs across the measured repetitions:

    • normal: 今日の予定を教えてください。 — content preserved
    • short: 予定话。 — mixed-script degradation
    • quiet: . — lexical content lost
    • low-SNR: 今日の予定を教えてください。 — content preserved
    • off-axis: 今日の 予定 を 教え てください。 — intent preserved
    • background: 今日の予定を教えてください。 — content preserved
    • clear target speech (ごめん): 救命。 — target meaning lost
    • ambiguous non-lexical input: 哦。 — non-empty false output
    • digital silence: I. in 5/5 runs — non-empty false transcription

    For my particular Japanese voice-assistant use case, the runtime is now stable and fast, but these quality results — especially the clear short utterance and silence behavior — mean I will not advance SenseVoice to my live evaluation stage at this time.

    That is now an application-level model-quality decision rather than a Vulkan runtime blocker.

    ---

    Why this fix mattered for my use case

    For some context, I encountered this issue while evaluating local STT candidates for a Japanese voice assistant running primarily on my Windows PC with a Radeon RX 9070 XT.

    The simplified local voice pipeline is:

    Microphone → VAD → Speaker Identification → STT → Authorization / Context → Local LLM → TTS → Playback

    The STT candidates I have been evaluating include:

    • Kotoba-Whisper v2.2 — current local baseline
    • Qwen3-ASR 0.6B — ROCm evaluation candidate
    • SenseVoiceSmall — Vulkan evaluation candidate

    Before this fix, the Vulkan 0xC0000005 crash prevented SenseVoice from even reaching model inference, so I could not evaluate it on the same latency and quality criteria as the other candidates.

    With runtime-llamacpp-v0.2.5, SenseVoice can now complete Vulkan inference reliably on this RX 9070 XT system.

    That allowed me to move past runtime compatibility testing and finally evaluate what I originally wanted to measure: actual latency, Japanese transcription quality, difficult audio conditions, and silence behavior.

    So even though I am currently holding SenseVoice for my specific application because of transcription quality, the important difference is that it is no longer excluded by the Vulkan runtime. I can now evaluate it properly based on its actual model behavior.

    Thank you for continuing to investigate the Windows AMD reports, adding the diagnostic boundaries, narrowing the failure down step by step, and ultimately fixing the host-side buffer path.

    The detailed retest instructions were also very helpful for providing reproducible hardware evidence from a normal user environment.

    I hope this RX 9070 XT result is useful for other Windows AMD users as well.
24. LauraGPT commented on Aug 29, 2026

    

    LauraGPT

    on Aug 29, 2026

    Collaborator

    More actions

    Thank you @rjffrwy16-ux for the unusually complete hardware retest. This is strong evidence that the official v0.2.5 asset crosses the previous host-buffer crash boundary on RX 9070 XT: 3/3 focused runs and 57/57 benchmark processes completed without crash, fallback, or compute failure.

    I am keeping this issue open. Your driver changed at the same time, so this result validates the combined v0.2.5 + AMD 26.8.1 configuration but does not isolate causality. We also still need a retest from the original report path, especially @Fy0808fhyj and @terrytotalk if available.

    The Japanese short/quiet/silence observations are also useful, but I will keep them separate from the Vulkan crash conclusion: they are model/decoding quality evidence for this fixture set, not a runtime regression or a general quality claim. In particular, non-empty output on digital silence deserves a focused reproducible issue if it persists with exact audio/model hashes.

    For the original reporters: please use the same v0.2.5 Windows Vulkan asset and include the final initialization boundary, driver version, exact command, and exit code. We will close only after the affected paths have enough confirmation.
25. LauraGPT

    added

    help wantedExtra attention is neededExtra attention is needed

    on Aug 31, 2026