Repository navigation
[torch Issue]: system freeze with torch.compile on gfx1151Â #1937
Description
Problem Description
Running torch.compile freezes the entire system without much output. Minimum code to reproduce:
import torch
import torch.nn as nn
class SimpleModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(1024, 512)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(512, 256)
self.fc3 = nn.Linear(256, 10)
def forward(self, x):
x = self.relu(self.fc1(x))
x = self.relu(self.fc2(x))
return self.fc3(x)
model = SimpleModel().to("cuda")
compiled_model = torch.compile(model)
x = torch.randn(32, 1024, device="cuda")
output = compiled_model(x)
Fresh Ubuntu 24.04.3 on Strix Halo (Bosgame M5, 128G). Installed the rock with
python -m pip install \
--index-url https://rocm.nightlies.amd.com/v2/gfx1151/ \
"rocm[libraries,devel]"
python -m pip install \
--index-url https://rocm.nightlies.amd.com/v2/gfx1151/ \
--pre torch torchaudio torchvision
Ran into the same issue when using the official torch installation this.
It also seems like the issue might be related to garbage collecting & cleaning, because I do not have trouble running anything with the compiled model, but as soon as the script (that has a torch.compile in it) quits, the whole system then freezes.
Operating System
Ubuntu 24.04.3
CPU
AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
GPU
Radeon 8060S / gfx1151
ROCm Version
7.1.0
ROCm Component
No response
Steps to Reproduce
No response
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
No response
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
Epliz commentedon Oct 29, 2025
I can confirm, same issue (gfx1151 too) with even a simpler code:
taylding-amd commentedon Oct 29, 2025
Hi @alanzjl, is this issue the same as the one you submitted earlier? Also, can you verify whether the issue has been resolved in PyTorch 2.10?
taylding-amd commentedon Oct 29, 2025
Hi @Epliz, I tried your code on my machine and was not able to reproduce the issue, could you please try again with PyTorch 2.10?
Epliz commentedon Oct 29, 2025
Hi @taylding-amd ,
To check, I installed torch 2.10 and rocm 7.10 from the v2 nightlies with:
The reduced version of the program I shared above did not trigger a hang with torch 2.10.
However the original program from the original post still did.
If I install from v2-staging
Then the reduced version exits correctly, and the first version sometimes hangs, sometimes segfaults.
Here is the end of the logs I could capture with AMD_LOG_LEVEL=7. I guess they stop when it actually hangs, but the fact that you can see "exit" at the end indicates that it reached the print statement I put at the end of the python script, and that it was exiting.
I am not sure what logs would be useful. I guess it is a hang when trying to synchronize the CPU and GPU.
logs_end.txt
(The full logs at 40MB and could not download)
Best,
Epliz
taylding-amd commentedon Oct 30, 2025
Thanks, @Epliz, for checking. I wasnât able to reproduce the hang on my machine. Which OS are you using?
Epliz commentedon Oct 31, 2025
Hi @taylding-amd ,
I am on Ubuntu 24.04.3 LTS, kernel 6.14.0-34-generic, ROCM 7.0.2 "host" installed from the installation packages, ROCM 7.10 through pip.
Epliz commentedon Oct 31, 2025
So, I saw there are other issues being reported (e.g. https://github.com/ROCm/legacy-rocm-build/issues/5590 ), so I updated my firmware with linux-firmware.git .
While that did not solve fully the hang of the application, it made the hang to not be a total system hang and I could get some interesting dmesg output that seems to give an idea of where the cpu gets blocked:
Full dmesg output:
dmesg.txt
Epliz commentedon Oct 31, 2025
I will try to install a newer kernel over the weekend to see if it solves the issue
taylding-amd commentedon Oct 31, 2025
You can also try the PyTorch docker images: https://hub.docker.com/r/rocm/pytorch
Epliz commentedon Nov 1, 2025
Updating the kernel to 6.17 + using amdgpu-dkms from ROCM 7.1 + using the latest firmware from linux-firmware.git solves the hang for me at least.
taylding-amd commentedon Nov 5, 2025
Hi, this issue will be closed now as it's solved in the newest release. Please feel free to reopen it or ask follow-up questions should further assistance be required. Thanks!