You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
device_map="auto" silently loads the model on the CPU instead of Neuron devices #1024
On a trn2.48xlarge with torch_neuronx imported, torch.neuron.device_count() reports 64 devices. But AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") puts no module on a Neuron device:
model.hf_device_map is empty ({}).
The weights stay on the CPU. Generation runs on the CPU and produces correct text, so nothing visibly fails. No error or warning says that the Neuron devices were not used.
Impact:
device_map="auto" is the usual way to load a model that is too large for one accelerator. On Neuron it falls back to the CPU without saying so, which looks like a working run.
We noticed only from the speed (about 1.4 tokens/s for a 235 GB MoE) and the empty hf_device_map.
Reproduction
import torch, torch_neuronx
from transformers import AutoModelForCausalLM
print(torch.neuron.device_count()) # 64 on trn2.48xlarge (LNC=2)
model = AutoModelForCausalLM.from_pretrained("poolside/Laguna-S-2.1", dtype=torch.bfloat16, device_map="auto").eval()
print(getattr(model, "hf_device_map", None)) # EXPECTED modules on neuron:N, GOT {}
print(next(model.parameters()).device) # expected neuron:N
--Output observed
neuron devices visible: 64
"device_map": {} # collections.Counter of model.hf_device_map values
generation: correct text at ~1.4 tokens/s (64 new tokens in ~22 s per prompt)
Expected Behavior
Expected Behavior
Either outcome would be acceptable:
device_map="auto" sees the Neuron devices and their memory (as it does for cuda, xpu, npu and mlu) and spreads the modules over them; or
it raises an error or warns that it found no supported accelerator and is loading on the CPU. It should not fall back to the CPU silently.
Please also document which device_map values are supported for Neuron (e.g. "neuron", explicit {"module": "neuron:N"} maps, max_memory keys).
Note: the device detection may live in Hugging Face Accelerate (get_max_memory / infer_auto_device_map) rather than in torch-neuronx. If so, please tell us and we will file it there as well.
Environment
Instance: trn2.48xlarge (16 devices, 64 logical cores at LNC=2), region ap-south-2, 21 Oct 2026.
AMI: Deep Learning AMI Neuron PyTorch Inference vLLM 0.29.0.1.2.0 (Ubuntu 26.04), SDK 2.33.0 (ami-0ea44cc75110b4afe).
torch-neuronx 2.14.3.0.282+dc70ffa, torch 2.14.0, neuronx-cc 2.28.7631.0+8f7b0cf5, Python 3.14.4. Standalone venv.
transformers: 5.20.2, accelerate: 1.15.0
aws-neuronx-runtime-lib 2.35.30.0, aws-neuronx-collectives 2.35.43.0, aws-neuronx-dkms 2.31.7.0.
We reproduced this issue on Neuron. Accelerate can't read device memory from torch-neuronx, so it finds no devices and device_map="auto" puts the model on CPU without a warning.
We're working on a fix in torch-neuronx so Accelerate can see Neuron device memory. Splitting one model across several Neuron devices also needs the fix for #1021. We'll update this issue when both land.
Activity
tdalmia-aws commentedon Oct 28, 2026
Thanks for the report, @GauravM1781.
We reproduced this issue on Neuron. Accelerate can't read device memory from torch-neuronx, so it finds no devices and
device_map="auto"puts the model on CPU without a warning.We're working on a fix in torch-neuronx so Accelerate can see Neuron device memory. Splitting one model across several Neuron devices also needs the fix for #1021. We'll update this issue when both land.