Skip to content

device_map="auto" silently loads the model on the CPU instead of Neuron devices #1024

Open

Description

@GauravM1781

🐛 Describe the bug

On a trn2.48xlarge with torch_neuronx imported, torch.neuron.device_count() reports 64 devices. But AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") puts no module on a Neuron device:

model.hf_device_map is empty ({}).
The weights stay on the CPU. Generation runs on the CPU and produces correct text, so nothing visibly fails. No error or warning says that the Neuron devices were not used.
Impact:
device_map="auto" is the usual way to load a model that is too large for one accelerator. On Neuron it falls back to the CPU without saying so, which looks like a working run.
We noticed only from the speed (about 1.4 tokens/s for a 235 GB MoE) and the empty hf_device_map.

Reproduction

import torch, torch_neuronx
from transformers import AutoModelForCausalLM

print(torch.neuron.device_count()) # 64 on trn2.48xlarge (LNC=2)
model = AutoModelForCausalLM.from_pretrained("poolside/Laguna-S-2.1", dtype=torch.bfloat16, device_map="auto").eval()
print(getattr(model, "hf_device_map", None)) # EXPECTED modules on neuron:N, GOT {}
print(next(model.parameters()).device) # expected neuron:N

--Output observed
neuron devices visible: 64
"device_map": {} # collections.Counter of model.hf_device_map values
generation: correct text at ~1.4 tokens/s (64 new tokens in ~22 s per prompt)

Expected Behavior

Expected Behavior
Either outcome would be acceptable:

  1. device_map="auto" sees the Neuron devices and their memory (as it does for cuda, xpu, npu and mlu) and spreads the modules over them; or
  2. it raises an error or warns that it found no supported accelerator and is loading on the CPU. It should not fall back to the CPU silently.

Please also document which device_map values are supported for Neuron (e.g. "neuron", explicit {"module": "neuron:N"} maps, max_memory keys).
Note: the device detection may live in Hugging Face Accelerate (get_max_memory / infer_auto_device_map) rather than in torch-neuronx. If so, please tell us and we will file it there as well.

Environment

Instance: trn2.48xlarge (16 devices, 64 logical cores at LNC=2), region ap-south-2, 21 Oct 2026.
AMI: Deep Learning AMI Neuron PyTorch Inference vLLM 0.29.0.1.2.0 (Ubuntu 26.04), SDK 2.33.0 (ami-0ea44cc75110b4afe).
torch-neuronx 2.14.3.0.282+dc70ffa, torch 2.14.0, neuronx-cc 2.28.7631.0+8f7b0cf5, Python 3.14.4. Standalone venv.
transformers: 5.20.2, accelerate: 1.15.0
aws-neuronx-runtime-lib 2.35.30.0, aws-neuronx-collectives 2.35.43.0, aws-neuronx-dkms 2.31.7.0.

Activity

  1. tdalmia-aws commented on Oct 28, 2026

    @tdalmia-aws
    Contributor

    Thanks for the report, @GauravM1781.

    We reproduced this issue on Neuron. Accelerate can't read device memory from torch-neuronx, so it finds no devices and device_map="auto" puts the model on CPU without a warning.

    We're working on a fix in torch-neuronx so Accelerate can see Neuron device memory. Splitting one model across several Neuron devices also needs the fix for #1021. We'll update this issue when both land.

  2. self-assigned this
    on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions