[David-Wen2025]

• Changed layer.moe to layer in AWQ MoE method
• Changed layer.moe to layer in GPTQ MoE method
• FusedMoE object doesn't have .moe attribute, should use layer directly

This fixes AttributeError: 'FusedMoE' object has no attribute 'moe' when using "auto_round" quantization with FusedMoE layers."

Purpose

To fix the issue #24803
Model：Qwen3-Next-80B-A3B-Instruct-int4-mixed AutoRound

Test Plan

Hardware: NVIDIA RTX PRO 6000 WorkStation 96G
vLLM version: v0.10.2（torch2.8+cu129）
Start Code:
export VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve "/home/model/Qwen3-Next-80B-A3B-Instruct-AutoRound" --host 0.0.0.0 --port 3001 --gpu-memory-utilization 0.8 --served-model-name "qwen3" --quantization "auto-round" --dtype "bfloat16" --max-model-len 131072

Test Result

Before:
info:
AttributeError: 'FusedMoE' object has no attribute 'moe' /root/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/auto_round.py", line 330

After:
info: Application startup complete

Model Test

Qwen3-Next-80B-A3B-Instruct-AutoRound

GSM8K（limit 250）

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.884	±	0.0203
		strict-match	5	exact_match	↑	0.844	±	0.0230

MMLU（limit 250）

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.8687	±	0.0032
- humanities	2	none		acc	↑	0.8656	±	0.0066
- other	2	none		acc	↑	0.8530	±	0.0068
- social sciences	2	none		acc	↑	0.9022	±	0.0061
- stem	2	none		acc	↑	0.8584	±	0.0063

Qwen3-Coder-30B-A3B-Instruct-int4-AutoRound

GSM8K（limit 250）

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.804	±	0.0252
		strict-match	5	exact_match	↑	0.904	±	0.0187

MMLU（limit 250）

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.7941	±	0.0039
- humanities	2	none		acc	↑	0.7697	±	0.0082
- other	2	none		acc	↑	0.7726	±	0.0081
- social sciences	2	none		acc	↑	0.8539	±	0.0073
- stem	2	none		acc	↑	0.7868	±	0.0074

GLM-4.5-Air-AWQ-FP16Mix

GSM8K（limit 250）

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.944	±	0.0146
		strict-match	5	exact_match	↑	0.944	±	0.0146

Qwen3-32B-AWQ

GSM8K（limit 250）

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.916	±	0.0176
		strict-match	5	exact_match	↑	0.904	±	0.0187

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request correctly identifies and fixes an AttributeError in auto_round.py for FusedMoE layers by removing the incorrect .moe attribute access. However, the fix introduces a new issue by passing the FusedMoE layer object directly to AWQMoEMethod and GPTQMarlinMoEMethod, which expect a FusedMoEConfig object according to their definitions and usage elsewhere in the codebase. My review includes suggestions to pass layer.moe_config instead, which ensures type correctness, improves code clarity, and maintains consistency with other parts of the vLLM framework.

[David-Wen2025]

The engine argument --kv-cache-dtype fp8 --calculate-kv-scales currently fails to work due to a conflict with Torch dynamo.

Real Start command:
export VLLM_ATTENTION_BACKEND=FLASHINFER
vllm serve "/home/model/Qwen3-Next-80B-A3B-Instruct-AutoRound"
--host 0.0.0.0
--port 3001
--gpu-memory-utilization 0.8
--served-model-name "qwen3"
--quantization "auto-round"
--dtype "bfloat16"
--max-model-len 131072

[yewentao256]

Thanks for the fix! Could you also test on some of the other models to make sure this won't affect them?
Also, please also show the accuracy result using lm_eval

[David-Wen2025]

Thanks for the fix! Could you also test on some of the other models to make sure this won't affect them? Also, please also show the accuracy result using lm_eval

Okay, I will refine the information.

[drrros]

Can confirm this changes work fine for Qwen3-Coder-30B-A3B-Instruct-int4-AutoRound

[David-Wen2025]

Can confirm this changes work fine for Qwen3-Coder-30B-A3B-Instruct-int4-AutoRound

Yes, I have confirmed that the model is runnable. I will submit the model testing data later.

[David-Wen2025]

Can confirm this changes work fine for Qwen3-Coder-30B-A3B-Instruct-int4-AutoRound

You can take a look at my updated model test results.

[David-Wen2025]

Thanks for the fix! Could you also test on some of the other models to make sure this won't affect them? Also, please also show the accuracy result using lm_eval

I have added the test results of four models, including MMLU and GSM8K.

[yewentao256]

LGTM, thanks for the work!

[chensuyue]

Can we merge this PR?

[mgoin]

Doesn't this need to pass in layer.moe_config?

[David-Wen2025]

Doesn't this need to pass in layer.moe_config?

I haven't tested the modification suggestions provided for Gemini Bot before, but after the recent testing, I confirmed that the modifications made to Gemini Bot are also usable.