Overview

vLLM-Ascend (vllm-ascend) is an Out-Of-Tree (OOT) hardware plugin that enables vLLM to run seamlessly on Huawei Ascend NPU hardware README.md51-53 It adheres to the hardware-pluggable architecture principles defined in upstream vLLM README.md55 providing a decoupled, modular integration layer. By leveraging torch-npu and the CANN (Compute Architecture for Neural Networks) software stack, vLLM-Ascend allows popular open-source architectures—including Transformer-like models, Mixture-of-Experts (MoE), embedding models, and multi-modal LLMs—to achieve high-throughput and low-latency inference on Ascend NPUs without modifying core model definitions README.md57-58

Releases are version-aligned with upstream vLLM releases to ensure API compatibility, feature parity, and continuous regression testing via automated CI pipelines docs/source/community/versioning_policy.md3-10 docs/source/faqs.md109-112

Sources: README.md51-58 docs/source/community/versioning_policy.md3-10 docs/source/faqs.md109-112

1.1 Features and Capabilities

vLLM-Ascend delivers a comprehensive suite of optimization features tailored for Ascend NPUs. This includes advanced attention backends such as Standard attention, Multi-level Attention (AscendMLABackend), Sparse Flash Attention (AscendSFABackend), and Distributed Sparse Attention (AscendDSABackend) docs/source/user_guide/release_notes.md9-12 It supports speculative decoding mechanisms (e.g., AscendEagleProposer, MTP), hardware-accelerated quantization schemes (W8A8, W4A8, W4A4, and MXFP8) docs/source/faqs.md117-119 execution graph compilation modes (ACLGraph, NPUGraph-EX via torchair) docs/source/developer_guide/Design_Documents/npugraph_ex.md and distributed scaling strategies including Tensor Parallelism (TP), Pipeline Parallelism (PP), Expert Parallelism (EP), and Context Parallelism (PCP/DCP).

For comprehensive technical details, implementation specifics, and configuration guides, see Features and Capabilities.

Sources: docs/source/user_guide/release_notes.md9-12 docs/source/faqs.md117-119 docs/source/developer_guide/Design_Documents/npugraph_ex.md

1.2 Supported Hardware and Models

The plugin supports a broad range of Ascend NPU hardware series, including the Atlas A2 training/inference series, Atlas A3 training/inference series, experimental Atlas 300I Duo (310P) series, and Atlas A5 (Ascend 950) high-density inference series README.md62 docs/source/faqs.md16-24 Compatibility is tightly bound to specific software versions, requiring coordinated releases of PyTorch, torch-npu, CANN toolkit, NNAL libraries, and Triton Ascend docs/source/community/versioning_policy.md24-60

For detailed hardware compatibility matrices, model support lists, and software version pairing tables, see Supported Hardware and Models.

Sources: README.md62 docs/source/faqs.md16-24 docs/source/community/versioning_policy.md24-60

1.3 System Architecture Overview

vLLM-Ascend integrates deeply into the vLLM lifecycle through a platform registration mechanism, custom workers, model runners, and a two-phase patching system vllm_ascend/platform.py78-79 vllm_ascend/utils.py52-199

Architecture and Code Entity Mapping

Title: Architecture and Code Entity Mapping

Sources: vllm_ascend/platform.py78-79 vllm_ascend/ascend_config.py27-60 vllm_ascend/utils.py52-199

Component Execution Flow

Title: Component Execution Flow

Sources: vllm_ascend/platform.py237-302 vllm_ascend/utils.py52-199

For detailed architectural breakdowns covering NPUPlatform, NPUWorker, NPUModelRunner, forward context flow, and the patch system, see System Architecture Overview.

Sources: vllm_ascend/platform.py78-302 vllm_ascend/ascend_config.py27-60 vllm_ascend/utils.py52-199