Skip to content

[Feature] minicpm eagle support - #18943

Merged
vllm-bot merged 5 commits into
vllm-project:mainfrom
huangyuxiang03:main
May 30, 2025
Merged

[Feature] minicpm eagle support#18943
vllm-bot merged 5 commits into
vllm-project:mainfrom
huangyuxiang03:main

Conversation

@huangyuxiang03

@huangyuxiang03 huangyuxiang03 commented May 30, 2025 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

I'm adding the support of running eagle speculative decoding on MiniCPM series. A new model file minicpm_eagle.py is added, as the eagle head of MiniCPM has a distinct architecture compared with vanilla eagle head. We also adapt the register.py to make sure the EagleMiniCPM can be used. We slightly modify the minicpm.py (moving the scaling factor from LM head to the last hidden state), which does not affect the usage of existing minicpm models.

I'm submitting this PR on behalf of ModelBest.Inc, the producer of MiniCPM series. We have conducted abundant tests to make sure the modification does not affect other parts of vLLM.

huangyuxiang03
minicpm eagle support
7f5b761 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
@huangyuxiang03
huangyuxiang03 marked this pull request as ready for review May 30, 2025 07:15
@github-actions

github-actions Bot commented May 30, 2025

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

huangyuxiang03
minicpm_eagle style fix
74da1cc 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
@DarkLight1337

DarkLight1337 commented May 30, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

It looks like there are some duplicate modules, can you import them directly from minicpm.py? Only keep in minicpm_eagle.py the parts that are different from regular MiniCPM model.

huangyuxiang03
minicpm eagle reuse existing classes
2bc9a53 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
@huangyuxiang03

huangyuxiang03 commented May 30, 2025

Copy link
Copy Markdown
Contributor Author

It looks like there are some duplicate modules, can you import them directly from minicpm.py? Only keep in minicpm_eagle.py the parts that are different from regular MiniCPM model.

Fixed. Now we only keep the diffferent classes in minicpm_eagle.py

DarkLight1337
DarkLight1337 reviewed May 30, 2025
View reviewed changes
Comment thread vllm/model_executor/models/minicpm_eagle.py
hidden_states=hidden_states,
)
hidden_states = residual + hidden_states * \
(self.config.scale_depth / math.sqrt(self.config.mup_denominator))

@DarkLight1337 DarkLight1337 May 30, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QQ: Where is mup_denominator defined? I can't seem to find it on HF Hub

@huangyuxiang03 huangyuxiang03 May 30, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are planning to support eagle speculative decoding in our next release. This hyperparameter will be added to the config in the next version of our model. Currently, since HF model config also supports plugging in new fields by **kwargs, so simply putting this into config.json would be working.

@DarkLight1337 DarkLight1337 May 30, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for the explanation! Should be good to go then.

@DarkLight1337

DarkLight1337 commented May 30, 2025

Copy link
Copy Markdown
Member

Can you add this model to _SPECULATIVE_DECODING_EXAMPLE_MODELS in the test registry?

@DarkLight1337

DarkLight1337 commented May 30, 2025

Copy link
Copy Markdown
Member

Otherwise I think the CI may fail

huangyuxiang03
fix registry test
574fa0c 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
@huangyuxiang03
huangyuxiang03 requested a review from ywang96 as a code owner May 30, 2025 09:33
@huangyuxiang03

huangyuxiang03 commented May 30, 2025

Copy link
Copy Markdown
Contributor Author

Otherwise I think the CI may fail

Sure, added.

DarkLight1337
DarkLight1337 reviewed May 30, 2025
View reviewed changes
Comment thread tests/models/registry.py
huangyuxiang03
fix minicpm is_available_online
6e98087 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
@DarkLight1337
DarkLight1337 enabled auto-merge (squash) May 30, 2025 10:34
@DarkLight1337

DarkLight1337 commented May 30, 2025

Copy link
Copy Markdown
Member

Thanks for your patience!

@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label May 30, 2025
@huangyuxiang03

huangyuxiang03 commented May 30, 2025

Copy link
Copy Markdown
Contributor Author

Thank you!

@vllm-bot
vllm-bot merged commit e1fadf1 into vllm-project:main May 30, 2025
amitm02 pushed a commit to amitm02/vllm that referenced this pull request Jun 1, 2025
@huangyuxiang03 huangyuxiang03 @amitm02
[Feature] minicpm eagle support (vllm-project#18943)
f3941c8 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
Co-authored-by: huangyuxiang03 <huangyx0321@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
amitm02 pushed a commit to amitm02/vllm that referenced this pull request Jun 1, 2025
@huangyuxiang03 huangyuxiang03 @amitm02
[Feature] minicpm eagle support (vllm-project#18943)
9b7eef2 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
Co-authored-by: huangyuxiang03 <huangyx0321@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
0826joyce pushed a commit to 0826joyce/vllm-serving-optimization that referenced this pull request May 19, 2026
@huangyuxiang03 huangyuxiang03
[Feature] minicpm eagle support (vllm-project#18943)
bf782f8 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
Co-authored-by: huangyuxiang03 <huangyx0321@gmail.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
@huangyuxiang03 huangyuxiang03
[Feature] minicpm eagle support (vllm-project#18943)
7556829 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
Co-authored-by: huangyuxiang03 <huangyx0321@gmail.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
@huangyuxiang03 huangyuxiang03
[Feature] minicpm eagle support (vllm-project#18943)
eaed9bb 
Signed-off-by: huangyuxiang03 <huangyx0321@gmail.com>
Co-authored-by: huangyuxiang03 <huangyx0321@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes
@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@huangyuxiang03 @DarkLight1337 @vllm-bot