Skip to content

Feature Request: Support Kimi K2 #14642

Closed
Closed
Feature Request: Support Kimi K2#14642
Labels
enhancementNew feature or request

Description

@yeahdongcn
yeahdongcn
opened on Jul 12, 2025

Prerequisites

  • I am running the latest code. Mention the version if possible as well.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Kimi K2 is a new open-source model available.

I tried to load this converted gguf but failed due to two issues:

  1. Number of experts exceeds 256
llama_model_loader: - kv  24:                     deepseek2.expert_count u32              = 384
  1. Missing tokenizer.ggml.model
llama_model_load: error loading model: error loading model vocabulary: key not found in model: tokenizer.ggml.model
llama_model_load_from_file_impl: failed to load model

Motivation

From the benchmark data, Kimi K2 turns to be the most powerful oss model now.

Possible Implementation

No response

Activity

  1. yeahdongcn
    added
    enhancementNew feature or request
    on Jul 12, 2025

  2. jukofyork commented on Jul 12, 2025

    @jukofyork
    jukofyork
    on Jul 12, 2025
    Collaborator
    1. Number of experts exceeds 256

    llama_model_loader: - kv 24: deepseek2.expert_count u32 = 384

    This should be an easy fix:

    llama.cpp/src/llama-hparams.h

    Line 9 in f5e96b3

    #define LLAMA_MAX_EXPERTS 256 // DeepSeekV3 

    2. Missing `tokenizer.ggml.model`

    llama_model_load: error loading model: error loading model vocabulary: key not found in model: tokenizer.ggml.model
    llama_model_load_from_file_impl: failed to load model

    https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/tokenization_kimi.py

    As far as I can see, everything else seems to be exactly the same as deepseek-v3 architecture (it has only 1 dense layer at the start instead of 3 though).

    I'm downloading it now - very slowly lol.

  3. jukofyork commented on Jul 12, 2025

    @jukofyork
    jukofyork
    on Jul 12, 2025
    Collaborator

    It might also be worth trying the hack suggested here: #12610

  4. yeahdongcn commented on Jul 12, 2025

    @yeahdongcn
    yeahdongcn
    on Jul 12, 2025
    CollaboratorAuthor

    Yeah, I got the first one working. However, I’m using a cloud instance with limited storage, so I’m unable to perform the conversion on it.

    I was able to extend the storage. Here is the new issue in converting from HF:

    root@rocm-jupyter-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# python3 convert_hf_to_gguf.py /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e
INFO:hf-to-gguf:Loading model: 2f7e011530aa89b56ab4277c26fc4aed3c52822e
WARNING:hf-to-gguf:Failed to load model config from /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e: The repository /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e contains custom code which must be executed to correctly load the model. You can inspect the repository content at /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e .
 You can inspect the repository content at https://hf.co//scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: DeepseekV3ForCausalLM
WARNING:hf-to-gguf:Failed to load model config from /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e: The repository /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e contains custom code which must be executed to correctly load the model. You can inspect the repository content at /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e .
 You can inspect the repository content at https://hf.co//scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-1-of-61.safetensors'
INFO:hf-to-gguf:token_embd.weight,            torch.bfloat16 --> F16, shape = {7168, 163840}
INFO:hf-to-gguf:blk.0.attn_norm.weight,       torch.bfloat16 --> F32, shape = {7168}
INFO:hf-to-gguf:blk.0.ffn_down.weight,        torch.float8_e4m3fn --> F16, shape = {18432, 7168}
Traceback (most recent call last):
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 7411, in <module>
    main()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 7405, in main
    model_instance.write()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 410, in write
    self.prepare_tensors()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 5679, in prepare_tensors
    super().prepare_tensors()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 277, in prepare_tensors
    for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 5676, in modify_tensors
    return [(self.map_tensor_name(name), data_torch)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 236, in map_tensor_name
    raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'
root@rocm-jupyter-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp#

  5. danielhanchen commented on Jul 12, 2025

    @danielhanchen
    danielhanchen
    on Jul 12, 2025
    Contributor

    The tiktoken tokenizer does look like an issue - similar to Hunyuan's changes - I think it's mostly similar to GPT4o's tokenizer maybe?

  6. CISC commented on Jul 12, 2025

    @CISC
    CISC
    on Jul 12, 2025
    Member

    I was able to extend the storage. Here is the new issue in converting from HF:
    ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'

    They have uploaded FP8 tensors, you have to convert them to BF16 first, using DeepSeek's convert.pyfp8_cast_bf16.py.

  7. jukofyork commented on Jul 12, 2025

    @jukofyork
    jukofyork
    on Jul 12, 2025
    Collaborator

    I was able to extend the storage. Here is the new issue in converting from HF:
    ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'

    They have uploaded FP8 tensors, you have to convert them to BF16 first, using DeepSeek's convert.py.

    This won't work on Ampere or lower cards sadly. I hacked out the Triton stuff and created this version:

    #11446 (comment)

    that should work just using CPU if needed.

  8. jukofyork commented on Jul 12, 2025

    @jukofyork
    jukofyork
    on Jul 12, 2025
    Collaborator

    Actually, I'm not sure what convert.py is for. It's this code that is used to convert form fp8 to bf16:

    https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py

    that sadly relies on this Triton kerel:

    https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py

    which won't work unless your GPU has native fp8 (ie: Ada or newer).

  9. CISC commented on Jul 12, 2025

    @CISC
    CISC
    on Jul 12, 2025
    Member

    Actually, I'm not sure what convert.py is for. It's this code that is used to convert form fp8 to bf16:

    https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py

    Ah, sorry, I just remembered it was in that repo and picked the first reasonably looking file. :)

  10. danielhanchen commented on Jul 12, 2025

    @danielhanchen
    danielhanchen
    on Jul 12, 2025
    Contributor

    By the way if it helps, I made BF16 via the DeepSeek script conversion - https://huggingface.co/unsloth/Kimi-K2-Instruct-BF16

    The tokenizer needs a new regex it seems, but I'm assuming it's similar to Hunyuan's conversion process, but more similar to GPT4o's tokenizer. Also they seem to tokenize Chinese characters more differently

  11. artraze commented on Jul 12, 2025

    @artraze
    artraze
    on Jul 12, 2025

    After to converting to bf16 (and trusting remote) the tokenizer seems to be a problem.

    convert_hf_to_gguf.py fails on vocab_size = self.hparams.get("vocab_size", len(tokenizer.vocab)) due to vocab not existing on tokenizer. If you bypass that you get

    WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:**          There are 2 possible reasons for this:
WARNING:hf-to-gguf:**          - the model has not been added to convert_hf_to_gguf_update.py yet
WARNING:hf-to-gguf:**          - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.
WARNING:hf-to-gguf:** ref:     https://github.com/ggml-org/llama.cpp/pull/6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh:  81212dc7cdb7e0c1074ca62c5aeab0d43c9f52b8a737be7b12a777c953027890

    Hacking kimi-k2 into ./convert_hf_to_gguf_update.py and running yields:

    Traceback (most recent call last):
  File "/home/aicli/llama/llama.cpp/./convert_hf_to_gguf_update.py", line 268, in <module>
    tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}", trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aicli/llama/llama.cpp/venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 949, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.kimi-k2.configuration_deepseek.DeepseekV3Config'> to build an AutoTokenizer.
Model type should be one of AlbertConfig, AlignConfig, ...

    I would assume that this is not finding the configuration_deepseek.py file or that isn't properly registering its contents but I don't know any of these libraries so I tapped out at that point.

  12. nicoboss commented on Jul 12, 2025

    @nicoboss
    nicoboss
    on Jul 12, 2025
    Contributor

    Here my patches to get convert_hf_to_gguf .py to work for Kimi-K2-Instruct. I have absolutely no idea if this is correct but it converts. Please correct me if I did anything wrong.

    I downloaded the original model and used the following command using the script from original DeepSeek V3 to conveart it to BF16: CUDA_VISIBLE_DEVICES=1 venv/bin/python fp8_cast_bf16.py --input-fp8-hf-path /cpool/Kimi-K2-Instruct --output-bf16-hf-path /bpool/Kimi-K2-Instruct_BF16

    diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index 8afb425b..80046c2a 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -608,12 +608,12 @@ class TextModel(ModelBase):

         from transformers import AutoTokenizer
         tokenizer = AutoTokenizer.from_pretrained(self.dir_model)
-        vocab_size = self.hparams.get("vocab_size", len(tokenizer.vocab))
-        assert max(tokenizer.vocab.values()) < vocab_size
+        vocab_size = self.hparams.get("vocab_size", tokenizer.vocab_size)
+        #assert max(tokenizer.vocab.values()) < vocab_size

         tokpre = self.get_vocab_base_pre(tokenizer)

-        reverse_vocab = {id_: encoded_tok for encoded_tok, id_ in tokenizer.vocab.items()}
+        reverse_vocab = {id_: encoded_tok for encoded_tok, id_ in tokenizer.get_vocab().items()}
         added_vocab = tokenizer.get_added_vocab()

         added_tokens_decoder = tokenizer.added_tokens_decoder
@@ -780,6 +780,9 @@ class TextModel(ModelBase):
         if chkhsh == "877081d19cf6996e2c4ff0e1236341e9b7bde288f5311a56a937f0afbbb3aeb5":
             # ref: https://huggingface.co/deepseek-ai/DeepSeek-V3
             res = "deepseek-v3"
+        if chkhsh == "81212dc7cdb7e0c1074ca62c5aeab0d43c9f52b8a737be7b12a777c953027890":
+            # ref: https://huggingface.co/moonshotai/Kimi-K2-Instruct
+            res = "deepseek-v3"
         if chkhsh == "b3f499bb4255f8ca19fccd664443283318f2fd2414d5e0b040fbdd0cc195d6c5":
             # ref: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
             res = "deepseek-r1-qwen"

  13. nicoboss commented on Jul 13, 2025

    @nicoboss
    nicoboss
    on Jul 13, 2025
    Contributor

    Simply converting the model as described above and increasing LLAMA_MAX_EXPERTS is not enough to get the model to load as it fails with lama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file.

    root@AI:/apool/llama.cpp/build/bin# ./llama-cli -m /cpool/Kimi-K2-Instruct.gguf -c 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
build: 5962 (46c3d25b) with cc (Debian 12.2.0-14+deb12u1) 12.2.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) - 23686 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) - 2572 MiB free
llama_model_loader: loaded meta data with 43 key-value pairs and 1096 tensors from /cpool/Kimi-K2-Instruct.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Kimi K2 Instruct_BF16
llama_model_loader: - kv   3:                         general.size_label str              = 384x14B
llama_model_loader: - kv   4:                            general.license str              = other
llama_model_loader: - kv   5:                       general.license.name str              = modified-mit
llama_model_loader: - kv   6:                      deepseek2.block_count u32              = 61
llama_model_loader: - kv   7:                   deepseek2.context_length u32              = 131072
llama_model_loader: - kv   8:                 deepseek2.embedding_length u32              = 7168
llama_model_loader: - kv   9:              deepseek2.feed_forward_length u32              = 18432

llama_model_loader: - kv  22:         deepseek2.attention.key_length_mla u32              = 192
llama_model_loader: - kv  23:       deepseek2.attention.value_length_mla u32              = 128
llama_model_loader: - kv  24:       deepseek2.expert_feed_forward_length u32              = 2048
llama_model_loader: - kv  25:                     deepseek2.expert_count u32              = 384
llama_model_loader: - kv  26:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  27:             deepseek2.expert_weights_scale f32              = 2.827000
llama_model_loader: - kv  28:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  29:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  30:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  31:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  32:              deepseek2.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  33: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  34: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  37:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  38:                      tokenizer.ggml.tokens arr[str,163840]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  39:                  tokenizer.ggml.token_type arr[i32,163840]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  40:                tokenizer.ggml.bos_token_id u32              = 163584
llama_model_loader: - kv  41:                tokenizer.ggml.eos_token_id u32              = 163585
llama_model_loader: - kv  42:                    tokenizer.chat_template str              = {% if tools -%}\n    {{ '<|im_system|>...
llama_model_loader: - type  f32:  365 tensors
llama_model_loader: - type  f16:  731 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = F16
print_info: file size   = 1912.14 GiB (16.00 BPW)
llama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file

llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model '/cpool/Kimi-K2-Instruct.gguf'
main: error: unable to load model

  14. nyuuzyou commented on Jul 13, 2025

    @nyuuzyou
    nyuuzyou
    on Jul 13, 2025

    This won't work on Ampere or lower cards sadly. I hacked out the Triton stuff and created this version:

    #11446 (comment)

    that should work just using CPU if needed.

    I used triton-cpu for the conversion without GPU and it worked well with minimal modifications to the original script.

  15. yeahdongcn commented on Jul 13, 2025

    @yeahdongcn
    yeahdongcn
    on Jul 13, 2025
    CollaboratorAuthor

    It looks like I was able to convert unsloth/Kimi-K2-Instruct-BF16 to F16 GGUF. I'm now quantizing the F16 model to Q4_K_M. (I also tried loading the F16 model directly, but it's way too large.)

    root@rocm-jupyter-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-quantize /scratch/hub/models--unsloth--Kimi-K2-Instruct-BF16/snapshots/98b2879bb5ebba8e9bac21cce3e0286e1082205f/98b2879bb5ebba8e9bac21cce3e0286e1082205f-384x14B-98b2879bb5ebba8e9bac21cce3e0286e1082205f-F16-00001-of-00011.gguf /scratch/hub/models--unsloth--Kimi-K2-Instruct-BF16/snapshots/98b2879bb5ebba8e9bac21cce3e0286e1082205f/abc-Q4_K_M.gguf Q4_K_M
main: build = 5884 (c31e6064)
main: built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
main: quantizing '/scratch/hub/models--unsloth--Kimi-K2-Instruct-BF16/snapshots/98b2879bb5ebba8e9bac21cce3e0286e1082205f/98b2879bb5ebba8e9bac21cce3e0286e1082205f-384x14B-98b2879bb5ebba8e9bac21cce3e0286e1082205f-F16-00001-of-00011.gguf' to '/scratch/hub/models--unsloth--Kimi-K2-Instruct-BF16/snapshots/98b2879bb5ebba8e9bac21cce3e0286e1082205f/abc-Q4_K_M.gguf' as Q4_K_M
llama_model_loader: additional 10 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 54 key-value pairs and 1096 tensors from /scratch/hub/models--unsloth--Kimi-K2-Instruct-BF16/snapshots/98b2879bb5ebba8e9bac21cce3e0286e1082205f/98b2879bb5ebba8e9bac21cce3e0286e1082205f-384x14B-98b2879bb5ebba8e9bac21cce3e0286e1082205f-F16-00001-of-00011.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = 98b2879bb5ebba8e9bac21cce3e0286e1082205f
llama_model_loader: - kv   3:                           general.finetune str              = 98b2879bb5ebba8e9bac21cce3e0286e1082205f
llama_model_loader: - kv   4:                         general.size_label str              = 384x14B
llama_model_loader: - kv   5:                            general.license str              = other
llama_model_loader: - kv   6:                       general.license.name str              = modified-mit
llama_model_loader: - kv   7:                   general.base_model.count u32              = 1
llama_model_loader: - kv   8:                  general.base_model.0.name str              = Kimi K2 Instruct
llama_model_loader: - kv   9:          general.base_model.0.organization str              = Moonshotai
llama_model_loader: - kv  10:              general.base_model.0.repo_url str              = https://huggingface.co/moonshotai/Kim...
llama_model_loader: - kv  11:                               general.tags arr[str,3]       = ["unsloth", "unsloth", "unsloth"]
llama_model_loader: - kv  12:                      deepseek2.block_count u32              = 61
llama_model_loader: - kv  13:                   deepseek2.context_length u32              = 131072
llama_model_loader: - kv  14:                 deepseek2.embedding_length u32              = 7168
llama_model_loader: - kv  15:              deepseek2.feed_forward_length u32              = 18432
llama_model_loader: - kv  16:             deepseek2.attention.head_count u32              = 64
llama_model_loader: - kv  17:          deepseek2.attention.head_count_kv u32              = 1
llama_model_loader: - kv  18:                   deepseek2.rope.freq_base f32              = 50000.000000
llama_model_loader: - kv  19: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  20:                deepseek2.expert_used_count u32              = 8
llama_model_loader: - kv  21:             deepseek2.attention.key_length u32              = 576
llama_model_loader: - kv  22:           deepseek2.attention.value_length u32              = 512
llama_model_loader: - kv  23:                          general.file_type u32              = 1
llama_model_loader: - kv  24:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  25:                       deepseek2.vocab_size u32              = 163840
llama_model_loader: - kv  26:            deepseek2.attention.q_lora_rank u32              = 1536
llama_model_loader: - kv  27:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  28:         deepseek2.attention.key_length_mla u32              = 192
llama_model_loader: - kv  29:       deepseek2.attention.value_length_mla u32              = 128
llama_model_loader: - kv  30:       deepseek2.expert_feed_forward_length u32              = 2048
llama_model_loader: - kv  31:                     deepseek2.expert_count u32              = 384
llama_model_loader: - kv  32:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  33:             deepseek2.expert_weights_scale f32              = 2.827000
llama_model_loader: - kv  34:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  35:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  36:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  37:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  38:              deepseek2.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  39: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  40: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
llama_model_loader: - kv  41:               general.quantization_version u32              = 2
llama_model_loader: - kv  42:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  43:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  44:                      tokenizer.ggml.tokens arr[str,163840]  = ["[PAD0]", "[PAD1]", "[PAD2]", "[PAD3...
llama_model_loader: - kv  45:                  tokenizer.ggml.token_type arr[i32,163840]  = [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, ...
llama_model_loader: - kv  46:                      tokenizer.ggml.merges arr[str,1]       = ["<|endoftext|> <|endoftext|>"]
llama_model_loader: - kv  47:                tokenizer.ggml.bos_token_id u32              = 163584
llama_model_loader: - kv  48:                tokenizer.ggml.eos_token_id u32              = 163585
llama_model_loader: - kv  49:            tokenizer.ggml.padding_token_id u32              = 163839
llama_model_loader: - kv  50:                    tokenizer.chat_template str              = {% if tools -%}\n    {{ '<|im_system|>...
llama_model_loader: - kv  51:                                   split.no u16              = 0
llama_model_loader: - kv  52:                                split.count u16              = 11
llama_model_loader: - kv  53:                        split.tensors.count i32              = 1096
llama_model_loader: - type  f32:  365 tensors
llama_model_loader: - type  f16:  731 tensors
    llama_tensor_get_type : tensor cols 128 x 512 are not divisible by 256, required for q4_K - using fallback quantization q5_0
converting to q5_0 .. size =     8.00 MiB ->     2.75 MiB
[   5/1096]           blk.0.attn_kv_a_mqa.weight - [ 7168,   576,     1,     1], type =    f16, converting to q4_K .. size =     7.88 MiB ->     2.21 MiB
[   6/1096]          blk.0.attn_kv_a_norm.weight - [  512,     1,     1,     1], type =    f32, size =    0.002 MB
[   7/1096]               blk.0.attn_norm.weight - [ 7168,     1,     1,     1], type =    f32, size =    0.027 MB
[   8/1096]             blk.0.attn_output.weight - [ 8192,  7168,     1,     1], type =    f16, converting to q4_K .. size =   112.00 MiB ->    31.50 MiB
[   9/1096]                blk.0.attn_q_a.weight - [ 7168,  1536,     1,     1], type =    f16, converting to q4_K .. size =    21.00 MiB ->     5.91 MiB
[  10/1096]           blk.0.attn_q_a_norm.weight - [ 1536,     1,     1,     1], type =    f32, size =    0.006 MB
[  11/1096]                blk.0.attn_q_b.weight - [ 1536, 12288,     1,     1], type =    f16, converting to q4_K .. size =    36.00 MiB ->    10.12 MiB
[  12/1096]                blk.0.attn_v_b.weight - [  512,   128,    64,     1], type =    f16, converting to q4_K .. size =     8.00 MiB ->     2.25 MiB
[  13/1096]                blk.0.ffn_down.weight - [18432,  7168,     1,     1], type =    f16, converting to q6_K .. size =   252.00 MiB ->   103.36 MiB
[  14/1096]                blk.0.ffn_gate.weight - [ 7168, 18432,     1,     1], type =    f16, converting to q4_K .. size =   252.00 MiB ->    70.88 MiB
[  15/1096]                blk.0.ffn_norm.weight - [ 7168,     1,     1,     1], type =    f32, size =    0.027 MB
[  16/1096]                  blk.0.ffn_up.weight - [ 7168, 18432,     1,     1], type =    f16, converting to q4_K .. size =   252.00 MiB ->    70.88 MiB
[  17/1096]                blk.1.attn_k_b.weight - [  128,   512,    64,     1], type =    f16,

  16. 19 remaining items

  17. yeahdongcn commented on Jul 14, 2025

    @yeahdongcn
    yeahdongcn
    on Jul 14, 2025
    CollaboratorAuthor

    I think the jinja error has something to do with bf16 repo i converted from including a .jinja file, I'm guessing converting to gguf without it leads to no error

    I think you will need to implement a new chat template, such as LLM_CHAT_TEMPLATE_KIMI_K2. I just added one locally, and it resolved the crash — but it's not fully working yet.

  18. danielhanchen commented on Jul 14, 2025

    @danielhanchen
    danielhanchen
    on Jul 14, 2025
    Contributor

    @gabriellarson Nice change - testing now! Note I use --verbose --verbose-prompt --jinja to test tokenization - but yes llama-tokenize also works!

  19. artraze commented on Jul 14, 2025

    @artraze
    artraze
    on Jul 14, 2025

    I think the jinja error has something to do with bf16 repo i converted from including a .jinja file, I'm guessing converting to gguf without it leads to no error

    I converted my own without a .jinja and need --jinja too but I had also been hacking on it for a bit so IDK if I broke something that caused a similar symptom :).

    Alright I think i fixed it with the latest commit, @artraze can you test again and see if you still see the double spaces?

    Definitely healthier! I only evaluated with English but I'm not seeing the huge changes in size on retokenizing output and the doubling of spaces has stopped. Overall it seems competent enough for a Q2 without imatrix. (I might try to run an imatrix tonight, though it'll probably take a while...)

  20. gabriellarson commented on Jul 14, 2025

    @gabriellarson
    gabriellarson
    on Jul 14, 2025
    Contributor

    @fernandaspets the Q2_K gguf should work, just make sure to use --jinja

  21. danielhanchen commented on Jul 14, 2025

    @danielhanchen
    danielhanchen
    on Jul 14, 2025
    Contributor

    I made some 245GB, 281GB (IQ1_S) dynamic quants + Q2_K_XL, Q4_K_XL quants at https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF - it should work fine with @gabriellarson 's PR #14654 or using my fork https://github.com/unslothai/llama.cpp - guide to run them here: https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally#run-kimi-k2-tutorials

  22. createthis commented on Jul 15, 2025

    @createthis
    createthis
    on Jul 15, 2025
    Contributor

    #14654 With a dual EPYC 9355 with 736gb ram in 23 channels (one of my rdimms is on RMA) and a blackwell 6000 pro with Kimi-K2-Instruct:1t:Q4_K_XL with NPS4:

    ./build/bin/llama-batched-bench \
    --model /data/Kimi-K2-Instruct-GGUF/UD-Q4_K_XL/Kimi-K2-Instruct-UD-Q4_K_XL-00001-of-00013.gguf \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 62 \
    -ot "blk\.(3|4|5|6|7|8)\.ffn_.*=CUDA0" \
    -ot exps=CPU \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.6 \
    --min-p 0.01 \
    --flash-attn \
    -npp 4096 \
    -ntg 1024 \
    -npl 1

main: n_kv_max = 131072, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  4096 |   1024 |    1 |   5120 |   38.594 |   106.13 |   46.432 |    22.05 |   85.026 |    60.22 |

    A few days later with the RMA'd rdimm installed again:

    main: n_kv_max = 131072, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 62, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  4096 |   1024 |    1 |   5120 |   35.969 |   113.88 |   47.150 |    21.72 |   83.119 |    61.60 |

    Normal run:

    ./build/bin/llama-server \
    --model /data/Kimi-K2-Instruct-GGUF/UD-Q4_K_XL/Kimi-K2-Instruct-UD-Q4_K_XL-00001-of-00013.gguf \
    --alias Kimi-K2-Instruct:1t:Q4_K_XL \
    --no-webui \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 62 \
    -ot "blk\.(3|4|5|6|7|8)\.ffn_.*=CUDA0" \
    -ot exps=CPU \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.6 \
    --min-p 0.01 \
    --log-colors \
    --flash-attn \
    --host 0.0.0.0 \
    --jinja \
    --port 11434

    89.6gb nvram
    491.9g system ram

    Image Image

    Some time passes...

    Image

    Image

  23. danielhanchen commented on Jul 15, 2025

    @danielhanchen
    danielhanchen
    on Jul 15, 2025
    Contributor

    @createthis Nice 24 tokens / s is pretty sick for Q4_K_XL!!

  24. yeahdongcn commented on Jul 15, 2025

    @yeahdongcn
    yeahdongcn
    on Jul 15, 2025
    CollaboratorAuthor

    Tested on an AMD Developer Cloud instance with 8 × MI300X GPUs, using #14654 and the Q4_K_M variant of https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF:

    root@rocm-jupyter-gpu-mi300x8-1536gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-batched-bench \
    --model ./unsloth/Kimi-K2-Instruct-GGUF/unsloth_Kimi-K2-Instruct-GGUF_Q4_K_M_Kimi-K2-Instruct-Q4_K_M-00001-of-00013.gguf \
    --numa numactl \
    --threads 32 \
    --ctx-size 131072 \
    --n-gpu-layers 999 \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.6 \
    --min-p 0.01 \
    --flash-attn \
    -npp 4096 \
    -ntg 1024 \
    -npl 1

main: n_kv_max = 131072, n_batch = 4096, n_ubatch = 4096, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 999, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  4096 |   1024 |    1 |   5120 |  109.385 |    37.45 |  183.934 |     5.57 |  293.319 |    17.46 |

llama_perf_context_print:        load time =  102419.85 ms
llama_perf_context_print: prompt eval time =  110295.55 ms /  4112 tokens (   26.82 ms per token,    37.28 tokens per second)
llama_perf_context_print:        eval time =  183907.84 ms /  1024 runs   (  179.60 ms per token,     5.57 tokens per second)
llama_perf_context_print:       total time =  395738.95 ms /  5136 tokens

    root@rocm-jupyter-gpu-mi300x8-1536gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m ./unsloth/Kimi-K2-Instruct-GGUF/unsloth_Kimi-K2-Instruct-GGUF_Q4_K_M_Kimi-K2-Instruct-Q4_K_M-00001-of-00013.gguf -ngl 999
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 8 ROCm devices:
  Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 1: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 2: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 3: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 4: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 5: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 6: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
  Device 7: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| deepseek2 671B Q4_K - Medium   | 578.14 GiB |  1026.41 B | ROCm       | 999 |           pp512 |        118.32 ± 0.51 |
| deepseek2 671B Q4_K - Medium   | 578.14 GiB |  1026.41 B | ROCm       | 999 |           tg128 |         38.09 ± 0.09 |

build: 6ef6aa1c (5898)

  25. nyuuzyou commented on Jul 15, 2025

    @nyuuzyou
    nyuuzyou
    on Jul 15, 2025

    Could this be the cause of tokenization issues??
    https://x.com/Kimi_Moonshot/status/1945050874067476962

    We've just fixed 2 bugs in Kimi-K2-Instruct huggingface repo. Please update the following files to apply the fix:

    ❤️ Big thanks to "bchenfireworks" and "cassanof" for reporting this!

    Related issues:
    https://huggingface.co/moonshotai/Kimi-K2-Instruct/discussions/14
    https://huggingface.co/moonshotai/Kim

  26. danielhanchen commented on Jul 15, 2025

    @danielhanchen
    danielhanchen
    on Jul 15, 2025
    Contributor

    @nyuuzyou Re their changes:

    1. tokenizer.encode will now encode special tokens - this does NOT affect the PR at Model : Add support for Kimi-K2 #14654 see Feature Request: Support Kimi K2 #14642 (comment) which shows the tokenization of special tokens is done correctly. Only HF's tokenizer.encode will be incorrect.
    2. The multi turn tool calling chat template does need an update. One has to enable the new one via --chat-template-file PATH_TO_KIMI_K2_CHAT_TEMPLATE.jinja - I will have to re-update the quants to bake the new chat template over the next few days

  27. createthis commented on Jul 19, 2025

    @createthis
    createthis
    on Jul 19, 2025
    Contributor

    @danielhanchen did you just update all the files in the UD_Q4_K_XL model yesterday? What was that about? I tried to grab the first file again to grab the new chat template and now it's giving me llama_model_load: error loading model: invalid model: tensor 'blk.5.ffn_gate_exps.weight' is duplicated.

  28. jukofyork commented on Jul 19, 2025

    @jukofyork
    jukofyork
    on Jul 19, 2025
    Collaborator

    We also find the correct EOS token should not be [EOS], but rather <|im_end|>, which we have also fixed in our model conversions.

    https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally

    @danielhanchen sorry to ping you, but can I ask where you got this info from?

    They still appear to use [EOS] in all their files:

    https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/tokenization_kimi.py

    https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/tokenizer_config.json

    https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/config.json

    apart from in generation_config.json:

    {
  "max_length": 131072,
  "eos_token_id": 163586
}

    I did notice this when I was making the draft models and thought it looked like <|im_end|> makes more sense (with [EOS] being the pre-training document separator), but can't find any confirmation on this and it looks like they even went out of their way to make [EOS] tokenise properly in their only change since the repo was created!?

  29. yeahdongcn commented on Jul 22, 2025

    @yeahdongcn
    yeahdongcn
    on Jul 22, 2025
    CollaboratorAuthor

    Since #14654 has been merged into master, I’m closing this issue as completed.

  30. yeahdongcn
    closed this as completedon Jul 22, 2025

  31. danielhanchen commented on Jul 22, 2025

    @danielhanchen
    danielhanchen
    on Jul 22, 2025
    Contributor

    @jukofyork Sorry on the delay - so the issue is [EOS] is not in the chat template, whilst <|im_end|> is, so the EOS for the instruct model should be <|im_end|>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions