saihai-v1-8b

日本語での紹介記事はこちら: 日本語 Decision モデル "saihai" (采配) と、日本語 Decision モデルベンチマークの公開

saihai (采配) is a Japanese Typed Decision Model. Give it a state (text or JSON) and named questions; it answers each question in one forward pass, without generating text:

saihai-v1-8b is fine-tuned from llm-jp/llm-jp-4.1-8b-thinking, an open LLM with strong Japanese ability. It is trained on Japanese and English decision data and is released under Apache-2.0, the same license as the base model.

日本語: saihai (采配) は、はい/いいえ (Noul)・選択 (Choice)・段階評価 (Score) の判断を、 文章を生成せずに確率として返す日本語の Decision モデルです。saihai-v1-8b は llm-jp-4.1-8b-thinking をベースにした 8B モデルです。

saihai family

Quick start

Use Python 3.12+ with PyTorch and Transformers. The BF16 weights require approximately 17.2 GB, plus inference memory. The bundled inference_saihai.py runs standalone; installing the saihai package is unnecessary.

uv pip install torch 'transformers==5.17.0' 'tokenizers>=0.23,<0.24' 'huggingface-hub>=1.0,<2' 'accelerate>=1.0,<2'

The example asks whether a customer wants a refund (Noul), which department should handle the request (Choice), and how urgent it is (Score).

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "hotchpotch/saihai-v1-8b",
    trust_remote_code=True,
    device_map="auto",
    dtype="auto",
)
response = model.systemone({
    'state': '昨日購入した商品を返品したいです。返金をお願いします。',
    'questions': {
        'refund': {'type': 'noul', 'instructions': '顧客は返金を求めているか？'},
        'department': {
            'type': 'choice', 'instructions': '問い合わせを担当部署に振り分ける。',
            'criteria': {
                'refunds': '返品・返金の対応',
                'delivery': '配送状況・配達の対応',
                'technical': 'ログイン・システム障害の対応',
            },
        },
        'urgency': {
            'type': 'score', 'instructions': '問い合わせの緊急度を評価する。',
            'criteria': [
                '通常対応：一般的な問い合わせや依頼',
                '優先対応：利用できず困っている',
                '至急対応：不正利用や被害が進行している',
            ],
        },
    },
})
answers = response["answers"]

print(f"返金要求あり：{answers['refund']['noul']:.2%}")

department = answers["department"]
department_names = {
    "refunds": "返品・返金",
    "delivery": "配送",
    "technical": "技術サポート",
}
selected = department["choice"]
print(
    f"担当部署：{department_names[selected]}"
    f"（{department['probabilities'][selected]:.2%}）"
)

urgency = answers["urgency"]
probabilities = urgency["probabilities"]
print(
    f"緊急度：通常対応 {probabilities['0']:.2%}、"
    f"優先対応 {probabilities['1']:.2%}、"
    f"至急対応 {probabilities['2']:.2%}"
    f"（スコア：{urgency['score']:.3f}）"
)

Example output, verified with CPU / BF16 (small numerical differences may occur on other devices):

返金要求あり：97.84%
担当部署：返品・返金（99.61%）
緊急度：通常対応 82.53%、優先対応 15.27%、至急対応 2.20%（スコア：0.197）

The urgency score is the probability-weighted average of normal 0, priority 1, and immediate 2. Probabilities are model outputs, not guaranteed correctness rates.

Pin revision="<full commit hash>" for reproducibility. A single question can use model.predict(state, question). For llama.cpp, use saihai-v1-8b-gguf.

Interface and limits

systemone() accepts state and named questions, and returns answers, model and usage. Questions are evaluated independently.

• Noul: noul is the probability of true; candidates are false then true.
• Choice: choice is the selected criterion key; probabilities contains every candidate's probability. Supports 1–255 candidates.
• Score: score is the expected zero-based rubric index. Supports 2–10 levels. Explicit numeric levels also return expected_value and numeric_values.

Choice and Score also return confidence; probabilities are not separately calibrated. Input JSON numbers use six significant digits; represent precise quantities as strings.

The limit is 65,536 input tokens per question, including instructions, state, candidate definitions and model delimiters. Overflow is rejected without truncation. Training used an 8,192-token limit; quality on longer inputs, including 64K inputs, has not been established.

Evaluation

Results on S1MB (System One Mosaic Benchmark). Scores are baseline-adjusted 0–100 values, not raw accuracy. Task Avg is the equal-weight average of the Noul, Choice and Score means; "general" columns average only the synthetic generalization tasks.

Compare with other decision models on the S1MB leaderboard. The Japanese evaluation covers 8,067 judgments with zero inference failures.

Training includes train-split examples from ten source datasets evaluated by S1MB japanese-v1 (for example JNLI, JSTS and JCommonsenseQA), and English source families from the Bekko System One dataset. Evaluation examples were excluded, but scores on those benchmarks are likely higher than on unseen tasks. These results do not establish generalization to unseen source families.

Training

Trained for one epoch (2,096 optimizer steps) on all 33,532 train judgments from hotchpotch/saihai-v1-dataset at revision 906504ac4b72e133a613011318afd8206fb5e8ec: 17,509 Japanese and 16,023 English cases. Training used BF16 LoRA (rank 16, alpha 32), an FP32 decision head, effective batch size 16, an 8,192-token input limit and seed 42. The final checkpoint was merged into BF16; validation did not select an earlier checkpoint. Final validation soft-target cross entropy was 0.462753 across 279 fixed validation cases.

The decision head reads candidate-code logits in the same format as llama.cpp's /v1/systemone endpoint (pplx-decider compatible), so the same weights also run in llama.cpp after GGUF conversion.

See training-provenance.json for base/data revisions and training conditions, and release-manifest.json for artifact hashes. The training, evaluation and GGUF conversion code is at hotchpotch/saihai.

License

The model weights are licensed under Apache-2.0, the same as the base model. The training data has source-specific licenses; see the dataset card.