ministral3-3b-reasoning-typed-decisions (Specialist)

Specialist model: fine-tuned for the Typed Decisions benchmark only; it is not a general-purpose chat or reasoning model.

Ministral 3 3B Reasoning fully fine-tuned (text decoder only; vision tower dropped, input embedding / tied LM head frozen) as a probabilistic decision model for LocalLLaMA/typed-decisions. Mode: fine-tuned on train (all four workflows, 1080 cases for training and 120 held out for epoch selection on validation KL).

Results

Scored on LocalLLaMA/typed-decisions all/test (400 cases, 2000 decisions), never seen in training.

ECE is computed per question and averaged over the 20 questions (closest to the README's reference rows); pooled over all decisions it is 0.1742.

By question type

By workflow

Baselines on the same split: Prior accuracy 0.478 / KL 0.327; zero-shot Ministral 3 3B Reasoning accuracy 0.340 / KL 1.659.

Full test distributions: test_predictions.jsonl; metrics: test_metrics.json. Gold is the mean of three teacher samples, so scores measure agreement with that teacher (self-agreement ≈ 0.735).

How it works

Each (case, question) pair is one prompt: a short system turn, then the case JSON, the question's instructions, and its options listed as letters A, B, C, … with their descriptions. The assistant turn is pre-filled with an empty [THINK][/THINK] block, so no reasoning is generated: the next-token logits at that position, restricted to those letter tokens and softmaxed, are the answer distribution. Training minimises soft cross-entropy against the gold distribution. One forward pass per question (5 per case).

Usage

import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "rony/ministral3-3b-reasoning-typed-decisions"
tok = AutoTokenizer.from_pretrained(repo, fix_mistral_regex=True)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).eval()
LETTERS = "ABCDEFGHIJ"
SYSTEM = 'You are a precise decision model. Read the case, then answer the question with the single letter of the best option.'
uv run jupyter lab ministral3_3b_reasoning_decisions.ipynb
def options(q):
    if q["type"] == "choice": return list(q["criteria"].items())
    if q["type"] == "noul":
        c = q.get("criteria") or {}
        return [("true", c.get("true", "The statement is true.")), ("false", c.get("false", "The statement is false."))]
    return [(str(i), d) for i, d in enumerate(q["criteria"])]

@torch.no_grad()
def decide(state, questions):
    out = {}
    for name, q in questions.items():
        opts = options(q)
        kind = {"choice": "Choose one option.", "noul": "Is the statement true or false?",
                "score": "Pick the level on the rubric."}[q["type"]]
        prompt = (f"### Case\n{json.dumps(state, separators=(',', ':'))}\n\n"
                  f"### Question ({name})\n{q['instructions']}\n{kind}\n\n### Options\n"
                  + "\n".join(f"{LETTERS[i]}. {l}: {d}" for i, (l, d) in enumerate(opts))
                  + "\n\nAnswer with one letter.")
        text = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}],
                                       tokenize=False, add_generation_prompt=True) + "[THINK][/THINK]"
        logits = model(**tok(text, add_special_tokens=False, return_tensors="pt")).logits[0, -1]
        ids = [tok.encode(LETTERS[i], add_special_tokens=False)[0] for i in range(len(opts))]
        p = torch.softmax(logits[ids].float(), -1).tolist()
        out[name] = dict(zip([l for l, _ in opts], p))
    return out