Crivo-68M

A 68M-parameter typed-decision encoder, distilled from Crivo-400M.

Paper: Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters (arXiv:2610.30970)

Crivo-68M reads a state and a set of typed questions (choice, noul, score) and returns a probability for every option in a single encoder pass. It does not generate text. It is built for the classify, route, gate and verify decisions that agents and pipelines make thousands of times, and sits in the middle of the Crivo family, between Crivo-32M and Crivo-150M. It was trained on the data of Crivo-400M, with Crivo-400M as its teacher.

Crivo follows the typed-decision interface popularized by TypeSafe's Jev (a state plus typed questions in, a probability for every option out) and runs unmodified in the open Decision Index kit. It is an independent open model, not affiliated with or endorsed by TypeSafe, and not trained on Jev outputs. Crivo is Portuguese for "sieve"; passar pelo crivo, to pass through the sieve, means to be judged and sorted.

Resumo em português

Crivo-68M é um encoder de decisões tipadas com 68M de parâmetros (Ettin/ModernBERT), destilado do Crivo-400M e treinado com os mesmos dados dele. Ele lê um state e perguntas choice, noul ou score e devolve uma probabilidade para cada opção numa única passada, sem gerar texto. No índice público do Decision Index 0.3 marca 32,82 (run completo publicado, sem truncamento), entre o Crivo-32M (25,06) e o Crivo-150M (36,39), com mediana de 16,4 ms por requisição numa A100. No Decision Index 0.4, que conta só os testes privados, o Full score é 17,69 (mesmas skills 25,27; domínios novos 5,03). Foi treinado só com dados abertos, com verificação de contaminação contra a suíte. Funciona em inglês. Licença CC BY-NC 4.0 (uso não comercial).

Results

Decision Index 0.4 — Full score (board)

Scored by the maintainers on the board in the Decision Index 0.4 batch of 2026-10-20. In edition 0.4 the Full score is 0.60 × private tests of the same skills + 0.40 × private tasks from new domains, both mapped onto the public scale using untuned stock models. The public index is still run and shown, but it no longer counts toward the Full score, and the new-domain tasks were revised. Edition 0.3.2 (0.20 public + 0.50 same skills + 0.30 new domains) stays available as an archive. Models within 0.9 points of the first model of a group share a rank group.

Rank group 42 is shared with vLLM-SR Decision 2.0 Eos 0.8B (Full 17.73), Crivo-150M v2 (Full 17.70), vLLM-SR d3-edge (0.6B) (Full 17.12), Valen-0.8B (Full 17.07) and DecisionTune 1.0 (Full 16.88).

Compared with models of similar size (board, Decision Index 0.4):

Crivo-68M is on the board's Pareto frontier of size × Full score: it is the highest-scoring model of its size, and the next model above it, Crivo-150M v2, is 2.2× larger and only 0.01 points higher.

Decision Index 0.3 — public index (self-run, complete)

Run with the official kit (apolinario/decision-index, commit 9eb2dbe, edition 0.3) on the complete public suite, with the engine shipped in this repository: every one of the 140,178 scoreable requests answered, none unsupported, nothing truncated. Scores are chance-corrected skill (0 = random guessing, 100 = perfect), weighted as the board does. Full results: tardellirs/decision-index-results → runs/crivo-68m.

In edition 0.4 the board shows this public index next to the model, but it no longer counts toward the Full score.

Per-benchmark skill (0–100, chance-corrected) — click to expand

★ = gold benchmark (weight 1.2 within its area). The index uses the native metric listed, chance-corrected.

Benchmark	Native metric	Crivo-68M	Crivo-32M	Crivo-150M
Knowledge & Reasoning
GPQA Diamond ★	accuracy	0.0	0.0	0.0
GSM8K	accuracy	0.5	0.3	2.0
ChessBench	accuracy	0.0	0.0	0.0
MuSR	accuracy	20.1	0.0	26.7
SATA-Bench	case exact accuracy	0.0	0.0	0.0
CRUXEval	accuracy	12.3	8.4	7.3
CLadder	accuracy	8.1	8.4	61.0
HLE ★	accuracy	0.0	0.0	0.0
MMLU-Pro ★	accuracy	3.5	1.1	3.4
BBH fixed-option tasks ★	accuracy	6.1	1.4	11.6
WinoGrande ★	accuracy	5.9	3.4	21.9
Language Understanding
ContractNLI	macro-F1	70.8	59.0	74.7
ANLI ★	macro-F1	4.1	8.4	19.0
HellaSwag ★	accuracy	19.0	11.0	48.9
ACOS	per-review F1	19.6	25.4	38.9
FinEntity	macro-F1	78.4	52.2	76.7
iSarcasmEval	sarcasm F1, track A English	0.0	0.0	0.0
VAST	macro-F1	54.3	51.0	61.4
NLI4CT	macro-F1	12.9	0.0	15.1
RAGTruth	F1 on hallucinated class	26.1	30.0	44.3
Retrieval & Classification
BANKING77 ★	macro-F1	85.8	86.7	90.6
CLINC150+OOS ★	macro-F1	87.6	86.2	91.8
BRIGHT ★	nDCG@10	20.9	17.5	18.6
Amazon ESCI	macro-F1	25.7	21.3	29.8
PhishNChips	accuracy (verdict)	0.4	0.0	4.0
HoVer	accuracy	62.9	58.5	69.4
Tools & Automation
BFCL ★	case exact accuracy	76.4	54.2	79.0
ToolRet	nDCG@10	53.8	49.4	55.7
API-Bank ★	accuracy	59.7	7.9	29.4
Home appliance simulator	case exact accuracy	75.0	27.3	79.5
When2Call	accuracy	53.7	47.2	62.5
Arts & Human Taste
BPoMP	accuracy	72.7	67.3	0.0
Humicroedit	accuracy	0.0	0.0	4.5
POP909-CL	accuracy	0.6	0.0	1.2
cfcolor	accuracy	0.8	0.0	0.6
Habermas Machine	accuracy	12.4	11.6	10.1
New Yorker caption matching	accuracy	38.9	32.3	44.1

• Strong: tool selection and API calls (BFCL 76.4, API-Bank 59.7, ToolRet 53.8, smart-home control 75.0), intent classification (BANKING77 85.8, CLINC150 87.6), entity sentiment (FinEntity 78.4), long-document NLI (ContractNLI 70.8, HoVer 62.9) and original-vs-perturbed text (BPoMP 72.7). Its Tools & Automation area (64.0) is above Crivo-150M's (60.7).
• Weak: expert knowledge and multi-step reasoning (GPQA, HLE, MMLU-Pro, GSM8K, chess, code execution), causal inference (CLadder 8.1, against 61.0 for Crivo-150M), adversarial NLI and commonsense completion (ANLI 4.1, HellaSwag 19.0), sarcasm and subjective "taste" benchmarks.

Quick start

Standalone (no kit)

from huggingface_hub import snapshot_download
import sys; path = snapshot_download("tardellirs/crivo-68m"); sys.path.insert(0, path)
from crivo import Crivo

model = Crivo(path)                                 # CUDA if available, else CPU
out = model.answer(
    state={"ticket": "My card was charged twice for the same order."},
    questions={
        "intent": {"type": "choice", "instructions": "What does the customer want?",
                   "criteria": {"refund": "Money back for the duplicate charge",
                                "cancel": "Cancel the order",
                                "info": "Just asking for information"}},
        "urgent": {"type": "noul", "instructions": "The customer needs a fast response."},
    })
print(out["intent"]["choice"], out["intent"]["probabilities"])   # refund {'refund': 0.80, 'cancel': 0.17, 'info': 0.03}
print(out["urgent"]["noul"])                                     # 0.06

state and instructions can be strings or JSON objects (rendered as compact JSON with sorted keys, exactly as in training).

Decision Index kit

pip install "decision-index[transformers] @ git+https://github.com/apolinario/decision-index@9eb2dbe"
hf download tardellirs/crivo-68m --local-dir crivo-68m && cd crivo-68m
python -m decision_index pipeline --engine decision_index_engine:DecisionEngine --edition 0.3 --out runs/crivo-68m

The engine defaults are the settings of the reported run: one option order (tta=1), fp32 scoring head, SDPA attention, batch_size=32, token_budget=65536, max_length=32768, bf16 autocast on CUDA. The state is tokenized once per request and all questions are batched. It raises Unsupported only for a question above max_length tokens, and there is none in the 0.3 suite.

Model details

Same design as the other Crivo models. Each question is one sequence

[CLS] instructions [SEP] state [SEP] [OPT] option_1 [OPT] option_2 … [OPT] option_n

(option_i = key: description). The hidden state at each [OPT] marker plus a question-type embedding goes through a small MLP scorer (run in fp32) to one logit per option; a softmax over the question's options gives the distribution. Options of a question see each other, and option order was randomized during training. Training sequences were up to 8,192 tokens; longer inputs are read whole with rotary positions up to 32,768.

Files: model.safetensors (fp32 weights), config.json (ModernBERT config + decision block), tokenizer files, model.py, crivo.py (standalone API), decision_index_engine.py (kit engine).

Training

Full fine-tune of cross-encoder/ettin-reranker-68m-v1 in three stages, on exactly the data of Crivo-400M, with Crivo-400M as the teacher in stages 1 and 2. Every example is one question with a target distribution over its options.

Data

No data labeled by Jev or any other proprietary decision API was used.

Distillation

Crivo-400M's logits were computed once over the stage-1 and stage-2 training rows (it agrees with the gold label on 82.7% and 87.8% of them). The loss is

α · (CE + 0.5 · Brier) + (1 − α) · τ² · KL(teacher ‖ student), with α = 0.5,

τ = 2 for questions with more than two options and τ = 1 otherwise. Sources where the teacher agrees with the gold label on fewer than 60% of their training rows (mostly math, logic and spatial reasoning, human-preference ratings and many tasksource families), plus smart-home control, CLadder and RAGTruth, train on the gold label only: 163 sources in stage 1 and 46 in stage 2. Stage 3 (long context) trains on the gold label only.

Recipe

AdamW (weight decay 0.01), linear warmup (5% stage 1, 3% after) and linear decay, bf16 autocast, option order shuffled for every choice example. Every checkpoint choice (stage-1 final vs EMA as the stage-2 init; stage 2, stage 3 or their average; interpolations of 10% and 20% toward the base model, WiSE-FT) was made on our own panels: held-out new-domain decision families, a development set of the board's skills, an external evaluation-only benchmark and a public-suite sample. None of them used the board. The stage-2/3 average was best; both WiSE interpolations were worse.

Compute. One NVIDIA A100 40 GB (Lambda Cloud). Crivo-68M and Crivo-17M were trained and evaluated in the same run of about 7 hours: teacher logits, the 68M, the three 17M arms, the panels and both complete public suites.

Contamination firewall

The training files are the ones of Crivo-400M, checked at three levels:

1. Suite test rows. Every training example was checked against the 0.3 suite: same normalized state and question, same distinctive state, or a shared passage of ≥120 characters were dropped.
2. Our own panels. Strict leaf-level filter (any string of ≥25 normalized characters) against the held-out panels used for checkpoint selection, so the selection stayed honest.
3. Final audit of the exact training files: every question against all 563,686 questions of the 0.3 suite (normalized state + instructions): 0 matches; the rule above re-applied to every row: 0 hits.

The API-Bank train split was not used: its dialogues and API catalogs repeat those of the test pool. The API-Bank skill comes only from the synthetic dialogues above, built from Hermes function-calling data and filtered against the suite. Benchmarks whose sources have no clean train split were used for evaluation only. The teacher only labeled these same training rows; it never saw suite items.

Intended use and limitations

Intended use: fast, local, probabilistic decisions over text or JSON state, as a first pass or a component inside an agent or pipeline: routing, intent and topic classification, policy gates, tool and API selection, relevance checks, NLI-style verification — where Crivo-150M or Crivo-400M is too heavy.

• Modest on new domains. On the board's private tasks from new domains it scores 5.03 (edition 0.4). Validate on your own data before relying on it for an unfamiliar task.
• English only.
• Little world knowledge, no step-by-step reasoning: near chance on expert QA, math word problems, chess, causal inference and code execution. Use Crivo-150M or Crivo-400M for those.
• Distilled model: it inherits Crivo-400M's biases and errors; where the teacher was wrong, the student often is too.
• The smart-home score is optimistic: the synthetic training cases reuse the suite's request templates (new houses only).
• Long inputs beyond 8,192 tokens are read without truncation but were never seen in training.
• No temperature fitted: treat probabilities as scores to threshold on your own data.
• Not for high-stakes decisions about people without human review.

License

CC BY-NC 4.0, the same as its teacher, Crivo-400M: several training sources are licensed for non-commercial use only (for example ANLI, CC BY-NC 4.0, and some Bekko dataset subsets). Upstream dataset licenses and terms continue to apply to the data. The base model (Ettin reranker 68M) is Apache-2.0.

See also

• The Crivo family, by size: Crivo-17M · Crivo-32M · Crivo-68M · Crivo-150M · Crivo-400M (Full 0.4 22.96, its teacher) · Crivo-1B (Full 0.4 28.96).
• Crivo collection.

Acknowledgements

• Ettin (JHU CLSP) and the Ettin reranker 68M used as the starting point.
• Bekko System One (hotchpotch) for the open training dataset.
• Dinah-0 (Lukitaduarte) for the [OPT] readout design this model follows.
• tasksource for procedural and multi-task typed-decision data.
• Decider (Mapika) for its open teacher data.
• NousResearch, MadeAgents, Team-ACE, Glaive, Galileo, The Atticus Project, bitext and BigCode for the open datasets behind the sibling sources.
• The Decision Index maintainers and its open kit, which made a reproducible comparison possible.

Citation

@misc{stekel2026jointperoption,
  title         = {Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters},
  author        = {Tardelli Ronan Coelho Stekel},
  year          = {2026},
  eprint        = {2610.30970},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.30970}
}

@misc{crivo68m2026,
  title        = {Crivo-68M: a typed-decision encoder distilled from Crivo-400M},
  author       = {Tardelli Stekel},
  year         = {2026},
  howpublished = {Hugging Face model card},
  url          = {https://huggingface.co/tardellirs/crivo-68m}
}