yotambr/safedecide-140m
A small decision model that knows when not to decide.
Give it a state and a typed question (yes/no, pick-one, or score). It returns a
probability per option and an automate / escalate flag backed by a
statistical guarantee:
With probability >= 90% over the calibration draw, the error rate among decisions marked 'automate' is <= 10%, for data exchangeable with the calibration set. Everything else is marked 'escalate'.
Certified levels (pick one with decide(request, alpha=...)):
| Error level α | Threshold | Automated on the calibration carve |
|---|---|---|
| 5% | not certified | 0% (escalates everything) |
| 10% (default) | 0.700 | 22.7% |
| 20% | 0.500 | 66.0% |
Out of the box, the default threshold applied unchanged to typed-decisions test (2,000 decisions, never used for training or calibration) automated 26.2% of them, and 4.4% of those were wrong (budget 10%).
The guarantee does not rely on the model's probabilities being calibrated. It comes from Learn-then-Test risk control (exact Clopper–Pearson bounds, fixed-sequence testing over a fixed threshold grid) on a held-out calibration set.
Code: github.com/yotambraun/safedecide · raw predictions behind every number: yotambr/safedecide-results
Why this exists
System One decision models (Jev, Clef, d3, Laya, Phocinae, …) return probabilities your code can branch on. Their calibration moves with task and phrasing (one audit measured a 0.125 mean gap between the same question asked as yes/no and as a two-option choice), so a threshold tuned on someone else's data says little about yours. This model ships with the tool that turns a re-check on your own data into a guarantee.
Results
Evaluated on typed-decisions
test (400 cases, 2,000 decisions), the jev-distill test_set_30k and its ood
split. The model was fitted on typed-decisions train (it belongs in that
benchmark's "fitted on train" table) and never saw test.
Headline metric: automation rate at a guaranteed error ≤ 10%, i.e. the share of decisions the model may take alone while the error among those decisions stays within budget (δ = 10%).
| Model | n | Accuracy | ECE ↓ | NLL ↓ | Automation @ 10% ↑ | Realised error | Violation rate |
|---|---|---|---|---|---|---|---|
| Phocinae-Largha-150M-v1 (test-exposed) | 2000 | 0.906 [0.892, 0.919] | 0.252 [0.239, 0.265] | 0.822 | 93.9% ± 2.9% | 7.84% | 3% |
| SafeDecide (no ConfTr) | 2000 | 0.754 [0.734, 0.776] | 0.147 [0.130, 0.167] | 0.894 | 33.2% ± 5.2% | 6.33% | 7% |
| SafeDecide (ConfTr) | 2000 | 0.753 [0.731, 0.775] | 0.153 [0.135, 0.173] | 0.898 | 29.7% ± 5.2% | 6.27% | 10% |
| SafeDecide, typed-decisions only | 2000 | 0.745 [0.722, 0.766] | 0.146 [0.128, 0.166] | 0.894 | 23.6% ± 10.6% | 5.46% | 6% |
| SafeDecide general (no ConfTr) | 2000 | 0.684 [0.660, 0.708] | 0.096 [0.075, 0.118] | 0.951 | 2.4% ± 5.8% | 1.47% | 5% |
| SafeDecide general (ConfTr) | 2000 | 0.657 [0.634, 0.680] | 0.065 [0.050, 0.091] | 0.968 | 1.9% ± 4.6% | 1.59% | 4% |
| Prior (reference) | 2000 | 0.487 [0.460, 0.515] | 0.037 [0.026, 0.066] | 1.095 | 0.0% ± 0.0% | 0.00% | 0% |
| Uniform (reference) | 2000 | 0.269 [0.243, 0.293] | 0.049 [0.032, 0.074] | 1.212 | 0.0% ± 0.0% | 0.00% | 0% |
Numbers are means over 100 random calibration/test splits of the same labelled pool, split by case. Brackets are 95% bootstrap intervals over cases. "Violation rate" is how often the realised error on the test half exceeded the budget. It is an empirical sanity check of the guarantee (which promises the true error exceeds the budget with probability ≤ δ); the finite test half adds some sampling noise of its own.
Compared with the models Hugging Face scored
Hugging Face published per-decision results for the models it ran on typed-decisions
test (bucket SaylorTwift/typed-decisions-evals).
The same calibration, case splits and correctness rule (that scorer counts a tie in the
gold as correct, so accuracies differ slightly from the table above) are applied to
every row, ours included. Read it with the training labels: this model was fitted on
typed-decisions train, most rows are general models scored zero-shot, and
Phocinae-Largha's shipped weights saw 500 of the test decisions.
The full table, with 5% and 20% and the Bonferroni comparison, is automation_board.md
in yotambr/safedecide-results; scripts/automation_board.py
in the GitHub repo rebuilds it.
Ablation: does Conformal Training help?
typed_decisions_test, stage 2
| α | Automation without ConfTr | Automation with ConfTr | Paired Δ (mean ± std over splits) |
|---|---|---|---|
| 1% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 2% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 5% | 4.4% | 4.0% | -0.4% ± 7.0% |
| 10% | 33.2% | 29.7% | -3.5% ± 3.6% |
| 20% | 70.8% | 70.4% | -0.4% ± 4.1% |
typed_decisions_test, stage 1
| α | Automation without ConfTr | Automation with ConfTr | Paired Δ (mean ± std over splits) |
|---|---|---|---|
| 1% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 2% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 5% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 10% | 2.4% | 1.9% | -0.5% ± 4.3% |
| 20% | 41.2% | 32.6% | -8.6% ± 3.4% |
jev_test_30k, stage 2
| α | Automation without ConfTr | Automation with ConfTr | Paired Δ (mean ± std over splits) |
|---|---|---|---|
| 1% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 2% | 9.7% | 9.4% | -0.3% ± 17.6% |
| 5% | 44.2% | 42.7% | -1.5% ± 2.4% |
| 10% | 68.9% | 68.7% | -0.1% ± 2.1% |
| 20% | 93.8% | 94.0% | +0.2% ± 1.1% |
jev_test_30k, stage 1
| α | Automation without ConfTr | Automation with ConfTr | Paired Δ (mean ± std over splits) |
|---|---|---|---|
| 1% | 2.7% | 2.8% | +0.2% ± 11.3% |
| 2% | 35.8% | 38.8% | +2.9% ± 19.7% |
| 5% | 60.3% | 68.7% | +8.4% ± 3.1% |
| 10% | 76.0% | 82.4% | +6.5% ± 2.2% |
| 20% | 99.3% | 100.0% | +0.7% ± 1.1% |
jev_ood, stage 2
| α | Automation without ConfTr | Automation with ConfTr | Paired Δ (mean ± std over splits) |
|---|---|---|---|
| 1% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 2% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 5% | 0.8% | 3.6% | +2.8% ± 7.8% |
| 10% | 37.1% | 36.3% | -0.8% ± 3.5% |
| 20% | 79.4% | 80.7% | +1.3% ± 3.4% |
jev_ood, stage 1
| α | Automation without ConfTr | Automation with ConfTr | Paired Δ (mean ± std over splits) |
|---|---|---|---|
| 1% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 2% | 0.0% | 0.0% | +0.0% ± 0.0% |
| 5% | 3.2% | 0.0% | -3.2% ± 8.9% |
| 10% | 43.1% | 12.8% | -30.3% ± 23.3% |
| 20% | 83.1% | 93.0% | +9.9% ± 5.5% |
What happens under distribution shift
Threshold calibrated in-distribution at α=5%. In-distribution: 45.8% automated, 3.27% error. Shifted (OOD split): 39.6% automated, 8.39% error.
The guarantee assumes deployment data looks like the calibration data. If it
doesn't, re-calibrate on labelled examples from your own traffic
(safedecide calibrate). Stricter levels need more of them: on typed-decisions,
600 calibration decisions certified 10% but not 5%. Ticket systems usually
already hold these labels: which team closed the ticket, whether the refund was
approved.
Speed
onnxruntime on 1 CPU thread, one 3-option decision.
| Precision | Size | p50 | p90 | Max logit diff vs PyTorch |
|---|---|---|---|---|
| fp32 | 563.1 MB | 80.0 ms | 89.1 ms | 2.9e-06 |
| int8 | 141.6 MB | 62.7 ms | 68.3 ms | 0.63 |
The guarantee is calibrated on the full-precision model (PyTorch or fp32 ONNX). int8 is smaller and faster but changes the logits (last column), so its automate/escalate flags are not covered unless you re-calibrate it.
Usage
A held-out customer-service case from typed-decisions, in the /v1/systemone shape:
from safedecide import SafeDecider
decider = SafeDecider.from_pretrained("yotambr/safedecide-140m")
response = decider.decide(
{
"state": {
"account": {
"lifetime_value_usd": 228,
"prior_tickets_90d": 0,
"seats": 1,
"tenure_months": 3,
"tier": "standard",
},
"thread": [
{"role": "customer", "text": "My payment didn’t go through, and I’m now locked out of my account."}
],
},
"questions": {
"topic": {
"type": "choice",
"instructions": "What is this customer conversation primarily about?",
"criteria": {
"account": "Login, plan changes, profile or access management.",
"billing": "A charge, invoice, subscription or payment problem.",
"delivery": "Shipping, fulfilment or delivery of a physical item.",
"refund": "The customer is explicitly asking for money back.",
"technical": "The product or service is not working as expected.",
},
},
"needs_human": {
"type": "noul",
"instructions": "This conversation requires a human agent rather than automated handling.",
"criteria": {
"false": "Automation can carry this to resolution.",
"true": "A human must take over: judgement, authority or empathy is required.",
},
},
},
}
)
response["answers"]["topic"] # choice "billing", confidence 0.92, action "automate"
response["answers"]["needs_human"] # noul 0.23 (no), confidence 0.77, action "automate"
response["guarantee"]["statement"] # the promise behind every "automate", in words
Inputs unlike typed-decisions (free text, options without descriptions) are outside
the shipped guarantee: re-certify on your own labelled decisions with
safedecide calibrate --model yotambr/safedecide-140m --data my_decisions.jsonl (format in the
README).
Serve it behind the same /v1/systemone API as Jev: safedecide serve --model yotambr/safedecide-140m.
How it works
- Architecture: each (state, question, option) triple is scored by a
cross-encoder with one logit; softmax over a question's options. Option-order
invariance is exact by construction. Standard
AutoModelForSequenceClassification, so it exports to ONNX and runs in the browser with transformers.js. - Training: soft-label cross-entropy against teacher distributions + Brier
score (proper scoring) + ranked probability score for ordinal questions.
Conformal Training (Stutz et al., ICLR 2022) was tested as the ablation above;
this checkpoint does not use it, since it did not help on typed-decisions. Two stages: jev-distill
openjev_v2plus the typed-decisions train carve, then the typed-decisions train carve alone at 640 tokens (no typed-decisions input is truncated). - Calibration: per-kind temperature scaling (temperatures:
{"noul": 1.1006431532659389, "choice": 1.1134611876261922, "score": 1.089708955367956}), then the selective thresholds above on the typed-decisions calibration carve (600 decisions by case, disjoint from the temperature fit; nevertest).
Data and license
Data profile: clean.
clean: theopenjev_v2stream of jev-distill-corpus-v3 (CC0 at source) plus the typed-decisions train split (Apache-2.0). Released under Apache-2.0.research: additionally theyuri_v1andyuri_v3streams.yuri_v3labels were distilled from TypeSafe Jev 1.13; checkpoints trained on it are released for research and non-commercial use only.
Test data: test_set_30k and ood splits of jev-distill (never trained on) and
the typed-decisions test split.
Limitations
- Decisions, not text. Questions that need multi-step reasoning or world knowledge belong to a generative model.
- The guarantee is marginal over the calibration distribution, not per class and not per customer.
- English-first; the base encoder is multilingual (1800+ languages) but other languages are not evaluated here.
- Not for fully automated decisions with legal, medical, financial or employment effects on people.
Citation
@misc{braun2026safedecide,
title = {SafeDecide: small decision models with guaranteed automation},
author = {Yotam Braun},
year = {2026},
url = {https://huggingface.co/yotambr/safedecide-140m}
}
- Downloads last month
- 27
Model tree for yotambr/safedecide-140m
Base model
jhu-clsp/mmBERT-small