yotambr/safedecide-400m

The larger SafeDecide: a decision model that knows when not to decide.

Give it a state and a typed question (yes/no, pick-one, or score). It returns a probability per option and an automate / escalate flag backed by a statistical guarantee:

With probability >= 90% over the calibration draw, the error rate among decisions marked 'automate' is <= 10%, for data exchangeable with the calibration set. Everything else is marked 'escalate'.

Certified levels (pick one with decide(request, alpha=...)):

Out of the box, the default threshold applied unchanged to typed-decisions test (2,000 decisions, never used for training or calibration) automated 39.7% of them, and 4.3% of those were wrong (budget 10%).

The guarantee does not rely on the model's probabilities being calibrated. It comes from Learn-then-Test risk control (exact Clopper–Pearson bounds, fixed-sequence testing over a fixed threshold grid) on a held-out calibration set.

Code: github.com/yotambraun/safedecide · raw predictions behind every number: yotambr/safedecide-results · the smaller model: yotambr/safedecide-140m

Results

typed-decisions test: 400 cases, 2,000 decisions. The model was fine-tuned on typed-decisions train, so it belongs in that benchmark's "fitted or fine-tuned on train" table. It never saw test.

Paired on the same decisions, this model's accuracy is +3.3 points over v0.1 (95% bootstrap interval over cases: +1.4 to +5.2). It is ahead on every question type (v0.1: noul 0.840, choice 0.717, score 0.718) and on three of the four workflows; on agent_trace_observability it is behind (0.706 vs 0.738):

How to read these:

• Accuracy is agreement with the gold label, as in the 140M model's row. On 35 test decisions the gold probabilities tie; counting a tie as correct gives 0.794, and the benchmark's bench/score_with_jobs.py (first tied option) gives 0.784.
• ECE is 15 equal-width bins on top-option confidence, pooled over the 2,000 decisions (10 bins: 0.172). It is higher than v0.1's: this model's probabilities are sharper than its accuracy warrants. The automate/escalate guarantee does not depend on it.
• Automation @ 10% is the share of decisions the model may take alone with the error among them certified at most 10% (δ = 10%), averaged over 100 random calibration/test splits of test by case (± is the standard deviation over splits).
• Precision. The numbers above are the sealed test pass, run once on an A100 with bf16 autocast. In full precision on CPU (what SafeDecider runs there, and what the thresholds were certified on) the same weights give accuracy 0.7875, KL 0.0825, Brier 0.0455 and ECE 0.172, the same as the sealed pass to within 0.001. Against v0.1, also in full precision (0.7545), the paired gain is the same +3.3 points [+1.4, +5.2].

Chosen on train, sealed before test

Every choice was made on train. The training recipe was chosen by 5-fold cross-validation of the 140M model on the 1,080 train cases outside a 600-decision calibration carve (out-of-fold accuracy: 0.658 for v0.1's schedule, 0.748 for the chosen recipe). This model reuses that recipe, fixed in advance, with no cross-validation of its own. Before test was scored, the SHA-256 of recipe.json (the code commit, every out-of-fold score, the choice, and the hash of each final model's weights and configuration) was posted in discussion #39: 11a60a13db03e1e8c831d7003d7a51bb8e20c4acc2a15b8d6a013872251812e7. test was then scored once, by a notebook pass that refuses to run if any hash has changed. Check it:

sha256sum recipe.json model.safetensors safedecide_config.json

One disclosure: before posting the seal, both models were compared with v0.1 on the 600-decision calibration carve (this model +4.8 points there, +3.3 on test). The same carve certifies the thresholds, so it was looked at twice; test was not looked at.

Speed

PyTorch on 1 CPU thread, one 3-option decision: 605 ms p50 (622 ms p90). The 140M model takes 80 ms on the same input and thread, so pick it when CPU latency matters. There is no ONNX export of this model yet.

Usage

pip install safedecide

A held-out customer-service case from typed-decisions test, in the /v1/systemone shape:

from safedecide import SafeDecider

decider = SafeDecider.from_pretrained("yotambr/safedecide-400m")
response = decider.decide(
    {
        "state": {
            "account": {
                "lifetime_value_usd": 228,
                "prior_tickets_90d": 0,
                "seats": 1,
                "tenure_months": 3,
                "tier": "standard",
            },
            "thread": [
                {"role": "customer", "text": "My payment didn’t go through, and I’m now locked out of my account."}
            ],
        },
        "questions": {
            "topic": {
                "type": "choice",
                "instructions": "What is this customer conversation primarily about?",
                "criteria": {
                    "account": "Login, plan changes, profile or access management.",
                    "billing": "A charge, invoice, subscription or payment problem.",
                    "delivery": "Shipping, fulfilment or delivery of a physical item.",
                    "refund": "The customer is explicitly asking for money back.",
                    "technical": "The product or service is not working as expected.",
                },
            },
            "needs_human": {
                "type": "noul",
                "instructions": "This conversation requires a human agent rather than automated handling.",
                "criteria": {
                    "false": "Automation can carry this to resolution.",
                    "true": "A human must take over: judgement, authority or empathy is required.",
                },
            },
        },
    }
)

response["answers"]["topic"]  # choice "billing", confidence 0.54 ("account" 0.43), action "escalate"
response["answers"]["needs_human"]  # noul 0.37 (no), confidence 0.63, action "automate"
response["guarantee"]["statement"]  # the promise behind every "automate", in words

The case really is split: the gold distribution for its topic is billing 0.58, account 0.33 (and 0.30 for needs a human), so the model hands the topic to a person instead of guessing.

Inputs unlike typed-decisions (free text, options without descriptions) are outside the shipped guarantee: re-certify on your own labelled decisions with safedecide calibrate --model yotambr/safedecide-400m --data my_decisions.jsonl (format in the README).

Serve it behind the same /v1/systemone API as Jev: safedecide serve --model yotambr/safedecide-400m.

How it works

• Architecture: each (state, question, option) triple is scored by a cross-encoder with one logit; softmax over a question's options. Option-order invariance is exact by construction. Standard AutoModelForSequenceClassification on Ettin-encoder-400m (MIT).
• Training: one stage, straight from the base encoder: soft-label cross-entropy against the teacher distributions + Brier score + ranked probability score for ordinal questions, at 640 tokens (no typed-decisions input is truncated). Each epoch also draws about one training-only factor question per two real decisions: questions built from the typed-decisions factors of the training cases, never model input and never from test. Trained on 90% of the 1,080 cases (972) with early stopping on the other 10%; best validation NLL at step 3,000, stopped at 3,500.
• Calibration: per-kind temperature scaling fitted on that 10% (540 decisions; temperatures: {"noul": 1.168, "choice": 1.025, "score": 1.021}), then the selective thresholds above on the calibration carve (600 decisions by case, never trained on; never test).

Data and license

Trained only on the typed-decisions train split (Apache-2.0), with no other decision data. Base encoder MIT. Released under Apache-2.0.

Limitations

• Decisions, not text. Questions that need multi-step reasoning or world knowledge belong to a generative model.
• The guarantee is marginal over the calibration distribution, not per class and not per customer, and only for data like the calibration set.
• Trained on four typed-decisions workflows only; other workflows need re-calibration at the very least.
• English only as evaluated.
• Not for fully automated decisions with legal, medical, financial or employment effects on people.

Citation

@misc{braun2026safedecide,
  title  = {SafeDecide: small decision models with guaranteed automation},
  author = {Yotam Braun},
  year   = {2026},
  url    = {https://huggingface.co/yotambr/safedecide-400m}
}