LLM as a Judge: 13 Steps, 60 Min [2026]

A judge model that agrees with your reviewers most of the time can still be a bad measuring stick. If it favors whatever answer sits in slot A, or whatever answer runs longer, that agreement number mostly reflects luck in your dataset. This tutorial shows how to build an LLM as a judge harness that catches those failures before you trust a single score. The timing matters: in the 11 days between October 2 and October 13, 2026, OpenAI documented its GPT-6 family, Anthropic shipped Claude Haiku 5.5 into a lineup that now reads Opus 5.5, Sonnet 5.5 and Haiku 5.5, and Google launched Gemini 4 Argon, with Gemini app access limited to Google AI Pro and Google AI Ultra subscribers.

You will build a small Python lab in 13 steps and about 60 minutes. It sends every answer pair to a judge twice, once in the original order and once swapped, then scores the verdicts against gold labels with Cohen’s kappa. It also measures position bias, verbosity bias and the flip rate, and it ends with a pass/fail gate you can run in CI. If you already run a general LLM eval pipeline, treat this project as the calibration layer that sits underneath it.

One caveat shapes the whole design. As of October 25, 2026, I found no official figure from Anthropic, OpenAI or Google for how often their models agree with human raters, favor the first answer, or reward length. Vendors publish capability and price, not judge calibration. So this tutorial measures those numbers on your data instead of quoting someone else’s.

The project runs offline first. A built-in mock judge with adjustable biases lets you test the harness for free, and every output block below comes from a real run of that exact code on Python 3.12 against 96 synthetic pairs. You swap in a paid model at step 13, once the plumbing works. You need basic Python and a terminal, nothing else.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

Prerequisites, Versions and Judge Budget for an LLM as a Judge Lab

Start with the boring part, because most failed judge projects die on setup. The table below lists what the lab needs. Python 3.10 is the floor because the code uses the X | Y type-hint syntax, and I tested everything on Python 3.12. I list the three libraries as the latest release on purpose: the code touches only stable, long-lived calls (chat.completions.create, load_dotenv, pytest.approx), so pinning an exact version here would add risk without adding safety.

Requirement	Version or detail	Why you need it
Python	3.10 or newer (tested on 3.12)	Type-hint syntax and the standard library only for the metrics
openai (Python SDK)	Latest release	One client class that talks to any OpenAI-compatible endpoint
python-dotenv	Latest release	Loads judge keys and model IDs from a local .env file
pytest	Latest release	Unit tests that lock the kappa math in place
API access (optional until step 13)	One key per judge vendor	The mock judge covers steps 1 to 12 offline
Time and disk	About 60 minutes, under 5 MB	Dataset, code and result files are tiny

The 2026 judge lineup and what a run costs

Three vendors now offer judge-ready models at very different prices. Anthropic’s Claude Haiku 5.5 lists at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, and Anthropic says it costs roughly 75% less to run than Haiku 4.5 on the workload it cited. OpenAI’s GPT-6 family spans three tiers: GPT-6 Astra at $10 and $50, GPT-6.1 Sol at $2 and $10, and GPT-6 Luna at $0.10 and $0.50, according to a pricing roundup that matches OpenAI’s own rate pages. Gemini 4 Argon launched globally on October 13 at an introductory $2 and $10, with cached input discounted 95%.

Two dates matter for your budget. Argon’s introductory price runs through November 30, 2026 and then moves to $4 per million input tokens and $20 per million output tokens, so any cost model you build today doubles for that judge on December 1. Also, Google’s consumer app access to Argon is limited to paying subscribers, which does not affect the API but explains why some teammates will not see it in the app.

The cost column below uses a planning assumption, not a measured number: 200 pairs judged in both orders is 400 calls, at roughly 600 input tokens and 80 output tokens per call. Multiply by ten for a 2,000-pair audit.

Judge model	Input per 1M tokens	Output per 1M tokens	Cost of 400 calls	Note
Claude Haiku 5.5	$0.10	$0.50	$0.04	Prompts up to 100,000 tokens
GPT-6 Luna	$0.10	$0.50	$0.04	OpenAI’s fast, efficient tier
GPT-6.1 Sol	$2.00	$10.00	$0.80	Lower-cost tier near Astra capability
Gemini 4 Argon (intro price)	$2.00	$10.00	$0.80	Through November 30, 2026
Gemini 4 Argon (from December 1)	$4.00	$20.00	$1.60	Announced post-introduction price
GPT-6 Astra	$10.00	$50.00	$4.00	OpenAI’s highest-intelligence tier

Claude Sonnet 5.5 and Opus 5.5 are also current. Anthropic says Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work, per a roundup of its October releases. I left both models out of the table because I could not confirm their per-token prices from a primary source today. The practical takeaway: judging at this scale costs pennies to a few dollars, so cost should never push you toward skipping the bias tests. One more status note from the research: Anthropic reportedly plans to retire Claude Sonnet 4.5 from its API on November 30, 2026, so do not build a long-lived judge on that model.

Finally, a naming warning. Consumer names such as GPT-6 Luna and API model IDs are not always the same string, and I could not confirm every API ID as of today. That is why the project reads each model ID from your .env file instead of hard-coding it. Copy the exact ID from each vendor’s model list.

Steps 1 and 2: Create the Project and Install Dependencies

Step 1 is the project skeleton. Create the folder tree, then a virtual environment so the lab never touches your system Python. The commands below work on Linux and macOS. On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 instead.

mkdir -p judge-lab/{data,judgelab,tests,results} && cd judge-lab
python3 -m venv .venv
source .venv/bin/activate
python --version      # expect 3.10 or newer
touch judgelab/__init__.py
printf '.env\n.cache/\n.venv/\nresults/\n' > .gitignore

Notice the .gitignore line. Your .env file will hold live API keys, and a judge project invites you to paste three of them into one place. Ignoring the file now costs five seconds. Rotating a leaked key later costs an afternoon.

Step 2 installs the three libraries and creates the environment template. Save the first block as requirements.txt and the second as .env.example, then copy the template to .env and fill in the vendors you actually use. Leave the others blank, because the harness only reads the letter you pass on the command line.

openai
python-dotenv
pytest

# One block per judge. Paste exact model IDs from each vendor's model list.
# Base URLs: confirm the OpenAI-compatible endpoint in each vendor's docs.
JUDGE_A_NAME=haiku-5-5
JUDGE_A_MODEL=<exact model ID from Anthropic's model list>
JUDGE_A_BASE_URL=https://api.anthropic.com/v1/
JUDGE_A_API_KEY=sk-ant-...

JUDGE_B_NAME=gpt-6-luna
JUDGE_B_MODEL=<exact model ID from OpenAI's model list>
JUDGE_B_BASE_URL=https://api.openai.com/v1/
JUDGE_B_API_KEY=sk-...

JUDGE_C_NAME=gemini-4-argon
JUDGE_C_MODEL=<exact model ID from Google's model list>
JUDGE_C_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
JUDGE_C_API_KEY=...

pip install -r requirements.txt
cp .env.example .env
printf '[pytest]\npythonpath = .\ntestpaths = tests\n' > pytest.ini

Why one OpenAI-style client for every vendor? The three judges you will compare each document an OpenAI-compatible chat endpoint, and a single client class means the exact same request code runs against each one. That removes a whole class of bugs where a Claude adapter and a Gemini adapter quietly format the prompt differently. Confirm each base URL in the vendor’s current docs before you rely on the examples in .env.example, since endpoints are the part of an API most likely to change.

The pytest.ini file looks trivial but fixes the most common first-run failure in this layout: without pythonpath = ., pytest cannot import the judgelab package and dies at collection time. You will see that exact error in the troubleshooting section.

Steps 3 and 4: Build a Gold-Labeled Pair Dataset

Step 3 is the step people skip, and it is the one that decides whether your judge numbers mean anything. A judge needs something to be measured against. That something is a set of answer pairs where a trusted human already decided which answer wins. Everything else in this lab, from kappa to the bias tests, compares the judge’s verdicts to those gold labels.

Each record in data/pairs.jsonl holds five fields: an id, the question, answer_a, answer_b, and gold, which is either A or B. For a real project, collect 100 to 200 pairs from your own production traffic and have one domain expert label them. Binary labels keep kappa simple. If your experts want a tie option, add it later (the advanced tips cover that).

Step 4 generates a demo dataset so you can run everything today without labeling anything. The generator is deliberately counterbalanced, which is the key design idea. Half of the correct answers sit in slot A and half in slot B, so a judge that always says A scores 50% accuracy and a kappa near zero. Half of the correct answers are also padded with filler text and half of the wrong answers are, so length tells the judge nothing about truth. A fair test has to break those two shortcuts by construction.

import json
import random
from pathlib import Path

FACTS = [
    ("What is the capital of Australia?", "Canberra", "Sydney"),
    ("How many bits are in a byte?", "8", "10"),
    ("Which protocol secures HTTP traffic?", "TLS", "FTP"),
    ("What does DNS stand for?", "Domain Name System", "Digital Network Service"),
    ("Which data structure is LIFO?", "A stack", "A queue"),
    ("What is 12 * 12?", "144", "124"),
    ("Which HTTP status means Not Found?", "404", "403"),
    ("What does SQL stand for?", "Structured Query Language", "Simple Question Language"),
    ("Which sorting algorithm has O(n log n) average case?", "Merge sort", "Bubble sort"),
    ("What does CPU stand for?", "Central Processing Unit", "Core Process Utility"),
    ("Which port does HTTPS use by default?", "443", "80"),
    ("What is the boiling point of water at sea level in Celsius?", "100", "90"),
]

PADDING = (
    " To give some background, this is a topic that comes up often, and it is worth "
    "being careful, because many people mix it up. In practice, the details matter, "
    "and the surrounding context can change how the answer is read in production."
)


def build_demo(path: str = "data/pairs.jsonl", n: int = 96, seed: int = 7) -> None:
    rng = random.Random(seed)
    rows = []
    for i in range(n):
        q, good, bad = FACTS[i % len(FACTS)]
        good_long = rng.random() < 0.5  # half the time the CORRECT answer is the wordy one
        a_good = rng.random() < 0.5     # gold winner sits in slot A half the time
        good_txt = good + (PADDING * 2 if good_long else "")
        bad_txt = bad + ("" if good_long else PADDING * 2)
        rows.append({
            "id": f"p{i:03d}",
            "question": q,
            "answer_a": good_txt if a_good else bad_txt,
            "answer_b": bad_txt if a_good else good_txt,
            "gold": "A" if a_good else "B",
        })
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    with open(path, "w") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")


def load(path: str) -> list[dict]:
    return [json.loads(line) for line in open(path) if line.strip()]


if __name__ == "__main__":
    build_demo()
    print("wrote data/pairs.jsonl")

Run it once. The seed is fixed at 7, so you get the same 96 pairs I did.

$ python -m judgelab.data
wrote data/pairs.jsonl
96 data/pairs.jsonl

Two honest limits apply to this demo set. The twelve facts repeat eight times each, so it tests the harness and not the model’s knowledge, and the gold label is ground truth rather than a human opinion. Real tasks are fuzzier: two answers can both be acceptable, and your experts will disagree with each other. When that happens, have two people label a sample and compute kappa between them first. A judge cannot reasonably beat the agreement your own humans reach.

Step 5: Write a Judge Prompt That Ignores Style

The prompt is the judge’s entire job description, so write it like a contract. Mine does five things. It tells the model to judge only factual correctness and task fulfilment. It states outright that length, politeness and formatting earn nothing. It asks for a one-sentence reason before the winner, which gives the model a moment to reason without burning many output tokens. It demands JSON only, so parsing stays a one-line regex. And it never mentions the gold label, the model that wrote either answer, or any score.

JUDGE_SYSTEM = """You are a strict evaluation judge. You compare two answers to the same question.
Judge ONLY factual correctness and whether the answer does what the question asks.
Do NOT reward length, politeness, or formatting. A short correct answer beats a long wrong one.
Reply with JSON only: {"reason": "<one sentence>", "winner": "A" or "B"}"""

JUDGE_USER = """Question:
{question}

Answer A:
{answer_a}

Answer B:
{answer_b}

Which answer is better? Reply with JSON only."""

Notice the explicit sentence a short correct answer beats a long wrong one. That line targets verbosity bias directly, and you will test whether it works at step 10. Prompt instructions reduce a bias, but they rarely erase it, which is why the measurement matters more than the wording.

Keep the criterion to one thing per judge. A prompt that asks for correctness, tone, safety and concision in a single verdict produces a verdict nobody can debug. If you need four criteria, run four judges with four short prompts and four kappa scores. Each one stays testable, and you can see which dimension a model handles badly.

Also keep the system prompt stable between runs. The cache key in the next step includes a hash of both prompt templates, so any edit invalidates stored verdicts automatically. That is the behavior you want, because verdicts produced under an old prompt describe a different judge.

Step 6: Build the Judge Client With Caching and Retries

This file holds two judge classes behind one method, pick(uid, question, ans_a, ans_b, gold_slot), which returns "A" or "B" for the slot the judge preferred. The harness never needs to know which class it is talking to.

MockJudge is the offline test double. It samples a verdict from a logistic model with three dials: skill (how strongly it follows the truth), pos_bias (a push toward slot A) and len_bias (a push toward the longer answer). Because you set the dials, you know the right answer to every test. If the harness reports a position bias on mock-first and none on mock-clean, the harness works. That is the same logic as testing a thermometer in ice water before you trust it on a patient.

ApiJudge is the real one. It sends the prompt at temperature 0, caps output at 200 tokens, parses the winner with a regex, and retries up to four times with exponential backoff (1, 2 and 4 seconds). Every successful verdict is appended to .cache/verdicts.jsonl under a hash of the judge name, model ID, prompt version, question and both answers. Re-running an evaluation then costs nothing for pairs you already judged, which makes iterating on the report logic painless.

from __future__ import annotations

import hashlib
import json
import math
import os
import random
import re
import time
from pathlib import Path

from .prompts import JUDGE_SYSTEM, JUDGE_USER

CACHE = Path(".cache/verdicts.jsonl")
PROMPT_ID = hashlib.sha256((JUDGE_SYSTEM + JUDGE_USER).encode()).hexdigest()[:12]


def _key(*parts: str) -> str:
    return hashlib.sha256("||".join(parts).encode()).hexdigest()[:24]


class MockJudge:
    """Offline judge with tunable, known biases. Lets you test the harness for free."""

    def __init__(self, name="mock", skill=2.0, pos_bias=0.0, len_bias=0.0, seed=7):
        self.name, self.skill, self.pos_bias, self.len_bias, self.seed = (
            name, skill, pos_bias, len_bias, seed)

    def pick(self, uid, question, ans_a, ans_b, gold_slot):
        rng = random.Random(_key(self.name, uid, question, ans_a, ans_b, str(self.seed)))
        sign = 1 if gold_slot == "A" else -1
        logit = self.skill * sign + self.pos_bias
        logit += self.len_bias * math.log(max(len(ans_a), 1) / max(len(ans_b), 1))
        p_a = 1 / (1 + math.exp(-logit))
        return "A" if rng.random() < p_a else "B"


class ApiJudge:
    def __init__(self, name, model, base_url, api_key):
        from openai import OpenAI
        self.name, self.model = name, model
        self.client = OpenAI(base_url=base_url, api_key=api_key)

    def pick(self, uid, question, ans_a, ans_b, gold_slot=None):
        ck = _key(self.name, self.model, PROMPT_ID, question, ans_a, ans_b)
        CACHE.parent.mkdir(exist_ok=True)
        if CACHE.exists():
            for line in CACHE.read_text().splitlines():
                rec = json.loads(line)
                if rec["k"] == ck:
                    return rec["w"]
        msg = JUDGE_USER.format(question=question, answer_a=ans_a, answer_b=ans_b)
        for attempt in range(4):
            try:
                resp = self.client.chat.completions.create(
                    model=self.model,
                    temperature=0,
                    max_tokens=200,
                    messages=[{"role": "system", "content": JUDGE_SYSTEM},
                              {"role": "user", "content": msg}],
                )
                text = resp.choices[0].message.content or ""
                m = re.search(r'"winner"\s*:\s*"([AB])"', text)
                if not m:
                    raise ValueError(f"unparseable verdict: {text[:120]!r}")
                with CACHE.open("a") as f:
                    f.write(json.dumps({"k": ck, "w": m.group(1)}) + "\n")
                return m.group(1)
            except Exception:
                if attempt == 3:
                    raise
                time.sleep(2 ** attempt)


def from_env(letter: str) -> ApiJudge:
    from dotenv import load_dotenv
    load_dotenv()
    p = f"JUDGE_{letter.upper()}_"
    return ApiJudge(os.environ[p + "NAME"], os.environ[p + "MODEL"],
                    os.environ[p + "BASE_URL"], os.environ[p + "API_KEY"])

Two details deserve a second look. First, the uid argument feeds the mock’s random seed so that repeated question and answer combinations still get independent draws. My first version omitted it, and the 96-pair demo set produced lumpy statistics because identical inputs received identical coin flips. Second, the model ID lives in the cache key, so switching from one model snapshot to another never serves stale verdicts.

Pinning matters beyond caching. OpenAI’s documentation recommends pinning production applications to specific model snapshots for consistent behavior, and the same advice applies to a judge more than to almost any other component. A floating alias that silently upgrades will shift your kappa between Tuesday and Wednesday, and you will blame your own code.

Step 7: Score Agreement With Cohen’s Kappa and Lock It In With Tests

Raw accuracy lies when labels are lopsided or when a judge has a habit. Cohen’s kappa corrects for the agreement two raters would reach by chance. The formula is short: kappa equals observed agreement minus expected agreement, divided by one minus expected agreement. Observed agreement is the share of items where the two raters match. Expected agreement multiplies each rater’s label frequencies and sums them.

Work one example by hand. Suppose 50 items, with the gold rater saying yes 25 times and no 25 times, and the judge matching on 20 yes and 15 no items. Observed agreement is 35 of 50, or 0.70. A judge that says yes 30 times and no 20 times against a 25/25 gold set has expected agreement of (0.5 x 0.6) + (0.5 x 0.4), which is 0.50. Kappa is (0.70 minus 0.50) divided by 0.50, so 0.40. The judge looks 70% accurate, yet kappa says it covers only 40% of the gap between chance and perfect agreement.

This is why an always-A judge scores zero kappa on a balanced set, even though it is right half the time. Kappa also punishes a judge that learned your label distribution instead of your task. The module below adds a bootstrap confidence interval too. With only 96 pairs, that interval around kappa can be wide, and you should know that before you celebrate or panic.

The summarize function turns per-pair rows into the full report. It computes kappa for the forward pass, the reversed pass and a final verdict that only counts when both orders agree (disagreements become a tie label, which kappa then treats as a miss). It also returns the flip rate, the slot-A pick rate, and the longer-answer win rate on trap pairs. Steps 8 to 10 explain each of those.

from __future__ import annotations

import random
from collections import Counter


def cohen_kappa(a: list[str], b: list[str]) -> float:
    """Cohen's kappa for two raters over the same items."""
    if len(a) != len(b) or not a:
        raise ValueError("need two equal-length, non-empty label lists")
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[k] / n) * (cb[k] / n) for k in set(ca) | set(cb))
    if pe == 1.0:
        return 1.0 if po == 1.0 else 0.0
    return (po - pe) / (1 - pe)


def bootstrap_ci(a, b, reps=2000, seed=11):
    rng = random.Random(seed)
    idx = range(len(a))
    stats = []
    for _ in range(reps):
        pick = [rng.choice(idx) for _ in idx]
        stats.append(cohen_kappa([a[i] for i in pick], [b[i] for i in pick]))
    stats.sort()
    return stats[int(0.025 * reps)], stats[int(0.975 * reps)]


def summarize(rows: list[dict]) -> dict:
    """rows: one dict per pair with gold, v_fwd, v_rev, slot_fwd, slot_rev, len_ratio."""
    gold = [r["gold"] for r in rows]
    fwd = [r["v_fwd"] for r in rows]
    rev = [r["v_rev"] for r in rows]
    final = [r["v_fwd"] if r["v_fwd"] == r["v_rev"] else "tie" for r in rows]

    flips = sum(r["v_fwd"] != r["v_rev"] for r in rows)
    slots = [r["slot_fwd"] for r in rows] + [r["slot_rev"] for r in rows]
    first_slot = sum(s == "A" for s in slots) / len(slots)

    # pairs where gold prefers the SHORTER answer and the other is >=1.3x longer
    trap = [r for r in rows if r["gold_is_shorter"] and r["len_ratio"] >= 1.3]
    longer_wins = (
        sum(r["v_fwd"] == r["longer"] for r in trap) / len(trap) if trap else float("nan")
    )
    lo, hi = bootstrap_ci(gold, final)
    return {
        "n": len(rows),
        "kappa_fwd": cohen_kappa(gold, fwd),
        "kappa_rev": cohen_kappa(gold, rev),
        "kappa_final": cohen_kappa(gold, final),
        "kappa_ci95": (lo, hi),
        "flip_rate": flips / len(rows),
        "first_slot_rate": first_slot,
        "trap_n": len(trap),
        "longer_win_rate": longer_wins,
    }

Never ship a metric without a test. These three tests pin the math: perfect agreement gives 1.0, the textbook example above gives exactly 0.4, and an always-A judge on a balanced set gives 0.0. If you later refactor the kappa function, the tests catch a broken formula in under a second.

import pytest
from judgelab.metrics import cohen_kappa


def test_perfect_agreement():
    assert cohen_kappa(list("ABAB"), list("ABAB")) == pytest.approx(1.0)


def test_textbook_example():
    # 20 yes/yes, 5 yes/no, 10 no/yes, 15 no/no -> po=0.7, pe=0.5, kappa=0.4
    a = ["Y"] * 25 + ["N"] * 25
    b = ["Y"] * 20 + ["N"] * 5 + ["Y"] * 10 + ["N"] * 15
    assert cohen_kappa(a, b) == pytest.approx(0.4)


def test_always_a_judge_scores_zero():
    gold = list("ABABABAB")
    assert cohen_kappa(gold, ["A"] * 8) == pytest.approx(0.0)

$ pytest -v
tests/test_metrics.py::test_perfect_agreement PASSED                     [ 33%]
tests/test_metrics.py::test_textbook_example PASSED                      [ 66%]
tests/test_metrics.py::test_always_a_judge_scores_zero PASSED            [100%]
============================== 3 passed in 0.01s ===============================

Steps 8 and 9: Run Both Orders and Measure Position Bias

Step 8 is the evaluation loop. For each pair, the harness asks the judge twice. In the forward pass, answers appear as stored. In the reversed pass, the two answers swap slots, and the harness flips the verdict back into original labels so both passes speak the same language. A consistent judge names the same underlying answer both times. An inconsistent one just reveals which slot it likes.

import argparse
import json
from pathlib import Path

from judgelab.data import load
from judgelab.judges import MockJudge, from_env
from judgelab.metrics import summarize

MOCKS = {
    "mock-clean": dict(skill=2.0, pos_bias=0.0, len_bias=0.0),
    "mock-first": dict(skill=2.0, pos_bias=1.6, len_bias=0.0),
    "mock-wordy": dict(skill=2.0, pos_bias=0.0, len_bias=0.9),
}


def build_judge(name):
    if name in MOCKS:
        return MockJudge(name=name, **MOCKS[name])
    return from_env(name)


def evaluate(judge, pairs):
    rows = []
    for p in pairs:
        # forward: as stored. reverse: answers swapped.
        slot_fwd = judge.pick(p["id"] + "-fwd", p["question"], p["answer_a"], p["answer_b"], p["gold"])
        swap_gold = "B" if p["gold"] == "A" else "A"
        slot_rev = judge.pick(p["id"] + "-rev", p["question"], p["answer_b"], p["answer_a"], swap_gold)
        v_rev = "B" if slot_rev == "A" else "A"   # map back to original labels
        la, lb = len(p["answer_a"]), len(p["answer_b"])
        longer = "A" if la >= lb else "B"
        rows.append({
            "id": p["id"], "gold": p["gold"],
            "v_fwd": slot_fwd, "v_rev": v_rev,
            "slot_fwd": slot_fwd, "slot_rev": slot_rev,
            "len_ratio": max(la, lb) / max(min(la, lb), 1),
            "longer": longer,
            "gold_is_shorter": p["gold"] != longer,
        })
    return rows


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--judge", required=True, help="a|b|c or mock-clean|mock-first|mock-wordy")
    ap.add_argument("--data", default="data/pairs.jsonl")
    ap.add_argument("--out", default="results")
    a = ap.parse_args()

    judge = build_judge(a.judge)
    rows = evaluate(judge, load(a.data))
    s = summarize(rows)
    Path(a.out).mkdir(exist_ok=True)
    Path(a.out, f"{judge.name}.jsonl").write_text("\n".join(json.dumps(r) for r in rows))

    print(f"judge            : {judge.name}")
    print(f"pairs            : {s['n']}")
    print(f"kappa (forward)  : {s['kappa_fwd']:.3f}")
    print(f"kappa (reversed) : {s['kappa_rev']:.3f}")
    print(f"kappa (final)    : {s['kappa_final']:.3f}  95% CI {s['kappa_ci95'][0]:.3f} to {s['kappa_ci95'][1]:.3f}")
    print(f"flip rate        : {s['flip_rate']:.1%}")
    print(f"slot-A pick rate : {s['first_slot_rate']:.1%}")
    print(f"longer wins trap : {s['longer_win_rate']:.1%} of {s['trap_n']} pairs")


if __name__ == "__main__":
    main()

The mapping line deserves attention: v_rev = "B" if slot_rev == "A" else "A". When the judge picks slot A in the reversed pass, it picked the original answer B. Get this wrong and every metric after it is garbage, so it is worth reading twice.

Step 9 reads the position-bias signals. Three numbers matter. The flip rate is the share of pairs where the two passes disagree. The slot-A pick rate is the share of all verdicts that chose the first slot, and on a counterbalanced dataset an unbiased judge sits near 50%. The kappa gap between forward and reversed passes shows which order the judge handles better. Run the clean mock first as your baseline.

$ python run_eval.py --judge mock-clean
judge            : mock-clean
pairs            : 96
kappa (forward)  : 0.787
kappa (reversed) : 0.724
kappa (final)    : 0.640  95% CI 0.523 to 0.761
flip rate        : 17.7%
slot-A pick rate : 50.5%
longer wins trap : 11.6% of 43 pairs

A clean judge with real skill still flips on some pairs here. That is ordinary noise, because even an honest judge sometimes errs on one pass and not the other. The tell is the slot-A rate: a value near half means no preference for position. Now run the judge with a built-in push toward the first slot.

$ python run_eval.py --judge mock-first
judge            : mock-first
pairs            : 96
kappa (forward)  : 0.487
kappa (reversed) : 0.617
kappa (final)    : 0.390  95% CI 0.291 to 0.491
flip rate        : 41.7%
slot-A pick rate : 69.8%
longer wins trap : 25.6% of 43 pairs

The bias announces itself. The slot-A pick rate jumps, the flip rate climbs sharply, and the final kappa falls well below the clean baseline. Notice the forward and reversed kappas differ even though the underlying judge is the same. A single-order evaluation would have reported either number and missed the problem. That asymmetry is the strongest argument for always running both orders.

Reversed runs double your call count, so they double your bill. At the prices above that is a few cents to a few dollars for a 200-pair set, and the swap is the only test here that no amount of prompt wording can replace.

Step 10: Test Verbosity Bias With Length Traps

Verbosity bias is the tendency to rate a longer answer higher regardless of quality. Position swaps cannot catch it, because length travels with the answer into whichever slot it lands in. You need a separate probe.

The probe is built into the dataset. In half of the pairs, the correct answer is the padded one, and in the other half the correct answer is the short one. The trap pairs are the second group: gold prefers the shorter answer, and the other answer is at least 1.3 times longer. On those pairs, a length-blind judge should pick the longer answer rarely, since it is wrong. The longer_win_rate metric counts how often it does anyway.

$ python run_eval.py --judge mock-wordy
judge            : mock-wordy
pairs            : 96
kappa (forward)  : 0.257
kappa (reversed) : 0.269
kappa (final)    : 0.229  95% CI 0.078 to 0.390
flip rate        : 14.6%
slot-A pick rate : 53.1%
longer wins trap : 79.1% of 43 pairs

Compare that to the clean baseline. The clean mock picks the longer (wrong) answer on only a small share of the 43 trap pairs, which is its ordinary error rate. The wordy mock does it on most of them. Its slot-A rate looks perfectly healthy and its flip rate is low, so a position-only test would have given it a clean bill of health. Its final kappa of 0.229 tells the real story.

This is the central lesson of the lab: each bias needs its own probe. A judge can pass one and fail the other. If you only swap orders, you will approve a judge that loves long answers. If you only check length, you will approve one that loves slot A.

One refinement for real data: bucket your trap pairs by length ratio (1.3 to 2x, 2 to 4x, above 4x). Verbosity bias usually grows with the ratio, and a bucketed table shows whether a judge only stumbles on extreme padding or starts failing at mild differences, which is where real-world answers actually live.

Steps 11 and 12: Compare Judges and Set a Ship Gate

Step 11 turns the per-judge result files into one comparison table. compare.py reads every file in results/, recomputes the metrics, and applies a gate. A judge passes only if all four conditions hold. The script then exits with code 1 if any judge fails, which is exactly what a CI job needs.

import json
import sys
from pathlib import Path

from judgelab.metrics import summarize

GATE = {"kappa_final": 0.60, "flip_rate": 0.20, "slot_gap": 0.05, "longer_win_rate": 0.25}


def passes(s):
    return (
        s["kappa_final"] >= GATE["kappa_final"]
        and s["flip_rate"] <= GATE["flip_rate"]
        and abs(s["first_slot_rate"] - 0.5) <= GATE["slot_gap"]
        and s["longer_win_rate"] <= GATE["longer_win_rate"]
    )


def main(folder="results"):
    print(f"{'judge':<14}{'kappa':>7}{'flip':>8}{'slot-A':>8}{'longer':>8}  gate")
    ok_all = True
    for f in sorted(Path(folder).glob("*.jsonl")):
        rows = [json.loads(x) for x in f.read_text().splitlines()]
        s = summarize(rows)
        ok = passes(s)
        ok_all &= ok
        print(f"{f.stem:<14}{s['kappa_final']:>7.3f}{s['flip_rate']:>8.1%}"
              f"{s['first_slot_rate']:>8.1%}{s['longer_win_rate']:>8.1%}  {'PASS' if ok else 'FAIL'}")
    sys.exit(0 if ok_all else 1)


if __name__ == "__main__":
    main(*sys.argv[1:])

$ python compare.py
judge           kappa    flip  slot-A  longer  gate
mock-clean      0.640   17.7%   50.5%   11.6%  PASS
mock-first      0.390   41.7%   69.8%   25.6%  FAIL
mock-wordy      0.229   14.6%   53.1%   79.1%  FAIL
exit=1

Step 12 is choosing your own gate, and I want to be direct about where my numbers come from. The thresholds in GATE are starting points I picked for this demo, not industry standards: final kappa of at least 0.60, flip rate no higher than 20%, slot-A rate within 5 points of 50%, and a longer-answer win rate on trap pairs of at most 25%. Tune them to the stakes. A judge that gates a nightly regression dashboard can be looser than one that decides which model ships to customers.

Mock judge	Built-in flaw	Final kappa	Flip rate	Slot-A rate	Longer wins trap	Gate
mock-clean	None	Not listed	Not listed	Not listed	Not listed	PASS
mock-first	Pushes toward slot A	Not listed	Not listed	Not listed	Not listed	FAIL
mock-wordy	Rewards length	0.229	Not listed	Not listed	Not listed	FAIL

Read the table as a diagnostic matrix. Each flawed judge fails on the column that matches its flaw, and each one fails the gate. The clean judge passes, but only just on flip rate, which is the signal to run a larger sample before you trust it. Wire the script into CI with a single line such as python run_eval.py --judge a && python compare.py so a prompt edit or a model swap that degrades calibration blocks the merge.

The gate also protects you from the quiet failure: a vendor-side model update. Pin the snapshot where the API allows it, store the verdict file with the date, and rerun the gate on a schedule. Calibration is a property of a model and a prompt and a day, not a permanent badge.

Step 13: Point the Harness at Real Judge Models

With the plumbing proven, swap the mock for a paid model. Open .env, paste the exact model ID for each vendor from its console, and run one judge at a time. Letters a, b and c map to the three blocks in the template.

python run_eval.py --judge a      # e.g. Claude Haiku 5.5
python run_eval.py --judge b      # e.g. GPT-6 Luna
python run_eval.py --judge c      # e.g. Gemini 4 Argon
python compare.py

Start with the cheapest model. At $0.04 per 400-call run on Haiku 5.5 or GPT-6 Luna, you can afford to rerun while you debug the parser and the prompt. Only then pay for Sol, Argon or Astra. A useful pattern is a tiered setup: a cheap judge screens everything, and an expensive judge re-judges only the pairs where the cheap one flipped or disagreed with gold.

Expect your first real run to look worse than the mock-clean baseline, and do not panic. The demo set is easy by design. Your own data will include borderline pairs and gold labels that two humans would argue about. If a real judge lands near kappa 0.3 on your hard set, that is information about your task, not necessarily a defect in the harness.

Watch for self-preference too. A judge can favor answers written by its own model family, and the harness will not see it unless your pairs include answers from several sources. When you judge outputs from model X, include at least one judge from a different vendor, then compare their verdicts on the pairs that involve X. If you must use a single judge, keep it out of the candidate pool.

Finally, log the run. For every result file, record the exact model ID, the date, the prompt version, the temperature, and the pricing tier you paid. Prices move: Argon’s rate doubles on December 1, and the Claude and GPT lineups have each changed within the past month. A judge result without that metadata cannot be reproduced six weeks later.

Common Pitfalls When You Build an LLM as a Judge

These six mistakes cause most broken judge setups. Each one produces a plausible-looking number, which is what makes them dangerous.

• Reporting accuracy instead of kappa. On a lopsided dataset, a judge that always picks the majority label scores high accuracy and zero skill. Report kappa, and report the confidence interval beside it.
• Single-order evaluation. Running only the original order hides position bias completely. Always run the swap, and always report the flip rate.
• Tuning the prompt on the pairs you report on. Edit the prompt against the same 96 pairs for an hour and your kappa will rise for reasons unrelated to quality. Split off a hold-out set that you touch once.
• Unbalanced gold labels. If 90% of your gold winners sit in slot A, a position-biased judge looks great. Counterbalance slots and lengths the way data.py does.
• Floating model aliases. A name that points at “the latest model” changes under you. Pin a snapshot where one exists, and store the ID with every result.
• Leaking context to the judge. Model names, previous scores, or a hint about which answer is “new” all bias the verdict. Keep the judge blind to anything that is not the question and the two answers.

A seventh trap deserves a mention: tiny samples. With 20 pairs, a single flipped verdict moves kappa by several points, and the bootstrap interval will span most of the useful range. Treat any result under about 100 pairs as a smoke test, not a measurement.

Troubleshooting: 10 Errors and Their Fixes

Here are the failures you are most likely to hit, with the cause and the fix for each.

1. ModuleNotFoundError: No module named ‘judgelab’. You ran Python from the wrong folder, or pytest cannot see the package. Run commands from the project root, keep pytest.ini with pythonpath = ., and use python -m judgelab.data for the generator.
2. KeyError: ‘JUDGE_A_NAME’. The .env file is missing, empty, or you launched the script from another directory. Copy .env.example to .env and fill the block for the letter you pass.
3. NotFoundError: model not found (404). The model string is wrong. Consumer product names and API IDs differ, so copy the exact ID from the vendor’s model list instead of typing the marketing name.
4. AuthenticationError (401). Either the key is wrong or it belongs to a different vendor than the base URL. Check that each block pairs one vendor’s key with that vendor’s endpoint.
5. RateLimitError (429). The loop is sequential, but small accounts still hit limits. The retry helper backs off up to 4 attempts. If you still fail, add a time.sleep(0.5) between pairs or switch to the vendor’s batch endpoint.
6. ValueError: unparseable verdict. The model wrapped its answer in prose or markdown, or the output got cut off. Raise max_tokens (reasoning-style models spend part of the budget thinking), and check that the reply contains "winner": "A" or "B".
7. Unsupported parameter: temperature or max_tokens. Some reasoning-class endpoints reject those fields. Remove temperature from the call, or swap max_tokens for the endpoint’s own limit parameter, and keep your repeat-run flip rate in mind since you lose determinism.
8. Kappa prints 0.000 for every judge. The judge returns the same label every time. Open the result file, count the slot-A rate, and look for a prompt that reveals nothing or a parser that defaults to A.
9. Results did not change after a prompt edit. You are reading cached verdicts. The cache key includes a prompt hash, so editing a template invalidates it, but if you edited a copy of the prompt elsewhere, delete .cache/ and rerun.
10. The kappa interval is huge. Your sample is too small. Add pairs until the 95% interval narrows enough to separate judges, and compare judges on the same pairs, not different subsets.

When a failure is not on this list, shrink it first. Run python run_eval.py --judge mock-clean. If the mock works and the real judge fails, the bug is in the API layer (key, URL, model ID, parameters). If the mock fails too, it is in your data or metrics.

Advanced Tips for a Production-Grade Judge

Once the basic gate passes, these upgrades earn their keep. I ordered them by payoff per hour of work.

• Use a judge panel. Run two or three judges from different vendors and take the majority verdict. Disagreements between vendors are your best candidates for human review, and a panel dilutes any single model’s self-preference.
• Add a tie option. Real answer pairs are sometimes equal. Extend the prompt with "tie", add a third label to the gold file, and kappa will handle three categories with no code change.
• Cache the rubric. The system prompt repeats on every call. Providers that discount cached input (Gemini 4 Argon lists a 95% discount on cached input tokens) reward a long, stable rubric placed first in the prompt.
• Use batch endpoints for audits. Anthropic and OpenAI both list a 50% saving for batch processing on their pricing pages. A nightly 2,000-pair audit does not need an instant answer.
• Audit the disagreements, not the agreements. Sample the pairs where the judge and gold differ, read twenty, and sort them into judge error versus label error. Label errors are common and fixing them lifts kappa without touching the model.
• Slice your kappa. Compute it per topic, per answer length bucket, and per source model. A judge with an overall kappa of 0.6 can hide a 0.2 on one slice that matters to you.

For broader context on where this fits, the AI and machine learning section tracks the model releases that keep resetting your baseline. Each new release is a reason to rerun the gate, not a reason to trust the new model’s marketing.

Finally, schedule the calibration. Rerun the full lab monthly and after any vendor model change. Store each run’s summary in a table and chart kappa over time. A slow downward drift is how you discover that your judge, or your data distribution, moved.

The Complete Working Project and Run Order

Everything lives in one small folder. The tree below is the final layout, and every file in it appeared in full in the steps above.

judge-lab/
  .env.example
  requirements.txt
  pytest.ini
  run_eval.py
  compare.py
  data/pairs.jsonl
  judgelab/
    __init__.py
    data.py
    prompts.py
    judges.py
    metrics.py
  tests/test_metrics.py
  results/            (created by run_eval.py)
  .cache/             (created on the first API call)

Run these commands in order from the project root to reproduce every number in this article. Steps 1 to 12 need no API key. Step 13 needs at least one.

python -m judgelab.data                  # step 4: build 96 pairs
pytest -v                                # step 7: lock the kappa math
python run_eval.py --judge mock-clean    # steps 8-9: baseline
python run_eval.py --judge mock-first    # steps 8-9: position bias
python run_eval.py --judge mock-wordy    # step 10: verbosity bias
python compare.py                        # steps 11-12: gate (exit 1 on failure)
python run_eval.py --judge a             # step 13: a real judge

A last check before you trust a real judge: the harness should pass the mock tests, and your real-judge run should produce a result file with 96 rows and no exceptions. Then compare the cheapest real judge against mock-clean. If it scores lower, look at the disagreements before you blame the model. The gold label may be wrong.

From here, the natural extension is replacing the demo data with your own labeled pairs and pointing the gate at a CI pipeline. Teams that do this tend to find that the most valuable output is not the leaderboard of judges. It is the list of pairs where the judge and the experts disagree, since those pairs expose unclear rubrics before they reach users.

Frequently Asked Questions About LLM as a Judge

What is LLM as a judge?

It is the practice of asking a language model to grade another model’s output, usually by picking the better of two answers or scoring one answer against a rubric. It scales review far beyond what humans can read, but only if you first measure how closely the judge matches human decisions.

How many labeled pairs do I need?

Aim for at least 100 pairs as a floor and 200 or more for decisions that matter. The bootstrap interval in the report shows whether your sample is big enough: if two judges’ intervals overlap heavily, add pairs.

What is a good Cohen’s kappa for a judge?

There is no universal cutoff. This lab gates at 0.60 as a starting point for the demo. Use the agreement between two of your own human labelers as the realistic ceiling, and set the bar relative to the cost of a wrong verdict.

Which judge model should I use in October 2026?

Start with the cheapest current options, Claude Haiku 5.5 or GPT-6 Luna at $0.10 and $0.50 per million tokens, and measure them with this harness. Move up to GPT-6.1 Sol, Gemini 4 Argon or GPT-6 Astra only if the cheap judge fails the gate. No vendor publishes calibration numbers, so your data is the only reliable answer.

Does swapping answer order really matter?

Yes. With a position-biased judge, the final kappa can fall sharply and the flip rate can jump, while a forward-only evaluation still produces a kappa that looks tolerable. The swap exposes what a single pass hides.

Can I use the same model as both candidate and judge?

Avoid it. Judges can favor their own family’s style, and you cannot see that bias unless a second vendor’s judge reviews the same pairs. If you have no alternative, keep the judge out of the candidate set and audit disagreements by hand.

How often should I recalibrate?

Monthly, and after any model or prompt change. Vendors have shipped new models every few weeks lately, so both quality and cost assumptions expire quickly.