How to Build an LLM Eval Pipeline: 12 Steps, 90 Min [2026]

Every team shipping an LLM-powered feature eventually hits the same wall: the model works great in the demo, then quietly breaks in production after a prompt tweak, a model swap, or a routine dependency update. Without a repeatable way to test outputs, engineers end up eyeballing a handful of responses and shipping on vibes. That approach falls apart fast once a product has more than a few hundred users.

This tutorial walks through building a working LLM evaluation pipeline from scratch, using the same open-source tools AI engineering teams reach for in 2026: DeepEval, Ragas, and Promptfoo. By the end you’ll have a test suite that catches regressions before they reach users, a golden dataset you can version alongside your code, and a CI/CD gate that blocks bad prompt changes automatically. Budget about 90 minutes for the full build, including time to install dependencies and run your first evaluation batch.

None of this requires a data science background. If you can write a Pytest test or a YAML config, you already have the skills this tutorial assumes. What it does require is discipline: treating evaluation as a first-class part of your codebase rather than a script you run once before a demo and forget about.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

Why Your AI Product Needs an LLM Evaluation Pipeline

Model quality varies more than most teams expect, and it varies by task. On the Artificial Analysis Intelligence Index, Claude Opus 4.8 led the field for much of the year at a score of 61.4, ahead of GPT-5.5 at 60.2, Gemini 3.1 Pro at 57, and Grok 4.3 at 53. Claude Opus 4.8 took the top spot on May 28, 2026, becoming the first model to break above 60 on that index’s v4.0 scale — though by August 27, 2026, LLM Stats was already tracking a different pair of leaders on its own leaderboard, with GPT-5.6 Sol topping GPQA Diamond at 94.6% and Claude Fable 5 topping SWE-Bench Verified at 95.0%, a reminder of how quickly the top of these charts turns over. Category winners shift depending on what you’re testing: Gemini 3.1 Pro wins reasoning tasks outright, posting a 94.3% score on GPQA Diamond, while GPT-5.5 leads agentic terminal work at 82.7% on Terminal-Bench 2.0 as of May 14, 2026, and other models pull ahead elsewhere on coding.

That spread matters because a model that tests well on one benchmark can still fail your specific use case. MiniMax M2.5, released in February 2026, scored 80.2% on SWE-Bench Verified, landing within 0.6 points of Claude Opus 4.6 while pricing far below it — though by May 2026, OFox’s benchmark roundup had GPT-5.5 well ahead of that mark at 88.7% on SWE-bench Verified, DeepSeek V4 Pro matching MiniMax’s territory at 80.6% SWE-bench for a strikingly low $0.43/$0.87 per million input/output tokens, and Claude Opus 4.7 posting 64.3% on the harder SWE-bench Pro variant as of May 14, 2026, per Presenc.ai. OpenAI has reported that GPT-5.5 Instant produces 52.5% fewer hallucinated claims than its predecessor, GPT-5.3 Instant. None of these numbers tell you how a model performs on your prompts, your data, and your users’ edge cases. An evaluation pipeline is how you find out.

Treat evaluation as a layered system rather than a single check. Cheap deterministic tests run first and catch obvious breakage. Heuristic and statistical scoring catch semantic drift. LLM-as-a-judge evaluation, the most expensive layer, only runs on what survives the earlier stages. Build it in that order and the pipeline stays fast and affordable even as your test suite grows. Stanford’s 2026 AI Index report tracks the same underlying shift industry-wide: as more products ship an LLM at the core, testing and evaluation practices are becoming as standard as unit testing was for traditional software a decade ago. Teams that treat evaluation as optional tend to find out the hard way, usually from a support ticket or a public screenshot rather than a failing test.

Prerequisites: Tools, Versions, and Accounts You’ll Need

Gather these before starting so you’re not context-switching mid-tutorial:

• Python 3.9 or newer: DeepEval and Ragas are both Python-first frameworks.
• Node.js 18 or newer: required only if you’re following the Promptfoo steps.
• DeepEval, currently on the 2.2.x release line: the pytest-style evaluation framework this tutorial centers on.
• Ragas, on the 0.2.x release line: for RAG-specific metrics like faithfulness and context precision.
• Promptfoo, on the 0.92.x release line: for side-by-side model comparison and CLI-driven evals.
• An API key for at least one LLM provider (OpenAI, Anthropic, or a local model server) to act as the model under test and, separately, the judge model.
• A code editor and a terminal. A GitHub account if you plan to wire evaluations into CI/CD in Step 10.
• Roughly $5-15 in API credits for running the example evaluations in this tutorial. Judge-model calls are the main cost driver, covered in Step 11.

You don’t need a RAG application or an agent already built to follow along. The examples below use a small support-chatbot scenario, but every step applies whether you’re testing a chatbot, a summarizer, a coding assistant, or a classification pipeline.

Step 1: Define Your Evaluation Goals and Success Metrics

Before installing anything, write down what “good” means for your application. Skip this step and you’ll end up with a test suite that measures things nobody cares about. Split your goals into four buckets:

• Functional: does the output answer the question, follow the instruction, or return valid structured data?
• Quality: is it coherent, relevant, and appropriately concise?
• Safety: does it avoid hallucinating facts, leaking system prompts, or producing unsafe content?
• Operational: does it respond within your latency budget and stay within your per-request cost target?

Write two or three measurable statements per bucket. “The bot should be helpful” isn’t testable. “The bot must cite a source from the retrieved context in at least 90% of factual answers” is. Pin these down now, because every metric you choose in Step 5 should trace back to one of these statements.

A quick worked example: a support chatbot might land on “must not invent a refund window that contradicts the retrieved policy document” (functional), “responses should stay under 150 words unless the user asks for detail” (quality), “must never state a specific dollar refund amount without a matching source” (safety), and “median response time under 3 seconds” (operational). Each of those maps directly to a metric you’ll wire up later: faithfulness, a length heuristic, a custom G-Eval rubric, and a simple latency assertion, respectively. Teams that skip writing these down tend to reach for whatever metric is easiest to implement rather than the one that actually reflects what they care about.

Step 2: Install DeepEval and Verify Your Setup

DeepEval works like Pytest, but for LLM outputs instead of application code. It’s Apache 2.0 licensed, has passed 16,000 GitHub stars as of mid-2026, and according to Confident AI’s own published figures has run over 20 million evaluations with more than 400,000 monthly downloads. Install it and confirm the CLI responds:

pip install -U deepeval
deepeval --version

# Point DeepEval at your judge model provider
export OPENAI_API_KEY="your-api-key-here"

DeepEval needs an LLM to act as the judge for most of its metrics, separate from whatever model you’re testing. Using OpenAI’s API here is the path of least resistance, but DeepEval also supports Anthropic, local models through Ollama, and custom judge classes if you want to keep judging in-house.

Step 3: Write and Run Your First Test Case

Create a file called test_chatbot.py. This first test checks a single response for relevancy, using DeepEval’s built-in AnswerRelevancyMetric:

from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_refund_policy_answer():
    relevancy_metric = AnswerRelevancyMetric(threshold=0.7)

    test_case = LLMTestCase(
        input="What is your refund policy?",
        actual_output="We offer full refunds within 30 days of purchase, "
                       "no questions asked. Contact support to start a claim.",
        retrieval_context=[
            "Refunds are available within 30 days of the original purchase date."
        ]
    )

    assert_test(test_case, [relevancy_metric])

Run it with deepeval test run test_chatbot.py. On a passing run you’ll see output close to this:

Evaluating 1 test case(s) sequentially...

✓ test_refund_policy_answer
  - Answer Relevancy: 0.92 (threshold=0.7) PASSED
  - Reason: The response directly and completely answers the
    question about refund policy without extraneous detail.

Test Results: 1 passed, 0 failed, 0 errored (100.0%)
Run duration: 3.41s

That single test doesn’t prove much on its own. The value shows up once you have dozens of these running automatically every time the prompt, model, or retrieval logic changes.

Step 4: Build a Golden Dataset for Regression Testing

A golden dataset is a versioned, curated set of inputs paired with expected behavior. It’s what turns one-off tests into a repeatable regression suite. Aim for 15-20 examples to start, covering your most common use cases plus every failure mode you’ve already seen in production logs or support tickets.

from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase

dataset = EvaluationDataset()

dataset.add_test_case(LLMTestCase(
    input="Can I return an item after 45 days?",
    actual_output="No, our return window closes at 30 days from purchase.",
    retrieval_context=["Refunds are available within 30 days of purchase."],
    expected_output="No, returns are only accepted within 30 days."
))

dataset.add_test_case(LLMTestCase(
    input="Do you ship to Canada?",
    actual_output="Yes, we ship to Canada with a 5-7 business day delivery window.",
    retrieval_context=["International shipping is available to Canada and the UK."],
))

# Save it so it's versioned alongside your code
dataset.save_as(file_type="json", directory="./golden_dataset")

Commit this file to your repository. Treat changes to it the way you’d treat a database migration: reviewed, versioned, and never edited silently. A dataset that drifts without anyone noticing is one of the most common reasons evaluation pipelines quietly stop meaning anything, covered more in the pitfalls section below.

Step 5: Choose Metrics That Map to Your Goals

Metrics fall into a few families, and picking the wrong one for the job is a common source of noisy results. The table below covers the ones you’ll use most often.

Metric	What It Measures	Best For	Tool Support
Answer Relevancy	Does the output address the input question directly	Chatbots, Q&A systems	DeepEval, Ragas
Faithfulness	Is every claim in the output backed by retrieved context	RAG pipelines	Ragas, DeepEval
Context Precision / Recall	Is retrieved context relevant and complete	Retrieval quality tuning	Ragas
G-Eval	Custom, rubric-based scoring via chain-of-thought judging	Open-ended, subjective criteria	DeepEval
Hallucination	Does the output contain unsupported or fabricated claims	Factual accuracy checks	DeepEval
Exact Match / BLEU / ROUGE	N-gram or string overlap against a reference answer	Summarization, translation, structured extraction	DeepEval, custom scripts

G-Eval deserves a special mention because it’s become the default way teams handle criteria that don’t fit a formula, like “is this response appropriately empathetic” or “does this code follow our style guide.” The method comes from a widely cited NLG evaluation paper using GPT-4 with chain-of-thought reasoning, and it’s now the basis for custom rubric metrics in DeepEval and several competing frameworks. You write a plain-language rubric, and the judge model scores against it step by step instead of returning a single opaque number.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

empathy_metric = GEval(
    name="Support Tone",
    criteria="Determine whether the response acknowledges the customer's "
             "frustration before providing a solution, and avoids blaming "
             "the customer for the issue.",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT
    ],
    threshold=0.75
)

Step 6: Add RAG-Specific Evaluation with Ragas

If your application retrieves documents before generating an answer, DeepEval’s general-purpose metrics won’t tell you whether the retrieval step itself is working. That’s where Ragas comes in. It’s Apache 2.0 licensed, sits around 14,900 GitHub stars, and focuses specifically on scoring the retrieval-then-generation loop with metrics like faithfulness and context precision.

pip install ragas

from ragas import evaluate
from ragas.metrics import faithfulness, context_precision, context_recall
from datasets import Dataset

data = {
    "question": ["What is your refund policy?"],
    "answer": ["Refunds are available within 30 days of purchase."],
    "contexts": [["Our return policy allows refunds within 30 days of the "
                   "original purchase date, provided the item is unused."]],
    "ground_truth": ["Refunds are accepted within 30 days of purchase."]
}

results = evaluate(
    Dataset.from_dict(data),
    metrics=[faithfulness, context_precision, context_recall]
)

print(results)
# {'faithfulness': 1.0, 'context_precision': 0.95, 'context_recall': 1.0}

Run this after every change to your chunking strategy, embedding model, or retriever configuration. A faithfulness score that drops even slightly usually means your model has started answering from parametric memory instead of the retrieved documents, which is exactly the kind of silent drift that’s hard to catch by reading transcripts manually.

Step 7: Compare Models Side-by-Side with Promptfoo

DeepEval and Ragas are built for testing one system against a fixed bar. Promptfoo is built for the other common question: which model or prompt variant performs best. It’s MIT licensed, has around 24,000 GitHub stars, and per its own project description is used internally at both OpenAI and Anthropic for prompt testing. It runs from a declarative YAML config rather than Python code, which makes it easier to hand to a non-engineer on the team.

npm install -g promptfoo

Create promptfooconfig.yaml:

prompts:
  - "Answer the customer's question using only the provided context: {{context}}\n\nQuestion: {{question}}"

providers:
  - openai:gpt-4o-mini
  - anthropic:claude-opus-4-8
  - ollama:chat:llama3.1

tests:
  - vars:
      question: "Can I return an item after 45 days?"
      context: "Refunds are available within 30 days of purchase."
    assert:
      - type: contains
        value: "30 days"
      - type: llm-rubric
        value: "Response should NOT imply a 45-day return window exists"

Run promptfoo eval, then promptfoo view to open a local dashboard comparing every provider side-by-side on every test case. The CLI summary looks like this:

Evaluating 3 providers across 1 test case...

Provider                  Pass Rate   Avg Score   Avg Latency
openai:gpt-4o-mini        100%        0.94        1.2s
anthropic:claude-opus-4-8 100%        0.98        2.1s
ollama:chat:llama3.1      0%          0.41        0.8s

Done. View results: promptfoo view

This is the fastest way to answer “should we switch models” with data instead of a hunch, and it’s cheap enough to rerun every time a new model ships.

Step 8: Layer Cheap Checks Before Expensive LLM-as-Judge Calls

LLM-as-a-judge metrics are the most flexible tool in this pipeline and also the slowest and most expensive, since every test case triggers an extra API call to the judge model. Running G-Eval or a hallucination check on every single test case, every single CI run, adds up fast once your suite grows past a few hundred cases. Structure your checks in three tiers instead:

1. Deterministic checks first: valid JSON schema, required fields present, string length within bounds, regex pattern matches. These run in milliseconds and cost nothing.
2. Heuristic and embedding-based checks second: semantic similarity against a reference answer using a lightweight embedding model, keyword presence, format validation.
3. LLM-as-judge last: reserved for cases that pass the first two tiers, or for the subset of your suite that genuinely needs subjective judgment.

def layered_check(test_case):
    # Tier 1: deterministic
    if not test_case.actual_output.strip():
        return {"passed": False, "tier": "deterministic", "reason": "empty output"}

    # Tier 2: heuristic
    if len(test_case.actual_output) > 2000:
        return {"passed": False, "tier": "heuristic", "reason": "output too long"}

    # Tier 3: only reached if the cheap checks passed
    return run_llm_judge(test_case)

This ordering alone typically cuts judge-model API calls by more than half on a mature test suite, since a meaningful share of failures are caught by tier one and two before they’d ever reach the expensive step.

Step 9: Log Results and Catch Regressions Over Time

A single evaluation run tells you about right now. A logged history tells you whether things are getting better or worse. Save every run’s results to a structured file with metadata about the model version, prompt version, and timestamp, so you can plot trends instead of comparing isolated snapshots.

import json
from datetime import datetime, timezone

def log_run(results, model_version, prompt_version):
    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "model_version": model_version,
        "prompt_version": prompt_version,
        "pass_rate": results["pass_rate"],
        "avg_score": results["avg_score"],
        "total_cases": results["total_cases"],
    }
    with open("eval_history.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Tools like MLflow or Weights & Biases work well here if you already use them for model training, since they give you dashboards and diffing for free. A plain JSONL file works just as well for smaller teams and costs nothing to maintain.

What matters is what you do with the history, not just that it exists. Once you have a few weeks of runs logged, plot pass rate against model version and prompt version side by side. A slow downward drift across otherwise-unrelated commits usually points to dataset staleness rather than a real regression, since the world your model answers questions about keeps moving even when your code doesn’t. A sharp single-run drop almost always traces back to one specific change, and having the history makes that change easy to find instead of a guessing game across a week of commits.

Step 10: Wire Evaluations into Your CI/CD Pipeline

An evaluation suite that only runs when someone remembers to run it manually will eventually get skipped under deadline pressure. Wiring it into CI/CD makes it a gate instead of a suggestion. Here’s a minimal GitHub Actions workflow that runs DeepEval on every pull request and fails the build if the pass rate drops below your threshold:

name: LLM Evaluation

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/chatbot/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -U deepeval
      - name: Run evaluation suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_chatbot.py --min-success 0.9

Scope the trigger with paths so the eval job only runs when prompt or model-related files actually change, not on every unrelated commit. That keeps CI fast and keeps your API bill predictable.

Step 11: Set Cost and Latency Budgets for Your Eval Suite

Judge-model calls are the line item that surprises teams first, and the frontier-model price spread is wide enough to matter: BestAI’s Q2 2026 report puts GPT-5.5 (xhigh) at $4.35 per million tokens, versus $1.74 per million tokens for Gemini 3.1 Pro Preview and just $1.43 per million tokens for Qwen3.7 Max, roughly a 3x gap between the priciest and cheapest options scoring in the same range. As a rough illustration: if a single G-Eval call uses roughly 500-1,000 combined input and output tokens including the rubric and chain-of-thought reasoning, a 1,000-case evaluation run lands around 0.5-1 million tokens. Priced against Claude Opus 4.6’s published rate of $0.30 per million input tokens, that single run costs somewhere in the low single-digit dollars, though picking a pricier frontier judge like GPT-5.5 (xhigh) instead could push that same run several times higher. Run it on every PR across a busy repo and it adds up, especially if your judge model is a frontier model rather than a cheaper one reserved for this purpose.

Three practical guardrails keep this under control:

• Sample instead of running everything. Run your full golden dataset nightly, and a representative sample of 20-30 cases on every PR.
• Use a cheaper judge for high-volume tiers. Reserve your most capable model for the final LLM-as-judge tier only, after deterministic and heuristic checks have already filtered out the easy failures.
• Set a hard per-run token cap in your CI job so a misconfigured test loop can’t silently burn through your budget overnight.

Step 12: Review, Iterate, and Expand Coverage as You Scale

Treat your evaluation suite as a living part of the codebase, not a one-time setup task. Every production incident, every weird support ticket, and every case where a user’s report didn’t match what your tests predicted is a candidate for a new golden dataset entry. Revisit thresholds quarterly. A 0.7 relevancy threshold that made sense when you launched might be too lenient once you have six months of real traffic to learn from.

Schedule a recurring review, even informally, where someone on the team reads through a sample of production transcripts that scored well on automated metrics. This catches the gap between what your metrics measure and what your users actually experience, which no amount of tooling closes on its own.

LLM Evaluation Tools Compared

The tools used in this tutorial aren’t the only options, but they cover the most common needs. The field is bigger than any single tutorial can cover — BenchLM.ai’s May 2026 leaderboard alone tracks 228 LLMs across 186 benchmarks just to make cost-versus-quality tradeoffs legible. Here’s how the broader field of evaluation and observability tooling stacks up as of mid-2026:

Tool	License	GitHub Stars	Best For	Interface
DeepEval	Apache 2.0	~16,000+	General-purpose, pytest-style testing	Python + CLI
Ragas	Apache 2.0	~14,900	RAG-specific evaluation	Python
Promptfoo	MIT	~10,800	Model/prompt comparison, red-teaming	YAML + CLI
Arize Phoenix	Open source (paid tier from $50/mo)	~10,600	Trace-based observability + eval	Python/TS + UI
TruLens	MIT	~3,100	Feedback functions on chains and agents	Python + dashboard

Most teams end up running more than one of these rather than picking a single winner. DeepEval or Ragas for the automated test suite, Promptfoo for one-off model bake-offs, and an observability layer like Arize Phoenix or TruLens for watching production traffic once the feature ships. None of these tools are mutually exclusive, and the cost of running more than one is mostly engineering time rather than license fees, since the core of each project is open source.

How Evaluation Fits Into the Broader AI Engineering Workflow

It helps to place evaluation on a timeline instead of treating it as one isolated task. Before you write a single test, you’re doing prompt engineering and manual spot-checking. That phase is fine for a prototype, but it doesn’t scale past a handful of examples a human can hold in their head. Evaluation is what lets a team move from “I tried it a few times and it seemed fine” to “this change improved our faithfulness score by four points across 200 held-out cases.”

After evaluation comes observability, which watches the same system once real users are sending real traffic. The two disciplines feed each other. Production traces surface the edge cases your golden dataset missed, and those cases graduate into new test cases, which is exactly the canary-evaluation pattern covered in the advanced tips section below. Teams that skip straight from prototype to production without building this loop tend to end up debugging blind, guessing at root causes from user complaints instead of reading a clear failure report.

None of this needs to be built in a single sprint. Start with Steps 1 through 4 below, a handful of test cases and one metric, and add tooling as the gaps become obvious. A pipeline that covers 80% of your use cases and actually runs on every pull request beats an ambitious framework that never gets past a proof of concept.

Common Pitfalls That Sink LLM Evaluation Pipelines

Most failed evaluation pipelines don’t fail loudly. They quietly stop being useful while everyone assumes they’re still working. Watch for these:

• Testing only happy paths. A suite full of easy, well-formed questions will pass consistently and tell you nothing about how the system handles ambiguous, adversarial, or malformed input. Deliberately include edge cases pulled from real user behavior.
• Defaulting to LLM-as-judge for everything. It’s tempting because it’s flexible, but running it on every check inflates cost and slows CI without adding proportional signal. Reserve it for genuinely subjective criteria.
• Letting the golden dataset drift silently. If anyone can edit test cases without review, your pass rate can improve simply because the bar moved, not because the model got better. Version it and require review on changes, same as production code.
• Ignoring judge-model inconsistency. LLM judges don’t always return the same score for the same input on repeated runs. Run judge calls at temperature 0 and, for high-stakes metrics, average across two or three runs rather than trusting a single call.
• Confusing observability with evaluation. Logging every trace tells you what happened. It doesn’t tell you whether what happened was correct. Teams that stop at tracing often believe they have coverage they don’t actually have.
• Never gating CI/CD on the results. An evaluation suite that runs but doesn’t block a merge on failure becomes decorative within a few months. Set a real threshold and enforce it.

Troubleshooting Common LLM Evaluation Errors

These are the issues that come up most often once a pipeline moves from a first test case to a real, running suite.

• Same test passes and fails on repeated runs. Almost always caused by non-zero temperature on either the model under test or the judge model. Set temperature to 0 for judge calls, and if the model under test is inherently non-deterministic, average results across three runs before scoring pass/fail.
• Evaluation run hangs or times out on larger datasets. DeepEval and Ragas both support async execution. Switch to a_measure() methods or parallel test execution instead of looping through cases sequentially.
• 429 rate-limit errors mid-run. Reduce concurrency in your test runner config, or add exponential backoff around the judge-model client. This gets more common as your dataset grows past a few hundred cases.
• All metric scores cluster near the same value with no discrimination between good and bad outputs. Usually means your G-Eval rubric is too vague. Rewrite the criteria with concrete, checkable conditions instead of general adjectives like “helpful” or “accurate.”
• Ragas context precision returns None or NaN. Typically happens when a test case has an empty retrieved-context list. Filter these out before the eval run or handle them as automatic failures rather than letting them corrupt the aggregate score.
• Promptfoo config fails to parse. YAML is indentation-sensitive and the most common cause is mixing tabs and spaces, or misaligning list items under tests. Validate with a YAML linter before running promptfoo eval.
• CI job times out before the evaluation finishes. Increase the job timeout, but also check whether you’re running your full dataset on every commit when a sampled subset would do, per the budgeting advice in Step 11.
• Judge model consistently disagrees with human reviewers. Pull 20-30 cases where humans have already labeled a verdict, run them through your judge prompt, and calibrate the rubric until agreement is reasonably close before trusting the judge on the full suite.
• Pass rate drops sharply after a routine dependency update. Check whether an underlying model version changed silently, for instance if your provider auto-upgrades a model alias. Pin explicit model versions in provider configs rather than using “latest” aliases in production evaluation.

Advanced Tips for Scaling Your Evaluation Pipeline

Once the basics are solid, a few techniques help the pipeline hold up as your application and team grow:

• Use a multi-judge ensemble for high-stakes metrics. Run the same evaluation through two different judge models and flag disagreements for human review, rather than trusting a single model’s opinion on subjective criteria.
• Apply statistical significance testing between model versions instead of comparing raw pass-rate percentages. A jump from 91% to 93% on a 30-case sample often isn’t meaningful. A paired significance test tells you whether it’s real.
• Generate synthetic test cases to expand coverage. DeepEval and several competing tools include synthetic data generators that can expand a hand-written 20-case dataset into hundreds of variations covering paraphrases and edge cases you didn’t think to write manually.
• Run canary evaluations on sampled production traffic, not just your static golden dataset. Real user queries surface failure patterns a hand-curated dataset never will.
• Build a lightweight human-in-the-loop review queue for the cases where automated metrics disagree with each other or land close to the threshold. This is where most rubric refinement work happens over time.

Complete Working Project: A Minimal Eval Suite You Can Copy Today

Here’s the full project structure, combining everything above into a working repository layout:

llm-eval-starter/
├── golden_dataset/
│   └── support_cases.json
├── test_chatbot.py
├── test_rag_faithfulness.py
├── promptfooconfig.yaml
├── requirements.txt
├── eval_history.jsonl
└── .github/
    └── workflows/
        └── llm-eval.yml

requirements.txt:

deepeval>=2.2.0
ragas>=0.2.0
datasets
pytest

test_chatbot.py: pulls in the golden dataset built in Step 4 and runs the full metric set from Step 5 against every case:

import pytest
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

dataset = EvaluationDataset()
dataset.add_test_cases_from_json_file(
    file_path="./golden_dataset/support_cases.json",
    input_key_name="input",
    actual_output_key_name="actual_output",
    retrieval_context_key_name="retrieval_context",
)

relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.8)

@pytest.mark.parametrize("test_case", dataset.test_cases)
def test_support_bot(test_case):
    assert_test(test_case, [relevancy, faithfulness])

Run the whole suite with deepeval test run test_chatbot.py -n 4 to parallelize across four workers, then check it into version control alongside your application code. From here, Step 10’s GitHub Actions workflow turns this into an automatic gate on every pull request that touches your prompts or model configuration.

Treat this starter layout as a floor, not a ceiling. Swap in your own domain (support, coding, summarization) by replacing support_cases.json and adjusting the metric thresholds to match what you defined back in Step 1. Add test_rag_faithfulness.py once you have a retrieval step to test, following the Ragas pattern from Step 6. The point of a starter project is that every piece is small enough to understand in one sitting, and swapping any single piece doesn’t require rewriting the rest.

Frequently Asked Questions

What is an LLM evaluation pipeline?

It’s an automated system that scores an AI model’s outputs against defined criteria, ranging from deterministic checks like format validation to LLM-as-a-judge scoring for subjective qualities like tone or helpfulness. The goal is catching regressions before they reach users, the same way a unit test suite catches bugs in traditional software.

Do I need an evaluation pipeline if I’m only using one model?

Yes. Even with a single fixed model, your prompts, retrieval logic, and application code will change over time, and any of those changes can silently degrade output quality. Evaluation isn’t primarily about comparing models, it’s about catching regressions in your own system.

What’s the difference between LLM evaluation and LLM observability?

Observability tools like tracing dashboards show you what happened: the inputs, outputs, latency, and token usage for every request. Evaluation goes a step further and scores whether what happened was actually correct or good. You typically want both, with observability watching production and evaluation gating changes before they ship.

Which tool should I start with: DeepEval, Ragas, or Promptfoo?

Start with DeepEval if you want a general-purpose, pytest-style suite integrated into your existing test infrastructure. Add Ragas once you have a RAG pipeline whose retrieval quality you need to isolate from generation quality. Reach for Promptfoo when the question is model or prompt selection rather than ongoing regression testing.

How much does it cost to run LLM-as-judge evaluations at scale?

It depends heavily on judge model choice, rubric length, and how often you run the full suite versus a sample. As a rough illustration, a 1,000-case run at roughly 500-1,000 tokens per judge call lands in the low single-digit dollars against a moderately priced model. Full runs on every commit across a busy repo can add up quickly, which is why sampling and tiered checks (Steps 8 and 11) matter.

Can I evaluate an LLM without writing code?

Promptfoo comes closest, since its configuration is declarative YAML rather than Python, and its dashboard gives non-engineers a way to review results. Most other frameworks in this space, including DeepEval and Ragas, assume at least basic Python familiarity.

How many test cases do I need for a reliable golden dataset?

Start with 15-20 covering your core use cases and known failure modes, which is enough to catch obvious regressions. Grow it over time by adding real production cases that revealed a gap, aiming for coverage of every distinct scenario your application handles rather than a specific target number.

How do I stop evaluation results from being flaky or inconsistent?

Set judge-model temperature to 0, average scores across two or three runs for any metric feeding a CI gate, and calibrate your rubrics against human-labeled examples before trusting them at scale. Flaky evaluations are almost always a temperature or rubric-clarity problem, not a fundamental limitation of the approach.