# Semantic Conventions for GenAI Evaluation organization into Experiments and Test Cases

RFC: Semantic Conventions for Evaluation Experiments and Test Cases

Tracking Issue: #3398
Status: Draft
Component: test, gen-ai

Problem

Building quality agents is hard. Evaluation is rarely binary pass/fail; test cases are subjective, scores are nuanced, and developers need tight feedback loops to debug agent quality and iterate. The gen_ai.evaluation.result event (PR open-telemetry/semantic-conventions#2563) captures individual evaluation scores on spans, but there's no standard way to:

Group evaluations into experiment runs
Identify test cases with machine-readable IDs
Correlate all spans belonging to a single evaluation execution
Track evaluations online and offline to close the feedback loop

Without this, observability platforms can't aggregate scores by experiment, compare runs side-by-side, or build experiment tracking UIs. Debugging multi-agent evaluation workflows also becomes significantly harder without standardized conventions. Each framework invents its own schema, fragmenting the ecosystem.

Evolution of This Proposal

The original issue proposed 13 new gen_ai.experiment.* / gen_ai.test_case.* attributes at the resource level. After maintainer feedback pointing to existing test.* and feature_flag.* conventions, the proposal was revised:

graph LR
    A["Original: 13 new gen_ai.* attrs<br/>Resource-level<br/>GenAI-only"] -->|community feedback| B["Revised: 2 new test.* attrs<br/>Span-level<br/>Generic + reusable"]

What Changed

Original Proposal	Revised	Why
`gen_ai.experiment.id`	`test.suite.run.id` (new)	Reuse `test.*` namespace
`gen_ai.experiment.name`	`test.suite.name` (exists)	Already in registry
`gen_ai.test_case.id`	`test.case.id` (new)	Reuse `test.*` namespace
`gen_ai.test_case.name`	`test.case.name` (exists)	Already in registry
`gen_ai.experiment.version/tags`	Dropped	Keep lean
`gen_ai.test_case.metadata` (JSON)	Dropped	No unstructured blobs
`gen_ai.test_case.tags`	Dropped	Keep lean, add later if needed
`gen_ai.test_case.input/expected_output`	Deferred	PII concerns, needs separate design
`gen_ai.dataset.*`	Deferred	Separate follow-up issue
Resource-level placement	Span-level (parent span)	Process may run multiple experiments

Net result: 2 new attributes + 4 existing reused. No new events, spans, or model groups.

Proposal

New Attributes

Added to model/test/registry.yaml:

Attribute	Type	Stability	Description
`test.suite.run.id`	`string`	`development`	Unique identifier for a specific test suite run. Scope: globally unique (e.g., UUID, timestamp-based ID).
`test.case.id`	`string`	`development`	Machine-readable unique identifier for a test case. Scope: unique within a `test.suite.run.id`. Distinct from the human-readable `test.case.name`.

These are generic, useful for GenAI evaluation, CI/CD test reporting, and ML evaluation alike.

Why Span-Level, Not Resource-Level

graph TD
    subgraph "❌ Resource-level (original)"
        R["Resource<br/>experiment.id = exp_001<br/>⚠️ immutable for SDK lifetime"]
        R --> S1["invoke_agent"]
        R --> S2["invoke_agent"]
    end

    subgraph "✅ Span-level (revised)"
        P1["test_suite_run span<br/>test.suite.run.id = run_001"]
        P2["test_suite_run span<br/>test.suite.run.id = run_002"]
        P1 --> C1["invoke_agent"]
        P1 --> C2["invoke_agent"]
        P2 --> C3["invoke_agent"]
    end

Resource attributes are immutable per SDK instance, so you can't run multiple experiments sequentially
Span-level enables natural parent-child correlation via trace context
Consistent with how gen_ai.conversation.id already works on spans

Trace Structure

The test_suite_run span shown below is an application-created span with no normative span kind or naming requirements. It is simply the span the application creates to represent the experiment run. The only convention is that it carries test.suite.* attributes.

Span: test_suite_run
  test.suite.name: "GPT-4 vs Claude Accuracy"
  test.suite.run.id: "run_20250205_001"
  test.suite.run.status: "success"
  │
  ├── Span: invoke_agent [SpanKind: CLIENT]
  │     test.case.id: "case_001"
  │     test.case.name: "greeting_response"
  │     test.case.result.status: "pass"
  │     gen_ai.operation.name: "chat"
  │     gen_ai.request.model: "gpt-4"
  │     │
  │     ├── Event: gen_ai.evaluation.result
  │     │     gen_ai.evaluation.name: "accuracy"
  │     │     gen_ai.evaluation.score.value: 0.95
  │     │
  │     └── Event: gen_ai.evaluation.result
  │           gen_ai.evaluation.name: "tone"
  │           gen_ai.evaluation.score.value: 0.88
  │
  ├── Span: invoke_agent [SpanKind: CLIENT]
  │     test.case.id: "case_002"
  │     test.case.name: "summarization_quality"
  │     test.case.result.status: "pass"
  │     gen_ai.operation.name: "chat"
  │     gen_ai.request.model: "gpt-4"
  │     │
  │     └── Event: gen_ai.evaluation.result
  │           gen_ai.evaluation.name: "accuracy"
  │           gen_ai.evaluation.score.value: 0.87
  │
  └── Span: invoke_agent [SpanKind: CLIENT]
        test.case.id: "case_003"
        test.case.name: "code_generation"
        test.case.result.status: "pass"
        │
        └── Event: gen_ai.evaluation.result
              gen_ai.evaluation.name: "correctness"
              gen_ai.evaluation.score.value: 1.0

Attribute Summary

graph TD
    P["<b>Parent Span: test_suite_run</b><br/>test.suite.name<br/>test.suite.run.id ← NEW<br/>test.suite.run.status"]
    C["<b>Child Spans: invoke_agent</b><br/>test.case.id ← NEW<br/>test.case.name<br/>test.case.result.status<br/>gen_ai.operation.name<br/>gen_ai.request.model"]
    E["<b>Events: gen_ai.evaluation.result</b><br/>gen_ai.evaluation.name<br/>gen_ai.evaluation.score.value<br/>gen_ai.evaluation.score.label"]

    P --> C
    C -.- E

What This Enables

Experiments
├── GPT-4 vs Claude Accuracy (Feb 5) - 50 cases, 92% pass
│   ├── accuracy: avg 0.95
│   ├── relevance: avg 0.91
│   └── [View details →]
├── RAG Retrieval Test (Feb 4) - 120 cases, 87% pass
└── Tool Calling Test (Feb 3) - 30 cases, 95% pass

Compare: GPT-4 vs Claude-3
┌────────────────┬─────────┬─────────┐
│ Metric         │ GPT-4   │ Claude  │
├────────────────┼─────────┼─────────┤
│ Pass rate      │ 85%     │ 92% ↑   │
│ Avg accuracy   │ 0.82    │ 0.91 ↑  │
│ Avg latency    │ 2.3s    │ 1.8s ↓  │
└────────────────┴─────────┴─────────┘

Instrumentation Flow

The app sets test.* attributes on spans as it iterates through the dataset. Evaluators inherit this context from the trace. No attribute copying needed.

Pattern 1: Inline Evaluation

Evaluator runs in the same process, attaches events directly to the span that already carries test.case.id.

┌─ eval-runner process ─────────────────────────────────────────┐
│                                                               │
│  Span: test_suite_run                                         │
│    test.suite.run.id: "run_001"                               │
│    │                                                          │
│    └── Span: invoke_agent          ← app sets test.case.id    │
│          test.case.id: "case_001"                             │
│          test.case.name: "greeting"                           │
│          │                                                    │
│          │  1. app calls agent, gets response                 │
│          │  2. evaluator scores response                      │
│          │  3. evaluator attaches event to SAME span          │
│          │                                                    │
│          └── Event: gen_ai.evaluation.result  ← inherits ctx  │
│                gen_ai.evaluation.name: "accuracy"             │
│                gen_ai.evaluation.score.value: 0.95            │
│                                                               │
└───────────────────────────────────────────────────────────────┘

Pattern 2: Post-Hoc Evaluation

Evaluator runs asynchronously (different process or later in time). It creates child spans or uses span links to reference the original trace. Note: in this pattern, the evaluator must independently know the test.case.id (e.g., from a database or exported span data). Trace context propagation does not apply across process boundaries in async scenarios.

┌─ app process ──────────────────────┐
│                                    │
│  Span: test_suite_run              │
│    test.suite.run.id: "run_001"    │
│    │                               │
│    └── Span: invoke_agent          │
│          test.case.id: "case_001"  │
│          span_id: "abc123"  ───────┼──┐
│                                    │  │
└────────────────────────────────────┘  │
                                        │  span link
┌─ evaluator process ───────────────┐   │
│                                   │   │
│  Span: evaluate                   │   │
│    links: ["abc123"]  ←───────────┼───┘
│    test.case.id: "case_001"       │
│    │                              │
│    └── Event: gen_ai.eval.result  │
│          gen_ai.evaluation.name:  │
│            "accuracy"             │
│          gen_ai.evaluation.       │
│            score.value: 0.95      │
│                                   │
└───────────────────────────────────┘

Pattern 3: Session/Trace-Level Evaluation

Evaluator scores an entire conversation. It creates a span under the test_suite_run parent, inheriting test.suite.run.id from the trace.

Span: test_suite_run
  test.suite.run.id: "run_001"
  │
  ├── Span: invoke_agent (case_001)     ← agent call
  │     test.case.id: "case_001"
  │
  ├── Span: invoke_agent (case_002)     ← agent call
  │     test.case.id: "case_002"
  │
  └── Span: evaluate_session            ← evaluator scores whole session
        test.suite.run.id: "run_001"    ← inherited from parent
        │
        └── Event: gen_ai.evaluation.result
              gen_ai.evaluation.name: "overall_coherence"
              gen_ai.evaluation.score.value: 0.91

The key point: because experiment context is on spans, evaluators just need to be part of the same trace (as children or via links). The test.* attributes propagate naturally through the trace tree.

CI/CD Integration

These attributes are not GenAI-specific. The same test.* conventions apply to CI/CD pipelines running evaluation suites as part of deployment gates or regression checks.

┌─ CI Pipeline ─────────────────────────────────────────────┐
│                                                           │
│  Step: Run evaluation suite                               │
│                                                           │
│  Span: test_suite_run                                     │
│    test.suite.name: "pre-deploy-eval"                     │
│    test.suite.run.id: "ci-run-4521"                       │
│    │                                                      │
│    ├── Span: invoke_agent                                 │
│    │     test.case.id: "case_001"                         │
│    │     test.case.result.status: "pass"                  │
│    │     └── Event: gen_ai.evaluation.result              │
│    │                                                      │
│    ├── Span: invoke_agent                                 │
│    │     test.case.id: "case_002"                         │
│    │     test.case.result.status: "fail"                  │
│    │     └── Event: gen_ai.evaluation.result              │
│    │                                                      │
│    test.suite.run.status: "failure"                        │
│                                                           │
│  Step: Gate decision                                      │
│    Query: test.suite.run.status == "failure" → block      │
│                                                           │
└───────────────────────────────────────────────────────────┘

Because test.suite.run.id correlates all spans in a run, CI systems can:

Query evaluation results by pipeline run
Compare scores across deployments (run N vs run N-1)
Gate deployments on evaluation thresholds
Track regression trends over time using test.suite.name as the stable key

This works the same way for ML model evaluation pipelines, not just GenAI agents.

FAQ

Does this wrap multi-turn conversations into a single span?

No. test.* attributes are additive metadata on existing spans. For multi-turn conversations, each turn remains its own span; test.case.id provides correlation without structural changes. Group turns by gen_ai.conversation.id as usual.

Span: test_suite_run
  test.suite.run.id: "run_001"
  │
  └── Span: test_case (multi-turn conversation)
        test.case.id: "case_001"
        gen_ai.conversation.id: "conv_abc"
        │
        ├── Span: invoke_agent (turn 1)
        ├── Span: invoke_agent (turn 2)
        └── Span: invoke_agent (turn 3)
              └── Event: gen_ai.evaluation.result

What about backends that don't support event attributes?

The test.* attributes (test.suite.run.id, test.case.id, etc.) are all span attributes, so they work everywhere. The evaluation scores use the already-merged gen_ai.evaluation.result event convention. If a backend doesn't support event attributes well, instrumentations can fall back to recording evaluation data as span attributes instead, since the spec allows both. The core experiment/test case context is always on spans regardless.

How do evaluators inherit experiment context from the app?

The app sets test.suite.run.id and test.case.id on spans as it iterates through the dataset. Evaluators don't copy these attributes; they attach gen_ai.evaluation.result events to the same spans (or child spans) that already carry the test context. The attributes propagate naturally through the trace tree.

This works at session, trace, and span level:

┌─ App sets test context ───────────────────────────────┐
│                                                       │
│  Span: test_suite_run                                 │
│    test.suite.run.id: "run_001"                       │
│    test.suite.name: "accuracy_eval"                   │
│    │                                                  │
│    └── Span: invoke_agent                             │
│          test.case.id: "case_001"                     │
│          test.case.name: "greeting"                   │
│          gen_ai.conversation.id: "conv_abc"           │
│          │                                            │
│          │  ← evaluator attaches to this span         │
│          │     no attribute copying needed             │
│          │                                            │
│          └── Event: gen_ai.evaluation.result          │
│                gen_ai.evaluation.name: "relevance"    │
│                gen_ai.evaluation.score.value: 0.95    │
│                                                       │
└───────────────────────────────────────────────────────┘

For async/post-hoc evaluation, the evaluator creates spans linked to the original via span links, referencing the same test.case.id. See the "Instrumentation Flow" section above for detailed patterns.

Why not use baggage for this?

Baggage is for propagation context, not analytical attributes. Values in baggage are not automatically recorded as span attributes, so backends can't query or aggregate on them. test.* span attributes are directly queryable, indexable, and visible in trace UIs without extra processing.

Closing as duplicate of #79

Navigation Menu

Repository navigation

# Semantic Conventions for GenAI Evaluation organization into Experiments and Test Cases #80

RFC: Semantic Conventions for Evaluation Experiments and Test Cases

Problem

Evolution of This Proposal

What Changed

Proposal

New Attributes

Why Span-Level, Not Resource-Level

Trace Structure

Attribute Summary

What This Enables

Instrumentation Flow

Pattern 1: Inline Evaluation

Pattern 2: Post-Hoc Evaluation

Pattern 3: Session/Trace-Level Evaluation

CI/CD Integration

FAQ

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

# Semantic Conventions for GenAI Evaluation organization into Experiments and Test Cases #80

Description

RFC: Semantic Conventions for Evaluation Experiments and Test Cases

Problem

Evolution of This Proposal

What Changed

Proposal

New Attributes

Why Span-Level, Not Resource-Level

Trace Structure

Attribute Summary

What This Enables

Instrumentation Flow

Pattern 1: Inline Evaluation

Pattern 2: Post-Hoc Evaluation

Pattern 3: Session/Trace-Level Evaluation

CI/CD Integration

FAQ

Activity

trask commented on May 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions