Skip to content

Support OpenTelemetry consistent tracestate sampling - #12397

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 39 commits into
masterfrom
milan.garnier/ot.th
Sep 29, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 39 commits into
masterfrom
milan.garnier/ot.th

Conversation

@MilanGarnier

@MilanGarnier MilanGarnier commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Implements OpenTelemetry consistent probability sampling with the ot
tracestate member.

  • Records locally made probability decisions as rv and th for agent-rate
    and rule-based sampling (and forwards them is they were extracted and valid).
  • Keeps rate-limiter outcomes and manual overrides from creating contradictory
    probability state.
  • Validates and normalizes inbound ot fields, rebuilding W3C tracestate
    within its member and size limits while preserving unmanaged vendor members.
  • Publishes sampling priority, Datadog sampling tags, and OTel sampling state
    together so W3C injection and OTLP JSON/protobuf export use one decision.
  • Clears inherited managed OTel state when Org Guard resamples a trace.

Adds coverage for parsing and normalization, W3C extraction and
reinjection, sampling decisions and overrides, Org Guard, and OTLP export.

Motivation

OpenTelemetry consistent probability sampling lets tracers make compatible
sampling decisions using a shared random value and threshold. The Java tracer
needs to retain compatible inbound state and publish locally made probability
decisions without conflicting with Datadog sampling behavior.

Additional Notes

Contributor Checklist

  • Format the title according to the contribution guidelines.
  • Assign the required type: and comp: labels.

Jira ticket: APMAPI-2171

@MilanGarnier MilanGarnier added type: feature Enhancements and improvements comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Sep 3, 2026
@MilanGarnier MilanGarnier changed the title Support OpenTelemetry tracestate sampling feat(sampling): Support for OpenTelemetry consistent tracestate sampling Sep 3, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 89.22%
• Overall Coverage: 59.24% (+0.14%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c2f6051 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.79 s 14.70 s [-0.3%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent 13.67 s 13.63 s [-0.5%; +1.1%] (no difference)
startup:petclinic:appsec:Agent 17.59 s 17.36 s [+0.4%; +2.3%] (maybe worse)
startup:petclinic:iast:Agent 16.72 s 17.61 s [-9.2%; -0.8%] (maybe better)
startup:petclinic:profiling:Agent 17.25 s 17.25 s [-1.2%; +1.3%] (no difference)
startup:petclinic:sca:Agent 17.70 s 17.53 s [-0.1%; +2.0%] (no difference)
startup:petclinic:tracing:Agent 16.69 s 16.85 s [-1.9%; +0.0%] (no difference)

Commit: c2f6051e · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@MilanGarnier
MilanGarnier changed the base branch from master to milan.garnier/preserve-deterministic-sampler-rate September 3, 2026 13:59
Base automatically changed from milan.garnier/preserve-deterministic-sampler-rate to master September 3, 2026 18:45
@MilanGarnier MilanGarnier mentioned this pull request Sep 4, 2026
3 tasks done
@MilanGarnier
MilanGarnier changed the base branch from master to milan.garnier/otel-tracestate-refactor September 4, 2026 09:30
@MilanGarnier
MilanGarnier force-pushed the milan.garnier/ot.th branch 5 times, most recently from 7a1ddea to a0176c9 Compare September 8, 2026 14:26
@MilanGarnier

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@MilanGarnier
MilanGarnier marked this pull request as ready for review September 10, 2026 09:02
@MilanGarnier
MilanGarnier requested review from a team as code owners September 10, 2026 09:02
@MilanGarnier
MilanGarnier requested review from mhlidd and removed request for a team September 10, 2026 09:02
@MilanGarnier
MilanGarnier force-pushed the milan.garnier/otel-tracestate-refactor branch from db321c0 to 1c02400 Compare September 10, 2026 09:02
chatgpt-codex-connector[bot]

This comment was marked as resolved.

datadog-datadog-prod-us1-2[bot]

This comment was marked as resolved.

@mcculls
mcculls force-pushed the milan.garnier/ot.th branch from ea1fb86 to cdde406 Compare September 28, 2026 22:24

@vandonr vandonr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok for IDM

@MilanGarnier
MilanGarnier added this pull request to the merge queue Sep 29, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-09-29 09:55:33 UTC ℹ️ Start processing command /merge


2026-09-29 09:55:37 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-09-29 11:08:30 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 29, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 79a1710 into master Sep 29, 2026
606 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the milan.garnier/ot.th branch September 29, 2026 11:08
@github-actions github-actions Bot added this to the 1.67.0 milestone Sep 29, 2026
@ValentinZakharov ValentinZakharov changed the title feat(sampling): Support for OpenTelemetry consistent tracestate sampling Support OpenTelemetry consistent tracestate sampling Oct 5, 2026
dougqh added a commit that referenced this pull request Oct 21, 2026
…hape cost

SamplingDecisionBenchmark gains arms that follow a probability decision with
the Datadog header, the W3C tracestate header and the OTLP tracestate, so the
deferred OtelTraceState is measured where it's built or skipped.

ServiceShapePropagationBenchmark measures head, leaf and intermediate
services with a default-configured tracer. It only uses API unchanged since
v1.66.0, so it can be built onto release tags to put a cost on #12397.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dougqh added a commit that referenced this pull request Oct 22, 2026
…ported (#12982)

* Defer building the OpenTelemetry tracestate for fresh probability decisions

Every root probability decision built an OtelTraceState inside the sampling
lock, though only W3C injection and OTLP export read it. Install a
ProbabilitySamplingState that keeps the trace id bits and rate, and builds
the ot member on first read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Clarify why lazy otel state readers agree on one instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add benchmarks for the lazy otel state's inject, export and service-shape cost

SamplingDecisionBenchmark gains arms that follow a probability decision with
the Datadog header, the W3C tracestate header and the OTLP tracestate, so the
deferred OtelTraceState is measured where it's built or skipped.

ServiceShapePropagationBenchmark measures head, leaf and intermediate
services with a default-configured tracer. It only uses API unchanged since
v1.66.0, so it can be built onto release tags to put a cost on #12397.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rename ServiceShapePropagationBenchmark to PropagationScenariosBenchmark

Describe it as a scenario benchmark: each arm is a whole request driven
through the tracer's public API, rather than one method in isolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test that racing readers get one OpenTelemetry state instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Oct 22, 2026
…etry tracestate (#12981)

Precompute decision maker tag values per sampling mechanism

Every keep decision built "-" + samplingMechanism and then validated, hashed
and looked it up in TagValue's cache. The mechanisms are a small fixed set, so
build their TagValues once and index by mechanism, falling back to
TagValue.from for anything beyond the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add a sampling decision benchmark

Measures the per-trace propagation-tag bookkeeping done on a root sampling
decision, both in a bare start/sample/finish span and on fresh
PropagationTags with no span around it, for the default, rule and manual
keep paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Defer building the OpenTelemetry tracestate until it's injected or exported (#12982)

* Defer building the OpenTelemetry tracestate for fresh probability decisions

Every root probability decision built an OtelTraceState inside the sampling
lock, though only W3C injection and OTLP export read it. Install a
ProbabilitySamplingState that keeps the trace id bits and rate, and builds
the ot member on first read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Clarify why lazy otel state readers agree on one instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add benchmarks for the lazy otel state's inject, export and service-shape cost

SamplingDecisionBenchmark gains arms that follow a probability decision with
the Datadog header, the W3C tracestate header and the OTLP tracestate, so the
deferred OtelTraceState is measured where it's built or skipped.

ServiceShapePropagationBenchmark measures head, leaf and intermediate
services with a default-configured tracer. It only uses API unchanged since
v1.66.0, so it can be built onto release tags to put a cost on #12397.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rename ServiceShapePropagationBenchmark to PropagationScenariosBenchmark

Describe it as a scenario benchmark: each arm is a whole request driven
through the tracer's public API, rather than one method in isolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test that racing readers get one OpenTelemetry state instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants