Every observability vendor used to ship its own proprietary agent, and switching from Datadog to Honeycomb meant re-instrumenting every service from scratch. That changed once OpenTelemetry reached CNCF Graduated status on On May 21, 2026, OpenTelemetry’s CNCF graduation was announced, cementing its position as the default instrumentation layer for traces, metrics, and logs across nearly every major backend on the market, with CNCF listing May 11, 2026 as the date it moved to the Graduated maturity level. This tutorial walks through installing the OpenTelemetry Collector, instrumenting real applications in Python and Node.js, and routing that telemetry into Grafana Tempo, Prometheus, and a SaaS backend, all without locking your code to a single vendor.
By the end you’ll have a working Collector pipeline receiving OTLP data over gRPC and HTTP, two instrumented sample services exporting traces and metrics, and a dual-export setup sending the same telemetry to an open source backend and a commercial one simultaneously. Total setup time runs about 90 minutes, most of it spent watching your first traces land in a dashboard rather than fighting configuration files.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Is OpenTelemetry and Why It Replaced Vendor Agents
OpenTelemetry (often shortened to OTel) is a CNCF observability framework that provides one set of APIs, SDKs, and a Collector service for capturing traces, metrics, and logs. Instead of installing a separate proprietary agent for every backend you might want to try, you instrument your code once against the OpenTelemetry API and route the resulting data wherever you like: Jaeger, Grafana Tempo, Prometheus, Honeycomb, Datadog, or half a dozen destinations at once.
The project’s graduation to CNCF’s top maturity tier in May 2026 wasn’t a formality. At the time of graduation, OpenTelemetry counted more than 26,000 contributors and roughly 5,100 contributing companies, making it the second-highest-velocity CNCF project after Kubernetes itself — and the traffic backs that up: opentelemetry.io logged 13 million page views across 5 million sessions between December 2024 and November 2025, according to CNCF/OpenTelemetry usage data reported in January 2026. The CNCF’s own materials describe the milestone as “solidifying status as the de facto observability standard,” and the ecosystem backs that up too: nearly every major observability vendor now accepts the OpenTelemetry Protocol (OTLP) natively, whether that’s an open source stack like Tempo and Prometheus or a commercial platform like Datadog and Honeycomb.
The practical case for adopting OpenTelemetry in 2026 comes down to three things. First, vendor neutrality: your instrumentation code doesn’t change even if your company switches observability vendors next year. Second, a single pipeline: the Collector can receive, transform, sample, and route telemetry to multiple destinations from one place, so you’re not running five different agents on every host. Third, standardization: semantic conventions define consistent attribute names (service.name, http.request.method, db.system) across every language and framework, so dashboards and alerts built against one service work the same way against the next.
Traces, Metrics, and Logs: The Three Signals
OpenTelemetry organizes telemetry into three core signals, and understanding the maturity of each matters before you build a production pipeline around them. Traces track a single request as it moves through multiple services, showing you exactly where latency accumulates. Metrics capture aggregated numeric measurements over time, like request counts or queue depth. Logs record discrete timestamped events, the same role they’ve always played, but now correlated with trace and span IDs so you can jump from a slow trace straight to the log lines that explain it.
As of August 2026, traces and metrics are stable across every major language SDK, meaning their APIs won’t introduce breaking changes. Logs support is further along than it was a year ago but still catching up in a few languages, and Go instrumentation specifically is mid-transition: the Go compile-time instrumentation project shipped v0.1.0 in December 2025, v0.2.0 in February 2026, and v0.3.0 in March 2026, with Datadog’s April 2026 OTel News roundup tracking it toward a v1.0 release targeted for June 1, 2026. A fourth signal, continuous profiling, entered public alpha on March 26, 2026, with an OTLP Profiles data model, but it’s not yet something to build production alerting around. This tutorial focuses on traces and metrics, the two signals ready for real production use today.
OpenTelemetry Adoption in 2026: Why It Matters Now
The adoption numbers behind OpenTelemetry explain why so many teams are migrating off vendor-specific agents this year. The CNCF’s 2025 Annual Survey, fielded in September 2025 and published in January 2026, found 49% of respondents already running OpenTelemetry in production and another 26% actively evaluating it, figures gathered before the May graduation announcement pushed adoption even further. An ecosystem atlas tracking the Collector core repository alone counted roughly 7,159 stars, 2,116 forks, and about 607 contributors as of late June 2026, on top of the more than 26,000 contributors spread across the wider OpenTelemetry project.
AWS’s own observability team logged more than 40 launches across CloudWatch, X-Ray, Amazon Managed Grafana, and Amazon Managed Service for Prometheus in just the first five months of 2026, and explicitly told customers to “start your OpenTelemetry migration now” rather than wait. That’s a notable shift for a company that spent years pushing its own proprietary X-Ray SDK. Regional survey data tells the same story: the OpenTelemetry Japanese Community Survey, current as of April 2026, found 61.47% of respondents already running OpenTelemetry in production and another 25.69% evaluating it, both well above the global figures reported just a few months earlier. When the cloud vendors that used to compete on proprietary agents start recommending the vendor-neutral standard instead, it’s a reasonable signal that OTLP has become the safer long-term bet for any team instrumenting new services in 2026.
Prerequisites and Requirements
You don’t need a Kubernetes cluster to follow this tutorial, though it works there too. A single machine with Docker installed and one of the two sample applications is enough to see the full pipeline working end to end. If you’re planning to run the Collector inside an existing Kubernetes cluster, the Helm-based deployment in Step 10 covers that path as well.
| Requirement | Minimum Version | Notes |
|---|---|---|
| Docker Engine | 24.0+ | Used to run the Collector and backend containers locally |
| Docker Compose | v2.20+ | Orchestrates the Collector, Tempo, and Prometheus containers together |
| OpenTelemetry Collector | v0.158.0 | Core or Contrib distribution; this tutorial uses Contrib |
| Python | 3.10+ | For the Python auto-instrumentation example |
| Node.js | 20 LTS+ | For the Node.js auto-instrumentation example |
| Helm (optional) | 3.14+ | Only needed for the Kubernetes deployment path |
| kubectl (optional) | 1.29+ | Only needed for the Kubernetes deployment path |
| Free disk space | 2 GB+ | Collector images, Tempo/Prometheus storage, sample app dependencies |
Version pinning matters more with OpenTelemetry than with most tools, because the Collector, the language SDKs, and your backend all need to speak compatible versions of the OTLP wire protocol and semantic conventions. This tutorial pins the Collector to 0.158.0, the latest stable release as of mid-August 2026. If you copy commands from older blog posts referencing 0.153.0 or earlier, expect some configuration keys and exporter names to differ slightly.
Step 1: Plan Your Observability Architecture
Before installing anything, decide where telemetry will flow. The simplest architecture has each application send OTLP data directly to a backend, but that couples your app to a specific vendor’s endpoint and gives you no place to sample, filter, or fan data out to multiple destinations. The pattern nearly every production OpenTelemetry deployment converges on instead is: application SDKs instrument the code and emit OTLP, a Collector sits between the app and any backend to batch, filter, and route, and one or more backends store and visualize the data.
This tutorial builds that three-tier pattern: two sample services (one Python, one Node.js) instrumented with OpenTelemetry SDKs, a Collector running in Contrib mode receiving their OTLP traffic, and a dual export to Grafana Tempo (traces) and Prometheus (metrics) locally, with a commented-out block showing how to add a SaaS backend like Honeycomb or Datadog alongside it. Using the Collector as the single control point means you can add a third backend later without touching application code at all, which is the entire point of adopting OpenTelemetry over a vendor-specific agent.
Step 2: Choose Between Core and Contrib Collector Distributions
The OpenTelemetry Collector ships in two distributions, and picking the wrong one is one of the most common early mistakes. The core distribution (otelcol) bundles a small, tightly curated set of receivers, processors, and exporters maintained directly by the OpenTelemetry core team. The contrib distribution (otelcol-contrib) adds well over 100 community and vendor-maintained components, including exporters for AWS CloudWatch, Google Cloud Monitoring, Kafka, and dozens of other systems.
| Aspect | Core Collector | Contrib Collector |
|---|---|---|
| Component count | Small, curated set | 100+ receivers/processors/exporters |
| Typical image size | ~50 MB | 120 MB+ |
| Maintenance | OpenTelemetry core team | Community and vendor contributors |
| Stability profile | Conservative, slower change cadence | Mixed; some components experimental |
| Best for | OTLP-only pipelines, regulated environments | Cloud vendor exporters, broad integrations |
If your entire pipeline sends OTLP to OTLP-native backends like Tempo, Honeycomb, or SigNoz, core is enough and keeps the attack surface and image size small. This tutorial uses contrib, because the Prometheus remote-write exporter and a few of the processors used later live in the contrib bundle. Regulated environments that need to audit every component in the pipeline often prefer starting from core and adding only the specific contrib components they need through a custom builder configuration (the ocb tool), rather than shipping the entire contrib bundle.
Step 3: Install the OpenTelemetry Collector
There are three common ways to run the OpenTelemetry Collector: as a Docker container, as a standalone binary managed by systemd, or as a Kubernetes Helm chart. This tutorial’s main path uses Docker, since it’s the fastest way to get a working pipeline on a laptop. Pull the contrib image pinned to the version you’re standardizing on.
# Docker: pull the pinned Contrib image
docker pull otel/opentelemetry-collector-contrib:0.158.0
# Confirm the image is present and check its version
docker run --rm otel/opentelemetry-collector-contrib:0.158.0 --version
If you’d rather run the Collector as a native binary on a Linux host (common for VM-based deployments outside Kubernetes), download the release tarball directly from GitHub and verify it starts.
# Binary install on Linux (amd64)
OTEL_VERSION="0.158.0"
curl -LO "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTEL_VERSION}/otelcol-contrib_${OTEL_VERSION}_linux_amd64.tar.gz"
tar -xzf "otelcol-contrib_${OTEL_VERSION}_linux_amd64.tar.gz"
sudo mv otelcol-contrib /usr/local/bin/otelcol
sudo chmod +x /usr/local/bin/otelcol
otelcol --version
Either path gets you the same binary underneath. The Docker route is what the rest of this tutorial builds on, since it composes cleanly with the Tempo and Prometheus containers you’ll add in Step 9.
Step 4: Configure OTLP Receivers
The Collector’s configuration file is a YAML document split into receivers, processors, exporters, and service pipelines that wire them together. Start with the receiver block, which defines how the Collector accepts incoming telemetry. OTLP supports two transports: gRPC on port 4317 and HTTP on port 4318. Configure both, since different SDKs and languages default to different transports.
# otel-collector-config.yaml (receivers section)
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
Mismatched ports and protocols are the single most common OpenTelemetry setup error. If an SDK is configured to send gRPC but points at port 4318 (the HTTP port), or sends HTTP to port 4317, the connection fails silently in a lot of client libraries rather than throwing an obvious error. Double-check that your application’s OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL environment variables agree with each other before assuming the Collector itself is broken.
Step 5: Configure Processors for Batching and Filtering
Processors sit between receivers and exporters, transforming data in flight. The two you’ll want in nearly every pipeline are batch, which groups spans and metrics before sending them onward to reduce the number of network calls, and memory_limiter, which protects the Collector from running out of memory during a traffic spike by dropping data gracefully instead of crashing.
# otel-collector-config.yaml (processors section)
processors:
batch:
timeout: 5s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
resource:
attributes:
- key: deployment.environment
value: production
action: upsert
The resource processor above stamps a consistent deployment.environment attribute onto every piece of telemetry passing through, which matters once you’re running the same Collector config across staging and production and need to tell the resulting data apart in your backend. Add sampling here too once trace volume grows past what your backend’s pricing plan comfortably handles; the tail sampling processor covered in the advanced tips section further down lets you keep 100% of error traces while dropping the majority of routine successful ones.
Step 6: Configure Exporters to Your Backends
Exporters define where processed telemetry goes. This tutorial routes traces to Grafana Tempo and metrics to Prometheus, both running locally via Docker Compose in Step 9. The otlp exporter type works for any OTLP-native backend (Tempo, Jaeger, Honeycomb, SigNoz, OpenObserve), while Prometheus specifically uses either its native OTLP receiver or the dedicated prometheusremotewrite exporter depending on which version you’re running.
# otel-collector-config.yaml (exporters section)
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
prometheusremotewrite:
endpoint: "http://prometheus:9090/api/v1/write"
# Uncomment to dual-export traces to a SaaS backend as well:
# otlp/honeycomb:
# endpoint: "api.honeycomb.io:443"
# headers:
# x-honeycomb-team: "${HONEYCOMB_API_KEY}"
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [prometheusremotewrite]
Notice that traces and metrics use entirely separate pipeline definitions under service.pipelines, each with its own receivers, processors, and exporters. This is a common point of confusion: adding an exporter to the exporters block alone does nothing until it’s also referenced inside a pipeline. If telemetry is arriving at the Collector but never reaching your backend, check the pipeline wiring before assuming the exporter itself is misconfigured.
Step 7: Instrument a Python Application
Python’s tracing and metrics APIs are stable as of August 2026, with logging still catching up, so this example focuses on traces and metrics. Consult the official Python getting-started guide for the current package names, then install the automatic instrumentation package, which detects popular frameworks (Flask, Django, FastAPI, requests) and instruments them without touching application code.
# Install the OpenTelemetry distro and auto-instrumentation
pip install opentelemetry-distro opentelemetry-exporter-otlp
# Auto-detect installed libraries and add instrumentation packages
opentelemetry-bootstrap -a install
# Run your app with auto-instrumentation enabled
OTEL_SERVICE_NAME=checkout-api
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
opentelemetry-instrument python app.py
Auto-instrumentation covers HTTP servers, HTTP clients, and most database drivers out of the box, giving you request-level traces immediately. For business logic that auto-instrumentation can’t see, add a manual span around the specific operation you care about.
from opentelemetry import trace
tracer = trace.get_tracer("checkout-api")
def process_payment(order_id, amount):
with tracer.start_as_current_span("process_payment") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("payment.amount", amount)
# existing payment logic runs unchanged here
result = charge_card(order_id, amount)
span.set_attribute("payment.status", result.status)
return result
This pairing, auto-instrumentation for broad framework coverage plus manual spans for the handful of business operations that matter most, is the pattern current OpenTelemetry guidance recommends across every language. Auto-instrumentation gets you 80% of the value in five minutes; the manual spans are where you encode what “success” and “failure” actually mean for your specific workflow.
Step 8: Instrument a Node.js Application
Node.js follows the same auto-instrumentation-plus-manual-spans pattern, with a slightly different setup mechanic since Node.js instrumentation needs to load before your application code does. Install the SDK and auto-instrumentation packages, then create a dedicated tracing bootstrap file.
npm install @opentelemetry/sdk-node
@opentelemetry/auto-instrumentations-node
@opentelemetry/exporter-trace-otlp-grpc
@opentelemetry/exporter-metrics-otlp-grpc
// tracing.js — must be required before your app entrypoint
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const sdk = new NodeSDK({
serviceName: 'orders-service',
traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4317' }),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({ url: 'http://localhost:4317' }),
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
# Start the app with the tracing bootstrap preloaded
node --require ./tracing.js server.js
The --require ./tracing.js flag is not optional. If instrumentation loads after Express, pg, or another library has already been imported, auto-instrumentation can’t patch it and those spans simply never appear, with no error message to explain why. This is one of the most reported Node.js-specific OpenTelemetry issues, so it’s worth checking first if traces are missing for one library but present for others.
Step 9: Run the Full Stack With Docker Compose
With the Collector configured and both sample apps instrumented, wire everything together with Docker Compose so the Collector, Tempo, and Prometheus start as one stack. This gives you a complete local observability pipeline you can tear down and rebuild in seconds.
# docker-compose.yaml
version: "3.9"
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.158.0
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
depends_on:
- tempo
- prometheus
tempo:
image: grafana/tempo:latest
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
ports:
- "3200:3200"
prometheus:
image: prom/prometheus:latest
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--web.enable-otlp-receiver"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
docker compose up -d
docker compose ps
# Watch Collector logs for incoming OTLP data
docker compose logs -f otel-collector
Note the --web.enable-otlp-receiver flag on Prometheus. Prometheus 3.x added native OTLP metrics ingestion over HTTP, exposed at /api/v1/otlp/v1/metrics, which means you can send metrics straight to Prometheus without the separate remote-write exporter if you prefer; both paths work, and this tutorial’s Collector config uses remote-write for wider compatibility with older Prometheus deployments. Once the stack is up, generate a few requests against your instrumented apps and open Grafana at localhost:3000 to add Tempo and Prometheus as data sources.
Step 10: Deploy the Collector to Kubernetes With Helm
Once you’ve validated the pipeline locally, moving the Collector into a Kubernetes cluster (whether that’s Azure AKS, Google Kubernetes Engine, or a self-managed cluster) uses the official Helm chart. Add the repository and install with the deployment mode that matches your use case.
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm install otel-collector open-telemetry/opentelemetry-collector
--namespace observability --create-namespace
--set image.repository="otel/opentelemetry-collector-contrib"
--set image.tag="0.158.0"
--set mode=daemonset
--values otel-values.yaml
kubectl get pods -n observability --watch
The mode flag is the setting that trips up the most first-time deployments. Use daemonset when you want a Collector instance on every node collecting local telemetry (typical for infrastructure and host metrics). Use deployment when you want a small, independently-scalable pool of Collectors acting as a central gateway that every application in the cluster sends its OTLP traffic to. A common production pattern runs both: a per-node daemonset Collector for local collection and processing, forwarding to a smaller central deployment-mode Collector that handles sampling, dedup, and export to external backends.
Step 11: Add Custom Metrics and Business-Level Spans
Auto-instrumentation captures infrastructure-level signals (HTTP status codes, database query latency), but it has no way of knowing which of those requests represent a completed checkout versus an abandoned cart. Custom metrics fill that gap, letting you build service-level indicators around what actually matters to the business rather than what a generic instrumentation library happens to see.
from opentelemetry import metrics
meter = metrics.get_meter("checkout-api")
checkout_counter = meter.create_counter(
"checkout.completed",
description="Number of successfully completed checkouts",
)
checkout_value = meter.create_histogram(
"checkout.value_usd",
description="Dollar value of completed checkouts",
)
def complete_checkout(order):
# existing checkout logic
checkout_counter.add(1, {"payment.method": order.payment_method})
checkout_value.record(order.total, {"payment.method": order.payment_method})
These custom metrics flow through the exact same Collector pipeline as your auto-instrumented traces, since they use the same OTLP exporter under the hood. That’s the practical payoff of the unified pipeline built in this tutorial: business metrics, infrastructure traces, and framework-level spans all land in the same backend, correlated by the same service.name and trace context, instead of living in three disconnected tools.
Step 12: Validate and Test the Full Pipeline
Before trusting the pipeline in production, run a deliberate end-to-end test rather than assuming silence means success. Generate a known request, then trace it through every layer: the application logs a span, the Collector logs it receiving and exporting data, and the backend shows it in a query.
# 1. Enable debug logging on the Collector temporarily
# (add a "logging" exporter to the traces pipeline in otel-collector-config.yaml)
# 2. Fire a test request against your instrumented app
curl -X POST http://localhost:8080/checkout -d '{"order_id":"test-001","amount":42.50}'
# 3. Confirm the Collector logged receipt of the span
docker compose logs otel-collector | grep "test-001"
# 4. Query Tempo directly for the trace
curl -s "http://localhost:3200/api/search?tags=order.id=test-001" | jq
If the span shows up in Collector logs but never reaches Tempo, the problem is almost always in the exporter or pipeline wiring from Step 6. If it never reaches the Collector at all, the problem is upstream in the SDK configuration or a port/protocol mismatch from Step 4. Isolating which hop failed by checking each layer in order is faster than guessing at the whole pipeline at once.
Step 13: Set Up Monitoring for the Collector Itself
The Collector is now a critical piece of infrastructure, and it needs its own monitoring like any other production service. The Collector exposes its own internal metrics (queue depth, dropped spans, memory usage) on a Prometheus-compatible endpoint by default, which you can scrape the same way you scrape any other service.
# otel-collector-config.yaml (add a telemetry block at the top level)
service:
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
pipelines:
traces: { ... }
metrics: { ... }
Watch otelcol_processor_dropped_spans and otelcol_exporter_send_failed_spans closely once this is live. A rising trend on either metric means the Collector is silently losing data, usually because memory_limiter is dropping telemetry under load or a downstream exporter is failing intermittently. Both are easy to miss without dedicated alerting, since a Collector that’s silently dropping 10% of spans looks identical from the application’s perspective to one that’s working perfectly.
The Complete Working Project, End to End
Stepping back, here’s what the finished project actually contains once all thirteen steps are done. A docker-compose.yaml file starts four containers together: the Collector, Tempo, Prometheus, and Grafana. An otel-collector-config.yaml file defines two receivers (OTLP gRPC and HTTP), three processors (memory_limiter, resource, batch, plus optionally tail_sampling), and two pipelines routing traces to Tempo and metrics to Prometheus. Two sample applications, one in Python and one in Node.js, ship with auto-instrumentation enabled and at least one manual span apiece modeling a real business operation, alongside a custom counter and histogram tracking domain-specific outcomes.
None of the individual pieces are exotic on their own. What makes the project worth building is that every piece speaks the same OTLP wire protocol, so the Collector configuration is the only place that changes when priorities shift, whether that means adding tail sampling to cut costs, routing a copy of production traces to a SaaS backend for a 30-day trial, or migrating an entire company off a legacy proprietary agent one service at a time. That’s a meaningfully different maintenance story than the old world of separate agents per vendor, each with its own instrumentation API, its own configuration format, and its own migration cost the day someone decides to switch.
Common Pitfalls When Setting Up OpenTelemetry
- Confusing core and contrib images. Pulling the wrong image and then wondering why the
prometheusremotewriteor a cloud vendor exporter isn’t recognized is one of the most common early mistakes. Check which distribution a config was written for before copying it. - Mismatched OTLP ports and protocols. Sending gRPC traffic to the HTTP port (4318) or vice versa fails with unhelpful errors in many client libraries. Always verify
OTEL_EXPORTER_OTLP_ENDPOINTandOTEL_EXPORTER_OTLP_PROTOCOLagree. - Missing or inconsistent resource attributes. Telemetry without a consistent
service.nameanddeployment.environmentis nearly impossible to filter cleanly once you have more than a handful of services reporting. - Exporting everything with no sampling. Shipping 100% of trace volume to a SaaS backend with usage-based pricing gets expensive fast. Cost-aware telemetry, meaning deliberate sampling and attribute filtering, is standard practice by 2026, not an optional optimization.
- Version drift between Collector, SDKs, and backend. Mixing an old SDK with a much newer Collector, or vice versa, can cause components incompatible with a given spec revision to silently misbehave. Pin and upgrade all three together.
- Wrong Kubernetes Collector mode. Choosing
deploymentwhen you neededdaemonset(or the reverse) leads to either missing node-local telemetry or an unnecessarily expensive central bottleneck. - Instrumenting only traces, ignoring metrics and logs. A trace-only pipeline misses the aggregate view metrics provide and the detailed event context logs provide; most production setups need at least traces and metrics together.
Troubleshooting OpenTelemetry Issues
No spans arriving at the Collector at all. Check that the application’s OTLP endpoint matches the Collector’s actual host and port, and confirm no firewall or container network policy is blocking 4317/4318.
Spans arrive at the Collector but never reach the backend. Verify the exporter is referenced inside the relevant pipeline under service.pipelines, not just declared in the top-level exporters block. This is the single most common cause of “silent” export failures.
Some libraries are traced, others aren’t (Node.js). The tracing bootstrap file almost certainly loaded after the untraced library was imported. Confirm --require ./tracing.js comes before your application entrypoint, not after.
Collector crashes or restarts under load. Check memory_limiter settings; a limit set too close to the container’s actual memory allocation causes OOM kills instead of graceful backpressure. Give the Collector headroom above the configured limit.
Metrics show up in Prometheus with wrong or missing labels. Confirm the resource processor is actually wired into the metrics pipeline, not just the traces pipeline. Each pipeline needs its own processor list.
Traces are fragmented across multiple disconnected spans. This usually means context propagation is broken somewhere between services, often because a message queue or async job hop doesn’t forward the trace context headers. Check that every network hop, including internal queues, propagates the traceparent header.
Prometheus OTLP receiver returns errors. Confirm the --web.enable-otlp-receiver flag is actually set on the Prometheus container; without it, the OTLP ingestion endpoint doesn’t exist and requests fail with a 404.
High cardinality is blowing up backend costs or crashing Prometheus. Audit custom attributes for anything with unbounded values (raw user IDs, full URLs with query strings) and either drop them with an attribute processor or replace them with bucketed/templated versions.
Helm install succeeds but no pods appear. Check kubectl describe pod for image pull errors first; a typo in the image.tag value (referencing a version that doesn’t exist) is a common cause after copying config from an older tutorial.
Advanced Tips: Tail Sampling and Cost Control
Once your pipeline is stable, the next problem most teams hit is volume. Shipping every single trace to a usage-priced backend gets expensive well before it gets genuinely more useful, since the overwhelming majority of traces represent routine, successful requests that nobody will ever look at. Tail sampling solves this by making the keep-or-drop decision after a trace completes, rather than at the start, so you can keep 100% of traces that contain an error or exceed a latency threshold while sampling down the routine traffic.
# otel-collector-config.yaml (add to processors, then reference in the traces pipeline)
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: errors-always-keep
type: status_code
status_code:
status_codes: [ERROR]
- name: slow-requests-keep
type: latency
latency:
threshold_ms: 1000
- name: baseline-sample
type: probabilistic
probabilistic:
sampling_percentage: 10
Tail sampling requires all spans belonging to a single trace to reach the same Collector instance before a sampling decision can be made, which matters if you’re running multiple Collector replicas behind a load balancer. The standard fix is routing traffic through a load-balancing exporter that hashes on trace ID, ensuring every span for a given trace lands on the same downstream Collector for the sampling decision. Combined with semantic convention attributes for consistent filtering, this is the pattern that lets teams keep full-fidelity error visibility while cutting routine trace volume by 80-90%, which is usually the difference between an OpenTelemetry pipeline that fits the budget and one that doesn’t.
OpenTelemetry-Compatible Backends Compared
Not every backend treats OTLP data the same way once it arrives. OTel-native backends preserve OpenTelemetry’s semantic conventions and attributes without translation, while OTel-compatible commercial platforms convert incoming OTLP into their own internal proprietary format, which can occasionally drop or rename attributes that don’t map cleanly. Grafana Tempo and Jaeger both fall into the OTel-native camp for traces, while Prometheus’s native OTLP receiver is documented as a feature flag you enable explicitly rather than a default-on behavior.
| Backend | Type | Signals Supported | Pricing Model |
|---|---|---|---|
| Grafana Tempo | OTel-native, open source | Traces | Free self-hosted; Grafana Cloud usage-based tiers |
| Prometheus 3.x | OTel-compatible, open source | Metrics | Free self-hosted |
| Jaeger | OTel-compatible, open source | Traces | Free self-hosted, CNCF project |
| Honeycomb | OTel-compatible, SaaS | Traces, metrics, logs | Free tier; usage-based paid plans |
| Datadog | OTel-compatible, SaaS | Traces, metrics, logs | Usage-based, proprietary pricing |
| OpenObserve / SigNoz | OTel-native, open source | Traces, metrics, logs | Free self-hosted; cloud plans available |
The setup in this tutorial exports to Tempo and Prometheus, both self-hosted and free, which is a reasonable default for evaluating OpenTelemetry before committing budget to a SaaS platform. Because the Collector controls routing rather than the application code, adding a Honeycomb or Datadog exporter alongside the existing ones later is a configuration change, not a re-instrumentation project.
Frequently Asked Questions
Is OpenTelemetry free to use?
Yes. The SDKs, the Collector, and the specification are all open source under the CNCF. Costs come from whichever backend you choose to store and query the data, whether that’s a free self-hosted stack like Tempo and Prometheus or a usage-based SaaS platform.
Do I need the Collector, or can apps send telemetry straight to a backend?
Direct export works for small setups, but the Collector adds batching, sampling, retry logic, and the ability to route to multiple backends without touching application code. Most production deployments use it once they have more than a couple of services.
What’s the difference between core and contrib Collector distributions?
Core ships a small, tightly maintained set of components; contrib adds 100+ community and vendor exporters, receivers, and processors, including most cloud vendor integrations. Use contrib unless you specifically need the smaller image size and narrower change surface of core.
Can I switch observability vendors later without re-instrumenting my code?
That’s the core design goal of OpenTelemetry. Since instrumentation targets the vendor-neutral OTel API rather than a specific backend’s SDK, switching vendors is a Collector exporter configuration change, not an application code change.
Are logs supported yet?
Support varies by language. Traces and metrics are stable everywhere as of August 2026; logs support is further along than a year ago but still catching up in some SDKs. Check your specific language’s status page before building critical alerting around OpenTelemetry logs today.
How much does auto-instrumentation actually cover?
Auto-instrumentation reliably covers HTTP servers and clients, most popular database drivers, and common messaging libraries. It has no visibility into business logic, so manual spans and custom metrics are still necessary for anything domain-specific, like tracking completed checkouts or subscription renewals.
Why is my Collector dropping data under load?
Check the memory_limiter processor settings first. If its limit is too close to the container’s actual available memory, the Collector drops telemetry to protect itself rather than crashing outright. Give it headroom and monitor otelcol_processor_dropped_spans to catch this early.
What is OpenTelemetry Profiling, and should I use it yet?
Continuous profiling is OpenTelemetry’s newest signal, in public alpha since March 26, 2026, with its own OTLP Profiles data model. It’s promising for connecting CPU/memory profiles to traces, but it’s not yet mature enough to build production alerting around; traces and metrics remain the stable foundation for now.





