Coder Legion - Recent posts tagged metrics

Centralizing Observability: Introducing the NKE Metrics Agent

Mon, 14 Sep 2026 04:39:27 +0000

Monitoring is a critical part of any cloud-native application. As Kubernetes environments grow, so does the complexity and resource cost of observing them. Today, we are excited to announce the general availability of Metrics Agent for the Nine Kubernetes Engine (NKE).

This release marks a significant step forward in our “Global Metrics” initiative, designed to provide a centralized, efficient, and scalable observability platform for all Nine products.

The Challenge with Prometheus

Monitoring in NKE involved running a full Prometheus instance within each cluster. While effective, this approach has downsides:

Resource Consumption: Prometheus can be memory and CPU intensive,
eating into the resources available for your actual applications.
Storage Complexity: Managing long-term persistent storage for
metrics within the cluster is challenging and can be costly.
Fragmentation: Metrics are locked inside each cluster, making it
hard to get a unified view of your infrastructure.

Because of these reasons, Metrics Agent will eventually replace the standalone Prometheus product for NKE customers, providing a more efficient and centralized solution.

Enter the Metrics Agent

Metrics Agent is a lightweight component based on VictoriaMetrics. Instead of storing metrics locally, it collects them and pushes them to our centralized Global Metrics cluster.

This architecture brings several immediate benefits:

Reduced Footprint: The agent runs on the NKE control plane, freeing
up your worker nodes for your workloads.
Centralized Storage: Your metrics are stored securely in our managed
Global Metrics backend, ensuring durability and high availability
without you needing to manage Persistent Volumes for monitoring.
Standard Configuration: We leverage the popular Prometheus Operator
ecosystem. You can continue to define your scraping targets using
standard ServiceMonitor and PodMonitor resources.

Important: Metrics Agent is currently limited to 100,000 unique time series per day. See the documentation for details.

How It Works

When you enable Metrics Agent on your NKE cluster, it deploys a vmagent (VictoriaMetrics Agent) into the nine-system namespace. This agent automatically discovers targets based on your configuration.

To start scraping an application, you simply create a ServiceMonitor or PodMonitor and add the required label:

labels:
  prometheus.nine.ch/<your-agent-name>: scrape

Note: Replace <your-agent-name> with the specific name of your metrics agent.

The agent then scrapes the targets and forwards the data to our central store, where it becomes available for querying via Grafana.

Note: Metrics Agent does not currently support Grafana Alerting. Use Alertmanager instead for alerting rules.

Pricing

Metrics Agent billing is based on your actual usage, plus a fixed fee per instance. For current pricing details, see the Metrics Agent documentation.

Migrating from Prometheus

If you’re currently using Prometheus in your NKE cluster, migrating to Metrics Agent is straightforward. By naming your Metrics Agent the same as your existing Prometheus instance, it will automatically pick up all your existing ServiceMonitors and PodMonitors without any configuration changes.

For detailed migration steps, see the migration guide in our documentation.

Getting Started

Metrics Agent is available now for NKE. To get started, read the full Metrics Agent guide for detailed setup instructions.

One Platform for All Metrics

The release of Metrics Agent is part of our broader vision for a unified observability platform. We are actively working on adding metrics from more of our managed services to this global storage, giving you a single pane of glass for your entire infrastructure at Nine.

Users of our PaaS product, deplo.io, benefit from a similar observability experience: application metrics there can also be visualized via your own Grafana instance. You can learn more about this in the deplo.io observability documentation.

We believe this new approach will significantly simplify how you monitor your applications on NKE. Activate Metrics Agent directly in Cockpit or follow the migration guide to switch from Prometheus. As always, we are here to help - if you have questions or need assistance, please reach out to our support team.

My Phone Was Talking Behind My Back. NextDNS Caught Every Word

Sun, 23 Aug 2026 18:29:06 +0000

A few weeks ago I did something embarrassingly simple: I pointed my Android phone's DNS at NextDNS, turned on logging, and just left it alone. No apps opened, no browsing, phone face-down on the desk. I wanted to see what my phone talks about when I'm not the one talking to it.

Turns out my phone has a lot to say. And almost none of it was to me.

What actually showed up in the logs

Within the first hour, before I'd touched a single app, the query log was already scrolling. Analytics pings. Crash reporting handshakes. Push notification services checking in like clockwork. By the time I had a real sample size, close to a third of all outbound DNS queries from a phone that was, by any reasonable definition, just sitting there — were getting flagged and blocked as ads or trackers. Not "I was scrolling Instagram" traffic. Idle traffic. Screen off, no interaction, sitting on a desk being a small glowing brick.

And here's the part that actually got under my skin: the single biggest source of that traffic, by a wide margin, wasn't some sketchy third-party SDK I'd never heard of. It was Google. Analytics, Ads, DoubleClick, Crashlytics — the whole in-house stack, running quietly underneath apps that had nothing to do with search or ads at all. Then a handful of the usual push-notification and analytics middlemen behind that. My phone runs Android. Of course a chunk of that traffic is Google's own OS talking to Google's own servers. But watching the actual volume, in real time, with domain names instead of vague reassurances, hits different than reading a privacy policy that says "we may collect usage data to improve your experience."

This is the part where I'd normally tell you I'm exaggerating for effect. I'm not. This is just what a stock Android phone does by default, and I hadn't even gotten to the browser yet.

Turns out this isn't a vibe, it's a measured, peer-reviewed fact

Here's where I stopped feeling paranoid and started feeling validated. Researcher Douglas Leith at Trinity College Dublin ran a formal study comparing telemetry traffic from Android and iOS handsets back to Google and Apple. The findings: both operating systems transmit telemetry even after a user explicitly opts out in settings. Both send data before you've ever logged in, in some cases before you've logged in at all. And Android sends dramatically more of it — on an idle handset, a Pixel checks in with Google's servers roughly every four and a half minutes, and Google collects something in the neighborhood of 20 times more telemetry volume than Apple does from an equivalent iPhone.

This isn't a fringe theory some guy posted on a forum. It's peer-reviewed academic research, and it got expensive for Google to ignore: in July 2025, a California jury awarded $314.6 million against Google specifically over Android phones transferring data to Google's servers while idle — over cellular data, at the phone owner's own expense, functionally billing you for the privilege of being tracked. Google's spokesperson called the verdict a misunderstanding of services "critical to the security, performance, and reliability of Android devices." Maybe. It's also, technically, the exact behavior an independent academic study measured and published years before a jury agreed a real person got harmed by it. A second case covering the other 49 U.S. states followed.

Google isn't just an app on your phone, it's most of the pipe your phone talks through

Step back from the DNS logs for a second, because the telemetry itself is only half the story. The other half is just how much of the internet's plumbing runs through one company by default, which is what makes "opting out" such a hilarious concept in practice.

As of 2026, Android sits around 72-73% of global mobile OS market share. Chrome holds somewhere around 68-70% of the global browser market, meaning the majority of humans reading anything online right now are doing it through a piece of software built by the same company that runs the ad network deciding what you see next. Google Search still commands somewhere north of 90% of the search market globally, and in the U.S., a federal judge already ruled in August 2024 that this dominance constitutes an illegal monopoly under the Sherman Act — a ruling whose remedies trial is still working its way through 2026. The EU fined Google $3.5 billion in September 2025 for separate competition violations. None of that dented Search revenue through the most recent quarter on record. You can win the ruling and still lose nothing that matters financially, which tells you everything about how enforcement currently scales against companies this size.

Put plainly: it's not that Google collects a lot of data. It's that Google built, owns, and operates the road, the car, and the tollbooth, and then also runs the billboard company. When one entity is the operating system, the browser, the search engine, and the ad exchange all at once, "just don't use their products" stops being a real option for the overwhelming majority of people on Earth, including, until three weeks ago, me.

The short list of companies who actually run your digital life

Google isn't alone up there, it's just the one with the most complete vertical stack. Zoom out and the internet most people experience daily runs through a genuinely small handful of companies: Google for search, Android, and the ad exchange underneath a huge share of the web's monetization. Meta for the social graph most people's real-world relationships are mapped onto. Amazon for a meaningful chunk of the cloud infrastructure the rest of the internet physically runs on, AWS included. Apple for the hardware and app-store gate on the other half of the world's phones. Microsoft for enterprise identity, GitHub, and now a deepening AI stake across the stack. Five companies, functionally deciding the default conditions under which a few billion people work, communicate, and get recommended what to buy next.

None of that requires a conspiracy. It's just what happens when infrastructure businesses win, at scale, for two decades straight, mostly unchallenged, while regulators wrote reports about it.

So what do you actually do with this

I'm not going to tell you to abandon Android and disappear into a GrapheneOS bunker, that's not a realistic ask for most people reading this, myself included some days. But NextDNS, or Pi-hole if you'd rather self-host it, is a genuinely low-effort way to see the shape of what's happening instead of just trusting a privacy policy summary. It won't stop everything — plenty of this telemetry survives the opt-out toggle entirely, that's the whole point of the Trinity College study. But watching the actual query log for even 48 hours will change how you think about the phrase "we only collect anonymous usage data to improve your experience," because you'll have a domain name and a timestamp proving otherwise, sitting right there in front of you.

Turns out the most educational thing you can do with a smartphone isn't installing a new app. It's just watching, for once, what it does when you're not.

Prometheus 3.0 and OpenTelemetry: Native OTLP Support Explained

Mon, 27 Apr 2026 23:49:40 +0000

Seven years is a long time in observability. Since Prometheus 2.0 landed in 2017, the ecosystem has been transformed by cloud-native adoption, the rise of distributed tracing, and the emergence of OpenTelemetry as the de facto standard for instrumentation. Prometheus 3.0, released in November 2024, is the project’s answer to that transformation — and its most significant change is the native ability to ingest OpenTelemetry metrics directly, without an intermediary collector standing in the way.

This article goes deep on what Prometheus 3.0 actually changes for platform engineers and cloud architects who are running — or planning to run — OTel-instrumented workloads alongside Prometheus-based monitoring stacks. We will cover the native OTLP ingestion endpoint, UTF-8 metric name support, Remote Write 2.0, migration considerations, and the architectural patterns that still make sense even when native OTLP is available.

What Changed in Prometheus 3.0: The OTel-Relevant Picture

Prometheus 3.0 ships a substantial set of changes. Not all of them are equally relevant to OpenTelemetry integration, so let’s focus on what actually moves the needle for OTel users before diving into each area in detail.

Native OTLP Ingestion

The flagship feature: Prometheus 3.0 ships with a built-in OTLP receiver that exposes an HTTP endpoint accepting metrics in the OpenTelemetry Protocol format. Applications instrumented with any OTel SDK can now push metrics directly to Prometheus without routing through an OpenTelemetry Collector. This is not a sidecar, not a plugin, not an external adapter — it is a first-class endpoint in the Prometheus binary itself.

UTF-8 Metric Names

Prometheus historically restricted metric names to [a-zA-Z_:][a-zA-Z0-9_:]*. OpenTelemetry uses dots and slashes in metric names by convention — http.server.request.duration is a canonical OTel metric name. Prometheus 3.0 lifts this restriction and supports arbitrary UTF-8 characters in metric names and label names, which is the single most important compatibility change for OTel interoperability.

Remote Write 2.0

Remote Write 2.0 replaces the original protocol with a more efficient encoding based on protobuf, adds native histogram support in the wire format, and reduces bandwidth consumption significantly for large-scale deployments. If you are federating metrics to Thanos, Mimir, or Cortex, this matters for operational cost.

New UI

The Prometheus web UI has been completely rewritten. The new UI uses React, supports metric metadata exploration, and provides a significantly improved query-building experience. This is a quality-of-life improvement rather than an architectural change, but it reduces the dependency on external tools like Grafana for ad-hoc investigation.

Breaking Changes Summary

Prometheus 3.0 removes several features that were deprecated in 2.x. The most operationally significant are: removal of the --web.enable-admin-api deprecated flag path, removal of certain legacy storage format options, changes to default scrape timeouts, and stricter validation of configuration that was previously silently accepted. We cover a migration checklist later in this article.

The OTLP Receiver: How It Works and What It Accepts

The OTLP receiver in Prometheus 3.0 is implemented as an optional feature that must be explicitly enabled. Once enabled, it exposes an HTTP endpoint at /api/v1/otlp/v1/metrics that accepts protobuf-encoded OTLP ExportMetricsServiceRequest payloads — the same wire format used by the OpenTelemetry Collector’s OTLP exporter.

What It Accepts (and What It Does Not)

This is critical to understand before you architect around native OTLP ingestion: Prometheus 3.0 OTLP support is metrics-only. It does not accept traces or logs. OTLP is a unified protocol covering all three signals, but Prometheus is a metrics store — the receiver handles only the metrics portion of the OTLP specification.

Supported metric types in the OTLP receiver:

Gauge — maps directly to a Prometheus Gauge
Sum (monotonic) — maps to a Prometheus Counter
Sum (non-monotonic) — maps to a Prometheus Gauge
Histogram (explicit bucket) — maps to a Prometheus Histogram
ExponentialHistogram — maps to Prometheus Native Histograms (a 3.0 feature)
Summary — maps to a Prometheus Summary

Resource attributes from the OTLP payload — things like service.name, k8s.pod.name, cloud.region — are converted to Prometheus labels. This conversion is configurable, and by default Prometheus applies a promotion strategy that converts the most common resource attributes to labels while discarding ones that would create extremely high cardinality.

Enabling the OTLP Receiver

Enabling native OTLP ingestion requires two things: a feature flag and a configuration block in prometheus.yml.

Start the Prometheus binary with the feature flag:

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=otlp-write-receiver

Then add the OTLP receiver configuration to your prometheus.yml:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

otlp:
  # Promote these OTLP resource attributes to Prometheus labels
  promote_resource_attributes:
    - service.name
    - service.namespace
    - service.instance.id
    - k8s.namespace.name
    - k8s.pod.name
    - k8s.node.name
    - cloud.region
    - deployment.environment

With this configuration, Prometheus will listen on port 9090 (default) and accept OTLP metrics at http://<prometheus-host>:9090/api/v1/otlp/v1/metrics.

Resource Attribute Promotion Strategy

The promote_resource_attributes list deserves careful thought. OTLP carries rich resource-level context — every metric payload includes a ResourceMetrics object with attributes describing the source: service name, version, environment, Kubernetes pod, node, cluster, cloud provider details, and more. Prometheus labels are flat key-value pairs on each time series. Promoting too many resource attributes explodes cardinality; promoting too few loses important context.

A pragmatic starting list for Kubernetes deployments:

otlp:
  promote_resource_attributes:
    - service.name          # Critical: identifies the service
    - service.namespace     # Logical grouping
    - deployment.environment  # prod/staging/dev
    - k8s.namespace.name    # Kubernetes namespace
    - k8s.pod.name          # Pod-level cardinality — consider omitting in high-scale
    - k8s.node.name         # Useful for infrastructure correlation

Avoid blindly promoting k8s.pod.name at scale — in a cluster with thousands of short-lived pods, this creates significant cardinality pressure. Prefer service.name and service.namespace for most alerting use cases, reserving pod-level labels for debugging dashboards.

UTF-8 Metric Names: Why This Is the Real Game-Changer

To appreciate why UTF-8 metric name support matters so much, you need to understand the friction it eliminates. OpenTelemetry semantic conventions define metric names using dots as namespace separators. The canonical HTTP server duration metric is http.server.request.duration. The canonical database query duration is db.client.operation.duration. These names are standardized across languages and frameworks — your Go service and your Java service and your Python service all emit the same metric name when instrumented with OTel.

Prometheus 2.x could not store these names. The dots are illegal characters in Prometheus metric naming. Every OTel-to-Prometheus bridge — the OpenTelemetry Collector’s Prometheus exporter, prom-client compatibility layers, the older prometheusremotewrite exporter — had to translate these names, typically by replacing dots with underscores: http_server_request_duration.

This translation is lossy and creates multiple problems:

Name collisions: http.server.request_duration and http.server.request.duration both become http_server_request_duration
Dashboard breakage: Grafana dashboards built against OTel semantic conventions don’t work against translated Prometheus metrics without modification
Cross-signal correlation: Trace attributes use dot notation; when metric names differ, automated correlation tools lose the thread
Vendor lock-in pressure: Teams end up with separate naming conventions for “Prometheus metrics” vs “OTel metrics” and maintain both

Prometheus 3.0 with UTF-8 support stores http.server.request.duration natively. No translation. No collision. The metric name you instrument with is the metric name you query.

Enabling UTF-8 Metric Names

UTF-8 metric names require the utf8-names feature flag:

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=utf8-names \
  --enable-feature=otlp-write-receiver

Once enabled, PromQL queries must use quoted metric names when the name contains characters outside the legacy character set:

# Legacy metric name — unquoted works fine
http_server_requests_total

# OTel metric name with dots — requires quoting in PromQL
{"__name__"="http.server.request.duration"}

# Or using the new PromQL syntax in Prometheus 3.0
http.server.request.duration{service_name="api-gateway"}

The PromQL parser in Prometheus 3.0 has been updated to handle quoted metric names as a first-class construct. Grafana’s PromQL engine has also been updated to handle this syntax — verify your Grafana version (10.3+ has full support) before deploying.

OTel SDK to Prometheus 3.0 Directly: No Collector Required

For teams that only need to get application metrics into Prometheus, native OTLP ingestion enables a dramatically simpler architecture. Here’s what it looks like with different OTel SDKs.

Go (OpenTelemetry SDK)

package main

import (
    "context"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func initMetrics(ctx context.Context) (*metric.MeterProvider, error) {
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("my-api"),
            semconv.ServiceNamespace("platform"),
            semconv.DeploymentEnvironment("production"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Point directly at Prometheus 3.0 OTLP endpoint
    exporter, err := otlpmetrichttp.New(ctx,
        otlpmetrichttp.WithEndpoint("prometheus:9090"),
        otlpmetrichttp.WithURLPath("/api/v1/otlp/v1/metrics"),
        otlpmetrichttp.WithInsecure(), // Use WithTLSClientConfig for production
    )
    if err != nil {
        return nil, err
    }

    provider := metric.NewMeterProvider(
        metric.WithResource(res),
        metric.WithReader(
            metric.NewPeriodicReader(exporter,
                metric.WithInterval(30*time.Second),
            ),
        ),
    )

    otel.SetMeterProvider(provider)
    return provider, nil
}

Python (OpenTelemetry SDK)

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_NAMESPACE

resource = Resource.create({
    SERVICE_NAME: "my-api",
    SERVICE_NAMESPACE: "platform",
    "deployment.environment": "production",
})

exporter = OTLPMetricExporter(
    endpoint="http://prometheus:9090/api/v1/otlp/v1/metrics",
)

reader = PeriodicExportingMetricReader(
    exporter,
    export_interval_millis=30_000,
)

provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

# Use the meter
meter = metrics.get_meter("my-api")
request_counter = meter.create_counter(
    name="http.server.request.count",
    description="Total HTTP server requests",
    unit="1",
)
request_duration = meter.create_histogram(
    name="http.server.request.duration",
    description="HTTP server request duration",
    unit="s",
)

Java (OpenTelemetry SDK with Spring Boot)

# application.properties (Spring Boot with OTel auto-instrumentation)
otel.service.name=my-api
otel.resource.attributes=service.namespace=platform,deployment.environment=production

# Configure OTLP exporter to push directly to Prometheus
otel.metrics.exporter=otlp
otel.exporter.otlp.metrics.endpoint=http://prometheus:9090/api/v1/otlp/v1/metrics
otel.exporter.otlp.metrics.protocol=http/protobuf

# Export interval
otel.metric.export.interval=30000

With Spring Boot and the OTel Java agent, no code changes are required beyond configuration — the agent instruments your HTTP server, database clients, and messaging systems automatically and pushes metrics using the names defined in OTel semantic conventions.

OTel Collector to Prometheus 3.0: When You Need the Intermediary

Native OTLP ingestion is compelling, but the OpenTelemetry Collector remains relevant for a significant set of use cases. Understanding when each pattern is appropriate is the core architectural decision you will face when adopting Prometheus 3.0 in an OTel environment.

Pattern 1: OTel Collector as Fan-Out Gateway

When you need to send metrics to multiple backends simultaneously — Prometheus for alerting, a long-term store like Thanos for historical analysis, and a commercial observability platform for full-stack correlation — the OTel Collector handles fan-out efficiently. Applications push once to the Collector; the Collector distributes to all backends.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  # Push to Prometheus 3.0 via OTLP
  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
    tls:
      insecure: true

  # Fan-out to Thanos via remote_write
  prometheusremotewrite/thanos:
    endpoint: http://thanos-receive:10908/api/v1/receive
    resource_to_telemetry_conversion:
      enabled: true

  # Fan-out to commercial backend
  otlp/datadog:
    endpoint: https://otel-intake.datadoghq.com
    headers:
      DD-API-KEY: "${DD_API_KEY}"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/prometheus, prometheusremotewrite/thanos, otlp/datadog]

Pattern 2: Collector for Metric Transformation

The OTel Collector’s transform processor and metricstransform processor allow you to reshape metrics before they reach Prometheus: rename labels, add static attributes, filter out high-cardinality series, aggregate metrics to reduce storage cost, or apply unit conversions. These operations are not available in Prometheus’s native OTLP receiver.

processors:
  transform/metrics:
    metric_statements:
      - context: metric
        statements:
          # Drop internal debug metrics
          - delete_matching_keys(attributes, "internal.*")
          # Normalize environment label values
          - set(attributes["deployment.environment"], "prod")
            where attributes["deployment.environment"] == "production"

  filter/drop_debug:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - ".*\.debug\..*"
          - "runtime\.go\.internal\..*"

  metricstransform:
    transforms:
      # Rename a metric to match your existing Prometheus naming convention
      - include: http.server.request.duration
        action: update
        new_name: http_server_request_duration_seconds

Pattern 3: Collector for Traces and Logs (Always Required)

If your architecture includes traces and logs alongside metrics — and in 2025 it almost certainly does — you need an OTel Collector regardless of what you do with metrics. Prometheus does not accept traces or logs. Jaeger, Tempo, and Loki all have their own ingestion protocols. The Collector is the universal routing layer for the three pillars of observability.

In this architecture, it is usually simpler to route all three signals through the Collector and let it push metrics to Prometheus via OTLP or remote_write, rather than splitting metrics to go directly and everything else through the Collector.

When to Use Native OTLP vs. OTel Collector: Decision Framework

<td

Scenario	Native OTLP	OTel Collector
Single metrics backend (Prometheus only)	Preferred	Overkill
Multiple metrics backends

Measuring productivity without freaking people out

Mon, 27 Apr 2026 15:39:18 +0000

You’ve settled on the key metrics to assess engineering team performance and satisfaction. Now, how do you ensure those metrics don’t become a source of stress or resentment?

Any time you set a target – be it deployment frequency or mean time to recover – the spectre of Goodhart’s Law looms, where metrics can become gameable, toxic, or prioritized beyond the health of the wider team or organization. Engineering leaders need to be vigilant to balance the value of ambitious targets against the risk of damaging team health.

This discussion explores tactics to choose and communicate metrics that balance high performance with a long term culture of continuous improvement.

Key takeaways:

Choose metrics your engineers won’t hate
Finding a balance between productivity and collaboration
How to set goals and use metrics to motivate your team

How To Measure AI’s Organizational Impact

Mon, 20 Apr 2026 12:00:09 +0000

When organizations introduce AI, they often make a critical error: they create entirely new metrics to measure its impact. This approach misses the fundamental truth that AI is a tool to help achieve existing goals, not a reason to change what success looks like.

Your Goals Haven’t Changed

Consider the difference between Formula 1 racing and EcoRally Scotland. Formula 1 teams optimize for speed — whoever crosses the finish line first wins. EcoRally teams have a completely different challenge: complete a 500-kilometer route with the best regularity score while using the least energy possible.

These teams need different strategies, different driving styles, and different metrics. The goals determine everything else.

The same principle applies to your organization. When you introduce AI, your fundamental purpose remains unchanged. You still want to create the best quality speakers, save bees, or deliver whatever value you were creating before. AI is simply a new tool to help you achieve those existing goals more effectively.

Stick With What Already Works

Organizations often have sophisticated measurement systems in place — financial metrics, mission-based indicators, and proxy measures that track different parts of their value stream. If you’ve already established that software delivery performance correlates with organizational outcomes, for example, then continue using those same measures to evaluate AI’s impact.

The danger lies in creating new metrics specifically for AI adoption. These measures rarely connect to meaningful business outcomes and can lead you to optimize for activities that don’t actually move the needle on what matters most.

The Local Optimization Trap

Here’s a common scenario: A development team starts using AI and reduces their feature delivery time from 16 hours to 12 hours — a 25% improvement that looks impressive on paper. However, when you examine the entire value stream, the lead time from customer request to delivered value remains unchanged at two weeks.

This isn’t a new problem. Eli Goldratt explored this in “The Goal,” and Lean Software Development emphasizes optimizing for the whole system, not individual parts. AI amplifies this challenge because it’s easy to see immediate productivity gains in specific areas while missing the broader organizational impact.

Focus On What Truly Matters

Most teams collect numerous metrics that help them improve their work and maintain standards. But organizationally, only a few metrics are truly critical — usually some combination of financial performance and mission-based indicators that track whether you’re making the intended difference in the world.

AI only delivers real value when its benefits flow through to these crucial numbers. Everything else is just interesting data.

Research-Driven Implementation

The most effective approach follows basic research principles: form a hypothesis, design a test, then evaluate the results. Before implementing AI, articulate clearly how you expect it to impact your mission-level metrics. If you’ve already established relationships between local measures (like software delivery performance) and organizational outcomes, you can build your hypothesis on these proven connections.

Too many organizations reverse this process — they implement AI first, then scramble to find metrics that show improvement. This backwards approach leads to hockey-stick charts that look impressive but don’t translate to meaningful business value. It’s the difference between running a business and running a marketing campaign.

The Path Forward

AI will impact your business — that’s inevitable. But whether that impact is positive depends largely on how thoughtfully you approach adoption. By maintaining focus on your existing goals and proven metrics, you can ensure that AI becomes a genuine accelerator of your mission rather than an expensive distraction.

The organizations that will see the greatest benefit from AI are those that resist the temptation to change their definition of success and instead use AI to achieve their existing definition of success more effectively.

The top 5 features of internal developer platforms

Tue, 20 Jan 2026 14:59:13 +0000

Our Platform Engineering Pulse report looked at many aspects of real-world practice, but one interesting study area was the themes for common platform features. We're launching a broader study to investigate this further, but here are the top 5 features the platform teams we asked had added to internal developer platforms.

Build automation
Deployment automation
Infrastructure automation
Test automation
Monitoring and observability

1. Build automation

Build automation should be triggered each time you change the code and provide you with fast feedback on your changes. If you're practicing Continuous Delivery, you'll commit changes to the main branch every few hours, with the feedback from the build process, including fast-running tests, arriving in around 5 minutes.

The build process includes compilation, linking, and bundling. It usually consists of a suite of fast-running tests that validate the functionality and fitness of the application. In practice, this is typically a sequence of scripts or commands that run each tool in the build chain, but some build processes are more complex.

The benefit of bringing build automation into your internal developer platform is the opportunity to harmonize pipelines and bring them all up to a common standard for fast and secure builds. For example, instead of each team having to invent a way to sign their package so it can be verified later in the deployment pipeline, platform teams can make this a standard part of the build.

Development teams rarely touch their build process, compared to how much focus a platform team might put into builds. That means they are less familiar with the tools and may not be aware of features that could improve build performance or security.

2. Deployment automation

Manual deployments are one of the primary reasons for bad paths to production. Even simple manual deployments can go drastically wrong, and when they do, the organization responds by introducing process steps that slow down the flow of change. This is well-meaning, but we've learned that reducing deployment frequency results in larger batches, which carry more risk and cause bigger problems.

Deployment automation is the heart of change in your software delivery. When deployments are reliable, repeatable, and low-effort, you can teach the organization to do them more often. Automated deployments help you build trust and reduce heavyweight processes like change approval by committee, which universally clogs the ability to deploy small batches frequently and safely.

The steps, process templates, project templates, and policies related to deployments are a perfect fit for Platform Engineering. The platform team can ensure crucial steps are included in all deployments, like verifying the provenance of the packages being deployed and ensuring the packages and deployment process have progressed through appropriate environments before reaching production.

When the organization needs to level up security across all deployments, a platform team using the right deployment tools can quickly bring the whole organization into compliance with the new standards.

3. Infrastructure automation

There was a time when ClickOps was the default for infrastructure changes. Modern software teams have rejected the idea of managing their infrastructure by clicking around in administration consoles, cloud portals, and command lines, as the result is that no two instances of anything are the same. These differences, though sometimes subtle, can cause production incidents that are difficult to identify and resolve.

By treating infrastructure as code, the definitions can be stored in version control and follow the same review and promotion patterns developers apply to application code. When infrastructure is created and managed this way, each instance is an identical copy, and it's easier to scale out, recover from a disaster, or create ephemeral environments for testing.

Platform teams can create standard setups for teams to use that are aligned to the technology choices in their golden pathways. Reducing the breadth of choice makes it easier to secure the infrastructure, control costs, and apply patches and updates.

4. Test automation

Sometimes it seems there are automated tests everywhere, from unit tests that run continuously in the background as a developer makes changes, to tests used in the build process, to validate end-to-end functionality, and test the fitness of the application's performance and security. The reason for their prevalence is that testing really works.

Without test automation, your team would need to perform the same checks manually, and when you don't do it, your customers do, which is incredibly frustrating.

While platform teams don't typically write tests (the research shows the most effective automated tests are written by the developer creating the feature), they can provide the tools that make it easy to define tests and have them run throughout the software delivery process. Platform teams may also provide easy ways to consume test services, making it easy to spin up end-to-end tests or performance tests without managing infrastructure.

To shorten feedback loops, all types of testing should be performed continuously and ideally represented by fast, reliable test automation suites. You should cover functional, security, and performance tests within your deployment pipeline.

You should consider how test data is managed as part of your test automation strategy. The ability to set up data in a known state will help you make tests less flaky. Test automation lets developers know early if their change causes a fault, reduces team burnout, and increases software stability.

5. Monitoring and observability

While test automation covers a suite of expected scenarios, monitoring and observability help you expand your view to the entirety of your real-world software use. Monitoring implementations tend to start with resource usage metrics, but mature into measuring software from the customer and business perspective.

The ability to see information-rich logs can help you understand how faults occur so you can design a more robust system.

Other popular features

Though they didn't make it into the top 5, documentation, security scanning, one-click setup for new projects, artifact management, and secrets management are common features of internal developer platforms.

The features are useful on their own, but when combined, they amplify their effects on overall software delivery and operational performance. Platform teams who provide a seamless toolchain from commit through to production and on to day 2 operations will find they have lifted a massive burden from developers.

What high performers do differently

The high-performing organizations used the amplification effect to provide a complete toolchain for deployment pipelines. Platforms that provided this set of functionality had more impact on the organization and its goals for Platform Engineering.

Builds
Test automation
Artifact management
Deployment automation
Infrastructure automation
Monitoring and observability

By creating a strong pipeline with equal consideration for long-term sustainability of the platform, you can create an offering that helps developers today and continues to help them as the organization navigates changing requirements, regulations, and competition.

We also looked at organizations with low performance; they were often missing 3 key features:

Test automation
Artifact management
Monitoring and observability

If your platform is missing any of these, you may be missing an opportunity to amplify the benefits of platform adoption.

Platform success

Throughout our study, we used each organization's individual definition of success for its platform. There are many different motivators for funding a platform initiative, and the ultimate test of a platform is whether it meets the specific goals it was founded to achieve.

You can use the MONK metrics to measure your Platform Engineering initiative. They balance benchmarkable metrics with contextual measures that track the platform's progress against the organization's goals.

Despite the many different ways platforms are measured, the presence of a complete feature set increased the chances of success.

Happy deployments!

How organizations measure Platform Engineering

Fri, 09 Jan 2026 07:40:46 +0000

Our Platform Engineering Pulse report is coming soon, containing insights, strategies, and real-world data on how organizations adopt and succeed with Platform Engineering. We’ll also launch a survey to deepen our understanding of the patterns, challenges, and future direction of successful platform building.

One of the areas we explored in the Platform Engineering Pulse report was how organizations measure their internal developer platforms, with results that varied from technical measures to not collecting any metrics at all.

This post looks at which metrics were popular and how you can use them to structure your measurement approach to avoid the zero-metric illusion.

Measurement is essential

At the start of a Platform Engineering initiative, the immediate problems take precedence, and measurement becomes an afterthought. This leaves platform teams without a pre-platform baseline, which makes it challenging to demonstrate the platform’s impact.

The lack of measurement also increases the risk of platforms adding features that don’t align with the organization’s core motivations. For instance, a platform team might focus on standardization, only to discover that the primary driver for investing in Platform Engineering was to improve developer experience.

Platforms often fail not because they are inherently bad, but because they target the wrong personas and attempt to solve the wrong problems.

Even successful platforms struggle with ongoing justification and optimization when they lack clear goals, a robust measurement system that reflects those goals, and baseline data. The platform team may find leadership challenges continued investment if they can’t show the platform’s value.

Therefore, it is crucial to measure platform performance against its established goals. But how are organizations currently approaching this?

What organizations measure

The survey found organizations focused on 3 key areas for measurement: Software delivery, operational performance, and user experience. The variety of metrics likely reflects the variety of contexts that platforms can assist, though is also evidence that metric systems are skewed to the elusive productivity problem rather than true developer experience and engagement.

Software delivery metrics

The software delivery metrics assess the speed, quality, and throughput of development teams using the platform. They often align with DORA measurements, highlighting how platforms are seen as a route to shipping software more often and at higher quality. The metrics encompass throughput (e.g., deployment frequency, build time, features delivered) and stability (e.g., change failure rate, build success rate).

Deployment frequency
Deployment times
Change failure rate
Recovery time
Build success rate
Build time
Features delivered

While valuable for understanding delivery performance, these metrics primarily reflect downstream outcomes rather than the platform’s direct contribution to developer productivity. They are most effective when measured before and after platform adoption to demonstrate improvement.

Operational performance metrics

Metrics for operational performance track the cost, performance, and efficiency of applications that use the platform. They combine traditional infrastructure monitoring (e.g., reliability, error rates, system performance) with resource optimization (e.g., usage efficiency, cost management). Project count is a basic adoption indicator, but it lacks the context of the addressable market size.

Reliability
Error rates
System performance
Resource usage
Infrastructure cost
Project count

These metrics are crucial for demonstrating that the platform is operationally sound and cost-effective, but they don’t necessarily indicate developer value or ease of use.

User experience metrics

These metrics directly measure how developers and teams perceive and interact with the platform, treating internal developers as customers. They focus on satisfaction and onboarding journeys. Net promoter score (NPS) offers benchmarkable sentiment data, while user satisfaction provides broader feedback. Onboarding time is critical as it represents the first impression and adoption barrier.

User satisfaction
Net promoter score (NPS)
Onboarding time

These metrics are vital for platforms run as products with optional adoption, indicating whether the platform is compelling enough for developers to choose and continue using voluntarily.

Most organizations prioritize technical and delivery metrics, with fewer focusing on user experience or business outcomes. This suggests a risk that platform teams are measuring what’s easy to collect rather than what truly demonstrates business value. The heavy emphasis on technical metrics indicates that many organizations still measure platforms like infrastructure rather than products serving internal customers.

How measurement improves platform performance

For metrics to improve platform performance, you need to measure multiple dimensions, not just one. Internal developer platforms offer many benefits, so the measurement system should cover all platform goals.

We found organizations that measure more dimensions were more likely to be successful. A single metric gives you a one-in-three chance of creating a successful platform, while 2 metrics make it 50/50. Organizations measuring 6 or more metrics were most likely to be successful.

Platform sponsors may expect you to deliver technical reliability, boost developer productivity, offer a positive user experience, manage costs, encourage adoption, and align with business objectives. Relying on just a handful of metrics cannot adequately capture the interplay of all these dimensions.

Breaking the success illusion

The data shows a zero-metric high-success effect. This occurs when organizations that don’t collect any concrete measures of the Platform Engineering effort report high success rates. This phenomenon comes from two very different situations masquerading as the same outcome.

There may be easy and obvious success criteria the platform can address, such as a critical and evident problem where success is undeniable without formal measurement. Where the effectiveness is apparent, formal measurements may be unnecessary. A more likely explanation is that there’s an illusion of success caused by a lack of measurement.

Without concrete metrics, platform teams can focus on outputs (e.g., features built, no outages) rather than actual outcomes (e.g., increased developer productivity, achievement of business goals). Stakeholders may mistake activity for impact, especially if the team is busy and there are no apparent failures.

This also helps explain the dramatic increase in success rates observed when moving from one metric to three or more. Teams relying on a single metric might choose one that doesn’t capture holistic success, leading to a false sense of achievement. However, when multiple dimensions are measured, maintaining illusions becomes difficult. For instance, you can’t claim unqualified success if deployment frequency is high but Net Promoter Score (NPS) is low.

The data also suggests a dangerous middle ground where minimal measurement can create more problems than no measurement. It provides false confidence without the comprehensive feedback necessary for genuine improvement.

Using MONK metrics

MONK metrics offer a balanced approach to measuring Platform Engineering success, combining external validation and internal alignment. This framework is more adaptable than purely technical metrics, yet still provides a standardized basis for comparison and improvement.

The MONK metrics are:

Market share
Onboarding times
Net Promoter Score (NPS)
Key customer metrics

The first three, market share, onboarding times, and NPS, form a benchmarkable trio. These metrics allow for comparison against industry standards and are broadly applicable to Platform Engineering initiatives, serving as a starting point to understand high performance in other organizations. For example, if your onboarding times are slower than those of your industry peers, you can investigate their methods and implement similar improvements.

Including “key customer metrics” is crucial for preventing the measurement framework from losing touch with business realities. Organizations kick off Platform Engineering to solve specific organizational problems, but their success is measured using generic technical metrics that don’t reflect the original investment’s purpose. Instead, it’s essential to translate the motivations behind adopting Platform Engineering into the key customer metrics to track progress towards the platform’s objectives effectively.

MONK metrics address the common issue of platform teams optimizing for metrics that appear favorable but don’t contribute to actual business value. For instance, if you introduce Platform Engineering to reduce time-to-market, measures of infrastructure uptime or build failures fail to track tangible improvements in delivery flow.

Lack of measurement makes your platform vulnerable

Without measurement, platform teams end up in a vulnerable position. They can’t demonstrate their value when budget discussions come up, they can’t identify what’s working versus what needs improvement, and they can’t make compelling cases for continued investment or expansion.

The absence of data also hinders course correction. Platform Engineering involves complex trade-offs between developer experience, operational efficiency, security, and cost. Without clear metrics, teams may prioritize visible or politically expedient solutions over those that deliver actual value.

MONK metrics offer a practical solution. They are accessible, enabling organizations to begin measuring with minimal tooling or data infrastructure. You can gather basic market share data through surveys, track onboarding times simply, and get NPS using lightweight feedback tools.

Their benchmarkable nature also helps address the common “what’s good enough?” dilemma. Instead of abstractly debating whether a two-day onboarding time is acceptable, teams can compare their performance to similar organizations.

SHIPPED25: DORA... Beyond the metrics

Wed, 10 Dec 2025 10:18:24 +0000

Most software teams have heard of the "DORA Metrics" or the "DORA 4 Keys", but there is so much more to the research. This session looks at those well-known metrics, but then takes you beyond them to some even more crucial insights into what great software delivery looks like.

Nathen Harvey and Steve Fenton introduce some key concepts, then abandon all slides to get hands on with some tools and models you can use to go beyond throughput and stability and out into the world of capabilities and continuous improvement.

Key insights

Throughput and stability aren't trade offs when you set teams up in the right way
How time to restore has evolved into failed deployment recovery time and rework rate
How to use the model to get better and getting better
How to use the assessment tool with your team to decide where to focus

Common mistakes in DevOps metrics

Tue, 18 Nov 2025 12:41:44 +0000

Metrics are crucial to DevOps and Continuous Delivery as part of the process of continuous improvement. However, you must balance collecting and displaying data with being flooded with information. You need to decide what data to collect and what smaller set of data you pay attention to at any time.

If your car had a dashboard that displayed every metric it collects through the engine management system, there'd be no space for a windscreen.

Early cars only featured an ammeter on their dashboard, which measured the electric current between the battery and voltage regulator. This was important as it told you the charging system was working. There was no speedometer. Cars could only achieve a top speed of 35 miles per hour, and the suspension discouraged traveling at this speed. It wasn't necessary to measure velocity.

In a modern car, no dashboard space is given to an ammeter (though a battery light will illuminate if there's a problem). But, you will find a speedometer in almost every car. The current dashboard design reflects the development of cars and the broader system they operate in. Engines are more powerful, suspension systems are better, roads are generally smoother, and more cars are on the road. Attitudes towards safety have also changed.

Similarly, the metrics you collect and show will change over time as you respond to differences in your team and organization.

As you create and evolve your measurement system, there are some common mistakes you should avoid.

Ignoring the data

The first problem with metrics is that you expend great effort collecting them, but they're not always used. This happens even in organizations that regularly review the numbers.

Your data needs some process that generates insights. You can turn what you discover from the information into theories you use for experiments. The experiments should then provide new data to start the cycle again.

The only good reason to collect DevOps metrics is to learn more about your work and find ways to improve it. If you collect data just in case, the chances are it will be misused or not used at all.

Activity bias

There are 4 levels of measurement, with activity often being the earliest and easiest to collect. You can usually track activity in near real-time. The result of the activity is not typically available as an output or outcome-based metric until some later date.

Measurement level	Software example	Heating example
Activity	Lines of code	Power consumption
Outputs	Features per week	Heating element temperature
System output	Lead time	Room temperature
Outcomes	User value	People are comfortable

Activity metrics are usually a built-in feature of your existing tools, so they're already available. The problem is not all activity represents progress. Some activities might even reduce output and result in worse outcomes.

You can use activity metrics to predict future changes to outputs and outcomes. To do this, you must continually test the relationship between your activity metrics and your related output or outcome metrics.

If you only measure activity, you get a lot of motion but no progress.

Tracking too much at once

The number of metrics you collect and display can increase, often rapidly. Before long, you've filled your dashboard with charts, and you don't know what's important and what isn't.

You need to keep your tracked metrics lean, current, and relevant. When a chart is no longer helpful, you should remove it from your dashboard. You should also consider whether the metric still needs to be collected and retire it if you don't have a good reason to track it.

If you already have a dashboard, open it, and for each chart ask, "What would I do differently if this number went up or down?". Revisit this question frequently, deleting any chart where you don't have an answer.

Your metric set should focus on key long-term output and outcome metrics, with dashboards showing short-term metrics across all categories in your current improvement efforts.

Jumping into tools

Data visualization tools like Microsoft Power BI, Tableau, or Google Data Studio are among the most useful software products you'll have in your organization. Many business tools have a grid or text-based interface, but data tools come stacked with colorful animated charts.

It's easy to be distracted by creating an engaging dashboard. If you don't begin with metric design, you end up with lots of pleasing dashboards that have no impact on your daily work. You need dashboards and charting tools to help you make sense of the information, but design the metrics first.

It's better to start in low-fidelity to collect meaningful metrics. It's okay to start with a simple spreadsheet or even a whiteboard. After you work out which measurements are helpful to your team and organization, start automating the collection and creating slick displays.

If you invest too much time creating a stunning dashboard, you'll find it harder to delete charts when they're no longer required.

Standardization

Some organizations try to replicate the success of a high-performing team by making other teams follow the same process. This is rarely successful, as each team works on different problems and has different skill levels. Just as the process and practices need to be context-specific, so do the metrics.

You should address metrics as part of your continuous improvement activities. For a team with long lead times, you measure batch size, queue times, and the time work spends in each state. These metrics are unsuitable for a team whose primary problem is quality.

This requires the data, insight, theory, experiment cycle where you look at the information you have about the problem you want to solve, form a theory about what might improve the issue, then run an experiment to test your idea.

The metrics you collect also signal to the team what's important at the current time. You often see improvements simply because your metrics communicate that you care about some aspect of software delivery.

Relying on eyeballs

You should create a simple dashboard to display the metrics you're tracking for experiments. This should be displayed on an information radiator so everyone on the team can see the data.

However, if you only react to data when someone looks at a dashboard, you'll crowd the dashboard with too much information and miss key events. Long-term metrics you keep to track progress over time will seem as important as the short-term metrics you use to improve your software delivery.

The key to resolving the problem of retaining long-term metrics without cluttering your dashboard is to put in place a monitoring and alerting process for the data. Automated alerts should tell you if the metrics cross a threshold, and you could use anomaly detection to tell you if something interesting happened.

With metrics tracked automatically, you can remove them from the dashboard to free up space.

Rewarding performance

If your team is working to increase their deployment rate, it can be tempting to incentivize them with a reward if they achieve daily deployments. This approach to motivation leads to poor outcomes. A team might let other critical work slip to achieve the goal - not to cheat the system but because you made daily deployments more important than anything else.

In the landmark book Punished By Rewards, Alfie Kohn explains that attempting to manage people with incentives leads to long-term harm for your organization. Hundreds of studies have found that people do worse work when offered rewards.

Using metrics to create a competitive atmosphere, either for individual performance, comparison of different teams, or to gamify the workplace (where you introduce game elements as a form of "fun" competition), all lead to trouble.

Competition conflicts with what you really need in your organization: collaboration. If you work on the assumption that people want to do good work, you'll find you don't need to use rewards or punishments to get them to improve their performance.

Conclusion

The 5 DORA metrics and the SPACE framework provide pre-built, balanced ways to measure software delivery performance. (There used to be 4 DORA metrics, but an additional reliability metric was added.)

A good set of metrics will mix leading indicators to predict performance with lagging indicators that check the accuracy of predictions. Measurements should span activity, output, system output, and outcome categories.

We cover DevOps and Continuous Delivery metrics in detail in our white paper on measuring Continuous Delivery.

Our DevOps Insights in Octopus give you better visibility into your company's DevOps performance by surfacing insights based on 4 key DORA metrics. These metrics help you qualify the results of your DevOps performance, and gain insights into areas for improvement.

Whatever you measure, you need to constantly refine your metrics to ensure they remain useful to your team and organization. Ideally, the metrics you collect are relevant to a specific experiment you're running to test a theory.

If you use metrics well, you amplify performance and learning as you seek to become one of the elite performers in software delivery.

Measuring Platform Engineering with MONK metrics

Wed, 05 Nov 2025 14:29:12 +0000

When you create an internal developer platform, treating the platform as a product is crucial. That means you need to measure progress with a product mindset, too. The MONK metrics help measure your progress and prove the value of Platform Engineering to your organization.

Measure Platform Engineering with MONK metrics

MONK metrics mix platform adoption and performance with metrics to show impact on the platform’s customers; the developers:

Market share
Onboarding time
Net Promoter Score (NPS)
Key customer metrics

Your market share is the number of developers who use the platform rather than an alternative. You must run a regulated market, so you can’t force people to use your platform. Developers who opt not to use the platform must have the same responsibility to deliver security, reliability, and governance requirements.

Developers should have the option not to use the internal developer platform. The platform team should respond competitively by improving the platform to gain market share. If developers move off the platform, find out why. Take a second look at your relationship with developers and explore why they didn’t flag an issue before moving away.

To understand your market share, you should track the number of developers:

In the organization
The platform could help
Using the platform

When you need to choose between possible features for the platform, it helps to consider the impact it will have on market share. What problems are causing pain for development teams that your platform could solve? Understanding the share, and the trend over time, can help your decision-making.

You can increase your market share by supporting more scenarios or convincing teams to move onto an existing ‘golden pathway’. Avoid adding workflows you can’t do well and focus on spreading the joy developers will experience by adopting one you can handle.

Onboarding time

Your internal developer platform should shorten the time it takes to get a new developer’s first change into Production. New developers should find it easier to start as the platform simplifies workflows and removes the need to learn some underlying technologies.

You should also track the onboarding experience for teams adopting the platform. Opting in should be easy, and the benefits should be quickly apparent. Developers don’t have weeks to invest in platform adoption, so you need to make the onboarding process swift and easy. Building case studies with teams using the platform will help convince others to consider it.

Net Promoter Score

Net Promoter Score (NPS) is a popular market research metric used by two-thirds of the Fortune 1000. Your organization may already collect this from end customers, but you must also collect the platform NPS from the developers. You only need to ask one question to collect NPS, collecting a score from 0 to 10 as the answer:

How likely are you to recommend the internal developer platform to another developer?

You can group responses based on the answer:

0–6: Detractors
7–8: Passives
9–10: Promoters

To calculate the score, take the percentage of promoters, and subtract the detractors.

Score	% of responses
0-6	20
7-8	35
9-10	45

With the table above, you can ignore passives (35%) and calculate the NPS as promoters (45%) minus detractors (20%). So, a net promoter score of 25.

Positive scores are good and negative scores need improvement. The software-as-a-service industry has an NPS in the range of 30 to 40, according to Beamer and userpilot.

A downward trend in NPS can show the platform fails to address an essential developer pain. It may also suggest developers are considering a move away from the platform.

When you collect NPS, you should ask further questions, not least why the user chose their score. This provides valuable product feedback. Don’t pester the same customer for feedback too often and limit how many survey questions you ask as you want to show you respect their time.

Key customer metrics

Your customers, the developers, are also tracking their own success metrics. For example, they may use DORA metrics or the SPACE framework. When talking with your customers, ask what measurements they care about and find ways to track them.

Many platforms are introduced to improve developer experience, reduce cognitive overload, and prevent burnout. Whatever motivated your organization to invest in a platform will be something to track over time.

You can help developers achieve or exceed goals by adding needed features to the internal developer platform. If you do this, they’ll love the platform and encourage others to adopt it. The DevOps capability model can help find techniques or practices you can add to your golden pathways to improve software delivery and operational performance.

Summary

Don’t only collect numbers. Regular conversations with developers are crucial whether they use the platform or not. Your customers (and potential customers) know their pain points and can help you find areas to help.

Regular customer interaction shows you run a healthy Platform Engineering effort.

MONK metrics help you:

Define the product roadmap for your internal developer platform
Start essential conversations with customers
Show the platform’s value to the people who pay to build it

MONK metrics should be part of your product management strategy. You also must speak to developers to find out more and keep track of the problems they care about.

Developer experience metrics

Mon, 03 Nov 2025 15:19:44 +0000

Developer experience (DevEx) is a key focus of platform engineering. It's also a crucial success factor for any organization developing software.

A recent paper by Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Freiler called DevEx: What actually drives productivity provides a framework for measuring and improving DevEx.

Developer experience

Developers tend to be happier when they can be productive. When a developer experiences friction, it’s a sign that they can’t deliver as much value as they feel they could.

The following can cause poor developer experience:

Working on low-value tasks
Interruptions
Poor tools slowing them down
Unrealistic deadlines

It’s hard to calculate the volume of an irregular object. Archimedes discovered that submerging an object in water provides a simple way to find its volume. Similarly, improving how developers feel and think about their work and its value can increase productivity.

Some benefits of improving developer experience are:

Increased productivity
Higher satisfaction
More engagement
Higher employee retention

Core dimensions

The 3 core dimensions categorize the friction developers experience:

Feedback loops
Cognitive load
Flow state

These dimensions are a valuable model for improvement.

Feedback loops

Software delivery involves many feedback loops:

Build and test times
Code review waits
Manual testing
Real user feedback

Each of these loops needs to be short to reduce friction. Ideally, all feedback loops should complete while the task is still active. If a task pauses during a feedback loop, it will interrupt the following work when it restarts, disrupting flow state and increasing cognitive load.

Organizations are twice as likely to exceed performance goals with short lead times and frequent deployments. Work in smaller batches to get shorter feedback loops.

Cognitive load

Writing and maintaining software involves a large amount of mental processing. This natural cognitive load for the task increases when developers have too many tools and technologies. The architecture of the software can also add to the burden.

You can reduce cognitive load if you:

Reduce toolchain friction
Keep documentation updated
Improve the system’s architecture
Remove delays in the process

Platform engineering is a is a strategy for reducing cognitive overload caused by tools and technologies.

Flow state

You can probably remember when you last felt in the flow. You may have had a good chunk of uninterrupted time and could focus on the problem. The code seemed to write itself and you made great progress. You enjoyed the work.

Psychologist Mihaly Csikszentmihalyi studied the concept. He described it as a fully immersed feeling of energized focus.

Flow happens naturally when you have:

Control over your work structure
Clear goals
Engaging work

It’s not a case of making flow happen but preventing disruption from interruptions and delays.

Studies show developers who enjoy their work perform better and produce higher-quality products.

You can make flow more likely by:

Using asynchronous communication to avoid interruptions
Clustering meetings to create large blocks of meeting-free time
Avoiding unplanned work
Batching questions

DevEx metrics

You can create DevEx metrics by measuring 3 core dimensions with 2 methods:

Perceptions — Gathered through surveys.
Workflows — Gathered from systems.

Many people question the value of perceptual measurements. They think they’re too subjective to be useful. However, as our white paper on metrics explains, they’re a vital reality check.

The whole point of DevEx metrics is to uncover ways to increase the 3 core dimensions. They’re not a measure of developers but for the benefit of developers.

The matrix below shows examples of the metrics you can collect. It’s intended to be a framework, so you can pick and change metrics as needed.

Dimension	Perceptions	Workflows
Feedback loops	How satisfied are you with (build times, test run speed, the time it takes to get a change live)	Build times, code review waits, deployment frequency, lead time for changes
Cognitive load	How complex is (the codebase, debugging the system, understanding the documentation)	The number of manual stages in the deployment pipeline, lead time for technical questions
Flow state	Are you able to (focus on work, avoid interruptions, understand your tasks)	Meeting free blocks, incident frequency, number of unplanned tasks

If you’re already collecting the DORA metrics, they’re helpful here.

DevEx key performance indicators

You must pick key performance indicators (KPIs) to track success. You can base these on the metrics you selected as part of the dimension and method matrix.

Perceived ease of delivering software
Perceived productivity
Employee satisfaction

The KPIs make sure you remember the purpose of measuring developer experience. They also help balance the trade-offs in the improvement process. You should measure the outcomes you want from your efforts.

Summary

Developer productivity measurements are traditionally elusive. However, you can use perceptual and workflow measurements to track and improve feedback loops, cognitive load, and flow state.

Using KPIs to improve these 3 dimensions will get you happier developers who help your organization achieve its goals.

Relax About Your DORA Metrics

Tue, 09 Sep 2025 13:04:02 +0000

Our jobs require a certain level of precision, so it’s natural that we’ll want to take an exact approach to our software delivery metrics. Some folks might even like a detailed standard to pin their measurement to so they can be sure it’s exactly right.

Investing too much in precision is wasteful and takes time away from something far more important: doing something in response to the data.

Why Precision Doesn’t Matter

The DORA metrics are the most popular DevOps metrics. The precision needed for DORA metrics is low, and gets even lower the higher you perform. In general, you want to get a temperature check for the kind of performance you have. For the throughput metrics, the unit of measurement is more informative than the number.

It’s more interesting whether you measure throughput in months, weeks, days or hours than it is to split the hairs to get the exact number. If you deploy once a month, your next milestone will be when you can deploy once a week. When your lead time for changes is a week, your next milestone is to make it a day.

Whether you deploy once every 26 days or every 30 days, you’re effectively deploying monthly. This matters because the improvements you’ll make to shave a day or two off a monthly deployment aren’t the changes you need to get to weekly deployments.

What About Marginal Gains

The Pareto principle, or the 80/20 rule, states that you get roughly 80% of consequences from just 20% of causes. For your improvement efforts, doing relatively simple things like adding deployment and test automation will provide a significant boost. As you expand your adoption of practices, you’ll find it harder to get the same returns for a similar level of investment.

The concept of marginal gains comes from elite sports. By making tiny improvements to technique and equipment, it’s possible to gain a cumulative effect that wins races. The British cycling team pursued this technique, and by looking for 1% gains in many areas, they achieved an incredible streak of wins in the Olympics and the Tour de France.

Marginal gains may be helpful to you in a highly competitive situation, but while you’re deploying monthly, you’re not yet performing at a level where you need to look for marginal gains. Marginal gains answer the Pareto paradox, which only presents itself once you’ve done all the obvious things and must now consider the other 80% to get the next improvement.

To refer back to the cycling example, if your bicycle is still made of heavy steel tubes, you are better off switching to a lighter material than trying to modify the aerodynamics. You should be working on the 20%, not looking for marginal gains among the often-ignored 80%.

When you have on-demand deployments, a low change failure rate, and lead times and recovery times measured in hours or minutes, you can switch to marginal gains if software delivery is still the constraint in your organization.

Unsurprisingly, if you can deliver to this level, the appetite for further improvements diminishes. Focus usually moves elsewhere, such as improving the hit rate for new features by connecting more closely to the software’s users.

It’s All About Improvements

The most common reason precise metrics are requested is that they’re about to be used in an unhealthy way. If you want to stack-rank teams based on software delivery performance, you’ll want to ensure the measurement is done the exact same way in every team. Without precision, the comparison may not be correct.

The four key DORA metrics don’t serve this purpose. The goal of tracking software delivery performance is to improve it. The only valid comparison for improvement is to track the same team and application over time to see how it trends.

When you collect the metrics for improvement, you only need to use them to inform your retrospectives or continuous improvement process. The unintentional side effect of our calendar system is that quarterly to monthly is a three times improvement. It requires an impactful change, and it’s easy to see the difference it makes. Similarly, monthly to weekly is 4x, weekly to daily is 5x, and then you deploy on demand, and it’s no longer the area to focus on.

When you’ve reached high-performance levels, you must zoom back up to view the whole value stream to find where improvement efforts are best directed. Increasing your lead time for changes from 1 hour and 20 minutes to 1 hour and 10 minutes isn’t the best use of your time.

How To Define Your Metrics

To get started, you don’t need any formal metric solution to use DORA’s four keys. You can open the quick check and use the questions to start a conversation with your team. For something like deployment frequency, you probably know off the top of your head whether you deploy quarterly, monthly or weekly. Other metrics, like change failure rate, will need more discussion.

You’ll discover many interesting details as your team develops its thoughts around defining the metrics and where to get the data with minimal fuss. Should the change failure be measured from when the fault is reported, when it was introduced or when it first affected a user? Is it possible to be sure of the exact date and time? Are pull requests included in the lead time for changes?

While those of us in the community have strong opinions on these, nobody is closer to the work than your team. You understand how things work and are smart enough to collaborate on a lightweight definition that’s easy to obtain and good enough to inform your improvement process.

If your team all agrees on the definition, it doesn’t matter what I say you should do. For example, I strongly believe pull requests and code reviews should be included in the lead time for changes because manual stages in your deployment pipeline are the most fruitful targets for improvement. If your team disagrees, that’s OK.

Teams must be allowed to vary their measurement methods to decide what is useful for them and what they need for their improvement process. They might choose a wrong path, but a team applying continuous improvement will self-correct over time.

Beyond Software Delivery

Software delivery is part of a wider system. You are working on a code change because there is some value to be unlocked. It’s worth introducing measures that make the state of the end-to-end system visible so you can work out where to target your improvement efforts.

Lead and cycle times can help you find the bottlenecks, and understanding the success of features can bring vital knowledge to your business so you don’t waste time delivering software changes that aren’t useful to your users.

When you have great performance against the DORA metrics, spend your time looking for something new to measure instead of trying to achieve even higher performance for no meaningful gain.

How Measurement Elevation and Aggregation Change Behaviors

Thu, 12 Jun 2025 14:54:14 +0000

The costs of measuring the wrong thing or the right things in the wrong way vastly outweigh the costs of collecting and processing the data.

In software delivery, conversations about metrics and measurement wax and wane as people discover useful techniques or get hurt by unintended consequences. Understanding how measurement elevation and aggregation levels change people’s behaviors and business outcomes will help you create a healthier metrics system that better informs your decisions.

The Economics of Measurement

A measurement system is subject to economics. There is a cost of collecting data and a value in using it. The economics, particularly the cost, are commonly overlooked. The widespread nature of computer systems means many leaders believe the cost of collecting metrics is negligible, which causes them to misunderstand these trade-offs.

For example, the cost of getting a time series of data onto a graph is very cheap, but the consequences of concentrating on this data can be surprisingly high. When you publish metrics, you signal that they are important. It’s natural for people to believe you want to see some form of improvement on those metrics, and they will make decisions based on this signal (whether or not that’s your intent).

If you shift people’s attention to the wrong measurements, the cost to your business could be huge.

The Trouble with Measurement Elevation

One classic problem with metrics occurs when they are elevated inappropriately. Measurements useful to one team can cause problems when shared higher up the organization. Software delivery metrics can be a helpful way for teams to inform their continuous improvement (CI) activities. However, when the same metrics are elevated throughout the organization, they may cause undesirable effects on behavior, especially if they become key performance indicators (KPIs).

Metrics are a way to make sense of abstract concepts. In business, they are used to connect actions across the organization back to the company strategy. Problems occur if you put more emphasis on the metrics themselves than the concept they represent or if they don’t appropriately represent the concept. When you fixate on metrics, you can force people to do clearly unwise things in their service.

For example, I worked on a team that missed a crucial release date for a finance system. One of the lead developers stopped working on the project to write a training guide. This behavior damaged the team and organization, but the developer was given a goal to write five training guides and their annual review was coming up. If they had done the right thing and helped with the release, they would have missed their goal-related bonus.

Was this the team member’s fault, or were they doing exactly what the system demanded?

I recently collaborated with Google and DORA on an article about how business leaders can empower software delivery teams that discusses how to avoid the elevation problem.

Problems with Measurement Aggregation

Aggregation levels are another crucial factor to consider in designing measurement systems. Measurement aggregation refers to the data you can measure at different organizational layers. On a basic level, you could measure at any of these levels:

Individual
Team
Business unit
Organization

I’ve simplified the levels to make it easier to apply this concept across multiple organizations. You’ll need to determine how this concept fits your organization’s context and structure.

Measurement aggregation could involve combining data or displaying multiple sets. For example, aggregating data at the business-unit level might mean calculating averages across all teams in the unit, or it might involve plotting each team as a separate line on a chart.

Aggregation changes the economics because the effect on behavior will change based on which organizational level you apply the measurement. For example, measuring performance at the team level encourages team members to help each other succeed, whereas measuring this at an individual level will reduce attention on crucial tasks if they don’t directly impact individual performance. A rough sketch of this change is shown below, with the measurement’s value dramatically changing with each movement through the layers.

This chart is easiest to explain by starting at the team level. When you measure a team, you encourage the members to help each other improve their performance over time. Junior team members are coached by more experienced people, and the process adapts over time. Lots of great things happen as the team finds better ways of working. This metric has high value at the team level.

But if you apply the same metric at the individual level, many of those beneficial behaviors stop. Time spent coaching other team members directly lowers an individual’s score. Time spent on automation and process improvement is more relevant to hitting your own numbers than increasing someone else’s.

You can’t blame someone for doing things that result in achieving the numbers you decided to measure — even if those things may seem wrong. When you make a metric too important, it changes how people perceive their work in ways you might not predict.

But what if you move up a level from the team to the business unit? You now have a problem with the measurement competing with contextual noise. One team may be working on a modern application with a loosely coupled architecture, another might be wrestling with heritage software that’s not factored well. One team’s struggles might be hidden by the another team’s improvements. The measurement’s value is reduced because the signal isn’t detectable.

A common solution to this problem is to measure at the team level and elevate the metrics for comparison at the business-unit level. However, great care must be taken here to avoid causing unintended competition or creating teams nobody wants to work on.

If you shift up to the organizational level, you once again obtain highly valuable information. This is the level where you are closest to measuring outcomes. Not only can these metrics help you tell whether you are profitable and hitting goals, but you can also assess whether improvements at the team level are translating into tangible results. (Although there is usually an unfortunate delay in confirming that leading indicators are converting into bankable results.)

Balancing Competition and Collaboration

The key to understanding the complete reversals at each level of aggregation is that they naturally tend toward either competition or collaboration. In knowledge work, collaboration is usually the best path to positive outcomes for the people, the organization and its customers.

The team and organization levels are units of collaboration. Individuals and business units are often units of competition. There are healthy places for competition — usually between your organization and others offering a similar product or service. That’s why they’re called competitors.

In business, collaboration gets you further than competition.

Healthy Measures Make a Healthy Organization

Understanding measurement economics, aggregation and elevation will help you design a system that encourages positive behaviors, those that leverage human ingenuity and collaboration to create value.

For more insight on designing healthy measurement systems, consult my white paper Measuring Continuous Delivery and DevOps.

When it comes to the economics of measurement, the costs of measuring the wrong thing or the right things in the wrong way vastly outweigh the costs of collecting and processing the data.

Coder Legion - Recent posts tagged metrics

Centralizing Observability: Introducing the NKE Metrics Agent

The Challenge with Prometheus

Enter the Metrics Agent

How It Works

Pricing

Migrating from Prometheus

Getting Started

One Platform for All Metrics

My Phone Was Talking Behind My Back. NextDNS Caught Every Word

What actually showed up in the logs

Turns out this isn't a vibe, it's a measured, peer-reviewed fact

Google isn't just an app on your phone, it's most of the pipe your phone talks through

The short list of companies who actually run your digital life

So what do you actually do with this

Prometheus 3.0 and OpenTelemetry: Native OTLP Support Explained

What Changed in Prometheus 3.0: The OTel-Relevant Picture

Native OTLP Ingestion

UTF-8 Metric Names

Remote Write 2.0

Remote Write 2.0

New UI

Breaking Changes Summary

The OTLP Receiver: How It Works and What It Accepts

What It Accepts (and What It Does Not)

Enabling the OTLP Receiver

Resource Attribute Promotion Strategy

UTF-8 Metric Names: Why This Is the Real Game-Changer

Enabling UTF-8 Metric Names

OTel SDK to Prometheus 3.0 Directly: No Collector Required

Go (OpenTelemetry SDK)

Python (OpenTelemetry SDK)

Java (OpenTelemetry SDK with Spring Boot)

OTel Collector to Prometheus 3.0: When You Need the Intermediary

Pattern 1: OTel Collector as Fan-Out Gateway

Pattern 2: Collector for Metric Transformation

Pattern 3: Collector for Traces and Logs (Always Required)

When to Use Native OTLP vs. OTel Collector: Decision Framework

Measuring productivity without freaking people out

How To Measure AI’s Organizational Impact

Your Goals Haven’t Changed

Stick With What Already Works

The Local Optimization Trap

Focus On What Truly Matters

Research-Driven Implementation

The Path Forward

The top 5 features of internal developer platforms

1. Build automation

2. Deployment automation

3. Infrastructure automation

4. Test automation

5. Monitoring and observability

Other popular features

What high performers do differently

Platform success

How organizations measure Platform Engineering

Measurement is essential

What organizations measure

Software delivery metrics

Operational performance metrics

User experience metrics

How measurement improves platform performance

Breaking the success illusion

Using MONK metrics

Lack of measurement makes your platform vulnerable

SHIPPED25: DORA... Beyond the metrics

Common mistakes in DevOps metrics

Ignoring the data

Activity bias

Tracking too much at once

Jumping into tools

Standardization

Relying on eyeballs

Rewarding performance

Conclusion

Further reading

Measuring Platform Engineering with MONK metrics

Measure Platform Engineering with MONK metrics

Market share

Onboarding time