[simlarsen]

The customer's report, checked

A customer running oneuptime/docker-agent:release (agent 12.0.20) saw container.cpu.utilization as a flat line (0.07% on one host, 7.17% on the other, for the same container using about 1% of a core) and container.memory.percent exactly 100 times too high (traefik: 175 MB of a 4.01 GB limit, 4.36%, shown as 436.64%). Both are real, both are still there on master's agent, and they have different causes.

1. CPU: the docker_stats receiver computes it from one reading (agent / upstream)

The Docker, Podman and Docker Swarm agents sent container.cpu.utilization as the collector's docker_stats receiver computes it: (Δ container CPU / Δ host CPU) × online CPUs × 100, with the previous reading taken from precpu_stats of the same stats response. Since collector-contrib 0.150.0 the receiver's Docker client (moby/moby/client v0.4+) calls ContainerStats with ContainerStatsOptions{}, which sends stream=false&one-shot=true: the daemon returns a single sample and precpu_stats is empty. The receiver then divides all the CPU the container used since it started by all the CPU the host used since it booted - a lifetime figure, diluted by host uptime, that barely moves and never follows the load. (Checked in the contrib source for every tag 0.118-0.163 and on main: 0.149.0 and earlier asked for the previous sample, 0.150.0 and later do not; nothing upstream fixes it yet.) OneUptime's agents ran 0.118.0 until June 2026 and 0.154.0 / 0.161.0 since, so every Docker and Swarm agent from June on has the flat line. Podman hosts always had it: Podman's Docker-compatible API fills precpu_stats only when streaming.

Reproduced on this machine (Docker 29.8.1, collector 0.161.0, the shipped receiver block): a busybox container held to a quarter of a core (--cpus=0.25, busy loop) read 0.0140, then 0.0141, while container.cpu.usage.total rose by exactly 25% of a core per second.

2. Memory (and CPU) percent: declared "1", read as a 0-1 fraction (OneUptime)

docker_stats (and podman_stats) send container.memory.percent and container.cpu.utilization as a percent (0-100, CPU above 100 on several cores) but declare the unit 1, the UCUM unit of a 0-1 fraction. OneUptime keeps one unit per metric name per project (MetricType) and read them as fractions everywhere a unit decides a number: the metrics explorer and dashboards (ValueFormatter: unit 1 + a .percent / .utilization name → ×100), the Docker / Podman host Metrics tab, container detail charts and Swarm Insights (MetricResultUnitConverter: 1 → % legend is ×100), generic Metrics monitors' % thresholds (MetricUnitUtil), and the container / Swarm task rows ingest mirrors (cpuValueToPercent multiplied every 1 by 100). The values themselves were right: 4.3664 stored, 436.64% drawn. The Docker/Podman Overview and Containers tabs, which draw the raw value with a %, and the platform monitors, which use the Docker/Podman/Swarm catalogs (%), were the only places that read them right.

Expected behaviour (decided here)

• container.cpu.utilization (Docker, Podman, Swarm) = the CPU the container used over each collection interval (30 s), as a percent of one core: 100% is one core busy for the whole interval, 250% two and a half cores. It is the per-second rate of container.cpu.usage.total divided by 10^7, so the two charts always agree and short bursts count in proportion. Percent of one core because that is what docker stats / podman stats show, what OneUptime's Docker/Podman/Swarm catalogs, pages, alert templates ("Above 80 (% of one core)") and docs already promise, and what the customer expects (10 ms/s ≈ 1%). Not % of the host (a container pinning one core on a 32-core host would read 3%) and not % of its limit (most containers have none). An interval average rather than docker stats' one-second sample, because a one-second sample every 30 s misses most spikes - exactly what the customer saw.
• container.memory.percent = memory in use (inactive file cache excluded) as a percent of the container's memory limit, or of the host's memory when it has none, 0-100, unit %, shown once as a percent.

What changed

Agents (agents/DockerAgent, agents/PodmanAgent, agents/DockerSwarmAgent, otel-collector-config.yaml):

• The receiver's own container.cpu.utilization is off. Four processors build it from container.cpu.usage.total: transform/cpu-utilization-source (copy), cumulative_to_delta/cpu-utilization (CPU time since the previous scrape; initial_value: drop, max_staleness: 10m; a counter reset after a container restart drops one point instead of sending a negative one), delta_to_rate/cpu-utilization (per second), transform/percent-units (scale_metric(1e-7, "%"), a description that says what it is). The same last processor gives container.memory.percent the unit % and a description that says what it is a percent of (the container's memory limit, or the host's memory without one; docker_stats only says "Percentage of memory used"). container.cpu.usage.total goes on unchanged. Identical in all three agents; only the metrics pipeline runs them.
• Verified against the real daemon (collector 0.161.0): 25.0% for a quarter core, 150.0% for one and a half cores, 0 / 30 / 100 / 69% for a container busy 10 s of every 20, nothing (instead of -858%) on the scrape after docker restart; memory percent equal to usage / limit × 100.

Ingest (OtelMetricsIngestService, new Common/Utils/Metrics/ContainerStatsPercentUnit):

• A docker_stats or podman_stats percent declared 1 - told apart by its instrumentation scope (.../receiver/dockerstatsreceiver, .../receiver/podmanreceiver, and the older otelcol/... names) - is recorded as % in the metric catalog and read as % by the container snapshots. This fixes agents already in the field and anyone running these receivers in their own collector, without an agent upgrade. The Kubernetes agent's container.cpu.utilization (CPU cores, declared 1, scope kubeletstatsreceiver) is left alone.
• cpuValueToPercent no longer multiplies docker_stats' percent: container rows (Docker, Podman, Swarm tasks, cloud instances from docker_stats sidecars) hold 7.17, not 717.
• One metric name, one unit per project: a project running the Kubernetes agent and a Docker/Podman agent gets container.cpu.utilization from both (cores 1 / percent %). The catalog keeps the % a container stats receiver set (TelemetryUtil reconcile, keepsStoredPercentUnit) instead of flipping - and writing - on every other batch.
• Data migration RecordContainerPercentMetricUnits: every project's stored container.memory.percent / container.cpu.percent rows that say 1, and container.cpu.utilization rows with docker_stats' own description, become % at once (compare-and-set; Kubernetes rows untouched), so stored memory percents read right from the moment this deploys, including projects whose hosts no longer report.

The same flaw elsewhere, fixed:

• Kubernetes pages, percent scaled twice. Every Kubernetes detail CPU chart (pod, container, node, deployment, statefulset, daemonset, job, cronjob, namespace) charts cores declared 1 with legend % and a transformValue that divides by the node's allocatable cores ×100 - after the fetch had already converted 1 → % (×100). A pod using 1.25% of a 4-core node was drawn at 125%. A query with its own transformValue now gets its values as stored (MetricResultUnitConverter); queries without one convert exactly as before. Every Kubernetes CPU query carries its transform from the first fetch (percent of one core until the node's allocatable CPU is known), and MetricView fetches again when a query gains or loses a transform (UI review, below). Mutation-checked: removing the skip fails the Kubernetes and Proxmox cases.
• Host agent, lifetime average. agents/InfrastructureAgent reported each process's cpuPercent with gopsutil's Process.CPUPercent, which is CPU time since the process started / its age - the same flat line. It now keeps each process's CPU time from the previous 30-second report (keyed by PID + start time, so a reused PID is not compared with another process) and reports the CPU used between the two as a percent of one core (as top). First report after start: no CPU%. Mutation-checked.

Display texts and docs:

• Docker Swarm task CPU tooltips (task page CPU field, Tasks list CPU column) deliberately named no scale while ingest scaled the value twice; they now say "100% is one full CPU core" like the Docker/Podman pages. Renamed in place in all 17 Dashboard locales and translated in all 16.
• Docs, all 17 languages, line for line, anchors localized: the Docker/Podman/Swarm monitor pages describe CPU as averaged over the 30-second interval; their "A CPU alert never fires" entries say a flat line whatever the load means an older agent and link to the upgrade; the Docker / Podman / Swarm agent guides get a troubleshooting entry "Container CPU is a flat line" ("Task CPU" for Swarm). Agent READMEs listed container.cpu.percent (docker_stats never sends it) and now say how container.cpu.utilization is computed.

What changes for existing customers

• Memory percent: fixed by the OneUptime update alone. On OneUptime Cloud as soon as this deploys; self-hosted on upgrading OneUptime. The data migration corrects the unit for every project, so past and new readings show 4.36%, not 436.64%, everywhere. No agent change needed.
• Container CPU: needs the agent upgraded (Docker / Podman: the warning sign beside the agent version on the host's Overview opens the commands, e.g. docker pull oneuptime/docker-agent:release and recreate). From the second reading after the upgrade, CPU follows the load. Swarm: the agent's version is its collector's, which did not change, so no warning sign appears; re-run the installer or download the agent's files again (documented).
• Stored CPU readings stay as they were: the old agents' figures were wrong at the source (a lifetime ratio) and cannot be recovered; they now show at their own small scale instead of ×100, and age out with retention.
• Alerts on container CPU (Docker "High Container CPU Usage" etc.) will start firing where they should: they were comparing 80 against a flat ~0.
• Kubernetes detail CPU charts now show the right percent of the node (they read 100× too high when the node's allocatable was known).
• Mixed Kubernetes + Docker projects: in the generic metrics explorer, container.cpu.utilization is one metric name with two quantities (Kubernetes cores, Docker percent); its unit settles on %, so Kubernetes series of that name read as raw cores there. The Kubernetes pages compute their own percent and are not affected.
• Host agent: per-process CPU% in the Server / VM monitor's process list is the CPU used since the previous report (a process's first report has none).
• While a project has both upgraded and older Docker / Podman / Swarm agents, the metric catalog's description of the two percents follows whichever agent reported last (the unit stays % either way); it settles once every agent is upgraded.

Left out, deliberately

• SQL Database monitor ratios (cache hit %, rollback %) are ratios of counters since the engine's statistics reset - the definition PostgreSQL/MySQL document, but they move slowly on long-running servers. Per-interval versions need the previous check's counters (state per monitor and probe); recommended as a follow-up, not changed here.
• Host CPU in the host agent is a one-second sample per report (it follows the load); not changed.
• Upstream: the docker_stats receiver's one-shot regression should be reported to opentelemetry-collector-contrib (IncludePreviousSample: true or stream_stats); I did not open an upstream issue. If upstream fixes it, the agents can keep these processors (they are the interval average, which is what we want either way).
• Swarm "outdated" sign for a config-only change (its version is the collector's) - a separate change to how the Swarm agent reports its version.
• A real Podman daemon run: none here; the Podman config's processors run in the synthetic part of the behaviour test, and its whole pipeline runs against the Docker socket mounted where it looks for Podman's.
• monitor/server-monitor.md says the process list carries "CPU %" - still true; not reworded (would need 16 translations for no change of meaning).

Tests (what each proves)

• Tests/Ops/ContainerAgentCpuUtilization.test.js (24): the three configs turn the receiver's CPU percent off and keep the counter, define the four processors exactly and identically (units and descriptions included), run them in order after memory_limiter in the metrics pipeline only, under the 0.161.0 names (not the deprecated aliases), staleness ≥ 10 scrapes; the arithmetic (10^9 ns/s = 100%; 25 / 150 / 0 / restart / 10 / 30%; 1.2% for 12 ms/s at any interval).
• Tests/Ops/container-agent-cpu-utilization-behaviour.sh (new CI step in Ops Config Test, ~1 min): each agent's processors in the real collector image, fed six scrapes of two containers (25, 150, 0, nothing after a restart, 10; 1% throughout), the counter unchanged and the memory percent the same values in %; then each agent's whole metrics pipeline against the runner's Docker daemon with a container held to half a core: 50% ±15, equal to the rate of container.cpu.usage.total, memory percent = usage / limit × 100 in %; both percents carry the new descriptions in both parts. Its checks were each broken once (old descriptions in a real capture) and failed.
• packages/App/Tests/Telemetry/OtelMetricsIngestContainerPercent.test.ts (19) with a new real capture of the fixed agent (Fixtures/DockerAgentCpuCapture.ts, two consecutive scrapes, 25% and 150%; every block it builds equals the captured one field for field) and the existing capture of the old one: catalog units % for both percents (old and new agent), the customer's 4.3664 stored as sent under %, container rows not ×100 (Docker, Podman, Swarm task), the captured CPU percent equals the rate of the captured counter, Kubernetes' cores stay 1, other producers/units untouched.
• packages/Common/Tests/Utils/Metrics/ContainerStatsPercentUnit.test.ts (38): the scope/name/unit table, the customer's 436.64% vs 4.37% through ValueFormatter, % thresholds no longer ×100, Kubernetes cores still a percent of one core, the sticky rule's truth table.
• packages/Common/Tests/Server/Utils/MetricTypeContainerPercentUnit.test.ts (10): a stored 1 becomes % with the first batch; interleaved Kubernetes/Docker batches settle on % with exactly one write; Kubernetes-only projects keep 1; other metrics' unit changes unchanged.
• packages/Common/Tests/App/Dashboard/ChartPercentScaledOnce.test.ts (12): the fetch's conversion then the chart's transform, as the pages run them: Kubernetes container/pod/node CPU 1.25% / 75% (not 125% / 7500%) whatever the catalog unit, Proxmox ratio once, docker_stats percents as themselves, and queries without a transform still converting.
• packages/App/Tests/Workers/DataMigrations/RecordContainerPercentMetricUnits.test.ts (8): registration and slot, the exact row selection (docker_stats' description for CPU), ids collected across pages before writing, compare-and-set % over 1, a failed write costs no other row, a failed read stops the chain, idempotent, rollback no-op.
• agents/InfrastructureAgent/utils/procs_test.go (4): the per-process arithmetic and its no-previous cases, CPU% only from the second report, an idle interval reads near zero where gopsutil's lifetime average reads 34%, exited processes forgotten.
• Updated: OtelMetricsIngestContainerSnapshot (rows no longer ×100), OtelMetricsCloudSnapshot comments, DockerSwarmMetricTooltips (the scale test now reads the fixed ingest and requires "100% is one full CPU core").

Mutation checks (one break at a time, each caught): ingest scope normalization removed (7 tests fail), sticky rule removed (2), converter transform-skip removed (6), host agent back to CPUPercent (2).

Ran locally

• Real collector 0.161.0 against Docker 29.8.1: the bug reproduced on master's config, and the fix measured (numbers above); otelcol validate on all three configs; the new behaviour script end to end after the last change (all three agents, synthetic + real daemon, 56 checks).
• Jest, after merging master: the Common suites that read the changed modules (95 suites, 4,814 tests), Common Guard (149 suites, 4,730 tests) and Locale (30 suites); the App suites that read them (99 suites), App root-level tests (17) and Locale (9). After the final merge (master at a104399, which brought only tests, the NormalizeListOrder fix and the Dashboard tsc heap): every App DataMigrations suite (44 suites, 718 tests), the docs suites for the six pages (9 suites, 6,348 tests) and the general docs suites (29 suites, 1,419 tests), docs:check-anchors, i18n:validate, the Ops jest suite (62 suites, 3,594 tests), and the App ingest suites with the re-captured fixture (3 suites, 70 tests).
• After the final merge: npm run compile (Common, App), compile-tests (Common), Dashboard tsc, Dashboard i18n:check, DashboardLocalesGuard and the Common suites for the changed modules (4 suites, 143 tests). gofmt, go vet (linux, windows, darwin) and go test ./... -race for the host agent in golang:1.25. prettier and eslint on every changed code file: clean, except two lines of the PodmanAgent config that come from master (prettier would requote a regex and rewrap a list; master's copy fails the same way, left as they are) and the Ops .js suites, which eslint's TypeScript project does not include (as for every Ops suite).
• Not run: a real Podman daemon; the full App/Common suites; a live OneUptime stack (the ingest and display paths are covered by the suites above with real captured payloads).

Seen failing on master while this was built, not caused here: Tests/Workers/DataMigrations/NormalizeListOrder ("covers exactly the lists that are put in order by dragging": AIPostmortemRule was not covered since #5071). Master fixed it in 6d2cec5; it passes after the merge.

Reply to the customer (draft)

Thank you for the detailed report - you were right on both counts.

Memory percent (436% instead of 4.36%): the Docker agent sends container.memory.percent as a percent but labels it with the unit "1" (a 0-1 fraction), and OneUptime multiplied it by 100 a second time when drawing it. The stored numbers were correct. This is fixed on our side: once you are on the OneUptime release with the fix (on OneUptime Cloud, as soon as we deploy it; self-hosted, when you upgrade OneUptime), your existing and new memory charts show the right value without any change to the agent.

CPU utilization (a flat line): the OpenTelemetry collector inside the agent changed how it reads container stats (since collector 0.150.0, which the agent has used since June): it takes a single reading and divides all the CPU a container ever used by all the CPU the host used since boot, so the line barely moves and does not follow the load. The new agent computes container.cpu.utilization itself from container.cpu.usage.total: the CPU each container used over each 30-second interval, as a percent of one core (100% = one full core), so it matches the per-second rate you charted by hand (about 1% for your FastAPI container) and moves with spikes. Please update the agent on both hosts: docker pull oneuptime/docker-agent:release and recreate the container (or docker compose pull && docker compose up -d). CPU readings are correct from the second scrape after the restart; readings stored before stay as they were. Your CPU alerts will now see the real values too.

Thanks again - this helped us fix the same issue on the Podman and Docker Swarm agents as well.

UI review

Looked at. I rendered the real components in an esbuild harness with the production Tailwind build and Inter, and mocked the network so it answers like the real endpoints (AnalyticsModelAPI.aggregate, the metric catalog, ModelAPI). Shots were taken in headless Chromium at 1280 px and 390 px, light and dark, in English and German:

• Docker host Metrics tab, with the customer's data: "before" (catalog unit 1, old agent) reads memory 436.64% and CPU a flat 7.17%, as in the report. "After" reads 4.37%, and CPU follows the load (bursts to 6.2%).
• Kubernetes pod Metrics tab: 0.05 cores on a 4-core node draws 1.25% (was 125%). It draws the same whether the allocatable lookup is known from the start or arrives 1.5 s after the page, and whether the catalog says 1 or %. With no allocatable reading it draws percent of one core.
• Swarm task page (CPU card and its tooltip) and the Tasks list (CPU column and header tooltip), desktop cards and phone cards.
• Metrics list rows and the explorer's chart headers, which show the agent's metric descriptions.
• The changed docs sections, rendered with the docs' own renderer and stylesheet in en, de, ja and fa (right to left) at 1280 and 390 px, light and dark: the three troubleshooting entries, "A CPU alert never fires", and the NOTE callout.

Fixed.

1. Kubernetes CPU charts: two regressions of the transform skip (functional, found in review):

   • No allocatable CPU (a cluster without k8s.node.allocatable_cpu, or a node page before its node object loads): the transform returned cores unchanged, which used to arrive already ×100 from the fetch. 0.05 cores drew as "0.05%", 100 times too low. It now draws percent of one core (5%), as before this PR (KubernetesCpuUtils.coresToPercentOfOneCore).
   • A race: the pages passed the transform only once the allocatable lookup had loaded. The first fetch came back converted (1 to %, ×100), and MetricView did not fetch again when the transform arrived, so the transform ran on converted values: 125% again. Every Kubernetes CPU query now carries its transform from the first fetch. MetricView also fetches again whenever a query gains or loses its own transform, which keeps the converter's "values as stored" contract for any page.

2. Metric descriptions cut off. Both new descriptions were cut at 1280 px, in the metrics list and in the explorer's chart header, before the part that says what the number is ("...one CPU core (100% is on..."). They now fit on one line and are worded like the docs:

   • container.cpu.utilization: "CPU used over each collection interval, where 100% is one full CPU core."
   • container.memory.percent: "Memory in use as a percent of the container's limit, or of host memory if it has none."

   Same in all three agents. I captured the fixed-agent fixture again with the real collector 0.161.0 against Docker 29.8.1 (25.0% and 150.0% again; every block equals the capture field for field). The behaviour script passed for all three agents with the new descriptions.

3. Docs: "a flat line near zero" is now "a flat line", in the three troubleshooting titles and the monitor pages' sentence, in all 17 languages. The old figure is a container's lifetime average over the host's uptime, so it can sit at any level: the customer's staging host read a flat 7.17%. Each edit deletes the phrase; no anchor moves.

4. READMEs: the CPU paragraph sat in the middle of the "Collected Metrics" list and split it in two. The list is whole again, with two short notes after it.

Checked and left alone. The Swarm CPU tooltips read well in English and the 16 translations, at both widths and in both themes. The Persian docs render right to left correctly.

Follow-ups I saw that this change did not cause:

• On phones, the metrics list row gives the metric name no room: the value and sparkline take the row.
• The Swarm task page's 64-character Container ID runs out of its card.
• The Docker and Podman Metrics tabs repeat the percent sign in their legends ("Max CPU % 6.20%").
• Those tabs' chart titles and the series controls ("1 series", "Percent") are not translated.
• The area chart passes chart-only props (referenceLines, formattedExemplarPoints, onExemplarClick, anomalyBand*) to a div, which only shows up as React dev warnings.

Tests the review added or changed:

• packages/Common/Tests/App/Dashboard/MetricViewTransformRefetch.test.tsx (new, 3 tests). A transform that arrives after the first fetch is fetched with. A transform made anew on every render does not refetch. A transform that goes away is fetched without.
• packages/Common/Tests/App/Dashboard/ChartPercentScaledOnce.test.ts (+5 tests):
  • no allocatable reading, while the lookup loads or at all, draws percent of one core whatever the catalog unit;
  • a node page before its node loads does the same;
  • coresToPercentOfOneCore;
  • a source guard that every Kubernetes CPU query passes the lookup as it is, never only once it has loaded.
• Each fix was broken once and its test failed (5 mutations: no refetch, raw fallback with an empty lookup, raw fallback while loading, raw fallback on the node page, one page back to a conditional transform).
• Tests/Ops/ContainerAgentCpuUtilization.test.js, Tests/Ops/Utils/ContainerAgentCpuBehaviour.js and packages/App/Tests/Fixtures/DockerAgentCpuCapture.ts: the new descriptions, and the re-captured values.

Ran after the review's last change (master had not moved since a104399):

• Common: the suites that read the changed modules (41 suites, 1,595 tests) and Guard (149 suites, 4,730 tests).
• App: the suites that read them (18 suites, 885 tests), the docs suites of the five pages and their neighbours (15 suites, 6,794 tests), the general docs suites (29 suites, 1,419 tests) and the root-level tests (48 suites).
• Ops jest (62 suites, 3,594 tests), plus docs:check-anchors and i18n:validate.
• The changed suites again with ee/ hidden.
• Compiles: Common compile and compile-tests, App compile, Dashboard tsc, and --listFiles for Common and App (no source outside the package and Common).
• The container behaviour script against Docker (all three agents).
• prettier and eslint on every changed file. They are clean apart from master's own lines in the Podman config (a log regex's quotes, the logs pipeline list) and the Ops .js suites, which eslint's TypeScript project does not include.

🤖 Generated with Claude Code

[simlarsen]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.