Repository navigation
fix(opentelemetry-demo): set the Prometheus memory limit to 512Mi - #2422
Conversation
The bundled Prometheus ships with a 400Mi memory limit, set in 2023 after a long run peaked at 250Mi. The demo now sends far more metrics through the OTLP receiver, with a long list of promoted resource attributes, and the data lives in an emptyDir that survives container restarts. Once the TSDB grows past what 400Mi can hold, Prometheus is OOMKilled, replays the same WAL on restart and is killed again, so a default install ends up in CrashLoopBackOff within a day. 1Gi keeps a default install running and matches what the reporter verified on a 4 vCPU / 8 GiB node. Fixes open-telemetry/opentelemetry-demo#3811 Assisted-by: Claude Fable 5.1
marcschaeferger
added
chart:demo
|
@adityareddy-dev Thanks for that.
@open-telemetry/demo-maintainers any objections? |
|
@julianocosta89 Done, went with option 2. The limit is back at 400Mi and the chart now sets retentionSize: 500MB, which lands as --storage.tsdb.retention.size=500MB on the server, so the TSDB in the emptyDir can't keep growing for the whole 7d. Picked 500MB as a starting point, happy to go lower if you'd rather. Haven't had it running for a day under the cap yet though, so if you want to see that first just say. Chart bumped to 0.42.2 since main moved to 0.42.1 with #2383, examples regenerated. Retitling the PR to match. |
adityareddy-dev
changed the title
There was a problem hiding this comment.
With the memory limit set to 400, wouldn't that still cause a CrashLoopBackOff?
There was a problem hiding this comment.
Fair question, and the honest answer is yes it still can. retention.size only deletes finished blocks on disk, it doesn't touch the head block or the WAL replay, and the replay is what #3811 died on. So the cap bounds the emptyDir, not memory.
I'd rather go back to the 1Gi bump from the original description, that's the number the reporter actually ran. I'll push that and drop the cap unless you want the disk bounded too.
… the TSDB cap retention.size only deletes finished blocks on disk. It does not bound the head block or the WAL replay, and the replay is what the crash loop in open-telemetry/opentelemetry-demo#3811 died on, so the cap kept the emptyDir small and left the OOM loop in place. 1Gi is the number the reporter ran.
adityareddy-dev
changed the title
|
@julianocosta89 @cyrille-leclerc the current diff is back to 1Gi with the retention cap removed, following the WAL replay concern. The chart is at 0.42.5. Could you review this version? |
|
Correction to my earlier reply: the issue says WAL/WBL replay completed and Prometheus became ready before it was OOM-killed. I shouldn't have called replay the point of failure. The reporter also added a 512Mi request and recreated the Pod alongside the 1Gi limit, so that observation doesn't prove the limit change alone fixes it. I've corrected the description. The current diff raises the limit to 1Gi, with requests and retention unchanged, and still needs runtime confirmation. |
|
@adityareddy-dev I don't think 1Gi is something we actually want. The docker version of Prometheus is running with 200Mi. I know in k8s there are more data coming through, but 5x more data? |
|
Fair, 1Gi was the reporter's workaround, not something I measured. The chart's collector also runs |
adityareddy-dev
changed the title
|
@julianocosta89 @cyrille-leclerc ran the chart from this branch on kind for 24 hours, 291 samples five minutes apart, no restarts or OOM kills in the window. Prometheus peaked at 317 MiB working set and 280 MiB RSS, with about 60k head series at the top. That run had the 1Gi limit, so it shows what Prometheus uses, not that it survives on less. Samples five minutes apart can miss a short spike, so I left room above the peak and set the limit to 512Mi. That is about 1.6x the measured peak, and it was 400Mi on main and 1Gi earlier in this PR. Pushed, examples regenerated, the chart stays at 0.42.5. If you'd rather sit closer to compose I can run 400Mi on kind for a day and see. |
|
Thank @adityareddy-dev! |
|
@adityareddy-dev I merged another demo pr first you'll need to bump the version and regenerate the examples |
Description
Raise the bundled Prometheus memory limit from 400Mi to 512Mi. Bump the chart from 0.42.4 to 0.42.5 and refresh the rendered examples. Requests and retention settings stay unchanged.
The issue reports OOMKilled/exit 137 within the first day with the default 400Mi limit. A 24-hour run of this chart on kind, with a 1Gi limit, peaked at 317 MiB working set and 280 MiB RSS with about 60k head series and no restarts. 512Mi is about 1.6x that peak, since samples five minutes apart can miss a short spike. The run did not use 512Mi itself.
Link to tracking issue
Fixes open-telemetry/opentelemetry-demo#3811
Authorship