filter_multiline: per-tag streams never freed, unbounded memory growth on high pod churn #12670

Bug Report

Describe the bug

When running filter_multiline in buffered mode (buffer true / buffer on), Fluent Bit allocates internal stream tracking structures (struct ml_stream and per-parser struct flb_ml_stream groups) for each unique (input_instance, tag) tuple via get_or_create_stream().

In Kubernetes environments where tags are derived from container metadata (e.g., kube.<namespace>.<pod>.<container>), every distinct container produces a unique stream key. In upstream Fluent Bit, these streams are appended to ctx->ml_streams and are never evicted or freed until process termination (cb_ml_exit() in plugins/filter_multiline/ml.c).

As a result, on nodes with container churn (such as CI/CD runners, batch workloads, or short-lived serverless tasks), dead streams accumulate indefinitely. Resident memory (RSS) grows linearly without bound as new container tags appear, eventually causing the host or container runtime to kill the process (OOMKill) and lose in-flight logs.

To Reproduce

• Rubular link if applicable: N/A (the issue is in stream allocation lifetime and heap retention, not regular expression matching).
• Example log message if applicable:

{
  "time": "2026-10-28T01:15:30.123456789Z",
  "stream": "stderr",
  "_p": "F",
  "log": "Exception in thread \"main\" java.lang.NullPointerException\n    at com.example.service.Worker.process(Worker.java:42)\n    at com.example.service.Main.main(Main.java:10)"
}

• Steps to reproduce the problem:

1. Deploy Fluent Bit v4.2.8 (or any release through master) as a DaemonSet on a Kubernetes cluster with the following configuration:

fluent-bit.conf:

[SERVICE]
    Flush         1
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            cri
    DB                /var/log/fluentbit.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  1
    Read_From_Head    On

[FILTER]
    Name                  multiline
    Match                 kube.*
    buffer                true
    flush_ms              500
    multiline.key_content log
    multiline.parser      go, java, python

[OUTPUT]
    Name                  null
    Match                 *

parsers.conf:

[PARSER]
    Name        cri
    Format      regex
    Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<_p>F|P) (?<log>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Set the container memory limit to 64Mi or 128Mi to observe rapid OOMKills, or leave unconstrained to observe linear monotonic heap retention.

1. Generate short-lived container churn emitting multi-line stack traces across unique pod and container names:

#!/bin/bash
# Generate 50 batches of short-lived pods with unique pod and container names
for b in $(seq 1 50); do
  echo "Launching batch ${b}/50..."
  for p in $(seq 1 20); do
    pod="churn-b${b}-p${p}-${RANDOM}"
    kubectl run "${pod}" --restart=Never --image=busybox -- sh -c '
      echo "Exception in thread \"main\" java.lang.NullPointerException"
      echo "    at com.example.service.Worker.process(Worker.java:42)"
      echo "    at com.example.service.Main.main(Main.java:10)"
      sleep 2
    ' >/dev/null 2>&1
  done

  # Wait for batch containers to complete execution
  kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod -l run=churn --timeout=30s >/dev/null 2>&1

  # Delete completed pods (containers already exited naturally, avoiding SIGTERM stalls)
  kubectl delete pod --all --grace-period=0 >/dev/null 2>&1
  sleep 1
done

(Note: A standalone client-go test driver is available upon request to automate batch orchestration, track actual started streams from Kubelet CRI metrics, and sample memory).

1. Monitor Fluent Bit resident memory (RSS), cgroup memory usage, and container restart count:

kubectl top pod -l k8s-app=fluent-bit
kubectl get pod -l k8s-app=fluent-bit -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'

1. Observe that after all churn pods terminate and /var/log/containers/*.log files are completely deleted, Fluent Bit memory does not return to baseline. Instead, memory remains permanently elevated at its post-churn plateau. Under typical container memory limits, the process is repeatedly OOMKilled (exit code 137).

Expected behavior

Streams that have completed multiline concatenation and have remained idle for a configurable period should be evicted, and their associated message buffers and parser stream groups freed. When active container churn ceases, resident memory should return to its baseline. If a container with a previously evicted tag logs again in the future, the stream should be re-created transparently without data loss.

Screenshots

Empirical Benchmark Results on Kubernetes 1.37

Below are measured memory dynamics from live benchmark runs generating 3,000 scheduled pods (5,348 actual started container streams) on a dedicated single-node cluster:

(Time-series RSS metrics and Valgrind Massif memory snapshot trees can be provided upon request).

Your Environment

• Version used: 4.2.8 (also verified against master branch commit f68b7b15d56373f28fe95b562ce91b26d5c298e9 / v5.1.5 logic)
• Configuration: in_tail (cri parser) → filter_multiline (mode parser, buffer true, flush_ms 500) → out_null
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.37 (GKE Standard)
• Server type and version: Linux 6.6
• Operating System and version: Container-Optimized OS (cos-117-18613-242-29)
• Filters and plugins: in_tail, filter_multiline, out_null

Additional context

1. Root Cause & Code Analysis

1. Linear Stream Accumulation (plugins/filter_multiline/ml.c): In buffered mode, incoming records are processed by cb_ml_filter(), which calls get_or_create_stream():

   mk_list_foreach_safe(head, tmp, &ctx->ml_streams) {
    stream = mk_list_entry(head, struct ml_stream, _head);
    name_check = strcmp(stream->input_name, i_ins->name);
    tag_check = strcmp(stream->tag, tag);
    if (tag_check == 0 && name_check == 0) {
        return stream;
    }
}

On a cache miss, it allocates a new struct ml_stream, calls flb_ml_stream_create() to instantiate stream state across all configured parsers, and appends it to ctx->ml_streams via mk_list_add(&stream->_head, &ctx->ml_streams).

1. Streams Only Freed on Process Shutdown: In cb_ml_exit():

   mk_list_foreach_safe(head, tmp, &ctx->ml_streams) {
    stream = mk_list_entry(head, struct ml_stream, _head);
    mk_list_del(&stream->_head);
    ml_stream_destroy(stream);
}

There is no mechanism in filter_multiline to evict streams whose containers have terminated or that have been idle. This is explicitly acknowledged in the source header at plugins/filter_multiline/ml.h#L35:

/*
 * input instance + tag is the unique identifier
 * for a multiline stream
 * TODO: implement clean up of streams that haven't been used recently
 */

1. Auto-Flush Timer Overhead: The auto-flush timer callback (cb_ml_flush_timer / flb_ml_flush_parser_instance in src/multiline/flb_ml.c) iterates over all registered streams every 500ms (flush_ms). As dead streams accumulate into the thousands, CPU time per tick increases steadily even when the node is completely idle.

2. Valgrind Massif DWARF Heap Attribution (94.27% of Post-Churn Heap Is filter_multiline)

Heap snapshots captured 10 minutes after churn completed (T+10m idle, N = 5,348 streams) cleanly isolate the permanent leak: while in_tail freed 100% of its closed-file buffers, filter_multiline retained 100% of all streams ever seen (115.92 MB out of 121.41 MB useful heap = 95.48% of useful heap / 94.27% of total heap):

3. Related Upstream Issues

• Slow memory leak after adding multiline filter - 1.9.9 #6192 ("Slow memory leak after adding multiline filter - 1.9.9"): Closed as stale/not_planned. A 2026 comment highlighted this exact mechanism and the ml.h#L35 TODO.
• Steady memory increase on high pod churn #6881 ("Steady memory increase on high pod churn"): Closed as not_planned, reporting the identical symptom on Kubernetes nodes with churning pods.

4. Proposed Fix: Opt-In Idle Stream Eviction (stream_idle_timeout)

We have implemented and tested a fix that resolves the leak by introducing opt-in idle stream eviction:

1. Configuration Property: Add stream_idle_timeout (type FLB_CONFIG_MAP_TIME, default "0" to preserve existing behavior for current deployments).
2. Monotonic Inactivity Tracking: Track last_used timestamps on struct ml_stream using CLOCK_MONOTONIC (ml_stream_clock_ms()) so system wall-clock steps do not trigger premature eviction.
3. Throttled Sweep: During cb_ml_filter(), run an eviction sweep at most once every max(1s, stream_idle_timeout / 2).
4. Safe Eviction (Zero Log Loss): For idle streams, invoke flb_ml_stream_id_destroy_all() to force-flush any pending unmerged content before destroying the core parser stream groups, and then unlink and free struct ml_stream.
5. Bounded Latency: Limit eviction to a maximum of 256 streams per filter call (FLB_MULTILINE_STREAM_EVICT_MAX) to prevent latency spikes on the event loop, continuing on subsequent calls if additional streams remain pending.
6. Metrics: Expose an eviction counter (multiline_streams_evicted_total) for Prometheus observability.
7. Transparent Re-creation: If an evicted tag logs again in the future, get_or_create_stream() automatically allocates a fresh stream.

The fix includes a runtime test (tests/runtime/filter_multiline.c) covering eviction, re-creation of evicted streams, and multiline concatenation on a re-created stream.