Fluent Bit 5.1.3: Accumulated DLQ chunks exhaust storage.max_chunks_up and block the OTLP output pipeline #12574

Bug Report

Describe the bug

When Dead Letter Queue (DLQ) is enabled with filesystem buffering, accumulated rejected chunks appear to consume the available storage.max_chunks_up capacity.

Once fs_chunks_up reaches the configured limit, Fluent Bit stops forwarding new records to the OpenTelemetry output, while the OpenTelemetry input continues accepting incoming requests and returning HTTP 201 Created.

The output pipeline remains blocked even though new records are accepted and filesystem buffering is enabled.

The issue was reproduced with storage.max_chunks_up values of 128 and 256.

Increasing the limit from 128 to 256 temporarily restored output processing. However, after accumulating more DLQ chunks and restarting Fluent Bit with the existing storage volume, the issue occurred again.

To Reproduce

Steps to reproduce the problem:

The reproduction environment uses Fluent Bit 5.1.3 with an OpenTelemetry input, OpenTelemetry output, filesystem buffering, and a mock OTLP receiver that returns HTTP 400 or 500.

1. Configure Fluent Bit

service:
  flush: 1
  log_level: debug
  scheduler.base: 1
  scheduler.cap: 2
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  storage.path: /fluent-bit/storage
  storage.sync: normal
  storage.max_chunks_up: 256
  storage.keep.rejected: on
  storage.rejected.path: rejected

pipeline:
  inputs:
    - name: opentelemetry
      listen: 0.0.0.0
      port: 4318
      tag: dlq.test
      tag_from_uri: false
      storage.type: filesystem

  outputs:
    - name: opentelemetry
      match: "*"
      host: mock-otel
      port: 8090
      logs_uri: /v1/logs
      tls: off
      retry_limit: ${RETRY_LIMIT}
      log_response_payload: on

2. Start the reproduction environment

docker compose up -d --build

3. Send OTLP log records repeatedly

curl -i \
  -H "Content-Type: application/json" \
  --data-binary @test-record.json \
  http://localhost:4318/v1/logs

The mock OTLP receiver returns HTTP 400 or 500, causing rejected chunks to accumulate in:

/fluent-bit/storage/rejected

4. Accumulate rejected chunks until the storage limit is reached

Monitor the storage metrics:

curl -s localhost:2020/api/v1/storage |
  jq '.storage_layer.chunks'

With storage.max_chunks_up: 256, more than 250 DLQ files were accumulated.

5. Observe the pipeline becoming blocked (with or without restart)

The issue can occur in two scenarios:

• Without restarting Fluent Bit: As rejected chunks accumulate, the output pipeline may stop processing new records once fs_chunks_up reaches the configured limit.
• After restarting Fluent Bit: In some tests, the pipeline continued processing until Fluent Bit was restarted with the existing DLQ files. After restart, fs_chunks_up immediately reached the configured limit, and the output pipeline became blocked.

To reproduce the second scenario, restart Fluent Bit without deleting the storage volume:

docker compose restart fluent-bit

The following storage metrics were observed after restart:

{
  "total_chunks": 260,
  "fs_chunks": 260,
  "fs_chunks_up": 256,
  "fs_chunks_down": 4
}

The configured storage.max_chunks_up limit was fully occupied immediately after startup.

Note: A restart is not always required to reproduce the issue. The pipeline can become blocked during normal operation as DLQ chunks accumulate.

6. Send another OTLP record

The OpenTelemetry input still responds with HTTP 201 Created.

The input metrics increase:

input.records: 11 -> 12
input.bytes:   3124 -> 3408

However, the output metrics remain unchanged:

proc_records:    0
errors:          0
retries:         0
dropped_records: 0

No new requests reach the mock OTLP receiver.

Expected behavior

Persisted DLQ chunks should not indefinitely block the active output pipeline.

Even when a large number of rejected chunks have accumulated, Fluent Bit should be able to continue processing and flushing new filesystem-buffered records.

Ideally, DLQ storage should not consume all available UP chunk capacity required by active pipeline processing.

Your Environment

• Version used: Fluent Bit 5.1.3
• Configuration: YAML configuration above
• Environment name and version: Local container environment using docker compose
• Server type and version: Fluent Bit container and a Python ThreadingHTTPServer mock receiver
• Operating System and version: macOS host with a Linux container runtime
• Filters and plugins:
  • Input: opentelemetry
  • Output: opentelemetry
  • Storage: filesystem
  • DLQ: storage.keep.rejected: on
  • Filters: none