Skip to content

rewrite_tag loses records when emitter buffer limit is reached, even with filesystem buffering #12667

Open

Description

@art333

Fluent Bit version: 5.1.3

Describe the bug

When emitter_mem_buf_limit is reached, rewrite_tag stops re-emitting records, even when emitter_storage.type filesystem is enabled.

Issue #12026 was addressed by PR #12029, which preserves the original record when re-emission fails. However, the record keeps its original tag, effectively skipping the intended rewrite_tag processing.

If downstream outputs only match the rewritten tag, those records are never delivered.

Observed behavior

In our tests:

230 records sent through the OpenTelemetry input.

Only 49 records reached S3.

Increasing emitter_mem_buf_limit resulted in all 230 records being delivered.

Filesystem buffering was enabled on the input and all rewrite_tag emitters.

Debug log:

[input:emitter:datacontenttype_emitter] emitter memory buffer limit reached. Not accepting record.

Expected behavior

Reaching the emitter memory limit should not cause records to skip the required rewrite_tag processing.

With filesystem buffering enabled, Fluent Bit should queue records on disk and retry re-emission, or propagate backpressure upstream until the records can be processed.

Preserving the original record without retrying the failed rewrite is not sufficient to prevent end-to-end data loss.

Related: #12026, #12029.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions