Records flushed by in_tail's multiline at shutdown skip all filters (or are lost with memory storage) #12628

Bug Report

Describe the bug

When fluent-bit shuts down, in_tail flushes the records still pending in its multiline buffers. These records are not filtered. By the time they are produced, the engine has already destroyed every filter instance:

/* src/flb_engine.c, flb_engine_shutdown() — v4.2.8 ~L1332, master ~L1685 */
flb_filter_exit(config);      /* filter instances destroyed, removed from config->filters */
flb_output_exit(config);
flb_custom_exit(config);
flb_input_exit_all(config);   /* in_tail exit flushes pending multiline records */

The call path for the late records:

1. in_tail_exit() → flb_tail_file_remove_all() → flb_tail_file_remove().
2. flb_ml_stream_id_destroy_all() (src/multiline/flb_ml_stream.c) runs flb_ml_flush_parser_instance(..., FLB_TRUE), a forced flush.
3. ml_flush_callback() → ml_stream_buffer_flush() → flb_input_log_append().
4. flb_input_chunk_append_raw() calls flb_filter_do(), but config->filters is now empty, so the records are written to the chunk unchanged.

What happens next depends on the input's storage type:

• storage.type filesystem: the unfiltered chunk stays on disk. The next process's storage_backlog sends it to the outputs. Records that a grep filter should have dropped are delivered, and fields that filters add (e.g. record_modifier, lua, kubernetes) are missing.
• storage.type memory: the outputs have already exited, so the records are discarded. The tail DB offset is already past them, so they are not re-read after a restart either. Those lines are silently lost.

Either way, the result depends on whether a line happened to be waiting in a multiline buffer when the shutdown came. With a flush_timeout of a few seconds, that is roughly the last record of every active file.

To Reproduce

repro.sh (attached, needs Docker) tails one file with a multiline parser, a grep filter (exclude log INFO), a record_modifier that adds filtered: yes, and storage.type filesystem. It writes four lines. The last one is still pending in the multiline buffer when the container is stopped. Then it starts the container again:

$ ./repro.sh fluent/fluent-bit:5.1.5
== before shutdown
{"log":"2026-01-01 ERROR one\n","filtered":"yes"}
{"log":"2026-01-01 ERROR three\n","filtered":"yes"}
== chunks left in storage after shutdown
/storage/tail.0/1-1792604103.519191611.flb
== after restart (storage backlog)
{"log":"2026-01-01 ERROR one\n","filtered":"yes"}
{"log":"2026-01-01 ERROR three\n","filtered":"yes"}
{"log":"2026-01-01 INFO four\n"}

INFO four should have been dropped by grep, and it also lacks filtered. The same result on 4.2.8.

With storage.type memory and the last line changed to ERROR four (which should pass the filter), that line is never delivered, either before or after the restart.

Expected behavior

Records an input emits while shutting down go through the same filters as every other record. Alternatively, the input is never asked to flush once the filters are gone.

Possible fix

In flb_engine_shutdown(), exit the inputs (or at least flush their pending multiline state) before flb_filter_exit(). With filesystem storage the filtered chunk would be stored and sent by the next process. Exiting outputs before inputs would still lose the records with memory storage. Fully fixing that case needs the pending flush to happen during the grace period, while the outputs are still running.

Workaround

• Our hosts: we restart fluent-bit hourly to work around in_tail: truncation (copytruncate) is missed because file->size is refreshed without a shrink check — read position stays past EOF #12617. With storage.type filesystem this delivered ~22,700 unfiltered records in 4 hours across 14 hosts, of which 3 would have passed grep. We drop them with a Cloud Logging exclusion that keys on the label our filters add.
• Lowering the multiline flush_timeout shrinks the window but cannot close it.

Your Environment

• Version: 4.2.8 (production, official package); reproduced on 4.2.8 and 5.1.5 (official Docker images). flb_engine_shutdown() on master has the same order.
• Configuration: tail + multiline.parser (custom regex parser, flush_timeout 5000), storage.type filesystem, grep, parser, modify and nest filters, stackdriver output.
• OS: Ubuntu 26.04 (production), Docker Desktop on macOS (repro).

Additional context

Related, separate in_tail shutdown issue: a partial line (no trailing newline yet) buffered at shutdown is dropped, but the DB offset already includes it, so the restart resumes mid-line (#11269).

repro.sh