enable_input_metrics: inaccurate values

Describe the bug

We can take metrics of input plugins by setting enable_input_metrics.
However, the value would be inaccurate.

I don't directly confirm it yet, but I have confirmed @caller_plugin_id of EventRouter has race condition.

Error event logs should include plugin id #4567 (comment)

So, it would be possible that a wrong metric_callbacks is selected.

Lines 96 to 102 in 2d8c9d4

    
           def find_callback 
        
             if @caller_plugin_id 
        
               @metric_callbacks[@caller_plugin_id] 
        
             else 
        
               nil 
        
             end 
        
           end

It would cause wrong metrics calculation.

To Reproduce

I haven't checked it yet, but the following settings should result in a slight error in the metric values.

<system>
  enable_input_metrics
</system>

<source>
  @type monitor_agent
</source>

<source>
  @type sample
  tag test.foo
  rate 100
</source>

<source>
  @type sample
  tag test.bar
  rate 100
</source>

<source>
  @type sample
  tag test.boo
  rate 100
</source>

<match test.**>
  @type null
</match>

Wait a few minutes and check the metrics.

curl http://localhost:24220/api/plugins.json | jq

{
  "plugins": [
    {
      "plugin_id": "object:d34",
      "plugin_category": "input",
      "type": "monitor_agent",
      "config": {
        "@type": "monitor_agent"
      },
      "output_plugin": false,
      "retry_count": null,
      "emit_records": 0,
      "emit_size": 0
    },
    {
      "plugin_id": "object:d48",
      "plugin_category": "input",
      "type": "sample",
      "config": {
        "@type": "sample",
        "tag": "test.foo",
        "rate": "100"
      },
      "output_plugin": false,
      "retry_count": null,
      "emit_records": 43112,
      "emit_size": 0
    },
    {
      "plugin_id": "object:d5c",
      "plugin_category": "input",
      "type": "sample",
      "config": {
        "@type": "sample",
        "tag": "test.bar",
        "rate": "100"
      },
      "output_plugin": false,
      "retry_count": null,
      "emit_records": 43109,
      "emit_size": 0
    },
    {
      "plugin_id": "object:d70",
      "plugin_category": "input",
      "type": "sample",
      "config": {
        "@type": "sample",
        "tag": "test.boo",
        "rate": "100"
      },
      "output_plugin": false,
      "retry_count": null,
      "emit_records": 43109,
      "emit_size": 0
    },
    {
      "plugin_id": "object:d0c",
      "plugin_category": "output",
      "type": "null",
      "config": {
        "@type": "null"
      },
      "output_plugin": true,
      "retry_count": 0,
      "emit_records": 129330,
      "emit_size": 0,
      "emit_count": 129330,
      "write_count": 0,
      "rollback_count": 0,
      "slow_flush_count": 0,
      "flush_time_count": 0,
      "retry": {}
    }
  ]
}

You can confirm a difference in the value of emit_records for each in_sample.

Expected behavior

There is no difference in the value of emit_records for each in_sample.

Your Environment

- Fluentd version: 1.18.0
- Package version:
- Operating system: Ubuntu 20.04.6 LTS (Focal Fossa)
- Kernel version: 5.15.0-124-generic

Your Configuration

Noted in `To Reproduce`.

Your Error Log

No error.

Additional context

No response

@daipom

@daipom is it open to work

Sure!
We need to take care of performance impact as well.
Adding exclusive lock may resolve this issue, but it may cause performance regression.
As Fluentd, it can be concluded that a certain degree of error should be tolerated and performance should be prioritized.

I looked into this. The race is real, and in most cases it is a misattribution rather than a loss.

Mechanism

PluginHelper::EventEmitter#router writes @router.caller_plugin_id = plugin_id on every call, and EventRouter#emit_stream reads it back via find_callback only after match(tag).emit_events(tag, es) has returned. The id lives in one instance variable shared by every plugin on the same router (all inputs in the root agent), so any router call in between changes which input gets the records. There are three ways to hit it:

Two input threads interleaving. The records are credited to the other input, so the sum over all inputs still matches the output side. The sum of the three counts in the report (129330) matches out_null, which is consistent with this, although from those numbers alone I cannot tell whether the small differences come from the race or from the inputs starting at slightly different times.
Re-entrancy in a single thread. When a filter raises, Filter#filter_stream and FilterOptimizer call filter.router.emit_error_event, which overwrites the id with the filter's id, so the outer emit finds no callback and the input loses that batch. This one is deterministic, so a pipeline with frequent filter errors can under-count its inputs.
Fiber interleaving within one thread, for plugins that emit from an http_server (async) handler.

Frequency

With the configuration from this issue (in_sample x3 to out_null) I could not trigger case 1 at all. With a buffered output and 6 inputs it happened at about 0.0006% of records, and in a CPU-bound micro benchmark at about 1 in 1,000,000 emits.

Possible fix

A Mutex around the read and write does not help, because the problem is the gap between them, not a torn access. What works without changing the public API is to keep the caller id in fiber-local storage, resolve the callback before emit_events, and keep the shared instance variable as a fallback for plugins that call @router.emit directly (for example in_kafka and in_sql), so they keep today's behavior. With enable_input_metrics disabled this costs nothing. With it enabled, which is the default since v1.19.0 (#4966), it adds roughly 70 to 100 ns per emit_stream call in a micro benchmark of the cheapest possible emit path.

Given the low frequency and that the cost lands on the default configuration, I will hold the PR for now. If anyone is affected by the under-counting in case 2, please let us know.

Separately, I found that inputs with @label are never counted at all, because the metric callback is registered on the root router while the input emits into the label's router. I will file that as a separate issue.

root_agent: fix emit_records and emit_size never counted for inputs w…

Backport(v1.19): root_agent: fix emit_records and emit_size never cou…

Navigation Menu

Repository navigation

enable_input_metrics: inaccurate values #4717

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def find_callback
	if @caller_plugin_id
	@metric_callbacks[@caller_plugin_id]
	else
	nil
	end
	end

enable_input_metrics: inaccurate values #4717

Description

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

Activity

blazethunderstorm commented on Jul 4, 2025

daipom commented on Jul 5, 2025

Watson1978 commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions