Skip to content

Compatibility issues with fluentd plugins #4645

Open
Open
Compatibility issues with fluentd plugins#4645
Assignees
daipom
Labels
plugin issuework-in-progress

Description

@PranavBhatSF
PranavBhatSF
opened on Sep 25, 2024

Usecase

We need to push kubernetes pod logs to Azure, S3 or Splunk based on the configuration.

What we have done

Built a custom image with Fluentd image as base and included the required plugins.
Dockerfile

FROM ghcr.io/kube-logging/fluentd:v1.16-4.9-full as base-fluentd

# add azure storage append blob plugin
RUN gem install fluent-plugin-azure-storage-append-blob -v 0.2.1
# add aws s3 plugin
RUN gem install fluent-plugin-s3 -v 1.7.2
# add splunk plugin
RUN gem install fluent-plugin-splunk-hec -v 1.3.2

Tried with https://github.com/fluent/fluent-plugin-splunk as well. But hitting the same issue.

Issue

Fluentd is pushing logs to S3 and Azure. But when we configure splunk as destination, fluentd-configcheck pod fails with the below error

....
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
....
/usr/lib/ruby/3.1.0/rubygems/specification.rb:2288:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)

To Reproduce

  • Use a fluentd image with Azure, S3 and Splunk plugins installed
  • Configure Splunk as destination by creating ClusterOutput and ClusterFlow objects. Configurations are added below.

Expected behavior

Fluentd should send logs to Splunk server

Your Environment

- Fluentd version: 1.16.3

This is a kubernetes 1.30.1 environment. But this is reproducible in older kubernetes versions as well.
We are using kube-logging stack. kube-logging version is 4.9.1

Your Configuration

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: splunk-output
spec:
  splunkHec:
    buffer:
      tags: '[]'
      timekey: <splunk_hec_timekey>
      timekey_use_utc: true
      timekey_wait: 10s
      type: file
    hec_host: <splunk_hec_host>
    hec_port: <splunk_hec_port>
    hec_token: <splunk_hec_token>
    index: <splunk_hec_index>
    insecure_ssl: true
    protocol: <splunk_hec_protocol>
    source: <splunk_hec_source>
    sourcetype: <splunk_hec_source_type>
EOF

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: splunk-flow
  namespace: logging
spec:
  filters:
  - tag_normaliser:
      format: ${namespace_name}/${pod_name}.${container_name}
  globalOutputRefs:
  - splunk-output
  match:
  - select:
      container_names:
      - istio-proxy
      namespaces:
      - istio-system
  - exclude:
      container_names:
      - istio-proxy
      - istio-init
      - aicenter-hit-count-update
      - istio-configure-executor
      - on-prem-tenant-license-update
      - curl
      - recovery
      - aicenter-oob-scheduler
      - cert-trustor
  - exclude:
      namespaces:
      - default
  - exclude:
      labels:
        app: csi-snapshotter
  - exclude:
      labels:
        app: csi-resizer
  - select: {}
EOF

Your Error Log

➜  ~ kubectl logs logging-fluentd-configcheck-691ab8a2  -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run
2024-09-25 07:20:53 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-09-25 07:20:53 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-09-25 07:20:53 +0000 [info]: gem 'fluentd' version '1.16.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-datapoint' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-events' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-gelf-best' version '1.3.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.5.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-prometheus-format' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-protobuf' version '2.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.9.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.3'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-ua-parser' version '1.2.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2024-09-25 07:20:53 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.6.0'
2024-09-25 07:20:53 +0000 [info]: gem 'fluentd' version '1.16.2'
2024-09-25 07:20:53 +0000 [info]: starting fluentd-1.16.3 as dry run mode ruby="3.2.4"
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:210:in `rescue in try_activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:203:in `try_activate'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/lib/ruby/3.2.0/rubygems/specification.rb:2303:in `raise_if_conflicts': Unable to activate swd-2.0.2, because faraday-1.10.3 conflicts with faraday (~> 2.0) (Gem::ConflictError)
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1432:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1470:in `block in activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `each'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1452:in `activate_dependencies'
	from /usr/local/lib/ruby/3.2.0/rubygems/specification.rb:1434:in `activate'
	from /usr/local/lib/ruby/3.2.0/rubygems.rb:204:in `try_activate'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:155:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': cannot load such file -- fluent/plugin/out_splunk (LoadError)
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluent-plugin-splunk-hec-1.3.3/lib/fluent/plugin/out_splunk_hec.rb:7:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/label.rb:31:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `block in configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `each'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:146:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
	from /usr/local/bundle/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /usr/local/bundle/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'

Activity

  1. PranavBhatSF
    added
    waiting-for-triage
    on Sep 25, 2024
  2. daipom
    added
    work-in-progress
    and removed
    waiting-for-triage
    on Sep 26, 2024

  3. daipom commented on Sep 26, 2024

    @daipom
    daipom
    on Sep 26, 2024
    Contributor

    Thanks for your report!
    It appears that some dependencies conflict with each other.
    We need to check it.

  4. daipom
    self-assigned this
    on Sep 26, 2024
  5. daipom
    added
    plugin issue
    on Sep 26, 2024

  6. daipom commented on Sep 26, 2024

    @daipom
    daipom
    on Sep 26, 2024
    Contributor

    At least, fluent-plugin-azure-storage-append-blob requires faraday v1.
    On the other hand, swd v2 requires faraday v2.
    Using swd v1 may solve the error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

plugin issuework-in-progress

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions