Skip to content

Podman failed to mount runtime directory for rootless netns: no such file or directory #13703

Closed
Closed
Podman failed to mount runtime directory for rootless netns: no such file or directory#13703
Labels
kind/bugCategorizes issue or PR as related to a bug.locked - please file new issue/PRAssist humans wanting to comment on an old issue or PR with locked comments.networkNetworking related issue or feature

Description

@fraschm1998
fraschm1998
opened on Mar 29, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. podman build .
  2. podman-compose up

Describe the results you received:
Podman failed to mount runtime directory for rootless netns: no such file or directory. I'm not sure if it's an issue with SELinux: https://dpaste.com/ANRM893KY

I found where the error message is printed (lines 128-138): https://fossies.org/linux/podman/libpod/networking_linux.go

echo $XDG_RUNTIME_DIR prints: /run/user/1000

Edit according to lines 114-119 it seems as tho I need to have the following binds mounted:

// The following bind mounts are needed
// 1. XDG_RUNTIME_DIR -> XDG_RUNTIME_DIR/rootless-netns/XDG_RUNTIME_DIR
// 2. /run/systemd -> XDG_RUNTIME_DIR/rootless-netns/run/systemd (only if it exists)
// 3. XDG_RUNTIME_DIR/rootless-netns/resolv.conf -> /etc/resolv.conf or XDG_RUNTIME_DIR/rootless-netns/run/symlink/target
// 4. XDG_RUNTIME_DIR/rootless-netns/var/lib/cni -> /var/lib/cni (if /var/lib/cni does not exists use the parent dir)
// 5. XDG_RUNTIME_DIR/rootless-netns/run -> /run

However in /run/user/1000 the only file related to netns is /run/user/1000/netns/rootless-netns-*

fd rootless in /

root@asus-g14 / # fd rootless
home/massimo/podman/cni/rootless-cni-infra
run/user/1000/libpod/tmp/rootless-netns.lock
run/user/1000/libpod/tmp/rootless-netns
run/user/1000/netns/rootless-netns-987c9a1aa493ae43558a
dev/shm/libpod_rootless_lock_1000
run/user/1000/libpod/tmp/rootless-netns/rootless-netns-slirp4netns.pid
var/db/repos/gentoo/metadata/md5-cache/sys-apps/rootlesskit-0.14.2
var/db/repos/gentoo/sys-apps/rootlesskit
var/db/repos/gentoo/sys-apps/rootlesskit/rootlesskit-0.14.2.ebuild
usr/share/doc/containerd-1.5.11/rootless.md.bz2
usr/share/selinux/targeted/include/services/rootlesskit.if
usr/share/selinux/strict/include/services/rootlesskit.if
usr/libexec/podman/rootlessport

root@asus-g14 / # fd rootless-netns
run/user/1000/libpod/tmp/rootless-netns.lock
run/user/1000/libpod/tmp/rootless-netns
run/user/1000/libpod/tmp/rootless-netns/rootless-netns-slirp4netns.pid
run/user/1000/netns/rootless-netns-987c9a1aa493ae43558a
root@asus-g14 / # v /run/user/1000/libpod/tmp/rootless-netns/resolv.conf

Edit I tried starting up the container again after doing podman system reset and get this:

ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: The name org.freedesktop.systemd1 was not provided by any .service files
Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/
rootless-netns/run: invalid argument

Output of podman version:

podman version 4.0.2

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: app-containers/conmon-2.1.0
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: v2.1.0'
  cpus: 16
  distribution:
    distribution: gentoo
    version: "2.8"
  eventLogger: file
  hostname: asus-g14
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.16-gentoo-x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 34120912896
  memTotal: 41527205888
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.4
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.1.12
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 27m 10.13s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  docker.io/library:
    Blocked: false
    Insecure: false
    Location: docker.io/library
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io/library
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /home/massimo/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/massimo/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 32
  runRoot: /run/user/1000/containers
  volumePath: /home/massimo/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646619707
  BuiltTime: Sun Mar  6 21:21:47 2022
  GitCommit: 342c8259381b63296e96ad29519bd4b9c7afbf97
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

app-containers/podman-4.0.2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Gentoo SeLinux Openrc

Activity

  1. openshift-ci
    added
    kind/bugCategorizes issue or PR as related to a bug.
    on Mar 29, 2022

  2. Luap99 commented on Mar 30, 2022

    @Luap99
    Luap99
    on Mar 30, 2022
    Member

    The error ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: The name org.freedesktop.systemd1 was not provided by any .service files should be ignored since you do not run systemd. I will fix this so that we do not log the error when you do not use systemd.

    Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/ rootless-netns/run: invalid argument sound like selinux problem. Is selinux enabled on your system?

  3. Luap99
    added
    networkNetworking related issue or feature
    on Mar 30, 2022

  4. fraschm1998 commented on Mar 30, 2022

    @fraschm1998
    fraschm1998
    on Mar 30, 2022
    Author

    Error: unable to start container 159d3d6111ff17c9c33108d37a547fb382e4909300ee07873f842ae2ac505fb7: could not create relabel rootless-netns run directory: setxattr /run/user/1000/libpod/tmp/ rootless-netns/run: invalid argument sound like selinux problem. Is selinux enabled on your system?

    I do have selinux installed but getenforce reports that it's in permissive mode. These are the logs I could find related to podman in /var/log/messages: http://dpaste.com/FRZSBJKZL

    Edit and podman related entried in audit.log: http://dpaste.com/4BEM47HHL

  5. Luap99 commented on Mar 30, 2022

    @Luap99
    Luap99
    on Mar 30, 2022
    Member

    We are tying to label it with the iptables_var_run_t type, is that available on your system?
    @rhatdan Any idea how to check what package provides this type?

  6. fraschm1998 commented on Mar 30, 2022

    @fraschm1998
    fraschm1998
    on Mar 30, 2022
    Author

    We are tying to label it with the iptables_var_run_t type, is that available on your system? @rhatdan Any idea how to check what package provides this type?

    These are the packages I have related to iptables:

     ~  ip                                                                                                                                                                      ok | 10:11:41
ip                        ip6tables-legacy-restore  ipcmk                     ippeveprinter             iptables-legacy           iptables-save
ip6tables                 ip6tables-legacy-save     ipcrm                     ipptool                   iptables-legacy-restore   iptables-xml
ip6tables-apply           ip6tables-restore         ipcs                      iptables                  iptables-legacy-save      iptunnel
ip6tables-legacy          ip6tables-save            ipmaddr                   iptables-apply            iptables-restore

  7. Luap99 commented on Mar 30, 2022

    @Luap99
    Luap99
    on Mar 30, 2022
    Member

    This is not about missing binaries, you are missing a selinux policy which provides this selinux type.
    @rhatdan ideas how to debug this?

  8. rhatdan commented on Mar 30, 2022

    @rhatdan
    rhatdan
    on Mar 30, 2022
    Contributor

    This looks like you have a custom policy and not the default policy. Your policy does not support MLS labels (MCS) which Podman relies on for container separation.

    Someone wrote podman_t types, and the person writing and shipping this policy needs to diagnose and figure out what is going on. This is not something the upstream podman team can fix.

  9. rhatdan
    closed this as completedon Mar 30, 2022

  10. rhatdan commented on Mar 30, 2022

    @rhatdan
    rhatdan
    on Mar 30, 2022
    Contributor

    I would be willing to work with whomever wrote this policy, but this is not a Podman issue.

  11. fraschm1998 commented on Mar 30, 2022

    @fraschm1998
    fraschm1998
    on Mar 30, 2022
    Author

    I would be willing to work with whomever wrote this policy, but this is not a Podman issue.

    Could it be related to this? opencontainers/selinux#171

  12. rhatdan commented on Mar 30, 2022

    @rhatdan
    rhatdan
    on Mar 30, 2022
    Contributor

    It could be, but I thought some fixes had gone in for this.

  13. devinrsmith commented on May 17, 2022

    @devinrsmith
    devinrsmith
    on May 17, 2022

    I ran into this, found the ultimate issue was

    exec: "slirp4netns": executable file not found in $PATH

    sudo dnf install slirp4netns seemed to fix the issue.

  14. 0xC0ncord commented on Jun 21, 2022

    @0xC0ncord
    0xC0ncord
    on Jun 21, 2022
    Contributor

    I would be willing to work with whomever wrote this policy, but this is not a Podman issue.

    @rhatdan For the record, I'm certain this is the policy I originally wrote and upstreamed to refpolicy which is now part of Gentoo's policy. I agree I don't think it's a bug in podman but unfortunately I can't provide more details without seeing any AVCs, and I never ran into this issue during the policy's development.

    opencontainers/selinux#171 is a separate issue.

  15. fraschm1998 commented on Jun 23, 2022

    @fraschm1998
    fraschm1998
    on Jun 23, 2022
    Author

    I can't provide more details without seeing any AVCs

    This is the most recent audit log: http://sprunge.us/2a3zQC

  16. rhatdan commented on Jun 23, 2022

    @rhatdan
    rhatdan
    on Jun 23, 2022
    Contributor

    @fraschm1998 Looks like your system is badly mislabeled. touch /.autorelabel; reboot
    Will cleanup labeling.

  17. 0xC0ncord commented on Jun 23, 2022

    @0xC0ncord
    0xC0ncord
    on Jun 23, 2022
    Contributor

    This is the most recent audit log: http://sprunge.us/2a3zQC

    The behavior for running podman as an unconfined user in refpolicy has changed recently, but that change is not part of Gentoo's policy yet.

    If you aren't hitting this issue while SELinux is in permissive mode, I suggest filing a bug in the Gentoo bug tracker since that would indicate the problem is outside the scope of Podman.

  18. github-actions
    added
    locked - please file new issue/PRAssist humans wanting to comment on an old issue or PR with locked comments.
    on Sep 20, 2023
  19. github-actions
    locked as resolved and limited conversation to collaborators on Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.locked - please file new issue/PRAssist humans wanting to comment on an old issue or PR with locked comments.networkNetworking related issue or feature

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions