Use single virtiofs device for all shares - #41129
Conversation
There was a problem hiding this comment.
Pull request overview
This PR changes WSLâs virtio-fs-backed drvfs sharing model from âone virtio-fs device per shareâ to a single aggregate virtio-fs device per server (separate devices for admin vs non-admin), avoiding MMIO aperture exhaustion that can otherwise cause subsequent mounts to hang.
Changes:
- Introduces an aggregate virtio-fs device that exposes per-share children keyed by GUID, and updates the hostâguest protocol to return the child name for binding.
- Updates Linux init to mount the aggregate root once under a hidden directory and bind-mount per-share children to user-visible targets; updates mount enumeration / reverse path translation to ignore internal mounts.
- Updates Windows and Linux tests to reflect aggregate virtio-fs behavior (mount root semantics, mount-id lookup, and exercising read-only mounts).
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/windows/DrvFsTests.cpp | Expands virtiofs loop test and adds alternating read-only mounts with validation. |
| test/linux/unit_tests/lxtmount.c | Adjusts mount root checks and mount-id lookup for aggregate virtio-fs semantics. |
| test/linux/unit_tests/drvfs.c | Updates test applicability/behavior notes for virtiofs parity with Plan9 limitations. |
| src/windows/service/exe/WslCoreVm.h | Changes AddVirtioFsShare return type and adds aggregate device instance tracking. |
| src/windows/service/exe/WslCoreVm.cpp | Implements aggregate virtio-fs device + per-share child creation; includes protocol response updates. |
| src/windows/common/GuestDeviceManager.h | Adds AddVirtiofsChild API. |
| src/windows/common/GuestDeviceManager.cpp | Implements AddVirtiofsChild forwarding to the device host. |
| src/windows/common/DeviceHostProxy.h | Adds AddVirtiofsChild and GetVirtiofsDevice helpers. |
| src/windows/common/DeviceHostProxy.cpp | Implements virtio-fs device child creation and virtiofs device lookup. |
| src/shared/inc/lxinitshared.h | Extends add-virtiofs-share response with ChildNameOffset. |
| src/linux/init/util.cpp | Skips internal aggregate mounts during reverse translation; adjusts virtiofs source resolution. |
| src/linux/init/drvfs.h | Defines VIRTIOFS_MOUNT_DIR and extends QueryVirtiofsMountSource signature. |
| src/linux/init/drvfs.cpp | Adds aggregate-root + child bind mounting logic; updates tag mapping and source query behavior. |
| src/linux/init/config.cpp | Updates mount enumeration/remount logic to handle aggregate virtio-fs and skip internal mounts. |
| packages.config | Updates Microsoft.WSL.DeviceHost package version. |
|
I suggest we make this a config option and set it to enabled by default, because (1) there may be issues that require disabling it; (2) longer term we are probably moving back to individual devices so it would probably be good to maintain and test that code. Agreed though that it is a pain to have to test both variations. |
|
It does not look like WSLC was updated to use the new logic. |
That can be done as a follow-up if you like. |
| if (UtilIsPathPrefix(MountEnum.Current().MountPoint, VIRTIOFS_MOUNT_DIR, false) > 0) | ||
| { | ||
| continue; | ||
| } |
| // | ||
| // Validate the tag is a GUID. | ||
| // | ||
|
|
Ben Hillis (benhillis)
left a comment
There was a problem hiding this comment.
Looks good, thanks for doing this!
Now that DrvFs shares are carried by a single aggregate virtiofs device (#41129 / #41151), only a small, fixed number of virtiofs devices exist per VM. That leaves enough memory-aperture headroom to give each aggregate multiple request queues, which significantly improves concurrent DrvFs throughput. Non-aggregate (per-share file-backed) and section-backed devices stay at a single queue: many of them can exist, and raising their queue count would reintroduce the aperture exhaustion that the one-queue default was added to avoid. fio on /mnt/c (16 parallel jobs, psync), 4 queues vs 1: randread 4k +56% IOPS (-36% latency) randwrite 4k +622% IOPS (-85% latency) smallfile randrw +48% IOPS (-33% latency) seqread 1M +15% IOPS seqwrite 1M -10% IOPS (single-stream, bandwidth-bound; within noise) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4ab64eb9-4271-4241-b838-3da58ec39305
Now that DrvFs shares are carried by a single aggregate virtiofs device (#41129 / #41151), only a small, fixed number of virtiofs devices exist per VM. That leaves enough memory-aperture headroom to give each aggregate multiple request queues, which significantly improves concurrent DrvFs throughput. Non-aggregate (per-share file-backed) and section-backed devices stay at a single queue: many of them can exist, and raising their queue count would reintroduce the aperture exhaustion that the one-queue default was added to avoid. fio on /mnt/c (16 parallel jobs, psync), 4 queues vs 1: randread 4k +56% IOPS (-36% latency) randwrite 4k +622% IOPS (-85% latency) smallfile randrw +48% IOPS (-33% latency) seqread 1M +15% IOPS seqwrite 1M -10% IOPS (single-stream, bandwidth-bound; within noise) Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4ab64eb9-4271-4241-b838-3da58ec39305
With virtio-fs enabled, each drvfs share was backed by its own virtio-fs device. Each device consumes an MMIO aperture, of which there are a fixed amount available for a VM. When a VM runs out of them subsequent calls to mount shares will hang.
To work around this limit, all drvfs shares are backed by a single aggregate virtio-fs device per server (one for the non-admin drvfs, and another for the admin). The aggregate device exposes a synthetic root with one child per share keyed by a GUID subname. Adding a share now adds a child to the existing device instead of creating a new device. In the guest the aggregate device is mounted once at a hidden location (
/run/wsl/virtiofs-mounts/drvfsa) and each share is a bind-mount of<device>/<child-name>-> target.Key Changes:
Wsl Service:
WslCoreVm::AddVirtioFsSharenow creates the aggregate device on first use (VirtiofsShareKind_Aggregate) and callsAddVirtiofsChild(device, childName, options, path)for every shareGuestDeviceManager::AddVirtiofsChild/DeviceHostProxy::AddVirtiofsChild+GetVirtiofsDevice, forwarding to the device-host IWslVirtiofsDevice::AddChild.Protocol changes:
LX_INIT_ADD_VIRTIOFS_SHARE_RESPONSE_MESSAGEgainsChildNameOffsetso the guest learns which child to bind.Guest init changes:
MountVirtioFsmounts the aggregate root once, then bind-mounts <VIRTIOFS_MOUNT_DIR>// â target. Falls back to Plan 9 if the share-add or the bind fails (so shares on volumes the server can't fully back still mount).QueryVirtiofsMountSourcetakes a Root argument to resolve an aggregate child back to its Windows path; config.cpp mount enumeration/remount handles the aggregate (non-/ root) case and skips the internal hidden mounts.UtilFindMountexcludes internal mounts under VIRTIOFS_MOUNT_DIR from reverse (WindowsâLinux) translation, so wslpath resolves to the user-facing/mnt/cinstead of the internal/run/wsl/virtiofs-mounts/... path.