Nightly Linux DEB/RPM packaging fails in Fetch Artifacts for all GPU families on az-linux-scale-rocm-small-prod; reruns stay pinned to the same pool #9556

Priority: Urgent

The Linux multi-arch nightly fails in both native package jobs at the same step, on every attempt. All builds, tarballs, Python packages and test triggers pass, but Linux publish and the Linux PyTorch/JAX release triggers are skipped, so no Linux nightly deb/rpm repos and no Linux release wheels for 2026-10-28.

Fetch Artifacts for all GPU families
Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 1
Process completed with exit code 1.
Executing the custom container implementation failed. Please contact your self hosted runner administrator.

The container hook message is a symptom: the job container started fine and checkout, Python setup, pip and apt-get all ran in it. The failing command is artifact_manager.py fetch, which ran 8–16 minutes before exiting 1.

Every attempt failed the same way

Eight different runner pods, all in az-linux-scale-rocm-small-prod-822rr.

What changed

The artifact volume did not change:

• fix(ci): Right-size Linux runners for release and compiler workflows #9463 (merged 10-23) moved build_native_deb_packages / build_native_rpm_packages in multi_arch_release_linux.yml from the default runner to build_runs_on_small.
• feat(runner-config): increase Azure prod Linux build traffic to 20% therock-ci-config#64 (merged 10-26) raised Azure's share of the Linux small pool from 10% to 20%.
• The only earlier Azure-small pass was on scale set c7p8q. Today's failures are on 822rr, which first served native packaging today. Other jobs on 822rr (Storage Libs, Debug Tools, CV Libs, Media Libs, DC Tools Core, Emulation, Profiler Apps) passed, but they fetch only their stage's inbound artifacts, not every family at once.

Why reruns could not recover

The build runner label is chosen once, at random, in release / setup / setup (amdgpu_family_matrix.py select_weighted_label). "Re-run failed jobs" reuses setup's output, so attempts 2–4 all went back to the same Azure small pool. The setup job in this run completed once, at 00:01 UTC.

Likely cause

Confirmed from the latest attempt's step logs. This fetch is the heaviest single-runner download in the release: all 761 archives for every Linux family and expanded gfx target. They download into output/artifacts/.download_cache and extract into output/artifacts, and the cache is only removed after every extraction finishes. Peak usage is therefore about 14 GiB of archives plus the full extracted tree at the same time.

Both jobs finished the downloads, then failed during extract with Errno 28 (No space left on device):

DEB (job 122675723536):

!! Failed to extract sysdeps-expat_dev_generic.tar.zst: [Errno 28] No space left on device: 'output/artifacts/sysdeps-expat_dev_generic'
ERROR: Only extracted 695/761 artifacts - 66 failed

RPM (job 122675723550):

!! Failed to extract support_dev_generic.tar.zst: [Errno 28] No space left on device: 'output/artifacts/support_dev_generic'
ERROR: Only extracted 686/761 artifacts - 75 failed

This is a disk limit on 822rr, not an S3 download or credentials failure. c7p8q and AWS small did not hit it. It is not the same problem as #8562: on this run, seven other jobs on the same 822rr pool read from and wrote to S3 successfully.

Fix

• Unblock 10-28. "Re-run failed jobs" will keep landing on Azure small. Use "Re-run all jobs" so setup picks the pool again, or accept no Linux deb/rpm/publish for 10-28.
• Infra. Compare az-linux-scale-rocm-small-prod-822rr with c7p8q and aws-linux-scale-rocm-small (work volume size, ephemeral storage limit, memory). Until it matches, set the Azure weight for the Linux small pool to 0 in runner-config-v2.json.
• CI. Run build_native_deb_packages / build_native_rpm_packages on build_runs_on (default) again in multi_arch_release_linux.yml and compiler_bump.yml. Their peak disk is the full multi-arch tree, not a small-job footprint.
• CI hardening. Have artifact_manager.py fetch delete each archive after it extracts, and log free disk before and after, as build_tarballs.py does.
• CI hardening. Make reruns choose the runner pool again (or fall back to AWS) when a job fails on a weighted pool, so one bad scale set can't block every retry.

cc @kailash-khalasi (Azure prod ramp), @dysu-amd (#9463)