Changes since v0.20.2:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.19.0, v0.20.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.20.1, v0.20.2.

Changes by Kind

Feature

• MultiKueue: Added support for RayService in-tree autoscaling on worker clusters using elastic workload slices. Requires the MultiKueueRayInTreeAutoscaling feature gate. (#17466, @kevin85421)
• Scheduling: Improved classical preemption performance by reusing candidate ordering and generating candidates on demand. Enable it with the alpha ClassicalPreemptionCandidateOrderCache feature gate (disabled by default). (#17296, @zhengchenyu)

Bug or Regression

• ConcurrentAdmission: Fixed a bug where a parent Workload could remain admitted on a flavor removed from its ClusterQueue when a surviving variant was already admitted. The parent is now marked for eviction so it can be re-admitted on a remaining flavor. (#17337, @yuluo-yx)

• DRA: Fixed a bug where a workload requesting a DeviceClass by its implicit extended resource name deviceclass.resource.kubernetes.io/<class> was treated as an ordinary extended resource, so its quota was not charged to the deviceClassMappings logical resource and the workload could stay pending. (#17434, @henry3260)

• Deployment, StatefulSet & LeaderWorkerSet: Fixed a bug where, once a WorkloadPriorityClass named default existed, updates to an object with ready replicas and no kueue.x-k8s.io/priority-class label were rejected, even when Kueue did not manage the object. Kueue no longer adds the default WorkloadPriorityClass to such objects on update. (#17309, @henry3260)

• Elastic workloads: Fixed Pods remaining Pending after scale-up when ElasticJobsViaWorkloadSlicesForProvisioningRequests is enabled. ProvisioningRequests belonging to a replaced workload slice are now retained while their capacity bookings remain valid. (#17359, @neilb-dotcom)

• ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic Workload replacement slice that grows one PodSet and shrinks another could remain pending when the growing PodSet appeared before the shrinking PodSet in spec.podSets. (#17561, @abhayjoshi201)

• Fixed a bug where setting a ClusterQueue's stopPolicy to HoldAndDrain did not evict admitted Workloads whose LocalQueue was held or deleted. (#17480, @henry3260)

• Jobs: Fixed a bug where a Job could adopt an unrelated, already finished prebuilt Workload and run without quota management. Kueue now stops the Job instead. (#16994, @bolubo)

• KueueViz: Fix the workload details page failing when the Workload's LocalQueue was deleted. (#17275, @onkar717)

• KueueViz: Fixed a bug where a backend component failure could prevent graceful HTTP server shutdown. KueueViz now shuts down the server and stops the components before reporting the failure. (#17471, @fat-catTW)

• KueueViz: Fixed a bug where an unknown KUEUEVIZ_AUTH_MODE value could start the backend without authentication and authorization middleware. KueueViz now exits with an error unless the value is Disabled or TokenReview. (#17311, @fat-catTW)

• KueueViz: Fixed a bug where distinct user group or extra attribute sets could incorrectly share a cached authorization result. (#17408, @fat-catTW)

• KueueViz: Fixed a bug where the cohorts list exposed ClusterQueue names to users without permission to list ClusterQueues. (#17300, @henry3260)

• KueueViz: Fixed links to missing ClusterQueues and Cohorts, such as "N/A" on the dashboard linking to /cluster-queue/undefined. (#17405, @onkar717)

• KueueViz: Fixed the ResourceFlavor detail page always showing "None" for node labels and node taints. (#17329, @onkar717)

• KueueViz: Fixed the dashboard tooltip of a preempted workload not linking to the preemptor workload. (#17428, @onkar717)

• LeaderWorkerSet: Fixed a bug where changing the kueue.x-k8s.io/queue-name label while no replica was Ready, but a Workload already held a quota reservation, made the LeaderWorkerSet reconciler fail on every retry and left the Workload in the old queue. Kueue now rejects the change in that case. (#17307, @Andrushika)

• MultiKueue: Fixed a bug where serveConfigV2 updates to a RayService with in-tree autoscaling were not propagated to the worker cluster when MultiKueueRayInTreeAutoscaling was enabled. (#17513, @kevin85421)

• MultiKueue: Fixed a regression where MultiKueueOrchestratedPreemption, enabled by default since v0.20, added two status writes for every Workload replicated to a worker cluster, reducing MultiKueue admission throughput. status.preemptionGates no longer lists a Closed state for gates that have not been opened; such gates remain closed. (#17468, @YQ-Wang)

• MultiKueue: Fixed batch/Job re-dispatch eviction loop caused by status.failed counter rejection on Kubernetes 1.35+. (#17569, @mishradwaterlaw)

• MultiKueue: Fixed manager startup failure when the MultiKueueClusterProfile feature gate is enabled but the ClusterProfile CRD is not installed. (#17228, @yindia)

• Observability: Fixed kueue_local_queue_admitted_active_workloads doubling after a ClusterQueue custom label change, and going negative after a LocalQueue label change. (#17435, @Andrushika)

• Observability: Improved Workload controller logging:

  • Log Workload inadmissibility and recovery from DRA validation errors at verbosity 3, after the corresponding status update succeeds. DRA validation failures no longer produce error logs on every reconcile.
  • Move routine per-Workload messages from verbosity 2 to 3 or higher to reduce log noise.
  • Rename the queue and prevQueue log keys to localQueue and prevLocalQueue, and include the namespace in LocalQueue references.
  • Identify the failed operation in reconciliation errors. If updating the Workload status after a DRA failure also fails, report both errors.

  DRA: Fixed a case where a Workload kept Requeued=True after DRA validation failed again. The controller now sets Requeued=False even if the same validation error is already recorded in the Workload status. (#17332, @mimowo)

• Pod integration: Fixed a bug where a pod group's new Workload could be built from a pod still terminating on a node, pinning it there so it could not be admitted. The fix is on by default through the Beta FinalizeTerminatingPodGroups feature gate. (#17501, @anguszzzz)

• RayCluster: Fixed a bug where worker PodSet counts could fall below KubeRay’s minimum replica count, leaving workers scheduling-gated when capacity became available. Kueue now respects worker replica bounds when calculating PodSet counts. (#17366, @kevin85421)

• Scheduling: Fixed a bug that could delay workload admission after leader failover when the new leader lacked Cohort quota. Kueue now keeps Cohorts in the scheduler cache on every replica. (#17404, @dundysm)

• Scheduling: Fixed a bug where every Workload event built the Workload's effective resources and scheduling hash for the DRA check even when no DeviceClass backs an extended resource, and compared whole PodSpecs to detect resource defaults, adding avoidable CPU overhead to the Workload controller. (#17444, @sohankunkerkar)

• Scheduling: Fixed a bug where fair-sharing preemption could repeatedly evict and re-admit Workloads when a preemptor was within nominal quota for one requested resource but still borrowing another. (#17349, @fat-catTW)

• Scheduling: Fixed a bug where the scheduler cache could drop an admitted Workload when its admission write was committed but the client observed an error, such as a timeout. This left the ClusterQueue's usage under-counted until the Workload's next update. (#17474, @kaushik229)

• StatefulSet: Fixed a bug where scaling a StatefulSet to zero released its Workload's quota while its Pods were still running, which could let other Workloads be admitted onto resources still in use. (#17540, @gola)

• TAS: Fixed a bug where Pods in a PodSet group whose indexes start from 0, such as JobSet or Ray workers, were ungated without rank ordering, which could split a kueue.x-k8s.io/podset-slice-required-topology slice across topology domains. Kueue now sets kueue.x-k8s.io/pod-index-offset: "0" on grouped JobSet and Ray Pod templates when it starts the job, which replaces the PodSet group offset. This applies to workloads admitted after the upgrade. (#17512, @henry3260)

• TAS: Fixed a bug where a Workload could be admitted onto a node whose Pod slots had run out earlier in the same scheduling cycle, or onto a node or topology domain with room for only some of its PodSets, leaving Pods Pending. In a ClusterQueue with a pods quota, a PodSet whose Pods request no resources can now be admitted. (#17327, @thc1006)

• TAS: Fixed a bug where a cold-started scheduler could admit workloads onto occupied nodes before existing ClusterQueues and their admitted Workloads were loaded into its cache. Kueue now waits for cache-filling handlers to process their initial data before scheduling. (#17455, @dundysm)

• TAS: Fixed a bug where replacing a failed node never completed for a Workload with some Pods already succeeded, leaving the recreated Pod gated. Kueue now keeps the admitted Pod count and resource usage when it updates the topology assignment. (#17502, @Narwhal-fish)

• TAS: Fixed a bug where waiting Workloads could remain inadmissible after a deactivated Workload's Pods finished terminating when SchedulerLibraryIntegration was enabled. Kueue now requeues them for admission. (#17608, @kshalot)

• TAS: Fixed a bug where, with `ElasticJobsViaWorkloadSlic...

Changes since v0.19.9:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.19.1, v0.19.2, v0.19.3, v0.19.4, v0.19.5, v0.19.6, v0.19.7, v0.19.8, v0.19.9.

Changes by Kind

Feature

• Scheduling: Improved classical preemption performance by reusing candidate ordering and generating candidates on demand. Enable it with the alpha ClassicalPreemptionCandidateOrderCache feature gate (disabled by default). (#17295, @zhengchenyu)

Bug or Regression

• ConcurrentAdmission: Fixed a bug where a parent Workload could remain admitted on a flavor removed from its ClusterQueue when a surviving variant was already admitted. The parent is now marked for eviction so it can be re-admitted on a remaining flavor. (#17338, @yuluo-yx)

• Deployment, StatefulSet & LeaderWorkerSet: Fixed a bug where, with the WorkloadPriorityClassDefaulting feature gate enabled, updates to an object with ready replicas and no kueue.x-k8s.io/priority-class label were rejected once a WorkloadPriorityClass named default existed, even when Kueue did not manage the object. Kueue no longer adds the default WorkloadPriorityClass to such objects on update. (#17313, @henry3260)

• ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic Workload replacement slice that grows one PodSet and shrinks another could remain pending when the growing PodSet appeared before the shrinking PodSet in spec.podSets. (#17580, @abhayjoshi201)

• Fixed a bug where setting a ClusterQueue's stopPolicy to HoldAndDrain did not evict admitted Workloads whose LocalQueue was held or deleted. (#17479, @henry3260)

• Jobs: Fixed a bug where a Job could adopt an unrelated, already finished prebuilt Workload and run without quota management. Kueue now stops the Job instead. (#16994, @bolubo)

• KueueViz: Fixed a bug where a backend component failure could prevent graceful HTTP server shutdown. KueueViz now shuts down the server and stops the components before reporting the failure. (#17472, @fat-catTW)

• KueueViz: Fixed a bug where an unknown KUEUEVIZ_AUTH_MODE value could start the backend without authentication and authorization middleware. KueueViz now exits with an error unless the value is Disabled or TokenReview. (#17312, @fat-catTW)

• KueueViz: Fixed links to missing ClusterQueues and Cohorts, such as "N/A" on the dashboard linking to /cluster-queue/undefined. (#17406, @onkar717)

• KueueViz: Fixed the ResourceFlavor detail page always showing "None" for node labels and node taints. (#17489, @onkar717)

• KueueViz: Fixed the dashboard tooltip of a preempted workload not linking to the preemptor workload. (#17490, @onkar717)

• KueueViz: Fixed the workload details page failing when the Workload's LocalQueue was deleted. (#17364, @onkar717)

• LeaderWorkerSet: Fixed a bug where changing the kueue.x-k8s.io/queue-name label while no replica was Ready, but a Workload already held a quota reservation, made the LeaderWorkerSet reconciler fail on every retry and left the Workload in the old queue. Kueue now rejects the change in that case. (#17310, @Andrushika)

• MultiKueue: Fixed a bug where, with the Alpha MultiKueueOrchestratedPreemption feature gate enabled, Kueue added two extra status writes on the worker cluster for every replicated Workload, reducing MultiKueue admission throughput. status.preemptionGates no longer lists a Closed state for gates that have not been opened; such gates remain closed. (#17469, @YQ-Wang)

• MultiKueue: Fixed batch/Job re-dispatch eviction loop caused by status.failed counter rejection on Kubernetes 1.35+. (#17059, @mishradwaterlaw)

• MultiKueue: Fixed manager startup failure when the MultiKueueClusterProfile feature gate is enabled but the ClusterProfile CRD is not installed. (#17228, @yindia)

• Observability: Improved Workload controller logging:

  • Log Workload inadmissibility and recovery from DRA validation errors at verbosity 3, after the corresponding status update succeeds. DRA validation failures no longer produce error logs on every reconcile.
  • Move routine per-Workload messages from verbosity 2 to 3 or higher to reduce log noise.
  • Rename the queue and prevQueue log keys to localQueue and prevLocalQueue, and include the namespace in LocalQueue references.
  • Identify the failed operation in reconciliation errors. If updating the Workload status after a DRA failure also fails, report both errors.

  DRA: Fixed a case where a Workload kept Requeued=True after DRA validation failed again. The controller now sets Requeued=False even if the same validation error is already recorded in the Workload status. (#17410, @mimowo)

• Pod integration: Fixed a bug where a pod group's new Workload could be built from a pod still terminating on a node, pinning it there so it could not be admitted. The fix is on by default through the Beta FinalizeTerminatingPodGroups feature gate. (#17454, @anguszzzz)

• RayCluster: Fixed a bug where worker PodSet counts could fall below KubeRay’s minimum replica count, leaving workers scheduling-gated when capacity became available. Kueue now respects worker replica bounds when calculating PodSet counts. (#17367, @kevin85421)

• Scheduling: Fixed a bug that could delay workload admission after leader failover when the new leader lacked Cohort quota. Kueue now keeps Cohorts in the scheduler cache on every replica. (#17346, @dundysm)

• Scheduling: Fixed a bug where an admission write that the apiserver committed but the client reported as failed (for example, a timeout) could remove the admitted Workload from the scheduler cache, so the ClusterQueue under-counted its usage until the Workload's next update. (#17578, @kaushik229)

• Scheduling: Fixed a bug where every Workload event built the Workload's effective resources and scheduling hash for the DRA check even when no DeviceClass backs an extended resource, and compared whole PodSpecs to detect resource defaults, adding avoidable CPU overhead to the Workload controller. (#17443, @sohankunkerkar)

• Scheduling: Fixed a bug where fair-sharing preemption could repeatedly evict and re-admit Workloads when a preemptor was within nominal quota for one requested resource but still borrowing another. (#17390, @fat-catTW)

• StatefulSet: Fixed a bug where scaling a StatefulSet to zero released its Workload's quota while its Pods were still running, which could let other Workloads be admitted onto resources still in use. (#17541, @gola)

• TAS: Fixed a bug where Pods in a PodSet group whose indexes start from 0, such as JobSet or Ray workers, were ungated without rank ordering, which could split a kueue.x-k8s.io/podset-slice-required-topology slice across topology domains. Kueue now sets kueue.x-k8s.io/pod-index-offset: "0" on grouped JobSet and Ray Pod templates when it starts the job, which replaces the PodSet group offset. This applies to workloads admitted after the upgrade. (#17516, @henry3260)

• TAS: Fixed a bug where a Workload could be admitted onto a node whose Pod slots had run out earlier in the same scheduling cycle, or onto a node or topology domain with room for only some of its PodSets, leaving Pods Pending. In a ClusterQueue with a pods quota, a PodSet whose Pods request no resources can now be admitted. (#17335, @thc1006)

• TAS: Fixed a bug where a cold-started scheduler could admit workloads onto occupied nodes before existing ClusterQueues and their admitted Workloads were loaded into its cache. Kueue now waits for cache-filling handlers to process their initial data before scheduling. (#16630, @dundysm)

• TAS: Fixed a bug where replacing a failed node never completed for a Workload with some Pods already succeeded, leaving the recreated Pod gated. Kueue now keeps the admitted Pod count and resource usage when it updates the topology assignment. (#17503, @Narwhal-fish)

• TAS: Fixed a bug where, with ElasticJobsViaWorkloadSlicesWithTAS enabled, Pods left after an elastic Job scale-down could be ungated into topology domains assigned to Pods at lower ranks. Kueue also no longer logs the failed to read rank information from pods error on every reconcile of a TAS Indexed Job with completions above parallelism. (#17341, @ivnovakov)

• TrainJob: Fixed a bug where a TrainJob was never admitted when a Workload with the same name as the TrainJob existed in the same namespace. (#17572, @henry3260)

Other (Cleanup or Flake)

• Logging: Added the Alpha feature gate LogLevelForSchedulerMainLoop (disabled by default) to move eight scheduler progress messages from verbosity 2 to verbosity 3. Enable it with --feature-gates=LogLevelForSchedulerMainLoop=true to reduce scheduler log volume at verbosity 2. If you enable this gate and rely on these logs for observability dashboards, set logging verbosity to 3 or higher. (#17371, @kevin85421)
• Observability: Improved the performance of LocalQueue pending-workload metrics by avoiding a scan of all pending workloads in the ClusterQueue on each queue change. (#17417, @...

Changes since v0.20.1:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.19.0, v0.20.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.20.1.

Changes by Kind

Feature

• KueueViz: The workloads dashboard now shows the Pods of pod group Workloads, including StatefulSet and LeaderWorkerSet Workloads. (#17186, @henry3260)

Bug or Regression

• CLI: Fixed a bug that caused kueuectl create clusterqueue to reject quota specifications with hyphenated resource names. Kueue now accepts these resource names. (#16987, @Dasmat13)
• CLI: Fixed a bug where kueuectl list workload failed with "resource name may not be empty" when a pending Workload had no spec.queueName. Such Workloads are now listed with empty LocalQueue and ClusterQueue columns. (#16986, @tenzen-y)
• CLI: Fixed a bug where kueuectl version printed only the client version, without the controller manager image, when Kueue was installed with Helm under a release name other than kueue. (#16953, @Eason09053360)
• CLI: Fixed a bug where kueuectl list workload --for could list workloads more than once when a paginated job-UID search ended with an empty page. (#17084, @Huangpinhan)
• CLI: Skip remote ClusterQueue lookup in create localqueue when using --dry-run=client. (#17109, @Dasmat13)
• Configuration: Fixed the admissionFairSharing.usageSamplingInterval and admissionFairSharing.resourceWeights validation errors to report the value at the field they name. (#16943, @thc1006)
• DRA: Fixed a bug where a Workload updated while the scheduler was processing it could be requeued with quota charges computed for its previous spec, causing over- or under-admission. Kueue now hands such a Workload back to the workload controller to recompute its charges. (#17208, @Anjali-Chauhan1)
• ElasticJobsViaWorkloadSlices: Fixed a bug where an admission retry could cause a zero-count PodSet to lose a suitable flavor and block later scale-up. Kueue now selects a suitable flavor across retries. (#17107, @prash2512)
• FairSharing: Fixed the preemption bug where workloads within nominal quota were preempted if the ClusterQueue borrowed on an uncontested flavor or within an internal sub-cohort. (#17031, @akshat-lakhera)
• JobSet: Fixed a bug where an evicted Workload could keep its quota reserved, and was not requeued, if a JobSet restart had recreated child Jobs that had already succeeded. (#17215, @MaysaMacedo)
• KueueViz: Fix the workload Status column showing the first condition type instead of the workload status. (#17191, @onkar717)
• KueueViz: Fixed a bug where the backend HTTP server could hang indefinitely during graceful shutdown if active HTTP requests did not complete. The server now enforces a 30-second shutdown timeout. (#17198, @fat-catTW)
• KueueViz: Fixed indefinite loading on the Local Queues, Cluster Queues, Cohorts, and Resource Flavors pages when no resources are found. (#17175, @weizhoublue)
• KueueViz: The namespaces websocket now only provides LocalQueue-backed live updates to users with cluster-wide watch access to LocalQueues. Users without that permission still receive the initial namespace snapshot filtered by their list permissions. (#17180, @fat-catTW)
• Observability: Fixed a bug where kueue_cluster_queue_weighted_share and kueue_replaced_workload_slices_total kept reporting a deleted ClusterQueue, and kueue_cluster_queue_weighted_share kept a stale series with the old cohort label after a ClusterQueue moved to another Cohort. (#17239, @tenzen-y)
• Observability: Fixed a bug where custom-label updates to a ClusterQueue or LocalQueue could leave stale metric gauge series in Prometheus. (#17219, @akshat-lakhera)
• Observability: Fixed a bug where deleting a ClusterQueue could remove the kueue_cohort_subtree_quota and kueue_cohort_subtree_resource_reservations series of its Cohort while other ClusterQueues or child Cohorts remained in it. (#16941, @henry3260)
• Pod, AppWrapper & Kubeflow Jobs: Fixed a bug where changing the kueue.x-k8s.io/queue-name label of a suspended job to an invalid LocalQueue name was accepted, so the Workload could not move to the new queue and the job reconciler failed on every retry. Kueue now rejects the update. (#16972, @tenzen-y)
• Pod: Avoid counting a terminating Pod's deletion grace period twice when deciding whether its PodGroup still holds quota. (#17077, @gola)
• Pod: Fixed a bug where, with SkipReassignmentForPodOwnedWorkloads enabled, a Workload owned by a single Pod that was preempted or evicted due to node failures while its Pod was still scheduling-gated (for example while an AdmissionCheck was pending) stayed parked with Requeued=False forever and its Pod was never deleted. Such Workloads are now requeued, and Workloads already left in that state by an earlier version are requeued on upgrade. (#17173, @nsega)
• Pods: Fixed a bug where a Pod could bypass ClusterQueue quota by setting the kueue.x-k8s.io/role-hash annotation to another role's value and running an oversized Pod under that role's reservation. Kueue now verifies, before removing the scheduling gate, that each Pod of a group fits the effective spec (LimitRange and RuntimeClass adjustments applied) of the PodSet its role hash names; a Pod that does not fit is kept gated and a Warning event is emitted. The gated Pod does not fail the Workload or disturb running peers, but the group cannot become PodsReady until the Pod is replaced by one that fits; with waitForPodsReady, the Workload is evicted and requeued (up to the requeue limit).
  The check is controlled by the Alpha PodIntegrationVerifyRoleRequests feature gate, disabled by default on this release. (#17123, @vladikkuzn)
• RayService: Fixed webhook validation errors referencing spec.rayClusterSpec instead of the actual API field spec.rayClusterConfig. (#17097, @cryo-zd)
• Scheduling: Fixed a bug that could prevent ClusterQueues in the same tree from requeueing workloads when quota became available after a Cohort cycle was resolved. (#17079, @henry3260)
• Scheduling: Fixed a bug where a Workload that needed to borrow quota could not preempt Workloads in its own ClusterQueue, even when withinClusterQueue allowed preemption. Kueue now preempts those Workloads. (#17010, @zhengchenyu)
• SparkApplication: Fixed a bug where a SparkApplication without spec.executor.instances could not be suspended when its Workload was evicted or deactivated, because Kueue wrote back an invalid instances: 0. (#16968, @tenzen-y)
• TAS: Fixed a bug where a workload with non-adjacent PodSet group members (for example, a JobSet) could start with another PodSet's labels and node selectors, leaving some Pods scheduling-gated. (#17089, @apullo777)
• TAS: Fixed a bug where topology preemption could exclude workloads on overlapping TAS flavors, even when removing them could make the incoming workload fit. With TASHandleOverlappingFlavors enabled, Kueue now considers those workloads as candidates. (#16919, @Nilsachy)
• TAS: Fixed a bug with FlavorFungibilityPreserveScanProgress enabled where workloads using TryNextFlavor could repeatedly retry a topology-blocked ResourceFlavor or skip viable borrowing flavors across scheduling cycles when all flavors required borrowing or when a later flavor had nominal quota. (#17070, @j-skiba)
• TAS: Improved required-topology placement failure messages to identify the required level and explain that all Pods must fit in a single topology domain. (#17172, @HasonoCell)
• Workload: Fixed the validation message shown when changing spec.maximumExecutionTimeSeconds on an admitted Workload to say "immutable while the workload is admitted" instead of "while workload quota reserved". (#16946, @thc1006)
• Workload: Reject negative Pod overhead when WorkloadValidateResourcesAreNonNegative is enabled. (#17201, @nareshpatel1015)

Changes since v0.19.8:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.19.1, v0.19.2, v0.19.3, v0.19.4, v0.19.5, v0.19.6, v0.19.7, v0.19.8.

• Observability: Kueue now rejects invalid metrics.localQueueMetrics.localQueueSelector values at startup. Previously, enabled LocalQueue metrics included all queues when this selector was invalid.

  Correct invalid selectors before upgrading if metrics.localQueueMetrics.enable is true. (#16977, @tenzen-y)

Changes by Kind

Feature

• KueueViz: The workloads dashboard now shows the Pods of pod group Workloads, including StatefulSet and LeaderWorkerSet Workloads. (#17189, @henry3260)

Bug or Regression

• CLI: Fixed a bug that caused kueuectl create clusterqueue to reject quota specifications with hyphenated resource names. Kueue now accepts these resource names. (#16988, @Dasmat13)
• CLI: Fixed a bug where kueuectl list workload failed with "resource name may not be empty" when a pending Workload had no spec.queueName. Such Workloads are now listed with empty LocalQueue and ClusterQueue columns. (#16978, @tenzen-y)
• CLI: Fixed a bug where kueuectl version printed only the client version, without the controller manager image, when Kueue was installed with Helm under a release name other than kueue. (#16966, @tenzen-y)
• CLI: Fixed a bug where kueuectl list workload --for could list workloads more than once when a paginated job-UID search ended with an empty page. (#16850, @Huangpinhan)
• CLI: Fixed a bug where kueuectl list workload --for did not finish when -l included kueue.x-k8s.io/job-uid and no Workloads matched. It now reports no matching Workloads and exits. (#16963, @tenzen-y)
• CLI: Skip remote ClusterQueue lookup in create localqueue when using --dry-run=client. (#17110, @Dasmat13)
• Configuration: Fixed the admissionFairSharing.usageSamplingInterval and admissionFairSharing.resourceWeights validation errors to report the value at the field they name. (#16942, @thc1006)
• DRA: Fixed a bug where a Workload updated while the scheduler was processing it could be requeued with quota charges computed for its previous spec, causing over- or under-admission. Kueue now hands such a Workload back to the workload controller to recompute its charges. (#17211, @Anjali-Chauhan1)
• ElasticJobsViaWorkloadSlices: Fixed a bug where an admission retry could cause a zero-count PodSet to lose a suitable flavor and block later scale-up. Kueue now selects a suitable flavor across retries. (#17146, @prash2512)
• FairSharing: Fixed the preemption bug where workloads within nominal quota were preempted if the ClusterQueue borrowed on an uncontested flavor or within an internal sub-cohort. (#17095, @akshat-lakhera)
• JobSet: Fixed a bug where an evicted Workload could keep its quota reserved, and was not requeued, if a JobSet restart had recreated child Jobs that had already succeeded. (#17220, @MaysaMacedo)
• KueueViz: Fix the workload Status column showing the first condition type instead of the workload status. (#17192, @onkar717)
• KueueViz: Fixed a bug where the backend HTTP server could hang indefinitely during graceful shutdown if active HTTP requests did not complete. The server now enforces a 30-second shutdown timeout. (#17199, @fat-catTW)
• KueueViz: Fixed indefinite loading on the Local Queues, Cluster Queues, Cohorts, and Resource Flavors pages when no resources are found. (#17179, @weizhoublue)
• Observability: Fixed a bug where kueue_cluster_queue_weighted_share and kueue_replaced_workload_slices_total kept reporting a deleted ClusterQueue, and kueue_cluster_queue_weighted_share kept a stale series with the old cohort label after a ClusterQueue moved to another Cohort. (#17238, @tenzen-y)
• Observability: Fixed a bug where custom-label updates to a ClusterQueue or LocalQueue could leave stale metric gauge series in Prometheus. (#17046, @akshat-lakhera)
• Observability: Fixed a bug where deleting a ClusterQueue could remove the kueue_cohort_subtree_quota and kueue_cohort_subtree_resource_reservations series of its Cohort while other ClusterQueues or child Cohorts remained in it. (#17240, @tenzen-y)
• Pod, AppWrapper & Kubeflow Jobs: Fixed a bug where changing the kueue.x-k8s.io/queue-name label of a suspended job to an invalid LocalQueue name was accepted, so the Workload could not move to the new queue and the job reconciler failed on every retry. Kueue now rejects the update. (#16973, @tenzen-y)
• Pod: Avoid counting a terminating Pod's deletion grace period twice when deciding whether its PodGroup still holds quota. (#17087, @gola)
• Pod: Fixed a bug where, with SkipReassignmentForPodOwnedWorkloads enabled, a Workload owned by a single Pod that was preempted or evicted due to node failures while its Pod was still scheduling-gated (for example while an AdmissionCheck was pending) stayed parked with Requeued=False forever and its Pod was never deleted. Such Workloads are now requeued, and Workloads already left in that state by an earlier version are requeued on upgrade. (#17204, @nsega)
• Pods: Fixed a bug where a Pod could bypass ClusterQueue quota by setting the kueue.x-k8s.io/role-hash annotation to another role's value and running an oversized Pod under that role's reservation. Kueue now verifies, before removing the scheduling gate, that each Pod of a group fits the effective spec (LimitRange and RuntimeClass adjustments applied) of the PodSet its role hash names; a Pod that does not fit is kept gated and a Warning event is emitted. The gated Pod does not fail the Workload or disturb running peers, but the group cannot become PodsReady until the Pod is replaced by one that fits; with waitForPodsReady, the Workload is evicted and requeued (up to the requeue limit).
  The check is controlled by the Alpha PodIntegrationVerifyRoleRequests feature gate, disabled by default on this release. (#17124, @vladikkuzn)
• RayService: Fixed webhook validation errors referencing spec.rayClusterSpec instead of the actual API field spec.rayClusterConfig. (#17205, @cryo-zd)
• Scheduling: Fixed a bug that could prevent ClusterQueues in the same tree from requeueing workloads when quota became available after a Cohort cycle was resolved. (#17080, @henry3260)
• Scheduling: Fixed a bug where a Workload that needed to borrow quota could not preempt Workloads in its own ClusterQueue, even when withinClusterQueue allowed preemption. Kueue now preempts those Workloads. (#17016, @zhengchenyu)
• SparkApplication: Fixed a bug where a SparkApplication with restartPolicy OnFailure or Always could run outside ClusterQueue quota after the Spark operator retried a failed submission. Kueue now keeps the Workload admitted while the application is in SUBMISSION_FAILED. (#16976, @tenzen-y)
• SparkApplication: Fixed a bug where a SparkApplication without spec.executor.instances could not be suspended when its Workload was evicted or deactivated, because Kueue wrote back an invalid instances: 0. (#16967, @tenzen-y)
• TAS: Fixed a bug where a workload with non-adjacent PodSet group members (for example, a JobSet) could start with another PodSet's labels and node selectors, leaving some Pods scheduling-gated. (#17092, @apullo777)
• TAS: Fixed a bug with FlavorFungibilityPreserveScanProgress enabled where workloads using TryNextFlavor could repeatedly retry a topology-blocked ResourceFlavor or skip viable borrowing flavors across scheduling cycles when all flavors required borrowing or when a later flavor had nominal quota. (#17072, @j-skiba)
• TAS: Improved required-topology placement failure messages to identify the required level and explain that all Pods must fit in a single topology domain. (#17241, @tenzen-y)
• Workload: Fixed the validation message shown when changing spec.maximumExecutionTimeSeconds on an admitted Workload to say "immutable while the workload is admitted" instead of "while workload quota reserved". (#16944, @thc1006)
• Workload: Reject negative Pod overhead when WorkloadValidateResourcesAreNonNegative is enabled. (#17202, @nareshpatel1015)

Changes since v0.20.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.19.0, v0.20.0.

Changes by Kind

Feature

• ElasticJobsViaWorkloadSlices: Fixed a bug where elastic Pods ungated for a ProvisioningRequest could carry the consume-provisioning-request annotation without provisioning-class-name when they were created before the job template was stamped. Kueue now applies both annotations, preventing such Pods from being pinned by the cluster autoscaler to nodes that are never created. (#16543, @neilb-dotcom)

Bug or Regression

• CLI: Fixed a bug where kueuectl create clusterqueue failed with "must match the name in coveredResources" when flavors listed the same resources in a different order, for example --nominal-quota "alpha:cpu=1;memory=1,beta:memory=2;cpu=2". (#16544, @Eason09053360)
• CLI: Fixed a bug where kueuectl delete workload deleted the owning batch/v1 Job but left its Pods running and the Workload in place. Owners are now deleted with background propagation, matching kubectl delete. (#16511, @Eason09053360)
• ElasticJobsViaWorkloadSlices: Fixed a bug where a replacement Workload could remain pending when one PodSet shrank below its predecessor's grant while another grew. (#16892, @weizhoublue)
• ElasticJobsViaWorkloadSlices: Fixed a bug where resizing an elastic job could let the scheduler admit workloads beyond a cohort's quota in the same scheduling cycle. With ElasticJobsViaWorkloadSlicesWithTAS enabled, the replaced slice's topology usage no longer defers other workloads to the next scheduling cycle. (#16601, @apullo777)
• FairSharing: Fixed kueue_cohort_weighted_share to report NaN instead of +Inf when a zero-weight Cohort is borrowing quota, matching its help text and kueue_cluster_queue_weighted_share. (#16818, @abhayjoshi201)
• FairSharing: Fixed a bug where a ClusterQueue or Cohort whose weighted share exceeded the int64 range could report a negative status.fairSharing.weightedShare. Such a share is now reported as the int64 maximum. (#16592, @thc1006)
• FairSharing: Fixed a bug where a ClusterQueue or Cohort with weight 0 that borrowed only resources its Cohort had nothing lendable for got a weighted share of 0 instead of the maximum, and was ranked as if it were within its quota. (#16567, @thc1006)
• FairSharing: Fixed a bug where a borrowing ClusterQueue or Cohort with a very large weight, such as 1e308, could report a status.fairSharing.weightedShare of 0 and be ranked as if it were within its quota. Kueue now keeps such a share above zero: the status reports 1, and the kueue_cluster_queue_weighted_share and kueue_cohort_weighted_share metrics report the smallest positive float64 value. (#16584, @thc1006)
• FairSharing: Fixed a bug where restoring same-ClusterQueue workloads could make cross-ClusterQueue preemption targets violate fair-sharing rules and cause preemption loops. Kueue now validates targets against the final simulated post-admission state. (#16912, @JanKaczmarski)
• JobFramework: Fixed Workloads inheriting Kueue's internal labels and annotations from the Job or Pod when they were listed in labelKeysToCopy or as custom metric sources. (#16773, @thc1006)
• KueueViz: Fix the LocalQueue detail page missing the Status cell in the workloads table and returning null instead of an empty array for queues with no workloads. (#16692, @Dasmat13)
• KueueViz: Fix the workload detail page not showing the ClusterQueue name and preemption state. (#16608, @onkar717)
• KueueViz: Fixed a bug where the workloads dashboard listed unrelated Pods for Workloads without a kueue.x-k8s.io/job-uid label, such as pod groups. Such Workloads now show an empty Pods list. (#16702, @henry3260)
• MultiKueue: Enabling MultiKueueRayInTreeAutoscaling now requires SkipChildJobSuspension to be enabled. Kueue reports a startup validation error if this dependency is disabled or omitted. (#16814, @kevin85421)
• MultiKueue: Fixed a bug where a replaced workload slice could keep holding quota on a worker cluster when the scheduler failed to finish it after admitting the replacement slice. (#16772, @kevin85421)
• Observability: Fix stale pending-workload metrics when LocalQueues are created, resumed, stopped, or deleted, including Workloads observed before their LocalQueue. (#16890, @tomsen02)
• Observability: Fixed a bug where the kueue_replaced_workload_slices_total metric was not exposed on the metrics endpoint. (#16763, @Narwhal-fish)
• Observability: Fixed stale kueue_pending_workloads series that remained after a ClusterQueue was stopped or resumed when CustomMetricLabels was enabled with a Workload-sourced custom label, which caused pending workloads to be double-counted. (#16871, @kevin85421)
• Observability: Refresh pending-workload metrics when ClusterQueues stop or resume, without relying on subsequent Workload updates. (#16891, @tomsen02)
• Ray integrations: Fixed a bug where workloads with worker groups using replicas and numOfHosts were not admitted because SubGroupCount and topology index labels were missing from the topology request. The KubeRayEvictOnInconsistentTopologyRequest feature gate is disabled by default, allowing existing workloads to continue running with old topology requests. (#15677, @spencer-p)
• Ray integrations: Use zero instead of one as the worker replica fallback when replicas is absent, matching the KubeRay CRD default. (#16908, @kevin85421)
• RayService: Added the Alpha KubeRayServiceUsingTopLevelSuspend feature gate, disabled by default, to opt in to top-level spec.suspend handling. Upgrade KubeRay to v1.7.0 or newer before enabling the gate. (#16882, @kevin85421)
• Scheduling: Fixed a bug where Workloads blocked by a closed preemption gate could retain a stale QuotaReserved condition and status.resourceRequests. Such Workloads now report the PreemptionGated reason on the QuotaReserved condition instead of AdmissionGated. (#16699, @henry3260)
• Scheduling: Fixed a bug where a CPU amount past the Quantity limit appeared in quota messages without the milli suffix, so it read as cores. (#16716, @HarshithaMS005)
• Scheduling: Fixed a bug where a Workload whose total request exceeded the int64 range could be admitted against a quota it did not fit. (#16549, @HarshithaMS005)
• Scheduling: Fixed a bug where partial admission of a PodSet whose total request overflowed int64 could admit more Pods than the quota allows, because the reduced request was derived from the saturated total instead of the per-Pod request. (#16519, @thc1006)
• StatefulSet: Fixed a bug where changing a StatefulSet’s queue while its Workload held quota could cause repeated Workload update failures when the StatefulSet had no ready replicas. Kueue now rejects the queue change until the Workload releases its quota. (#16705, @weizhoublue)
• StatefulSet: Fixed a bug where labels listed in integrations.labelKeysToCopy, and the labels and annotations used as CustomMetricLabels sources, were not copied from a StatefulSet into its Workload. (#16751, @nsega)
• TAS: Fixed a bug where a Workload using delayed topology assignment or failed node replacement could lose its quota reservation, and for a failed node replacement also its admission. (#16491, @apullo777)
• TAS: Fixed a bug where a Workload using delayed topology assignment or failed node replacement could miss a scheduling retry until it was updated again. (#16503, @apullo777)
• TAS: Fixed a bug where a second scheduling pass (failed-node replacement or delayed topology assignment) computed from an outdated Workload could overwrite a newer admission or drop status updated in the meantime, such as an Evicted condition. The scheduler now rejects the stale write, keeps the Workload's reservation, emits an OutdatedScheduleCycle event and retries the second pass. (#16858, @anguszzzz)
• TAS: Fixed a bug where an elastic Workload with a TAS PodSet group could reassign its running leader to a different domain when the workers scaled up from zero, with the ElasticJobsViaWorkloadSlicesWithTAS feature gate enabled. (#16820, @Narwhal-fish)
• TAS: Fixed a bug where stuck pods could break rank-based pod ungating after node replacement or workload re-admission. (#16796, @pajakd)
• TAS: Fixed a scheduler panic when an elastic Workload scaled the workers of a PodSet group from 0 to the same count as the leader, with the ElasticJobsViaWorkloadSlicesWithTAS feature gate enabled. (#16810, @Narwhal-fish)
• TAS: Fixed workloads with non-adjacent PodSet group members getting stuck during flavor-scan resume or delayed/failed-node topology reassignment. (#16757, @apullo777)
• TAS: With KueueDRADeviceFeasibility enabled, fixed a bug where PodSets of one Workload could be placed on a node whose device plugin publishes fewer devices of a DRA-backed extended resource than they need together, leaving Pods Pending. The same miscount could keep an elastic scale up pending. (#16809, @thc1006)

Other (Cleanup or Flake)

• AgentSkills: Do not require the acknowledgment that the skills are experimental. (#16865, @mimowo)
• ConfigAPI: Kueue logs configured internal labels in integrations.labelKeysToCopy at startup. Kueue does not copy these labels to a Workload. (#16874, @thc1006)

Changes since v0.19.7:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.19.1, v0.19.2, v0.19.3, v0.19.4, v0.19.5, v0.19.6, v0.19.7.

Changes by Kind

Bug or Regression

• CLI: Fixed a bug where kueuectl create clusterqueue failed with "must match the name in coveredResources" when flavors listed the same resources in a different order, for example --nominal-quota "alpha:cpu=1;memory=1,beta:memory=2;cpu=2". (#16545, @Eason09053360)
• CLI: Fixed a bug where kueuectl delete workload deleted the owning batch/v1 Job but left its Pods running and the Workload in place. Owners are now deleted with background propagation, matching kubectl delete. (#16512, @Eason09053360)
• ElasticJobsViaWorkloadSlices: Fixed a bug where resizing an elastic job could let the scheduler admit workloads beyond a cohort's quota in the same scheduling cycle. With ElasticJobsViaWorkloadSlicesWithTAS enabled, the replaced slice's topology usage no longer defers other workloads to the next scheduling cycle. (#16125, @apullo777)
• FairSharing: Fix kueue_cohort_weighted_share metric to report NaN instead of +Inf when a Cohort with fairSharing.weight: 0 is actively borrowing resources, matching kueue_cluster_queue_weighted_share and the metric documentation. (#16864, @abhayjoshi201)
• FairSharing: Fixed a bug where a ClusterQueue or Cohort whose weighted share exceeded the int64 range could report a negative status.fairSharing.weightedShare. Such a share is now reported as the int64 maximum. (#16593, @thc1006)
• FairSharing: Fixed a bug where a ClusterQueue or Cohort with weight 0 that borrowed only resources its Cohort had nothing lendable for got a weighted share of 0 instead of the maximum, and was ranked as if it were within its quota. (#16568, @thc1006)
• FairSharing: Fixed a bug where a borrowing ClusterQueue or Cohort with a very large weight, such as 1e308, could report a status.fairSharing.weightedShare of 0 and be ranked as if it were within its quota. Kueue now keeps such a share above zero: the status reports 1, and the kueue_cluster_queue_weighted_share and kueue_cohort_weighted_share metrics report the smallest positive float64 value. (#16587, @thc1006)
• FairSharing: Fixed a bug where restoring same-ClusterQueue workloads could make cross-ClusterQueue preemption targets violate fair-sharing rules and cause preemption loops. Kueue now validates targets against the final simulated post-admission state. (#16911, @JanKaczmarski)
• JobFramework: Added the Alpha SkipChildJobSuspension feature gate (disabled by default). When enabled, Kueue no longer suspends managed child jobs based on the ancestor Workload's state and leaves their lifecycle to the ancestor controller. (#16460, @kevin85421)
• JobFramework: Fixed Workloads inheriting Kueue's internal labels and annotations from the Job or Pod when they were listed in labelKeysToCopy or as custom metric sources. (#16774, @thc1006)
• KueueViz: Fix the LocalQueue detail page missing the Status cell in the workloads table and returning null instead of an empty array for queues with no workloads. (#16693, @Dasmat13)
• KueueViz: Fix the workload detail page not showing the ClusterQueue name and preemption state. (#16905, @tenzen-y)
• KueueViz: Fixed a bug where the workloads dashboard listed unrelated Pods for Workloads without a kueue.x-k8s.io/job-uid label, such as pod groups. Such Workloads now show an empty Pods list. (#16707, @henry3260)
• Observability: Fix stale pending-workload metrics when LocalQueues are created, resumed, stopped, or deleted, including Workloads observed before their LocalQueue. (#16888, @tomsen02)
• Observability: Fixed a bug where the kueue_replaced_workload_slices_total metric was not exposed on the metrics endpoint. (#16856, @Narwhal-fish)
• Observability: Refresh pending-workload metrics when ClusterQueues stop or resume, without relying on subsequent Workload updates. (#16889, @tomsen02)
• Ray integrations: Fixed a bug where workloads with worker groups using replicas and numOfHosts were not admitted because SubGroupCount and topology index labels were missing from the topology request. The KubeRayEvictOnInconsistentTopologyRequest feature gate is disabled by default, allowing existing workloads to continue running with old topology requests. (#15677, @spencer-p)
• Ray integrations: Use zero instead of one as the worker replica fallback when replicas is absent, matching the KubeRay CRD default. (#16915, @mbobrovskyi)
• Scheduling: Fixed a bug where Workloads blocked by a closed preemption gate could retain a stale QuotaReserved condition and status.resourceRequests. Such Workloads now report the PreemptionGated reason on the QuotaReserved condition instead of AdmissionGated. (#16761, @henry3260)
• Scheduling: Fixed a bug where a CPU amount past the Quantity limit appeared in quota messages without the milli suffix, so it read as cores. (#16717, @HarshithaMS005)
• Scheduling: Fixed a bug where a Workload whose total request exceeded the int64 range could be admitted against a quota it did not fit. (#16671, @HarshithaMS005)
• Scheduling: Fixed a bug where partial admission of a PodSet whose total request overflowed int64 could admit more Pods than the quota allows, because the reduced request was derived from the saturated total instead of the per-Pod request. (#16520, @thc1006)
• StatefulSet: Fixed a bug where changing a StatefulSet's queue while its Workload held quota could cause repeated Workload update failures when the StatefulSet had no ready replicas. Kueue now rejects the queue change until the Workload releases its quota. (#16710, @weizhoublue)
• StatefulSet: Fixed a bug where labels listed in integrations.labelKeysToCopy, and the labels and annotations used as CustomMetricLabels sources, were not copied from a StatefulSet into its Workload. (#16770, @nsega)
• TAS: Fixed a bug where a Workload using delayed topology assignment or failed node replacement could lose its quota reservation, and for a failed node replacement also its admission. (#16462, @apullo777)
• TAS: Fixed a bug where a Workload using delayed topology assignment or failed node replacement could miss a scheduling retry until it was updated again. (#16468, @apullo777)
• TAS: Fixed a bug where a second scheduling pass (failed-node replacement or delayed topology assignment) computed from an outdated Workload could overwrite a newer admission or drop status updated in the meantime, such as an Evicted condition. The scheduler now rejects the stale write, keeps the Workload's reservation, emits an OutdatedScheduleCycle event and retries the second pass. (#16859, @anguszzzz)
• TAS: Fixed a bug where an elastic Workload with a TAS PodSet group could reassign its running leader to a different domain when the workers scaled up from zero, with the ElasticJobsViaWorkloadSlicesWithTAS feature gate enabled. (#16817, @Narwhal-fish)
• TAS: Fixed a bug where stuck pods could break rank-based pod ungating after node replacement or workload re-admission. (#16795, @pajakd)
• TAS: Fixed a scheduler panic when an elastic Workload scaled the workers of a PodSet group from 0 to the same count as the leader, with the ElasticJobsViaWorkloadSlicesWithTAS feature gate enabled. (#16812, @Narwhal-fish)
• TAS: Fixed workloads with non-adjacent PodSet group members getting stuck during flavor-scan resume or delayed/failed-node topology reassignment. (#16658, @apullo777)

Other (Cleanup or Flake)

• AgentSkills: Do not require the acknowledgment that the skills are experimental. (#16869, @mimowo)
• ConfigAPI: Kueue logs configured internal labels in integrations.labelKeysToCopy at startup. Kueue does not copy these labels to a Workload. (#16878, @thc1006)

Changes since v0.19.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.

• AFS: LocalQueue usage is now accounted from quota reservation rather than admission for ClusterQueues using UsageBasedAdmissionFairSharing. AdmissionFairSharingAnchorAtQuotaReservation is Beta and enabled by default in v0.20.

  If you explicitly disable AdmissionFairSharing, also disable AdmissionFairSharingAnchorAtQuotaReservation before upgrading to v0.20, or Kueue will reject the configuration at startup. Before rolling back to a release that does not recognize AdmissionFairSharingAnchorAtQuotaReservation, remove it from the configuration. (#15000, @apullo777)

• API: Kueue no longer serves kueue.x-k8s.io/v1beta1.

Update manifests and API clients to v1beta2 before upgrading. Run the migration script before upgrading to rewrite existing objects in v1beta2 storage. (#14558, @mbobrovskyi)

• DRA & ResourceTransformation: Fixed a bug where DRA device-class mapping or a resource transformation under the reserved resource name pods was silently discarded or left the Workload permanently pending.

Remove or rename those entries before upgrading, or the kueue-controller-manager will fail to start. Renaming a mapping name or an outputs key also requires updating the matching ClusterQueue nominalQuota entries in the same change. (#13989, @thc1006)

• DRA: Fixed missing quota charges for Pod overhead and transformation outputs that share a DRA resource name. Restartable init-container requests are now counted alongside regular-container requests.

Review ClusterQueue quotas for mapped logical resources and contributions still charged under original resource names. When both use one name, quota must cover their sum. With BlockUndeclared, uncovered contributions leave Workloads pending; with IgnoreUndeclared, they are not charged. Corrected accounting applies when Workloads are evaluated. (#14154, @thc1006)

• KueueViz: Added Kubernetes RBAC checks through SubjectAccessReview for API and WebSocket requests when authentication is enabled.

Ensure users have the get, list, and watch permissions needed for their views. For non-Helm or custom RBAC installations, ensure the backend ServiceAccount can create subjectaccessreviews in the authorization.k8s.io API group. The Helm chart includes this permission. (#13810, @mykysha)

• LeaderWorkerSet: Fixed a quota bypass where raising spec.leaderWorkerTemplate.size on an already-admitted, Kueue-managed LeaderWorkerSet ran more pods per group than the reserved quota covered. spec.leaderWorkerTemplate.size is now immutable while the LeaderWorkerSet is managed by Kueue, behind the new LWSImmutableGroupSize feature gate (Beta, enabled by default). spec.replicas stays mutable.

If you change spec.leaderWorkerTemplate.size on a Kueue-managed LeaderWorkerSet, recreate it at the new size instead, or disable the LWSImmutableGroupSize feature gate to keep the previous behavior, which also restores the quota bypass. (#13279, @ivnovakov)

• Observability: Kueue now rejects invalid metrics.localQueueMetrics.localQueueSelector values at startup. Previously, enabled LocalQueue metrics included all queues when this selector was invalid.

Correct invalid selectors before upgrading if metrics.localQueueMetrics.enable is true. (#16379, @weizhoublue)

• RayService: Starting in v0.20, the Beta RayServiceValidateUpgradeStrategy feature gate is enabled by default.

Kueue-managed RayServices that use zero-downtime upgrades must enable the ElasticJobsViaWorkloadSlices feature gate and set the kueue.x-k8s.io/elastic-job annotation to "true". To disable zero-downtime upgrades, explicitly set spec.upgradeStrategy.type to None. (#16094, @kevin85421)

• SparkApplication: Fixed a bug where Workloads reserved less CPU and memory than Spark requests for the driver and executor Pods, because cores, memoryOverhead and Spark's default memory overhead were ignored. Kueue now reserves the same CPU and memory that Spark requests. Memory values must use the Java format accepted by Spark (for example 512m or 2g); Kubernetes-style values such as 512Mi are rejected.

If SparkApplications rely on cores or on the default memory overhead, raise the ClusterQueue quotas accordingly: each Pod now reserves its cores as CPU and at least 384Mi of additional memory. (#15833, @henry3260)

• TAS: Fix a bug where TASRecomputeAssignmentWithinSchedulingCycle can be enabled even if TopologyAwareScheduling is disabled.

If you disable TopologyAwareScheduling, also set TASRecomputeAssignmentWithinSchedulingCycle=false before upgrading. (#14237, @tenzen-y)

• TAS: Workload slice sizes must be positive, and podSetSliceRequiredTopology and podSetSliceSize must be set together.

Correct invalid manifests before upgrading. Disabling TASValidateWorkloadSliceSize disables only webhook checks; CRD minimum-size validation still applies. (#12728, @mszadkow)

• TrainJob: support Trainer v2.3.

If you use Trainer v2.2, you have to update it to v2.3 since Trainer v2.2 and v2.3 don't have API compatibility. (#14504, @SatishChoudhary642)

• Workloads: Fixed a bug that caused Workloads to be finalized as orphaned while their owner Job was being deleted. Kueue now waits until the owner Job no longer exists before finalizing the Workload.

If you implement the ComposableJob interface in a custom job integration, change your Load method to return (*jobframework.LoadResult, error) instead of (bool, error). Use jobframework.NewLoadResult(shouldFinalize, found) where found is false when the job is absent and true otherwise. (#12849, @mbobrovskyi)

Changes by Kind

API Change

• ConfigurablePreemptions: Added the alpha PreemptionConfig API for selecting additional Workloads to preempt when classical or Fair Sharing candidates are insufficient. Requires the ConfigurablePreemptions feature gate and alpha CRDs. (#15913, @danylott)
• PartialAdmission: Workload PodSet.minCount can now be set to 0, allowing PodSets with zero replicas. Previously, minCount had to be at least 1. (#14720, @vladikkuzn)
• TAS: Allowed mutating Topology.spec.levels when kubernetes.io/hostname is the lowest level both before and after the change. Workloads left pending under the previous levels are automatically retried, and the usage of already-admitted workloads is preserved across the change. (#13910, @tomsen02)
• TAS: Allowed updating spec.nodeLabels of a ResourceFlavor with topologyName set. Admitted workloads keep their quota reservation and topology assignment, pending workloads are requeued against the new node set, and usage of workloads admitted on overlapping flavors is counted exactly once. (#13738, @tomsen02)
• TAS: ResourceFlavor spec.tolerations can now be updated when spec.topologyName is set. (#13622, @tomsen02)

Feature

• CLI: Added ACTIVE and REASON columns to wide output for ClusterQueue and LocalQueue resources. (#15633, @prash2512)
• CLI: Added an ACTIVE column and an --active flag to kueuectl list localqueue, allowing users to see and filter LocalQueues by whether they have an active connection to their ClusterQueue (--active=true for active, --active=false for inactive). (#15866, @henry3260)
• CLI: The importer mapping file now supports a resources list in the match block, selecting only the Pods that request a non-zero amount of all the listed resources. (#15573, @Huang-Wei)
• ClusterQueue: Graduated RejectUpdatesToCQWithInvalidOnFlavors to Beta (enabled by default), which rejects ClusterQueue updates that reference invalid ResourceFlavor names in admissionCheckStrategy.onFlavors. (#14776, @tenzen-y)
• ConcurrentAdmission: Increased the maximum number of ResourceFlavors per ResourceGroup from 16 to 32 for ClusterQueues with ConcurrentAdmissionPolicy configured. (#15949, @ResidentMario)
• Controllers: Increased the maximum concurrent API updates per reconcile from 8 to 32 via the HighMaxParallelismWithinReconcile feature gate (Beta, enabled by default in v0.20). (#16336, @olekzabl)
• DRA: Added DeviceTaintRule support to the KueueDRADeviceFeasibility device check, so Kueue no longer admits Workloads onto tainted devices. (#16081, @sohankunkerkar)
• DRA: Added the KueueDRADeviceFeasibility feature gate, Alpha and disabled by default. Kueue checks device availability per node before admitting a Workload that uses ResourceClaimTemplates or DRA-backed extended resources, so quota is not reserved for a Workload kube-scheduler cannot place. (#15577, @sohankunkerkar)
• DRA: Added the KueueDRAIntegrationPrioritizedList feature gate (alpha, disabled by default). When enabled, a DRA firstAvailable request whose alternatives all ask for the same device count on DeviceClasses mapped to one logical resource is charged that count once. A request whose alternatives differ in count or in logical resource, or use a counter-backed or capacity-backed mapping, is refused. Kueue does not check that any alternative is satisfiable, and MultiKueue does not support firstAvailable requests. (#14130, @thc1006)
• DRA: Device feasibility checks now include taints from ResourceSlices and DeviceTaintRules. A Workload’s device requests must tolerate these taints. Requires both the KueueDRAIntegrationDeviceTaints and KueueDRADeviceFeasibility feature gates. (#16169, @sohankunkerkar)
• DRA: Device requests with adminAccess: true are now accepted without DRA quota charges. Other resource requests are counted normally. (#13642, @sohankunkerkar)
• DRA: With UnadmittedWorkloadsObser...

Changes since v0.19.6:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.19.1, v0.19.2, v0.19.3, v0.19.4, v0.19.5, v0.19.6.

Changes by Kind

Feature

• Controllers: Increased the maximum concurrent API updates per reconcile from 8 to 32. Controlled by the HighMaxParallelismWithinReconcile feature gate. (#16387, @mimowo)

Bug or Regression

• AFS: Fixed a bug where workloads could be scheduled in priority order instead of fair-sharing order when Admission Fair Sharing was enabled. (#16307, @abhayjoshi201)
• CLI: Fixed a bug where kueuectl list pods --for pod/NAME printed No resources found instead of listing the group members, when the Pod group name was stored in the kueue.x-k8s.io/pod-group-name annotation rather than the label. (#16408, @tenzen-y)
• CLI: Fixed a bug where list pods --for pod/NAME printed No resources found for a Pod that is not part of a pod group. The command now lists that Pod itself. (#16378, @tenzen-y)
• Cohorts: Fixed a possible controller hang after an invalid Cohort hierarchy cycle update. ClusterQueues in a cyclic Cohort now surface a clear CohortCycleDetected condition instead of hanging. (#16376, @tenzen-y)
• Helm: Fixed ServiceMonitor metrics scraping to verify the cert-manager-issued certificate when enableCertManager is true and no custom tlsConfig is set. (#16318, @HsiuChuanHsu)
• KueueViz: Fix Workloads page crash when no workloads exist by adding the missing Alert import. (#16396, @Dasmat13)
• LeaderWorkerSet & StatefulSet: Fixed a bug where, during a rolling update, Pods at the current revision were ungated before the Workload was admitted, so they could run without quota. Kueue now ungates these Pods only when the Workload is admitted. (#16401, @henry3260)
• LeaderWorkerSet: Fixed a bug where a rolling update with a percentage maxSurge (such as 50%) had its surge groups' Workloads deleted while those groups were still running. The value was read as zero, so Kueue treated the update as having no surge. (#14405, @thc1006)
• MultiKueue: Fixed a bug where evicted elastic workloads left stale remote objects on worker clusters, preventing re-admission from the manager-cluster specification. (#16178, @kevin85421)
• MultiKueue: Fixed a bug where the quota automation condition could retain a stale ObservedGeneration after a ClusterQueue spec update. Kueue now refreshes it after successful evaluation. (#16403, @mbobrovskyi)
• Observability: Fixed Admitted events reporting 0s when quota was reserved before admission. Events now show the actual time between quota reservation and admission. (#16412, @tenzen-y)
• Observability: Fixed kueue_cohort_subtree_admitted_active_workloads after Cohort hierarchy changes so it reflects admitted Workloads in the current subtree. (#16437, @tenzen-y)
• Pod: Fixed a bug where Pod groups could receive different ResourceFlavor assignments depending on client-provided role-hash values. PodSets are now ordered by scheduling shape rather than role-hash, making flavor assignment deterministic. Controlled by the PodGroupSchedulingShapeOrdering feature gate (Beta, enabled by default). (#14809, @amirialy)
• Scheduling: Fixed a bug that could leave workloads pending when preemption target selection omitted pods quota or used the spec count for a partially admitted workload-slice replacement. Kueue now sizes preemption targets using the assigned counts and resources. (#16200, @apullo777)
• Scheduling: Fixed a bug where preemption for workloads with non-adjacent members of a kueue.x-k8s.io/podset-group-name group could use incorrect resource requests or flavors, causing a scheduler panic or leaving a Workload pending. Kueue now matches PodSets by name. (#16273, @apullo777)
• StatefulSet: Fixed a bug where changing the kueue.x-k8s.io/queue-name label after the Pods were ungated but before any was Ready made the StatefulSet reconciler fail on every retry, and could keep a Pod scheduling-gated during a rolling update. Kueue now only updates the queue name on Pods that are still gated. (#16335, @henry3260)
• TAS: Fixed a bug where a Node or TAS ResourceFlavor change requeued inadmissible Workloads in every ClusterQueue instead of only the ClusterQueues that use that flavor. (#16152, @sohankunkerkar)

Other (Cleanup or Flake)

• FairSharing: Improved preemption performance by up to 20% by precomputing lendable Cohort capacity instead of recalculating it for each preemption candidate. (#15955, @venuchitta)

Changes since v0.18.10:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.18.1, v0.18.2, v0.18.3, v0.18.4, v0.18.5, v0.18.6, v0.18.7, v0.18.8, v0.18.9, v0.18.10.

Changes by Kind

Feature

• Controllers: Increased the maximum concurrent API updates per reconcile from 8 to 32. Controlled by the HighMaxParallelismWithinReconcile feature gate. (#16388, @mimowo)

Bug or Regression

• AFS: Fixed a bug where workloads could be scheduled in priority order instead of fair-sharing order when Admission Fair Sharing was enabled. (#16308, @abhayjoshi201)
• CLI: Fixed a bug where kueuectl list pods --for pod/NAME printed No resources found instead of listing the group members, when the Pod group name was stored in the kueue.x-k8s.io/pod-group-name annotation rather than the label. (#16407, @tenzen-y)
• CLI: Fixed a bug where list pods --for pod/NAME printed No resources found for a Pod that is not part of a pod group. The command now lists that Pod itself. (#16377, @tenzen-y)
• Cohorts: Fixed a possible controller hang after an invalid Cohort hierarchy cycle update. ClusterQueues in a cyclic Cohort now surface a clear CohortCycleDetected condition instead of hanging. (#16375, @tenzen-y)
• ConcurrentAdmission: Fixed a panic that left a parent Workload without variants when its name contained no "-". (#16380, @tenzen-y)
• Helm: Fixed ServiceMonitor metrics scraping to verify the cert-manager-issued certificate when enableCertManager is true and no custom tlsConfig is set. (#16319, @HsiuChuanHsu)
• KueueViz: Fix Workloads page crash when no workloads exist by adding the missing Alert import. (#16397, @Dasmat13)
• LeaderWorkerSet & StatefulSet: Fixed a bug where, during a rolling update, Pods at the current revision were ungated before the Workload was admitted, so they could run without quota. Kueue now ungates these Pods only when the Workload is admitted. (#16399, @henry3260)
• LeaderWorkerSet: Fixed a bug where a rolling update with a percentage maxSurge (such as 50%) had its surge groups' Workloads deleted while those groups were still running. The value was read as zero, so Kueue treated the update as having no surge. (#14405, @thc1006)
• MultiKueue: Fixed a bug where evicted elastic workloads left stale remote objects on worker clusters, preventing re-admission from the manager-cluster specification. (#16179, @kevin85421)
• MultiKueue: Fixed a bug where the quota automation condition could retain a stale ObservedGeneration after a ClusterQueue spec update. Kueue now refreshes it after successful evaluation. (#16404, @mbobrovskyi)
• Observability: Fixed Admitted events reporting 0s when quota was reserved before admission. Events now show the actual time between quota reservation and admission. (#16411, @tenzen-y)
• Observability: Fixed kueue_cohort_subtree_admitted_active_workloads after Cohort hierarchy changes so it reflects admitted Workloads in the current subtree. (#16398, @apullo777)
• Pod: Fixed a bug where Pod groups could receive different ResourceFlavor assignments depending on client-provided role-hash values. PodSets are now ordered by scheduling shape rather than role-hash, making flavor assignment deterministic. Controlled by the PodGroupSchedulingShapeOrdering feature gate (Alpha, disabled by default). (#16365, @amirialy)
• Scheduling: Fixed a bug that could leave workloads pending when preemption target selection omitted pods quota or used the spec count for a partially admitted workload-slice replacement. Kueue now sizes preemption targets using the assigned counts and resources. (#16201, @apullo777)
• Scheduling: Fixed a bug where preemption for workloads with non-adjacent members of a kueue.x-k8s.io/podset-group-name group could use incorrect resource requests or flavors, causing a scheduler panic or leaving a Workload pending. Kueue now matches PodSets by name. (#16273, @apullo777)
• StatefulSet: Fixed a bug where changing the kueue.x-k8s.io/queue-name label after the Pods were ungated but before any was Ready made the StatefulSet reconciler fail on every retry, and could keep a Pod scheduling-gated during a rolling update. Kueue now only updates the queue name on Pods that are still gated. (#16334, @henry3260)
• TAS: Fixed a bug where a Node or TAS ResourceFlavor change requeued inadmissible Workloads in every ClusterQueue instead of only the ClusterQueues that use that flavor. (#16152, @sohankunkerkar)

Other (Cleanup or Flake)

• FairSharing: Improved preemption performance by up to 20% by precomputing lendable Cohort capacity instead of recalculating it for each preemption candidate. (#15955, @venuchitta)

Changes since v0.19.5:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.

• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.19.1, v0.19.2, v0.19.3, v0.19.4, v0.19.5.

• DRA: Fixed a bug where a chargeable Pod overhead or a resource-transformation output that shared a name with a DRA-backed extended resource was taken back along with the DRA charge and dropped from quota accounting. Only the containers' own request is subtracted now, so any such contribution left under the original extendedResourceName after excludeResourcePrefixes and transformations is preserved. Also fixed a restartable init (sidecar) container being maxed against the total instead of added to it, so its devices are counted alongside the regular containers'.

  When a deviceClassMappings entry maps a DeviceClass to a different logical name, ClusterQueues must cover that logical name. A container-only request is fully translated and needs quota only on the logical name; cover the original extendedResourceName as well only if a Workload leaves chargeable Pod overhead or a resource-transformation output under it (normally in the same resource group). With such a residual uncovered, BlockUndeclared makes the Workload wait with the original name reported unavailable, while IgnoreUndeclared skips the residual and leaves it out of recorded usage. Where no mapping renames the DeviceClass, or the mapped name is the extendedResourceName itself, the device charge and the residual land on that one key, so its corrected usage can rise: a container asking for 1 with 1 of chargeable overhead now records 2 where it recorded 1. No second key is needed there, but the nominal quota has to cover the pair rather than a count of devices. The change applies as Workloads are evaluated, not to existing reservations at once. (#15987, @thc1006)

• SparkApplication: Fixed a bug where Workloads reserved less CPU and memory than Spark requests for the driver and executor Pods, because cores, memoryOverhead and Spark's default memory overhead were ignored. Kueue now reserves the same CPU and memory that Spark requests. Memory values must use the Java format accepted by Spark (for example 512m or 2g); Kubernetes-style values such as 512Mi are rejected.

If SparkApplications rely on cores or on the default memory overhead, raise the ClusterQueue quotas accordingly: each Pod now reserves its cores as CPU and at least 384Mi of additional memory. (#15923, @henry3260)

Changes by Kind

Bug or Regression

• AppWrapper: Added editor and viewer ClusterRoles for AppWrappers (#16101, @HsiuChuanHsu)
• CLI: Fix kueuectl version to honor --namespace when looking up the controller manager image. (#15977, @DevaanshPathak)
• ConcurrentAdmission: Fix a panic that left a parent Workload without variants when its name contained no "-". (#15850, @rjgoyln)
• DRA: Fixed a bug where a Workload that had been marked inadmissible kept reporting Requeued=False with reason Inadmissible after its DRA resources were resolved, because the condition was only cleared in memory and never written back. Kueue now persists Requeued=True with reason DRAResourcesResolved. (#15959, @PannagaRao)
• Deployment: The Workload of a Deployment-managed Pod carries the Deployment UID in the kueue.x-k8s.io/job-uid label instead of the Pod UID. Gated by DeploymentJobUIDLabel, Alpha and disabled by default. (#16004, @rjgoyln)
• ElasticJobsViaWorkloadSlices: Fixed a bug that could assign an incompatible ResourceFlavor when an elastic Workload was re-admitted with zero pods. Kueue now selects a flavor that supports the Workload’s per-pod resource requirements. (#16078, @prash2512)
• FailureRecovery: Fixed a bug where, with the FailureRecoveryPolicy feature gate enabled, Kueue added a duplicate KueueFailureRecovery Pod condition and emitted a duplicate KueueForcefullyDeleted warning event
  every time it reconciled a Pod stuck in termination. Kueue now sets the condition once and emits the event only when the condition changes. (#16113, @ErikJiang)
• Fixed a bug where an evicted Workload for a StatefulSet or LeaderWorkerSet that had no Pods (for example, scaled to zero) kept its quota reserved, which blocked other Workloads in the ClusterQueue. Kueue now releases the reservation. (#16128, @gola)
• Job: Fixed a bug where a Job with parallelism greater than completions could release the quota of its still-running Pods once some Pods succeeded, allowing the ClusterQueue to admit Workloads beyond its quota. (#16132, @henry3260)
• Job: Reject partial-admission minimum parallelism values outside the int32 range. (#16002, @cryo-zd)
• MultiKueue: Fixed a bug that left Active.ObservedGeneration stale after a healthy configuration update. Kueue now records the latest evaluated generation. (#15989, @weizhoublue)
• Pod: Added the PodIntegrationCountSucceededPodsAsReady feature gate (Alpha, disabled by default). When enabled, succeeded pods count as ready for PodsReady, which fixes a bug where a pod group could be evicted after waitForPodsReady.recoveryTimeout once one of its pods completed. (#15910, @henry3260)
• ProvisioningRequest: Fixed a bug where a pre-existing PodTemplate at the deterministic name could be adopted verbatim, letting a user who can write PodTemplate objects decouple the capacity requested from Cluster Autoscaler from the quota-checked Workload PodSets. Kueue now replaces divergent specs with Kueue-derived contents. Gated by EnforceProvisioningPodTemplateContents (Alpha, disabled by default). (#16070, @vladikkuzn)
• RBAC: Fixed a bug where a subject bound to kueue-batch-admin-role had no access to AdmissionCheck, MultiKueueCluster, MultiKueueConfig, ProvisioningRequestConfig and WorkloadPriorityClass (#16032, @HsiuChuanHsu)
• Ray integrations: Reject Kueue-managed jobs whose head Pod template has no containers during admission. (#16084, @kevin85421)
• RayJob: Fixed a bug where a RayJob using the default submitter Job (submissionMode: K8sJobMode without submitterPodTemplate) and a head Pod template with no containers caused the Kueue webhook and reconciler to panic. Kueue now returns an error instead. (#16065, @henry3260)
• ResourceTransformations: Fixed a bug where excluding a transformation input could let a Workload bypass ClusterQueue quota accounting. Kueue now accounts for the transformed output. (#16053, @weizhoublue)
• SparkApplication: Fixed a bug where a SparkApplication that only set the SparkApplication-level spec.nodeSelector permanently lost it after the first eviction, letting the driver and executor pods be scheduled onto any node once re-admitted. The selector is now recorded in the Workload PodSets, so it also participates in ResourceFlavor matching. (#15797, @henry3260)
• StatefulSet: Fixed a bug that allowed invalid TAS annotations in StatefulSet updates when TopologyAwareScheduling is enabled. Kueue now validates TAS metadata during updates. (#16104, @thuongvu)
• TAS: Fix a bug where StatefulSet integration didn't validate pod template metadata for topology annotations. (#15881, @weizhoublue)
• TAS: Fixed a bug that could cause workload admission to fail with topology level not specified when the unconstrained topology annotation was set to "false". Kueue now rejects this value when the TASRejectFalseUnconstrainedTopology feature gate is enabled. (#15847, @cryo-zd)
• TAS: Fixed a bug where, with the SchedulerLibraryIntegration feature gate enabled, ResourceFlavor tolerations and admission-check PodSetUpdates (for example a ProvisioningRequest nodeSelector) were not applied to the simulated pod, so a workload could stay pending on a tainted node or be assigned to a node its nodeSelector excludes. (#15836, @nsega)
• TAS: Introduced the alpha feature gate TASPartialSlices (enabled by default) to support scheduling PodSets whose count is not an exact multiple of the slice size. (#16121, @pajakd)
• TrainJob: Fixed a bug where an evicted or unadmitted TrainJob emitted a Stopped event and sent a redundant PATCH request on every reconcile. (#15754, @henry3260)
• WorkloadPriorityClassDefaulting: Fixed the defaulting webhook setting the kueue.x-k8s.io/priority-class: default label on jobs created in namespaces excluded by managedJobsNamespaceSelector. (#15998, @rjgoyln)
• Workloads: Fixed a bug that caused an incorrect pointer value in admission errors when resourceUsage was not divisible by the assigned pod count. Kueue now reports the actual pod count. (#15738, @Kunal241207)
• Workloads: Fixed a bug where a Workload with pod-level requests smaller than its aggregate container requests could reserve too little quota. Kueue now accounts for at least the aggregate container requests and adds Pod overhead. (#16047, @Dasmat13)

Other (Cleanup or Flake)

• Helm: Set the default Cohort controller concurrency to 1 for Helm installations. (#15876, @HsiuChuanHsu)