✨ อัปเดตล่าสุด: kubernetes sigs/kueue/pull/

hahahaheihei · 2026-05-27T08:25:45Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

this pr fixed sparkapplication can't requeue when driver pod or executor unready timeout

Which issue(s) this PR fixes:

Fixes #11675

Special notes for your reviewer:

Does this PR introduce a user-facing change?

SparkApplication: Fixed a bug where `PodsReady` returned true based on driver state alone, so workloads
with stuck executors never reached `waitForPodsReady.timeout`.

netlify · 2026-05-27T08:25:54Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`cad24f4`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/6a1812dc04b5af0008a61f32

k8s-ci-robot · 2026-05-27T08:25:54Z

Welcome @hahahaheihei!

It looks like this is your first PR to kubernetes-sigs/kueue 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kueue has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-05-27T08:25:55Z

Hi @hahahaheihei. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mimowo · 2026-05-27T08:35:01Z

/ok-to-test

mimowo · 2026-05-27T08:35:35Z

Please add the release-note block

hahahaheihei · 2026-05-27T09:25:17Z

/retest

hahahaheihei · 2026-05-27T09:42:59Z

Please add the release-note block
i have completed the release-note，please review the pr

mimowo · 2026-05-27T13:16:46Z

+			sparkcommon.LabelSparkRole:    sparkcommon.SparkRoleDriver,
+		},
+	); err == nil && len(driverList.Items) > 0 {
+		driverKeys := make([]client.ObjectKey, 0, len(driverList.Items))


do not silence errors, propagate up

You're right; thank you for your suggestion.

kannon92 · 2026-05-27T15:52:06Z

/cc @everpeace

mimowo · 2026-05-27T19:39:36Z

+		_ = wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, 5*time.Second, true, func(ctx context.Context) (bool, error) {
+			for _, key := range driverKeys {
+				var p corev1.Pod
+				err := c.Get(ctx, key, &p)
+				if err == nil {
+					return false, nil
+				}
+				if !apierrors.IsNotFound(err) {
+					return false, err
+				}
+			}
+			return true, nil
+		})


The periodic poll is not a good pattern, and is potentially racy.

I'm wondering - why not just return error if there are any pods which are not terminating? Then the job reconciler should re-enter Stop, and kill some more pods

in my test ，we shoul wait for driver pod be cleaned,then kill pending executor pod ,otherwise, the driver pod may occasionally be re-launched before being cleaned, and when a new driver pod starts up, it will launch a new executor pod as well. if return err

if ptr.Deref(j.Spec.Suspend, false) { return false, nil }

this code ,will never kill executor pod

Perhaps a more elegant implementation would be to delegate both the executor cleanup and driver cleanup to the Spark-operator, while the kueue only needs to manage the suspend operation at critical moments. @mimowo

Oh yes, this is how the integration k8s job, JobSet or other CRDs work - we just flip the suspend, and wait for termination in isActive

I have submitted the corresponding issue to the Spark community. kubeflow/spark-operator#2961

yeah but still waiting 5s seems arbitrary and racy im wondering if we can fix it better even if the code is in kueue

I have removed the stop-related logic. I believe that with proper configuration of ownerReferences, driver pod and executor pod should be cascaded for deletion.

sgtm it would be great if you can confirm this is working well on a live cluster, plus let's adjust the release note.

mimowo · 2026-05-28T08:20:31Z

 func (j *SparkApplication) PodsReady(ctx context.Context, _ client.Client) bool {
-	return j.Status.AppState.State == sparkv1beta2.ApplicationStateRunning
+	// Driver must be running.
+	if j.Status.AppState.State != sparkv1beta2.ApplicationStateRunning {


Could you add a unit test for the Reconcile, IIUC it will call PodsReady

sure ，i have added some unit test

Signed-off-by: hahahaheihei <1475660997@qq.com>

mimowo

/lgtm
/approve
Thank you for bugfixing the integration 👍
/cherrypick release-0.17

k8s-ci-robot · 2026-05-28T11:45:27Z

LGTM label has been added.

Details

Git tree hash: 26c347144f78cb46df301ec9f27daa32b92c9631

k8s-ci-robot · 2026-05-28T11:45:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hahahaheihei, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mimowo · 2026-05-28T20:28:11Z

/cherrypick release-0.17

k8s-infra-cherrypick-robot · 2026-05-28T20:28:48Z

@mimowo: #11676 failed to apply on top of branch "release-0.17":

Applying: fix: PodsReady and Stop for SparkApplication framework
Using index info to reconstruct a base tree...
M	pkg/controller/jobs/sparkapplication/sparkapplication_controller.go
M	pkg/controller/jobs/sparkapplication/sparkapplication_controller_test.go
M	test/integration/singlecluster/controller/jobs/sparkapplication/sparkapplication_test_util.go
Falling back to patching base and 3-way merge...
Auto-merging test/integration/singlecluster/controller/jobs/sparkapplication/sparkapplication_test_util.go
Auto-merging pkg/controller/jobs/sparkapplication/sparkapplication_controller_test.go
CONFLICT (content): Merge conflict in pkg/controller/jobs/sparkapplication/sparkapplication_controller_test.go
Auto-merging pkg/controller/jobs/sparkapplication/sparkapplication_controller.go
CONFLICT (content): Merge conflict in pkg/controller/jobs/sparkapplication/sparkapplication_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 fix: PodsReady and Stop for SparkApplication framework

Details

In response to this:

/cherrypick release-0.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mimowo · 2026-05-28T21:12:33Z

@hahahaheihei please prepare the cherrypick if you would like the fix to be included in patch release

hahahaheihei · 2026-05-29T01:44:18Z

@hahahaheihei please prepare the cherrypick if you would like the fix to be included in patch release

sure , once i resolve the conflict , i will submit a new pr

mimowo · 2026-05-29T05:51:51Z

Awesome. we are planning the patch releases today, around 11 am Polish time.

Otherwise it will go to the next patch release, in two weeks.

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 27, 2026

k8s-ci-robot requested review from mimowo and tenzen-y May 27, 2026 08:25

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 27, 2026

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 27, 2026

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 27, 2026

hahahaheihei force-pushed the fix/spark-podsready-stop branch from cfa4c1f to 59044ac Compare May 27, 2026 11:36

mimowo reviewed May 27, 2026

View reviewed changes

k8s-ci-robot requested a review from everpeace May 27, 2026 15:52

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 27, 2026

mimowo reviewed May 27, 2026

View reviewed changes

hahahaheihei mentioned this pull request May 28, 2026

Executor pods are not cascade-deleted when driver pod is deleted kubeflow/spark-operator#2961

Closed

hahahaheihei force-pushed the fix/spark-podsready-stop branch from 59044ac to 4ebed80 Compare May 28, 2026 05:54

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 28, 2026

hahahaheihei changed the title ~~fix: PodsReady and Stop for SparkApplication framework~~ fix: PodsReady for SparkApplication framework May 28, 2026

mimowo reviewed May 28, 2026

View reviewed changes

fix: PodsReady and Stop for SparkApplication framework

cad24f4

Signed-off-by: hahahaheihei <1475660997@qq.com>

hahahaheihei force-pushed the fix/spark-podsready-stop branch from 4ebed80 to cad24f4 Compare May 28, 2026 10:03

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 28, 2026

mimowo reviewed May 28, 2026

View reviewed changes

k8s-ci-robot assigned mimowo May 28, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 28, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 28, 2026

everpeace mentioned this pull request May 28, 2026

SparkApplication framework: PodsReady ignores executor state, and Stop leaves orphan executor pods across requeue cycles #11675

Closed

k8s-ci-robot merged commit 00d4c12 into kubernetes-sigs:main May 28, 2026
45 checks passed

k8s-ci-robot added this to the v0.18 milestone May 28, 2026

hahahaheihei mentioned this pull request May 29, 2026

[release-0.17] fix: PodsReady and Stop for SparkApplication framework #11777

Merged

Conversation

hahahaheihei commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

netlify Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Uh oh!

k8s-ci-robot commented May 27, 2026

Uh oh!

k8s-ci-robot commented May 27, 2026

Uh oh!

mimowo commented May 27, 2026

Uh oh!

mimowo commented May 27, 2026

Uh oh!

hahahaheihei commented May 27, 2026

Uh oh!

hahahaheihei commented May 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kannon92 commented May 27, 2026

Uh oh!

mimowo May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented May 28, 2026

Uh oh!

k8s-ci-robot commented May 28, 2026

Uh oh!

Uh oh!

mimowo commented May 28, 2026

Uh oh!

k8s-infra-cherrypick-robot commented May 28, 2026

Uh oh!

mimowo commented May 28, 2026

Uh oh!

hahahaheihei commented May 29, 2026

Uh oh!

mimowo commented May 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

hahahaheihei commented May 27, 2026 •

edited

Loading

netlify Bot commented May 27, 2026 •

edited

Loading

mimowo May 27, 2026 •

edited

Loading