Repository navigation
Executor pods are not cascade-deleted when driver pod is deleted #2961
Description
What happened?
When a SparkApplication is suspended (spec.suspend=true) or otherwise has its driver pod deleted, the executor pods are not cleaned up by Kubernetes garbage collection — even though the driver pod carries controller: true in the executor's ownerReferences. This is because Spark Operator additionally attaches the SparkApplication CR as a second owner of each executor pod, which prevents K8s GC from cascading the deletion when only the driver disappears.
Reproduction Code
Apply a SparkApplication and let it reach the running state:
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-test
spec:
type: Python
mode: cluster
image: docker.io/apache/spark:3.5.3-python3
mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
sparkVersion: "3.5.3"
driver:
cores: 1
coreRequest: "1"
memory: 512m
executor:
cores: 1
coreRequest: "1"
memory: 512m
instances: 2
restartPolicy:
type: Never
Inspect an executor pod's owner references:
$ kubectl get pod spark-test-<hash>-exec-1 \
-o jsonpath='{range .metadata.ownerReferences[*]}{.kind}/{.name} controller={.controller}{"\n"}{end}'
SparkApplication/spark-test controller=
Pod/spark-test-driver controller=trueThe executor has two owners: the SparkApplication CR (controller=false) and the driver pod (controller=true).
Now delete the driver pod and observe that executors are NOT garbage collected:
$ kubectl delete pod spark-test-driver
pod "spark-test-driver" deleted
$ sleep 40 && kubectl get pod
NAME READY STATUS RESTARTS AGE
spark-test-<hash>-exec-1 0/1 Completed 0 7m
spark-test
### Expected behavior
When the driver pod is deleted, the executor pod should be deleted in cascade.
### Actual behavior
executor pod still exist
### Environment & Versions
- Kubernetes: v1.33.x
- Spark Operator: v2.5.0
- Apache Spark: 3.5.3
- SparkApplication suspended via Kueue (or by manually setting spec.suspend=true / deleting driver)
### Additional context
_No response_
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Activity
github-actions commentedon May 28, 2026
🎉 Welcome to the Kubeflow Spark Operator! 🎉
Thanks for opening your first issue! We're happy to have you as part of our community 🚀
Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏
[-]Executor pods are not garbage collected when driver pod is deleted, due to dual ownerReferences[/-][+] Executor pods are not cascade-deleted when driver pod is deleted[/+]hahahaheihei commentedon May 28, 2026
kubernetes-sigs/kueue#11676
tariq-hasan commentedon Jun 1, 2026
Thanks for raising the issue @hahahaheihei.
The SparkApplication owner reference on the executor pods was added as part of this PR: #2670. Given the current issue I am wondering if it would be ideal to just drop the executor's SparkApplication reference and rely on GC controller cascade-deletion instead of adding operator-managed deletion on the executor pods. @everpeace
github-actions commentedon Aug 30, 2026
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Stop adding SparkApplication owner reference to executor pods
tariq-hasan commentedon Sep 9, 2026
/remove-lifecycle stale
fix: stop adding SparkApplication owner reference to executor pods (#…
Upstream sync 2026-09-14 (#181)
Harsh-2005d commentedon Sep 25, 2026
Hi @tariq-hasan , I went through the earlier PR and the discussion around the E2E testing for #3152 . I noticed there was a suggestion to add a dedicated test verifying that executor pods are cascade-deleted when the driver pod is deleted.
Is anyone currently working on that? If not, I’d be interested in working on the E2E test.
tariq-hasan commentedon Sep 28, 2026
This issue has been fixed by #3152.
Thanks @hahahaheihei for raising the issue and @KR-Ravindra for the fix.
/close
google-oss-prow commentedon Sep 28, 2026
@tariq-hasan: Closing this issue.
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.