Skip to content

fix: evict stale instance cache entries in List() - #9242

Merged
ryan-mist merged 2 commits into
aws:mainfrom
ryan-mist:fix-ghost-node
Aug 4, 2026
Merged

ryan-mist merged 2 commits into
aws:mainfrom
ryan-mist:fix-ghost-node

Conversation

@ryan-mist

@ryan-mist ryan-mist commented Jun 9, 2026 •

Copy link
Copy Markdown
Member

Fixes #9226

Description
The instance cache TTL was recently changed from 1 minute to NoExpiration to support zonal shift. However, List() only adds entries to the cache and never removes entries for instances that EC2 no longer returns. We rely on these instances being removed from the cache as part of node termination - https://github.com/kubernetes-sigs/karpenter/blob/7593c9c8e04d961788940047f70cd00411e83823/pkg/controllers/node/termination/controller.go#L117:L128

This can cause spot-interrupted, IODCR-interrupted, or manually terminated instances to persist as ghost Nodes / NodeClaims in some cases (I've only been able to do this by turning off graceful node shutdown).

This PR changes List() to remove entries for instances that EC2 no longer returns if we have not shifted out of the zone

How was this change tested?

  • unit tests

Before PR

Interrupted instance through reclaiming capacity in IODCRs:

{"level":"INFO","time":"2026-06-09T17:12:43.743Z","logger":"controller","caller":"interruption/utils.go:160","message":"initiating delete from interruption message","commit":"f22f2ea-dirty","controller":"interruption","namespace":"","name":"","reconcileID":"5d8f172f-9ba1-4605-9c56-d17a3138af9b","queue":"ryanmist-karpenter","messageKind":"capacity_reservation_interrupted","NodeClaim":{"name":"iodcr-pool-ct6j7"},"action":"CordonAndDrain","Node":{"name":"ip-xxx-xxx-xx-87.us-west-2.compute.internal"}}

Instance is terminated, but the NodeClaim and Node are still around:

aws ec2 describe-instances --instance-ids i-08580c5bce70e890c | grep "State\": {" -A 3
                    "State": {
                        "Code": 48,
                        "Name": "terminated"
                    },
k get nodeclaim
NAME               TYPE        CAPACITY   ZONE         NODE                                          READY   AGE
iodcr-pool-ct6j7   m5.xlarge   reserved   us-west-2c   ip-xxx-xxx-xx-87.us-west-2.compute.internal   True    15m

k get node
NAME                                           STATUS     ROLES    AGE    VERSION
ip-xxx-xx-xx-87.us-west-2.compute.internal    NotReady   <none>   14m    v1.36.1-eks-3385e9b

After PR

Interrupted instance through reclaiming capacity in IODCRs:

{"level":"INFO","time":"2026-06-09T18:34:15.836Z","logger":"controller","caller":"interruption/utils.go:160","message":"initiating delete from interruption message","commit":"af2924f-dirty","controller":"interruption","namespace":"","name":"","reconcileID":"8a2d659d-314d-40b5-89b8-bedc5759a1fb","queue":"ryanmist-karpenter","messageKind":"capacity_reservation_interrupted","NodeClaim":{"name":"iodcr-pool-fzhrg"},"action":"CordonAndDrain","Node":{"name":"ip-xxx-xxx-xx-52.us-west-2.compute.internal"}}

Instance is terminated, and the NodeClaim and Node are removed shortly after:

aws ec2 describe-instances --instance-ids i-06164f09161e05aa2 | grep "State\": {" -A 3
                    "State": {
                        "Code": 48,
                        "Name": "terminated"
                    },
k get nodeclaim iodcr-pool-fzhrg
Error from server (NotFound): nodeclaims.karpenter.sh "iodcr-pool-fzhrg" not found

k get node ip-xxx-xxx-xx-52.us-west-2.compute.internal
Error from server (NotFound): nodes "ip-xxx-xxx-xx-52.us-west-2.compute.internal" not found

Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: #
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@ryan-mist
ryan-mist requested a review from a team as a code owner June 9, 2026 18:19
@ryan-mist
ryan-mist requested a review from Youssef-Beltagy June 9, 2026 18:19

@ZS221 ZS221 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter snapshot

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-af2924fffe57e7321221ee8e84e565e5aa231907.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-af2924fffe57e7321221ee8e84e565e5aa231907" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

@Sarthug99 Sarthug99 self-assigned this Jul 20, 2026

@DerekFrank DerekFrank left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ryan-mist
ryan-mist merged commit 0b0cd60 into aws:main Aug 4, 2026
13 checks passed
@AndrewSirenko AndrewSirenko mentioned this pull request Aug 14, 2026
1 of 3 tasks
ryan-mist added a commit to ryan-mist/karpenter-provider-aws that referenced this pull request Aug 14, 2026
@ryan-mist ryan-mist mentioned this pull request Aug 14, 2026
1 of 3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spot interrupted instances persist as ghosts with all pods terminating

4 participants