Repository navigation
calico/node token is invalidated by Kubernetes when the pod is evicted, leading to CNI failures #4857
Description
Expected Behavior
Calico CNI plugin tears down Pod in a timely manner.
Current Behavior
Calico CNI plugin shows errors terminating Pods, and therefore eviction takes too long. Especially relevant in Kubernetes conformance testing.
Aug 18 18:19:04.521: INFO: At 2021-08-18 18:18:01 +0000 UTC - event for taint-eviction-a1: {kubelet ip-10-0-8-52} FailedKillPod: error killing pod: failed to "KillPodSandbox" for "0701ef9b-e
17d-43b5-a48f-89fa3ac00999" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"taint-eviction-a1_taint-multiple-pods-4011\" network: error
getting ClusterInformation: connection is unauthorized: Unauthorized"
The natural things to check are RBAC permissions, which match recommendations:
- apiGroups:
- crd.projectcalico.org
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
...
To be certain, we can use the actual kubeconfig Calico writes to the host's /etc/cni/net.d. It does indeed seem to have permission to get clusterinformations. The error above is unusual.
./kubectl --kubeconfig /etc/cni/net.d/calico-kubeconfig auth can-i get clusterinformations --all-namespaces
yes
Steps to Reproduce (for bugs)
sonobuoy run --e2e-focus="NoExecuteTaintManager Multiple Pods" --e2e-skip="" \
--plugin-env=e2e.E2E_EXTRA_ARGS="--non-blocking-taints=node-role.kubernetes.io/controller"
Context
This issue affects Kubernetes Conformance tests:
Summarizing 1 Failure:
[Fail] [sig-node] NoExecuteTaintManager Multiple Pods [Serial] [It] evicts pods with minTolerationSeconds [Disruptive] [Conformance]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113
The test in question creates two Pods that don't tolerate a taint, and expects them to be terminated within certain times. In Kubelet logs, the Calico CNI plugin is complaining with the logs above and termination takes too long.
Your Environment
- Calico version: v3.19.2 or v3.20.0
- Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes v1.22.0
- Operating System and version: Fedora CoreOS
- Link to your project (optional): https://github.com/poseidon/typhoon, Calico manifests https://github.com/poseidon/terraform-render-bootstrap/tree/master/resources/calico
Activity
caseydavenport commentedon Aug 31, 2021
Well that's certainly bizarre. Not sure what has changed in that area that might cause this issue, especially given the RBAC seems to have the correct permissions present! Will see what I can find..
caseydavenport commentedon Aug 31, 2021
@dghubble could you confirm that the
calico-kubeconfigfile is actually the one referenced in Calico CNI configuration json? Just to make sure the plugin is actually using the file.One thing that might be relevant here is that Kubernetes recently enabled service account projection by default: https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume
You may need manifest changes so that Calico can maintain the credentials on disk as they are rotated. I see that you have the correct volume mount here: https://github.com/poseidon/terraform-render-bootstrap/blob/master/resources/calico/daemonset.yaml#L174-L177
You may also need to set CALICO_MANAGE_CNI=true in the calico/node env to enable the right logic, though.
dghubble commentedon Sep 2, 2021
When
10-calico.conflistis written out,__KUBECONFIG_FILEPATH__is replaced with/etc/cni/net.d/calico-kubeconfig. Within thecalico-nodecontainer, the location of the mounted file is/host/cni/net.d/calico-kubeconfig. Maybe that's not whatcalico-nodewants, but it's been this way a while.That seems to match what Calico's release
calico.yamlshows here and Calico reports no troubles finding a kubeconfig either.Setting
calico-node'senvdidn't seem to alter the result.ilyesAj commentedon Oct 21, 2021
we're facing the same issue here . is there any suggested solutions ?
Our Environment:
Calico version: v3.19.3
Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes v1.21.5
Operating System and version: ubuntu 20.04
provisioning tool: kops v1.21.1
caseydavenport commentedon Nov 4, 2021
The error is certainly unusual. I haven't been able to reproduce this at all in my own rigs. Generally, and RBAC issue would show something more precise - e.g., "serviceaccount X is not allowed to Y resouce Z".
Issues with bad TLS credentials would also show up more clearly. I'm not really sure the root cause of this, and I think to figure it out we probably need to dig into the API server logs to see why it is rejecting the request as unauthorized.
caseydavenport commentedon Dec 7, 2021
@dghubble @ilyesAj it's been a long time on this one... did you ever make any headway on it? I haven't seen this anywhere else.
dghubble commentedon Dec 7, 2021
To pass Kubernetes v1.22 conformance testing, Typhoon used flannel instead of Calico. I'll re-run with Calico during the v1.23 cycle.
ilyesAj commentedon Dec 8, 2021
@caseydavenport we have rollbacked to k8s 1.20.11 , it's seems to be releated to the k8s version , we didn't have any problems since .
lwr20 commentedon Mar 18, 2022
I had a go at reproing this.
sonobuoy run --e2e-focus="evicts.pods.with.minTolerationSeconds" --e2e-skip=""That hit the same issue, I think:
Looking at calico-node logs from that node, I see that the calico-node pod has a start time that is after the end of the test. I wonder if the problem is that calico-node doesn't tolerate the taint that's being added and is killed?
lwr20 commentedon Mar 18, 2022
If I manually add the same taint that the test uses (
kubernetes.io/e2e-evict-taint-key=evictTaintVal:NoExecute), I do indeed see the calico-node pod on that node disappear.9 remaining items
caseydavenport commentedon Mar 21, 2022
Yep, this one at least is no longer the case (manifests are multi-arch now).
Yeah, this would only be hit if the CNI plugin on the host needs to make API calls and is doing so using the serviceaccount token of the daemonset pod that installed it.
One option here might be to use the TokenRequest API directly to provision a separate token not bound to the life of the calico/node pod.
In general I agree that we can't expect "Tolerate all" to be acceptable for every single cluster in existence, but I do think it is the correct default for what we ship because it will be right for the vast majority of clusters.
I believe for use cases like this we have switched to using node affinities rather than taints/tolerations. For example, this node affinity prevents us from running on fargate nodes: https://github.com/tigera/operator/blob/master/pkg/render/node.go#L711
dghubble commentedon Mar 21, 2022
Awesome to see the multi-arch manifest images. I'll check those out, that'll help remove one case.
I agree, for the vast majority of clusters your example seems great. I wouldn't advocate changing it either.
I may look at node affinities, but having those be conditional is a lot more tricky just with the Terraform logic available to us. And taints do seem to express the situation fairly clearly. I'm not sure affinities would have a different end result (e.g. Kubernetes E2E might have an equivalent test there that gets thrown off in the same way).
Thanks to your investigation @lwr20, this looks like a detail of how CNCF conformance tests run, you'd agree? That wouldn't affect real clusters as far as I can tell. We could just say, if we need to pass conformance, you need to tolerate
kubernetes.io/e2e-evict-taint-keyhttps://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/taints.go#L48 in our docs.lwr20 commentedon Mar 21, 2022
I agree. I certainly don't think that particular conformance test is intended to mandate that "CNIs must work without talking to the kubernetes apiserver" for example.
Of course if we could do this, that would be ideal. But will need some prototyping and testing of course.
dghubble commentedon Mar 23, 2022
Adding the DaemonSet toleration for
kubernetes.io/e2e-evict-taint-keygets this conformance test passing for me as well. I can update conformance testing nodes and go ahead and close this issue if that's alright.It would be nice to not hit this if that's a reasonable thing to do. Presumably that would be a separate issue if its desired.
ScheererJ commentedon Apr 7, 2022
@caseydavenport We have run into this issue in one of our clusters in a slightly different scenario. For bin packing reasons, we scale the resource requests of
calico-nodevertically in a cluster-proportional manner (pretty similar to https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/calico-policy-controller).As the cluster grew in size,
calico-nodewas supposed to be recreated with bigger memory requests, i.e. the existing pod was deleted and a new one with higher memory requests being created. However, as the node was nearly fully loaded there was not enough space for the new pod. Thanks to the priority class ofcalico-nodepre-emption occurred and thekube-schedulertried to get rid of a lower priority pod on the node.However, now we ran into the problem that the lower priority pod could not be deleted as network sandbox deletion via CNI fails with this error (
error getting ClusterInformation: connection is unauthorized: Unauthorized") as the token in calico's kubeconfig belongs to a deleted pod.The node cannot automatically recover from this as no pod can be completely removed due to the CNI error and
calico-nodecannot be scheduled as the memory requirements are not fulfilled.Is there a plan to resolve this issue by for example using the token api directly or otherwise decoupling the validity of the token used for CNI from the
calico-nodepod lifecycle?ScheererJ commentedon Apr 7, 2022
As we would like to have this issue fixed properly, we would like to contribute a solution to this.
It seems like the logic is scattered across two place:
Instead of using the token of the
calico-nodepod, we would propose create a separate token via https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/ either bound to no object or bound to the node object. The validity period can be rather small, e.g. 1h. The token would then be replaced with a simple timer based approach.@caseydavenport Would you be open to such a contribution?
musigma-admin commentedon Apr 7, 2022
Hi,
In our case on a slightly different setting. Even the new nodes with compartively less/no pods have this issue. Even when I had actually formatted and added the machine back We still figure get this CNI issue.
You can find more about this issue here
caseydavenport commentedon Apr 7, 2022
Yes, this is the approach that I was musing on as well. I think it is worth exploring this to see what it would look like and what limitations it might have (hopefully none)
[-]network: error getting ClusterInformation: connection is unauthorized[/-][+]calico/node token is invalidated by Kubernetes when the pod is evicted, leading to CNI failures[/+]ScheererJ commentedon Apr 8, 2022
@caseydavenport Should I create a corresponding pull request or do you plan to explore it yourself?
caseydavenport commentedon Apr 13, 2022
@ScheererJ I'd be happy to review a PR for this if you have one. Otherwise I will take a look at it myself once v3.23 is out the door (so in a couple of weeks).
ScheererJ commentedon Apr 14, 2022
@caseydavenport Feel free to take a look at #5910 if you have some time to spare. I will be on vacation next week, though. Hence, there is no hurry from my side.