Skip to content

Commit fc87a49

Browse files
alexeymoskalev-devopshisco
authored andcommitted
feat(nodes): drain plan — classifier, read-only endpoint and plan dialog (#1584)
Replaces the generic Drain Node confirmation with a server-backed plan. Backend (pkg/k8score, internal/server): - Pure classifier ClassifyPodForDrain returning a per-pod outcome (evict / skip / may-block) with an operator-facing reason. Rules follow kubectl drain's filters: terminal and mirror pods skipped, DaemonSet pods never evicted, no controller owner needs force, emptyDir needs deleteEmptyDirData. A pod counts as managed only with a controller owner reference (metav1.GetControllerOf); hasManagedOwner accepted any owner reference before. A pod that is already terminating is reported as evict without consulting PDBs, as the Eviction API admits it regardless. - POST /api/nodes/{name}/drain-plan: reads only, requesting user's client, same body as drain, returns evaluated options, generation time, summary, per-pod outcomes with reasons and an emptyDir flag; PDBs the caller cannot list are reported as not evaluated instead of counted as zero. - DrainNode runs on the same classifier and returns skippedPods with reasons (additive); the MCP manage_node tool is unchanged. Frontend (packages/k8s-ui, web): - DrainPlanDialog fetches the plan before enabling Drain, labels it an estimate re-evaluated on execution, lists every pod with its outcome and reason, starts with deleteEmptyDirData off and requires an explicit acknowledgement naming the affected pods, keeps force explicit. Both options are always sent explicitly; nothing relies on server defaults. - Post-drain toast lists evictions, skipped pods with reasons and individual failures. Tests: classifier table, PlanNodeDrain and the handler on a fake clientset asserting only get/list actions, status mapping, DrainNode skip reporting; frontend: confirm gating (plan required, emptyDir acknowledgement), dialog content rendering, plan request shape, drain result summary. Docs: endpoint list and RBAC note.
1 parent 6ae7fcf commit fc87a49

16 files changed

Lines changed: 1333 additions & 111 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,7 @@ After `make <name>-demo`, run `kubectl config use-context kind-radar-<name>-demo
166166
- GitOps controller actions: `/api/argo/applications/...` (sync, refresh, terminate, suspend, resume, rollback, selective-sync), `/api/flux/{kind}/...` (reconcile, suspend, resume, sync-with-source)
167167
- Argo CD API integration: `PUT /api/integrations/argocd` (URL/token, probe-before-persist, token preserved across GET-redaction round-trips); `/api/argo/applications/{ns}/{name}/resource-diff` (Git-rendered desired vs live via argocd-server managed-resources; dual RBAC gate + structural Secret redaction; see docs/gitops.md)
168168
- GitOps detail data: `/api/gitops/tree/{kind}/{ns}/{name}` (resource tree + ownership edges), `/api/gitops/insights/{kind}/{ns}/{name}` (curated diagnosis: summary + issues + drift + events + plan + history + capabilities)
169-
- Nodes: `/api/nodes/{name}/...` (cordon, uncordon, drain, debug)
169+
- Nodes: `/api/nodes/{name}/...` (cordon, uncordon, drain, drain-plan (read-only estimate: per-pod evict/skip/may-block with reasons), debug)
170170
- Audit: `/api/audit`, `/api/audit/resource/{kind}/{ns}/{name}`, `/api/settings/audit` (GET/PUT)
171171
- Network trace: `/api/trace/{kind}/{ns}/{name}` (path-shaped diagnosis for Service/Ingress/HTTPRoute/GRPCRoute/Gateway; `?probe=true` runs DNS/TCP/TLS/HTTP probes against the declared path - direct TCP in-cluster, K8s API server proxy from a laptop, gated by the user's `services/proxy` + `pods/proxy` RBAC).
172172
- Capacity (Karpenter): `/api/capacity` (overview), `/api/capacity/pools` (+ `/{name}`, `/{name}/members`), `/api/capacity/demand` (`?state=`, `?pool=`, `?owner=ns/Kind/name`, `?pod=ns/name`), `/api/capacity/activity` — all read-only, all gated on the caller's ability to list NodePools; deliberately cluster-wide (no namespace view-filter forwarding). See [docs/capacity.md](docs/capacity.md)

‎docs/in-cluster.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -331,7 +331,7 @@ Some features require additional permissions. Most are disabled by default for s
331331
| Node runtime evidence | `rbac.viewNodeRuntime: true` | `false` | Let upgrade-impact checks inspect kubelet metrics and effective configuration through `nodes/proxy`. Enable only for a trusted no-auth audience; authenticated users need this permission on their own Kubernetes identity |
332332
| Traffic TLS | `rbac.traffic: true` | `true` | Read Hubble relay TLS certs for Cilium traffic observation |
333333
334-
> **Node management** (cordon, uncordon, drain) is available via the MCP server and API. These operations require `patch` on nodes, `list` on pods, and `create` on `pods/eviction`, which are not included in the default ClusterRole. Add them via `rbac.additionalRules` or use [per-user authentication](authentication.md) so each user's own RBAC governs node operations.
334+
> **Node management** (cordon, uncordon, drain) is available via the MCP server and API. These operations require `patch` on nodes, `list` on pods, and `create` on `pods/eviction`, which are not included in the default ClusterRole. The read-only drain plan (`POST /api/nodes/{name}/drain-plan`) needs `get` on nodes, `list` on pods and `list` on `poddisruptionbudgets`; without the last one the plan is still returned, marked as evaluated without PDB knowledge. Add them via `rbac.additionalRules` or use [per-user authentication](authentication.md) so each user's own RBAC governs node operations.
335335
336336
Enable features as needed:
337337

‎internal/k8s/node_ops.go‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@ type DrainOptions = k8score.DrainOptions
1515
// DrainResult is an alias for the reusable DrainResult type.
1616
type DrainResult = k8score.DrainResult
1717

18+
// DrainPlan is an alias for the reusable DrainPlan type.
19+
type DrainPlan = k8score.DrainPlan
20+
1821
// CordonNode marks a node as unschedulable.
1922
func CordonNode(ctx context.Context, nodeName string) error {
2023
client := GetClient()
@@ -66,3 +69,12 @@ func DrainNodeWithClient(ctx context.Context, nodeName string, opts DrainOptions
6669
}
6770
return k8score.DrainNode(ctx, client, nodeName, opts)
6871
}
72+
73+
// PlanNodeDrainWithClient computes a read-only drain plan with a caller-supplied client
74+
// (so the plan is evaluated under the caller's RBAC identity). It never mutates the cluster.
75+
func PlanNodeDrainWithClient(ctx context.Context, nodeName string, opts DrainOptions, client kubernetes.Interface) (*DrainPlan, error) {
76+
if client == nil {
77+
return nil, fmt.Errorf("not connected to cluster")
78+
}
79+
return k8score.PlanNodeDrain(ctx, client, nodeName, opts)
80+
}

‎internal/server/node_handlers.go‎

Lines changed: 89 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,14 +9,17 @@ import (
99

1010
"github.com/go-chi/chi/v5"
1111
apierrors "k8s.io/apimachinery/pkg/api/errors"
12+
"k8s.io/client-go/kubernetes"
1213

1314
"github.com/skyhook-io/radar/internal/auth"
1415
"github.com/skyhook-io/radar/internal/k8s"
1516
)
1617

17-
// DrainRequest is the optional JSON body for the drain endpoint.
18+
// DrainRequest is the optional JSON body shared by the drain and drain-plan endpoints.
19+
// An omitted deleteEmptyDirData means true for the drain and false for the plan; see
20+
// drainOptionsFromRequest.
1821
type DrainRequest struct {
19-
DeleteEmptyDirData *bool `json:"deleteEmptyDirData,omitempty"` // Default true if omitted
22+
DeleteEmptyDirData *bool `json:"deleteEmptyDirData,omitempty"`
2023
Force bool `json:"force"`
2124
GracePeriodSeconds *int64 `json:"gracePeriodSeconds,omitempty"`
2225
Timeout int `json:"timeout,omitempty"` // seconds, default 60
@@ -92,6 +95,89 @@ func (s *Server) handleUncordonNode(w http.ResponseWriter, r *http.Request) {
9295
s.writeJSON(w, map[string]string{"status": "ok", "message": "Node uncordoned"})
9396
}
9497

98+
// drainOptionsFromRequest turns the JSON body shared by the drain and drain-plan
99+
// endpoints into DrainOptions. The two endpoints differ only in what an omitted
100+
// deleteEmptyDirData means: the drain keeps its historical default (true, matching
101+
// kubectl drain --delete-emptydir-data), the plan defaults to false so a preview
102+
// never silently includes emptyDir data. The plan echoes the options it used.
103+
func drainOptionsFromRequest(req DrainRequest, deleteEmptyDirDefault bool) k8s.DrainOptions {
104+
deleteLocal := deleteEmptyDirDefault
105+
if req.DeleteEmptyDirData != nil {
106+
deleteLocal = *req.DeleteEmptyDirData
107+
}
108+
opts := k8s.DrainOptions{
109+
IgnoreDaemonSets: true,
110+
DeleteEmptyDirData: deleteLocal,
111+
Force: req.Force,
112+
GracePeriodSeconds: req.GracePeriodSeconds,
113+
Timeout: 60 * time.Second,
114+
}
115+
if req.Timeout > 0 {
116+
opts.Timeout = time.Duration(req.Timeout) * time.Second
117+
}
118+
return opts
119+
}
120+
121+
// handleDrainPlan returns a read-only estimate of what draining the node would do:
122+
// per-pod evict / skip / may-block outcomes with reasons, evaluated with the requesting
123+
// user's client. Nothing is cordoned or evicted. The result is a snapshot; the drain
124+
// endpoint re-lists and re-evaluates live state when it runs.
125+
func (s *Server) handleDrainPlan(w http.ResponseWriter, r *http.Request) {
126+
if !s.requireConnected(w) {
127+
return
128+
}
129+
130+
nodeName := chi.URLParam(r, "name")
131+
if nodeName == "" {
132+
s.writeError(w, http.StatusBadRequest, "node name is required")
133+
return
134+
}
135+
136+
opts, err := drainPlanOptionsFromHTTP(r)
137+
if err != nil {
138+
s.writeError(w, http.StatusBadRequest, "invalid request body")
139+
return
140+
}
141+
142+
client := s.getClientForRequest(r)
143+
if client == nil {
144+
s.writeError(w, http.StatusServiceUnavailable, "cluster client not available — check cluster connection")
145+
return
146+
}
147+
s.writeDrainPlan(w, r, client, nodeName, opts)
148+
}
149+
150+
// drainPlanOptionsFromHTTP decodes the optional JSON body of a drain-plan request
151+
// and applies the plan's defaults (deleteEmptyDirData off unless asked).
152+
func drainPlanOptionsFromHTTP(r *http.Request) (k8s.DrainOptions, error) {
153+
var req DrainRequest
154+
if err := json.NewDecoder(r.Body).Decode(&req); err != nil && err != io.EOF {
155+
return k8s.DrainOptions{}, err
156+
}
157+
return drainOptionsFromRequest(req, false), nil
158+
}
159+
160+
// writeDrainPlan computes the plan with the given client and writes it, mapping
161+
// Kubernetes errors to HTTP status codes the same way the drain endpoint does.
162+
func (s *Server) writeDrainPlan(w http.ResponseWriter, r *http.Request, client kubernetes.Interface, nodeName string, opts k8s.DrainOptions) {
163+
plan, err := k8s.PlanNodeDrainWithClient(r.Context(), nodeName, opts, client)
164+
if err != nil {
165+
if apierrors.IsNotFound(err) {
166+
s.writeError(w, http.StatusNotFound, err.Error())
167+
return
168+
}
169+
if apierrors.IsForbidden(err) {
170+
s.writeError(w, http.StatusForbidden, err.Error())
171+
return
172+
}
173+
log.Printf("[node-ops] Failed to plan drain for node %s: %v", nodeName, err)
174+
s.writeError(w, http.StatusInternalServerError, err.Error())
175+
return
176+
}
177+
178+
s.writeJSON(w, plan)
179+
}
180+
95181
func (s *Server) handleDrainNode(w http.ResponseWriter, r *http.Request) {
96182
if !s.requireConnected(w) {
97183
return
@@ -111,20 +197,7 @@ func (s *Server) handleDrainNode(w http.ResponseWriter, r *http.Request) {
111197

112198
// DeleteEmptyDirData defaults true (matching kubectl drain --delete-emptydir-data).
113199
// Most pods use emptyDir for tmp/caches; without this, drain skips almost everything.
114-
deleteLocal := true
115-
if req.DeleteEmptyDirData != nil {
116-
deleteLocal = *req.DeleteEmptyDirData
117-
}
118-
opts := k8s.DrainOptions{
119-
IgnoreDaemonSets: true,
120-
DeleteEmptyDirData: deleteLocal,
121-
Force: req.Force,
122-
GracePeriodSeconds: req.GracePeriodSeconds,
123-
Timeout: 60 * time.Second,
124-
}
125-
if req.Timeout > 0 {
126-
opts.Timeout = time.Duration(req.Timeout) * time.Second
127-
}
200+
opts := drainOptionsFromRequest(req, true)
128201

129202
auth.AuditLog(r, "", nodeName)
130203
client := s.getClientForRequest(r)
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
package server
2+
3+
import (
4+
"encoding/json"
5+
"net/http"
6+
"net/http/httptest"
7+
"strings"
8+
"testing"
9+
"time"
10+
11+
corev1 "k8s.io/api/core/v1"
12+
apierrors "k8s.io/apimachinery/pkg/api/errors"
13+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
14+
"k8s.io/apimachinery/pkg/runtime"
15+
"k8s.io/apimachinery/pkg/runtime/schema"
16+
"k8s.io/client-go/kubernetes/fake"
17+
k8stesting "k8s.io/client-go/testing"
18+
19+
"github.com/skyhook-io/radar/internal/k8s"
20+
)
21+
22+
func TestDrainOptionsFromRequestDefaults(t *testing.T) {
23+
no := false
24+
yes := true
25+
grace := int64(5)
26+
27+
tests := []struct {
28+
name string
29+
req DrainRequest
30+
emptyDirDefault bool
31+
wantEmptyDir bool
32+
wantForce bool
33+
wantTimeout time.Duration
34+
}{
35+
{name: "drain keeps its historical emptyDir default", req: DrainRequest{}, emptyDirDefault: true, wantEmptyDir: true, wantTimeout: 60 * time.Second},
36+
{name: "plan never includes emptyDir data unless asked", req: DrainRequest{}, emptyDirDefault: false, wantEmptyDir: false, wantTimeout: 60 * time.Second},
37+
{name: "explicit false overrides the drain default", req: DrainRequest{DeleteEmptyDirData: &no}, emptyDirDefault: true, wantEmptyDir: false, wantTimeout: 60 * time.Second},
38+
{name: "explicit true overrides the plan default", req: DrainRequest{DeleteEmptyDirData: &yes, Force: true, Timeout: 120}, emptyDirDefault: false, wantEmptyDir: true, wantForce: true, wantTimeout: 120 * time.Second},
39+
}
40+
for _, tt := range tests {
41+
t.Run(tt.name, func(t *testing.T) {
42+
got := drainOptionsFromRequest(tt.req, tt.emptyDirDefault)
43+
if !got.IgnoreDaemonSets {
44+
t.Fatalf("DaemonSets must always be ignored")
45+
}
46+
if got.DeleteEmptyDirData != tt.wantEmptyDir || got.Force != tt.wantForce || got.Timeout != tt.wantTimeout {
47+
t.Fatalf("got %+v", got)
48+
}
49+
})
50+
}
51+
52+
got := drainOptionsFromRequest(DrainRequest{GracePeriodSeconds: &grace}, false)
53+
if got.GracePeriodSeconds == nil || *got.GracePeriodSeconds != 5 {
54+
t.Fatalf("grace period must pass through, got %+v", got.GracePeriodSeconds)
55+
}
56+
}
57+
58+
func drainPlanTestClient() *fake.Clientset {
59+
yes := true
60+
return fake.NewSimpleClientset(
61+
&corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "worker-1"}},
62+
&corev1.Pod{
63+
ObjectMeta: metav1.ObjectMeta{Name: "web-1", Namespace: "shop", OwnerReferences: []metav1.OwnerReference{{Kind: "ReplicaSet", Name: "web", Controller: &yes}}},
64+
Spec: corev1.PodSpec{NodeName: "worker-1"},
65+
Status: corev1.PodStatus{Phase: corev1.PodRunning},
66+
},
67+
)
68+
}
69+
70+
func TestWriteDrainPlanRespondsWithPlanAndNeverMutates(t *testing.T) {
71+
client := drainPlanTestClient()
72+
s := &Server{}
73+
rr := httptest.NewRecorder()
74+
req := httptest.NewRequest(http.MethodPost, "/api/nodes/worker-1/drain-plan", nil)
75+
76+
s.writeDrainPlan(rr, req, client, "worker-1", k8s.DrainOptions{IgnoreDaemonSets: true})
77+
78+
if rr.Code != http.StatusOK {
79+
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
80+
}
81+
var plan k8s.DrainPlan
82+
if err := json.Unmarshal(rr.Body.Bytes(), &plan); err != nil {
83+
t.Fatalf("decode plan: %v", err)
84+
}
85+
if !plan.Estimate || plan.Node != "worker-1" || plan.Summary.Evict != 1 || len(plan.Pods) != 1 {
86+
t.Fatalf("unexpected plan: %+v", plan)
87+
}
88+
for _, a := range client.Actions() {
89+
if a.GetVerb() != "get" && a.GetVerb() != "list" {
90+
t.Fatalf("plan endpoint must not mutate the cluster, saw %s %s", a.GetVerb(), a.GetResource().Resource)
91+
}
92+
}
93+
}
94+
95+
func TestWriteDrainPlanStatusMapping(t *testing.T) {
96+
t.Run("unknown node is 404", func(t *testing.T) {
97+
s := &Server{}
98+
rr := httptest.NewRecorder()
99+
s.writeDrainPlan(rr, httptest.NewRequest(http.MethodPost, "/", nil), drainPlanTestClient(), "ghost", k8s.DrainOptions{})
100+
if rr.Code != http.StatusNotFound {
101+
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
102+
}
103+
})
104+
t.Run("forbidden pod list is 403", func(t *testing.T) {
105+
client := drainPlanTestClient()
106+
client.PrependReactor("list", "pods", func(k8stesting.Action) (bool, runtime.Object, error) {
107+
return true, nil, apierrors.NewForbidden(schema.GroupResource{Resource: "pods"}, "", nil)
108+
})
109+
s := &Server{}
110+
rr := httptest.NewRecorder()
111+
s.writeDrainPlan(rr, httptest.NewRequest(http.MethodPost, "/", nil), client, "worker-1", k8s.DrainOptions{})
112+
if rr.Code != http.StatusForbidden {
113+
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
114+
}
115+
})
116+
t.Run("forbidden PDB list degrades to an unevaluated plan, not an error", func(t *testing.T) {
117+
client := drainPlanTestClient()
118+
client.PrependReactor("list", "poddisruptionbudgets", func(k8stesting.Action) (bool, runtime.Object, error) {
119+
return true, nil, apierrors.NewForbidden(schema.GroupResource{Group: "policy", Resource: "poddisruptionbudgets"}, "", nil)
120+
})
121+
s := &Server{}
122+
rr := httptest.NewRecorder()
123+
s.writeDrainPlan(rr, httptest.NewRequest(http.MethodPost, "/", nil), client, "worker-1", k8s.DrainOptions{})
124+
var plan k8s.DrainPlan
125+
if rr.Code != http.StatusOK || json.Unmarshal(rr.Body.Bytes(), &plan) != nil || plan.PDBsEvaluated || plan.PDBError == "" {
126+
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
127+
}
128+
})
129+
}
130+
131+
func TestDrainPlanOptionsFromHTTP(t *testing.T) {
132+
t.Run("empty body applies the plan defaults", func(t *testing.T) {
133+
opts, err := drainPlanOptionsFromHTTP(httptest.NewRequest(http.MethodPost, "/", nil))
134+
if err != nil || opts.DeleteEmptyDirData || opts.Force || !opts.IgnoreDaemonSets {
135+
t.Fatalf("got %+v, %v", opts, err)
136+
}
137+
})
138+
t.Run("explicit options are honoured", func(t *testing.T) {
139+
body := strings.NewReader(`{"deleteEmptyDirData":true,"force":true,"gracePeriodSeconds":7}`)
140+
opts, err := drainPlanOptionsFromHTTP(httptest.NewRequest(http.MethodPost, "/", body))
141+
if err != nil || !opts.DeleteEmptyDirData || !opts.Force || opts.GracePeriodSeconds == nil || *opts.GracePeriodSeconds != 7 {
142+
t.Fatalf("got %+v, %v", opts, err)
143+
}
144+
})
145+
t.Run("malformed body is rejected", func(t *testing.T) {
146+
if _, err := drainPlanOptionsFromHTTP(httptest.NewRequest(http.MethodPost, "/", strings.NewReader("{"))); err == nil {
147+
t.Fatalf("expected a decode error")
148+
}
149+
})
150+
}

‎internal/server/server.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -663,9 +663,11 @@ func (s *Server) setupAppRoutes(r chi.Router) {
663663
r.Post("/nodes/{name}/debug", s.handleNodeDebug)
664664
r.Delete("/nodes/{name}/debug", s.handleNodeDebugCleanup)
665665

666-
// Node operations (cordon/uncordon)
666+
// Node operations (cordon/uncordon) and the read-only drain plan.
667+
// The drain itself lives outside this timeout group, see above.
667668
r.Post("/nodes/{name}/cordon", s.handleCordonNode)
668669
r.Post("/nodes/{name}/uncordon", s.handleUncordonNode)
670+
r.Post("/nodes/{name}/drain-plan", s.handleDrainPlan)
669671

670672
// Pod file browser
671673
r.Get("/pods/{namespace}/{name}/files", s.handlePodFileList)

0 commit comments

Comments
 (0)