Skip to content

kill process group when a cancelled container hangs - #7087

Open
dany74q wants to merge 1 commit into
moby:masterfrom
dany74q:kill-process-group-no-process-sandbox
Open

dany74q wants to merge 1 commit into
moby:masterfrom
dany74q:kill-process-group-no-process-sandbox

Conversation

@dany74q

@dany74q dany74q commented Aug 27, 2026 •

Copy link
Copy Markdown

When a step is cancelled, buildkit SIGKILLs only the process it started: runc kill for a run, the exec'd pid for an exec. Its subprocesses can survive and keep the IO open, so runc never returns and the solve hangs. Under --oci-worker-no-process-sandbox the container shares the host pid namespace, and rootless runc can't enumerate the survivors without cgroups (opencontainers/runc#4398). An exec leaks the same way with the sandbox, since its children stay in the container's pid namespace. See #4483 and #2855.

The fix SIGKILLs the process group, since runc makes the process a session leader. Exec always kills the group first, and signals the pid only when there is no group. Run does this under NoProcessSandbox, which now passes a pidfile to runc run, before the usual runc kill, then also kills every process left in the init's mount namespace, pinned as soon as runc writes the pidfile, so a process that called setsid goes too. A SIGKILL sent by the client takes the same path.

@dany74q dany74q changed the title executor: kill process group when container does not exit under NoProcessSandbox kill process group when container hangs under NoProcessSandbox Aug 27, 2026
@dany74q
dany74q force-pushed the kill-process-group-no-process-sandbox branch from 839285e to c92b0ac Compare August 27, 2026 07:14
@AkihiroSuda
AkihiroSuda requested a balanced review from Copilot September 2, 2026 03:52
Comment thread executor/runcexecutor/executor.go

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

PID reuse can kill unrelated processes, failure paths can still hang indefinitely, and the regression test does not exercise rootless operation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds process-group cleanup for hung containers using NoProcessSandbox.

Changes:

  • Records run/exec process IDs and kills surviving process groups.
  • Adds cancellation coverage for run and exec paths.
File summaries
File Description
executor/runcexecutor/executor.go Implements process-group termination.
executor/runcexecutor/executor_linux.go Enables PID files in no-sandbox mode.
worker/runc/runc_test.go Adds cancellation regression coverage.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread executor/runcexecutor/executor.go Outdated
Comment thread executor/runcexecutor/executor.go Outdated
Comment thread worker/runc/runc_test.go Outdated
@ZRHann

ZRHann commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi, is this still being worked on? It would be very useful to me. Happy to help if needed.

@jsternberg

Copy link
Copy Markdown
Collaborator

Would there be any negative if, instead of only doing this with NoProcessSandbox, we just always did it?

@dany74q
dany74q force-pushed the kill-process-group-no-process-sandbox branch from c92b0ac to b0bdaa7 Compare October 21, 2026 10:25
@dany74q

dany74q commented Oct 21, 2026

Copy link
Copy Markdown
Author

Would there be any negative if, instead of only doing this with NoProcessSandbox, we just always did it?

none that i found, exec leaked the same way with the sandbox so it kills the group in both modes now. run already gets that from the pid namespace

@AkihiroSuda
AkihiroSuda requested a balanced review from Copilot October 21, 2026 10:30
@AkihiroSuda AkihiroSuda added this to the v0.35.0 milestone Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread executor/runcexecutor/executor.go
Comment thread executor/runcexecutor/executor.go
@dany74q dany74q changed the title kill process group when container hangs under NoProcessSandbox kill process group when a cancelled container hangs Oct 21, 2026
@dany74q

dany74q commented Oct 21, 2026

Copy link
Copy Markdown
Author

friendly ping @AkihiroSuda @jsternberg

@dany74q
dany74q force-pushed the kill-process-group-no-process-sandbox branch from b0bdaa7 to fcb0861 Compare October 21, 2026 20:13
@dany74q
dany74q requested a review from AkihiroSuda October 21, 2026 20:37
@AkihiroSuda

Copy link
Copy Markdown
Member

needs rebasing

@dany74q
dany74q force-pushed the kill-process-group-no-process-sandbox branch from fcb0861 to 80fa839 Compare October 23, 2026 19:59
@dany74q

dany74q commented Oct 23, 2026

Copy link
Copy Markdown
Author

rebased

Signed-off-by: Danny Hershko Shemesh <danny@wiz.io>
@dany74q
dany74q force-pushed the kill-process-group-no-process-sandbox branch from 80fa839 to 323ce2c Compare October 23, 2026 23:54
@dany74q

dany74q commented Oct 24, 2026

Copy link
Copy Markdown
Author

pushed a fix for a race in the cleanup, could someone approve the ci runs on the new head?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants