nucleic/third_party/containerization/PATCHES.md

Vendored containerization — Nucleic patches

Overview of why these patches exist (the session-isolation model) + the build/validate workflow: docs/CONTAINER_ISOLATION.md. This file is the per-patch detail.

This is a vendored copy of apple/containerization at upstream commit 6b7b42ca3efeee8c706070e4355e6a807c5336ae, referenced by the root Package.swift via .package(path: "third_party/containerization") instead of the github URL.

It is vendored (not pulled) because we carry a local patch upstream doesn't have. Keeping it in-tree means the patch can't be lost to a dependency re-resolve.

What's changed vs. upstream

1. Sources/Containerization/LinuxContainer.swift — forward VM extensions. LinuxContainer.Configuration gains a vmExtensions: [any Sendable] field, and LinuxContainer assigns it into VMConfiguration.extensions when it builds the VM config. Upstream already supports VMConfiguration.extensions + the VZInstanceExtension hook (configureVZ/didCreate), but LinuxContainer — the only entry point we use — never forwarded it, so there was no way to attach a device (e.g. a virtio memory balloon) to a container's VM. Search for the marker comment [Nucleic vendored patch] to find both edit sites.

   Nucleic uses this to attach a VZVirtioTraditionalMemoryBalloonDeviceConfiguration and drive its target at runtime for automatic VM memory reclamation — see MemoryBalloon.swift / ContainerEngine in NucleicCore.

2. Sources/Containerization/LinuxProcess.swift — process-group kill. LinuxProcess gains killProcessGroup(_:), which signals the negative pid (-pid) so the guest's kill(2) targets the exec'd process's whole process group, not just the leader. Every exec is setsid()'d by vmexec, so the process is its own group leader (pgid == pid) and a group signal reaches the children it forked. Upstream only exposes the leader-only kill(_:), which let a forked child survive a Stop in a long-lived shared container. Marked with [Nucleic vendored patch]; used by ContainerizedProcessHandle.sendSignal in NucleicCore.

3. Sources/Containerization/LinuxProcess.swift — stdio-connection diagnostics (log-only). setupIO logs (os.Logger, subsystem com.nucleic, category container-io) when a configured stdio stream's guest side never connects — which leaves its host FileHandle nil, so the relay / readability handler is never wired and the agent's stdin is never delivered (it hangs) or its stdout is never read (the "no output, just a spinner" symptom in Nucleic Control containers). Behavior is unchanged; it only surfaces the failing stream. Marked [Nucleic vendored patch] (the import os, the nucleicIOLog static, and the per-stream check in setupIO). All three are wrapped in #if canImport(os) — non-Xcode toolchains (e.g. a swiftly Swift used to cross-build the host framework) resolve Foundation/Virtualization but not the os overlay, so the diagnostic degrades to a no-op there instead of failing the build; Xcode (the local make vminit-image path) builds keep it.

4. Trimmed for footprint (no behavior change). Tests/, docs/, examples/, and images/ were dropped, and the corresponding .testTarget(...) entries removed from Package.swift. The library/executable targets we build are untouched.

5. Sources/Containerization/LinuxProcess.swift — non-blocking stdio relay. Upstream's setupIO relays guest stdout/stderr with FileHandle.availableData, a blocking read, from inside a readabilityHandler. Those handlers run on Foundation's shared readability queue, so if one exec's guest stdout wedged mid-stream that blocking read parked the shared thread and head-of-line-blocked every other exec's stdout/stderr relay across all containers — one stuck session froze the others. The patch marks each connected fd O_NONBLOCK and drains it via a new nucleicDrainNonBlocking (returns bytes + EOF, never blocks; EAGAIN just waits for the next readable event). A wedged stream is now contained to its own exec. Marked [Nucleic vendored patch] (the two static helpers nucleicSetNonBlocking/nucleicDrainNonBlocking and the two rewritten readabilityHandler blocks). Requires host-side POSIX read/fcntl/errno.

6. Sources/Containerization/LinuxProcess.swift — atomic stdio-or-abort start. In start(), after setupIO returns, if a configured stdio stream never connected from the guest (its FileHandle is nil — patch #3's logged failure), the patch tears the just-created exec back down (agent.deleteProcess) and throws instead of calling startProcess. Upstream proceeds and runs a process with a dead stream (stdin never delivered → hangs; stdout never read → the "no output, just a spinner" 60s stall in Nucleic Control). Now that permanent silent stall surfaces as a clean, retryable start error. Marked [Nucleic vendored patch] (the guard block before startProcess).

7. Sources/Containerization/Vminitd.swift — bounded teardown RPC. deleteProcess now sends a 30s CallOptions.timeout (upstream sends none, so it can block forever on a wedged agent channel). Nucleic calls LinuxProcess.delete() after every turn to reclaim the per-exec vsock/gRPC connection exec() dials; an unbounded deleteProcess would let that reclaim hang and the connection leak. On the thrown deadline, performDeletion still closes the agent connection. Marked [Nucleic vendored patch] (the callOpts block in deleteProcess). NOTE: this pairs with a Nucleic-side change in ContainerizedProcessHandle (call delete() after the exec exits / on force-close) — without that caller, upstream never deletes execs at all and the shared control container leaks a connection + runConnections() task per turn.

8. Sources/ContainerizationOS/Socket/Socket.swift — acceptStream survives transient accept errors. Upstream cancelled the accept DispatchSource on ANY accept(2) failure, permanently ending accepting while the socket stayed bound and listening — a silent black hole: every later client connect(2) SUCCEEDED into the kernel backlog and hung forever unanswered. When the socket is the relayed control plane of a shared container, that is the "agent produced no output within 60s / stdio transport stalled" all-sessions wedge, fixable only by recreating the VM (an app restart). Transient errors — ECONNABORTED/ECONNRESET (a queued connection dying before accept, routine under connection churn), EMFILE/ENFILE/ENOBUFS/ENOMEM (resource pressure), EINTR/EAGAIN — now skip that one accept and keep listening (new isTransientAcceptError). This is SHARED code: the fix reaches the host by a normal build and the guest via the initfs rebuild. Marked [Nucleic vendored patch].

9. Sources/Containerization/UnixSocketRelay.swift — relay loops contain per-connection failures. Both accept loops (setupHostVsockListener — the host half of a container's relayed control socket — and setupHostVsockDial) used to let ONE thrown per-connection dial/connect (host server rebinding, backlog momentarily full → ECONNREFUSED, fd pressure) propagate out of the loop, whose teardown then removed the vsock listener — permanently severing every session in the container from the host control plane. Each connection is now handled on its own task with its error logged and contained (mirroring the guest VsockProxy), and the pre-relay failure paths close both ends so a failed connection fails FAST for the peer and leaks no fds. Marked [Nucleic vendored patch].

10. Sources/ContainerizationOS/Keychain/KeychainQuery.swift — prompt-free registry-credential reads. Upstream's get/list/exists call SecItemCopyMatching with the legacy login Keychain's interactive authorization panel enabled, so any process that isn't on a registry internet-password item's ACL raises the macOS "'cctl' wants to use your confidential information stored in 'ghcr.io' in your keychain" panel when it reads that item — e.g. a cctl binary re-signed ad-hoc by a fresh make vminit-image reading a GHCR token an earlier build stored, or any tool linking KeychainHelper.lookup during an image pull/push/list. Nucleic's rule is that no automatic credential lookup may ever raise a Keychain panel. This patch wraps the three SecItemCopyMatching reads in withoutInteractiveUI (SecKeychainSetUserInteractionAllowed(false) — the only switch that governs the legacy ACL/partition-list dialog; the data-protection kSecUseAuthenticationUI* flags do NOT), so an already-trusted item reads silently while anything else fails with errSecInteractionNotAllowed — which isQuerySuccessful now treats as "not found" so the caller falls back to anonymous / REGISTRY_HOST/USERNAME/TOKEN env auth. save (the cctl login write path) gains a delete-and-retry on errSecDuplicateItem, since the now-silent exists can under-report an unreadable pre-existing item. Mirrors KeychainOwnedAccess.withoutLegacyKeychainUI in NucleicCore. Host-side (shipped by a normal swift build). The three SecKeychain* symbols are formally deprecated but are the only API covering the legacy ACL panel; they're bound directly via @_silgen_name (nucleic_SecKeychain*, top of file) so the required calls compile without deprecation warnings. Marked [Nucleic vendored patch].

11. Sources/Containerization/Vminitd.swift — configure the gRPC pipeline before the channel goes active. Vminitd.init used the now-deprecated HTTP2ClientTransport.WrappedChannel.wrapping( channel:config:serviceConfig:), which wraps an already-connected channel best-effort and may drop early server frames such as SETTINGS. Migrated to wrapping(config:serviceConfig:makeChannel:), which invokes the transport's configure inside the bootstrap's channel initializer — before the vsock channel becomes active. init is now async throws (the new overload is async); its callers in VZVirtualMachineInstance (start/dialAgent, both already async) and the integration test now try await. Marked [Nucleic vendored patch].

GUEST-side patches (require rebuilding the initfs — see below)

Patches #1–#7 are host-side (the Containerization library), shipped by a normal swift build. Patches #8+ live in vminitd/ (the guest agent), which rides in the initfs OCI image. They are INERT until that image is rebuilt from this source and published, and ContainerEngine.vminitReference points at it. Build it with make vminit-image (root Makefile) — it builds cctl + the guest vminitd/vmexec from this vendored tree and packages ghcr.io/abkslm/vminit:<tag> into the local cctl store; make vminit-image-push publishes it (authenticate once with make vminit-image-login, which stores a GHCR token in the macOS Keychain — or set REGISTRY_HOST/USERNAME/TOKEN), and vminitReference is pinned to that custom image. First time on a machine, run make vminit-image-prep once (installs the swiftly toolchain + musl SDK the guest cross-build needs). Bump the -nucleicN tag suffix and rebuild whenever a guest patch changes. Built locally, not in CI: the host framework needs the macOS 26+ Virtualization SDK that GitHub-hosted runners lack.

8. vminitd/Sources/VminitdCore/ManagedProcess.swift — offload the blocking start off the event loop. ManagedProcess.start() did synchronous, potentially slow pipe reads (waiting for vmexec to return the pid, then for the error pipe to close) while holding state's Mutex, ON the calling task — which is the gRPC handler's event-loop thread. A slow start therefore parked the loop and head-of-line-blocked sibling execs' control RPCs sharing it. The patch splits the body into a synchronous startBlocking() and an async start() that runs it on DispatchQueue.global via a checked continuation, keeping the loop responsive. Safe because the body has no await and ManagedProcess is Sendable. Marked [Nucleic vendored patch].

9. Per-exec cgroups (OOM/CPU/pids isolation). Upstream puts the container init AND every exec in ONE cgroup (/container/<id>), so one session's runaway RSS trips the in-VM OOM-killer against a random sibling, and a fork bomb / CPU hog hits the whole box. This patch makes /container/<id> an intermediary: the resource ceiling stays on it, ManagedContainer.init moves the init into its own leaf (/container/<id>/init) which enables cgroup.subtree_control up the chain, and ManagedProcess.start places each exec in its OWN child (/container/<id>/<execID>) with memory.oom.group=1 (a runaway session's OOM kills only its tree), a fair cpu.weight, and a pids.max fork-bomb backstop. New Cgroup2Manager helpers: setOomGroup/setCpuWeight/ setPidsMax/remove. Best-effort with a graceful fallback: if any step of the per-exec setup fails it wipes the partial state and reverts to the flat layout, and ManagedProcess falls back to the container cgroup per exec — so a cgroup hiccup degrades to today's behavior, never a failed start. ManagedContainer.execCgroupParent == nil marks flat mode.

   Beyond scoped-OOM, a hard host-configured per-exec memory.max is also wired — WITHOUT a protobuf change, because the exec already ships the full OCI Spec and the guest just ignored linux.resources. Host: LinuxProcessConfiguration.memoryLimitInBytes → LinuxContainer.exec stamps it onto spec.linux.resources.memory.limit. Guest: Server+GRPC.createProcess reads that back and passes it to createExec/ManagedProcess, which sets memory.max (new Cgroup2Manager.setMemoryMax) on the exec's cgroup — so a session can't consume the whole box before its own OOM. Driven by Nucleic's ContainerServiceSettings.controlPerSessionMemoryGiB (default 0 = off; applied only to the shared control container, via ContainerManager.exec), so the default stays scoped-OOM-only. Marked [Nucleic vendored patch] across Cgroup2Manager.swift, ManagedContainer.swift, ManagedProcess.swift, Server+GRPC.swift (guest) and LinuxProcessConfiguration.swift, LinuxContainer.swift (host). COMPILE-VERIFIED (host + musl cross-build); NOT yet runtime-validated — a wrong cgroup-v2 hierarchy fails at runtime, so boot a container with the new image and confirm sessions start, /sys/fs/cgroup/container/<id>/<execID> exists per session, and a hog is contained, before pointing a shipping build at it. vmexec/RunCommand is unchanged: it still applies linux.resources at linux.cgroupsPath, which the patch repoints (init leaf) and clears accordingly.

10. vminitd/Sources/VminitdCore/VsockProxy.swift — leak-proof, crash-proof relay connections; no black-hole listener. Four fixes to the guest half of the relayed control socket (the path every session's MCP/approval traffic crosses in a shared container):

• fd leak (the root of the recurring all-sessions stall): cleanup ran its two epoll unregisters and two close(2)s in one do/catch, so a thrown unregister SKIPPED the closes — leaking both connection fds. Control-plane traffic is connection-churny by design (an SSE tools/call closes its connection every gated call; every intercepted git/gh/command event is a short-lived connection), so the leaks accumulated until vminitd hit EMFILE, its accept path began failing, and — before patch #10 — the accept stream died with the guest socket still bound: every session in the container then stalled ("produced no output within 60s") until the VM was recreated. Each cleanup step now runs independently.
• double-resume crash: both fds' epoll handlers can reach the cleanup condition; a second entry would resume the CheckedContinuation twice — a fatal trap in the VM's PID-1 agent. cleanup is now once-guarded.
• try! registrations: an epoll_ctl failure crashed vminitd outright; registration failures now fail only that connection, releasing whatever was already set up.
• no black-hole listener: if the accept loop ever ends unexpectedly, the proxy now closes its listener (new listenerLoopEnded), so peers get fail-fast refusals instead of connecting into a never-accepted backlog. A failed pre-relay connection is also closed explicitly. Marked [Nucleic vendored patch].

14. Non-blocking, non-spinning guest I/O plane (IOPair.swift, OSFile+Splice.swift, VsockProxy.swift) — the root cause of the "control plane unresponsive / all sessions stall" wedge. Every exec's stdio relay (IOPair) and every control-plane relay connection (VsockProxy) share ONE thread: ProcessSupervisor.default's epoll poller. Three defects let a single slow peer freeze that thread — and with it every session's stdio AND the whole container's control plane at once (probes then read "accepts connections but never answers"; before the manager-side recovery rework the host restarted the container over this, SIGKILLing every session):

• IOPair blocking write: only registered fds get O_NONBLOCK (set by Epoll.add), and the relay registers only its READ fd — the write fd (e.g. the vsock socket carrying an exec's stdout to the host) stayed blocking. One slow-drained stream parked the poller thread in write(2). The relay now sets the write fd non-blocking up front and implements real backpressure: a full destination stashes the remainder in a pending backlog, registers the write fd for EPOLLOUT, and suspends reads until the flush completes (throttling the producing process via its own pipe, not the shared thread). The old close-on-short-write path — dead while the fd was blocking, live and stdio-dropping once non-blocking — is subsumed by the backlog; genuine write errors still close the pair.
• OSFile.splice busy-spin: when the destination was full (EAGAIN) and the source idle, the outer loop had no exit — it spun the poller thread at 100% until the peer drained (or forever if it never did). The flush leg's EAGAIN branch now returns partial progress; the un-flushed bytes stay in the transfer pipe and the destination's EPOLLOUT edge resumes the flush (both VsockProxy handlers already pump both directions on both events).
• VsockProxy registration race: the client fd's epoll handler can fire — and splice toward the server fd — before the second registration makes that fd non-blocking; a full destination made that a genuinely blocking splice on the poller thread. Both fds are now set non-blocking before either is registered. All three are guest-side and INERT until the initfs image is rebuilt (make vminit-image, tag 0.34.0-nucleic4) and ContainerEngine.vminitReference is bumped after runtime validation. Marked [Nucleic vendored patch]. NOTE: the splice EAGAIN-return introduced a latent cross-direction accounting hazard fixed by patch #15.

15. Per-direction relay state in OSFile+Splice.swift + VsockProxy.swift — fixes the poller-thread spin patch #14 made reachable. The old SpliceFile design threaded ONE offset pair through BOTH directions of a proxied connection: each fd's struct served as the read-counter for one direction and the write-counter for the other, so the loop guards compared differences of two directions' counters. That was survivable only while every splice call fully drained its transfer pipe before returning. Once patch #14's EAGAIN branch let pending bytes persist across calls, one parked direction skewed the shared counters for the other: its write leg's to.offset < from.offset guard went false with data still in the pipe, and the outer while true then alternated read-EAGAIN/skip-write forever — a hard 100% spin on the single ProcessSupervisor poller thread (epoll never runs again, so the event that would un-skew the counters can never be processed). Container-wide dead control plane + frozen stdio until the VM is recreated; triggered in practice by host-side backpressure (any slow host reader), and bidirectional traffic on one connection (e.g. the control plane's SSE stream + requests). Replaced SpliceFile/OSFile.splice with OSFile.RelayDirection (each direction owns its OWN pipe, bytesIn/bytesOut, and sawSourceEOF) and OSFile.relay, which also fixes a second latent defect: source EOF is now reported only after the pipe fully drains, so the caller's SHUT_WR can never truncate a parked tail (the old read leg returned .eof immediately, dropping pending bytes). VsockProxy.handleConn now tracks two directions with independent done flags; a broken destination ends both. Guest-side and INERT until the initfs image is rebuilt and repointed (mind the vminit.ext4.reference cache sidecar). Marked [Nucleic vendored patch].

16. Loss-free stdio teardown (IOPair.swift, ManagedProcess.swift, VsockProxy.swift, StandardIO.swift, TerminalIO.swift) — the truncated-final-output class. Four related defects dropped the tail of a stream at teardown, plus two fd leaks:

• IOPair's relay handler closed on a bare EPOLLHUP even with a backpressure flush in flight, dropping the parked remainder (the CLI's final result line) when the destination was momentarily full at process exit. It now returns instead; the destination's EPOLLOUT flushes the backlog, the pump then reads EOF and closes loss-free.
• ManagedProcess.setExit force-closed ALL stdio on SIGCHLD, racing the poller thread's final EPOLLIN drain (drain() silently ignores short writes). It now closes only stdin; stdout/stderr self-close on EOF (loss-free per the previous fix), with an 8s delayed full close as a backstop for pipe-inheriting grandchildren. ManagedProcess.IO is now Sendable for that delayed capture.
• VsockProxy full-hangup teardown dropped bytes already read off the dead peer that were parked toward the SURVIVING peer; it now makes one best-effort relay pass toward the survivor before cleanup.
• VsockProxy leaked the relayTo fd (closeOnDeinit: false) on a failed backend connect(); StandardIO.start/TerminalIO.start/attach leaked live pairs/sockets (retained forever by the supervisor's handler map) on partial setup failure — all paths now close what they created before rethrowing. Marked [Nucleic vendored patch].

17. Epoll registration integrity (Epoll.swift in ContainerizationOS, ProcessSupervisor.swift, IOCloser.swift, TerminalIO.swift).

• Registration generations: epoll events now carry the registration's generation in epoll_data (high 32 bits), and the supervisor dispatches only when it matches the live entry — a stale event queued for a closed registration of a RECYCLED fd number can no longer fire the new registration's handler (it could tear down a brand-new healthy connection mid-batch).
• Non-clobbering registerFd: registering an already-registered fd now fails fast (EEXIST) instead of overwriting the existing handler and then, on the epoll EEXIST, deleting the map entry — which left the fd armed with NO handler (a silently dead relay).
• TerminalIO shared-fd collision: the stdin relay's write destination is now a dup(2) of the terminal fd (DupIOCloser), so its EPOLLOUT backpressure registration can't collide with the stdout relay's read registration on the same number — one bulk paste used to permanently kill the terminal's stdout relay.
• Epoll.add now ORs O_NONBLOCK into existing flags instead of replacing the flag set. Marked [Nucleic vendored patch].

18. Host-side stdin relay off the Swift cooperative pool (LinuxProcess.swift). The stdin fd is blocking (only the read fds get O_NONBLOCK), and startStdinRelay's FileHandle.write parked a width-limited cooperative-pool thread — non-cancellably — whenever the guest stopped reading with the vsock buffer full (large prompt lines). A few wedged sessions starved the whole Swift concurrency runtime (every decode loop and watchdog: an app-wide stall). Writes are now offloaded to a per-process GCD queue via a checked continuation; a wedged write costs one expendable GCD thread, and process deletion closing the fd still unwedges it. Marked [Nucleic vendored patch].

Re-vendoring a newer upstream commit

1. git clone upstream (or copy .build/checkouts/containerization after bumping the URL pin temporarily), check out the desired commit.
2. rsync -a --exclude=.git --exclude=.build --exclude=.swiftpm --exclude=Tests/ --exclude=docs/ \ --exclude=images/ <upstream>/ third_party/containerization/
3. Remove the .testTarget(...) blocks from third_party/containerization/Package.swift.
4. Re-apply patch #1 (the vmExtensions field + the vmConfig.extensions = … forward), patch #2 (LinuxProcess.killProcessGroup(_:)), patch #3 (the setupIO stdio-connection log + its import os / nucleicIOLog), patch #5 (the non-blocking stdio relay: nucleicSetNonBlocking / nucleicDrainNonBlocking + the rewritten readabilityHandler blocks), and patch #6 (the atomic stdio-or-abort guard in start()), patch #7 (the bounded deleteProcess timeout in Vminitd.swift), and patch #8 (the ManagedProcess.start event-loop offload in vminitd/). Grep for [Nucleic vendored patch] to find every site, and patch #9 (per-exec cgroups) across Cgroup2Manager.swift / ManagedContainer.swift / ManagedProcess.swift, patch #10 (Socket.acceptStream transient-error tolerance + isTransientAcceptError), patch #11 (the UnixSocketRelay per-connection containment + fail-fast closes), patch #12 (the VsockProxy cleanup/try!/listener hardening in vminitd/), patch #13 (the prompt-free KeychainQuery reads: withoutInteractiveUI + the errSecInteractionNotAllowed handling + the save duplicate retry), and patch #14 (the non-blocking/non-spinning guest I/O plane: IOPair backpressure, the OSFile.splice EAGAIN return, and the VsockProxy pre-registration non-blocking fds — all in vminitd/), patch #15 (the per-direction OSFile.RelayDirection/OSFile.relay rewrite + the two-direction VsockProxy.handleConn, which supersede the upstream SpliceFile/splice shapes entirely — in vminitd/), patch #16 (loss-free stdio teardown: the IOPair HUP-with-pending return, the setExit stdin-only close + grace pass, the VsockProxy hangup flush + connect-leak close, and the StandardIO/TerminalIO partial-failure cleanup — in vminitd/), patch #17 (epoll registration generations in ContainerizationOS/Linux/Epoll.swift + the generation-checked, non-clobbering ProcessSupervisor handler table + DupIOCloser and the TerminalIO dup destination — spans Sources/ AND vminitd/), and patch #18 (the startStdinRelay GCD write offload in Sources/Containerization/LinuxProcess.swift — host side). After re-applying any vminitd/ patch, rebuild + publish the custom init image with make vminit-image + make vminit-image-push, and bump ContainerEngine.vminitReference.
5. Update the commit hash above and in the root Package.swift comment.
6. swift build and run the balloon tests.