nucleic/third_party/containerization/PATCHES.md

Vendored containerization — Nucleic patches

Overview of why these patches exist (the session-isolation model) + the build/validate workflow: docs/CONTAINER_ISOLATION.md. This file is the per-patch detail.

This is a vendored copy of apple/containerization at upstream commit 6b7b42ca3efeee8c706070e4355e6a807c5336ae, referenced by the root Package.swift via .package(path: "third_party/containerization") instead of the github URL.

It is vendored (not pulled) because we carry a local patch upstream doesn't have. Keeping it in-tree means the patch can't be lost to a dependency re-resolve.

What's changed vs. upstream

1. Sources/Containerization/LinuxContainer.swift — forward VM extensions. LinuxContainer.Configuration gains a vmExtensions: [any Sendable] field, and LinuxContainer assigns it into VMConfiguration.extensions when it builds the VM config. Upstream already supports VMConfiguration.extensions + the VZInstanceExtension hook (configureVZ/didCreate), but LinuxContainer — the only entry point we use — never forwarded it, so there was no way to attach a device (e.g. a virtio memory balloon) to a container's VM. Search for the marker comment [Nucleic vendored patch] to find both edit sites.

   Nucleic uses this to attach a VZVirtioTraditionalMemoryBalloonDeviceConfiguration and drive its target at runtime for automatic VM memory reclamation — see MemoryBalloon.swift / ContainerEngine in NucleicCore.

2. Sources/Containerization/LinuxProcess.swift — process-group kill. LinuxProcess gains killProcessGroup(_:), which signals the negative pid (-pid) so the guest's kill(2) targets the exec'd process's whole process group, not just the leader. Every exec is setsid()'d by vmexec, so the process is its own group leader (pgid == pid) and a group signal reaches the children it forked. Upstream only exposes the leader-only kill(_:), which let a forked child survive a Stop in a long-lived shared container. Marked with [Nucleic vendored patch]; used by ContainerizedProcessHandle.sendSignal in NucleicCore.

3. Sources/Containerization/LinuxProcess.swift — stdio-connection diagnostics (log-only). setupIO logs (os.Logger, subsystem com.nucleic, category container-io) when a configured stdio stream's guest side never connects — which leaves its host FileHandle nil, so the relay / readability handler is never wired and the agent's stdin is never delivered (it hangs) or its stdout is never read (the "no output, just a spinner" symptom in Nucleic Control containers). Behavior is unchanged; it only surfaces the failing stream. Marked [Nucleic vendored patch] (the import os, the nucleicIOLog static, and the per-stream check in setupIO). All three are wrapped in #if canImport(os) — non-Xcode toolchains (e.g. a swiftly Swift used to cross-build the host framework) resolve Foundation/Virtualization but not the os overlay, so the diagnostic degrades to a no-op there instead of failing the build; Xcode (the local make vminit-image path) builds keep it.

4. Trimmed for footprint (no behavior change). Tests/, docs/, examples/, and images/ were dropped, and the corresponding .testTarget(...) entries removed from Package.swift. The library/executable targets we build are untouched.

5. Sources/Containerization/LinuxProcess.swift — non-blocking stdio relay. Upstream's setupIO relays guest stdout/stderr with FileHandle.availableData, a blocking read, from inside a readabilityHandler. Those handlers run on Foundation's shared readability queue, so if one exec's guest stdout wedged mid-stream that blocking read parked the shared thread and head-of-line-blocked every other exec's stdout/stderr relay across all containers — one stuck session froze the others. The patch marks each connected fd O_NONBLOCK and drains it via a new nucleicDrainNonBlocking (returns bytes + EOF, never blocks; EAGAIN just waits for the next readable event). A wedged stream is now contained to its own exec. Marked [Nucleic vendored patch] (the two static helpers nucleicSetNonBlocking/nucleicDrainNonBlocking and the two rewritten readabilityHandler blocks). Requires host-side POSIX read/fcntl/errno.

6. Sources/Containerization/LinuxProcess.swift — atomic stdio-or-abort start. In start(), after setupIO returns, if a configured stdio stream never connected from the guest (its FileHandle is nil — patch #3's logged failure), the patch tears the just-created exec back down (agent.deleteProcess) and throws instead of calling startProcess. Upstream proceeds and runs a process with a dead stream (stdin never delivered → hangs; stdout never read → the "no output, just a spinner" 60s stall in Nucleic Control). Now that permanent silent stall surfaces as a clean, retryable start error. Marked [Nucleic vendored patch] (the guard block before startProcess).

7. Sources/Containerization/Vminitd.swift — bounded teardown RPC. deleteProcess now sends a 30s CallOptions.timeout (upstream sends none, so it can block forever on a wedged agent channel). Nucleic calls LinuxProcess.delete() after every turn to reclaim the per-exec vsock/gRPC connection exec() dials; an unbounded deleteProcess would let that reclaim hang and the connection leak. On the thrown deadline, performDeletion still closes the agent connection. Marked [Nucleic vendored patch] (the callOpts block in deleteProcess). NOTE: this pairs with a Nucleic-side change in ContainerizedProcessHandle (call delete() after the exec exits / on force-close) — without that caller, upstream never deletes execs at all and the shared control container leaks a connection + runConnections() task per turn.

8. Sources/ContainerizationOS/Socket/Socket.swift — acceptStream survives transient accept errors. Upstream cancelled the accept DispatchSource on ANY accept(2) failure, permanently ending accepting while the socket stayed bound and listening — a silent black hole: every later client connect(2) SUCCEEDED into the kernel backlog and hung forever unanswered. When the socket is the relayed control plane of a shared container, that is the "agent produced no output within 60s / stdio transport stalled" all-sessions wedge, fixable only by recreating the VM (an app restart). Transient errors — ECONNABORTED/ECONNRESET (a queued connection dying before accept, routine under connection churn), EMFILE/ENFILE/ENOBUFS/ENOMEM (resource pressure), EINTR/EAGAIN — now skip that one accept and keep listening (new isTransientAcceptError). This is SHARED code: the fix reaches the host by a normal build and the guest via the initfs rebuild. Marked [Nucleic vendored patch].

9. Sources/Containerization/UnixSocketRelay.swift — relay loops contain per-connection failures. Both accept loops (setupHostVsockListener — the host half of a container's relayed control socket — and setupHostVsockDial) used to let ONE thrown per-connection dial/connect (host server rebinding, backlog momentarily full → ECONNREFUSED, fd pressure) propagate out of the loop, whose teardown then removed the vsock listener — permanently severing every session in the container from the host control plane. Each connection is now handled on its own task with its error logged and contained (mirroring the guest VsockProxy), and the pre-relay failure paths close both ends so a failed connection fails FAST for the peer and leaks no fds. Marked [Nucleic vendored patch].

10. Sources/ContainerizationOS/Keychain/KeychainQuery.swift — prompt-free registry-credential reads. Upstream's get/list/exists call SecItemCopyMatching with the legacy login Keychain's interactive authorization panel enabled, so any process that isn't on a registry internet-password item's ACL raises the macOS "'cctl' wants to use your confidential information stored in 'ghcr.io' in your keychain" panel when it reads that item — e.g. a cctl binary re-signed ad-hoc by a fresh make vminit-image reading a GHCR token an earlier build stored, or any tool linking KeychainHelper.lookup during an image pull/push/list. Nucleic's rule is that no automatic credential lookup may ever raise a Keychain panel. This patch wraps the three SecItemCopyMatching reads in withoutInteractiveUI (SecKeychainSetUserInteractionAllowed(false) — the only switch that governs the legacy ACL/partition-list dialog; the data-protection kSecUseAuthenticationUI* flags do NOT), so an already-trusted item reads silently while anything else fails with errSecInteractionNotAllowed — which isQuerySuccessful now treats as "not found" so the caller falls back to anonymous / REGISTRY_HOST/USERNAME/TOKEN env auth. save (the cctl login write path) gains a delete-and-retry on errSecDuplicateItem, since the now-silent exists can under-report an unreadable pre-existing item. Mirrors KeychainOwnedAccess.withoutLegacyKeychainUI in NucleicCore. Host-side (shipped by a normal swift build); SecKeychain* deprecation warnings are expected (built with WARNINGS_AS_ERRORS=false). Marked [Nucleic vendored patch].

GUEST-side patches (require rebuilding the initfs — see below)

Patches #1–#7 are host-side (the Containerization library), shipped by a normal swift build. Patches #8+ live in vminitd/ (the guest agent), which rides in the initfs OCI image. They are INERT until that image is rebuilt from this source and published, and ContainerEngine.vminitReference points at it. Build it with make vminit-image (root Makefile) — it builds cctl + the guest vminitd/vmexec from this vendored tree and packages ghcr.io/abkslm/vminit:<tag> into the local cctl store; make vminit-image-push publishes it (authenticate once with make vminit-image-login, which stores a GHCR token in the macOS Keychain — or set REGISTRY_HOST/USERNAME/TOKEN), and vminitReference is pinned to that custom image. First time on a machine, run make vminit-image-prep once (installs the swiftly toolchain + musl SDK the guest cross-build needs). Bump the -nucleicN tag suffix and rebuild whenever a guest patch changes. Built locally, not in CI: the host framework needs the macOS 26+ Virtualization SDK that GitHub-hosted runners lack.

8. vminitd/Sources/VminitdCore/ManagedProcess.swift — offload the blocking start off the event loop. ManagedProcess.start() did synchronous, potentially slow pipe reads (waiting for vmexec to return the pid, then for the error pipe to close) while holding state's Mutex, ON the calling task — which is the gRPC handler's event-loop thread. A slow start therefore parked the loop and head-of-line-blocked sibling execs' control RPCs sharing it. The patch splits the body into a synchronous startBlocking() and an async start() that runs it on DispatchQueue.global via a checked continuation, keeping the loop responsive. Safe because the body has no await and ManagedProcess is Sendable. Marked [Nucleic vendored patch].

9. Per-exec cgroups (OOM/CPU/pids isolation). Upstream puts the container init AND every exec in ONE cgroup (/container/<id>), so one session's runaway RSS trips the in-VM OOM-killer against a random sibling, and a fork bomb / CPU hog hits the whole box. This patch makes /container/<id> an intermediary: the resource ceiling stays on it, ManagedContainer.init moves the init into its own leaf (/container/<id>/init) which enables cgroup.subtree_control up the chain, and ManagedProcess.start places each exec in its OWN child (/container/<id>/<execID>) with memory.oom.group=1 (a runaway session's OOM kills only its tree), a fair cpu.weight, and a pids.max fork-bomb backstop. New Cgroup2Manager helpers: setOomGroup/setCpuWeight/ setPidsMax/remove. Best-effort with a graceful fallback: if any step of the per-exec setup fails it wipes the partial state and reverts to the flat layout, and ManagedProcess falls back to the container cgroup per exec — so a cgroup hiccup degrades to today's behavior, never a failed start. ManagedContainer.execCgroupParent == nil marks flat mode.

   Beyond scoped-OOM, a hard host-configured per-exec memory.max is also wired — WITHOUT a protobuf change, because the exec already ships the full OCI Spec and the guest just ignored linux.resources. Host: LinuxProcessConfiguration.memoryLimitInBytes → LinuxContainer.exec stamps it onto spec.linux.resources.memory.limit. Guest: Server+GRPC.createProcess reads that back and passes it to createExec/ManagedProcess, which sets memory.max (new Cgroup2Manager.setMemoryMax) on the exec's cgroup — so a session can't consume the whole box before its own OOM. Driven by Nucleic's ContainerServiceSettings.controlPerSessionMemoryGiB (default 0 = off; applied only to the shared control container, via ContainerManager.exec), so the default stays scoped-OOM-only. Marked [Nucleic vendored patch] across Cgroup2Manager.swift, ManagedContainer.swift, ManagedProcess.swift, Server+GRPC.swift (guest) and LinuxProcessConfiguration.swift, LinuxContainer.swift (host). COMPILE-VERIFIED (host + musl cross-build); NOT yet runtime-validated — a wrong cgroup-v2 hierarchy fails at runtime, so boot a container with the new image and confirm sessions start, /sys/fs/cgroup/container/<id>/<execID> exists per session, and a hog is contained, before pointing a shipping build at it. vmexec/RunCommand is unchanged: it still applies linux.resources at linux.cgroupsPath, which the patch repoints (init leaf) and clears accordingly.

10. vminitd/Sources/VminitdCore/VsockProxy.swift — leak-proof, crash-proof relay connections; no black-hole listener. Four fixes to the guest half of the relayed control socket (the path every session's MCP/approval traffic crosses in a shared container):

• fd leak (the root of the recurring all-sessions stall): cleanup ran its two epoll unregisters and two close(2)s in one do/catch, so a thrown unregister SKIPPED the closes — leaking both connection fds. Control-plane traffic is connection-churny by design (an SSE tools/call closes its connection every gated call; every intercepted git/gh/command event is a short-lived connection), so the leaks accumulated until vminitd hit EMFILE, its accept path began failing, and — before patch #10 — the accept stream died with the guest socket still bound: every session in the container then stalled ("produced no output within 60s") until the VM was recreated. Each cleanup step now runs independently.
• double-resume crash: both fds' epoll handlers can reach the cleanup condition; a second entry would resume the CheckedContinuation twice — a fatal trap in the VM's PID-1 agent. cleanup is now once-guarded.
• try! registrations: an epoll_ctl failure crashed vminitd outright; registration failures now fail only that connection, releasing whatever was already set up.
• no black-hole listener: if the accept loop ever ends unexpectedly, the proxy now closes its listener (new listenerLoopEnded), so peers get fail-fast refusals instead of connecting into a never-accepted backlog. A failed pre-relay connection is also closed explicitly. Marked [Nucleic vendored patch].

Re-vendoring a newer upstream commit

1. git clone upstream (or copy .build/checkouts/containerization after bumping the URL pin temporarily), check out the desired commit.
2. rsync -a --exclude=.git --exclude=.build --exclude=.swiftpm --exclude=Tests/ --exclude=docs/ \ --exclude=images/ <upstream>/ third_party/containerization/
3. Remove the .testTarget(...) blocks from third_party/containerization/Package.swift.
4. Re-apply patch #1 (the vmExtensions field + the vmConfig.extensions = … forward), patch #2 (LinuxProcess.killProcessGroup(_:)), patch #3 (the setupIO stdio-connection log + its import os / nucleicIOLog), patch #5 (the non-blocking stdio relay: nucleicSetNonBlocking / nucleicDrainNonBlocking + the rewritten readabilityHandler blocks), and patch #6 (the atomic stdio-or-abort guard in start()), patch #7 (the bounded deleteProcess timeout in Vminitd.swift), and patch #8 (the ManagedProcess.start event-loop offload in vminitd/). Grep for [Nucleic vendored patch] to find every site, and patch #9 (per-exec cgroups) across Cgroup2Manager.swift / ManagedContainer.swift / ManagedProcess.swift, patch #10 (Socket.acceptStream transient-error tolerance + isTransientAcceptError), patch #11 (the UnixSocketRelay per-connection containment + fail-fast closes), patch #12 (the VsockProxy cleanup/try!/listener hardening in vminitd/), and patch #13 (the prompt-free KeychainQuery reads: withoutInteractiveUI + the errSecInteractionNotAllowed handling + the save duplicate retry). After re-applying any vminitd/ patch, rebuild + publish the custom init image with make vminit-image + make vminit-image-push, and bump ContainerEngine.vminitReference.
5. Update the commit hash above and in the root Package.swift comment.
6. swift build and run the balloon tests.