Files
containerization/PATCHES.md
T
abkslmandClaude Fable 5 6f950d859b Fix the recurring session-stall pair: MainActor lock-reconcile hang + vminitd relay spin
Host (dominant): AppStore.reconcileLocks polls every 3s on the MainActor
while any lock is held — effectively forever, since interrupted/errored
sessions deliberately retain locks. Each pass ran heldPathDisposition's
diverges() as three held×unmerged scans with two split-allocations per
pathsOverlap call, pinning the main thread for tens of seconds per pass
on a diverged trunk (hang-reports 2026-07-28: 100% of samples in
reconcileLocks→heldPathDisposition→pathsOverlap). That froze running
sessions' transcripts and starved the spawn path into the 60s
"produced no output" watchdog. pathsOverlap is now allocation-free
bytewise comparison with identical semantics, and divergentHeldPaths
answers all three questions from one O((held+unmerged)·depth) set.

Guest (persistence): VsockProxy threaded ONE offset pair through BOTH
relay directions; once the EAGAIN-return backpressure patch let pending
bytes persist, traffic in the other direction skewed the shared
counters, made the write leg unreachable, and spun the single
ProcessSupervisor poller thread forever — container-wide dead control
plane until VM recreation, triggered by exactly the backpressure the
host hang created. Each direction now owns its own pipe and counters
(OSFile.RelayDirection), and source EOF is only surfaced after the
pipe drains so SHUT_WR can't truncate a parked tail. Vendored patch
docs updated (#15); inert until the initfs image is rebuilt+repointed.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-28 15:05:37 -07:00

289 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Vendored `containerization` — Nucleic patches
> Overview of *why* these patches exist (the session-isolation model) + the build/validate workflow:
> [`docs/CONTAINER_ISOLATION.md`](../../docs/CONTAINER_ISOLATION.md). This file is the per-patch detail.
This is a **vendored copy** of [apple/containerization](https://github.com/apple/containerization)
at upstream commit `6b7b42ca3efeee8c706070e4355e6a807c5336ae`, referenced by the root `Package.swift`
via `.package(path: "third_party/containerization")` instead of the github URL.
It is vendored (not pulled) because we carry a local patch upstream doesn't have. Keeping it
in-tree means the patch can't be lost to a dependency re-resolve.
## What's changed vs. upstream
1. **`Sources/Containerization/LinuxContainer.swift` — forward VM extensions.**
`LinuxContainer.Configuration` gains a `vmExtensions: [any Sendable]` field, and
`LinuxContainer` assigns it into `VMConfiguration.extensions` when it builds the VM config.
Upstream already supports `VMConfiguration.extensions` + the `VZInstanceExtension` hook
(`configureVZ`/`didCreate`), but `LinuxContainer` — the only entry point we use — never forwarded
it, so there was no way to attach a device (e.g. a virtio memory balloon) to a container's VM.
Search for the marker comment `[Nucleic vendored patch]` to find both edit sites.
Nucleic uses this to attach a `VZVirtioTraditionalMemoryBalloonDeviceConfiguration` and drive its
target at runtime for automatic VM memory reclamation — see `MemoryBalloon.swift` /
`ContainerEngine` in NucleicCore.
2. **`Sources/Containerization/LinuxProcess.swift` — process-group kill.**
`LinuxProcess` gains `killProcessGroup(_:)`, which signals the negative pid (`-pid`) so the
guest's `kill(2)` targets the exec'd process's whole **process group**, not just the leader.
Every exec is `setsid()`'d by `vmexec`, so the process is its own group leader (pgid == pid) and
a group signal reaches the children it forked. Upstream only exposes the leader-only `kill(_:)`,
which let a forked child survive a Stop in a long-lived shared container. Marked with
`[Nucleic vendored patch]`; used by `ContainerizedProcessHandle.sendSignal` in NucleicCore.
3. **`Sources/Containerization/LinuxProcess.swift` — stdio-connection diagnostics (log-only).**
`setupIO` logs (`os.Logger`, subsystem `com.nucleic`, category `container-io`) when a *configured*
stdio stream's guest side never connects — which leaves its host `FileHandle` nil, so the relay /
readability handler is never wired and the agent's stdin is never delivered (it hangs) or its
stdout is never read (the "no output, just a spinner" symptom in Nucleic Control containers).
Behavior is unchanged; it only surfaces the failing stream. Marked `[Nucleic vendored patch]`
(the `import os`, the `nucleicIOLog` static, and the per-stream check in `setupIO`). All three are
wrapped in `#if canImport(os)` — non-Xcode toolchains (e.g. a swiftly Swift used to cross-build the
host framework) resolve Foundation/Virtualization but not the `os` overlay, so the diagnostic
degrades to a no-op there instead of failing the build; Xcode (the local `make vminit-image` path)
builds keep it.
4. **Trimmed for footprint (no behavior change).** `Tests/`, `docs/`, `examples/`, and `images/`
were dropped, and the corresponding `.testTarget(...)` entries removed from `Package.swift`. The
library/executable targets we build are untouched.
5. **`Sources/Containerization/LinuxProcess.swift` — non-blocking stdio relay.**
Upstream's `setupIO` relays guest stdout/stderr with `FileHandle.availableData`, a **blocking**
read, from inside a `readabilityHandler`. Those handlers run on Foundation's shared readability
queue, so if one exec's guest stdout wedged mid-stream that blocking read parked the shared thread
and head-of-line-blocked **every** other exec's stdout/stderr relay across all containers — one
stuck session froze the others. The patch marks each connected fd `O_NONBLOCK` and drains it via a
new `nucleicDrainNonBlocking` (returns bytes + EOF, never blocks; EAGAIN just waits for the next
readable event). A wedged stream is now contained to its own exec. Marked `[Nucleic vendored patch]`
(the two static helpers `nucleicSetNonBlocking`/`nucleicDrainNonBlocking` and the two rewritten
`readabilityHandler` blocks). Requires host-side POSIX `read`/`fcntl`/`errno`.
6. **`Sources/Containerization/LinuxProcess.swift` — atomic stdio-or-abort start.**
In `start()`, after `setupIO` returns, if a *configured* stdio stream never connected from the
guest (its `FileHandle` is nil — patch #3's logged failure), the patch tears the just-created exec
back down (`agent.deleteProcess`) and throws instead of calling `startProcess`. Upstream proceeds
and runs a process with a dead stream (stdin never delivered → hangs; stdout never read → the "no
output, just a spinner" 60s stall in Nucleic Control). Now that permanent silent stall surfaces as
a clean, retryable start error. Marked `[Nucleic vendored patch]` (the guard block before
`startProcess`).
7. **`Sources/Containerization/Vminitd.swift` — bounded teardown RPC.**
`deleteProcess` now sends a 30s `CallOptions.timeout` (upstream sends none, so it can block
forever on a wedged agent channel). Nucleic calls `LinuxProcess.delete()` after every turn to
reclaim the per-exec vsock/gRPC connection `exec()` dials; an unbounded `deleteProcess` would let
that reclaim hang and the connection leak. On the thrown deadline, `performDeletion` still closes
the agent connection. Marked `[Nucleic vendored patch]` (the `callOpts` block in `deleteProcess`).
NOTE: this pairs with a Nucleic-side change in `ContainerizedProcessHandle` (call `delete()` after
the exec exits / on force-close) — without that caller, upstream never deletes execs at all and
the shared control container leaks a connection + `runConnections()` task per turn.
10. **`Sources/ContainerizationOS/Socket/Socket.swift` — `acceptStream` survives transient accept
errors.** Upstream cancelled the accept `DispatchSource` on ANY `accept(2)` failure, permanently
ending accepting while the socket stayed **bound and listening** — a silent black hole: every
later client `connect(2)` SUCCEEDED into the kernel backlog and hung forever unanswered. When
the socket is the relayed control plane of a shared container, that is the "agent produced no
output within 60s / stdio transport stalled" all-sessions wedge, fixable only by recreating the
VM (an app restart). Transient errors — `ECONNABORTED`/`ECONNRESET` (a queued connection dying
before accept, routine under connection churn), `EMFILE`/`ENFILE`/`ENOBUFS`/`ENOMEM` (resource
pressure), `EINTR`/`EAGAIN` — now skip that one accept and keep listening (new
`isTransientAcceptError`). This is SHARED code: the fix reaches the host by a normal build and
the guest via the initfs rebuild. Marked `[Nucleic vendored patch]`.
11. **`Sources/Containerization/UnixSocketRelay.swift` — relay loops contain per-connection
failures.** Both accept loops (`setupHostVsockListener` — the host half of a container's relayed
control socket — and `setupHostVsockDial`) used to let ONE thrown per-connection dial/connect
(host server rebinding, backlog momentarily full → `ECONNREFUSED`, fd pressure) propagate out of
the loop, whose teardown then removed the vsock listener — permanently severing every session in
the container from the host control plane. Each connection is now handled on its own task with
its error logged and contained (mirroring the guest `VsockProxy`), and the pre-relay failure
paths close both ends so a failed connection fails FAST for the peer and leaks no fds. Marked
`[Nucleic vendored patch]`.
13. **`Sources/ContainerizationOS/Keychain/KeychainQuery.swift` — prompt-free registry-credential
reads.** Upstream's `get`/`list`/`exists` call `SecItemCopyMatching` with the legacy login
Keychain's interactive authorization panel enabled, so any process that isn't on a registry
internet-password item's ACL raises the macOS *"'cctl' wants to use your confidential information
stored in 'ghcr.io' in your keychain"* panel when it reads that item — e.g. a `cctl` binary
re-signed ad-hoc by a fresh `make vminit-image` reading a GHCR token an earlier build stored, or
any tool linking `KeychainHelper.lookup` during an image pull/push/list. Nucleic's rule is that no
automatic credential lookup may ever raise a Keychain panel. This patch wraps the three
`SecItemCopyMatching` reads in `withoutInteractiveUI` (`SecKeychainSetUserInteractionAllowed(false)`
— the only switch that governs the legacy ACL/partition-list dialog; the data-protection
`kSecUseAuthenticationUI*` flags do NOT), so an already-trusted item reads silently while anything
else fails with `errSecInteractionNotAllowed` — which `isQuerySuccessful` now treats as "not found"
so the caller falls back to anonymous / `REGISTRY_HOST`/`USERNAME`/`TOKEN` env auth. `save`
(the `cctl login` write path) gains a delete-and-retry on `errSecDuplicateItem`, since the
now-silent `exists` can under-report an unreadable pre-existing item. Mirrors
`KeychainOwnedAccess.withoutLegacyKeychainUI` in NucleicCore. Host-side (shipped by a normal
`swift build`). The three `SecKeychain*` symbols are formally deprecated but are the only API
covering the legacy ACL panel; they're bound directly via `@_silgen_name` (`nucleic_SecKeychain*`,
top of file) so the required calls compile without deprecation warnings. Marked
`[Nucleic vendored patch]`.
14. **`Sources/Containerization/Vminitd.swift` — configure the gRPC pipeline before the channel goes
active.** `Vminitd.init` used the now-deprecated `HTTP2ClientTransport.WrappedChannel.wrapping(
channel:config:serviceConfig:)`, which wraps an already-connected channel best-effort and may drop
early server frames such as SETTINGS. Migrated to `wrapping(config:serviceConfig:makeChannel:)`,
which invokes the transport's `configure` inside the bootstrap's channel initializer — before the
vsock channel becomes active. `init` is now `async throws` (the new overload is async); its callers
in `VZVirtualMachineInstance` (`start`/`dialAgent`, both already async) and the integration test now
`try await`. Marked `[Nucleic vendored patch]`.
### GUEST-side patches (require rebuilding the initfs — see below)
Patches #1–#7 are host-side (the `Containerization` library), shipped by a normal `swift build`.
Patches #8+ live in `vminitd/` (the guest agent), which rides in the initfs OCI image. They are INERT
until that image is rebuilt from this source and published, and `ContainerEngine.vminitReference`
points at it. Build it with **`make vminit-image`** (root Makefile) — it builds cctl + the guest
vminitd/vmexec from this vendored tree and packages `ghcr.io/abkslm/vminit:<tag>` into the local cctl
store; `make vminit-image-push` publishes it (authenticate once with `make vminit-image-login`, which
stores a GHCR token in the macOS Keychain — or set `REGISTRY_HOST`/`USERNAME`/`TOKEN`), and
`vminitReference` is pinned to that custom image. First time on a machine, run `make vminit-image-prep` once (installs
the swiftly toolchain + musl SDK the guest cross-build needs). Bump the `-nucleicN` tag suffix and
rebuild whenever a guest patch changes. Built locally, not in CI: the host framework needs the macOS
26+ Virtualization SDK that GitHub-hosted runners lack.
8. **`vminitd/Sources/VminitdCore/ManagedProcess.swift` — offload the blocking start off the event loop.**
`ManagedProcess.start()` did synchronous, potentially slow pipe reads (waiting for `vmexec` to
return the pid, then for the error pipe to close) while holding `state`'s Mutex, ON the calling
task — which is the gRPC handler's event-loop thread. A slow start therefore parked the loop and
head-of-line-blocked sibling execs' control RPCs sharing it. The patch splits the body into a
synchronous `startBlocking()` and an async `start()` that runs it on `DispatchQueue.global` via a
checked continuation, keeping the loop responsive. Safe because the body has no `await` and
`ManagedProcess` is `Sendable`. Marked `[Nucleic vendored patch]`.
9. **Per-exec cgroups (OOM/CPU/pids isolation).** Upstream puts the container init AND every exec in
ONE cgroup (`/container/<id>`), so one session's runaway RSS trips the in-VM OOM-killer against a
*random* sibling, and a fork bomb / CPU hog hits the whole box. This patch makes `/container/<id>`
an intermediary: the resource ceiling stays on it, `ManagedContainer.init` moves the init into its
own leaf (`/container/<id>/init`) which enables `cgroup.subtree_control` up the chain, and
`ManagedProcess.start` places each exec in its OWN child (`/container/<id>/<execID>`) with
`memory.oom.group=1` (a runaway session's OOM kills only *its* tree), a fair `cpu.weight`, and a
`pids.max` fork-bomb backstop. New `Cgroup2Manager` helpers: `setOomGroup`/`setCpuWeight`/
`setPidsMax`/`remove`. **Best-effort with a graceful fallback**: if any step of the per-exec setup
fails it wipes the partial state and reverts to the flat layout, and `ManagedProcess` falls back to
the container cgroup per exec — so a cgroup hiccup degrades to today's behavior, never a failed
start. `ManagedContainer.execCgroupParent == nil` marks flat mode.
Beyond scoped-OOM, a **hard host-configured per-exec `memory.max`** is also wired — WITHOUT a
protobuf change, because the exec already ships the full OCI `Spec` and the guest just ignored
`linux.resources`. Host: `LinuxProcessConfiguration.memoryLimitInBytes` → `LinuxContainer.exec`
stamps it onto `spec.linux.resources.memory.limit`. Guest: `Server+GRPC.createProcess` reads that
back and passes it to `createExec`/`ManagedProcess`, which sets `memory.max` (new
`Cgroup2Manager.setMemoryMax`) on the exec's cgroup — so a session can't consume the whole box
before its own OOM. Driven by Nucleic's `ContainerServiceSettings.controlPerSessionMemoryGiB`
(default 0 = off; applied only to the shared control container, via `ContainerManager.exec`), so the
default stays scoped-OOM-only. Marked `[Nucleic vendored patch]` across `Cgroup2Manager.swift`,
`ManagedContainer.swift`, `ManagedProcess.swift`, `Server+GRPC.swift` (guest) and
`LinuxProcessConfiguration.swift`, `LinuxContainer.swift` (host). **COMPILE-VERIFIED (host + musl
cross-build); NOT yet runtime-validated** — a
wrong cgroup-v2 hierarchy fails at runtime, so boot a container with the new image and confirm
sessions start, `/sys/fs/cgroup/container/<id>/<execID>` exists per session, and a hog is contained,
before pointing a shipping build at it. `vmexec/RunCommand` is unchanged: it still applies
`linux.resources` at `linux.cgroupsPath`, which the patch repoints (init leaf) and clears
accordingly.
12. **`vminitd/Sources/VminitdCore/VsockProxy.swift` — leak-proof, crash-proof relay connections;
no black-hole listener.** Four fixes to the guest half of the relayed control socket (the path
every session's MCP/approval traffic crosses in a shared container):
- **fd leak (the root of the recurring all-sessions stall):** `cleanup` ran its two epoll
unregisters and two `close(2)`s in one `do/catch`, so a thrown unregister SKIPPED the closes —
leaking both connection fds. Control-plane traffic is connection-churny by design (an SSE
`tools/call` closes its connection every gated call; every intercepted git/gh/command event is
a short-lived connection), so the leaks accumulated until vminitd hit `EMFILE`, its accept
path began failing, and — before patch #10 — the accept stream died with the guest socket
still bound: every session in the container then stalled ("produced no output within 60s")
until the VM was recreated. Each cleanup step now runs independently.
- **double-resume crash:** both fds' epoll handlers can reach the cleanup condition; a second
entry would resume the `CheckedContinuation` twice — a fatal trap in the VM's PID-1 agent.
`cleanup` is now once-guarded.
- **`try!` registrations:** an `epoll_ctl` failure crashed vminitd outright; registration
failures now fail only that connection, releasing whatever was already set up.
- **no black-hole listener:** if the accept loop ever ends unexpectedly, the proxy now closes
its listener (new `listenerLoopEnded`), so peers get fail-fast refusals instead of connecting
into a never-accepted backlog. A failed pre-relay connection is also closed explicitly.
Marked `[Nucleic vendored patch]`.
14. **Non-blocking, non-spinning guest I/O plane (`IOPair.swift`, `OSFile+Splice.swift`,
`VsockProxy.swift`) — the root cause of the "control plane unresponsive / all sessions stall"
wedge.** Every exec's stdio relay (`IOPair`) and every control-plane relay connection
(`VsockProxy`) share ONE thread: `ProcessSupervisor.default`'s epoll poller. Three defects let
a single slow peer freeze that thread — and with it every session's stdio AND the whole
container's control plane at once (probes then read "accepts connections but never answers";
before the manager-side recovery rework the host restarted the container over this, SIGKILLing
every session):
- **`IOPair` blocking write:** only *registered* fds get `O_NONBLOCK` (set by `Epoll.add`), and
the relay registers only its READ fd — the write fd (e.g. the vsock socket carrying an exec's
stdout to the host) stayed blocking. One slow-drained stream parked the poller thread in
`write(2)`. The relay now sets the write fd non-blocking up front and implements real
backpressure: a full destination stashes the remainder in a `pending` backlog, registers the
write fd for EPOLLOUT, and suspends reads until the flush completes (throttling the producing
process via its own pipe, not the shared thread). The old close-on-short-write path — dead
while the fd was blocking, live and stdio-dropping once non-blocking — is subsumed by the
backlog; genuine write errors still close the pair.
- **`OSFile.splice` busy-spin:** when the destination was full (EAGAIN) and the source idle,
the outer loop had no exit — it spun the poller thread at 100% until the peer drained (or
forever if it never did). The flush leg's EAGAIN branch now returns partial progress; the
un-flushed bytes stay in the transfer pipe and the destination's EPOLLOUT edge resumes the
flush (both `VsockProxy` handlers already pump both directions on both events).
- **`VsockProxy` registration race:** the client fd's epoll handler can fire — and splice
toward the server fd — before the second registration makes that fd non-blocking; a full
destination made that a genuinely blocking splice on the poller thread. Both fds are now set
non-blocking before either is registered.
All three are guest-side and INERT until the initfs image is rebuilt (`make vminit-image`, tag
`0.34.0-nucleic4`) and `ContainerEngine.vminitReference` is bumped after runtime validation.
Marked `[Nucleic vendored patch]`. **NOTE:** the splice EAGAIN-return introduced a latent
cross-direction accounting hazard fixed by patch #15.
15. **Per-direction relay state in `OSFile+Splice.swift` + `VsockProxy.swift` — fixes the
poller-thread spin patch #14 made reachable.** The old `SpliceFile` design threaded ONE offset
pair through BOTH directions of a proxied connection: each fd's struct served as the
read-counter for one direction and the write-counter for the other, so the loop guards compared
*differences of two directions' counters*. That was survivable only while every splice call
fully drained its transfer pipe before returning. Once patch #14's EAGAIN branch let pending
bytes persist across calls, one parked direction skewed the shared counters for the other: its
write leg's `to.offset < from.offset` guard went false with data still in the pipe, and the
outer `while true` then alternated read-EAGAIN/skip-write forever — a hard 100% spin on the
single `ProcessSupervisor` poller thread (epoll never runs again, so the event that would
un-skew the counters can never be processed). Container-wide dead control plane + frozen stdio
until the VM is recreated; triggered in practice by host-side backpressure (any slow host
reader), and bidirectional traffic on one connection (e.g. the control plane's SSE stream +
requests). Replaced `SpliceFile`/`OSFile.splice` with `OSFile.RelayDirection` (each direction
owns its OWN pipe, `bytesIn`/`bytesOut`, and `sawSourceEOF`) and `OSFile.relay`, which also
fixes a second latent defect: source EOF is now reported only after the pipe fully drains, so
the caller's SHUT_WR can never truncate a parked tail (the old read leg returned `.eof`
immediately, dropping pending bytes). `VsockProxy.handleConn` now tracks two directions with
independent done flags; a broken destination ends both. Guest-side and INERT until the initfs
image is rebuilt and repointed (mind the `vminit.ext4.reference` cache sidecar). Marked
`[Nucleic vendored patch]`.
## Re-vendoring a newer upstream commit
1. `git clone` upstream (or copy `.build/checkouts/containerization` after bumping the URL pin
temporarily), check out the desired commit.
2. `rsync -a --exclude=.git --exclude=.build --exclude=.swiftpm --exclude=Tests/ --exclude=docs/ \
--exclude=images/ <upstream>/ third_party/containerization/`
3. Remove the `.testTarget(...)` blocks from `third_party/containerization/Package.swift`.
4. Re-apply patch #1 (the `vmExtensions` field + the `vmConfig.extensions = …` forward), patch #2
(`LinuxProcess.killProcessGroup(_:)`), patch #3 (the `setupIO` stdio-connection log + its
`import os` / `nucleicIOLog`), patch #5 (the non-blocking stdio relay: `nucleicSetNonBlocking` /
`nucleicDrainNonBlocking` + the rewritten `readabilityHandler` blocks), and patch #6 (the atomic
stdio-or-abort guard in `start()`), patch #7 (the bounded `deleteProcess` timeout in
`Vminitd.swift`), and patch #8 (the `ManagedProcess.start` event-loop offload in `vminitd/`). Grep
for `[Nucleic vendored patch]` to find every site, and patch #9 (per-exec cgroups) across
`Cgroup2Manager.swift` / `ManagedContainer.swift` / `ManagedProcess.swift`, patch #10
(`Socket.acceptStream` transient-error tolerance + `isTransientAcceptError`), patch #11 (the
`UnixSocketRelay` per-connection containment + fail-fast closes), patch #12 (the `VsockProxy`
cleanup/`try!`/listener hardening in `vminitd/`), patch #13 (the prompt-free
`KeychainQuery` reads: `withoutInteractiveUI` + the `errSecInteractionNotAllowed` handling +
the `save` duplicate retry), and patch #14 (the non-blocking/non-spinning guest I/O plane:
`IOPair` backpressure, the `OSFile.splice` EAGAIN return, and the `VsockProxy` pre-registration
non-blocking fds — all in `vminitd/`), and patch #15 (the per-direction
`OSFile.RelayDirection`/`OSFile.relay` rewrite + the two-direction `VsockProxy.handleConn`,
which supersede the upstream `SpliceFile`/`splice` shapes entirely — in `vminitd/`). After
re-applying any `vminitd/` patch, rebuild + publish the custom init image
with `make vminit-image` + `make vminit-image-push`, and bump `ContainerEngine.vminitReference`.
5. Update the commit hash above and in the root `Package.swift` comment.
6. `swift build` and run the balloon tests.