nucleic/nucleic
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
98432fdfd06767a5c3f3b87a0fcfec7d6b23fb37
nucleic/docs/CONTAINER_ISOLATION.md
T

12 KiB
Raw Blame History

Nucleic — Container session isolation

Status (2026-07-15): host-side fixes are live (any dev build) — now including the control-plane accept-loop hardening (#10/#11) behind the recurring all-sessions "produced no output within 60s" stall. The custom vminit image -nucleic3 is built, pushed, and DEPLOYED (ContainerEngine.vminitReference points at it): it carries the offload (#8), per-exec cgroups (#9, with the graceful flat-layout fallback), and the guest halves of the control-plane hardening (#10/#12 — the VsockProxy fd-leak and black-hole-listener fixes). #9/#10/#12 are compile-verified but were deployed ahead of the §5 runtime validation by explicit decision — if containers misbehave, revert vminitReference to -nucleic1 (the previous validated tag); the reference-keyed initfs cache re-pulls on next launch either way. The spawn-stall watchdog's probe + container auto-recovery (below) additionally converts any control-plane wedge from "restart the app" into "retry the turn".

Why this exists: Nucleic Control runs several agent sessions concurrently, and to keep that affordable on an average Mac it runs them as separate execs inside one shared control container (per agent family) rather than one VM per session — see RUNTIME_ARCHITECTURE (ContainerEngine / ContainerManager) and VSOCK_CONTROL_PLANE. Sharing a container is a memory optimization, but it created a blast-radius problem: one session could wedge, leak, OOM-kill, starve, or fork-bomb the others. This document is the map of how each of those vectors is now contained without giving up the shared-container memory efficiency (project/coding sessions already get one container each — ContainerManager.containerName(for:) — so this is specifically about the shared control container).

The low-level per-patch detail lives in third_party/containerization/PATCHES.md (the vendored Apple containerization framework carries Nucleic patches #1–#12). This doc is the "why + where + how to build/validate" overview.

1. The vectors and the fixes

"One session takes down another" via… Fix Where State
stdio wedge — a stuck stream freezes sibling sessions' relays Non-blocking per-exec stdio relay (O_NONBLOCK + nucleicDrainNonBlocking) replacing blocking FileHandle.availableData; atomic stdio-or-abort start (fail fast + retryable instead of a silent 60s stall) vendored LinuxProcess.swift (patches #5, #6); recoverable-error mapping in ClaudeCodeBackend live (host build)
per-turn connection leak — the "degrades until you quit the app" driver exec() already dials a DEDICATED per-exec vsock/gRPC agent connection; the leak was that Nucleic never called LinuxProcess.delete(), so each turn leaked a connection + runConnections() task + guest exec record. Now ContainerizedProcessHandle fires delete() on exit and on force-close; deleteProcess is bounded by a 30s deadline (patch #7) ContainerizedProcessHandle.swift; vendored Vminitd.swift live (host build)
control-plane head-of-line block — a slow exec start stalls sibling RPCs on the shared event loop Offload ManagedProcess.start's blocking pipe reads off the gRPC event loop (DispatchQueue.global + continuation) vendored vminitd/ManagedProcess.swift (patch #8) in deployed -nucleic3
OOM cross-kill — a runaway session's OOM kills a random sibling Per-exec cgroup with memory.oom.group=1 so an OOM kills only the offending session's process tree vendored vminitd/… (patch #9) deployed in -nucleic3 (runtime validation pending)
CPU starvation / fork bomb Per-exec cpu.weight (fair share) + pids.max (backstop) vendored vminitd/… (patch #9) deployed in -nucleic3 (runtime validation pending)
one session eating all memory before its own OOM Opt-in host-configured per-exec memory.max (rides the exec's OCI spec — no protobuf change) vendored host + guest + ContainerManager/ContainerEngine; setting controlPerSessionMemoryGiB deployed in -nucleic3 (runtime validation pending)
control-plane black hole — one bad accept/connect permanently kills the shared control socket's relay while it stays bound; every session then hangs in MCP init ("produced no output within 60s") until an app restart Accept loops tolerate transient accept(2) errors (#10); host relay contains per-connection failures + closes both ends on failure (#11); guest VsockProxy cleanup is leak-proof/once-guarded, registrations aren't try!, and a dead accept loop closes its listener so peers fail fast (#12). Backstops: listen backlog 16→SOMAXCONN, the in-guest bridge is supervised (restart loop), and the spawn-stall watchdog now probes the control plane from inside the guest and auto-recreates the container when the probe confirms it dead vendored Socket.swift / UnixSocketRelay.swift / vminitd/VsockProxy.swift; MCPApprovalServer / ContainerEngine / ContainerManager / ClaudeCodeBackend host halves + probe/recovery live (host build); guest halves deployed in -nucleic3 (runtime validation pending)

Note the connection-leak finding also settled a design question: per-exec control channels already exist — every LinuxContainer.exec() dials its own agent connection. The "one shared channel per container" concern applies only to the container's init/lifecycle agent, not to session runs.

2. Two shipping surfaces: host vs guest

  • Host-side patches compile into the app via a normal swift build (the Containerization library) — they take effect the moment the code is in dev, no image work. This is the stdio fixes, the connection-leak fix, the bounded teardown, and the recoverable-error mapping.
  • Guest-side patches live in third_party/containerization/vminitd/ (the guest agent, PID 1). They ride the vminit initfs OCI image and are inert until that image is rebuilt and published and ContainerEngine.vminitReference points at it. This is the offload (#8) and the cgroups work (#9). The materialized initfs (…/Nucleic/containers/initfs/vminit.ext4) is cached keyed on the image reference (a vminit.ext4.reference sidecar): repointing vminitReference re-pulls on next launch. (Before the sidecar existed the first-ever materialized initfs was trusted forever, so a repoint silently kept booting the old guest — verify a rollout actually landed by checking the sidecar's contents.) The pull authenticates with the app's GitHub token when the image is on ghcr.io, so the package can stay private; if the pull fails and a cached initfs exists, the cache is used and the pull retries next launch.

3. The per-exec cgroup layout (#9)

cgroup v2's "no internal process" rule means a cgroup can hold processes or delegate controllers to children, not both. So the container cgroup is restructured into an intermediary:

/container/<id>            ← intermediary: resource ceiling (memory.max) + subtree_control delegated
├── init                   ← leaf: the container init (sleep infinity + control-bridge)
├── exec-aaaa              ← leaf: session A's agent   — memory.oom.group=1, cpu.weight, pids.max, [memory.max]
├── exec-bbbb              ← leaf: session B's agent   — …
└── …
  • ManagedContainer.init builds this: creates /container/<id>, creates /container/<id>/init, enables cgroup.subtree_control up the chain (by enabling controllers from the init leaf), applies the container ceiling to the parent, and repoints the init's cgroupsPath at the leaf.
  • ManagedProcess.start places each exec in its own /container/<id>/<execID> and sets memory.oom.group=1, cpu.weight=100, pids.max=4096, and (if a host limit was passed) memory.max.
  • Graceful fallback: if any per-exec setup step fails, ManagedContainer wipes the partial state and reverts to the upstream flat layout (init + all execs share /container/<id>), and each exec independently falls back to the container cgroup. execCgroupParent == nil marks flat mode. A cgroup hiccup therefore degrades to today's behavior — it can never block a container from starting. This is the key safety property given the change is compile-verified but not yet runtime-validated.

New Cgroup2Manager helpers: setOomGroup / setCpuWeight / setPidsMax / setMemoryMax / remove.

4. Hard per-exec memory.max (opt-in follow-up)

memory.oom.group scopes an OOM to the offending session, but without a cap a session can still consume the whole container before that OOM. The hard cap closes that — and needs no protobuf change, because the exec already ships the full OCI Spec and the guest was just ignoring linux.resources:

  • Host: LinuxProcessConfiguration.memoryLimitInBytes → LinuxContainer.exec stamps it onto spec.linux.resources.memory.limit.
  • Guest: Server+GRPC.createProcess reads it back and applies it as the exec cgroup's memory.max (ManagedProcess / Cgroup2Manager.setMemoryMax).
  • Nucleic: ContainerServiceSettings.controlPerSessionMemoryGiB — default 0 = off. When > 0 it applies (via ContainerManager.exec) only to the shared control container (per-session sandbox containers already run one session each). Off by default preserves #9's generous scoped-OOM behavior.

5. Building & validating the custom vminit image

The guest image is built locally, not in CI: make containerization compiles the host framework, which uses macOS 26+ Virtualization/vmnet symbols that GitHub-hosted runners' SDKs lack (and the app targets macOS 27). The content/image store is user-global (~/Library/Application Support/com.apple.containerization/), so the built image is reachable from any checkout.

Root-Makefile targets:

make vminit-image-prep     # ONE-TIME per machine: swiftly + the aarch64 musl static SDK (~1 GB)
make vminit-image          # build cctl (Xcode Swift) + cross-build vminitd/vmexec (musl) → package the image
make vminit-image-login    # ONE-TIME: cctl login → GHCR token stored in the macOS Keychain (no env vars)
make vminit-image-push     # push to ghcr.io/abkslm/vminit (auth from the Keychain)

(The targets force WARNINGS_AS_ERRORS=false — Xcode's Swift 6.4 rejects -warnings-as-errors alongside SwiftPM's -suppress-warnings on dependencies. cctl is built on demand so the targets work from a fresh checkout.)

Tag scheme & the current-vs-next invariant:

  • -nucleicN is our guest-patch revision (0.34.0 = the upstream vminit version we replace).
  • Makefile: NUCLEIC_VMINIT_REF = the tag you build (the next image).
  • ContainerEngine.vminitReference = the currently deployed, validated tag.
  • On a new guest patch: bump NUCLEIC_VMINIT_REF, make vminit-image + make vminit-image-push, validate, then set vminitReference to match. So NUCLEIC_VMINIT_REF may be one revision ahead while a build is validated. Currently both are on -nucleic3 (#9 + the #10/#12 control-plane hardening), deployed ahead of full validation by explicit decision — revert vminitReference to -nucleic1 if it misbehaves.

Validation gate (before pointing a shipping build at a new guest image): boot a real Control session and confirm (a) sessions start normally, (b) /sys/fs/cgroup/container/<id>/<execID> exists per session, and (c) — with controlPerSessionMemoryGiB set — a memory hog is capped/killed within its own session and siblings survive. If good, bump vminitReference; if not, the graceful fallback means it's still running (just flat) and you stay on the previous tag.

6. Re-vendoring note

The Apple containerization framework is vendored at a pinned commit with Nucleic patches #1–#12. When re-vendoring a newer upstream, re-apply every [Nucleic vendored patch] site (grep for that marker) per PATCHES.md, then rebuild + publish the image and bump vminitReference.

7. Open follow-up

  • A host-tunable per-exec cpu.max (hard CPU quota) and pids.max would be a small extension of the same OCI-spec-riding mechanism used for memory.max in §4.
  • The controlPerSessionMemoryGiB policy is a flat GiB cap; a fraction-of-container or concurrency-aware policy could be layered on if static caps prove too blunt.
Reference in New Issue View Git Blame Copy Permalink