Host — the two remaining app-wide stall mechanisms plus main-thread pins
found by mining all nine hang reports:
- LinuxProcess.startStdinRelay wrote to a BLOCKING stdin fd on a
width-limited cooperative-pool thread, non-cancellably; wedged guests
starved the whole concurrency runtime (decode loops, watchdogs — an
app-wide freeze surviving the reconcile fix). Writes now offload to a
per-process GCD queue (vendored patch #18).
- TranscriptWriter (actor) did blocking write/fsync on the cooperative
pool; it now runs on its own DispatchSerialQueue executor.
- UserMessageBubble's truncation probe typeset entire pasted-log-sized
messages through CoreText per layout pass (100% main-thread pins in
the 07-21 hang reports); certainly-long messages now skip the probe
and render a prefix while collapsed.
- toolGroupSignature JSON-encoded every tool input in the transcript up
to 12.5x/s on the MainActor; now a structural hash. The summary pass
is trailing-throttled to 0.4s, and flatItems joins streaming chunks
once instead of re-copying the prefix per delta.
- StatusFeedFetcher.parseDate allocated three formatters per call (86%
of a pool thread in the 07-26 report); now shared statics.
Guest (vminitd) — teardown data loss and epoll registration hazards:
- IOPair no longer closes on a bare EPOLLHUP with a backpressure flush
in flight (dropped the CLI's final output line); EPOLLOUT finishes the
flush, then EOF closes loss-free. ManagedProcess.setExit closes only
stdin, letting stdout/stderr self-close on EOF, with an 8s grace pass
(patch #16).
- Epoll events carry a registration generation; the supervisor ignores
stale events for recycled fd numbers. registerFd refuses EEXIST
instead of clobbering the existing handler. TerminalIO's stdin relay
writes a dup of the terminal fd so its backpressure registration
can't collide with the stdout relay's (patch #17).
- VsockProxy flushes bytes parked toward the surviving peer on hangup,
closes the dialing socket on a failed backend connect, and
StandardIO/TerminalIO clean up partially-created pairs on setup
failure (patch #16).
Full suite: 1451+292+74+20 tests, two failures — both pre-existing
environmental (MacVM base image absent on this machine; a load-flaky
liveness test that passes 3/3 in isolation).
Co-Authored-By: Claude Fable 5 <[email protected]>
Host (dominant): AppStore.reconcileLocks polls every 3s on the MainActor
while any lock is held — effectively forever, since interrupted/errored
sessions deliberately retain locks. Each pass ran heldPathDisposition's
diverges() as three held×unmerged scans with two split-allocations per
pathsOverlap call, pinning the main thread for tens of seconds per pass
on a diverged trunk (hang-reports 2026-07-28: 100% of samples in
reconcileLocks→heldPathDisposition→pathsOverlap). That froze running
sessions' transcripts and starved the spawn path into the 60s
"produced no output" watchdog. pathsOverlap is now allocation-free
bytewise comparison with identical semantics, and divergentHeldPaths
answers all three questions from one O((held+unmerged)·depth) set.
Guest (persistence): VsockProxy threaded ONE offset pair through BOTH
relay directions; once the EAGAIN-return backpressure patch let pending
bytes persist, traffic in the other direction skewed the shared
counters, made the write leg unreachable, and spun the single
ProcessSupervisor poller thread forever — container-wide dead control
plane until VM recreation, triggered by exactly the backpressure the
host hang created. Each direction now owns its own pipe and counters
(OSFile.RelayDirection), and source EOF is only surfaced after the
pipe drains so SHUT_WR can't truncate a parked tail. Vendored patch
docs updated (#15); inert until the initfs image is rebuilt+repointed.
Co-Authored-By: Claude Fable 5 <[email protected]>
Overview doc so other agents/humans understand the shared-control-container
isolation work: the failure vectors + fixes (stdio wedge, connection leak,
control-plane HOL, OOM cross-kill, CPU/fork-bomb, per-session memory.max), the
host-vs-guest shipping surfaces, the per-exec cgroup layout + graceful fallback,
and the local vminit-image build/validate workflow + -nucleicN tag invariant.
Cross-linked from the vendored PATCHES.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds an opt-in hard per-session memory ceiling on top of patch #9's scoped-OOM.
The exec already ships the full OCI Spec, so the limit rides
spec.linux.resources.memory.limit — no RPC/protobuf change:
- host framework: LinuxProcessConfiguration.memoryLimitInBytes; LinuxContainer.exec
stamps it onto the exec spec.
- guest: Server+GRPC.createProcess reads it back and applies it as the exec
cgroup's memory.max (new Cgroup2Manager.setMemoryMax) via createExec/ManagedProcess.
- Nucleic: ContainerServiceSettings.controlPerSessionMemoryGiB (default 0 = off),
applied only to the shared control container (ContainerManager.exec); wired
through ContainerEngine.exec.
So one session can't consume the whole shared container's memory before its own
(oom.group-scoped) OOM. Default off preserves #9's behavior. Compile-verified host
+ musl guest; rides the pending -nucleic2 image, still runtime-pending.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Restructures the guest cgroup layout so each exec gets its OWN child cgroup
(/container/<id>/<execID>) with memory.oom.group=1, a fair cpu.weight, and a
pids.max backstop — so one control session can't OOM-kill, starve, or fork-bomb
its siblings in the shared container. The container init moves to its own leaf
so the container cgroup can delegate controllers to children (cgroup v2
no-internal-process rule). New Cgroup2Manager helpers: setOomGroup/setCpuWeight/
setPidsMax/remove.
Best-effort with graceful fallback: any failure in the per-exec setup wipes the
partial state and reverts to today's flat layout, and each exec falls back to the
container cgroup — a cgroup hiccup degrades to current behavior, never a failed
start.
COMPILE-VERIFIED via the musl cross-build; NOT yet runtime-validated. Built as
image tag -nucleic2; vminitReference stays on the validated -nucleic1 until
-nucleic2 is checked in a real container. A hard host-configured per-exec
memory.max (exec-RPC resources field) remains a follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
cctl login stores the credential in the macOS Keychain, which cctl images push
reads automatically — so pushing needs no REGISTRY_* env vars after a one-time
login. Env vars remain as a fallback.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The swiftly toolchain used by the vminit-image CI resolves Foundation/
Virtualization but not the 'os' overlay, so 'import os' failed with
"no such module 'os'". Guarding the diagnostic logger degrades it to a
no-op under such toolchains while local (Xcode) builds keep it.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Host-side (ships with a normal swift build):
- LinuxProcess: non-blocking stdio relay (O_NONBLOCK + nucleicDrainNonBlocking)
so a wedged stream can't head-of-line-block sibling execs' relays; atomic
stdio-or-abort start (patches #5, #6).
- Vminitd: bounded deleteProcess timeout so teardown can't hang a wedged
channel (patch #7).
- ContainerizedProcessHandle: call LinuxProcess.delete() after exit and on
force-close — fixes a per-turn leak (per-exec vsock/gRPC connection +
runConnections() task) in the long-lived shared control container. Likely
the "degrades until app restart" root cause.
- ClaudeCodeBackend: map the atomic-start abort to a recoverable AgentError so
a failed launch settles as retryable instead of locking the composer.
Guest-side (rides the custom vminitd initfs; inert until the image is built):
- ManagedProcess: offload the blocking start off the gRPC event loop (patch #8).
- Per-exec cgroups (patch #9) recorded as design only — cross-cutting.
Pipeline:
- .github/workflows/vminit-image.yml builds vminitd from the vendored source
and pushes ghcr.io/abkslm/vminit; ContainerEngine.vminitReference repointed
at the custom image.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Containerized control sessions could hang with no output ("Working…" forever): the agent's exec
stdio over the vminitd vsock channel intermittently failed to carry bytes, so claude ran and
reached the approval server but its stdin/stdout never connected — it idled in interactive
stream-json mode and never exited. Forensics (nucleic.sqlite + per-session claude-home MCP logs)
showed every stdio layer byte-identical to a working state, i.e. a flaky framework race, not a
regression in our code. Make the failure recoverable and visible instead of an eternal spinner:
- ClaudeCodeBackend: a 60s spawn watchdog on containerized runs — no first stdout → emit a
recoverable error (with guest stderr + container probe), SIGKILL the wedged process, and finish
the run errored, instead of awaiting stdoutLines forever.
- Stop reliability: ProcessHandle.forceCloseStreams() (ContainerizedProcessHandle finishes its
line streams host-side; default no-op for the host pipe handle), wired into every kill
escalation (terminate/interruptThenKill/killGroupAfter) so a force-killed run always settles
even when the guest wait/stdio RPC wedges — the real cause of "Stop is inconsistent".
- Cap MCP_TIMEOUT (connection) to 60s so an unreachable approval server can't wedge startup for
~24.8 days; MCP_TOOL_TIMEOUT stays unbounded for human-answered approvals.
- LinuxProcess.setupIO logs which stdio stream fails to connect (os.Logger, com.nucleic /
container-io) so a stall pinpoints the failing stream. Vendored patch #3.
- Tests: stop-escalation force-close, responsive-process no-op, MCP timeout asymmetry.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root-cause and fix the four reported container regressions plus two
adjacent confirmed bugs.
- Memory balloon (CPU 100% + output freeze that never recovered): the
autoballoon drove the whole-VM target from a per-container cgroup figure
with no guest swap, spinning a swapless guest in perpetual direct
reclaim. Default memoryManagement to off; make the target whole-VM-aware
(reserveBytes) so it never inflates below the working set plus the
guest's non-cgroup footprint; deflate the balloon on a failed stats read
instead of freezing it inflated.
- Stop button: signal the agent's whole process group (new vendored
LinuxProcess.killProcessGroup, negative pid) so forked children die too;
replace the unbounded wait() in every teardown/shutdown with a bounded
terminate() that escalates SIGTERM -> SIGKILL; interrupt escalates to a
group kill so a wedged agent always stops.
- MCPApprovalServer port-0 race: single-flight start(host:) so concurrent
sessions sharing one control-container server all receive the real bound
port; publish listener+port only after .ready (a failed bind no longer
pins a stale port 0); guard the Claude call site against port 0.
- Container CPU metric: divide the CPU delta by the actual measured window
instead of a fixed 200 ms, so it stops over-reading under load.
- Command interceptor: drop the ~40 coreutil Node shims so cat/grep/etc
run their native binaries (no Node-per-command); the bash tracer still
records them as metadata.
- Make the bash command tracer opt-in (commandTracingEnabled, default
off) — the per-command DEBUG trap only activates when enabled; the
git/gh interception the conflict/merge system relies on stays always-on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Switch the containerization dependency from the github URL to a vendored copy
(third_party/containerization, upstream commit 6b7b42ca) referenced by path, so
we can carry a small local patch that upstream lacks: LinuxContainer.Configuration
gains a `vmExtensions` field forwarded into VMConfiguration.extensions. Upstream
already supports VMConfiguration.extensions + the VZInstanceExtension hook, but
LinuxContainer — our only entry point — never forwarded them, so there was no way
to attach a device (e.g. a memory balloon) to a container's VM.
Tests/, docs/, examples/, images/ and the corresponding test targets are trimmed
for footprint (we never build the dependency's tests). See PATCHES.md for the full
diff vs. upstream and the re-vendoring procedure. Also adds the ContainerizationExtras
product to NucleicCore (AddressAllocator, named in the configureVZ signature).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
abkslmandClaude Fable 5