nucleic/docs/VSOCK_CONTROL_PLANE.md

vsock control plane

Moving the container control plane off TCP/HTTP onto vsock, so macOS stops raising incoming-connection (Application Firewall) and Local Network ("network discovery") prompts when a sandboxed agent talks to the host.

Why

The agent's stdio (Channel 1 — the stream-json firehose) already rides virtio-vsock via the containerization framework and is silent. The control plane (Channel 2) does not: it runs over HTTP/TCP on the vmnet NAT gateway, which is the AF_INET path — the host binds an NWListener on the vmnet subnet and the guest connects to it over IP. That is exactly what trips the two macOS prompts. virtio-vsock is a VM device channel handled by Virtualization.framework, not the host IP stack, so moving the control plane onto it raises no prompts.

Egress (the agent's outbound internet — Anthropic API, git, npm, gh) stays on vmnet NAT; see §7.

Channels

	Transport	Carries	Status
stdio	virtio-vsock	stream-json / NDJSON prompt	already vsock — untouched
control plane	unix socket relayed over vsock (via in-guest loopback bridge)	approvals (MCP), git/gh/command interceptor events	mandatory (2026-07-09) for every containerized run — shared AND per-session; the toggle and the legacy gateway-TCP path are retired
egress	vmnet NAT (gateway)	agent's outbound internet — Anthropic API, git, npm, gh + DNS	stays on NAT by design (§7) — can't ride vsock

Current state of Channel 2 (before this work)

• Approval MCP server — MCPApprovalServer (Sources/NucleicCore/Claude/MCPApprovalServer.swift) bound an NWListener to 0.0.0.0 when sandboxed; the agent got --mcp-config → http://<gateway>:<port>/mcp + Authorization: Bearer <per-session-token>. Routes: /mcp (JSON-RPC: approve, check_conflict, host_exec).
• Interceptor hooks — in-container git/gh/command shims POST to http://<gateway>:<port>/{git,gh,command}-event; endpoint URLs + token injected as env (NUCLEIC_GIT_HOOK_URL …) wherever a backend launches an agent in a container.
• Networking — VmnetNetwork(VMNET_SHARED_MODE) NAT; host at the subnet gateway .1; DNS at the gateway.

Target architecture

• The host serves the control plane on a per-container unix domain socket, relayed into the guest over vsock with the framework's first-class UnixSocketConfiguration(.into) (LinuxContainer.Configuration.sockets). No host IP listener.
• The socket is per container, not per session: Nucleic Control runs sessions as execs inside a shared control container (ContainerManager.sharedControlContainerName and the per-family split-outs), so the relay is attached once at container create. The single MCPApprovalServer already multiplexes sessions by bearer token (handlers are token-keyed — "one server can serve all sessions"), which is exactly the disambiguator a shared socket needs.
• The agent reaches the socket via a stdio-MCP bridge or an in-guest loopback HTTP shim (§3).
• Interceptor shims write to the same in-guest socket (§4).
• Per-session bearer token preserved (§5).

Findings / hard constraints (discovered while implementing)

1. NWListener with a .unix endpoint is in-process-only. Network.framework matches a local .unix endpoint within the process and never materializes an on-disk socket (verified: stat/fileExists both fail, yet an in-process NWConnection still "connects"). The framework's host-side relay is an ordinary socket client, so the host control socket must be a real BSD AF_UNIX listener (socket/bind/listen + an accept loop), not NWListener(.unix).
2. AF_UNIX path cap ≈ 104 bytes (sun_path on macOS). The per-container socket must live at a short path — e.g. /tmp/nuc-<id>.sock — never under ~/Library/Application Support/Nucleic/sessions/<UUID>/…, which overflows. start(unixSocketPath:) throws a clear error if the path is too long; the caller (the container-socket path convention) owns choosing a short, container-stable path.

Implementation

1. Host: UDS transport for the control server — DONE

MCPApprovalServer gained start(unixSocketPath:) backed by a real AF_UNIX listener (socket/bind/listen + a DispatchSourceRead accept loop). The connection-serving pipeline was refactored onto a transport-agnostic ByteConn abstraction with two implementations — NWByteConn (TCP, unchanged) and UnixSocketByteConn (blocking POSIX read/write offloaded to a background queue, so a turn suspended on an approval holds no thread). stop() cancels the accept source and unlinks the socket. The HTTP/JSON-RPC dispatch is byte-for-byte the same on both transports. TCP path and public API unchanged — purely additive.

Tested: MCPApprovalServerTests.unixSocketTransportServesMCPAndInterceptorRoutes — a raw BSD client completes MCP initialize, a 401 auth rejection, and a /git-event post over the socket, and asserts a real on-disk S_ISSOCK node.

2. Attach the relay to the container — DONE

ContainerSpec gained controlSocketHostPath: String? (the host socket; nil → legacy TCP path) and static let controlSocketGuestPath = "/run/nucleic/control.sock" (fixed guest mount point). ContainerEngine.ensureRunning's config closure sets config.sockets = [UnixSocketConfiguration(source: <host>, destination: controlSocketGuestPath, direction: .into)] when the path is set; ContainerSpec.renamed carries it through the physical-name rename. The host side is dialed lazily per guest connection, so the server only needs to be listening before the agent execs — not at container create.

Tested: ContainerSandboxTests.controlSocketRelayPlumbing (default-off, survives renamed, guest path constant).

Open detail for step 5: the relay's guest-socket permissions is left default for now; set it so the non-root agent uid can connect when the bridge lands.

2b. Shared per-container approval server — DONE

ApprovalServerRegistry (Sources/NucleicCore/Claude/ApprovalServerRegistry.swift) vends one MCPApprovalServer per control-container logical name. A run whose container is a shared control container (ContainerManager.allSharedControlContainerNames) now resolves its server from the registry instead of a per-backend instance; host runs and per-session sandboxes keep the per-backend server. The server is already share-safe — start() is idempotent, dispatch is token-keyed, and teardown only unregisters the run's token (never stops the server), so it persists across container restarts (balloon stop+recreate). The bearer token disambiguates sessions on the shared socket. The registry server is long-lived (app lifetime / explicit teardown), which is correct: it must outlive container recreation so the relay re-attaches to the same still-listening host socket.

ApprovalServerRegistry.controlSocketPath(for:) defines the short, container-stable host path (/tmp/nucleic/<name>.sock) the relay's source and the server's UDS bind both use.

The gateway-TCP transport and mcpConfig are unchanged — converging to the shared server is a correctness-neutral refactor (no agent-visible change). The actual UDS repoint (set controlSocketHostPath, start the shared server on the UDS, flip the agent's MCP config to the guest socket) is inseparable from the bridge and lands in §3, where it is testable end to end.

Tested: MCPApprovalServerTests.registrySharesOneServerPerContainer / controlSocketPathIsShortAndStable.

3. Agent + interceptors → server, and the UDS repoint — DONE (behind a flag)

Implemented with an in-guest loopback bridge: containers/nucleic-sandbox/control-bridge.js is a tiny Node forwarder that listens on 127.0.0.1:<ContainerSpec.controlBridgePort> (loopback, inside the VM netns — invisible to macOS) and pipes each connection to the relayed unix socket (ContainerSpec.controlSocketGuestPath). The container's root init launches it only when a control socket is relayed in (ContainerEngine sets init to sh -c "node /opt/nucleic/control-bridge.js & exec sleep infinity"), so it lives exactly as long as the container.

Why a root loopback forwarder rather than direct-UDS clients: the bridge runs as the container's root init child, so it reaches the root-owned relayed socket, while the non-root agent and the interceptor shims only ever touch loopback TCP — no socket-permission juggling, and (crucially) the git/gh/command shims need no change: they keep POSTing HTTP, now to the loopback bridge.

Host side (ClaudeCodeBackend): the run's container always carries a controlSocketHostPath, so the server is started on the UDS only — no IP listener at all — and the control endpoint handed to the agent + shims becomes 127.0.0.1:<bridge port> (mcpConfig + the NUCLEIC_*_HOOK_URL env). A containerized spec without a control socket fails the run loudly (the invariant is enforced, not silently degraded).

4. Wiring — DONE; MANDATORY as of 2026-07-09 (was: gated, then default-on with image v4)

The gate is gone: ContainerServiceSettings.vsockControlPlaneEnabled and its defaults key were retired (the Settings toggle had already been removed). SessionController.containerSpec() sets controlSocketHostPath = ApprovalServerRegistry.controlSocketPath(for: name) for every containerized spec — shared control containers AND per-session sandbox containers; everything downstream keys off that one field (relay attach, init bridge, UDS-only server, loopback endpoint). A shared control container meets the long-lived registry server; a per-session container's socket is served by its backend's own per-backend server (stopped — and the socket unlinked — at backend shutdown). Because the path requires control-bridge.js + node in the image (sandbox image ≥ v4), ContainerEngine probes for both once per fresh clone and fails the start with an actionable error when a custom image lacks them.

5. Auth — unchanged

The per-session bearer token is unchanged: it still rides the HTTP Authorization header (now over loopback → bridge → UDS), validated host-side exactly as before, and still disambiguates sessions on the shared per-container socket.

6. Remove the IP listener — DONE

For every containerized run the server never binds TCP (start(unixSocketPath:) only), so there is no control-plane NWListener and mcpConfig carries no gateway host/port — the macOS incoming-connection / local-network prompts have nothing to fire on. The legacy gateway-TCP fallback is deleted; ClaudeCodeBackend refuses a containerized run whose spec somehow lacks a control socket (fail-loud, never silently degrade to an endpoint the guest can't reach). TCP survives only as the host (non-container) runs' loopback listener.

6b. Grok & Codex control containers — PARTIAL (behind the flag)

The container infra is backend-agnostic (registry keyed by name, the nucleic-control-codex / nucleic-control-xai names in allSharedControlContainerNames, the bridge/relay keyed off controlSocketHostPath), so the per-family control containers were a backend-side gap, not an infra gap. Done this pass:

• GrokACPBackend / CodexAppServerBackend gained a container-exec path: when a run carries a vsock control socket (now: every containerized run) and a ContainerManager is present, the agent execs inside the shared control container (stdio over vsock) instead of on the host — i.e. each agent family runs isolated in its own box ("agenticide" separation). Gated on that one signal, so default behavior is unchanged. Both backends' approvals are native over their own stdio (already vsock when containerized), so they need no host control server.
• CommandInterceptor.hookEnv(...) centralizes the interceptor endpoint env so every containerized backend wires it identically; ClaudeCodeBackend now uses it.
• Wired through NucleicApp (the shared ContainerManager is handed to the Grok + Codex backends).

Done for Grok/Codex:

• Image binaries (v4): containers/nucleic-sandbox/Dockerfile installs Codex (npm i -g @openai/codex) and Grok (curl -fsSL https://x.ai/cli/install.sh | bash). The Grok installer drops its toolchain under /root/.grok and symlinks /usr/local/bin/grok there — unreachable by the non-root agent (/root is 0700), and the original re-link made a self-referential symlink ("Too many levels of symbolic links") that silently failed CI, which is why the registry image was stuck on the pre-bridge build. Fixed by relocating the install to a world-traversable /opt/grok and relinking; a grok --version build-time check still fails loudly if the arm64 binary didn't land.
• Auth seeding: SessionController.containerSpec() seeds per backend. Both store auth as plain files (no Keychain), so seedAgentHome copies the host's ~/.codex / ~/.grok into the per-session writable home — and since the container's $HOME is that dir, the agents find $HOME/.codex / $HOME/.grok automatically (no config-dir env needed). API-key users are covered by forwarding OPENAI_API_KEY / XAI_API_KEY / GROK_CODE_XAI_API_KEY (non-blank) into the exec env.
• Grok interceptor observation: GrokACPBackend now resolves the shared per-container server, starts it on the relayed socket, registers the git/gh/command report handlers (→ its conflictCoordinator), and injects CommandInterceptor.hookEnv — so a Grok control session gets the same git/gh/command observation (locks + autoship) as Claude. Its approvals stay native ACP (no MCP), so only the report routes are registered.
• Codex conflict/lock integration: CodexAppServerBackend now joins the lock/arbitration system. Acquire: a item/fileChange/requestApproval arbitrates on the edited paths before the patch applies (deny on deferred/cancelled/re-ground, ahead of the always-rule cache) — the approval carries only the item id, so paths are captured from the fileChange item lifecycle (CodexAppServerDecoder.fileChangeItemPaths). Release/observe: the same shared-server + interceptor wiring as Grok. Codex's approvals stay native (no MCP), so only the report routes register.
• CodexExecBackend (unattended Codex) execs in its control container with the git/gh/command interceptor wired (observe/release → conflictCoordinator). It runs --ask-for-approval never, so there is no acquire seam — it joins on observe/release only (merges → release + autoship + the activity feed), not preventive lock acquisition. So all four backends (Claude, Grok, Codex app-server, Codex exec) are on the control-container + interceptor path.

Still pending for Grok/Codex:

• Verify on hardware: the Grok installer's arm64 install path, codex app-server auth from a seeded auth.json, and the Codex fileChange-path/arbitration timing need a real container-build + run check (part of the §6b rollout; Codex wire shapes are still synthetic-fixture-confirmed pending a live capture).

7. Egress (verify before doing more) — TODO

vmnet NAT stays for the agent's outbound internet. After §1–6, verify the firewall + Local Network prompts are gone with vmnet still present (egress is outbound + framework-managed, so it should not re-trigger them). Only if a Local Network prompt persists, tunnel egress over vsock too (in-guest HTTPS_PROXY → vsock → host forward proxy) and drop the IP interface — larger effort (DNS, non-HTTP protocols); do not start unless needed.

Now mandatory (2026-07-09; default since image v4)

Promoted to default once the bridge-bearing image existed:

1. nucleic-sandbox:v4 — built + pushed by CI (.github/workflows/sandbox-image.yml, IMAGE_TAG: v4), carrying control-bridge.js (+ the Codex/Grok CLIs). The earlier tags were never bumped precisely because bumping ahead of a published image makes every control session pull a non-existent tag.
2. ProjectSandbox.defaultImage bumped to :v4; the launch-time rootfs prune (reconcileDisk/pruneObsoleteRootfs) drops the stale v3 cache and pulls v4 fresh — no manual re-pull needed.
3. Then made mandatory: the Settings toggle, the nucleic.container.vsockControlPlane defaults key, and the legacy gateway-TCP fallback are all retired. Every containerized run — the shared control containers and per-session sandbox containers alike — rides the relayed socket; custom images must provide node + control-bridge.js (enforced by the engine's fresh-clone preflight).

Manual verification checklist (first real-hardware run)

• The control session works end to end: approval round-trip, AskUserQuestion, host_exec, and a git/gh interceptor event all land.
• No macOS prompt — neither "accept incoming network connections" nor Local Network — on a clean machine (reset with tccutil reset / a fresh user if needed).
• In the container: node /opt/nucleic/control-bridge.js is running under init (PID 1 child) and ss -tlnp shows it listening on 127.0.0.1:9099; /run/nucleic/control.sock exists.
• On the host: lsof -nP -iTCP -sTCP:LISTEN shows no new control-plane port for the session; /tmp/nucleic/<container>.sock exists and is a socket.
• Flag off (or old image): behavior is byte-for-byte the legacy gateway-TCP path.

Testing / verification (automated)

• Unit (done): MCPApprovalServer over a raw AF_UNIX client (unixSocketTransportServes…); ApprovalServerRegistry sharing + short-path (registrySharesOneServerPerContainer, controlSocketPathIsShortAndStable); ContainerSpec relay plumbing (controlSocketRelayPlumbing).
• Not auto-testable here (no VM / CI image): the in-guest bridge, the conditional init, and the end-to-end relay — covered by the manual checklist above.
• Regression: transcript/emission behavior unchanged (stdio path untouched); approval suspend/resume preserved; flag-off path unchanged.

Caveats / ordering

• The per-container socket must be a short path (≤ ~103 bytes) and container-stable (same path for every session of a shared control container), so the single token-multiplexed server is reachable.
• Ship nucleic-bridge (or the loopback shim) in containers/nucleic-sandbox/Dockerfile and bump ProjectSandbox.defaultImage.
• The control-plane mechanism is shared by all containerized backends — keep the host socket, relay attachment, and hook-env injection backend-agnostic so additional containerized backends pick it up without a parallel implementation.
• A persistent UDS connection holds an outstanding JSON-RPC request naturally. MCP_TIMEOUT (the initial connection) is now bounded to 60s so an unreachable server can't wedge startup; only MCP_TOOL_TIMEOUT stays pinned high, because a gated tool call legitimately suspends until a human answers.