# vsock control plane

Moving the container **control plane** off TCP/HTTP onto vsock, so macOS stops raising
incoming-connection (Application Firewall) and Local Network ("network discovery") prompts when a
sandboxed agent talks to the host.

## Why

The agent's **stdio** (Channel 1 — the `stream-json` firehose) already rides virtio-vsock via the
containerization framework and is silent. The **control plane** (Channel 2) does not: it runs over
HTTP/TCP on the vmnet NAT gateway, which is the *AF_INET path* — the host binds an `NWListener` on
the vmnet subnet and the guest connects to it over IP. That is exactly what trips the two macOS
prompts. virtio-vsock is a VM device channel handled by Virtualization.framework, not the host IP
stack, so moving the control plane onto it raises no prompts.

Egress (the agent's outbound internet — Anthropic API, `git`, `npm`, `gh`) stays on vmnet NAT; see
§7.

## Channels

| | Transport | Carries | Status |
|---|---|---|---|
| stdio | virtio-vsock | `stream-json` / NDJSON prompt | already vsock — untouched |
| control plane | unix socket relayed over vsock (via in-guest loopback bridge) | approvals (MCP), git/gh/command interceptor events | **default** as of sandbox image `v4`; legacy gateway-TCP kept behind the off switch |
| egress | vmnet NAT (gateway) | agent's outbound internet — Anthropic API, `git`, `npm`, `gh` + DNS | stays on NAT by design (§7) — can't ride vsock |

## Current state of Channel 2 (before this work)

- **Approval MCP server** — `MCPApprovalServer` (`Sources/NucleicCore/Claude/MCPApprovalServer.swift`)
  bound an `NWListener` to `0.0.0.0` when sandboxed; the agent got `--mcp-config` →
  `http://<gateway>:<port>/mcp` + `Authorization: Bearer <per-session-token>`. Routes: `/mcp`
  (JSON-RPC: `approve`, `check_conflict`, `host_exec`).
- **Interceptor hooks** — in-container `git`/`gh`/command shims `POST` to
  `http://<gateway>:<port>/{git,gh,command}-event`; endpoint URLs + token injected as env
  (`NUCLEIC_GIT_HOOK_URL` …) wherever a backend launches an agent in a container.
- **Networking** — `VmnetNetwork(VMNET_SHARED_MODE)` NAT; host at the subnet gateway `.1`; DNS at
  the gateway.

## Target architecture

- The host serves the control plane on a **per-container unix domain socket**, relayed into the
  guest over vsock with the framework's first-class
  `UnixSocketConfiguration(.into)` (`LinuxContainer.Configuration.sockets`). No host IP listener.
- The socket is **per container, not per session**: Nucleic Control runs sessions as `exec`s inside
  a **shared** control container (`ContainerManager.sharedControlContainerName` and the per-family
  split-outs), so the relay is attached once at container create. The single
  `MCPApprovalServer` already multiplexes sessions by **bearer token** (handlers are token-keyed —
  "one server can serve all sessions"), which is exactly the disambiguator a shared socket needs.
- The agent reaches the socket via a stdio-MCP bridge or an in-guest loopback HTTP shim (§3).
- Interceptor shims write to the same in-guest socket (§4).
- Per-session bearer token preserved (§5).

## Findings / hard constraints (discovered while implementing)

1. **`NWListener` with a `.unix` endpoint is in-process-only.** Network.framework matches a local
   `.unix` endpoint *within the process* and never materializes an on-disk socket (verified:
   `stat`/`fileExists` both fail, yet an in-process `NWConnection` still "connects"). The framework's
   host-side relay is an **ordinary socket client**, so the host control socket **must** be a real
   BSD `AF_UNIX` listener (`socket`/`bind`/`listen` + an accept loop), not `NWListener(.unix)`.
2. **`AF_UNIX` path cap ≈ 104 bytes** (`sun_path` on macOS). The per-container socket must live at a
   **short** path — e.g. `/tmp/nuc-<id>.sock` — never under
   `~/Library/Application Support/Nucleic/sessions/<UUID>/…`, which overflows. `start(unixSocketPath:)`
   throws a clear error if the path is too long; the caller (the container-socket path convention)
   owns choosing a short, container-stable path.

## Implementation

### 1. Host: UDS transport for the control server — DONE

`MCPApprovalServer` gained `start(unixSocketPath:)` backed by a real `AF_UNIX` listener
(`socket`/`bind`/`listen` + a `DispatchSourceRead` accept loop). The connection-serving pipeline was
refactored onto a transport-agnostic `ByteConn` abstraction with two implementations — `NWByteConn`
(TCP, unchanged) and `UnixSocketByteConn` (blocking POSIX read/write offloaded to a background
queue, so a turn suspended on an approval holds no thread). `stop()` cancels the accept source and
unlinks the socket. The HTTP/JSON-RPC dispatch is byte-for-byte the same on both transports. TCP path
and public API unchanged — purely additive.

Tested: `MCPApprovalServerTests.unixSocketTransportServesMCPAndInterceptorRoutes` — a **raw BSD
client** completes MCP `initialize`, a 401 auth rejection, and a `/git-event` post over the socket,
and asserts a real on-disk `S_ISSOCK` node.

### 2. Attach the relay to the container — DONE

`ContainerSpec` gained `controlSocketHostPath: String?` (the host socket; `nil` → legacy TCP path)
and `static let controlSocketGuestPath = "/run/nucleic/control.sock"` (fixed guest mount point).
`ContainerEngine.ensureRunning`'s config closure sets
`config.sockets = [UnixSocketConfiguration(source: <host>, destination: controlSocketGuestPath,
direction: .into)]` when the path is set; `ContainerSpec.renamed` carries it through the
physical-name rename. The host side is dialed **lazily per guest connection**, so the server only
needs to be listening before the agent execs — not at container create.

Tested: `ContainerSandboxTests.controlSocketRelayPlumbing` (default-off, survives `renamed`, guest
path constant).

> Open detail for step 5: the relay's guest-socket `permissions` is left default for now; set it so
> the non-root agent uid can connect when the bridge lands.

### 2b. Shared per-container approval server — DONE

`ApprovalServerRegistry` (`Sources/NucleicCore/Claude/ApprovalServerRegistry.swift`) vends one
`MCPApprovalServer` per control-container logical name. A run whose container is a shared control
container (`ContainerManager.allSharedControlContainerNames`) now resolves its server from the
registry instead of a per-backend instance; host runs and per-session sandboxes keep the per-backend
server. The server is already share-safe — `start()` is idempotent, dispatch is token-keyed, and
teardown only **unregisters the run's token** (never stops the server), so it persists across
container restarts (balloon stop+recreate). The bearer token disambiguates sessions on the shared
socket. The registry server is long-lived (app lifetime / explicit teardown), which is correct: it
must outlive container recreation so the relay re-attaches to the same still-listening host socket.

`ApprovalServerRegistry.controlSocketPath(for:)` defines the short, container-stable host path
(`/tmp/nucleic/<name>.sock`) the relay's `source` and the server's UDS bind both use.

The **gateway-TCP transport and `mcpConfig` are unchanged** — converging to the shared server is a
correctness-neutral refactor (no agent-visible change). The actual UDS repoint (set
`controlSocketHostPath`, start the shared server on the UDS, flip the agent's MCP config to the guest
socket) is inseparable from the bridge and lands in §3, where it is testable end to end.

Tested: `MCPApprovalServerTests.registrySharesOneServerPerContainer` /
`controlSocketPathIsShortAndStable`.

### 3. Agent + interceptors → server, and the UDS repoint — DONE (behind a flag)

Implemented with an **in-guest loopback bridge**: `containers/nucleic-sandbox/control-bridge.js` is a
tiny Node forwarder that listens on `127.0.0.1:<ContainerSpec.controlBridgePort>` (loopback, inside
the VM netns — invisible to macOS) and pipes each connection to the relayed unix socket
(`ContainerSpec.controlSocketGuestPath`). The container's root init launches it **only when a control
socket is relayed in** (`ContainerEngine` sets init to
`sh -c "node /opt/nucleic/control-bridge.js & exec sleep infinity"`), so it lives exactly as long as
the container.

Why a root loopback forwarder rather than direct-UDS clients: the bridge runs as the container's root
init child, so it reaches the **root-owned** relayed socket, while the non-root agent and the
interceptor shims only ever touch **loopback TCP** — no socket-permission juggling, and (crucially)
**the git/gh/command shims need no change**: they keep POSTing HTTP, now to the loopback bridge.

Host side (`ClaudeCodeBackend`): when the run's container carries a `controlSocketHostPath`, the
(shared) server is started on the **UDS only — no IP listener at all** — and the control endpoint
handed to the agent + shims becomes `127.0.0.1:<bridge port>` (`mcpConfig` + the `NUCLEIC_*_HOOK_URL`
env). Otherwise the legacy gateway-TCP path is used unchanged.

### 4. Wiring + the gate — DONE; default ON as of image `v4`

One flag gates the whole vertical: `ContainerServiceSettings.vsockControlPlaneEnabled` (now **default
on** — `true` when unset, an explicit stored `false` still wins; Settings → "Control plane over
vsock"). `SessionController.containerSpec()` sets
`controlSocketHostPath = ApprovalServerRegistry.controlSocketPath(for: name)` only for a shared
control container when the flag is on; everything downstream keys off that one field (relay attach,
init bridge, UDS-only server, loopback endpoint). Flag off → byte-for-byte the legacy path. The
default was moved in lockstep with `ProjectSandbox.defaultImage` reaching a bridge-bearing tag (`v4`),
since the path requires `control-bridge.js` in the image.

### 5. Auth — unchanged

The per-session bearer token is unchanged: it still rides the HTTP `Authorization` header (now over
loopback → bridge → UDS), validated host-side exactly as before, and still disambiguates sessions on
the shared per-container socket.

### 6. Remove the IP listener — DONE on the flag path

On the flag-on (now default) path the server never binds TCP (`start(unixSocketPath:)` only), so there
is no control-plane `NWListener` and `mcpConfig` carries no gateway host/port — the macOS
incoming-connection / local-network prompts have nothing to fire on. The legacy TCP path is retained
behind an explicit `vsockControlPlane = false` for fallback; it can be deleted once the vsock path has
soaked on real hardware.

### 6b. Grok & Codex control containers — PARTIAL (behind the flag)

The container infra is backend-agnostic (registry keyed by name, the `nucleic-control-codex` /
`nucleic-control-xai` names in `allSharedControlContainerNames`, the bridge/relay keyed off
`controlSocketHostPath`), so the per-family control containers were a backend-side gap, not an infra
gap. Done this pass:

- **`GrokACPBackend` / `CodexAppServerBackend`** gained a container-exec path: when a run carries a
  vsock control socket (so: flag on + shared control container) and a `ContainerManager` is present,
  the agent execs **inside the shared control container** (stdio over vsock) instead of on the host —
  i.e. each agent family runs isolated in its own box ("agenticide" separation). Gated on that one
  signal, so default behavior is unchanged. Both backends' approvals are native over their own stdio
  (already vsock when containerized), so they need **no host control server**.
- **`CommandInterceptor.hookEnv(...)`** centralizes the interceptor endpoint env so every
  containerized backend wires it identically; `ClaudeCodeBackend` now uses it.
- Wired through `NucleicApp` (the shared `ContainerManager` is handed to the Grok + Codex backends).

**Done for Grok/Codex:**

- **Image binaries (`v4`):** `containers/nucleic-sandbox/Dockerfile` installs Codex (`npm i -g
  @openai/codex`) and Grok (`curl -fsSL https://x.ai/cli/install.sh | bash`). The Grok installer drops
  its toolchain under `/root/.grok` and symlinks `/usr/local/bin/grok` there — unreachable by the
  non-root agent (`/root` is `0700`), and the original re-link made a self-referential symlink ("Too
  many levels of symbolic links") that silently failed CI, which is why the registry image was stuck on
  the pre-bridge build. Fixed by relocating the install to a world-traversable `/opt/grok` and
  relinking; a `grok --version` build-time check still fails loudly if the arm64 binary didn't land.
- **Auth seeding:** `SessionController.containerSpec()` seeds per backend. Both store auth as plain
  files (no Keychain), so `seedAgentHome` copies the host's `~/.codex` / `~/.grok` into the per-session
  writable home — and since the container's `$HOME` is that dir, the agents find `$HOME/.codex` /
  `$HOME/.grok` automatically (no config-dir env needed). API-key users are covered by forwarding
  `OPENAI_API_KEY` / `XAI_API_KEY` / `GROK_CODE_XAI_API_KEY` (non-blank) into the exec env.
- **Grok interceptor observation:** `GrokACPBackend` now resolves the shared per-container server,
  starts it on the relayed socket, registers the git/gh/command report handlers (→ its
  `conflictCoordinator`), and injects `CommandInterceptor.hookEnv` — so a Grok control session gets the
  same git/gh/command observation (locks + autoship) as Claude. Its approvals stay native ACP (no MCP),
  so only the report routes are registered.
- **Codex conflict/lock integration:** `CodexAppServerBackend` now joins the lock/arbitration system.
  *Acquire:* a `item/fileChange/requestApproval` arbitrates on the edited paths before the patch
  applies (deny on deferred/cancelled/re-ground, ahead of the always-rule cache) — the approval carries
  only the item id, so paths are captured from the fileChange item lifecycle
  (`CodexAppServerDecoder.fileChangeItemPaths`). *Release/observe:* the same shared-server + interceptor
  wiring as Grok. Codex's approvals stay native (no MCP), so only the report routes register.
- **`CodexExecBackend`** (unattended Codex) execs in its control container with the git/gh/command
  interceptor wired (observe/release → `conflictCoordinator`). It runs `--ask-for-approval never`, so
  there is **no acquire seam** — it joins on observe/release only (merges → release + autoship + the
  activity feed), not preventive lock acquisition. So all four backends (Claude, Grok, Codex
  app-server, Codex exec) are on the control-container + interceptor path.

**Still pending for Grok/Codex:**
- **Verify on hardware:** the Grok installer's arm64 install path, `codex app-server` auth from a
  seeded `auth.json`, and the Codex fileChange-path/arbitration timing need a real container-build + run
  check (part of the §6b rollout; Codex wire shapes are still synthetic-fixture-confirmed pending a
  live capture).

### 7. Egress (verify before doing more) — TODO

vmnet NAT stays for the agent's outbound internet. After §1–6, verify the firewall + Local Network
prompts are gone with vmnet still present (egress is outbound + framework-managed, so it should not
re-trigger them). **Only if** a Local Network prompt persists, tunnel egress over vsock too (in-guest
`HTTPS_PROXY` → vsock → host forward proxy) and drop the IP interface — larger effort (DNS, non-HTTP
protocols); do not start unless needed.

## Now the default (image `v4`)

Promoted to default once the bridge-bearing image existed:

1. `nucleic-sandbox:v4` — built + pushed by CI (`.github/workflows/sandbox-image.yml`, `IMAGE_TAG: v4`),
   carrying `control-bridge.js` (+ the Codex/Grok CLIs). The earlier tags were never bumped precisely
   because bumping ahead of a published image makes every control session pull a non-existent tag.
2. `ProjectSandbox.defaultImage` bumped to `:v4`; the launch-time rootfs prune
   (`reconcileDisk`/`pruneObsoleteRootfs`) drops the stale `v3` cache and pulls `v4` fresh — no manual
   re-pull needed.
3. `vsockControlPlaneEnabled` defaults on, so **Settings → Container → "Control plane over vsock"** is
   on. Turn it off (or `defaults write … nucleic.container.vsockControlPlane -bool NO`) to fall back to
   the legacy gateway-TCP path — no rebuild needed.

The first Nucleic Control session on real hardware is the validation: run the checklist below. Once it
has soaked, the legacy TCP path (§6) can be deleted.

### Manual verification checklist (first real-hardware run)

- The control session works end to end: approval round-trip, `AskUserQuestion`, `host_exec`, and a
  `git`/`gh` interceptor event all land.
- **No macOS prompt** — neither "accept incoming network connections" nor Local Network — on a clean
  machine (reset with `tccutil reset` / a fresh user if needed).
- In the container: `node /opt/nucleic/control-bridge.js` is running under init (PID 1 child) and
  `ss -tlnp` shows it listening on `127.0.0.1:9099`; `/run/nucleic/control.sock` exists.
- On the host: `lsof -nP -iTCP -sTCP:LISTEN` shows **no** new control-plane port for the session;
  `/tmp/nucleic/<container>.sock` exists and is a socket.
- Flag **off** (or old image): behavior is byte-for-byte the legacy gateway-TCP path.

## Testing / verification (automated)

- **Unit (done):** `MCPApprovalServer` over a raw `AF_UNIX` client (`unixSocketTransportServes…`);
  `ApprovalServerRegistry` sharing + short-path (`registrySharesOneServerPerContainer`,
  `controlSocketPathIsShortAndStable`); `ContainerSpec` relay plumbing (`controlSocketRelayPlumbing`).
- **Not auto-testable here (no VM / CI image):** the in-guest bridge, the conditional init, and the
  end-to-end relay — covered by the manual checklist above.
- **Regression:** transcript/emission behavior unchanged (stdio path untouched); approval
  suspend/resume preserved; flag-off path unchanged.

## Caveats / ordering

- The per-container socket must be a **short** path (≤ ~103 bytes) and container-stable (same path for
  every session of a shared control container), so the single token-multiplexed server is reachable.
- Ship `nucleic-bridge` (or the loopback shim) in `containers/nucleic-sandbox/Dockerfile` and bump
  `ProjectSandbox.defaultImage`.
- The control-plane mechanism is shared by all containerized backends — keep the host socket, relay
  attachment, and hook-env injection backend-agnostic so additional containerized backends pick it up
  without a parallel implementation.
- A persistent UDS connection holds an outstanding JSON-RPC request naturally. `MCP_TIMEOUT` (the
  initial connection) is now bounded to 60s so an unreachable server can't wedge startup; only
  `MCP_TOOL_TIMEOUT` stays pinned high, because a gated tool call legitimately suspends until a human
  answers.
