nucleic/nucleic
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b03dfb2c946702f8729eefc46b1d01c8c25c9a8b
nucleic/docs/COVALENCE_RUNNER.md
T

67 KiB
Raw Blame History

Nucleic — Covalence Cloud Runner

Status (2026-07-10): designed; foundation implemented and verified end to end — a Linux nucleicd (both arches, incl. Cloudflare's required linux/amd64) boots, enrolls with the deployed production relay, listens on it, and mints working pairing codes; the Cloudflare control plane is written and unit-tested; the wire-protocol extensions are landed and capability-gated. The dispatch gap is closed and the first real agent turns have run on a Linux runner: a fresh runner seeds projects at boot (NUCLEIC_RUNNER_PROJECTS) or over the wire (ClientMsg.createProject → HostMsg.projectCreated, gated on WireCapabilities.canCreateProjects, with the iOS Add Project sheet driving it), and real claude sessions — dispatched via startChat through the deployed production relay, streaming events back, with a destructive Bash call surfacing an approval that a remote client answered over the wire — completed end to end on Linux (nucleic-smoke, §11.1). Cloudflare container enablement (item 3) is code-complete: the loopback :9200 pairing endpoint is live (curl-verified), the presence→wake hook is built + tested on both Workers, and both image-publish CI workflows are authored (the runner image now publishes to Cloudflare's own managed registry, not GHCR). What remains is purely operational: set the CLOUDFLARE_API_TOKEN repo secret and run the two CI workflows (the containers block is now active, pinned to registry.cloudflare.com/<ACCOUNT_ID>/nucleic-runner:v1), then deploy with the secrets (RUNNER_TOKEN_SECRET, RUNNER_ADMIN_SECRET, RUNNER_WAKE_SECRET ×2 + RUNNER_WAKE_URL). Settings→pool (item 4) is complete: RunnerPoolClient PoP-enrolls, provisions, pushes the knobs, lists instances, pulls the one-shot pairing code — and the pool credential now rides the mesh to control-scope peers like relay memberships do. The credential mesh (item 5) is complete, both directions: a device (CredentialProvider) captures its Keychain credentials, gossips a manifest, and seals them to a runner answering credentialNeeded; the runner lands them 0600; and when a session's CLI refreshes the OAuth token, a digest-gated sweep mirrors the rotation back to each device (credentialUpdate), reconciled newest-wins into the Keychain. Proven E2E over the production relay (device→runner + a real authenticated claude turn) with the reverse-seal + change-gate unit-tested. All seven credential kinds now land (Claude/Codex logins, Grok config, GitHub token + Anthropic/OpenAI/xAI keys) via a table-driven vault, and Settings shows what a Mac shares. Next up: item 6 — intelligence delegation's device-side executor. §11 is the handoff brief for whoever picks this up. (2026-07-13) Production lifecycle fixes landed: the stale-host provision/wake wedge, the loopback-bind pairing failure inside Cloudflare Containers, and the missing auto-join-on-provision — plus heartbeat token adoption and image/Worker deploy CI. §11.2b is the postmortem. (2026-07-13) Zero-touch runners landed (auto-provision + scale-to-zero): manual provisioning is no longer required. The composer offers Covalence whenever a runner is live or Covalence Cloud is enabled (AppStore.meshDispatchPossible); selecting it with nothing running boots one automatically — dispatchChatToMesh calls AppStore.provisionAndJoinRunner (the provision→wait-ready→pull-pairing→joinMesh choreography, lifted out of RemoteAccessView and single-flighted) and the durable dispatch queue lands the parked chat once the runner publishes presence. A headless nucleicd now auto-approves the inbound pairing (AppStore.meshAutoApprovePairing, set in Nucleicd.swift) — it mints the one-shot code, so possession is the authorization and there is no operator to click the confirm; without this the runner declined the very device that provisioned it. The pool DO's alarm() now scales the host to zero when the whole pool is idle past idleTimeoutSeconds (no host sessions, no live sandbox); a later provision/wake or the relay-presence hook reboots it. Per-boot identity rotation leaves stale .cloud roster tombstones, so a successful join prunes older offline cloud peers (pruneStaleCloudRunnerPeers). Follow-up: the iOS composer still gates on a live runner (it dispatches through the Mac host, which owns the pool credential); originating a first provision from the phone needs a host-advertised "runner-enabled" capability.

The Covalence Cloud Runner is a headless Nucleic host a user can dispatch agents onto as if it were a macOS host. It joins the user's Covalence mesh as PeerKind.cloud, speaks the existing Noise/CBOR sync protocol through the deployed relay, and provides the same backend functionality the macOS app's backend provides. First deployment target is the Cloudflare developer platform (Workers + Durable Objects + Containers); self-hosting (plain container, or a spare Mac running nucleicd directly) is a stated goal and shapes every seam below.

Builds on CLOUD_RUNTIME.md (the nucleicd port thesis) and MESH_TRANSFER.md (the mesh the runner joins). Where CLOUD_RUNTIME is the business/platform plan, this doc is the runner's product + integration design: mesh membership, virtualization tiers, intelligence delegation, the credential mesh, and the runner fleet + Settings surface.

(2026-07-19) Pairing lifecycle hardened — re-mintable codes, orphan adoption, prune. The one-shot mint→serve→burn pairing endpoint composed badly with the fleet model: any consumer failure after the burn (a dropped proxy response, a failed join) dead-ended the runner in 410 "stop and provision again", and a wake-hook reboot (per-boot identity rotation, no device running the join choreography) produced a permanent orphan — "ready" in the pool, absent from the mesh, undispatchable, holding a slot. Changes: (1) nucleicd's GET /pairing now mints a fresh one-time code per call (each code is still single-use; the endpoint is only reachable through the pool-credential-gated DO proxy, and a credential holder could always provision a fresh runner anyway, so re-minting grants no new power); (2) heartbeats' roomId is now stored on the instance record and exposed in GET /v1/pool/status, so a device can tell a joined runner from an orphan; (3) provisionAndJoinRunner adopts (pairs + joins) a ready runner whose roomId isn't a paired device before booting another slot, and Settings flags such rows "not in mesh — Provision to link"; (4) new POST /v1/pool/prune (Worker) / POST /prune (DO) deletes stopped registry rows — Settings ▸ "Delete stopped" drives it. The 410 client path survives only for runners on older images and no longer claims success just because some other cloud peer is live. Follow-up, same day — the wake↔orphan loop is closed at both ends. (5) The pool's targeted wake (/wake {instanceId}, the relay-presence hook's path) now declines a dead/stale slot (identityRotated) instead of rebooting it: a reboot mints a fresh in-container identity that can never serve the room the device is dialing, so it only manufactured orphans (a wedged-but-alive runner that resumes heartbeating now recovers instead of being fenced; a bare untargeted wake still behaves like provision). (6) Devices now revoke every disconnected .cloud tombstone once observed disconnected past a 5-min grace (AppStore.cloudRunnerRevokeGrace, stamped by the peer-presence stream — not lastSeenAt, which only updates on connect), replacing the "spare the newest" policy that kept exactly one dead identity dialing (and wake-nudging) forever. Scale-from-zero is device-driven: a parked dispatch provisions/adopts via provisionAndJoinRunner, which completes the join — the presence hook alone can't, so it no longer boots anything.

(2026-07-18) Scaling model reshaped — fleet of runner peers, not sandboxes in one host. The "Maximum containers" setting became "Maximum runners" (nucleic.runner.maxRunners). A pool no longer runs one host-of-record nucleicd that fans sessions across sibling sandbox containers; it scales the number of independent runner peers (each a full nucleicd with its own mesh identity / relay enrollment / pairing) up to the cap on demand, and reaps idle ones to zero. Placement moved entirely to the owning device's mesh dispatcher (least-loaded across the peer fleet, same code that spreads chats across sibling Macs). The pool-side load balancer (placeSandbox/acquire/release) and the tier-1 sandbox seam (§0.2 item 7) are removed; §2/§4/§7/§8 describe the fleet model.

0. Status

0.1 Done (implemented + verified)

Piece Where State
Runner control plane (Worker + RunnerPool/RunnerContainer DOs) cloud/nucleic-runner/ Implemented + unit-tested (32 node:test tests; typecheck + wrangler dry-run clean; make runner-{test,build,deploy}). KV namespace provisioned; NOT yet wrangler deployed (needs the two secrets, §8)
Wire protocol: intelligence delegation + sealed credential mesh Sources/NucleicProtocol/Sync/RunnerMessages.swift + envelope/capability plumbing in MessageEnvelope/WireMessages/PeerTypes/SyncClient Implemented + tested; additive, capability-gated; every exhaustive switch handled (ConnectionHandler, iOS RemoteStore ×2, HostConnection)
Sealed-credential crypto (SealedCredentialBox) Sources/NucleicProtocol/Sync/SealedCredential.swift Implemented + tested (X25519 + HKDF + ChaChaPoly; AAD binds the credential kind); passes on Darwin AND Linux
Welcome.hostKind + AppStore.meshSelfKind + PeerClient.Config.selfKind protocol + core A runner truthfully pins as .cloud on peers; transfer destinations filter on canOwnSessions (not == .mac)
nucleicd headless host daemon Sources/nucleicd/ Runs on macOS (self-host rung) and Linux — the CLOUD_RUNTIME Phase-2 port landed (§9). Boot-verified on aarch64-glibc and on x86_64 (fully-static musl cross-build via scripts/build-nucleicd-linux.sh, run under qemu): enrolls with the production relay via PoP, listening on: relay, mints working pairing codes, clean SIGTERM shutdown, pool heartbeats with epoch fencing (409 → exit 75)
Linux protocol conformance swift test --filter NucleicProtocolTests on Linux All 162 tests green under swift-crypto — Noise handshake vectors, relay-enrollment PoP (byte-pinned against the Worker), sealed boxes. The CLOUD_RUNTIME go/no-go, answered go
Runner container image containers/nucleic-runner/ Dockerfile ships the static amd64 nucleicd; proven build recipe scripted (scripts/build-nucleicd-linux.sh)
Settings ▸ Covalence ▸ Runner UI, wired to the pool (item 4) RunnerSection in Sources/NucleicApp/RemoteAccessView.swift → AppStore.runnerPoolClient() → RunnerPoolClient (Sources/NucleicCore/Sync/RunnerPoolClient.swift); PoP enroll via RunnerEnrollment (NucleicProtocol, context nucleic-runner-enroll-v1:) Implemented: knob changes debounce-push PATCH /settings, Provision/Refresh drive /provision + /status (per-instance list), the one-shot pairing code surfaces after provision. Proof byte-pinned against the Worker beside the relay vectors (RelayEnrollmentTests — 169 protocol tests green on Linux); credential in Keychain (Darwin) / 0600 file (Linux) with the 401-re-enroll retry. Mesh-riding the pool credential remains (item 4 note)
Project seeding — both layers of the former dispatch gap NUCLEIC_RUNNER_PROJECTS boot seeding in Sources/nucleicd/Nucleicd.swift; ClientMsg.createProject (WireCreateProjectRequest in RunnerMessages.swift, gated on WireCapabilities.canCreateProjects) → SyncHostBridge.createProject → AppStore.addClonedProject (controlled clone; branch supported), answered by HostMsg.projectCreated correlated on requestID Implemented + tested (wire round-trip/legacy tests; iOS switches handled). Idempotent by normalized origin remote (AppStore.projectMatching(remote:)) — a reboot or client retry never stacks repo-2 clones. Boot-smoke-verified on Linux
Phone-initiated project creation (CLOUD_RUNTIME §6) iOS ProjectsView Add Project sheet → RemoteStore.createProject (host picker over live hosts advertising canCreateProjects; addProject state settled by the correlated projectCreated reply) Implemented; iOS builds. The project row itself arrives via the dashboard push
Loopback control endpoint (item 3d) Sources/nucleicd/ControlEndpoint.swift: GET /pairing (one-shot mint → serve → burn, then 410) + GET /health on 127.0.0.1:9200; auto-on in pool mode, NUCLEIC_RUNNER_CONTROL_PORT opts a self-host in Implemented; curl-verified on Linux (ok / payload / 410 / 404 / clean shutdown). This is what the Worker's GET /v1/pool/pairing proxy reaches
Pool credential rides the mesh (item 4 tail) HostMsg.runnerPoolCredential (WireRunnerPoolCredential) pushed post-hello via SyncHost.register → control-scope ConnectionHandler gate → defaulted SyncHostBridge.runnerPoolCredential; PeerClient routes into AppStore.mergeRunnerPoolCredential (updatedAt-monotonic); JSON credential store (legacy dot-form tolerated) Implemented + wire-tested; a shared credential drives the owner's pool + control-plane URL, and only the OWN pool auto-re-enrolls on 401
Credential mesh, both directions (item 5) Runner: RunnerCredentialVault (sealing keypair + inventory + 0600 landing + mirror-back). Device: CredentialProvider (Keychain/store capture, manifest with sealing key, seal-to-runner, land-update). Wiring: SyncHostBridge.credentialNeed push, PeerClient.setCredentialHandlers, AppStore.{receiveCredentialProvision,receiveCredentialManifest,mirrorRefreshedCredentials,credentialRefreshHolder}, nucleicd 30s mirror sweep; CredentialManifest.sealingPublicKey added to the wire E2E-verified over the production relay: device→runner provision + a real authenticated claude turn; runner→device reverse-seal, AAD binding, and the digest change-gate unit-tested (CredentialMeshTests); 171 protocol tests green on Linux. Refresh lease arbitrated by CredentialRefreshLease.merged. All seven kinds land via the table-driven RunnerCredentialVault.landings (Claude/Codex logins, Grok config, GitHub token + Anthropic/OpenAI/xAI keys → files + env vars); Settings ▸ Runner lists what the Mac shares
Presence→wake hook, both sides (item 3e) Runner: cloud/nucleic-runner/src/wake.ts (+ index.ts route POST /v1/hook/wake, env RUNNER_WAKE_SECRET); nucleicd heartbeats carry roomId, the Worker refreshes room:<roomId> → poolId in KV. Edge: handleRelay stamps x-nucleic-room; Room.maybeWakeRunner nudges the hook on client-connect-without-host (30s throttle, fire-and-forget) Implemented + unit-tested both sides (runner 40 green; edge 66 green; both typecheck + dry-run clean). Deploy gate: RUNNER_WAKE_SECRET on both Workers, RUNNER_WAKE_URL var on the edge
Image-publish CI (items 3a/3b) .github/workflows/sandbox-image.yml (now multi-arch arm64+amd64, GHCR) and .github/workflows/runner-image.yml (static amd64 nucleicd → boot smoke vs the deployed relay → wrangler containers push to Cloudflare's managed registry registry.cloudflare.com/<ACCOUNT_ID>/nucleic-runner, NOT GHCR — Wrangler/Containers can't pull GHCR without extra config) Authored; runner-image.yml needs the CLOUDFLARE_API_TOKEN CI secret and must run once to publish before the (now active) wrangler containers block deploys
First real agent turns on a Linux runner (was §0.2 item 2) nucleic-smoke (Sources/nucleic-smoke/, NucleicProtocol-only executable) plays the iPhone programmatically: pair from the --print-pairing code, dial the deployed production relay, startChat, stream events, answer approvals Verified end to end on Linux (glibc container): real claude turns ran Bash/Write/Read in a fresh worktree of the seeded controlled project and completed (awaitingInput); a destructive rm surfaced an approval even under Control-project auto mode (RiskClassifier) and the client's allow over the relay resumed the turn. Required fix: ContainerServiceSettings.serviceEnabled is hard-false on Linux (tier-0 in-place; the stub ContainerManager otherwise killed every dispatch because controlled projects pin the service on)
Darwin regression host + xcodebuild NucleicApp/nucleicd/NucleicCore build; 162 protocol + 100+ core sync/transfer tests pass; iOS app builds — all re-verified after every port wave

0.2 Remaining (ordered — this is the work queue)

  1. createProject — the dispatch gap. Done (kept here so items 2–8 keep their numbers; details in §0.1). Both layers landed: (a) NUCLEIC_RUNNER_PROJECTS boot seeding and (b) the capability-gated createProject wire verb, built via the §11.4 recipe. What remains of it is client UI: nothing sends createProject yet — the phone/Mac affordance ships with CLOUD_RUNTIME §6's phone-initiated project creation (the iOS RemoteStore switches already handle the case inertly).
  2. First real agent turn on a Linux runner. Done (details in §0.1; numbering kept). Real claude turns ran on a Linux nucleicd — dispatched, streamed, and approved over the deployed relay via nucleic-smoke (§11.1 has the recipe). Credentials were copied files (~/.claude/.credentials.json), exactly the interim posture item 5 replaces. Two behaviors learned the hard way: Control projects force auto on (createSession's resolvedAuto), so only destructive calls surface approvals on a runner's default project shape; and ContainerServiceSettings.serviceEnabled is now hard-false on Linux (tier-0 in-place) — see §11.3.
  3. Cloudflare container enablement — mostly landed; two ops gates + one edge edit remain.
    • (a) arm64-only base CI authored: sandbox-image.yml now publishes the nucleic-sandbox multi-arch index (arm64 + amd64, same v6 tag). Gate: the workflow must actually run on a CI branch and the per-CLI --version checks pass on amd64.
    • (b) build + push runner image CI authored: runner-image.yml cross-builds the static amd64 nucleicd, boot-smokes it against the deployed relay, and wrangler containers pushes it to Cloudflare's managed registry registry.cloudflare.com/<ACCOUNT_ID>/nucleic-runner:{v1,latest} (GHCR isn't a valid Cloudflare Containers pull source). Gate: needs the CLOUDFLARE_API_TOKEN repo secret and must run once.
    • (c) wrangler.jsonc carries the active containers block pinned to registry.cloudflare.com/<ACCOUNT_ID>/nucleic-runner:v1. The managed registry needs no wrangler containers registries configure and no pull credentials — Cloudflare pulls it directly. make runner-deploy after (a)+(b) publish (deploying before the image is pushed 404s on a missing image).
    • (d) loopback pairing endpoint Done: nucleicd serves GET /pairing (one-shot: mint → serve → burn, then 410) + GET /health on 127.0.0.1:9200 (Sources/nucleicd/ControlEndpoint.swift), on automatically in pool mode and via NUCLEIC_RUNNER_CONTROL_PORT on a self-host. Curl-verified on Linux end to end.
    • (e) wake hook Done, both sides. Host heartbeats carry the runner's relay roomId; the runner Worker registers room:<roomId> → poolId in KV (wake.ts, refreshed per heartbeat) and POST /v1/hook/wake (bearer: RUNNER_WAKE_SECRET) resolves it and forwards a pool-auth /wake — unit-tested (8 tests). Edge side: handleRelay stamps x-nucleic-room on the Room fetch, and Room.fetch nudges the hook (fire-and-forget, 30s-throttled, first nudge always fires) when a client connects to a room with no host socket — an ordinary Mac's room just 404s at the runner Worker. Unit-tested (3 tests: request shape + throttle, unconfigured no-op, unreachable-hook swallow). Deploy gate: set RUNNER_WAKE_SECRET on both Workers + RUNNER_WAKE_URL on the edge.
  4. Wire Settings → pool. Done, including the mesh ride. RunnerPoolClient (NucleicCore) mirrors RelayAccess: PoP enroll under the nucleic-runner-enroll-v1: context (RunnerEnrollment in NucleicProtocol, byte-pinned against the Worker next to the relay vectors), Keychain/0600-file credential, provision/wake/stopAll, debounce-pushed PATCH /settings, GET /status → RunnerPoolStatus, and the GET /v1/pool/pairing proxy pull; RunnerSection drives it all from Settings. The credential rides the mesh like relay memberships: HostMsg.runnerPoolCredential is pushed post-hello to control-scope peers; receivers merge on updatedAt (newest wins — AppStore.mergeRunnerPoolCredential), so every trusted Mac manages the SAME pool. A client holding a shared credential manages that pool (and its control-plane URL) and only auto-re-enrolls on 401 for its OWN pool — a rotated shared credential says "re-share from the owning Mac" instead of silently creating the wrong pool.
  5. Credential mesh, device side. Done — both directions, proven end to end.
    • Runner receiving: RunnerCredentialVault (NucleicCore) — the sealing keypair (minted first boot, secret-store-persisted, distinct from the Noise identity), the missing-kinds inventory, and 0600-file landing (claude-oauth → ~/.claude/.credentials.json; github-token → ~/.nucleic/github-token + GITHUB_TOKEN env, re-exported at boot). nucleicd sets the vault → canReceiveSealedCredentials; HostMsg.credentialNeeded is pushed post-hello to approve-scope peers; receiveCredentialProvision opens each box (kind-bound AAD) and lands it.
    • Device providing: CredentialProvider (NucleicCore) — captures the Mac's login credentials (Keychain OAuth via ClaudeLoginKeychain, the managed token via GitHubCredentialStore), gossips a CredentialManifest (records + the device's own sealing key, now on the wire) to a runner peer, and seals its held kinds to the runner's key answering credentialNeeded. PeerClient routes all three (manifest push on connect, seal-on-credentialNeeded, land-on-credentialUpdate) into AppStore.
    • Mirror-back (runner → device, the syncClaudeLoginBack analogue): a session's CLI refreshes the OAuth token in place during a turn; a 30s digest-gated sweep in nucleicd (AppStore.mirrorRefreshedCredentials) seals the rotation to each connected device's sealing key and pushes HostMsg.credentialUpdate; the device reconciles it newest-wins into its Keychain (CredentialProvider.landUpdate → ClaudeLoginKeychain).
    • Refresh lease: AppStore.credentialRefreshHolder arbitrates every device's gossiped leases with CredentialRefreshLease.merged (earliest live claim wins), so at most one device advances a rotating token; the runner's own session CLI is the de-facto refresher and the digest-gated mirror-back keeps everyone coherent.
    • Verified: device→runner E2E over the production relay (runner asked, nucleic-smoke provisioned, files landed 0600, a real claude turn authenticated); the runner→device seal + AAD binding and the digest change-gate are unit-tested (CredentialMeshTests); 171 protocol tests green on Linux.
    • All seven kinds land: the vault is table-driven (RunnerCredentialVault.landings) — file-native rotating logins (claude-oauth → ~/.claude/.credentials.json, codex-auth → ~/.codex/auth.json, both mirrorable), the Grok config (~/.grok/config.toml), and the env-backed secrets (GitHub token + Anthropic/OpenAI/xAI API keys, landed to a 0600 nucleic-owned file AND exported as their env var). CredentialProvider reads each from the Mac's real source (Keychain OAuth, ControlAPIKeyStore/CodexControlAPIKeyStore, the ~/.codex/~/.grok files, GitHubCredentialStore, process env), and landUpdate reconciles a mirrored-back Codex rotation newest-wins by last_refresh (CodexAuthFile) the way it does Claude by expiresAt. The two tables are lockstep-tested (providerAndVaultKindsAgree).
    • Mac UI: Settings ▸ Covalence ▸ Runner shows a read-only "Credentials shared with runners" list (AppStore.credentialKindsSharedWithRunners) — what a credentialNeeded would be answered with, sealed on request.
    • Fully done. Gotcha encoded in RunnerCredentialVault.processHome: on Linux homeDirectoryForCurrentUser reads passwd and ignores an overridden HOME, landing files where no CLI looks — resolve $HOME from the environment.
  6. Intelligence executor, device side (AFM behind canProvideIntelligence) + the runner's AgentIntelligenceProvider and nucleic.runner.intelligenceMode plumbing (§5). Done, generalized into a mesh-wide AFM queue (design + file map in §5). Landed: the host-side MeshIntelligenceQueue (priority-ordered, parallel across devices, top tiers desktop-only, Apple-Silicon power bias), DelegatedIntelligenceProvider (mesh + agent modes over the shared IntelligenceDelegate templates, heuristics as guardrails, fail-closed audit), executors on BOTH device platforms (macOS PeerClient → AppleIntelligenceProvider .generateText; iOS PhoneIntelligenceExecutor on FoundationModels), the wire additions (WireIntelligenceRequest.priority, WireClientCapabilities.intelligenceProfile), and nucleicd's NUCLEIC_RUNNER_INTELLIGENCE_MODE / nucleic.runner.intelligenceMode selection (default mesh). The tail is done too: Settings ▸ Covalence ▸ Runner has the mode picker (rides the debounced PATCH /settings push), the pool control plane carries intelligenceMode in PoolSettings (server-clamped to the three modes; older stored records read the default) and injects NUCLEIC_RUNNER_INTELLIGENCE_MODE into every container boot (host + sandboxes), and nucleic-smoke plays an intelligence worker (canProvideIntelligence, desktop profile, canned answers; --expect-intelligence PASSes on the round trip). E2E-verified over the production relay: a Linux nucleicd in mesh mode delegated the chat-naming job (interactive tier → the desktop-class worker), the smoke answered a canned title, and the session came back renamed with it — with the agent turn itself dying (no claude in the container), proving naming never blocks on a dead turn.
  7. Tier-1 sandboxes in nucleicd Retired. The old plan had one host-of-record nucleicd fan sessions across sibling RunnerPool containers via /v1/pool/sandbox/acquire/release. The fleet-of-runners model replaces it (§2, §7): the pool scales the number of full runner peers (each its own mesh identity, tier-0 in-place), and the owning device's mesh dispatcher places chats across them. maxContainers→maxRunners; placeSandbox/acquire/release removed.
  8. Hardening tail: NucleicCoreTests on Linux (only the protocol suite runs there today — the fakes need FileHandle.bytes alternatives); checkpoint/restore to R2 (CLOUD_RUNTIME §3.5); golden-transcript conformance; weekly image-rebuild CI.

1. Thesis

A runner is just another mesh host. PeerKind.cloud already exists, PeerCapabilities {canHost, canRunAgents} already describe it, the relay (live at relay.nucleic.blakeslee.xyz) already routes to any host that enrolls with proof-of-possession of its static key, and every dispatch verb a phone or sibling Mac uses against a Mac host (startChat, approvals, integrate, session transfer, transcript fetch) works unchanged against a host that happens to run in a container. So the runner is not a new protocol — it is:

  1. nucleicd — the headless NucleicCore host (single SessionController writer, TranscriptWriter seq authority, ApprovalCoordinator, WorktreeManager, the three+ adapters, SyncHost over a relay-only listener).
  2. A control plane (cloud/nucleic-runner) that provisions, wakes, scales, and caps the containers nucleicd runs in — the part that is Cloudflare-specific and deliberately thin, so a self-hosted docker/podman driver can replace it behind the same HTTP surface.
  3. Three integration seams that make a cloud host feel like a macOS host: sandbox tiering (§4), intelligence delegation (§5), and the credential mesh (§6).

2. Topology

 iPhone ─┐ Noise/CBOR (E2EE)          ┌────────────── Cloudflare ─────────────────┐
 Mac ────┼── WebSocket ───────────────►│ nucleic-edge Worker + Room DO (deployed)  │
 Mac #2 ─┘  (ciphertext only)         │        │ ciphertext frames                 │
    ▲  ▲                              │        ▼                                   │
    │  │ Noise per runner peer        │ nucleic-runner Worker + RunnerPool DO      │
    │  └──────────────────────────────┤   fleet registry · maxRunners · fencing    │
    │                                 │        │ start/stop (per runner slot)      │
 Settings ▸ Covalence Cloud ─ REST ──►│        ▼                                   │
    (pool credential)                 │ ┌ runner-1 ┐ ┌ runner-2 ┐   … ≤ maxRunners │
                                      │ │ nucleicd │ │ nucleicd │  (each a full     │
                                      │ │ SyncHost │ │ SyncHost │   mesh peer:      │
                                      │ │ ◄ Noise  │ │ ◄ Noise  │   own identity,   │
                                      │ │ + agents │ │ + agents │   room, pairing)  │
                                      │ └──────────┘ └──────────┘                   │
                                      └────────────────────────────────────────────┘
  • Each runner is its own mesh peer. A runner's nucleicd owns one DeviceIdentity; its HostID/relay roomId is sha256(staticKey) like any Mac. A fleet of N runners shows up as N "Cloud" peers in the mesh roster — the owning device pairs each one (roster gossip introduces them to the rest of the mesh). Each runner runs its chats in its own container (tier-0 in-place; RunSpec.container = nil), like the shared nucleic-control container locally.
  • The runner is the scaling unit, not a sandbox inside one peer. The RunnerPool DO scales the number of runner peers up to maxRunners on demand and reaps idle ones to zero (§7); it does not place sessions. Placement — which runner peer a chat lands on — is decided on the owning device by the existing least-loaded mesh dispatcher, exactly as it spreads chats across sibling Macs.

3. Mesh integration (how devices see the runner)

Pairing and dispatch reuse mesh mechanics verbatim:

  • Enrollment/pairing. Provisioning happens from the Mac app (Settings ▸ Covalence ▸ Runner) or, later, iOS. The provisioning device calls the runner control plane, which boots the container; nucleicd generates its identity inside the container (the private key never leaves it), enrolls with the relay via the existing X25519 PoP flow, and returns a PairingPayload (host static key + fresh PSK + relay room fields — the QR payload, minus the QR) over the provisioning channel. The device completes Noise XXpsk0 through the relay and pins the runner like any host. Because the runner is a MeshMember with kind: .cloud, capabilities: {canHost: true, canRunAgents: true}, roster gossip introduces it to every other mesh device automatically — one device provisions, the whole mesh can dial it.
  • Dispatch. startChat into a runner project, approvals (first-responder-wins), model/ effort/auto/ship changes, fetchDiff, transcript fetch, and session transfer (TransferOffer → chunks → 2-phase commit) are the existing verbs; the runner advertises the matching WireCapabilities and the Mac sidebar's unified local/remote representation renders its projects and sessions like a sibling Mac's. "Move to Cloud" is the same "Move to Mac" flow gated on canReceiveSessionTransfer + PeerKind.canOwnSessions (already true for .cloud). A fresh runner's empty store is populated by ClientMsg.createProject (clone a git URL as a controlled repo, gated on WireCapabilities.canCreateProjects — any AppStore-backed host advertises it, cockpit or runner) or by boot-time seeding (§9); the new project reaches devices through the dashboard broadcast.
  • Presence/wake. A device connecting to the runner's room signals the RunnerPool DO to wake the container if asleep (relay presence → pool wake hook), mirroring CLOUD_RUNTIME §3.3. (Runner side built — §0.2 item 3e: host heartbeats register room:<roomId> → poolId and the authenticated POST /v1/hook/wake forwards a pool /wake. The edge Room's nudge on client-connect-without-host is the one remaining edit, spelled out in item 3e.) Provisioning now has both channels: --print-pairing on stdout (self-host) and the loopback :9200 one-shot GET /pairing behind the Worker's GET /v1/pool/pairing proxy (container path).

4. Virtualization tiers (mirroring the macOS desktop's concept)

The macOS host tiers work as: lightweight containers (Apple containerization) for ordinary build/test/scripting, desktop VMs (Linux/macOS, Virtualization.framework) for GUI work, with per-tier caps (nucleic.macvm.maxConcurrent, enforced in MacVMManager.ensureRunning). The runner mirrors the concept on Linux infrastructure:

Tier Local (Mac) analogue Runner implementation Status
0 — in-place shared nucleic-control container agent runs in the runner's own container (RunSpec.container = nil) now
2 — Linux desktop VMs (GUI orchestration) linux_vm_exec / linux_vm_computer future — Cloudflare Containers cannot nest KVM, so tier 2 lands with self-host (QEMU/KVM on the operator's box) or an alternate provider slotted behind the same RunnerSandboxProvider seam future, planned-for

Scaling on the runner is by number of runner peers, not by fanning sessions across sandbox containers inside one peer: each runner runs its chats in-place (tier 0) in its own container, and the pool spins up more runners on demand (§7). The tier-1 "session sandbox inside a runner" seam (an earlier design where one host-of-record fanned out to sibling RunnerPool containers via /v1/pool/sandbox/acquire) has been retired — the fleet-of-runners model replaces it. Tier 2 still matters for GUI work: nucleicd asks an abstract sandbox provider for "a sandbox of tier T"; the Cloudflare provider serves T0 today and rejects T2 with a typed "unsupported on this provider" the UI can explain; a self-host provider maps T2 to a desktop VM. Agent-facing tools stay identical (linux_container now; linux_vm_* when tier 2 exists), so sessions are portable between a Mac host and a runner.

5. Intelligence: replacing Apple Foundation Models in the cloud — implemented (item 6)

IntelligenceProviding (12 methods — summaries, naming, turn classification, triage, bash map/reduce, trunk-summary folds, host-exec reason audits) is the seam; AFMs are app-side only. The runner offers the user's choice of replacement (nucleic.runner.intelligenceMode, env-overridable via NUCLEIC_RUNNER_INTELLIGENCE_MODE; default mesh):

  1. agent — DelegatedIntelligenceProvider.agent(...) (the doc's AgentIntelligenceProvider) backed by one of the user's authenticated agents (the credential mesh already puts Claude credentials on the runner): a one-shot claude -p --model <SKU> --output-format text per call (default SKU haiku, override NUCLEIC_RUNNER_INTELLIGENCE_MODEL), serialized through a private AFMRequestQueue, SIGKILLed at the tier's deadline, over the same strict templates and guardrails as mesh mode (AgentCLIIntelligenceBackend in Sources/NucleicCore/DelegatedIntelligence.swift).
  2. mesh — the mesh-wide AFM queue (MeshIntelligenceQueue, Sources/NucleicCore/Sync/MeshIntelligence.swift): the host submits jobs; every connected device that advertised WireClientCapabilities.canProvideIntelligence (+ its intelligenceProfile {deviceClass, chip}) is a registered worker, and jobs run in parallel across devices (one in flight per worker — the device-side AFM queue is serial anyway). Scheduling:
    • Priority tiers ride the wire (WireIntelligenceRequest.priority: interactive > bashSummary > completion > background, mirroring AFMRequestQueue.Priority). Jobs dispatch highest-tier-first, and the tier also sets the executor's slot on its local AFM queue and the submit deadline (20/30/45/90 s).
    • Device-class restriction: the two top tiers need desktop-class tok/s, so IntelligencePriority.allowsMobileExecution keeps them off iPhones/iPads — completion/background may run there.
    • Apple-Silicon power bias: each job takes the most powerful idle eligible worker by AppleSiliconChip.rank (every M > every A; within M the tier dominates — Ultra > Max > Pro > base, LLM decoding being bandwidth-bound — generation breaking ties), so the hottest work always lands on the strongest chip currently free.
    • Deadlines, worker-disconnect re-dispatch (one retry), stale-result correlation, and a no-eligible-worker fast-fail all resolve nil → the caller's heuristic fallback. Executors render the request through the shared templates (IntelligenceDelegate in Sources/NucleicProtocol/Sync/IntelligenceDelegation.swift — kept in lockstep with AppleIntelligenceProvider's prompts): macOS via PeerClient → IntelligenceDelegateExecutor → AppleIntelligenceProvider.generateText (the app advertises a desktop profile + DeviceCapability.chipName on macOS 26+); iOS via PhoneIntelligenceExecutor (FoundationModels on iOS 26, mobile profile + SoC name). Registration flows through hello → ConnectionHandler → SyncHost.register → SyncHostBridge.intelligenceWorkerConnected (connection-ID-guarded against the dedup race) into AppStore.meshIntelligence; answers route back via receiveIntelligenceResult.
  3. heuristic — HeuristicIntelligence, the same model-free fallback the Mac uses with AI off. Always the terminal fallback of modes 1–2 (guardrails reuse HeuristicSummary.echoesCommands / .isGrounded; a model title still passes HeuristicTitle.sanitizeModelTitle + looksLikeTitle); auditHostExecReason stays fail-closed — no model path ⇒ .notChecked, a garbled verdict token ⇒ .unclear.

Delegation is capability-gated both directions: the host advertises WireCapabilities.canDelegateIntelligence (set by nucleicd in mesh mode via AppStore.delegatesIntelligence, so a client never sends an unsolicited result at a host that would choke on the unknown ClientMsg tag), and only clients advertising canProvideIntelligence are asked. Tests: RunnerMessagesTests (wire + chip hierarchy + template coverage), MeshIntelligenceQueueTests (scheduling contracts), DelegatedIntelligenceTests (encoding/parsing/guardrails/executor), and SyncHostTests.intelligenceWorkerRegistersAndRoundTripsAJob (hello → registration → push → result, end to end over the real Noise/CBOR pipe). Live E2E: boot nucleicd with NUCLEIC_RUNNER_INTELLIGENCE_MODE=mesh + a seeded project and run nucleic-smoke '<pairing>' --expect-intelligence — it advertises the worker capability, serves the delegated sessionName job with a canned title, and PASSes when the sessionUpdated comes back carrying it (verified over the production relay on Linux).

6. The credential mesh

Today, credentials live on the Mac (Keychain: Claude OAuth JSON, control API keys, GitHub token; dot-dirs: ~/.codex/auth.json, ~/.grok/config.toml, …), get seeded into sandbox containers per turn, and refreshed tokens are synced back (syncClaudeLoginBack, syncCodexLoginBack). The runner generalizes exactly this, over the mesh's E2EE channel:

(2026-07-18) The phone can now originate a login, not just relay one — remote agent sign-in (REMOTE_AGENT_LOGIN.md): a device sends ClientMsg.agentLoginBegin at any host advertising canBrokerAgentLogin (Mac cockpit or runner), the host runs the Claude/Codex OAuth flow via its brokers' loginRemotely, the phone presents the consent page and captures the redirect on its own loopback, and the credential lands in the host's canonical store — from where everything below (first-landing mirror-back, credentialNeeded/provision, newest-wins) propagates it mesh-wide. This closes the "fresh runner, no credentialed Mac" bootstrap gap for logins; API keys from the phone shipped as REMOTE_AGENT_LOGIN §8 (one-shot sealed push, never stored on the phone).

(2026-07-18) The phone is now a full credential HOLDER, and any device can DELETE a credential mesh-wide. iOS keeps the two mirrorable logins in an encrypted vault (PhoneCredentialVault: device-local key, SE-P-256-wrapped at rest where available — the honesty note below still holds), gossips a CredentialManifest, answers credentialNeeded for requested kinds, and lands credentialUpdate with the shared newest-wins comparators (ClaudeCredentialFormat/CodexCredentialFormat in NucleicProtocol). It records refresh leases but never acquires one. Deletion rides CredentialTombstone — deletedAt-monotonic per kind, exactly the roster-tombstone discipline — via ClientMsg.credentialRevoke (WireCapabilities.canRevokeCredentials), the HostMsg.credentialRevoked fan-out + post-hello table push, and a new tombstones field on the manifest, so a stale holder's gossip can never resurrect a deleted kind; a landed stone deletes the runner's 0600 file + env var and clears the Mac's Keychain stores, then agentAuthStatus rebroadcasts as the implicit ack.

  • The vault. Each client device keeps a CredentialManifest — the mirrored set of credential records {kind, payload, updatedAt, provenance} for agents + platforms (Claude OAuth, Anthropic/OpenAI/xAI API keys, Codex auth.json, agent home files, GitHub token). The manifest is mirrored across all mesh devices (macOS, iOS) via the existing roster-gossip discipline (idempotent, updatedAt-monotonic per kind), so any single mesh member can serve a runner and any single member can run refreshes for the whole group.
  • At rest / Secure Enclave. The vault is encrypted at rest with a device-local key. Where Secure Enclave exists, the vault key is wrapped by an SE-resident P-256 key (kSecAttrTokenIDSecureEnclave); honesty note per CLOUD_RUNTIME §5: Curve25519 keys cannot live in the SE, so the sealing keypair (below) is Keychain-held, and what the SE protects is the at-rest wrap. Never claim more.
  • Sealed injection (device → runner). The runner's nucleicd generates an ephemeral X25519 credential-sealing keypair at provisioning (inside the container; distinct from the Noise identity so credentials can be re-sealed across identity rotation). When it lacks or loses a credential (fresh boot, expiry), it pushes HostMsg.credentialNeeded {kinds, sealingPublicKey} to connected clients; a holding device replies ClientMsg.credentialProvision(SealedCredentialEnvelope) — each record sealed directly to the runner's key with SealedCredentialBox (ephemeral X25519 → HKDF-SHA256 → ChaChaPoly-AEAD, context nucleic-sealed-credential-v1, recipient key + kind bound as AAD). The relay and control plane only ever carry ciphertext — same posture as sandboxed sessions today, where the host seeds credential files straight into the container.
  • On the runner. Records land as 0600 files/env exactly where the local sandbox seeding puts them (.credentials.json, ~/.codex/auth.json, agent homes, GITHUB_TOKEN), then flow into session sandboxes through the existing per-turn seeding.
  • Refresh, one member for the group. OAuth refresh rotates refresh tokens, so concurrent refreshers revoke each other (the exact race codexControlAuthMode == .apiKey exists to dodge). The mesh elects one refresher per credential kind via a CredentialRefreshLease {kind, holderDeviceID, expiresAt} — gossiped like roster state, preferring a Mac (Keychain, usually awake), falling back to the runner itself. The lease holder refreshes; the rotated record propagates as a manifest update (HostMsg.credentialUpdate to devices, sealed per-device; ordinary mirror gossip between devices); everyone else only ever reads. A stale lease (holder offline past expiresAt) is simply re-acquired — monotonic updatedAt makes late writes harmless.

7. Runner fleet + Settings

The scaling unit is a whole runner — a full nucleicd host with its own mesh identity / relay enrollment / pairing, appearing as its own .cloud peer in the roster. The pool scales the number of runners; it does not place sessions. Which runner a chat lands on is decided on the owning device by the existing mesh dispatcher (runnerPresenceByHost, least-loaded across the peer fleet) — the same code that spreads chats across sibling Macs. There is no host-of-record and no load balancer in the control plane.

  • The fleet registry. RunnerPool DO owns {instanceId, state, sessions, activeSessions, epoch, lastSeenAt, lastActiveAt, lastBusyAt} per runner slot (runner-<n>). Every boot stamps a monotonic fencing epoch (a stale slot's heartbeat is 409'd — CLOUD_RUNTIME §3.3's zombie-writer answer). sessions (total) and activeSessions (still doing something — total minus the Done-and-quiet chats) are reported on each heartbeat for the status list and reaping, not placement.
  • Demand-driven scale-up. POST /v1/pool/provision boots one more runner if the live count is below the cap (lowest free runner-<n> slot), else answers at_capacity. The owning device calls it when a mesh dispatch is parked and liveCloudRunnerCount < maxRunners (AppStore.dispatchChatToMesh → provisionAndJoinRunner, single-flighted); each newly joined runner publishes presence, which retries the parked dispatch. A later parked dispatch grows the fleet again until the cap is reached.
  • Max runners. Settings ▸ Covalence ▸ Covalence Cloud has a stepper (nucleic.runner.maxRunners, default 4, clamp 1…16 — mirroring nucleic.macvm.maxConcurrent's pattern). The Mac pushes it to PATCH /v1/pool/settings with the pool credential; the DO clamps server-side too (clampSettings, the same 1…16 bounds — keep RunnerSettings.maxRunnersRange and pool.ts MAX_RUNNERS_RANGE in lockstep). Past the cap, new chats simply queue on the mesh until a runner frees up — never a failed dispatch.
  • Scale-to-zero. The DO alarm reaps runners uniformly: idle (zero sessions past idleTimeoutSeconds — parity with ContainerManager's idle timers), all-done (still holds sessions but every one is Done and quiet — activeSessions reached 0 — for longer than DONE_SLEEP_MS = 300 s, so a container isn't torn down the instant an agent returns and the user can still follow up; a Done session holding a queued follow-up counts as active and blocks this), dead (silent past the staleness window, with a longer BOOT_GRACE_MS while a runner is still cold-pulling its image), or over a lowered cap (drained oldest-first, never evicting live work). Scale-up from zero is device-driven only: a parked dispatch calls provision and the device completes the mesh join (adopting any ready-but-unjoined runner first). The relay-presence wake hook is a no-op beyond confirming a live slot — it declines to reboot a dead one (identityRotated), because per-boot identity rotation means a reboot could never serve the room the device was dialing (see the 2026-07-19 note); devices likewise revoke disconnected .cloud tombstones after a 5-minute grace so dead rooms stop being dialed at all.

8. Control plane (cloud/nucleic-runner)

Follows nucleic-edge conventions exactly (strict TS, node:test, hash-only KV storage, constant-time compares, HMAC payload.signature tokens).

  • Auth. Two-tier, mirroring relayToken: a pool credential (poolId.poolSecret, minted at pool creation, hash-only in KV) and short-lived instance tokens the DO mints for container↔pool calls. Pool creation requires X25519 PoP of the provisioning device's static key (context nucleic-runner-enroll-v1: — same algorithm as relayEnroll, different context string so proofs can't be replayed across services); the pool credential is then shared to other mesh devices over the E2EE mesh channel, like relay memberships are today.
  • Routes.
    • POST /v1/runner/enroll/challenge / POST /v1/runner/enroll — PoP → {poolId, poolSecret}
    • GET /v1/pool/status · PATCH /v1/pool/settings (pool credential) — Settings surface
    • POST /v1/pool/provision (boot one more runner if under maxRunners) / POST /v1/pool/wake (reboot a specific slept runner slot) / POST /v1/pool/stop — runner fleet lifecycle
    • GET /v1/pool/pairing?instance=runner-<n> (pool credential) — proxy a runner's one-shot pairing payload from its nucleicd loopback control port
    • POST /v1/pool/heartbeat (instance token) — per-runner liveness + session counts + roomId (registers the room→pool→instance mapping the wake hook resolves)
  • Bindings. NUCLEIC_RUNNER_POOLS DO namespace (RunnerPool), NUCLEIC_RUNNER_CONTAINERS DO namespace (RunnerContainer, wrapping the Containers runtime; image containers/nucleic-runner), KV NUCLEIC_RUNNER_TOKENS (provisioned), secrets RUNNER_TOKEN_SECRET / RUNNER_ADMIN_SECRET (set via wrangler secret put before the first deploy). Domain: runner.nucleic.blakeslee.xyz.
  • Self-hosting. The Worker surface is deliberately plain HTTP + one DO; a self-host bundle is the same nucleicd image plus a small compose file where the "pool" is a local supervisor speaking the same routes (docker/podman). Nothing in nucleicd knows it's on Cloudflare.

9. nucleicd

The headless host daemon. It runs on macOS and on Linux: macOS for the spare-Mac self-host rung, Linux for the runner container. The CLOUD_RUNTIME Phase-2 port is done — the seams landed exactly where §3.1 predicted: CryptoKit→swift-crypto behind canImport(CryptoKit) (interop pinned by the protocol suite passing on Linux), Keychain→0600-file LinuxSecretStore, the approval server's TCP leg on BSD sockets (mirroring its AF_UNIX path), the relay WebSocket on SwiftNIO (corelibs' URLSession WS rides libcurl's experimental WS, absent from distro builds), os.Logger→a stderr shim, Apple containerization/Virtualization excluded with stub actors (LinuxSupport.swift) so the shared object graph compiles unchanged. Design rules:

  • Relay-first, LAN-optional. The runner listens on the relay (RelayListener) always; LAN and tailnet only when configured (a self-hosted Mac mini on your LAN wants both).
  • Config via env (NUCLEIC_RUNNER_*): control-plane URL + instance token, relay URL, data dir, transports, display name. No interactive setup; nucleicd --print-pairing emits the pairing payload for the provisioning channel. NUCLEIC_RUNNER_PROJECTS (comma/ whitespace-separated git URLs) clones + registers projects at boot — idempotent by normalized origin remote, and a failed seed logs and moves on rather than blocking boot.
  • Same single-writer runtime. It drives the same AppStore-rooted object graph headlessly (the CLOUD_RUNTIME "no TS twin / no fork" commitment) — identical semantics local and cloud.

10. Risks / honesty notes

  • Trust tier. A runner container holds plaintext code, transcripts, and injected credentials while running — it is a trusted endpoint like your Mac on operated infrastructure, never "E2EE execution." Publish the CLOUD_RUNTIME §5 three-tier table wherever runner marketing appears; the credential mesh's sealed injection means Nucleic's edge code never sees plaintext credentials, which is the honest, strong claim.
  • Linux port has landed its go/no-go (CLOUD_RUNTIME §11.1's tail risk is now bounded): the protocol vectors pass on Linux and a booted Linux nucleicd enrolls + listens on the deployed relay. Remaining Linux follow-ups are operational, not existential: run the NucleicCore test suite on Linux (only the protocol suite runs there today), verify a full agent turn in-container (claude on the BSD-socket approval server), image push + weekly rebuild CI, and the checkpoint/restore choreography (CLOUD_RUNTIME §3.5).
  • Anthropic ToS flux on BYO subscription tokens: unchanged mitigation — API-key/Console fallback first-class (controlAuthMode == .apiKey already exists end-to-end).
  • SE overclaim hazard: the SE wraps the vault at rest; sealing keys are Curve25519 in the Keychain. Copy must not blur this (see §6).

11. Handoff — field notes for the next agent

Everything below was learned the hard way in the implementation passes; read it before touching this subsystem.

11.1 How to build & verify each piece

What Where it runs Command
Control-plane worker any Linux/macOS with node make runner-test · make runner-build (typecheck + wrangler dry-run) · make runner-deploy
Darwin Swift macOS host swift build --target NucleicApp / --target nucleicd; swift test --filter NucleicProtocolTests (+ SyncHostTests, WireMessageTests, PeerClientTests, SessionTransferTests)
iOS macOS host, xcodebuild only xcodebuild -project ios/NucleicRemote/NucleicRemote.xcodeproj -scheme NucleicRemote -destination 'generic/platform=iOS' build — swift build never compiles this target, and protocol changes have silently broken it before (MESH_TRANSFER's own warning; it bit again here)
Linux (glibc, native arch) swift:6.3-bookworm container, repo mounted apt-get install -y unzip libsqlite3-dev (unzip: SwiftPM validates the Sparkle binary artifact at resolution even on Linux!), then swift build --scratch-path /build --product nucleicd and swift test --scratch-path /build --filter NucleicProtocolTests — always use a container-local --scratch-path, never the shared worktree's .build (it would poison the macOS build)
Linux amd64 static (the Cloudflare artifact) same container, any arch scripts/build-nucleicd-linux.sh (installs the Static Linux SDK 0.1.0 for Swift 6.3.3 with pinned checksum, compiles the SQLite amalgamation into its sysroot for GRDB, swift build --swift-sdk x86_64-swift-linux-musl -c release)
Boot smoke test any of the above NUCLEIC_RUNNER_DATA_DIR=/tmp/rd NUCLEIC_RUNNER_NAME=Test timeout --signal=TERM 25 <nucleicd> --print-pairing → expect listening on: relay, a nucleic://pair?d=… line on stdout, shutting down. On a non-native arch: prefix qemu-x86_64-static (static binaries need no binfmt setup). Note: this enrolls a throwaway room on the production relay — harmless (hash-derived roomIds can't squat; credentials age out)
Agent-turn smoke test (nucleic-smoke) Linux box/container with claude + credentials (~/.claude/.credentials.json) Boot nucleicd with a seeded project and --print-pairing (pipe through stdbuf -oL when redirecting — the pairing line is otherwise stuck in stdio buffering), then nucleic-smoke '<pairing-code>' --prompt '…' --model haiku --timeout 200 → pairs over the relay, starts a chat in the first project, streams events, answers approvals, exits 0 on turn completion. Gotchas: the pairing secret is one-time (restart nucleicd for a fresh code per run); make sure exactly ONE nucleicd is up — N daemons supersede each other's relay room socket and the room silently swallows client frames (and slim images lack pkill: kill by /proc/*/comm scan); a prompt needs a destructive call (e.g. rm) to surface an approval, because Control projects run auto

11.2 Port gotchas (why the code looks the way it does)

  • ClientMsg decode THROWS on unknown tags; HostMsg falls back to .unknown. Every new client→host verb must be gated on a WireCapabilities flag the host advertises (the relayMembership / canFetchDiff recipe). And every new ClientMsg/SyncClient.Event case breaks four exhaustive switches: ConnectionHandler.messageLoop (host), SyncClient.messageLoop (protocol), iOS RemoteStore (owner-routing + demo switches), and iOS HostConnection (event switch). The compiler only tells you about the iOS ones under xcodebuild.
  • swift-crypto ≠ CryptoKit in one visible way: Curve25519.*.PrivateKey isn't Sendable on Linux — hence DeviceIdentity's conditional @unchecked Sendable (SecureChannel.swift). Everything else was drop-in; the interop is proven by the vectors, not assumed.
  • corelibs-foundation's URLSessionWebSocketTask does not work on distro Linux (it rides libcurl's experimental WS protocol, compiled out of Debian's libcurl). Symptom: NSURLErrorDomain -1005 instantly at connect. Plain HTTPS via URLSession works fine (the PoP enrollment used it successfully). Hence the NIO-backed RelayWebSocket (#else leg of RelayTransport.swift): NIOSSL TLS → HTTP/1.1 upgrade → RFC 6455 client framing — client frames must be masked (maskKey: .random()), fragments reassembled, pings answered. WireFraming.maxFrameSize (16 MiB) equals NIO's decoder cap exactly; don't add headroom.
  • On Linux, URLSession lives in FoundationNetworking, XMLParser in FoundationXML — every file touching them needs the #if canImport(...) import.
  • Three libcs, not two: Darwin / Glibc / Musl (the static SDK). Gates that say #if canImport(Glibc) break under musl. Type spellings differ: SOCK_STREAM is Int32 (Darwin), __socket_type (glibc — needs .rawValue), plain int (musl); SHUT_RDWR needs Int32(...); there's no SO_NOSIGPIPE on Linux → MSG_NOSIGNAL per send (writeNoSigpipe in MCPApprovalServer.swift).
  • SwiftPM cross-platform manifest tricks that worked: keep every package dependency declared on all platforms (identical Package.resolved everywhere — the Linux resolve never dirtied it) and condition the products with .when(platforms:); compute the NucleicCore exclude: list per-platform for Apple-framework files; #if os(Linux) runs at manifest-evaluation time. Declare helper arrays before let package (top-level code executes in order). Node's strip-types runner (worker tests) rejects constructor parameter properties — that's why RunnerPoolCore declares fields explicitly.
  • The Linux stubs (Sources/NucleicCore/LinuxSupport.swift) mirror the FULL externally referenced surfaces of ContainerManager/MacVMManager plus the value types from excluded files (MacVMEntry, MacVMMaintenanceInfo, ContainerResourceSample, ContainerKillDiagnosis, UncheckedSendableBox). The container naming statics are duplicated there (channel suffix, shared/control names) — flagged "keep in lockstep" in both files; if you touch naming in Container/ContainerManager.swift, touch the stub. Portable files that stayed compiled on Linux: MacVMSpec/MacVMSurface/MacVMPackage, CommandInterceptor (its only Apple import was os).
  • GRDB on Linux: works. Glibc: libsqlite3-dev. Musl static: no sqlite in the sysroot — the build script compiles the amalgamation with Debian-equivalent flags (COLUMN_METADATA/FTS5/RTREE/SNAPSHOT/DBSTAT) and installs it into the SDK sysroot.
  • Static Linux SDK practicals: version is per-toolchain (6.3.3 → bundle 0.1.0; checksum from https://www.swift.org/api/v1/install/releases.json, platform static-sdk). The bundle carries BOTH arch sysroots (~2 GB unpacked) — extract selectively when disk-tight. swift sdk install of a local file needs no checksum flag; a URL does.
  • Host.current(), Observation, DispatchSource/DispatchIO, signal handling all work on Linux as used. iCloud APIs (startDownloadingUbiquitousItem, NSFileVersion) don't exist — ProjectRelocator's iCloud passes are #if canImport(Darwin)-gated to no-ops.
  • AppStore runs headless — the CLOUD_RUNTIME open question is answered: no AppKit in its transitive graph; nucleicd drives the same object graph as NucleicApp.init minus UI hooks and minus ModelCatalog (App-layer; a runner takes model/effort per-dispatch, and an empty Welcome.modelCatalog is the designed fallback).

11.2b Production lifecycle fixes (2026-07-13) — read before touching provisioning

A live investigation (throwaway pool enrolled against the deployed Worker) found the deployed control plane, image, and secrets all working — a provisioned container booted and heartbeated in seconds — while the user-visible flow was dead. Three causes, all fixed; the failure modes are worth knowing because they were invisible from every dashboard:

  • The stale-host wedge. provision() (and wake(), which is the same verb) answered alreadyRunning for ANY non-stopped host record — including one whose heartbeat was long stale (container crashed, evicted, or never really born; container.start() is fire-and-forget and nothing calls monitor()). The alarm never reaps the host slot, so one dead record wedged the pool forever: every Provision was a silent no-op and the presence→wake hook could never revive a slept container — the exact case it exists for. Now: a host record stale past HEARTBEAT_STALE_MS is treated as dead and re-booted with a bumped fencing epoch (zombies fence out via the 409 path); the alarm likewise reaps sandboxes whose heartbeat went stale even if their last report claimed live sessions (dead capacity otherwise held a maxContainers slot forever). Settings ▸ Covalence Cloud also gained a Stop runner button — the explicit reset.
  • The loopback-bind pairing failure. ControlEndpoint bound 127.0.0.1:9200, but the RunnerContainer DO's getTcpPort(9200).fetch(…) reaches the container over its virtual network interface — Cloudflare Containers require listening on 0.0.0.0; a loopback listener is unreachable and every GET /v1/pool/pairing 500d, so no runner could ever hand out its pairing payload (= never join the mesh). Pool/container mode now binds 0.0.0.0 (still private: container ports are DO-proxy-only, never public); a bare self-host keeps loopback; NUCLEIC_RUNNER_CONTROL_BIND overrides.
  • Nobody completed the join. The Settings flow pulled the one-shot pairing code exactly once, ~1s after provision (a race it usually lost even ignoring the bind bug), swallowed failures with try?, and only displayed the code. §3's design — the provisioning device completes XXpsk0 itself — is now implemented: provision polls /status until the host's first heartbeat, pulls the pairing payload with retries, and calls AppStore.joinMesh(with:) so the runner lands in the mesh with zero manual steps (the code stays visible as a manual fallback; a 410 means the one-shot was already consumed — Stop + re-provision re-arms it).

Related hardening from the same pass: nucleicd now adopts the re-minted instance token each heartbeat returns (the pool re-mints precisely so a live container's token never lapses; reusing the boot token forever would 401 after INSTANCE_TTL = 30 days) — the client became an actor for that. CI: runner-image.yml now also triggers on Sources/** + Package.* (the image's payload is the nucleicd binary; Swift-only changes used to ship a silently stale daemon), and every successful image push redeploys the Worker (Cloudflare pins the image digest at deploy — a re-pushed :v1 only rolls out with a deploy); runner-deploy.yml deploys on Worker-code changes and by hand. Worker secrets survive deploys.

Ops note: a pool wedged by the old bug shows a stale host row and inert Provision — after this ships, either press Stop then Provision, or just Provision (stale-recovery reboots it).

11.3 Trip-wires for the next tasks

  • createProject (§0.2 item 1): transfer import rejects unknown projects (resolveTransferProject — by UUID / root commit / normalized remote), so seeding projects is a prerequisite for BOTH dispatch paths, not just startChat.
  • Pairing proxy (§0.2 item 3d — done): RunnerContainer DO proxies GET /pairing to loopback :9200 (NUCLEICD_CONTROL_PORT in cloud/nucleic-runner/src/container.ts); nucleicd serves the payload there once (mint → serve → burn, then 410 — re-provisioning means a container restart), mirroring --print-pairing. ControlEndpoint.swift in the nucleicd target; the burn happens before the bytes go out, so a half-delivered payload counts as served rather than risking a double-armed secret.
  • Pool auth handshake: poolId = sha256(provisioning device's static key) — the same string as that device's HostID/relay roomId, PoP context nucleic-runner-enroll-v1: (proofs can't replay across services). Re-enrolling rotates the pool secret.
  • Epoch fencing contract: heartbeat 409 means "a newer boot of this slot exists" — nucleicd exits 75 (EX_TEMPFAIL) and the supervisor must NOT restart that epoch.
  • swift test on Linux builds the whole package graph — a compile break in any target blocks the protocol suite; the Darwin-only test fixtures (fake-claude/fake-grok, which use FileHandle.bytes) are excluded from the Linux manifest for exactly this reason.
  • ContainerServiceSettings.serviceEnabled is hard-false on Linux. Controlled projects — the runner's default project shape — pin the container service on via reconcileContainerService, and with it on, SessionController.isSandboxed routes every dispatch into the stub ContainerManager, which throws ("the Apple-containerization sandbox service does not exist on Linux"). The #if os(Linux) gate in Project.swift makes tier-0 in-place execution (RunSpec.container == nil) the only path, per §4. Don't "fix" a dead dispatch by enabling the service on Linux — wire the RunnerPool sandbox provider (item 7) instead.
  • swift build takes ONE --product — repeated flags silently keep only the last one (this shipped a stale nucleicd mid-verification once). Build products in separate invocations.

11.4 The additive-wire-change recipe (used for all runner verbs; reuse it)

  1. New payload struct in RunnerMessages.swift (Sendable/Codable/Equatable, decode-defaulted optionals). 2. New WireCapabilities flag (decode-default false) if a client will send a new tag. 3. ClientMsg/HostMsg cases + EnvelopeKey + both codec switches in MessageEnvelope.swift. 4. SyncClient.Event + messageLoop. 5. Handle in ConnectionHandler (host) with a defaulted SyncHostBridge hook. 6. Handle-or-inert in iOS RemoteStore (×2) + HostConnection. 7. Round-trip + legacy-tolerance tests in RunnerMessagesTests.swift. 8. Build macOS AND iOS (xcodebuild) AND Linux.
Reference in New Issue View Git Blame Copy Permalink