# Nucleic — Native in-guest automation agent (design) > **Superseded as the default; now an optional add-on.** Since the host-side virtual-IO path landed > (MACOS_VM.md §12.2 — capture the guest framebuffer via `VZVirtualMachineView` + inject `NSEvent` > HID, all host-side with no guest TCC), that is the **default** computer-use path. This native agent > is retained as an **opt-in** (`MacVMSettings.axAgentEnabled`, OFF by default) for its enduring win: > **semantic**, framebuffer-independent Accessibility control (`ax_*` — read the accessibility tree, > act on an element by identity), which is more robust than pixel targeting and lets you inspect an > app's AX tree when debugging. Only this agent needs the guest's TCC grants — which the supported > macOS 27 one-click base build writes automatically (no manual step; MACOS_VM.md §12.5). The pixel > ops it also implements are now redundant with the host-side surface. A **native in-guest agent** that observes and controls a macOS guest VM through Apple's frameworks directly — the **Accessibility API** (`AXUIElement`) for semantic control, **ScreenCaptureKit** for capture, **CGEvent** for raw input — reached from the host over **vsock**. It replaces the current `mac_vm_computer` transport (SSH → `launchctl asuser` → `screencapture`/`cliclick` → base64), which is a stack of shims over those same frameworks. **Status:** implemented through §13 phases 1–4 (code + provisioning), **pending real-guest validation** (§13 phase 5 / §14). What exists: the guest agent (`guest/NucleicVMAgent` — its own SwiftPM package, macOS 27 floor — vsock listener, all §4 ops), the host side (`MacVMAgentClient` + the `VZVirtioSocketDevice` config + the ping-probe/fallback routing in `MacVMEngine+ComputerAgent`), the extended `mac_vm_computer` tool surface (`ax_*` actions + `ref`/`value`), the app-bundle build (`scripts/build-vm-agent.sh`), and provisioning Phase 7d (`scripts/provision-macos-guest.sh`). The empirical claims to validate on a real guest (guest-side vsock `listen`, and AX/CGEvent behaviour) are called out in §9 and §10 — until then the SSH path remains the proven default and the agent is strictly additive. Companion to [MACOS_VM](MACOS_VM.md) (which this augments) and [VSOCK_CONTROL_PLANE](VSOCK_CONTROL_PLANE.md) (Nucleic's existing vsock work for the Linux container). --- ## 1. Why — the CLI shims are the wrong layer `mac_vm_computer` today (see [MACOS_VM](MACOS_VM.md) §12) reaches the guest over SSH and drives it with CLI tools that are themselves thin wrappers over Apple frameworks: | Concern | Today (janky) | Native agent | | --- | --- | --- | | Input | `cliclick` over SSH, `launchctl asuser`, coordinate clicks | **Accessibility API** (`AXUIElement`) — act on a control *by identity*; **CGEvent** for raw fallback | | Observe | `screencapture` CLI + base64 over SSH | **`AXUIElement` tree** (semantic) + **ScreenCaptureKit** (pixels), in-process | | Transport | SSH over NAT + `/var/db/dhcpd_leases` IP discovery | **vsock** — a VM device channel, no NAT/DHCP/SSH, no macOS network prompts | Three concrete wins, not just tidiness: 1. **Semantic control.** Instead of "the model looks at a screenshot, guesses `(x, y)`, and clicks blind," the agent reads the accessibility tree — every control's role, title, value, frame, and available actions — and acts on an element *by identity* (`AXPress` a button, set a field's `AXValue`). This is how professional macOS UI automation works, and it's dramatically more reliable than pixel targeting. 2. **The Accessibility tree is framebuffer-independent** — a robustness property of AX (§9). The AX tree is vended by AppKit from the app's *semantic* view model, in-process, independent of compositing and of any pixel readback — so an AX-driven agent observes and drives app UIs from the app's own state rather than from a captured image. This makes control robust even when a *capture* API returns nothing useful (a compositor hiccup, an off-screen or occluded window), and it's what lets you inspect an app's live AX tree when debugging. It's fundamentally more reliable than guessing coordinates off a screenshot. 3. **vsock, not SSH.** No NAT/DHCP/lease-parsing, no SSH keys, no `launchctl asuser` gymnastics, no macOS incoming-connection prompts — a VM device channel handled by `Virtualization.framework` (`docs/VSOCK_CONTROL_PLANE.md` moved the Linux container's control plane onto vsock for the same reason). **What it does *not* solve (honesty):** TCC is still required — the agent needs **Accessibility** and **Screen Recording** grants (§8); an active **Aqua/GUI session** is still required (AX, SCK and CGEvent all need WindowServer), so auto-login and a per-user **LaunchAgent** stay. --- ## 2. Architecture ``` Claude agent │ calls the mac_vm_computer tool (now with AX-semantic actions — §7) ▼ MCPApprovalServer ──▶ ClaudeCodeBackend.handleMacVMComputerCall │ │ │ ▼ │ MacVMManager ──▶ MacVMEngine │ │ preferred: native agent over vsock │ │ fallback: SSH + screencapture/cliclick (today's path, §10) │ ▼ │ VZVirtioSocketDevice.connect(toPort:) ── host end of vsock ──┐ ▼ │ (image + AX tree flow back up as MCP content the model sees) ▼ ┌──── macOS guest (the clone) ──────────────┐ │ NucleicVMAgent (LaunchAgent, Aqua session)│ │ AF_VSOCK listener :port │ │ • AXUIElement (observe + control) │ │ • ScreenCaptureKit (capture) │ │ • CGEvent (raw mouse/key/scroll) │ └─────────────────────────────────────────────┘ ``` The native path and the SSH/CLI path **coexist**: the engine prefers the agent (a vsock `ping` succeeds), and transparently falls back to today's SSH path when the agent isn't present or reachable, so nothing regresses and the base image can adopt the agent incrementally. --- ## 3. Transport — vsock Apple's `Virtualization` framework gives the host a direct socket into the guest, no IP stack involved. ### 3.1 Host side (`MacVMEngine`) - **Config:** add exactly one `VZVirtioSocketDeviceConfiguration()` to `config.socketDevices` in `buildConfiguration` (currently the mac-VM config has no socket device). - **Runtime:** after boot, on the VM's serial queue, `vm.socketDevices.first as? VZVirtioSocketDevice`. - **Connect:** `func connect(toPort: UInt32) async throws -> VZVirtioSocketConnection`. Apple: *"If the guest operating system doesn't listen for connections to the specified port, this method does nothing"* — so the **guest must be listening first** (the agent's listener, §5). The returned `VZVirtioSocketConnection` exposes a real POSIX `fileDescriptor` (write to send, read to receive; it becomes `-1` when closed) and a `close()`. Wrap the fd in `DispatchIO` (idiomatic for a socket) or a nonblocking `DispatchSource`; keep a strong reference to the connection while I/O is in flight. - **One device per VM.** A `VZVirtioSocketDevice` on macOS can't be told which VM connected on the host side (host CID is fixed at 2, guests at 3), but since Nucleic owns one device per VM instance, that's moot. ### 3.2 Guest side (`NucleicVMAgent`) Standard BSD vsock server (`#include `; `AF_VSOCK` is available *inside* a macOS guest — on the host `socket(AF_VSOCK, …)` fails `ENODEV`, which is why the host reaches it only via `VZVirtioSocketDevice`): ```c int s = socket(AF_VSOCK, SOCK_STREAM, 0); struct sockaddr_vm addr = {0}; addr.svm_len = sizeof(addr); // macOS-specific BSD length field (Linux has none) addr.svm_family = AF_VSOCK; addr.svm_cid = VMADDR_CID_ANY; // listen on any CID addr.svm_port = NUCLEIC_AGENT_PORT; bind(s, (struct sockaddr *)&addr, sizeof(addr)); listen(s, 4); int conn = accept(s, NULL, NULL); // the host's connect(toPort:) lands here ``` - **Port:** a fixed Nucleic-reserved port (e.g. `2035`), a shared constant on both sides. - **⚠ Verify on a real guest (§10):** Apple's `Virtualization` docs describe the host-connects / guest-listens topology in guest-OS-agnostic terms, and `` exists on macOS, but no single Apple doc names "*macOS* guest `listen()` + host `connect(toPort:)`" — smoke-test it on the target guest build before building on top. (Linux guests are proven; the macOS-guest listen path is the one inference.) If it doesn't hold, the reverse topology — host `VZVirtioSocketListener` + guest connects to `VMADDR_CID_HOST` — is the documented fallback. ### 3.3 Framing Newline-delimited JSON (NDJSON): one request object per line, one reply per line — the same framing Nucleic already uses for agent stdio (`LineSplitter`). Screenshots ride as base64 in the JSON (simple, matches the existing image path); a length-prefixed binary frame is a future efficiency option if base64 overhead on a screenshot→act loop matters. --- ## 4. Wire protocol (host ⇄ agent) Requests carry an `op`; replies carry `ok` plus op-specific fields (or `ok:false, error`). Handles to UI elements are opaque `ref` strings the agent assigns (§6.2). | `op` | Request fields | Reply | Backed by | | --- | --- | --- | --- | | `ping` | — | `{ok, version, displays:[…]}` | capability negotiation | | `screenshot` | `display?`, `format:"jpeg"` | `{ok, image:, width, height, scale}` | ScreenCaptureKit (§7) | | `ax_dump` | `target:"frontmost"\|pid`, `maxDepth?` | `{ok, tree:}` | AXUIElement (§6) | | `ax_element_at` | `x`, `y` | `{ok, node}` | `AXUIElementCopyElementAtPosition` | | `ax_action` | `ref`, `action:"AXPress"\|…` | `{ok}` | `AXUIElementPerformAction` | | `ax_set_value` | `ref`, `value` | `{ok}` | `AXUIElementSetAttributeValue` | | `ax_focus` | `ref` | `{ok}` | set `kAXFocusedAttribute` | | `click` | `x`, `y`, `button?`, `count?` | `{ok}` | CGEvent (§8) | | `move` | `x`, `y` | `{ok}` | CGEvent | | `drag` | `fromX,fromY,toX,toY` | `{ok}` | CGEvent | | `type` | `text` | `{ok}` | CGEvent keyboard | | `key` | `chord:"cmd+s"` | `{ok}` | CGEvent + flags | | `scroll` | `dx`, `dy` | `{ok}` | CGEvent scroll wheel (**real scroll**, unlike the cliclick arrow-key hack) | | `launch_app` | `name` | `{ok}` | `NSWorkspace.openApplication` | A `node` is: ```json { "ref": "e17", "role": "AXButton", "subrole": "AXCloseButton", "title": "Build", "value": null, "enabled": true, "focused": false, "frame": { "x": 812, "y": 44, "w": 74, "h": 22 }, "actions": ["AXPress", "AXShowMenu"], "children": [ … ] } ``` ### 4.4 Streaming exec (protocol v2) — the vsock replacement for SSH exec `exec` is the one **streaming** op and the transport for `mac_vm_exec` (retiring SSH-over-NAT). Unlike every request/reply op above, it takes over its connection for the life of the command, so the host opens a **dedicated** vsock connection per exec (the listener is one thread per connection → concurrent execs + the control channel never interfere). - **Request** (one line): `{"op":"exec","command":}`. The host composes env exports + `cd ` + the body into `command` and the guest runs it under `/bin/zsh -c` (so `/etc/zshenv`'s toolchain PATH resolves), exactly as sshd used to invoke it. - **Frames** (one JSON object per line, tagged by `t`), guest→host: `{"t":"o","d":}`, `{"t":"e","d":}`, then terminal `{"t":"x","code":}` (or `{"t":"err","m":}` if the spawn failed). Payload bytes are **base64** so arbitrary binary output can't break the NDJSON framing. - **Frames**, host→guest: `{"t":"i","d":}`, `{"t":"eof"}`, `{"t":"sig","n":}`. Host side is `MacVMExecChannel` (a `ProcessHandle` that re-splits the base64 chunks into newline lines); guest side is `ConnectionHandler.runExec` (spawns the process, pumps its pipes as frames, polls the connection for input frames). Constants are mirrored in `MacVMAgentWire.Exec` (host) and `VMAgentCore.AgentWire.Exec` (guest) — keep them in lockstep. The protocol version is bumped to **2** because a v1 agent doesn't speak `exec`. --- ## 5. The in-guest agent — `NucleicVMAgent` A small Swift program, packaged as a **signed `.app` bundle** (the correct TCC shape — §8), run as a per-user **LaunchAgent** under the `agent` account so it lives in the Aqua GUI session (AX, SCK and CGEvent all require an active WindowServer session). Responsibilities: - Bind the `AF_VSOCK` listener (§3.2); accept loop; per-connection NDJSON request loop. - Implement the §4 ops with `ApplicationServices`/`AXUIElement` (§6), `ScreenCaptureKit` (§7), and `CoreGraphics`/`CGEvent` (§8). - On start, self-check `AXIsProcessTrusted()` and Screen-Recording authorization; report readiness in the `ping` reply so the host can surface a clear "agent present but not TCC-granted" state. - `LSUIElement` (no Dock icon). Small, single-purpose, no network egress. It is **not** an MCP server itself — it speaks the private §4 protocol to `MacVMEngine`, which bridges it to the `mac_vm_computer` MCP tool. The host keeps ownership of VM lifecycle, gating, and the approval boundary; the agent is a dumb, well-scoped actuator. --- ## 6. Accessibility — observe + control Framework: `ApplicationServices` (re-exports `HIServices`). All calls are synchronous cross-process IPC into the target app, governed by `AXUIElementSetMessagingTimeout` (default ~6 s). ### 6.1 Reading the tree - Root: `AXUIElementCreateApplication(pid)` for the frontmost app (`NSWorkspace.shared.frontmostApplication?.processIdentifier`), or `AXUIElementCreateSystemWide()` for hit-testing. - Per node: `AXUIElementCopyAttributeValue` for `kAXRole`, `kAXSubrole`, `kAXTitle`, `kAXValue`, `kAXEnabled`, `kAXFocused`; `kAXPosition`/`kAXSize` come back as an `AXValueRef` box — unwrap with `AXValueGetValue(_, .cgPoint/.cgSize, _)` (AX uses **top-left-origin** screen coords). Actions via `AXUIElementCopyActionNames`. Recurse `kAXChildren`. - **Cost control:** a full walk of a rich app is thousands of IPC round-trips. Batch with `AXUIElementCopyMultipleAttributeValues`, cap `maxDepth`, dedupe by element identity, and set a tight messaging timeout so a hung app can't wedge the walker. ### 6.2 Element handles (`ref`) `AXUIElementRef`s go stale as the UI mutates, so the agent keeps a per-connection registry mapping a short `ref` id → the live `AXUIElementRef` it handed out in the last `ax_dump`/`ax_element_at`. Refs are valid until the next dump (or a short TTL); the host's convention is **dump → act on a ref → dump again**. A stale ref returns `kAXErrorInvalidUIElement`, which the host surfaces as "re-dump and retry." ### 6.3 Control - Press/menu: `AXUIElementPerformAction(el, kAXPressAction)` (also `kAXShowMenu`, `kAXConfirm`, `kAXCancel`, `kAXIncrement`/`kAXDecrement`). Treat `kAXErrorCannotComplete` as *possibly succeeded* — apps often run modal processing inside the action callback and don't return within the AX timeout. - Type into a field: set `kAXFocused = true`, then `kAXValue = ""` (check `AXUIElementIsAttributeSettable` first). Some non-AppKit fields reject `AXValue` writes — fall back to focusing the element then CGEvent keystrokes (§8). - **Prefer AX actions over CGEvent coordinate clicks** wherever a control exposes an action: AX drives the app in-process and is fully framebuffer-independent (§9); a CGEvent click at `kAXPosition` is closer to the pixel world (still geometry-correct, but less robust). --- ## 7. Screen capture — ScreenCaptureKit For pixels: `SCShareableContent.current` → `SCContentFilter` (full display or a specific `SCWindow`) → `SCScreenshotManager.captureImage(contentFilter:configuration:)` (macOS 14+) → `CGImage` → JPEG. In-process, no `screencapture` CLI, no `launchctl asuser`, no base64-over-SSH. Needs Screen Recording TCC + an Aqua session. Capture depends on the compositor rendering the window, so **AX (§6), not SCK, is the robust "look" primitive** whenever a control exposes a semantic identity — pixels are the fallback for custom-drawn UI. --- ## 8. Low-level input — CGEvent When AX has no action for a target (custom-drawn UI): `CGEvent(mouseEventSource:mouseType:…)` + `.post(tap: .cghidEventTap)` for clicks/moves/drags at screen coordinates; `CGEvent(keyboardEventSource: virtualKey:keyDown:)` with `.flags` for keystrokes and chords; `CGEvent(scrollWheelEvent2Source:…)` for **real** scroll (a proper wheel event — the current cliclick path fakes scroll with arrow keys). CGEvent needs Accessibility TCC and the Aqua session. Note CGEvent clicks target pixel coordinates, so they lean on window geometry being correct but are less framebuffer-independent than AX actions; prefer AX. --- ## 9. Framebuffer independence (the enduring robustness property — HIGH confidence) The **Accessibility tree is architecturally independent of the rendered framebuffer**: AppKit vends it from the view's *semantic* model (roles/titles/values inferred from `NSAccessibility` conformance), `accessibilityFrame` is a layout-geometry computation (not a pixel readback), and `AXUIElementPerformAction` invokes the control's action callback **in the app process** — no rendering, no synthetic on-screen click required. `NSAccessibilityElement` can even back UI with *no `NSView` at all*, proving AX is decoupled from any drawn surface. Consequently AX observe+control is driven by the app's own state, not by a captured image. It stays correct where a *capture* API is unreliable — an occluded or off-screen window, a compositor hiccup — because `CGWindowListCopyWindowInfo` bounds and the window server's geometry remain valid even when the pixels don't. That decoupling is exactly what makes semantic control more robust than pixel targeting. **Caveats (none framebuffer-related, but real):** lazy/virtualized views (table cells) may be absent from the tree until laid out; **non-AppKit apps** (Electron/Chromium, Java, GL/game UIs) expose poor or empty AX trees regardless of platform — Chromium needs `AXManualAccessibility`/`--force-renderer- accessibility`, Electron `app.setAccessibilitySupportEnabled(true)`; and some toolkits return `kAXErrorCannotComplete`/`kAXErrorAPIDisabled` intermittently — budget retries and agent-restart handling. **Still verify empirically on a real guest** — the architecture is confirmed but should be smoke-tested. --- ## 10. Tool surface & fallback `mac_vm_computer` gains AX-semantic actions alongside today's pixel actions ([MACOS_VM](MACOS_VM.md) §12.1): - `ax_dump` → returns the front app's element tree (roles/titles/values/frames/actions) as text the model reads — the semantic "look," and the framebuffer-independent one. Optionally also returns a screenshot. - `ax_press` / `ax_set_value` / `ax_focus` (by `ref`) → act on a control by identity. - `ax_element_at` (x,y) → resolve the element under a point. - Existing `screenshot` / `left_click` / `type` / `key` / `scroll` — now routed through the agent's ScreenCaptureKit/CGEvent when present (cleaner, and `scroll` becomes real). Guidance to the model: *prefer `ax_dump` + act-by-ref; fall back to screenshot + pixel click when a control has no AX action or isn't in the tree.* **Fallback / capability negotiation.** `MacVMEngine` tries a vsock `ping` when it needs the agent; on success it routes ops natively, on failure (no agent baked in, not launched, not TCC-granted, or vsock listen unsupported) it uses **today's SSH + screencapture/cliclick path** unchanged. Per-op fallback is possible (e.g. native input but SSH screenshot) but the simplest first cut is all-or-nothing by `ping`. AX-semantic ops have **no** SSH equivalent, so `ax_dump`/`ax_*` are only advertised when the agent is present. --- ## 11. TCC, packaging, provisioning - **TCC:** the agent needs **three** distinct grants, each a separate `TCC.db` service: `kTCCServiceAccessibility` (AXUIElement read/control — `AXIsProcessTrusted`), `kTCCServicePostEvent` (CGEvent input synthesis — surfaced under the "Accessibility" list but a distinct row; `CGPreflightPostEventAccess`/`CGRequestPostEventAccess`), and `kTCCServiceScreenCapture` (ScreenCaptureKit — `CGPreflight/CGRequestScreenCaptureAccess`). A proper signed **`.app` bundle** with a **native Mach-O** main executable is the *correct* shape (TCC mis-handles script-based executables, and recent macOS wants a real bundle for Screen Recording, attributed to the responsible process) — an improvement over granting a bare CLI. Pre-grant all three to the agent's bundle-id / code requirement via the same direct system-`TCC.db` write the provisioner does automatically on the supported macOS 27 base build ([MACOS_VM](MACOS_VM.md) §12.5); no other unattended path exists (PPPC can't silently grant Screen Recording; `tccutil` can't grant). Note a grant is often not picked up until the process **restarts** (esp. Screen Recording), so the LaunchAgent should be bounced after provisioning writes the rows. If a grant is missing at runtime the agent reports it in the `ping` reply (via `AXIsProcessTrusted` + the `CGPreflight*` checks) so the host can surface an actionable "agent present but not authorized" state rather than silently failing. - **Packaging:** a real signed **`.app` bundle** is mandatory-in-practice, not just tidy — TCC mis-handles bare (non-bundled) executables for Screen Recording (they may not appear in the Screen Recording settings list and can be declined even with a `auth_value=2` row), so a real bundle attributed to the responsible process is the correct shape. So: ship `NucleicVMAgent.app` with a `CFBundleIdentifier`, an `NSScreenCaptureUsageDescription`, and **hardened runtime** (`codesign --options runtime`). **Sign with a stable identity** (Developer ID, or a self-signed cert baked into the image) — **not ad-hoc**: TCC keys the grant to the app's *designated requirement* (the `csreq` blob), and an ad-hoc signature changes identity every build, so a pre-inserted TCC row stops matching after a rebuild. **Notarization is not required** (the app is never quarantined inside the guest); strip any quarantine xattr (`xattr -dr com.apple.quarantine`) when baking it in. Install to e.g. `/Applications/NucleicVMAgent.app` + a LaunchAgent at `/Library/LaunchAgents/xyz.blakeslee.nucleic.vmagent.plist` (`ProgramArguments` → the bundle's `Contents/MacOS/NucleicVMAgent`, `RunAtLoad`, `KeepAlive`, `LimitLoadToSessionType=Aqua`, `AssociatedBundleIdentifiers`) so it runs in the auto-login Aqua session. This extends `scripts/provision-macos-guest.sh` Phase 7. The TCC pre-grant rows use `client_type=0` (bundle id), `auth_value=2`, and the app's `csreq` blob (`codesign -d -r- … | csreq -r- -b …`) in the **system** `TCC.db`; reboot so `tccd` reloads. - **macOS 27 note:** on the supported macOS 27 host+guest, `VZMacGuestProvisioningOptions` (gated `#available(macOS 27, *)`) injects the account + enables auto-login declaratively during the one-click base build, and the same pass writes this agent's TCC grants — so there is no manual Setup-Assistant or SIP step. See [MACOS_VM](MACOS_VM.md) §4.4. --- ## 12. Fit with the existing code (as built) - **`guest/NucleicVMAgent`** — the agent, as its own SwiftPM **package** (not a target of the main one): it's a separate package because it runs inside the guest and SwiftPM has no per-target platform floors. Both it and the main package floor at macOS 27. Targets: `CVsock` (C shim — Swift's Darwin overlay doesn't surface `sockaddr_vm`), `VMAgentCore` (pure wire/keymap logic, unit-tested), and the `NucleicVMAgent` executable (`ApplicationServices`, `ScreenCaptureKit`, `CoreGraphics`, `AppKit`). `scripts/build-vm-agent.sh` wraps it into the signed `.app` (+ the LaunchAgent plist). - **`MacVMEngine`** — `buildConfiguration` attaches one `VZVirtioSocketDeviceConfiguration`; `MacVMAgentClient.swift` holds the NDJSON client (an actor over the vsock fd), the wire constants (`MacVMAgentWire`, port 2035 — mirrored in `VMAgentCore.AgentWire`), and the `ping`-probe capability cache per live VM (`LiveVM.agent`: unprobed → available/unavailable; a transport failure re-probes once, an absent agent stays absent for the VM's lifetime). - **`MacVMEngine+ComputerAgent`** — routes `performComputerAction` through the agent when present (action ran natively ⇒ never re-run over SSH; capture-only fallbacks after), SSH/cliclick otherwise; implements the `ax_*` ops (no SSH fallback for those). - **`MCPApprovalServer`** — the `mac_vm_computer` schema carries the AX actions + `ref`/`value` fields; the AX tree returns as text content (and screenshots as image content, as before). - **`scripts/provision-macos-guest.sh`** — Phase 7d installs the agent app + LaunchAgent + its three TCC grants (bundle-id-keyed rows), written automatically during the macOS 27 base build; it skips gracefully when the app isn't staged. - **Shipped embedded + auto-installed.** `scripts/package-app.sh` bakes the signed `NucleicVMAgent.app` (+ its LaunchAgent plist + the provisioner) into the host app under `Contents/Resources/macvm/` (signed inside-out, so its designated requirement is stable across builds — the TCC rows key on it). The one-click **Build base image** then stages it into the guest and runs Phase 7d over SSH (`MacVMEngine+Provision.swift`, resolved via `resolveVMAgentApp()`; dev/SwiftPM runs fall back to building it from source). No separate `scripts/build-vm-agent.sh` step for the operator. - **`MacVMSettings`** — no new toggle: the agent is auto-preferred when present (the ping probe is the switch), and the SSH path engages transparently otherwise. Whether the base *installs* the agent is gated on the existing computer-use opt-in. --- ## 13. Rollout plan (phased, each independently landable) 1. ✅ **Transport core** — `NucleicVMAgent` skeleton with the vsock listener + `ping`/`screenshot`; host `VZVirtioSocketDevice` config + client. *Code landed; the real-guest round-trip that proves §3.2 is still owed (see 5).* SSH path stays the default until this is proven. 2. ✅ **Input + capture over the agent** — `click`/`move`/`type`/`key`/`scroll` (real scroll) + SCK `screenshot`; `mac_vm_computer` prefers the agent, SSH fallback intact. 3. ✅ **Accessibility** — `ax_dump`/`ax_element_at`/`ax_action`/`ax_set_value` + the tool-schema actions + model guidance. This is where the semantic + framebuffer-independent wins land. 4. ✅ **Provisioning + base image** — the agent, LaunchAgent, and TCC grants in `provision-macos-guest.sh` Phase 7d (agent app staged from `scripts/build-vm-agent.sh` output). 5. ⬜ **Validate on a real guest** — smoke-test the vsock round-trip (§3.2), then confirm AX observe+control works on a macOS 27 guest (§9). --- ## 14. Open questions / verify-on-real-VM - **Guest-side vsock `listen()` on a macOS guest** + host `connect(toPort:)` — strongly inferred, not single-doc-confirmed (§3.2). Smoke-test; reverse topology is the fallback. - **AX + CGEvent behaviour** on a real macOS 27 guest — architecture confirmed (§9), but validate on the target build. - **AX ref lifetime / staleness** policy — dump-then-act TTL vs. re-resolving handles. - **`SCScreenshotManager` availability** (macOS 14+) and behaviour on the target guest. - **Non-AppKit app coverage** (Electron/Chromium/Java) — may need per-app AX-enable toggles; document the gaps. --- ## 15. Cross-references - [MACOS_VM](MACOS_VM.md) — the macOS-VM subsystem this augments (§12 computer-use, §4.4 guest versions, §12.5 TCC reality). - [VSOCK_CONTROL_PLANE](VSOCK_CONTROL_PLANE.md) — Nucleic's existing vsock control plane for the Linux container (the pattern this reuses for the macOS VM). - Apple: `VZVirtioSocketDevice` / `VZVirtioSocketConnection` / `VZVirtioSocketDeviceConfiguration`; `AXUIElement` (ApplicationServices/HIServices); `ScreenCaptureKit` (`SCScreenshotManager`); `CGEvent`.