nucleic/nucleic
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
27311320b20ea41bc2878a7b57bed8864a99ca7d
nucleic/docs/MACOS_VM_NATIVE_AGENT.md
T

29 KiB
Raw Blame History

Nucleic — Native in-guest automation agent (design)

Superseded as the default; now an optional add-on. Since the host-side virtual-IO path landed (MACOS_VM.md §12.2 — capture the guest framebuffer via VZVirtualMachineView + inject NSEvent HID, all host-side with no guest TCC), that is the default computer-use path. This native agent is retained as an opt-in (MacVMSettings.axAgentEnabled, OFF by default) for its enduring win: semantic, framebuffer-independent Accessibility control (ax_* — read the accessibility tree, act on an element by identity), which is more robust than pixel targeting and lets you inspect an app's AX tree when debugging. Only this agent needs the guest's TCC grants — which the supported macOS 27 one-click base build writes automatically (no manual step; MACOS_VM.md §12.5). The pixel ops it also implements are now redundant with the host-side surface.

A native in-guest agent that observes and controls a macOS guest VM through Apple's frameworks directly — the Accessibility API (AXUIElement) for semantic control, ScreenCaptureKit for capture, CGEvent for raw input — reached from the host over vsock. It replaces the current mac_vm_computer transport (SSH → launchctl asuser → screencapture/cliclick → base64), which is a stack of shims over those same frameworks.

Status: implemented through §13 phases 1–4 (code + provisioning), pending real-guest validation (§13 phase 5 / §14). What exists: the guest agent (guest/NucleicVMAgent — its own SwiftPM package, macOS 27 floor — vsock listener, all §4 ops), the host side (MacVMAgentClient + the VZVirtioSocketDevice config + the ping-probe/fallback routing in MacVMEngine+ComputerAgent), the extended mac_vm_computer tool surface (ax_* actions + ref/value), the app-bundle build (scripts/build-vm-agent.sh), and provisioning Phase 7d (scripts/provision-macos-guest.sh). The empirical claims to validate on a real guest (guest-side vsock listen, and AX/CGEvent behaviour) are called out in §9 and §10 — until then the SSH path remains the proven default and the agent is strictly additive. Companion to MACOS_VM (which this augments) and VSOCK_CONTROL_PLANE (Nucleic's existing vsock work for the Linux container).

1. Why — the CLI shims are the wrong layer

mac_vm_computer today (see MACOS_VM §12) reaches the guest over SSH and drives it with CLI tools that are themselves thin wrappers over Apple frameworks:

Concern Today (janky) Native agent
Input cliclick over SSH, launchctl asuser, coordinate clicks Accessibility API (AXUIElement) — act on a control by identity; CGEvent for raw fallback
Observe screencapture CLI + base64 over SSH AXUIElement tree (semantic) + ScreenCaptureKit (pixels), in-process
Transport SSH over NAT + /var/db/dhcpd_leases IP discovery vsock — a VM device channel, no NAT/DHCP/SSH, no macOS network prompts

Three concrete wins, not just tidiness:

  1. Semantic control. Instead of "the model looks at a screenshot, guesses (x, y), and clicks blind," the agent reads the accessibility tree — every control's role, title, value, frame, and available actions — and acts on an element by identity (AXPress a button, set a field's AXValue). This is how professional macOS UI automation works, and it's dramatically more reliable than pixel targeting.

  2. The Accessibility tree is framebuffer-independent — a robustness property of AX (§9). The AX tree is vended by AppKit from the app's semantic view model, in-process, independent of compositing and of any pixel readback — so an AX-driven agent observes and drives app UIs from the app's own state rather than from a captured image. This makes control robust even when a capture API returns nothing useful (a compositor hiccup, an off-screen or occluded window), and it's what lets you inspect an app's live AX tree when debugging. It's fundamentally more reliable than guessing coordinates off a screenshot.

  3. vsock, not SSH. No NAT/DHCP/lease-parsing, no SSH keys, no launchctl asuser gymnastics, no macOS incoming-connection prompts — a VM device channel handled by Virtualization.framework (docs/VSOCK_CONTROL_PLANE.md moved the Linux container's control plane onto vsock for the same reason).

What it does not solve (honesty): TCC is still required — the agent needs Accessibility and Screen Recording grants (§8); an active Aqua/GUI session is still required (AX, SCK and CGEvent all need WindowServer), so auto-login and a per-user LaunchAgent stay.

2. Architecture

  Claude agent
     │ calls the mac_vm_computer tool (now with AX-semantic actions — §7)
     ▼
  MCPApprovalServer ──▶ ClaudeCodeBackend.handleMacVMComputerCall
     │                       │
     │                       ▼
     │                 MacVMManager ──▶ MacVMEngine
     │                                     │  preferred: native agent over vsock
     │                                     │  fallback:  SSH + screencapture/cliclick (today's path, §10)
     │                                     ▼
     │                          VZVirtioSocketDevice.connect(toPort:)      ── host end of vsock ──┐
     ▼                                                                                            │
  (image + AX tree flow back up as MCP content the model sees)                                    ▼
                                                          ┌──── macOS guest (the clone) ──────────────┐
                                                          │  NucleicVMAgent  (LaunchAgent, Aqua session)│
                                                          │   AF_VSOCK listener :port                   │
                                                          │   • AXUIElement   (observe + control)       │
                                                          │   • ScreenCaptureKit (capture)              │
                                                          │   • CGEvent        (raw mouse/key/scroll)   │
                                                          └─────────────────────────────────────────────┘

The native path and the SSH/CLI path coexist: the engine prefers the agent (a vsock ping succeeds), and transparently falls back to today's SSH path when the agent isn't present or reachable, so nothing regresses and the base image can adopt the agent incrementally.

3. Transport — vsock

Apple's Virtualization framework gives the host a direct socket into the guest, no IP stack involved.

3.1 Host side (MacVMEngine)

  • Config: add exactly one VZVirtioSocketDeviceConfiguration() to config.socketDevices in buildConfiguration (currently the mac-VM config has no socket device).
  • Runtime: after boot, on the VM's serial queue, vm.socketDevices.first as? VZVirtioSocketDevice.
  • Connect: func connect(toPort: UInt32) async throws -> VZVirtioSocketConnection. Apple: "If the guest operating system doesn't listen for connections to the specified port, this method does nothing" — so the guest must be listening first (the agent's listener, §5). The returned VZVirtioSocketConnection exposes a real POSIX fileDescriptor (write to send, read to receive; it becomes -1 when closed) and a close(). Wrap the fd in DispatchIO (idiomatic for a socket) or a nonblocking DispatchSource; keep a strong reference to the connection while I/O is in flight.
  • One device per VM. A VZVirtioSocketDevice on macOS can't be told which VM connected on the host side (host CID is fixed at 2, guests at 3), but since Nucleic owns one device per VM instance, that's moot.

3.2 Guest side (NucleicVMAgent)

Standard BSD vsock server (#include <sys/vsock.h>; AF_VSOCK is available inside a macOS guest — on the host socket(AF_VSOCK, …) fails ENODEV, which is why the host reaches it only via VZVirtioSocketDevice):

int s = socket(AF_VSOCK, SOCK_STREAM, 0);
struct sockaddr_vm addr = {0};
addr.svm_len    = sizeof(addr);      // macOS-specific BSD length field (Linux has none)
addr.svm_family = AF_VSOCK;
addr.svm_cid    = VMADDR_CID_ANY;    // listen on any CID
addr.svm_port   = NUCLEIC_AGENT_PORT;
bind(s, (struct sockaddr *)&addr, sizeof(addr));
listen(s, 4);
int conn = accept(s, NULL, NULL);    // the host's connect(toPort:) lands here
  • Port: a fixed Nucleic-reserved port (e.g. 2035), a shared constant on both sides.
  • ⚠ Verify on a real guest (§10): Apple's Virtualization docs describe the host-connects / guest-listens topology in guest-OS-agnostic terms, and <sys/vsock.h> exists on macOS, but no single Apple doc names "macOS guest listen() + host connect(toPort:)" — smoke-test it on the target guest build before building on top. (Linux guests are proven; the macOS-guest listen path is the one inference.) If it doesn't hold, the reverse topology — host VZVirtioSocketListener + guest connects to VMADDR_CID_HOST — is the documented fallback.

3.3 Framing

Newline-delimited JSON (NDJSON): one request object per line, one reply per line — the same framing Nucleic already uses for agent stdio (LineSplitter). Screenshots ride as base64 in the JSON (simple, matches the existing image path); a length-prefixed binary frame is a future efficiency option if base64 overhead on a screenshot→act loop matters.

4. Wire protocol (host ⇄ agent)

Requests carry an op; replies carry ok plus op-specific fields (or ok:false, error). Handles to UI elements are opaque ref strings the agent assigns (§6.2).

op Request fields Reply Backed by
ping — {ok, version, displays:[…]} capability negotiation
screenshot display?, format:"jpeg" {ok, image:<b64>, width, height, scale} ScreenCaptureKit (§7)
ax_dump target:"frontmost"|pid, maxDepth? {ok, tree:<node>} AXUIElement (§6)
ax_element_at x, y {ok, node} AXUIElementCopyElementAtPosition
ax_action ref, action:"AXPress"|… {ok} AXUIElementPerformAction
ax_set_value ref, value {ok} AXUIElementSetAttributeValue
ax_focus ref {ok} set kAXFocusedAttribute
click x, y, button?, count? {ok} CGEvent (§8)
move x, y {ok} CGEvent
drag fromX,fromY,toX,toY {ok} CGEvent
type text {ok} CGEvent keyboard
key chord:"cmd+s" {ok} CGEvent + flags
scroll dx, dy, x?, y? {ok} CGEvent scroll wheel (real scroll, unlike the cliclick arrow-key hack). dx/dy are the gesture TOTAL, delivered as a paced stream of 2-line notches the way hardware reports one — a single outsized event is coalesced down to ~one notch by any app that animates its scrolling. x/y move the pointer first: a wheel event routes by pointer position, not by coordinate
launch_app name {ok} NSWorkspace.openApplication

A node is:

{ "ref": "e17", "role": "AXButton", "subrole": "AXCloseButton", "title": "Build",
  "value": null, "enabled": true, "focused": false,
  "frame": { "x": 812, "y": 44, "w": 74, "h": 22 },
  "actions": ["AXPress", "AXShowMenu"],
  "children": [ … ] }

4.4 Streaming exec (protocol v2) — the vsock replacement for SSH exec

exec is the one streaming op and the transport for mac_vm_exec (retiring SSH-over-NAT). Unlike every request/reply op above, it takes over its connection for the life of the command, so the host opens a dedicated vsock connection per exec (the listener is one thread per connection → concurrent execs + the control channel never interfere).

  • Request (one line): {"op":"exec","command":<shell script>}. The host composes env exports + cd <workdir> + the body into command and the guest runs it under /bin/zsh -c (so /etc/zshenv's toolchain PATH resolves), exactly as sshd used to invoke it.
  • Frames (one JSON object per line, tagged by t), guest→host: {"t":"o","d":<b64 stdout>}, {"t":"e","d":<b64 stderr>}, then terminal {"t":"x","code":<int>} (or {"t":"err","m":<msg>} if the spawn failed). Payload bytes are base64 so arbitrary binary output can't break the NDJSON framing.
  • Frames, host→guest: {"t":"i","d":<b64 stdin>}, {"t":"eof"}, {"t":"sig","n":<signal>}.

Host side is MacVMExecChannel (a ProcessHandle that re-splits the base64 chunks into newline lines); guest side is ConnectionHandler.runExec (spawns the process, pumps its pipes as frames, polls the connection for input frames). Constants are mirrored in MacVMAgentWire.Exec (host) and VMAgentCore.AgentWire.Exec (guest) — keep them in lockstep. The protocol version is bumped to 2 because a v1 agent doesn't speak exec.

5. The in-guest agent — NucleicVMAgent

A small Swift program, packaged as a signed .app bundle (the correct TCC shape — §8), run as a per-user LaunchAgent under the agent account so it lives in the Aqua GUI session (AX, SCK and CGEvent all require an active WindowServer session). Responsibilities:

  • Bind the AF_VSOCK listener (§3.2); accept loop; per-connection NDJSON request loop.
  • Implement the §4 ops with ApplicationServices/AXUIElement (§6), ScreenCaptureKit (§7), and CoreGraphics/CGEvent (§8).
  • On start, self-check AXIsProcessTrusted() and Screen-Recording authorization; report readiness in the ping reply so the host can surface a clear "agent present but not TCC-granted" state.
  • LSUIElement (no Dock icon). Small, single-purpose, no network egress.

It is not an MCP server itself — it speaks the private §4 protocol to MacVMEngine, which bridges it to the mac_vm_computer MCP tool. The host keeps ownership of VM lifecycle, gating, and the approval boundary; the agent is a dumb, well-scoped actuator.

6. Accessibility — observe + control

Framework: ApplicationServices (re-exports HIServices). All calls are synchronous cross-process IPC into the target app, governed by AXUIElementSetMessagingTimeout (default ~6 s).

6.1 Reading the tree

  • Root: AXUIElementCreateApplication(pid) for the frontmost app (NSWorkspace.shared.frontmostApplication?.processIdentifier), or AXUIElementCreateSystemWide() for hit-testing.
  • Per node: AXUIElementCopyAttributeValue for kAXRole, kAXSubrole, kAXTitle, kAXValue, kAXEnabled, kAXFocused; kAXPosition/kAXSize come back as an AXValueRef box — unwrap with AXValueGetValue(_, .cgPoint/.cgSize, _) (AX uses top-left-origin screen coords). Actions via AXUIElementCopyActionNames. Recurse kAXChildren.
  • Cost control: a full walk of a rich app is thousands of IPC round-trips. Batch with AXUIElementCopyMultipleAttributeValues, cap maxDepth, dedupe by element identity, and set a tight messaging timeout so a hung app can't wedge the walker.

6.2 Element handles (ref)

AXUIElementRefs go stale as the UI mutates, so the agent keeps a per-connection registry mapping a short ref id → the live AXUIElementRef it handed out in the last ax_dump/ax_element_at. Refs are valid until the next dump (or a short TTL); the host's convention is dump → act on a ref → dump again. A stale ref returns kAXErrorInvalidUIElement, which the host surfaces as "re-dump and retry."

6.3 Control

  • Press/menu: AXUIElementPerformAction(el, kAXPressAction) (also kAXShowMenu, kAXConfirm, kAXCancel, kAXIncrement/kAXDecrement). Treat kAXErrorCannotComplete as possibly succeeded — apps often run modal processing inside the action callback and don't return within the AX timeout.
  • Type into a field: set kAXFocused = true, then kAXValue = "<text>" (check AXUIElementIsAttributeSettable first). Some non-AppKit fields reject AXValue writes — fall back to focusing the element then CGEvent keystrokes (§8).
  • Prefer AX actions over CGEvent coordinate clicks wherever a control exposes an action: AX drives the app in-process and is fully framebuffer-independent (§9); a CGEvent click at kAXPosition is closer to the pixel world (still geometry-correct, but less robust).

7. Screen capture — ScreenCaptureKit

For pixels: SCShareableContent.current → SCContentFilter (full display or a specific SCWindow) → SCScreenshotManager.captureImage(contentFilter:configuration:) (macOS 14+) → CGImage → JPEG. In-process, no screencapture CLI, no launchctl asuser, no base64-over-SSH. Needs Screen Recording TCC + an Aqua session. Capture depends on the compositor rendering the window, so AX (§6), not SCK, is the robust "look" primitive whenever a control exposes a semantic identity — pixels are the fallback for custom-drawn UI.

8. Low-level input — CGEvent

When AX has no action for a target (custom-drawn UI): CGEvent(mouseEventSource:mouseType:…) + .post(tap: .cghidEventTap) for clicks/moves/drags at screen coordinates; CGEvent(keyboardEventSource: virtualKey:keyDown:) with .flags for keystrokes and chords; CGEvent(scrollWheelEvent2Source:…) for real scroll (a proper wheel event — the current cliclick path fakes scroll with arrow keys). CGEvent needs Accessibility TCC and the Aqua session. Note CGEvent clicks target pixel coordinates, so they lean on window geometry being correct but are less framebuffer-independent than AX actions; prefer AX.

9. Framebuffer independence (the enduring robustness property — HIGH confidence)

The Accessibility tree is architecturally independent of the rendered framebuffer: AppKit vends it from the view's semantic model (roles/titles/values inferred from NSAccessibility conformance), accessibilityFrame is a layout-geometry computation (not a pixel readback), and AXUIElementPerformAction invokes the control's action callback in the app process — no rendering, no synthetic on-screen click required. NSAccessibilityElement can even back UI with no NSView at all, proving AX is decoupled from any drawn surface.

Consequently AX observe+control is driven by the app's own state, not by a captured image. It stays correct where a capture API is unreliable — an occluded or off-screen window, a compositor hiccup — because CGWindowListCopyWindowInfo bounds and the window server's geometry remain valid even when the pixels don't. That decoupling is exactly what makes semantic control more robust than pixel targeting.

Caveats (none framebuffer-related, but real): lazy/virtualized views (table cells) may be absent from the tree until laid out; non-AppKit apps (Electron/Chromium, Java, GL/game UIs) expose poor or empty AX trees regardless of platform — Chromium needs AXManualAccessibility/--force-renderer- accessibility, Electron app.setAccessibilitySupportEnabled(true); and some toolkits return kAXErrorCannotComplete/kAXErrorAPIDisabled intermittently — budget retries and agent-restart handling. Still verify empirically on a real guest — the architecture is confirmed but should be smoke-tested.

10. Tool surface & fallback

mac_vm_computer gains AX-semantic actions alongside today's pixel actions (MACOS_VM §12.1):

  • ax_dump → returns the front app's element tree (roles/titles/values/frames/actions) as text the model reads — the semantic "look," and the framebuffer-independent one. Optionally also returns a screenshot.
  • ax_press / ax_set_value / ax_focus (by ref) → act on a control by identity.
  • ax_element_at (x,y) → resolve the element under a point.
  • Existing screenshot / left_click / type / key / scroll — now routed through the agent's ScreenCaptureKit/CGEvent when present (cleaner, and scroll becomes real).

Guidance to the model: prefer ax_dump + act-by-ref; fall back to screenshot + pixel click when a control has no AX action or isn't in the tree.

Fallback / capability negotiation. MacVMEngine tries a vsock ping when it needs the agent; on success it routes ops natively, on failure (no agent baked in, not launched, not TCC-granted, or vsock listen unsupported) it uses today's SSH + screencapture/cliclick path unchanged. Per-op fallback is possible (e.g. native input but SSH screenshot) but the simplest first cut is all-or-nothing by ping. AX-semantic ops have no SSH equivalent, so ax_dump/ax_* are only advertised when the agent is present.

11. TCC, packaging, provisioning

  • TCC: the agent needs three distinct grants, each a separate TCC.db service: kTCCServiceAccessibility (AXUIElement read/control — AXIsProcessTrusted), kTCCServicePostEvent (CGEvent input synthesis — surfaced under the "Accessibility" list but a distinct row; CGPreflightPostEventAccess/CGRequestPostEventAccess), and kTCCServiceScreenCapture (ScreenCaptureKit — CGPreflight/CGRequestScreenCaptureAccess). A proper signed .app bundle with a native Mach-O main executable is the correct shape (TCC mis-handles script-based executables, and recent macOS wants a real bundle for Screen Recording, attributed to the responsible process) — an improvement over granting a bare CLI. Pre-grant all three to the agent's bundle-id / code requirement via the same direct system-TCC.db write the provisioner does automatically on the supported macOS 27 base build (MACOS_VM §12.5); no other unattended path exists (PPPC can't silently grant Screen Recording; tccutil can't grant). Note a grant is often not picked up until the process restarts (esp. Screen Recording), so the LaunchAgent should be bounced after provisioning writes the rows. If a grant is missing at runtime the agent reports it in the ping reply (via AXIsProcessTrusted + the CGPreflight* checks) so the host can surface an actionable "agent present but not authorized" state rather than silently failing.
  • Packaging: a real signed .app bundle is mandatory-in-practice, not just tidy — TCC mis-handles bare (non-bundled) executables for Screen Recording (they may not appear in the Screen Recording settings list and can be declined even with a auth_value=2 row), so a real bundle attributed to the responsible process is the correct shape. So: ship NucleicVMAgent.app with a CFBundleIdentifier, an NSScreenCaptureUsageDescription, and hardened runtime (codesign --options runtime). Sign with a stable identity (Developer ID, or a self-signed cert baked into the image) — not ad-hoc: TCC keys the grant to the app's designated requirement (the csreq blob), and an ad-hoc signature changes identity every build, so a pre-inserted TCC row stops matching after a rebuild. Notarization is not required (the app is never quarantined inside the guest); strip any quarantine xattr (xattr -dr com.apple.quarantine) when baking it in. Install to e.g. /Applications/NucleicVMAgent.app + a LaunchAgent at /Library/LaunchAgents/xyz.blakeslee.nucleic.vmagent.plist (ProgramArguments → the bundle's Contents/MacOS/NucleicVMAgent, RunAtLoad, KeepAlive, LimitLoadToSessionType=Aqua, AssociatedBundleIdentifiers) so it runs in the auto-login Aqua session. This extends scripts/provision-macos-guest.sh Phase 7. The TCC pre-grant rows use client_type=0 (bundle id), auth_value=2, and the app's csreq blob (codesign -d -r- … | csreq -r- -b …) in the system TCC.db; reboot so tccd reloads.
  • macOS 27 note: on the supported macOS 27 host+guest, VZMacGuestProvisioningOptions (gated #available(macOS 27, *)) injects the account + enables auto-login declaratively during the one-click base build, and the same pass writes this agent's TCC grants — so there is no manual Setup-Assistant or SIP step. See MACOS_VM §4.4.

12. Fit with the existing code (as built)

  • guest/NucleicVMAgent — the agent, as its own SwiftPM package (not a target of the main one): it's a separate package because it runs inside the guest and SwiftPM has no per-target platform floors. Both it and the main package floor at macOS 27. Targets: CVsock (C shim — Swift's Darwin overlay doesn't surface sockaddr_vm), VMAgentCore (pure wire/keymap logic, unit-tested), and the NucleicVMAgent executable (ApplicationServices, ScreenCaptureKit, CoreGraphics, AppKit). scripts/build-vm-agent.sh wraps it into the signed .app (+ the LaunchAgent plist).
  • MacVMEngine — buildConfiguration attaches one VZVirtioSocketDeviceConfiguration; MacVMAgentClient.swift holds the NDJSON client (an actor over the vsock fd), the wire constants (MacVMAgentWire, port 2035 — mirrored in VMAgentCore.AgentWire), and the ping-probe capability cache per live VM (LiveVM.agent: unprobed → available/unavailable; a transport failure re-probes once, an absent agent stays absent for the VM's lifetime).
  • MacVMEngine+ComputerAgent — routes performComputerAction through the agent when present (action ran natively ⇒ never re-run over SSH; capture-only fallbacks after), SSH/cliclick otherwise; implements the ax_* ops (no SSH fallback for those).
  • MCPApprovalServer — the mac_vm_computer schema carries the AX actions + ref/value fields; the AX tree returns as text content (and screenshots as image content, as before).
  • scripts/provision-macos-guest.sh — Phase 7d installs the agent app + LaunchAgent + its three TCC grants (bundle-id-keyed rows), written automatically during the macOS 27 base build; it skips gracefully when the app isn't staged.
  • Shipped embedded + auto-installed. scripts/package-app.sh bakes the signed NucleicVMAgent.app (+ its LaunchAgent plist + the provisioner) into the host app under Contents/Resources/macvm/ (signed inside-out, so its designated requirement is stable across builds — the TCC rows key on it). The one-click Build base image then stages it into the guest and runs Phase 7d over SSH (MacVMEngine+Provision.swift, resolved via resolveVMAgentApp(); dev/SwiftPM runs fall back to building it from source). No separate scripts/build-vm-agent.sh step for the operator.
  • MacVMSettings — no new toggle: the agent is auto-preferred when present (the ping probe is the switch), and the SSH path engages transparently otherwise. Whether the base installs the agent is gated on the existing computer-use opt-in.

13. Rollout plan (phased, each independently landable)

  1. ✅ Transport core — NucleicVMAgent skeleton with the vsock listener + ping/screenshot; host VZVirtioSocketDevice config + client. Code landed; the real-guest round-trip that proves §3.2 is still owed (see 5). SSH path stays the default until this is proven.
  2. ✅ Input + capture over the agent — click/move/type/key/scroll (real scroll) + SCK screenshot; mac_vm_computer prefers the agent, SSH fallback intact.
  3. ✅ Accessibility — ax_dump/ax_element_at/ax_action/ax_set_value + the tool-schema actions + model guidance. This is where the semantic + framebuffer-independent wins land.
  4. ✅ Provisioning + base image — the agent, LaunchAgent, and TCC grants in provision-macos-guest.sh Phase 7d (agent app staged from scripts/build-vm-agent.sh output).
  5. ⬜ Validate on a real guest — smoke-test the vsock round-trip (§3.2), then confirm AX observe+control works on a macOS 27 guest (§9).

14. Open questions / verify-on-real-VM

  • Guest-side vsock listen() on a macOS guest + host connect(toPort:) — strongly inferred, not single-doc-confirmed (§3.2). Smoke-test; reverse topology is the fallback.
  • AX + CGEvent behaviour on a real macOS 27 guest — architecture confirmed (§9), but validate on the target build.
  • AX ref lifetime / staleness policy — dump-then-act TTL vs. re-resolving handles.
  • SCScreenshotManager availability (macOS 14+) and behaviour on the target guest.
  • Non-AppKit app coverage (Electron/Chromium/Java) — may need per-app AX-enable toggles; document the gaps.

15. Cross-references

  • MACOS_VM — the macOS-VM subsystem this augments (§12 computer-use, §4.4 guest versions, §12.5 TCC reality).
  • VSOCK_CONTROL_PLANE — Nucleic's existing vsock control plane for the Linux container (the pattern this reuses for the macOS VM).
  • Apple: VZVirtioSocketDevice / VZVirtioSocketConnection / VZVirtioSocketDeviceConfiguration; AXUIElement (ApplicationServices/HIServices); ScreenCaptureKit (SCScreenshotManager); CGEvent.
Reference in New Issue View Git Blame Copy Permalink