nucleic/docs/LINUX_VM_AX_AGENT.md

Nucleic — Linux in-guest semantic (AT-SPI) control agent (design)

Status: proposed. Today a Linux guest's computer-use is 100% host-side pixels (virtio-gpu capture + USB HID injection); the semantic ax_* ops short-circuit for Linux with a "use screenshot

• pixel actions" message (Sources/NucleicCore/MacVM/MacVMEngine+Computer.swift). This doc plans the Linux counterpart to the macOS native AX agent (MACOS_VM_NATIVE_AGENT.md): a semantic, framebuffer-independent control path that reads the accessibility tree and acts on controls by identity, using Linux's AT-SPI2 accessibility framework.

The macOS agent (guest/NucleicVMAgent) drives the guest through Apple's Accessibility API (AXUIElement) — read every control's role/title/value/frame/actions, then AXPress / set AXValue / set focus by identity. That is dramatically more robust than guessing (x, y) off a screenshot, and it keeps working when a capture API returns nothing useful. This doc brings the same capability to Linux guests via AT-SPI2, the de-facto Linux accessibility standard, slotting into the vsock + NDJSON + protocol-v2 transport and the ax_* tool surface Nucleic already ships.

Companion to LINUX_VM.md (the Linux guest subsystem this augments) and MACOS_VM_NATIVE_AGENT.md (the macOS agent this mirrors).

---

1. Why — semantic control, minus the macOS tax

The macOS agent's three wins (semantic control, framebuffer independence, vsock transport — see MACOS_VM_NATIVE_AGENT.md §1) apply verbatim to Linux, and Linux is easier on two axes:

• No permission model. macOS needs three TCC grants (kTCCServiceAccessibility, kTCCServicePostEvent, kTCCServiceScreenCapture) pre-written into TCC.db, plus a signed .app bundle and an Aqua session — the single hardest part of the macOS agent. AT-SPI needs none of it: just the AT-SPI D-Bus bus running in the session and the toolkit a11y bridges enabled.
• Input is already solved host-side. The Wayland/X11 community's biggest a11y pain — synthesizing input — does not bite us: clicks/keystrokes go in via host-side USB HID (VZUSBKeyboardConfiguration + VZUSBScreenCoordinatePointingDeviceConfiguration), not via the display server. So this agent's job is narrower than a general Linux automation tool: read the tree and act by identity. For actions it prefers AT-SPI in-process actions (coordinate-free); the pixel path (host-side) remains the fallback.

2. The macOS ↔ Linux mapping

AT-SPI2 sits between Windows UIA's rigid structures and macOS AX's flexibility: enum roles, but untyped action strings — the same shape as AX. The ax_* wire ops map cleanly:

Wire op (unchanged)	macOS backing (Ops+AX.swift)	Linux backing (AT-SPI2)
ax_dump	AXUIElementCopyAttributeValue (role/title/value/frame/actions), recurse kAXChildren	Atspi.Accessible tree: get_role_name, get_name, AtspiValue, AtspiComponent.get_extents, AtspiAction action names; recurse get_child_at_index
ax_element_at	AXUIElementCopyElementAtPosition	AtspiComponent.get_accessible_at_point(ATSPI_COORD_TYPE_SCREEN)
ax_press (action)	AXUIElementPerformAction("AXPress"/…)	AtspiAction.do_action(i) (map action name → index)
ax_set_value	AXUIElementSetAttributeValue(kAXValue)	AtspiEditableText.set_text_contents / AtspiValue.set_current_value
ax_focus	set kAXFocused	AtspiComponent.grab_focus
ping readiness	AXIsProcessTrusted() + CGPreflight*	AT-SPI bus reachable? (report atspi: true/false)

The node JSON is identical to macOS (§4 of MACOS_VM_NATIVE_AGENT.md): ref, role, subrole?, title, value, enabled, focused, frame{x,y,w,h}, actions[], children[]. The per-connection ref registry and dump→act→dump staleness contract (§6.2 there) carry over unchanged, so the host side (MacVMEngine+ComputerAgent, MCPApprovalServer schema) needs no protocol changes — only to stop short-circuiting Linux.

3. Display server — X11, not Wayland (decision)

The Linux base currently runs a Wayland session (labwc compositor via greetd auto-login, provision-linux-guest.sh). We are switching the automation base to X11.

Rationale: the AT-SPI tree is vended by the toolkit over D-Bus and is compositor-agnostic (tree reads and action-by-identity work under either), but global screen coordinates are the weak spot on Wayland. Wayland deliberately hides absolute window positions from clients, so AtspiComponent.get_extents(SCREEN) returns window-relative or unreliable coordinates — precisely the gap the GNOME "Newton" project is still closing in 2025-26. That coordinate is exactly what the pixel-fallback path needs (element → (x, y) for the host to click), and it's what lets ax_element_at hit-test correctly. Under X11, AT-SPI coordinate reporting is fully mature and global geometry is trivial. Since Nucleic builds the base image, choosing X11 for the automation guest sidesteps the entire Wayland-a11y-immaturity problem at the cost of a compositor swap we control.

The X11 session (see §6): Xorg (modesetting driver on virtio-gpu) + a minimal WM (openbox) + an auto-login greeter, with at-spi2-core + the GTK/Qt a11y bridges enabled. Host-side capture/HID are unchanged — virtio-gpu scanout and USB HID don't care whether X11 or Wayland is compositing.

4. In-guest agent shape — split, don't rewrite

guest/nucleic-linux-agent/src/agent.c is a dependency-free static C binary carrying ping + exec, started early by systemd/nucleic-linux-agent.service. Hand-rolling AT-SPI's D-Bus marshalling in zero-dependency C would be miserable, and we do not want to add heavy deps to the boot-critical exec path. So:

• Keep agent.c as-is for ping/exec — static, zero-dep, early boot. Extend only its ping reply to advertise AT-SPI availability ("atspi": true|false).
• Add a separate a11y helper linked against the real bindings — libatspi-2.0 + GLib (C) or a small python3-pyatspi process. It is apt-installed in the provision phase and runs inside the graphical session (it needs the session's D-Bus + AT_SPI_BUS). The C agent forwards ax_* op lines to the helper (a local unix socket or a spawned request/response), or the host reaches the helper on a second vsock port. Recommended first cut: helper listens on its own vsock port, agent.c unchanged except for the readiness bit; the host tries the a11y port for ax_* and the base port for ping/exec.

Language for the helper: Python + pyatspi for the spike and likely for v1 (fast to write, the canonical AT-SPI AT-side API); revisit a static C/libatspi or Rust xa11y rewrite only if startup cost or the extra runtime deps in the base become a problem.

5. Known limitations (be honest)

• Non-GTK/Qt apps expose poor/empty AT-SPI trees — same caveat the macOS doc calls out (§9) for Electron/Java/GL. Electron/Chromium need --force-renderer-accessibility; enable QT_ACCESSIBILITY=1 / GTK_MODULES bridges by default. Terminals and custom-drawn UIs give thin trees → fall back to the (host-side) pixel path.
• AT-SPI is chatty. Every property is an individual D-Bus round-trip; a full tree walk of a rich app can take seconds. Mirror the macOS cost controls: maxDepth, a node budget (macOS caps at 800), and a per-app timeout so a hung app can't wedge the walker. Batch where the bindings allow.
• Lazy/virtualized views (list/table rows) may be absent from the tree until scrolled into view — same as AX.

6. Provisioning changes (X11 base)

provision-linux-guest.sh phase-2 first-boot install changes from the Wayland stack to X11 + AT-SPI:

• Out: labwc, foot (Wayland-only terminal), xwayland, wlr-randr, seatd, the dbus-run-session -- labwc greetd session.
• In: xserver-xorg + xserver-xorg-video-*/modesetting, openbox (WM), xterm, x11-xserver-utils; at-spi2-core + gir1.2-atspi-2.0 + python3-pyatspi + python3-gi (the a11y helper + bridges); AT-SPI env (GTK_MODULES=gail:atk-bridge, QT_ACCESSIBILITY=1, QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1, Chromium --force-renderer-accessibility default).
• greetd session: auto-login into an X session (startx/xinit launching openbox-session) instead of labwc. Autostart xterm (and the a11y helper) so a fresh screenshot isn't an empty root window.

The two-phase base build, bootstrap initramfs, vsock agent (ping/exec), host-side capture/HID, and clone lifecycle are all unchanged — this is a session-package swap plus the a11y helper.

7. Host wiring

Minimal, because the transport + tool surface already exist and are OS-neutral:

• MacVMEngine+Computer.swift — remove/relax the Linux short-circuit (currently ~L45) so ax_* route to the agent for Linux guests once ping reports atspi:true.
• MacVMEngine+ComputerAgent.swift — the ax_dump/ax_element_at/ax_press/ax_set_value/ax_focus handlers are already OS-neutral; point them at the Linux a11y helper's port when the guest is Linux.
• agent.c send_ping_reply — add "atspi" to the readiness block so the host advertises ax_* only when the helper is present.
• MCPApprovalServer — no schema change (the ax_* actions already exist).

8. Rollout plan (phased, each independently landable)

0. Spike / de-risk — ✅ DONE (guest/nucleic-linux-agent/spike/). Under a headless X (Xvfb) a GTK probe app launches and pyatspi dumps the tree into the macOS-identical node JSON, then exercises do_action (press) + set_text_contents (set value) + grab_focus (focus). Ran green on Debian 12 arm64 (same toolkit stack as the Ubuntu 24.04 guest): all three primitives passed and get_extents(SCREEN) resolved to real geometry — confirming the bindings, the node shape, and the X11 coordinate assumption. See spike/README.md.
1. X11 base — swap provision-linux-guest.sh to the X11 + AT-SPI stack (§6); rebuild + smoke-test the base boots to an X session with the AT-SPI bus up.
2. a11y helper — implement ax_dump/ax_element_at/ax_press/ax_set_value/ax_focus in the helper, emitting the macOS-identical node JSON; ref registry + staleness; cost controls (§5).
3. Host wiring — §7: stop short-circuiting Linux, route ax_* to the helper, advertise on ping.atspi.
4. Validate + iterate — real-guest reliability across GTK/Qt/Electron; document coverage gaps.

9. Open questions / verify-on-real-VM

• get_extents(SCREEN) accuracy under our X11 session — expected good on X11; confirm on the real base (this is the load-bearing assumption behind the Wayland→X11 switch).
• a11y helper transport — second vsock port vs. agent.c-forwarded unix socket. Second port is the simplest first cut; revisit if port management is awkward.
• Helper language — Python/pyatspi (v1) vs. C/libatspi or Rust xa11y (if deps/startup bite).
• Non-GTK/Qt coverage — per-app a11y enable toggles; document the gaps like the macOS doc does.

10. Cross-references

• LINUX_VM.md — the Linux guest subsystem (two-phase base build, vsock agent, host-side computer-use) this augments.
• MACOS_VM_NATIVE_AGENT.md — the macOS AX agent this mirrors (wire protocol §3–4, AX details §6, framebuffer independence §9, TCC/packaging §11).
• AT-SPI2: freedesktop.org Accessibility/AT-SPI2; libatspi reference (AtspiAccessible, AtspiComponent, AtspiAction, AtspiValue, AtspiEditableText); pyatspi2 (GNOME).