# Nucleic — Linux in-guest semantic (AT-SPI) control agent (design)

> **Status: proposed.** Today a Linux guest's computer-use is 100% host-side pixels (virtio-gpu
> capture + USB HID injection); the semantic `ax_*` ops short-circuit for Linux with a "use screenshot
> + pixel actions" message (`Sources/NucleicCore/MacVM/MacVMEngine+Computer.swift`). This doc plans the
> Linux counterpart to the macOS native AX agent ([MACOS_VM_NATIVE_AGENT.md](MACOS_VM_NATIVE_AGENT.md)):
> a semantic, framebuffer-independent control path that reads the accessibility tree and acts on
> controls **by identity**, using Linux's **AT-SPI2** accessibility framework.

The macOS agent (`guest/NucleicVMAgent`) drives the guest through Apple's Accessibility API
(`AXUIElement`) — read every control's role/title/value/frame/actions, then `AXPress` / set `AXValue` /
set focus by identity. That is dramatically more robust than guessing `(x, y)` off a screenshot, and it
keeps working when a *capture* API returns nothing useful. This doc brings the same capability to Linux
guests via **AT-SPI2**, the de-facto Linux accessibility standard, slotting into the vsock + NDJSON +
protocol-v2 transport and the `ax_*` tool surface Nucleic already ships.

Companion to [LINUX_VM.md](LINUX_VM.md) (the Linux guest subsystem this augments) and
[MACOS_VM_NATIVE_AGENT.md](MACOS_VM_NATIVE_AGENT.md) (the macOS agent this mirrors).

---

## 1. Why — semantic control, minus the macOS tax

The macOS agent's three wins (semantic control, framebuffer independence, vsock transport — see
`MACOS_VM_NATIVE_AGENT.md` §1) apply verbatim to Linux, and Linux is *easier* on two axes:

- **No permission model.** macOS needs three TCC grants (`kTCCServiceAccessibility`,
  `kTCCServicePostEvent`, `kTCCServiceScreenCapture`) pre-written into `TCC.db`, plus a signed `.app`
  bundle and an Aqua session — the single hardest part of the macOS agent. AT-SPI needs **none of it**:
  just the AT-SPI D-Bus bus running in the session and the toolkit a11y bridges enabled.
- **Input is already solved host-side.** The Wayland/X11 community's biggest a11y pain — synthesizing
  input — does not bite us: clicks/keystrokes go in via **host-side USB HID**
  (`VZUSBKeyboardConfiguration` + `VZUSBScreenCoordinatePointingDeviceConfiguration`), not via the
  display server. So this agent's job is narrower than a general Linux automation tool: **read the tree
  and act by identity.** For actions it prefers AT-SPI in-process actions (coordinate-free); the pixel
  path (host-side) remains the fallback.

## 2. The macOS ↔ Linux mapping

AT-SPI2 sits between Windows UIA's rigid structures and macOS AX's flexibility: enum roles, but
**untyped action strings** — the same shape as AX. The `ax_*` wire ops map cleanly:

| Wire op (unchanged) | macOS backing (`Ops+AX.swift`) | Linux backing (AT-SPI2) |
|---|---|---|
| `ax_dump` | `AXUIElementCopyAttributeValue` (role/title/value/frame/actions), recurse `kAXChildren` | `Atspi.Accessible` tree: `get_role_name`, `get_name`, `AtspiValue`, `AtspiComponent.get_extents`, `AtspiAction` action names; recurse `get_child_at_index` |
| `ax_element_at` | `AXUIElementCopyElementAtPosition` | `AtspiComponent.get_accessible_at_point(ATSPI_COORD_TYPE_SCREEN)` |
| `ax_press` (action) | `AXUIElementPerformAction("AXPress"/…)` | `AtspiAction.do_action(i)` (map action name → index) |
| `ax_set_value` | `AXUIElementSetAttributeValue(kAXValue)` | `AtspiEditableText.set_text_contents` / `AtspiValue.set_current_value` |
| `ax_focus` | set `kAXFocused` | `AtspiComponent.grab_focus` |
| `ping` readiness | `AXIsProcessTrusted()` + `CGPreflight*` | AT-SPI bus reachable? (report `atspi: true/false`) |

The `node` JSON is **identical** to macOS (§4 of `MACOS_VM_NATIVE_AGENT.md`): `ref`, `role`, `subrole?`,
`title`, `value`, `enabled`, `focused`, `frame{x,y,w,h}`, `actions[]`, `children[]`. The per-connection
`ref` registry and dump→act→dump staleness contract (§6.2 there) carry over unchanged, so the host side
(`MacVMEngine+ComputerAgent`, `MCPApprovalServer` schema) needs no protocol changes — only to stop
short-circuiting Linux.

## 3. Display server — **X11, not Wayland** (decision)

The Linux base currently runs a **Wayland** session (`labwc` compositor via `greetd` auto-login,
`provision-linux-guest.sh`). **We are switching the automation base to X11.**

Rationale: the AT-SPI *tree* is vended by the toolkit over D-Bus and is compositor-agnostic (tree reads
and action-by-identity work under either), but **global screen coordinates are the weak spot on
Wayland**. Wayland deliberately hides absolute window positions from clients, so
`AtspiComponent.get_extents(SCREEN)` returns window-relative or unreliable coordinates — precisely the
gap the GNOME "Newton" project is still closing in 2025-26. That coordinate is exactly what the
pixel-fallback path needs (element → `(x, y)` for the host to click), and it's what lets `ax_element_at`
hit-test correctly. Under **X11, AT-SPI coordinate reporting is fully mature** and global geometry is
trivial. Since Nucleic builds the base image, choosing X11 for the automation guest sidesteps the entire
Wayland-a11y-immaturity problem at the cost of a compositor swap we control.

The X11 session (see §6): `Xorg` (modesetting driver on virtio-gpu) + a minimal WM (`openbox`) + an
auto-login greeter, with `at-spi2-core` + the GTK/Qt a11y bridges enabled. Host-side capture/HID are
unchanged — virtio-gpu scanout and USB HID don't care whether X11 or Wayland is compositing.

## 4. In-guest agent shape — split, don't rewrite

`guest/nucleic-linux-agent/src/agent.c` is a dependency-free static C binary carrying `ping` + `exec`,
started early by `systemd/nucleic-linux-agent.service`. Hand-rolling AT-SPI's D-Bus marshalling in
zero-dependency C would be miserable, and we do **not** want to add heavy deps to the boot-critical
exec path. So:

- **Keep `agent.c` as-is** for `ping`/`exec` — static, zero-dep, early boot. Extend only its `ping`
  reply to advertise AT-SPI availability (`"atspi": true|false`).
- **Add a separate a11y helper** linked against the real bindings — `libatspi-2.0` + GLib (C) or a small
  `python3-pyatspi` process. It is `apt`-installed in the provision phase and runs **inside the graphical
  session** (it needs the session's D-Bus + `AT_SPI_BUS`). The C agent forwards `ax_*` op lines to the
  helper (a local unix socket or a spawned request/response), or the host reaches the helper on a second
  vsock port. Recommended first cut: **helper listens on its own vsock port**, `agent.c` unchanged except
  for the readiness bit; the host tries the a11y port for `ax_*` and the base port for `ping`/`exec`.

Language for the helper: **Python + `pyatspi`** for the spike and likely for v1 (fast to write, the
canonical AT-SPI AT-side API); revisit a static C/`libatspi` or Rust `xa11y` rewrite only if startup
cost or the extra runtime deps in the base become a problem.

## 5. Known limitations (be honest)

- **Non-GTK/Qt apps expose poor/empty AT-SPI trees** — same caveat the macOS doc calls out (§9) for
  Electron/Java/GL. Electron/Chromium need `--force-renderer-accessibility`; enable
  `QT_ACCESSIBILITY=1` / `GTK_MODULES` bridges by default. Terminals and custom-drawn UIs give thin
  trees → fall back to the (host-side) pixel path.
- **AT-SPI is chatty.** Every property is an individual D-Bus round-trip; a full tree walk of a rich app
  can take seconds. Mirror the macOS cost controls: `maxDepth`, a node budget (macOS caps at 800), and a
  per-app timeout so a hung app can't wedge the walker. Batch where the bindings allow.
- **Lazy/virtualized views** (list/table rows) may be absent from the tree until scrolled into view —
  same as AX.

## 6. Provisioning changes (X11 base)

`provision-linux-guest.sh` phase-2 first-boot install changes from the Wayland stack to X11 + AT-SPI:

- **Out:** `labwc`, `foot` (Wayland-only terminal), `xwayland`, `wlr-randr`, `seatd`, the
  `dbus-run-session -- labwc` greetd session.
- **In:** `xserver-xorg` + `xserver-xorg-video-*`/modesetting, `openbox` (WM), `xterm`, `x11-xserver-utils`;
  `at-spi2-core` + `gir1.2-atspi-2.0` + `python3-pyatspi` + `python3-gi` (the a11y helper + bridges);
  AT-SPI env (`GTK_MODULES=gail:atk-bridge`, `QT_ACCESSIBILITY=1`,
  `QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1`, Chromium `--force-renderer-accessibility` default).
- **greetd session:** auto-login into an X session (`startx`/`xinit` launching `openbox-session`) instead
  of `labwc`. Autostart `xterm` (and the a11y helper) so a fresh screenshot isn't an empty root window.

The two-phase base build, bootstrap initramfs, vsock agent (`ping`/`exec`), host-side capture/HID, and
clone lifecycle are all unchanged — this is a session-package swap plus the a11y helper.

## 7. Host wiring

Minimal, because the transport + tool surface already exist and are OS-neutral:

- `MacVMEngine+Computer.swift` — remove/relax the Linux short-circuit (currently ~L45) so `ax_*` route
  to the agent for Linux guests once `ping` reports `atspi:true`.
- `MacVMEngine+ComputerAgent.swift` — the `ax_dump`/`ax_element_at`/`ax_press`/`ax_set_value`/`ax_focus`
  handlers are already OS-neutral; point them at the Linux a11y helper's port when the guest is Linux.
- `agent.c` `send_ping_reply` — add `"atspi"` to the readiness block so the host advertises `ax_*` only
  when the helper is present.
- `MCPApprovalServer` — no schema change (the `ax_*` actions already exist).

## 8. Rollout plan (phased, each independently landable)

0. **Spike / de-risk** — ✅ **DONE** (`guest/nucleic-linux-agent/spike/`). Under a headless X (Xvfb) a
   GTK probe app launches and `pyatspi` dumps the tree into the macOS-identical `node` JSON, then
   exercises `do_action` (press) + `set_text_contents` (set value) + `grab_focus` (focus). Ran green on
   Debian 12 arm64 (same toolkit stack as the Ubuntu 24.04 guest): all three primitives passed and
   `get_extents(SCREEN)` resolved to real geometry — confirming the bindings, the `node` shape, and the
   X11 coordinate assumption. See `spike/README.md`.
1. **X11 base** — swap `provision-linux-guest.sh` to the X11 + AT-SPI stack (§6); rebuild + smoke-test
   the base boots to an X session with the AT-SPI bus up.
2. **a11y helper** — implement `ax_dump`/`ax_element_at`/`ax_press`/`ax_set_value`/`ax_focus` in the
   helper, emitting the macOS-identical `node` JSON; ref registry + staleness; cost controls (§5).
3. **Host wiring** — §7: stop short-circuiting Linux, route `ax_*` to the helper, advertise on
   `ping.atspi`.
4. **Validate + iterate** — real-guest reliability across GTK/Qt/Electron; document coverage gaps.

## 9. Open questions / verify-on-real-VM

- **`get_extents(SCREEN)` accuracy under our X11 session** — expected good on X11; confirm on the real
  base (this is the load-bearing assumption behind the Wayland→X11 switch).
- **a11y helper transport** — second vsock port vs. `agent.c`-forwarded unix socket. Second port is the
  simplest first cut; revisit if port management is awkward.
- **Helper language** — Python/`pyatspi` (v1) vs. C/`libatspi` or Rust `xa11y` (if deps/startup bite).
- **Non-GTK/Qt coverage** — per-app a11y enable toggles; document the gaps like the macOS doc does.

## 10. Cross-references

- [LINUX_VM.md](LINUX_VM.md) — the Linux guest subsystem (two-phase base build, vsock agent, host-side
  computer-use) this augments.
- [MACOS_VM_NATIVE_AGENT.md](MACOS_VM_NATIVE_AGENT.md) — the macOS AX agent this mirrors (wire protocol
  §3–4, AX details §6, framebuffer independence §9, TCC/packaging §11).
- AT-SPI2: freedesktop.org `Accessibility/AT-SPI2`; `libatspi` reference (`AtspiAccessible`,
  `AtspiComponent`, `AtspiAction`, `AtspiValue`, `AtspiEditableText`); `pyatspi2` (GNOME).
</content>
</invoke>
