nucleic/docs/LINUX_VM_SEMANTIC_AGENT.md

Nucleic — Linux VM semantic (accessibility) agent, Wayland-native

Goal. Bring semantic computer-use to the Linux guest — read the accessibility tree and act on a control by identity (the ax_* surface, at parity with the macOS native agent in MACOS_VM_NATIVE_AGENT.md) — running under a Wayland compositor (GNOME 49 / Mutter), not X11.

Why this is a differentiator. Robust semantic computer-use on Wayland does not exist off the shelf: AT-SPI's tree works, but its geometry is broken under Wayland (a client can't know its own on-screen position, so Component.getExtents(SCREEN) returns garbage/zeros). The upstream fix (Newton) stalled in early 2025 when Sovereign Tech Fund money ran out; only its keyboard-shortcut slice landed (GNOME 48). By vendoring the consumer half of Newton — the compositor as the source of truth for surface geometry and focus — into our controlled VM, we get accurate, frame-consistent element geometry on Wayland that stock GNOME does not provide. That is a concrete accuracy advantage for Wayland computer-use that competitors relying on screenshots-and-pixels (or stuck on X11) do not have.

Companion to LINUX_VM.md (which this extends), MACOS_VM_NATIVE_AGENT.md (the macOS semantic agent we reach parity with), and VSOCK_CONTROL_PLANE.md.

Status: in progress. Sequenced so that every phase ships usable value and the expensive parts are only entered if measurement forces them.

Done so far:

• Base migrated + rebuild-validated on the plan's target compositor. The Linux base now provisions GNOME / Mutter on Wayland (was labwc), pinned to Ubuntu 26.04 LTS "resolute" (latest LTS, GNOME 50). A full host base rebuild passes and a clone boots with the AT-SPI bus live (toolkit-accessibility=true, gnome-shell running, vsock agent answering). Getting 26.04 to boot under Virtualization.framework required a kernel-packaging fix chain (EFI-zboot unwrap, block-device payload, module baking + depmod) — see docs/M_A11Y_RESULTS.md.

• Phase 0 probe written (scripts/a11y-probe.py) with the results template (docs/M_A11Y_RESULTS.md). The measurement run against GUI apps is still pending.

• Phase 1 agent implemented + validated end-to-end. guest/nucleic-a11y-agent/ is a Rust crate speaking AgentWire NDJSON over a second vsock port (2036); it implements ping / ax_dump / ax_action / ax_set_value / ax_focus against AT-SPI via hand-rolled zbus proxies, emitting the macOS agent's node schema. It's a systemd user service (runs in the GNOME session for the a11y bus); provisioning bakes it in and enables it. Host: MacVMEngine routes Linux ax_* to it (linuxA11yClient, cached on LiveVM.a11yAgent) with pixel fallback. Confirmed on a real 26.04 clone: mac_vm-style ax_dump over vsock 2036 returns the live gnome-shell tree (refs, roles, titles, actions, children — and populated frames). One zbus gotcha fixed: AT-SPI objects don't implement Properties.GetAll, so proxies must be built with CacheProperties::No (direct Get).

• Phase 0 measured + Phase 2 built & validated. The probe (scripts/a11y-probe.py) run showed 100% semantic-action coverage on GTK4 but broken AT-SPI SCREEN geometry (window on-screen origin lost → reported at 0,0). Per that data (lean Phase 2, docs/M_A11Y_RESULTS.md), the focused-window geometry gap is closed by a GNOME Shell extension (guest/mutter-geometry-helper/, org.nucleic.Geometry) that publishes the compositor's true window origin; the agent correlates the AT-SPI app to that window by title and reports frame = origin + getExtents(WINDOW). Validated on a 26.04 clone: gnome-text-editor frames went from broken {x:0,y:0} to true global {x:610,y:356} (matching Mutter). Plus a coordinate fallback (non-semantic element → agent returns its global frame → host clicks the center via the surface pointer). CI (.github/workflows/linux-vm-agents.yml) builds + distributes all guest artifacts. The heavy all-surface Newton port (Phase 3) is deferred — 100% semantic coverage means coordinates are the minority path, and the focused-window helper covers it.

Resolved decisions (see §9): a11y-agent language = Rust; compositor = GNOME/Mutter (26.04 LTS); transport = second vsock port (2036), C ping/exec agent untouched on 2035; Phase 3 provider = AT-SPI trees + compositor geometry (no toolkit push), deferred pending non-GTK data.

---

1. Background & current state (as of mid-2026)

What we established during research:

Layer	State on Wayland	Consequence for us
Semantic tree (roles, names, states, values, events)	AT-SPI works; GTK/Qt expose it regardless of display server	Reusable as-is. This is our provider.
Semantic actions (Action.doAction, EditableText, Value.setCurrentValue, grabFocus)	Work over AT-SPI D-Bus	Reusable as-is. No coordinates needed for most controls.
Global element geometry (getExtents(SCREEN))	Broken — client doesn't know its surface's on-screen position; only the compositor does	The gap. Must be sourced from the compositor.
Element ↔ window correlation	No shared global window id bridges AT-SPI and Mutter (X11 used the X window id; Wayland has none)	Trivial for the focused window; harder for background windows.
Input injection	No client-side global injection by design	Solved below/around Wayland — see §4.3.
Screen capture	Host-side already (virtio-gpu scanout, see LINUX_VM.md)	Already solved in Nucleic.

Newton, precisely: its design makes the compositor the source of truth, with a per-surface accessibility tree (not per-application) and surface-relative coordinates that the compositor combines with known surface positions. That model dissolves the correlation+geometry problem by construction. As of GNOME 48/49 only the keyboard-shortcut extraction landed (Tyrychtr/Garnacho); the geometry/tree protocol remains an unmerged prototype. We vendor the consumer half of that design; we do not attempt full Newton (provider-side toolkit rewrites, cross-desktop standardization).

What GNOME 49 gives us for free: the Mutter SDK (virtual monitor + input via the remote-desktop / screencast APIs) and libei input synthesis. Injection and capture are effectively handed to us.

---

2. Where this fits in Nucleic

Today (LINUX_VM.md): the Linux guest needs no in-guest agent for computer-use — capture is host-side virtio-gpu, input is a host-side USB absolute pointer, and guest/nucleic-linux-agent/ (static C, vsock 2035) implements only ping + exec. The host engine short-circuits the ax_* ops for Linux guests with a "use screenshot + pixel actions" message (MacVMEngine+ComputerAgent).

This plan replaces that short-circuit with a real Linux semantic path:

 host (Swift)                              Linux guest (Wayland / GNOME 49)
 ────────────                              ────────────────────────────────
 mac_vm_computer ─ ax_dump ─┐
                  ax_press  │   vsock      ┌────────────────────────────────┐
 MacVMEngine+     ax_set_.. ├─────2035────►│  nucleic-a11y-agent            │
 ComputerAgent    ax_focus  │  (AgentWire) │   ├─ AT-SPI client (tree+acts) │
                  screenshot┘              │   └─ geometry via compositor   │
 (host-side capture/pixels stay as fallback)│         │                     │
                                            │         ▼                     │
                                            │  Mutter geometry helper       │
                                            │  (Phase 2 plugin → Phase 3    │
                                            │   vendored-Newton protocol)   │
                                            └────────────────────────────────┘

Design intent: the host tool surface does not change. ax_dump / ax_press / ax_set_value / ax_focus (with ref / value) already exist for macOS; we make them resolve on Linux too. An agent using mac_vm_computer gets the same semantic API on either guest OS.

---

3. Key design decisions

1. Keep AT-SPI as the provider; do not rewrite toolkits. Full Newton's cost is the provider side (per-toolkit adapters). We reuse the trees apps already expose. We only replace the geometry source and, in Phase 3, the transport for geometry/focus.

2. The a11y agent is a separate in-guest component, not the static C binary. AT-SPI needs libatspi + GObject-Introspection + D-Bus, which breaks nucleic-linux-agent's dependency-free static-C model. Introduce a sibling guest/nucleic-a11y-agent/ (candidate: Rust + the async atspi crate from the Odilia project, or Python + pyatspi/GI). It speaks the same VMAgentCore.AgentWire NDJSON wire so the host treats it uniformly. The C agent keeps owning ping/exec; a11y ops route to the new component (same vsock port, op-based dispatch, or a second port — see Open Questions §9).

3. Target GNOME 49 on Wayland, pinned. Vendored-Newton work is version-coupled to Mutter internals; pin the guest to a known Mutter and treat upgrades as deliberate ports. Ubuntu base is already pinned (LINUX_VM.md).

4. Injection layering (in the controlled VM): prefer AT-SPI semantic actions (no coordinates); for coordinate fallback use the existing host-side USB absolute pointer (already 1:1 with the screenshot per LINUX_VM.md) — which sidesteps Wayland injection restrictions entirely. libei / Mutter-SDK input is a secondary option. We likely need no new injection mechanism.

5. Semantic-first, coordinates as fallback. Maximize doAction/EditableText/Value/grabFocus coverage so geometry is only consulted for elements with no exposed semantic action. This shrinks how much the (hard) geometry problem actually matters.

6. Every phase is shippable. Path A (Phases 1–2) delivers working semantic computer-use on Wayland on its own. The vendored-Newton subset (Phase 3) is a geometry/quality upgrade layered on top, not a prerequisite.

---

4. Phased plan

Phase 0 — Measurement (ground truth before building)

Goal: decide empirically how far semantic-only gets us and how broken geometry really is on our target apps, so later phases are scoped by data, not blog posts.

Work:

• Boot a GNOME 49 Wayland guest (pinned) in the VM.
• Ship a throwaway AT-SPI probe (scripts/a11y-probe.py or Rust): walk the full tree for a target app set (GTK4, Qt, an Electron app, a browser), dump per node: role, name, states, available actions, getExtents(WINDOW) and getExtents(SCREEN).
• Record, per app: (a) % of interactive nodes reachable by a semantic action, (b) whether getExtents(SCREEN) returns real values or zeros, (c) event latency for a full-tree walk.
• Smoke-test the Mutter SDK input/capture handle.

Deliverable: docs/M_A11Y_RESULTS.md (mirroring M0_RESULTS.md) with the coverage/geometry table.

Exit criteria: we know the semantic-action coverage %, and whether background-window geometry is even needed for our workloads. Agent fit: strong (scripted, measurable).

---

Phase 1 — Path A core: AT-SPI semantic agent (no geometry)

Goal: ax_dump / ax_press / ax_set_value / ax_focus resolve on Linux via AT-SPI, semantic actions only. Immediate parity-ish with macOS for everything actionable without pixels.

Work:

• Create guest/nucleic-a11y-agent/ (Rust atspi recommended). Implement:
  • ax_dump → walk AT-SPI, emit the same node schema the macOS agent emits (ref, role, title, value, states, actions; frame left null/window-relative for now). Stable ref allocation that survives across a dump→act round-trip (map ref → AT-SPI object path / accessible id).
  • ax_press → Action.doAction (or best-matching action) on the ref'd element.
  • ax_set_value → EditableText / Value.setCurrentValue.
  • ax_focus → Component.grabFocus.
  • focus/property change events (optional this phase) over the wire.
• Wire it to AgentWire NDJSON on vsock (port decision, §9).
• Host: replace the Linux ax_* short-circuit in MacVMEngine+ComputerAgent with routing to the a11y agent; ping-probe/fallback exactly like the macOS agent (fall back to host-side pixel path if the a11y agent is absent/unhealthy).
• Provisioning: install the a11y agent + its runtime deps into the Linux base image (MacVMEngine+LinuxProvision.swift), enable AT-SPI bus in the guest session, and force Chromium/Electron accessibility on (they lazy-build the tree only when an AT is detected).

Deliverable: semantic control of GTK/Qt apps on Wayland end-to-end through mac_vm_computer.

Exit criteria: on the Phase 0 app set, all semantically-actionable controls are drivable by ref with no screenshot. Agent fit: strong (bounded client code, testable against real apps in the VM).

---

Phase 2 — Path A geometry: focused-window fusion (Mutter helper)

Goal: real global frame for elements in the focused window, enabling accurate coordinate fallback + overlay/annotation, without touching Newton internals.

Work:

• Small Mutter/GNOME Shell helper (JS extension or C plugin) exposing, over a private D-Bus name: the focused window's global geometry (and ideally the geometry of all toplevels).
• In the a11y agent, compute frame_global = window_origin (from helper) + getExtents(WINDOW). Populate frame in ax_dump for the focused surface.
• Coordinate fallback: when an element has no semantic action, target frame_global via the existing host-side USB absolute pointer (already pixel-1:1).
• Correlation: focused-window only — AT-SPI's focused accessible ∩ Mutter's focused window. No general correlation needed yet.

Deliverable: accurate click/annotation for the active window; pixel fallback for non-semantic controls.

Exit criteria: for the focused window, frame_global matches on-screen pixels within tolerance for GTK4/Qt. Agent fit: strong (C/JS Mutter helper is bounded; geometry math is testable). Pacing item: the build→run→observe loop, which the VM makes autonomous.

At the end of Phase 2 we have working, differentiated Wayland semantic computer-use. Phase 3 is a quality/coverage upgrade, entered only if Phase 0/2 data show background-window geometry or frame-sync accuracy matter for our workloads.

---

Phase 3 — Vendored-Newton subset: compositor as source of truth

Goal: accurate, frame-consistent geometry and focus for all surfaces (not just focused), with element↔surface correlation dissolved by construction — the Newton consumer model, vendored and scoped to our VM.

Work:

• Port the Newton prototype consumer half onto pinned GNOME 49: the Mutter-side pieces that expose (a) the set of surfaces with accessibility trees, (b) the focused surface, (c) surface-relative node geometry combined with compositor-known surface positions → true global coordinates.
• Keep AT-SPI as the provider: rather than the full Newton app→compositor push (which needs toolkit adapters), feed the compositor geometry service from AT-SPI trees keyed per surface, OR carry the minimal Newton provider path for GTK4 only if measurement shows AT-SPI geometry is unusable. Decide from Phase 0 data.
• Replace the Phase 2 focused-window helper with the surface-keyed geometry service; the a11y agent now gets per-surface global frames for all windows and correct z-ordered hit-testing (Newton follows Firefox's z-ordered on-screen node list — reuse that for local hit-testing).
• Frame synchronization: consume geometry/focus updates so the reported tree state matches the visible frame (Newton's core promise), eliminating stale-geometry races during animations/scroll.

Deliverable: background-window semantic control, correct geometry under motion, no correlation heuristics.

Exit criteria: geometry correct for non-focused and animating windows; no reliance on title/z-order correlation guesses. Agent fit: strong on the mechanical port + service + client; weak on the few novel protocol-shape decisions (yours to make); medium on compositor timing debug (tractable via the VM observe loop).

Explicitly out of scope (full Newton, not needed): cross-toolkit provider adapters (Qt, Chromium), cross-desktop standardization, rigorous public protocol definition, Orca/screen-reader semantics (speech, braille, text attributes). We are not shipping a standard; we are vendoring a capability.

---

Phase 4 — Hardening & coverage

• Graceful degradation ladder: semantic action → focused-window geometry → all-surface geometry → host-side pixel fallback. Never dead-end; always fall back to the proven pixel path.
• App-coverage reporting: log when a target app exposes no/poor AT-SPI (the real, path-independent ceiling) so the agent knows to switch to vision.
• Version-pinning guardrails + a documented port procedure for GNOME upgrades.
• Perf: cache/diff AT-SPI walks (D-Bus is chatty on large trees like browsers).

---

5. Component inventory / repo layout

Path	New/changed	Role
guest/nucleic-a11y-agent/	new	AT-SPI semantic agent (Rust atspi), speaks AgentWire NDJSON
guest/mutter-geometry-helper/	new (Phase 2)	Mutter/Shell extension exposing surface geometry over private D-Bus
third_party/newton/	new (Phase 3)	Vendored + pinned Newton consumer-half port
Sources/NucleicCore/MacVM/MacVMEngine+ComputerAgent.*	changed	Route Linux ax_* to the a11y agent; ping/fallback
Sources/NucleicCore/MacVM/MacVMEngine+LinuxProvision.swift	changed	Install a11y agent + deps; enable AT-SPI; force Chromium a11y
scripts/a11y-probe.*, scripts/build-a11y-agent.sh	new	Phase 0 probe; build script (cf. build-vm-agent.sh)
docs/M_A11Y_RESULTS.md	new	Phase 0 measurement results

Wire protocol: unchanged (VMAgentCore.AgentWire / NDJSON over AF_VSOCK). Tool surface: unchanged (ax_* already defined for macOS).

---

6. Risks & mitigations

Risk	Severity	Mitigation
App-coverage ceiling — apps with no/poor AT-SPI (Electron mis-config, Flutter, games, custom-GL)	High, path-independent	Semantic-first + always-available host-side pixel fallback; report coverage; force Chromium a11y flag
Newton prototype is coupled to Mutter internals	Medium	Pin GNOME 49; isolate in third_party/newton/; documented port procedure; Phase 3 is optional
Newton protocol "not rigorously defined"	Medium	We don't standardize — private, frozen shape scoped to our VM
AT-SPI D-Bus latency on large trees	Medium	Cache/diff walks; lazy subtree expansion; Phase 4 perf work
ref stability across dump→act	Medium	Map ref→AT-SPI object path/id; re-resolve on stale ref, re-dump on miss
GNOME upgrade breaks vendored geometry	Medium	Pinned base; upgrades are deliberate ports, gated by the Phase 0 probe as a regression test

---

7. Testing & validation

• Phase 0 probe doubles as a regression harness — re-run per GNOME bump; geometry/coverage table must not regress.
• Per-phase real-guest validation in the VM (the macOS agent's §5-style "pending real-guest validation" discipline): assert ax_dump schema parity with macOS, ax_press round-trips, and (Phase 2+) frame_global vs. on-screen pixel tolerance measured against a known-geometry test app.
• Cross-OS parity test: same high-level automation script drives a macOS guest and a Linux guest through the identical mac_vm_computer ax_* surface.

---

8. Effort & sequencing summary

Phase	Outcome	Rough effort	Agent fit
0 Measurement	Coverage/geometry ground truth	days	strong
1 Semantic agent	Semantic control on Wayland (no geometry)	~1–2 weeks	strong
2 Focused geometry	Accurate active-window geometry + pixel fallback	~1–2 weeks	strong
3 Vendored Newton	All-surface, frame-consistent geometry	weeks–~2 months	strong code / you on design calls / medium on timing debug
4 Hardening	Degradation ladder, coverage, perf	ongoing	strong

Path A (Phases 0–2) is the shippable spine; Phase 3 is the differentiator, entered on data.

---

9. Open questions / decisions for the user

1. A11y-agent language: Rust (atspi, async, single static-ish binary, matches our "small agent" ethos) vs. Python (pyatspi/GI, faster to prototype, heavier runtime in the image). Recommendation: Rust. ✅ Decided: Rust.
2. Transport: reuse vsock 2035 with op-based dispatch to the a11y component, or a second vsock port dedicated to a11y (cleaner separation from the static C ping/exec agent)? Recommendation: second port, keep the C agent untouched. ✅ Decided: second port (2036); the C agent stays on 2035.
3. Phase 3 provider: feed the compositor geometry service from AT-SPI trees (no toolkit patches) vs. carry Newton's GTK4 provider push. Decide from Phase 0 geometry data.
4. Compositor: GNOME/Mutter (assumed) vs. evaluate KWin (its scripting D-Bus exposes window geometry more readily, which could make Phase 2 cheaper). Recommendation: confirm GNOME, spot- check KWin during Phase 0. ✅ Decided: GNOME 49 / Mutter — base migrated (Ubuntu 25.10, GDM, AT-SPI on). KWin spot-check deferred (not blocking).
5. Coverage bar: which apps must reach semantic parity (defines "done" and the Phase 0 app set)?