# Nucleic — Linux VM semantic (accessibility) agent, Wayland-native

> **Goal.** Bring **semantic** computer-use to the Linux guest — read the accessibility tree and act
> on a control *by identity* (the `ax_*` surface, at parity with the macOS native agent in
> [MACOS_VM_NATIVE_AGENT.md](MACOS_VM_NATIVE_AGENT.md)) — running under a **Wayland** compositor
> (GNOME 49 / Mutter), not X11.
>
> **Why this is a differentiator.** Robust semantic computer-use on Wayland does not exist off the
> shelf: AT-SPI's tree works, but its *geometry* is broken under Wayland (a client can't know its own
> on-screen position, so `Component.getExtents(SCREEN)` returns garbage/zeros). The upstream fix
> (**Newton**) stalled in early 2025 when Sovereign Tech Fund money ran out; only its keyboard-shortcut
> slice landed (GNOME 48). By **vendoring the consumer half of Newton** — the compositor as the source
> of truth for surface geometry and focus — into our controlled VM, we get accurate, frame-consistent
> element geometry on Wayland that stock GNOME does not provide. That is a concrete accuracy advantage
> for Wayland computer-use that competitors relying on screenshots-and-pixels (or stuck on X11) do not
> have.

Companion to [LINUX_VM.md](LINUX_VM.md) (which this extends), [MACOS_VM_NATIVE_AGENT.md](MACOS_VM_NATIVE_AGENT.md)
(the macOS semantic agent we reach parity with), and [VSOCK_CONTROL_PLANE.md](VSOCK_CONTROL_PLANE.md).

**Status:** in progress. Sequenced so that **every phase ships usable value** and the expensive parts
are only entered if measurement forces them.

**Done so far:**
- **Base migrated + rebuild-validated on the plan's target compositor.** The Linux base now provisions
  **GNOME / Mutter on Wayland** (was labwc), pinned to **Ubuntu 26.04 LTS "resolute"** (latest LTS,
  GNOME 50). A full host base rebuild **passes** and a clone boots with the AT-SPI bus live
  (`toolkit-accessibility=true`, `gnome-shell` running, vsock agent answering). Getting 26.04 to boot
  under Virtualization.framework required a kernel-packaging fix chain (EFI-zboot unwrap, block-device
  payload, module baking + depmod) — see `docs/M_A11Y_RESULTS.md`.
- **Phase 0 probe written** (`scripts/a11y-probe.py`) with the results template
  (`docs/M_A11Y_RESULTS.md`). The measurement *run* against GUI apps is still pending.
- **Phase 1 agent implemented + validated end-to-end.** `guest/nucleic-a11y-agent/` is a Rust crate
  speaking `AgentWire` NDJSON over a **second vsock port (2036)**; it implements `ping` / `ax_dump` /
  `ax_action` / `ax_set_value` / `ax_focus` against AT-SPI via hand-rolled `zbus` proxies, emitting the
  macOS agent's node schema. It's a systemd **user** service (runs in the GNOME session for the a11y
  bus); provisioning bakes it in and enables it. Host: `MacVMEngine` routes Linux `ax_*` to it
  (`linuxA11yClient`, cached on `LiveVM.a11yAgent`) with pixel fallback. **Confirmed on a real 26.04
  clone:** `mac_vm`-style `ax_dump` over vsock 2036 returns the live `gnome-shell` tree (refs, roles,
  titles, actions, children — and populated `frame`s). One zbus gotcha fixed: AT-SPI objects don't
  implement `Properties.GetAll`, so proxies must be built with `CacheProperties::No` (direct `Get`).

- **Phase 0 measured + Phase 2 built & validated.** The probe (`scripts/a11y-probe.py`) run showed
  **100% semantic-action coverage** on GTK4 but **broken AT-SPI SCREEN geometry** (window on-screen
  origin lost → reported at 0,0). Per that data (lean Phase 2, `docs/M_A11Y_RESULTS.md`), the
  focused-window geometry gap is closed by a **GNOME Shell extension** (`guest/mutter-geometry-helper/`,
  `org.nucleic.Geometry`) that publishes the compositor's true window origin; the agent correlates the
  AT-SPI app to that window by title and reports `frame = origin + getExtents(WINDOW)`. **Validated on a
  26.04 clone:** gnome-text-editor frames went from broken `{x:0,y:0}` to true global `{x:610,y:356}`
  (matching Mutter). Plus a **coordinate fallback** (non-semantic element → agent returns its global
  frame → host clicks the center via the surface pointer). CI (`.github/workflows/linux-vm-agents.yml`)
  builds + distributes all guest artifacts. The heavy all-surface Newton port (Phase 3) is **deferred** —
  100% semantic coverage means coordinates are the minority path, and the focused-window helper covers it.

**Resolved decisions (see §9):** a11y-agent language = **Rust**; compositor = **GNOME/Mutter (26.04
LTS)**; transport = **second vsock port (2036)**, C `ping`/`exec` agent untouched on 2035; Phase 3
provider = **AT-SPI trees + compositor geometry** (no toolkit push), deferred pending non-GTK data.

---

## 1. Background & current state (as of mid-2026)

What we established during research:

| Layer | State on Wayland | Consequence for us |
|---|---|---|
| **Semantic tree** (roles, names, states, values, events) | AT-SPI works; GTK/Qt expose it regardless of display server | **Reusable as-is.** This is our provider. |
| **Semantic actions** (`Action.doAction`, `EditableText`, `Value.setCurrentValue`, `grabFocus`) | Work over AT-SPI D-Bus | **Reusable as-is.** No coordinates needed for most controls. |
| **Global element geometry** (`getExtents(SCREEN)`) | **Broken** — client doesn't know its surface's on-screen position; only the compositor does | **The gap.** Must be sourced from the compositor. |
| **Element ↔ window correlation** | No shared global window id bridges AT-SPI and Mutter (X11 used the X window id; Wayland has none) | Trivial for the *focused* window; harder for background windows. |
| **Input injection** | No client-side global injection by design | Solved below/around Wayland — see §4.3. |
| **Screen capture** | Host-side already (virtio-gpu scanout, see LINUX_VM.md) | Already solved in Nucleic. |

**Newton, precisely:** its design makes the **compositor the source of truth**, with a **per-surface**
accessibility tree (not per-application) and **surface-relative** coordinates that the compositor
combines with known surface positions. That model *dissolves* the correlation+geometry problem by
construction. As of GNOME 48/49 only the keyboard-shortcut extraction landed (Tyrychtr/Garnacho); the
geometry/tree protocol remains an unmerged prototype. We vendor the **consumer half** of that design;
we do **not** attempt full Newton (provider-side toolkit rewrites, cross-desktop standardization).

**What GNOME 49 gives us for free:** the **Mutter SDK** (virtual monitor + input via the remote-desktop
/ screencast APIs) and **libei** input synthesis. Injection and capture are effectively handed to us.

---

## 2. Where this fits in Nucleic

Today (LINUX_VM.md): the Linux guest needs **no in-guest agent for computer-use** — capture is
host-side virtio-gpu, input is a host-side USB absolute pointer, and `guest/nucleic-linux-agent/`
(static C, vsock **2035**) implements only `ping` + `exec`. The host engine short-circuits the `ax_*`
ops for Linux guests with a "use screenshot + pixel actions" message (`MacVMEngine+ComputerAgent`).

This plan replaces that short-circuit with a real Linux semantic path:

```
 host (Swift)                              Linux guest (Wayland / GNOME 49)
 ────────────                              ────────────────────────────────
 mac_vm_computer ─ ax_dump ─┐
                  ax_press  │   vsock      ┌────────────────────────────────┐
 MacVMEngine+     ax_set_.. ├─────2035────►│  nucleic-a11y-agent            │
 ComputerAgent    ax_focus  │  (AgentWire) │   ├─ AT-SPI client (tree+acts) │
                  screenshot┘              │   └─ geometry via compositor   │
 (host-side capture/pixels stay as fallback)│         │                     │
                                            │         ▼                     │
                                            │  Mutter geometry helper       │
                                            │  (Phase 2 plugin → Phase 3    │
                                            │   vendored-Newton protocol)   │
                                            └────────────────────────────────┘
```

Design intent: **the host tool surface does not change.** `ax_dump` / `ax_press` / `ax_set_value` /
`ax_focus` (with `ref` / `value`) already exist for macOS; we make them resolve on Linux too. An agent
using `mac_vm_computer` gets the same semantic API on either guest OS.

---

## 3. Key design decisions

1. **Keep AT-SPI as the provider; do not rewrite toolkits.** Full Newton's cost is the provider side
   (per-toolkit adapters). We reuse the trees apps already expose. We only replace the *geometry*
   source and, in Phase 3, the *transport* for geometry/focus.

2. **The a11y agent is a separate in-guest component, not the static C binary.** AT-SPI needs
   libatspi + GObject-Introspection + D-Bus, which breaks `nucleic-linux-agent`'s dependency-free
   static-C model. Introduce a sibling **`guest/nucleic-a11y-agent/`** (candidate: **Rust + the async
   `atspi` crate** from the Odilia project, or Python + `pyatspi`/GI). It speaks the same
   `VMAgentCore.AgentWire` NDJSON wire so the host treats it uniformly. The C agent keeps owning
   `ping`/`exec`; a11y ops route to the new component (same vsock port, op-based dispatch, or a second
   port — see Open Questions §9).

3. **Target GNOME 49 on Wayland, pinned.** Vendored-Newton work is version-coupled to Mutter
   internals; pin the guest to a known Mutter and treat upgrades as deliberate ports. Ubuntu base is
   already pinned (LINUX_VM.md).

4. **Injection layering (in the controlled VM):** prefer AT-SPI **semantic** actions (no coordinates);
   for coordinate fallback use the existing **host-side USB absolute pointer** (already 1:1 with the
   screenshot per LINUX_VM.md) — which sidesteps Wayland injection restrictions entirely. `libei` /
   Mutter-SDK input is a secondary option. We likely need **no** new injection mechanism.

5. **Semantic-first, coordinates as fallback.** Maximize `doAction`/`EditableText`/`Value`/`grabFocus`
   coverage so geometry is only consulted for elements with no exposed semantic action. This shrinks
   how much the (hard) geometry problem actually matters.

6. **Every phase is shippable.** Path A (Phases 1–2) delivers working semantic computer-use on Wayland
   on its own. The vendored-Newton subset (Phase 3) is a geometry/quality upgrade layered on top, not a
   prerequisite.

---

## 4. Phased plan

### Phase 0 — Measurement (ground truth before building)

**Goal:** decide empirically how far semantic-only gets us and how broken geometry really is on our
target apps, so later phases are scoped by data, not blog posts.

**Work:**
- Boot a **GNOME 49 Wayland** guest (pinned) in the VM.
- Ship a throwaway **AT-SPI probe** (`scripts/a11y-probe.py` or Rust): walk the full tree for a target
  app set (GTK4, Qt, an Electron app, a browser), dump per node: role, name, states, available
  actions, `getExtents(WINDOW)` and `getExtents(SCREEN)`.
- Record, per app: (a) % of interactive nodes reachable by a **semantic action**, (b) whether
  `getExtents(SCREEN)` returns real values or zeros, (c) event latency for a full-tree walk.
- Smoke-test the **Mutter SDK** input/capture handle.

**Deliverable:** `docs/M_A11Y_RESULTS.md` (mirroring `M0_RESULTS.md`) with the coverage/geometry table.

**Exit criteria:** we know the semantic-action coverage %, and whether background-window geometry is
even needed for our workloads. **Agent fit: strong** (scripted, measurable).

---

### Phase 1 — Path A core: AT-SPI semantic agent (no geometry)

**Goal:** `ax_dump` / `ax_press` / `ax_set_value` / `ax_focus` resolve on Linux via AT-SPI, semantic
actions only. Immediate parity-ish with macOS for everything actionable without pixels.

**Work:**
- Create **`guest/nucleic-a11y-agent/`** (Rust `atspi` recommended). Implement:
  - `ax_dump` → walk AT-SPI, emit the same node schema the macOS agent emits (`ref`, role, title,
    value, states, actions; `frame` left null/window-relative for now). Stable `ref` allocation that
    survives across a dump→act round-trip (map `ref` → AT-SPI object path / accessible id).
  - `ax_press` → `Action.doAction` (or best-matching action) on the `ref`'d element.
  - `ax_set_value` → `EditableText` / `Value.setCurrentValue`.
  - `ax_focus` → `Component.grabFocus`.
  - focus/property change **events** (optional this phase) over the wire.
- Wire it to `AgentWire` NDJSON on vsock (port decision, §9).
- **Host:** replace the Linux `ax_*` short-circuit in `MacVMEngine+ComputerAgent` with routing to the
  a11y agent; ping-probe/fallback exactly like the macOS agent (fall back to host-side pixel path if
  the a11y agent is absent/unhealthy).
- Provisioning: install the a11y agent + its runtime deps into the Linux base image
  (`MacVMEngine+LinuxProvision.swift`), enable AT-SPI bus in the guest session, and **force
  Chromium/Electron accessibility on** (they lazy-build the tree only when an AT is detected).

**Deliverable:** semantic control of GTK/Qt apps on Wayland end-to-end through `mac_vm_computer`.

**Exit criteria:** on the Phase 0 app set, all semantically-actionable controls are drivable by `ref`
with no screenshot. **Agent fit: strong** (bounded client code, testable against real apps in the VM).

---

### Phase 2 — Path A geometry: focused-window fusion (Mutter helper)

**Goal:** real **global** `frame` for elements in the **focused** window, enabling accurate coordinate
fallback + overlay/annotation, without touching Newton internals.

**Work:**
- Small **Mutter/GNOME Shell helper** (JS extension or C plugin) exposing, over a private D-Bus name:
  the focused window's global geometry (and ideally the geometry of all toplevels).
- In the a11y agent, compute `frame_global = window_origin (from helper) + getExtents(WINDOW)`.
  Populate `frame` in `ax_dump` for the focused surface.
- Coordinate fallback: when an element has no semantic action, target `frame_global` via the existing
  **host-side USB absolute pointer** (already pixel-1:1).
- **Correlation:** focused-window only — AT-SPI's focused accessible ∩ Mutter's focused window. No
  general correlation needed yet.

**Deliverable:** accurate click/annotation for the active window; pixel fallback for
non-semantic controls.

**Exit criteria:** for the focused window, `frame_global` matches on-screen pixels within tolerance for
GTK4/Qt. **Agent fit: strong** (C/JS Mutter helper is bounded; geometry math is testable). Pacing item:
the build→run→observe loop, which the VM makes autonomous.

> At the end of Phase 2 we have **working, differentiated Wayland semantic computer-use.** Phase 3 is a
> quality/coverage upgrade, entered only if Phase 0/2 data show background-window geometry or
> frame-sync accuracy matter for our workloads.

---

### Phase 3 — Vendored-Newton subset: compositor as source of truth

**Goal:** accurate, **frame-consistent** geometry and focus for **all** surfaces (not just focused),
with element↔surface correlation dissolved by construction — the Newton consumer model, vendored and
scoped to our VM.

**Work:**
- **Port the Newton prototype consumer half** onto pinned GNOME 49: the Mutter-side pieces that expose
  (a) the set of surfaces with accessibility trees, (b) the focused surface, (c) **surface-relative
  node geometry combined with compositor-known surface positions** → true global coordinates.
- Keep **AT-SPI as the provider**: rather than the full Newton app→compositor push (which needs toolkit
  adapters), feed the compositor geometry service from AT-SPI trees keyed per surface, OR carry the
  minimal Newton provider path for GTK4 only if measurement shows AT-SPI geometry is unusable. Decide
  from Phase 0 data.
- Replace the Phase 2 focused-window helper with the surface-keyed geometry service; the a11y agent now
  gets per-surface global frames for **all** windows and correct z-ordered hit-testing (Newton follows
  Firefox's z-ordered on-screen node list — reuse that for local hit-testing).
- Frame synchronization: consume geometry/focus updates so the reported tree state matches the visible
  frame (Newton's core promise), eliminating stale-geometry races during animations/scroll.

**Deliverable:** background-window semantic control, correct geometry under motion, no correlation
heuristics.

**Exit criteria:** geometry correct for non-focused and animating windows; no reliance on
title/z-order correlation guesses. **Agent fit: strong on the mechanical port + service + client;
weak on the few novel protocol-shape decisions (yours to make); medium on compositor timing debug
(tractable via the VM observe loop).**

**Explicitly out of scope (full Newton, not needed):** cross-toolkit provider adapters (Qt, Chromium),
cross-desktop standardization, rigorous public protocol definition, Orca/screen-reader semantics
(speech, braille, text attributes). We are not shipping a standard; we are vendoring a capability.

---

### Phase 4 — Hardening & coverage

- Graceful degradation ladder: semantic action → focused-window geometry → all-surface geometry →
  host-side pixel fallback. Never dead-end; always fall back to the proven pixel path.
- App-coverage reporting: log when a target app exposes no/poor AT-SPI (the real, path-independent
  ceiling) so the agent knows to switch to vision.
- Version-pinning guardrails + a documented port procedure for GNOME upgrades.
- Perf: cache/diff AT-SPI walks (D-Bus is chatty on large trees like browsers).

---

## 5. Component inventory / repo layout

| Path | New/changed | Role |
|---|---|---|
| `guest/nucleic-a11y-agent/` | **new** | AT-SPI semantic agent (Rust `atspi`), speaks `AgentWire` NDJSON |
| `guest/mutter-geometry-helper/` | **new** (Phase 2) | Mutter/Shell extension exposing surface geometry over private D-Bus |
| `third_party/newton/` | **new** (Phase 3) | Vendored + pinned Newton consumer-half port |
| `Sources/NucleicCore/MacVM/MacVMEngine+ComputerAgent.*` | **changed** | Route Linux `ax_*` to the a11y agent; ping/fallback |
| `Sources/NucleicCore/MacVM/MacVMEngine+LinuxProvision.swift` | **changed** | Install a11y agent + deps; enable AT-SPI; force Chromium a11y |
| `scripts/a11y-probe.*`, `scripts/build-a11y-agent.sh` | **new** | Phase 0 probe; build script (cf. `build-vm-agent.sh`) |
| `docs/M_A11Y_RESULTS.md` | **new** | Phase 0 measurement results |

Wire protocol: unchanged (`VMAgentCore.AgentWire` / NDJSON over `AF_VSOCK`). Tool surface: unchanged
(`ax_*` already defined for macOS).

---

## 6. Risks & mitigations

| Risk | Severity | Mitigation |
|---|---|---|
| **App-coverage ceiling** — apps with no/poor AT-SPI (Electron mis-config, Flutter, games, custom-GL) | High, **path-independent** | Semantic-first + always-available host-side pixel fallback; report coverage; force Chromium a11y flag |
| Newton prototype is coupled to Mutter internals | Medium | Pin GNOME 49; isolate in `third_party/newton/`; documented port procedure; Phase 3 is optional |
| Newton protocol "not rigorously defined" | Medium | We don't standardize — private, frozen shape scoped to our VM |
| AT-SPI D-Bus latency on large trees | Medium | Cache/diff walks; lazy subtree expansion; Phase 4 perf work |
| `ref` stability across dump→act | Medium | Map `ref`→AT-SPI object path/id; re-resolve on stale ref, re-dump on miss |
| GNOME upgrade breaks vendored geometry | Medium | Pinned base; upgrades are deliberate ports, gated by the Phase 0 probe as a regression test |

---

## 7. Testing & validation

- **Phase 0 probe doubles as a regression harness** — re-run per GNOME bump; geometry/coverage table
  must not regress.
- Per-phase real-guest validation in the VM (the macOS agent's §5-style "pending real-guest
  validation" discipline): assert `ax_dump` schema parity with macOS, `ax_press` round-trips, and
  (Phase 2+) `frame_global` vs. on-screen pixel tolerance measured against a known-geometry test app.
- Cross-OS parity test: same high-level automation script drives a macOS guest and a Linux guest
  through the identical `mac_vm_computer` `ax_*` surface.

---

## 8. Effort & sequencing summary

| Phase | Outcome | Rough effort | Agent fit |
|---|---|---|---|
| 0 Measurement | Coverage/geometry ground truth | days | strong |
| 1 Semantic agent | Semantic control on Wayland (no geometry) | ~1–2 weeks | strong |
| 2 Focused geometry | Accurate active-window geometry + pixel fallback | ~1–2 weeks | strong |
| 3 Vendored Newton | All-surface, frame-consistent geometry | weeks–~2 months | strong code / you on design calls / medium on timing debug |
| 4 Hardening | Degradation ladder, coverage, perf | ongoing | strong |

Path A (Phases 0–2) is the shippable spine; Phase 3 is the differentiator, entered on data.

---

## 9. Open questions / decisions for the user

1. **A11y-agent language:** Rust (`atspi`, async, single static-ish binary, matches our "small agent"
   ethos) vs. Python (`pyatspi`/GI, faster to prototype, heavier runtime in the image). Recommendation:
   **Rust**. **✅ Decided: Rust.**
2. **Transport:** reuse vsock **2035** with op-based dispatch to the a11y component, or a **second
   vsock port** dedicated to a11y (cleaner separation from the static C `ping`/`exec` agent)?
   Recommendation: **second port**, keep the C agent untouched. **✅ Decided: second port (2036);** the
   C agent stays on 2035.
3. **Phase 3 provider:** feed the compositor geometry service from **AT-SPI trees** (no toolkit
   patches) vs. carry Newton's **GTK4 provider** push. Decide from Phase 0 geometry data.
4. **Compositor:** GNOME/Mutter (assumed) vs. evaluate **KWin** (its scripting D-Bus exposes window
   geometry more readily, which could make Phase 2 cheaper). Recommendation: **confirm GNOME**, spot-
   check KWin during Phase 0. **✅ Decided: GNOME 49 / Mutter** — base migrated (Ubuntu 25.10, GDM,
   AT-SPI on). KWin spot-check deferred (not blocking).
5. **Coverage bar:** which apps must reach semantic parity (defines "done" and the Phase 0 app set)?
