# Nucleic — Linux VM semantic-agent measurement (Phase 0)

Ground-truth measurements for [LINUX_VM_SEMANTIC_AGENT.md](LINUX_VM_SEMANTIC_AGENT.md) **Phase 0**:
before building the agent, decide empirically (a) how far semantic-only actions get us and (b) how
broken AT-SPI screen geometry actually is on our target apps, so Phases 1–3 are scoped by data rather
than by blog posts. Mirrors [M0_RESULTS.md](M0_RESULTS.md).

**Status: ✅ measured + geometry solved** (Ubuntu 26.04 LTS / GNOME 50 / Mutter). Headlines: **100%
semantic-action coverage** on GTK4 (coordinates rarely needed to *drive* the UI); the AT-SPI *client*
API can't report a window's on-screen origin on Wayland, but that's **not a real limitation for us** —
the whole display/input path is host-owned virtio in ONE coordinate space, so the compositor knows the
true geometry and Phase 2 sources it. Verified chain: VZ virtio-gpu scanout **1920×1200** = Mutter
current mode **1920×1200 @ scale 1.0** = host JPEG capture = USB pointer (1:1) = the geometry helper's
window coords. So `ax_dump` frames are **true, directly-clickable screen pixels** (validated: broken
`{0,0}` → correct `{610,356}`). Still to measure: Qt / Electron / browser apps.

---

## How to reproduce

Inside a booted Linux guest (see §Prerequisites), from the repo root on the shared volume:

```sh
# one-time deps in the guest
sudo apt-get install -y python3-gi gir1.2-atspi-2.0

# what can AT-SPI see?
scripts/a11y-probe.py --list

# full measurement for the app set, machine-readable dump alongside the table
scripts/a11y-probe.py --json /tmp/a11y.json --nodes-json
```

### Prerequisites / caveats that affect the numbers

- **AT-SPI bus must be live.** Under Wayland the session bus must expose `org.a11y.Bus`; the
  at-spi2 registry must be running in the desktop session the probe runs in.
- **Chromium / Electron lazily build their tree.** Launch them with accessibility forced on
  (`--force-renderer-accessibility`, or set the `ACCESSIBILITY_ENABLED` heuristics) or their subtree
  reads as empty and coverage looks artificially bad. Phase 1 provisioning must force this on.
- **Compositor.** The numbers are compositor-dependent for *geometry* only (the tree/actions are
  compositor-independent). Record which compositor the run used — this is the open decision
  (labwc today vs the GNOME 49/Mutter target the plan assumes; see §Open items).

---

## App set (§9 Q5 — the coverage bar)

The apps we require semantic parity on define "done" for Phase 1 and the regression bar for GNOME
bumps. Proposed set spanning the toolkit matrix (confirm/adjust with the run):

| App | Toolkit | Why it's in the set |
|---|---|---|
| _tbd (e.g. gedit / GNOME Text Editor)_ | GTK4 | Primary GTK a11y reference |
| _tbd (e.g. foot / a terminal)_ | — | The always-present terminal in the base |
| _tbd (a Qt app)_ | Qt | Second major toolkit |
| _tbd (an Electron app / VS Code)_ | Chromium | The lazy-tree / forced-a11y case |
| _tbd (Firefox or Chromium)_ | browser | Largest, chattiest tree (latency stress) |

---

## Results

### Semantic-action coverage & geometry (per app)

_Filled from `scripts/a11y-probe.py` output. `cover` = semantically-actionable ÷ interactive controls.
`scr=0` / `scr=win` = fraction of component nodes whose SCREEN extents are zero-sized / identical to
WINDOW-relative extents (the two Wayland-broken-geometry symptoms)._

First run — **Ubuntu 26.04 LTS, GNOME 50 / Mutter, Wayland** (2026-07-09), via `scripts/a11y-probe.py`
against the auto-login session's live apps:

| App | toolkit | nodes | interactive | coverage | walk(s) | comp-nodes | scr=0 | scr=win | geometry |
|---|---|---|---|---|---|---|---|---|---|
| gnome-shell | GTK/compositor | 1388 | 1101 | **100%** | 0.80 | 1387 | 92% | 100% | broken\* |
| gnome-text-editor | GTK4 | 83 | 11 | **100%** | 0.13 | 82 | 34% | 16% | **usable** |
| kgx (gnome-console) | GTK4 | 63 | 10 | **100%** | 0.10 | 62 | 23% | 18% | **usable** |
| xdg-desktop-portal-gtk | GTK | 1 | 0 | — | 0.00 | 0 | — | — | n/a |

\* gnome-shell **is** the compositor, so its internal nodes trivially have screen == window (scr=win
100%); this is not a client-geometry signal. The client apps (text-editor, console) are the real datum.

### Findings

1. **Semantic-action coverage = 100%** on every real app. Every interactive control is reachable by an
   `ax_*` semantic action (Action / EditableText / Value / Focusable) with **no coordinates**. This is
   the headline result: **semantic-first is fully sufficient for actionability on GTK4/Wayland**, so
   geometry barely matters for *driving* the UI (only for overlays/annotation or the rare non-semantic
   control). ⚠️ Still to measure: **Qt, Electron/Chromium, Firefox** (not installed in the base) — the
   real coverage ceiling is toolkit-dependent (risk table §6).
2. **AT-SPI's SCREEN extents are origin-blind — but the *system* geometry is fully known.** The
   AT-SPI *client* API can't report a window's on-screen origin on Wayland (by design). But this is
   **not a real limitation here**, because our entire display/input path is host-owned virtio in a
   single coordinate space: the VZ virtio-gpu scanout is 1920×1200, Mutter's current mode is 1920×1200
   @ scale 1.0, the host captures that scanout as a 1920×1200 JPEG, and the USB pointer targets those
   same pixels 1:1. So the compositor's window coordinates ARE screenshot pixels and pointer targets;
   we just have to read them from the compositor (which knows) rather than the AT-SPI client (which
   doesn't). That scale-1 invariant is pinned in provisioning so it can't drift.

   Dumping gnome-text-editor's raw AT-SPI frame values shows the client-side blindness (initially I
   misread the `scr=win` summary as "usable" — wrong):
   - the **window root reports `SCREEN = {x:0, y:0, w:700, h:520}`** — the size is right but the
     on-screen **origin is lost** (Mutter doesn't place a 700×520 window at 0,0; the client can't know
     its surface position, so `getExtents(SCREEN)` falls back to a 0,0 origin);
   - child nodes are **inconsistent** — e.g. `screen={x:0,y:0,…}` while `window={x:0,y:46,…}` (the
     screen coords dropped the offset).

   The **WINDOW-relative** extents ARE correct and self-consistent (`window={0,46,…}`). So the only
   missing piece is the **window's true on-screen origin**, which only the compositor knows. **This
   confirms the Phase 2 premise:** a focused-window Mutter helper (window origin) fused with AT-SPI's
   window-relative extents yields true global coordinates. Lesson: trust raw frame values, not the
   `scr=win` summary heuristic (which counted "screen≠window" as good even when the difference is
   garbage). Background/unfocused windows still to measure.

   **✅ Phase 2 closed this gap (validated).** The `guest/mutter-geometry-helper/` GNOME Shell extension
   publishes the compositor's true window frame over `org.nucleic.Geometry`; the agent correlates the
   AT-SPI app to that window by title and reports `frame = origin + getExtents(WINDOW)`. On a rebuilt
   26.04 clone the extension returned gnome-text-editor at global `(610, 356, 700, 520)`, and the agent's
   `ax_dump` frames went from the broken `{x:0,y:0}` to **true global `{x:610,y:356}`** (header) /
   `{x:610,y:402}` (content). The coordinate fallback (host clicks a non-semantic element's global frame
   center) rides on these now-accurate coordinates.
3. **Walk latency is small.** Full walk: client apps ~0.1 s; gnome-shell's ~1.4k-node tree ~0.8 s.
   Phase 4 cache/diff work isn't urgent for these; re-measure on a browser (largest tree).
4. **Mutter SDK input/capture smoke — not done** (the host-side USB pointer + virtio-gpu capture remain
   the injection/capture path regardless; only relevant if Phase 2 needs compositor input).

---

## Exit criteria (Phase 0)

- [x] Semantic-action coverage % known — **100%** on GTK4 (text-editor, console) + gnome-shell. Qt /
      Electron / browser still to measure (not in the base image).
- [x] Known whether `get_extents(SCREEN)` is usable or broken — **BROKEN** (window on-screen origin
      lost; reported at 0,0). WINDOW-relative extents are correct. → focused-window Mutter helper needed.
- [ ] Known whether **background-window** geometry is needed / works (only focused measured so far).
- [x] Full-tree walk latency recorded — client apps ~0.1 s, gnome-shell ~0.8 s. Browser still to measure.
- [x] Compositor decision recorded — **GNOME/Mutter, Ubuntu 26.04 LTS** (§9 Q4).

---

## Open items surfaced during Phase 0 setup

- **Compositor migration — done and rebuild-validated.** The base was provisioning **labwc**; it now
  provisions **GNOME / Mutter** on Wayland (Ubuntu pin **26.04 LTS "resolute"**, `gdm3` auto-login,
  AT-SPI on system-wide). A full host base rebuild now **passes end-to-end**: the two-boot build
  completes (`provisioned: true`), a clone boots to **Ubuntu 26.04 LTS (kernel 7.0.0-27-generic)**, and
  the in-guest agent answers over **vsock** (control-plane PASS).

  Getting there required fixing a cascade of 26.04-vs-24.04 kernel regressions (all landed): EFI-zboot
  kernel → pipeline unwraps to a raw Image; virtiofs-now-a-module → payload delivered as a raw tar on a
  2nd virtio-blk disk; busybox `mke2fs`/no-`install`; and — the key one — the external kernel's modules
  aren't in the cloud rootfs, so we **bake `linux-modules-<ver>` into the rootfs** (scripts/fetch-kernel-modules.sh)
  and run **depmod + load vsock/virtio_gpu early** (`nucleic-modsetup.service`) before the agent.

  **Liveness confirmed** (clone of the sealed base, checks over vsock): `graphical.target`, **GDM
  active**, `gnome-shell --mode=user` running; packages present (gnome-shell/session, gdm3,
  gnome-console, gnome-text-editor, at-spi2-core, python3-gi, gir1.2-atspi-2.0, xwayland);
  `toolkit-accessibility=true`; `python3 → Atspi import OK`; `modules.dep` present, `vsock` + `virtio_gpu`
  loaded. So the earlier empty screenshot was the headless CLI spike (no AppKit capture surface), not a
  missing GPU driver — verify the surface path in the app; the `ax_*` semantic path doesn't need it.

  **Still open (Phase 0 measurement proper):** run `scripts/a11y-probe.py` against GUI apps from within
  the session a11y bus to fill the coverage/geometry tables above.

- **Phase 1 end-to-end confirmed.** Driving `ax_dump` over vsock 2036 (host → Rust a11y agent → AT-SPI)
  on a live 26.04 clone returned the `gnome-shell` tree with the macOS node schema (refs, roles, titles,
  actions, children, frames). This validates the full semantic path. (The `frame`s on gnome-shell looked
  plausible, but gnome-shell is the compositor; the follow-up per-app frame inspection — §Findings 2 —
  showed **client-app SCREEN geometry is actually broken**.)
