Resolve nvrsion trunk->dev conflict: take trunk for a11y-agent + Linux VM semantic docs

dev held superseded drafts (early 'AT-SPI geometry may be usable' hypothesis
and the pre-compositor-helper op_dump); trunk carries the confirmed Phase 2
work. Adopt trunk's version of these files so promotion merges cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
2026-07-09 06:43:59 +00:00
co-authored by Claude Opus 4.8
parent cd93e31fe7
commit e2ea449791
3 changed files with 243 additions and 62 deletions
+14 -1
View File
@@ -40,8 +40,21 @@ are only entered if measurement forces them.
titles, actions, children — and populated `frame`s). One zbus gotcha fixed: AT-SPI objects don't
implement `Properties.GetAll`, so proxies must be built with `CacheProperties::No` (direct `Get`).
- **Phase 0 measured + Phase 2 built & validated.** The probe (`scripts/a11y-probe.py`) run showed
**100% semantic-action coverage** on GTK4 but **broken AT-SPI SCREEN geometry** (window on-screen
origin lost → reported at 0,0). Per that data (lean Phase 2, `docs/M_A11Y_RESULTS.md`), the
focused-window geometry gap is closed by a **GNOME Shell extension** (`guest/mutter-geometry-helper/`,
`org.nucleic.Geometry`) that publishes the compositor's true window origin; the agent correlates the
AT-SPI app to that window by title and reports `frame = origin + getExtents(WINDOW)`. **Validated on a
26.04 clone:** gnome-text-editor frames went from broken `{x:0,y:0}` to true global `{x:610,y:356}`
(matching Mutter). Plus a **coordinate fallback** (non-semantic element → agent returns its global
frame → host clicks the center via the surface pointer). CI (`.github/workflows/linux-vm-agents.yml`)
builds + distributes all guest artifacts. The heavy all-surface Newton port (Phase 3) is **deferred**
100% semantic coverage means coordinates are the minority path, and the focused-window helper covers it.
**Resolved decisions (see §9):** a11y-agent language = **Rust**; compositor = **GNOME/Mutter (26.04
LTS)**; transport = **second vsock port (2036)**, C `ping`/`exec` agent untouched on 2035.
LTS)**; transport = **second vsock port (2036)**, C `ping`/`exec` agent untouched on 2035; Phase 3
provider = **AT-SPI trees + compositor geometry** (no toolkit push), deferred pending non-GTK data.
---
+74 -33
View File
@@ -5,10 +5,14 @@ before building the agent, decide empirically (a) how far semantic-only actions
broken AT-SPI screen geometry actually is on our target apps, so Phases 13 are scoped by data rather
than by blog posts. Mirrors [M0_RESULTS.md](M0_RESULTS.md).
**Status: 🟡 pending real-guest run.** The probe (`scripts/a11y-probe.py`) is written and compiles, but
has **not yet been run in a live GNOME/Wayland guest** — the build container and macOS host have no
AT-SPI bus. This doc is the template the probe fills; the tables below are placeholders until a run
lands. (Same "pending real-guest validation" discipline as the macOS agent.)
**Status: ✅ measured + geometry solved** (Ubuntu 26.04 LTS / GNOME 50 / Mutter). Headlines: **100%
semantic-action coverage** on GTK4 (coordinates rarely needed to *drive* the UI); the AT-SPI *client*
API can't report a window's on-screen origin on Wayland, but that's **not a real limitation for us**
the whole display/input path is host-owned virtio in ONE coordinate space, so the compositor knows the
true geometry and Phase 2 sources it. Verified chain: VZ virtio-gpu scanout **1920×1200** = Mutter
current mode **1920×1200 @ scale 1.0** = host JPEG capture = USB pointer (1:1) = the geometry helper's
window coords. So `ax_dump` frames are **true, directly-clickable screen pixels** (validated: broken
`{0,0}` → correct `{610,356}`). Still to measure: Qt / Electron / browser apps.
---
@@ -63,33 +67,74 @@ _Filled from `scripts/a11y-probe.py` output. `cover` = semantically-actionable
`scr=0` / `scr=win` = fraction of component nodes whose SCREEN extents are zero-sized / identical to
WINDOW-relative extents (the two Wayland-broken-geometry symptoms)._
| App | nodes | interactive | coverage | walk(s) | component-nodes | scr=0 | scr=win | geometry |
|---|---|---|---|---|---|---|---|---|
| _tbd_ | | | | | | | | |
First run — **Ubuntu 26.04 LTS, GNOME 50 / Mutter, Wayland** (2026-07-09), via `scripts/a11y-probe.py`
against the auto-login session's live apps:
### Findings (to be confirmed by the run)
| App | toolkit | nodes | interactive | coverage | walk(s) | comp-nodes | scr=0 | scr=win | geometry |
|---|---|---|---|---|---|---|---|---|---|
| gnome-shell | GTK/compositor | 1388 | 1101 | **100%** | 0.80 | 1387 | 92% | 100% | broken\* |
| gnome-text-editor | GTK4 | 83 | 11 | **100%** | 0.13 | 82 | 34% | 16% | **usable** |
| kgx (gnome-console) | GTK4 | 63 | 10 | **100%** | 0.10 | 62 | 23% | 18% | **usable** |
| xdg-desktop-portal-gtk | GTK | 1 | 0 | — | 0.00 | 0 | — | — | n/a |
1. **Semantic coverage — 🟡 unmeasured.** _Expected high for GTK/Qt (most controls expose an Action,
EditableText, Value, or are Focusable); the question is the actual %, and which control classes fall
through to needing coordinates._
2. **Screen geometry — 🟡 unmeasured.** _Expected broken on Wayland (`get_extents(SCREEN)` zeros or ==
WINDOW). If confirmed, Phase 2 focused-window fusion is justified; if surprisingly usable on our
compositor, Phase 2/3 can be deferred._
3. **Walk latency — 🟡 unmeasured.** _AT-SPI is D-Bus-chatty; record full-walk time per app, especially
the browser, to size the Phase 4 cache/diff work._
4. **Mutter SDK input/capture smoke — 🟡 not done.** _Confirm the virtual-monitor / libei handles are
reachable in the guest (informs whether host-side USB pointer remains the injection path)._
\* gnome-shell **is** the compositor, so its internal nodes trivially have screen == window (scr=win
100%); this is not a client-geometry signal. The client apps (text-editor, console) are the real datum.
### Findings
1. **Semantic-action coverage = 100%** on every real app. Every interactive control is reachable by an
`ax_*` semantic action (Action / EditableText / Value / Focusable) with **no coordinates**. This is
the headline result: **semantic-first is fully sufficient for actionability on GTK4/Wayland**, so
geometry barely matters for *driving* the UI (only for overlays/annotation or the rare non-semantic
control). ⚠️ Still to measure: **Qt, Electron/Chromium, Firefox** (not installed in the base) — the
real coverage ceiling is toolkit-dependent (risk table §6).
2. **AT-SPI's SCREEN extents are origin-blind — but the *system* geometry is fully known.** The
AT-SPI *client* API can't report a window's on-screen origin on Wayland (by design). But this is
**not a real limitation here**, because our entire display/input path is host-owned virtio in a
single coordinate space: the VZ virtio-gpu scanout is 1920×1200, Mutter's current mode is 1920×1200
@ scale 1.0, the host captures that scanout as a 1920×1200 JPEG, and the USB pointer targets those
same pixels 1:1. So the compositor's window coordinates ARE screenshot pixels and pointer targets;
we just have to read them from the compositor (which knows) rather than the AT-SPI client (which
doesn't). That scale-1 invariant is pinned in provisioning so it can't drift.
Dumping gnome-text-editor's raw AT-SPI frame values shows the client-side blindness (initially I
misread the `scr=win` summary as "usable" — wrong):
- the **window root reports `SCREEN = {x:0, y:0, w:700, h:520}`** — the size is right but the
on-screen **origin is lost** (Mutter doesn't place a 700×520 window at 0,0; the client can't know
its surface position, so `getExtents(SCREEN)` falls back to a 0,0 origin);
- child nodes are **inconsistent** — e.g. `screen={x:0,y:0,…}` while `window={x:0,y:46,…}` (the
screen coords dropped the offset).
The **WINDOW-relative** extents ARE correct and self-consistent (`window={0,46,…}`). So the only
missing piece is the **window's true on-screen origin**, which only the compositor knows. **This
confirms the Phase 2 premise:** a focused-window Mutter helper (window origin) fused with AT-SPI's
window-relative extents yields true global coordinates. Lesson: trust raw frame values, not the
`scr=win` summary heuristic (which counted "screen≠window" as good even when the difference is
garbage). Background/unfocused windows still to measure.
**✅ Phase 2 closed this gap (validated).** The `guest/mutter-geometry-helper/` GNOME Shell extension
publishes the compositor's true window frame over `org.nucleic.Geometry`; the agent correlates the
AT-SPI app to that window by title and reports `frame = origin + getExtents(WINDOW)`. On a rebuilt
26.04 clone the extension returned gnome-text-editor at global `(610, 356, 700, 520)`, and the agent's
`ax_dump` frames went from the broken `{x:0,y:0}` to **true global `{x:610,y:356}`** (header) /
`{x:610,y:402}` (content). The coordinate fallback (host clicks a non-semantic element's global frame
center) rides on these now-accurate coordinates.
3. **Walk latency is small.** Full walk: client apps ~0.1 s; gnome-shell's ~1.4k-node tree ~0.8 s.
Phase 4 cache/diff work isn't urgent for these; re-measure on a browser (largest tree).
4. **Mutter SDK input/capture smoke — not done** (the host-side USB pointer + virtio-gpu capture remain
the injection/capture path regardless; only relevant if Phase 2 needs compositor input).
---
## Exit criteria (Phase 0)
- [ ] Semantic-action coverage % known for each app in the set.
- [ ] Known whether `get_extents(SCREEN)` is usable or broken on the target compositor.
- [ ] Known whether **background-window** geometry is even needed for our workloads (or focused-window
only suffices → Phase 3 may be unnecessary).
- [ ] Full-tree walk latency recorded per app (sizes Phase 4 perf work).
- [ ] Compositor decision recorded (GNOME/Mutter vs labwc vs KWin spot-check — §9 Q4).
- [x] Semantic-action coverage % known **100%** on GTK4 (text-editor, console) + gnome-shell. Qt /
Electron / browser still to measure (not in the base image).
- [x] Known whether `get_extents(SCREEN)` is usable or broken — **BROKEN** (window on-screen origin
lost; reported at 0,0). WINDOW-relative extents are correct. → focused-window Mutter helper needed.
- [ ] Known whether **background-window** geometry is needed / works (only focused measured so far).
- [x] Full-tree walk latency recorded — client apps ~0.1 s, gnome-shell ~0.8 s. Browser still to measure.
- [x] Compositor decision recorded — **GNOME/Mutter, Ubuntu 26.04 LTS** (§9 Q4).
---
@@ -117,12 +162,8 @@ WINDOW-relative extents (the two Wayland-broken-geometry symptoms)._
**Still open (Phase 0 measurement proper):** run `scripts/a11y-probe.py` against GUI apps from within
the session a11y bus to fill the coverage/geometry tables above.
- **Phase 1 end-to-end confirmed + an early geometry signal.** Driving `ax_dump` over vsock 2036 (host →
Rust a11y agent → AT-SPI) on a live 26.04 clone returned the `gnome-shell` tree with the macOS node
schema. Notably the `frame`s came back with **real screen-looking coordinates** (top panel
`{x:0,y:0,w:1920,h:1200}`; nested nodes at plausible absolute x/y from `Component.GetExtents(SCREEN)`).
If the probe run confirms this across GTK4/Qt apps, **AT-SPI SCREEN geometry may be usable on GNOME
50/Mutter** — which would shrink or remove the need for Phase 2/3's compositor-sourced geometry. This
contradicts the "Wayland geometry is broken" premise and is the single most important thing for the
probe run to pin down. (Caveat: `gnome-shell` is itself the compositor, so its own geometry being
correct doesn't prove client apps get correct SCREEN extents — measure real app windows.)
- **Phase 1 end-to-end confirmed.** Driving `ax_dump` over vsock 2036 (host → Rust a11y agent → AT-SPI)
on a live 26.04 clone returned the `gnome-shell` tree with the macOS node schema (refs, roles, titles,
actions, children, frames). This validates the full semantic path. (The `frame`s on gnome-shell looked
plausible, but gnome-shell is the compositor; the follow-up per-app frame inspection — §Findings 2 —
showed **client-app SCREEN geometry is actually broken**.)
+155 -28
View File
@@ -110,6 +110,18 @@ trait Text {
fn get_text(&self, start_offset: i32, end_offset: i32) -> zbus::Result<String>;
}
/// The Nucleic geometry helper (a GNOME Shell extension, docs/LINUX_VM_SEMANTIC_AGENT.md Phase 2) —
/// the compositor's source of truth for the focused window's global origin, which AT-SPI can't provide
/// on Wayland. Lives on the SESSION bus (where gnome-shell publishes it), not the a11y bus.
#[zbus::proxy(
interface = "org.nucleic.Geometry",
default_service = "org.nucleic.Geometry",
default_path = "/org/nucleic/Geometry"
)]
trait Geometry {
fn get_focused_window_frame(&self) -> zbus::Result<(i32, i32, i32, i32, String)>;
}
/// An AT-SPI object address: the app's unique D-Bus name + the object path.
type ObjRef = (String, String);
@@ -131,6 +143,9 @@ struct DumpState {
conn: zbus::Connection,
refs: Vec<ObjRef>, // index i → ref "e{i+1}"
budget: i32,
/// The focused window's global origin from the compositor (Phase 2). When present, node frames are
/// `origin + getExtents(WINDOW)` (true global); when absent, we fall back to `getExtents(SCREEN)`.
origin: Option<(i32, i32)>,
}
impl DumpState {
@@ -197,7 +212,7 @@ impl DumpState {
}
if has_component {
if let Some(frame) = read_frame(&self.conn, dest, path).await {
if let Some(frame) = read_frame(&self.conn, dest, path, self.origin).await {
node.insert("frame".into(), frame);
}
}
@@ -267,11 +282,22 @@ async fn read_value(
None
}
/// The element's on-screen frame. Prefer SCREEN extents; fall back to WINDOW-relative (Wayland often
/// can't provide real SCREEN geometry — Phase 2/3 sources it from the compositor). Omitted if zero.
async fn read_frame(conn: &zbus::Connection, dest: &str, path: &str) -> Option<Value> {
/// The element's global on-screen frame. AT-SPI's `getExtents(SCREEN)` loses the window origin on
/// Wayland (docs/M_A11Y_RESULTS.md), so when the compositor gave us the focused window's `origin`
/// (Phase 2) we compute the true global frame as `origin + getExtents(WINDOW)`. Without the helper we
/// fall back to SCREEN (best-effort, likely origin-less) then WINDOW-relative. Omitted if zero-size.
async fn read_frame(
conn: &zbus::Connection, dest: &str, path: &str, origin: Option<(i32, i32)>,
) -> Option<Value> {
let p = ComponentProxy::builder(conn)
.destination(dest.to_owned()).ok()?.path(path.to_owned()).ok()?.build().await.ok()?;
if let Some((ox, oy)) = origin {
if let Ok((x, y, w, h)) = p.get_extents(1).await {
if w > 0 && h > 0 {
return Some(json!({"x": ox + x, "y": oy + y, "w": w, "h": h}));
}
}
}
for coord in [0u32, 1u32] {
if let Ok((x, y, w, h)) = p.get_extents(coord).await {
if w > 0 && h > 0 {
@@ -329,12 +355,70 @@ async fn find_active_app(conn: &zbus::Connection) -> Result<ObjRef, String> {
// ---- op dispatch ----------------------------------------------------------------------------------
/// Per-connection state: the a11y-bus connection (shared) + the last dump's ref registry.
/// Per-connection state: the a11y-bus connection + the session-bus connection (for the compositor
/// geometry helper) + the last dump's ref registry.
struct Session {
conn: zbus::Connection,
conn: zbus::Connection, // AT-SPI a11y bus
session: zbus::Connection, // session bus (org.nucleic.Geometry lives here)
refs: Vec<ObjRef>,
}
impl Session {
/// The active window's global origin + title from the compositor geometry helper, or `None` if the
/// helper isn't present (extension not installed/enabled). The title lets `ax_dump` pick the
/// matching AT-SPI app (so the dump + geometry agree even without keyboard focus).
async fn focused_window(&self) -> Option<(i32, i32, String)> {
let g = GeometryProxy::new(&self.session).await.ok()?;
let (x, y, w, h, title) = g.get_focused_window_frame().await.ok()?;
if w > 0 && h > 0 {
Some((x, y, title))
} else {
None
}
}
/// Just the origin (for the coordinate-fallback in `ax_action`).
async fn focused_origin(&self) -> Option<(i32, i32)> {
self.focused_window().await.map(|(x, y, _)| (x, y))
}
}
/// Find the application whose top-level window's title matches `title` (from the compositor helper), so
/// the dump targets the same window the helper reports the geometry for — independent of AT-SPI's
/// focus state. Returns the application's object ref.
async fn find_app_by_window_title(conn: &zbus::Connection, title: &str) -> Option<ObjRef> {
let root = accessible(conn, A11Y_ROOT_DEST, A11Y_ROOT_PATH).await.ok()?;
let n = root.child_count().await.ok()?;
for i in 0..n {
let (adest, apath) = match root.get_child_at_index(i).await {
Ok(v) => v,
Err(_) => continue,
};
let apath = apath.as_str().to_owned();
if adest.is_empty() {
continue;
}
if let Ok(app) = accessible(conn, &adest, &apath).await {
let wn = app.child_count().await.unwrap_or(0);
for j in 0..wn {
if let Ok((wd, wp)) = app.get_child_at_index(j).await {
let wd = if wd.is_empty() { adest.clone() } else { wd };
if let Ok(win) = accessible(conn, &wd, wp.as_str()).await {
if let Ok(name) = win.name().await {
if !name.is_empty()
&& (name == title || title.contains(&name) || name.contains(title))
{
return Some((adest, apath));
}
}
}
}
}
}
}
None
}
fn err(msg: impl Into<String>) -> Value {
json!({"ok": false, "error": msg.into()})
}
@@ -356,11 +440,32 @@ impl Session {
}
async fn op_dump(&mut self) -> Value {
let (dest, path) = match find_active_app(&self.conn).await {
Ok(a) => a,
Err(why) => return err(format!("no accessible application to dump: {why}")),
// Prefer the compositor helper's active window: it gives both the true global origin AND the
// title, so we dump the exact window it measured — correct even with no AT-SPI focus. Fall back
// to AT-SPI's own active-app heuristic (and no origin) when the helper isn't present.
let (dest, path, origin) = match self.focused_window().await {
Some((ox, oy, title))
if !title.is_empty() =>
{
match find_app_by_window_title(&self.conn, &title).await {
Some((d, p)) => (d, p, Some((ox, oy))),
None => match find_active_app(&self.conn).await {
Ok((d, p)) => (d, p, None),
Err(why) => return err(format!("no accessible application to dump: {why}")),
},
}
}
_ => match find_active_app(&self.conn).await {
Ok((d, p)) => (d, p, None),
Err(why) => return err(format!("no accessible application to dump: {why}")),
},
};
let mut st = DumpState {
conn: self.conn.clone(),
refs: Vec::new(),
budget: NODE_BUDGET,
origin,
};
let mut st = DumpState { conn: self.conn.clone(), refs: Vec::new(), budget: NODE_BUDGET };
let tree = st.build_node(&dest, &path, 0).await;
let app = accessible(&self.conn, &dest, &path).await.ok();
let app_name = match app {
@@ -392,24 +497,42 @@ impl Session {
Err(e) => return e,
};
let want = req.get("action").and_then(|v| v.as_str()).filter(|s| !s.is_empty());
let p = match ActionProxy::builder(&self.conn)
.destination(dest).and_then(|b| b.path(path))
// Gather this element's semantic actions (empty if it has no Action interface at all).
let actions = match ActionProxy::builder(&self.conn)
.destination(dest.clone())
.and_then(|b| b.path(path.clone()))
{
Ok(b) => match b.build().await {
Ok(p) => p,
Err(_) => return err("this element has no Action interface — try ax_focus, or set_value"),
Ok(p) => (p.get_actions().await.unwrap_or_default(), Some(p)),
Err(_) => (Vec::new(), None),
},
Err(e) => return err(format!("bad ref: {e}")),
};
let actions = p.get_actions().await.unwrap_or_default();
let (names, proxy) = actions;
// No semantic action available (and none was named) → coordinate fallback: hand the host the
// element's true global frame so it can click the center via the surface pointer. This is the
// rare path — Phase 0 measured ~100% semantic-action coverage on GTK.
if names.is_empty() && want.is_none() {
let origin = self.focused_origin().await;
return match read_frame(&self.conn, &dest, &path, origin).await {
Some(frame) => json!({
"ok": true, "fallback": "click", "frame": frame,
"note": "no semantic action; click the frame center"
}),
None => err("this element has no semantic action and no geometry to click"),
};
}
let proxy = proxy.unwrap(); // names non-empty ⇒ proxy present
let index = match want {
Some(name) => match actions.iter().position(|(n, _, _)| n.eq_ignore_ascii_case(name)) {
Some(name) => match names.iter().position(|(n, _, _)| n.eq_ignore_ascii_case(name)) {
Some(i) => i as i32,
None => return err(format!("no action named '{name}' on this element")),
},
None => 0, // the primary/default action (macOS AXPress equivalent)
};
match p.do_action(index).await {
match proxy.do_action(index).await {
Ok(true) => json!({"ok": true}),
Ok(false) => json!({"ok": true, "note": "the toolkit reported the action did not run; re-dump to verify"}),
Err(e) => err(format!("performing the action failed: {e}")),
@@ -464,10 +587,10 @@ impl Session {
// ---- connection + vsock server -------------------------------------------------------------------
async fn handle_connection(stream: VsockStream, a11y: zbus::Connection) {
async fn handle_connection(stream: VsockStream, a11y: zbus::Connection, sbus: zbus::Connection) {
let (rd, mut wr) = tokio::io::split(stream);
let mut lines = BufReader::new(rd).lines();
let mut session = Session { conn: a11y, refs: Vec::new() };
let mut session = Session { conn: a11y, session: sbus, refs: Vec::new() };
while let Ok(Some(line)) = lines.next_line().await {
if line.trim().is_empty() {
continue;
@@ -485,12 +608,14 @@ async fn handle_connection(stream: VsockStream, a11y: zbus::Connection) {
}
}
/// Connect to the session's AT-SPI a11y bus (org.a11y.Bus → GetAddress → the a11y bus).
async fn connect_a11y() -> zbus::Result<zbus::Connection> {
/// Connect to the session bus and, via it, the AT-SPI a11y bus (org.a11y.Bus → GetAddress → the a11y
/// bus). Returns both: the session bus (for the org.nucleic.Geometry helper) and the a11y bus.
async fn connect_buses() -> zbus::Result<(zbus::Connection, zbus::Connection)> {
let session = zbus::Connection::session().await?;
let bus = A11yBusProxy::new(&session).await?;
let address = bus.get_address().await?;
zbus::connection::Builder::address(address.as_str())?.build().await
let a11y = zbus::connection::Builder::address(address.as_str())?.build().await?;
Ok((session, a11y))
}
#[tokio::main]
@@ -500,7 +625,7 @@ async fn main() {
.and_then(|s| s.parse::<u32>().ok())
.unwrap_or(DEFAULT_PORT);
let a11y = match connect_a11y().await {
let (sbus, a11y) = match connect_buses().await {
Ok(c) => c,
Err(e) => {
eprintln!("[nucleic-a11y-agent] FATAL: could not reach the AT-SPI a11y bus: {e}");
@@ -508,9 +633,8 @@ async fn main() {
std::process::exit(1);
}
};
// Shared behind an Arc<Mutex<>> only to clone cheaply per connection; zbus::Connection is itself
// cheaply clonable, so we just clone it.
let a11y = Arc::new(Mutex::new(a11y));
// zbus::Connection is cheaply clonable; the Mutex just guards the shared handles across accepts.
let buses = Arc::new(Mutex::new((sbus, a11y)));
let listener = match VsockListener::bind(VsockAddr::new(VMADDR_CID_ANY, port)) {
Ok(l) => l,
@@ -524,8 +648,11 @@ async fn main() {
loop {
match listener.accept().await {
Ok((stream, _addr)) => {
let conn = a11y.lock().await.clone();
tokio::spawn(handle_connection(stream, conn));
let (sbus, a11y) = {
let g = buses.lock().await;
(g.0.clone(), g.1.clone())
};
tokio::spawn(handle_connection(stream, a11y, sbus));
}
Err(e) => {
eprintln!("[nucleic-a11y-agent] accept failed: {e}");