Resolve nvrsion trunk->dev conflict: take trunk for a11y-agent + Linux VM semantic docs
dev held superseded drafts (early 'AT-SPI geometry may be usable' hypothesis and the pre-compositor-helper op_dump); trunk carries the confirmed Phase 2 work. Adopt trunk's version of these files so promotion merges cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Claude Opus 4.8
@@ -40,8 +40,21 @@ are only entered if measurement forces them.
|
||||
titles, actions, children — and populated `frame`s). One zbus gotcha fixed: AT-SPI objects don't
|
||||
implement `Properties.GetAll`, so proxies must be built with `CacheProperties::No` (direct `Get`).
|
||||
|
||||
- **Phase 0 measured + Phase 2 built & validated.** The probe (`scripts/a11y-probe.py`) run showed
|
||||
**100% semantic-action coverage** on GTK4 but **broken AT-SPI SCREEN geometry** (window on-screen
|
||||
origin lost → reported at 0,0). Per that data (lean Phase 2, `docs/M_A11Y_RESULTS.md`), the
|
||||
focused-window geometry gap is closed by a **GNOME Shell extension** (`guest/mutter-geometry-helper/`,
|
||||
`org.nucleic.Geometry`) that publishes the compositor's true window origin; the agent correlates the
|
||||
AT-SPI app to that window by title and reports `frame = origin + getExtents(WINDOW)`. **Validated on a
|
||||
26.04 clone:** gnome-text-editor frames went from broken `{x:0,y:0}` to true global `{x:610,y:356}`
|
||||
(matching Mutter). Plus a **coordinate fallback** (non-semantic element → agent returns its global
|
||||
frame → host clicks the center via the surface pointer). CI (`.github/workflows/linux-vm-agents.yml`)
|
||||
builds + distributes all guest artifacts. The heavy all-surface Newton port (Phase 3) is **deferred** —
|
||||
100% semantic coverage means coordinates are the minority path, and the focused-window helper covers it.
|
||||
|
||||
**Resolved decisions (see §9):** a11y-agent language = **Rust**; compositor = **GNOME/Mutter (26.04
|
||||
LTS)**; transport = **second vsock port (2036)**, C `ping`/`exec` agent untouched on 2035.
|
||||
LTS)**; transport = **second vsock port (2036)**, C `ping`/`exec` agent untouched on 2035; Phase 3
|
||||
provider = **AT-SPI trees + compositor geometry** (no toolkit push), deferred pending non-GTK data.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+74
-33
@@ -5,10 +5,14 @@ before building the agent, decide empirically (a) how far semantic-only actions
|
||||
broken AT-SPI screen geometry actually is on our target apps, so Phases 1–3 are scoped by data rather
|
||||
than by blog posts. Mirrors [M0_RESULTS.md](M0_RESULTS.md).
|
||||
|
||||
**Status: 🟡 pending real-guest run.** The probe (`scripts/a11y-probe.py`) is written and compiles, but
|
||||
has **not yet been run in a live GNOME/Wayland guest** — the build container and macOS host have no
|
||||
AT-SPI bus. This doc is the template the probe fills; the tables below are placeholders until a run
|
||||
lands. (Same "pending real-guest validation" discipline as the macOS agent.)
|
||||
**Status: ✅ measured + geometry solved** (Ubuntu 26.04 LTS / GNOME 50 / Mutter). Headlines: **100%
|
||||
semantic-action coverage** on GTK4 (coordinates rarely needed to *drive* the UI); the AT-SPI *client*
|
||||
API can't report a window's on-screen origin on Wayland, but that's **not a real limitation for us** —
|
||||
the whole display/input path is host-owned virtio in ONE coordinate space, so the compositor knows the
|
||||
true geometry and Phase 2 sources it. Verified chain: VZ virtio-gpu scanout **1920×1200** = Mutter
|
||||
current mode **1920×1200 @ scale 1.0** = host JPEG capture = USB pointer (1:1) = the geometry helper's
|
||||
window coords. So `ax_dump` frames are **true, directly-clickable screen pixels** (validated: broken
|
||||
`{0,0}` → correct `{610,356}`). Still to measure: Qt / Electron / browser apps.
|
||||
|
||||
---
|
||||
|
||||
@@ -63,33 +67,74 @@ _Filled from `scripts/a11y-probe.py` output. `cover` = semantically-actionable
|
||||
`scr=0` / `scr=win` = fraction of component nodes whose SCREEN extents are zero-sized / identical to
|
||||
WINDOW-relative extents (the two Wayland-broken-geometry symptoms)._
|
||||
|
||||
| App | nodes | interactive | coverage | walk(s) | component-nodes | scr=0 | scr=win | geometry |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| _tbd_ | | | | | | | | |
|
||||
First run — **Ubuntu 26.04 LTS, GNOME 50 / Mutter, Wayland** (2026-07-09), via `scripts/a11y-probe.py`
|
||||
against the auto-login session's live apps:
|
||||
|
||||
### Findings (to be confirmed by the run)
|
||||
| App | toolkit | nodes | interactive | coverage | walk(s) | comp-nodes | scr=0 | scr=win | geometry |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| gnome-shell | GTK/compositor | 1388 | 1101 | **100%** | 0.80 | 1387 | 92% | 100% | broken\* |
|
||||
| gnome-text-editor | GTK4 | 83 | 11 | **100%** | 0.13 | 82 | 34% | 16% | **usable** |
|
||||
| kgx (gnome-console) | GTK4 | 63 | 10 | **100%** | 0.10 | 62 | 23% | 18% | **usable** |
|
||||
| xdg-desktop-portal-gtk | GTK | 1 | 0 | — | 0.00 | 0 | — | — | n/a |
|
||||
|
||||
1. **Semantic coverage — 🟡 unmeasured.** _Expected high for GTK/Qt (most controls expose an Action,
|
||||
EditableText, Value, or are Focusable); the question is the actual %, and which control classes fall
|
||||
through to needing coordinates._
|
||||
2. **Screen geometry — 🟡 unmeasured.** _Expected broken on Wayland (`get_extents(SCREEN)` zeros or ==
|
||||
WINDOW). If confirmed, Phase 2 focused-window fusion is justified; if surprisingly usable on our
|
||||
compositor, Phase 2/3 can be deferred._
|
||||
3. **Walk latency — 🟡 unmeasured.** _AT-SPI is D-Bus-chatty; record full-walk time per app, especially
|
||||
the browser, to size the Phase 4 cache/diff work._
|
||||
4. **Mutter SDK input/capture smoke — 🟡 not done.** _Confirm the virtual-monitor / libei handles are
|
||||
reachable in the guest (informs whether host-side USB pointer remains the injection path)._
|
||||
\* gnome-shell **is** the compositor, so its internal nodes trivially have screen == window (scr=win
|
||||
100%); this is not a client-geometry signal. The client apps (text-editor, console) are the real datum.
|
||||
|
||||
### Findings
|
||||
|
||||
1. **Semantic-action coverage = 100%** on every real app. Every interactive control is reachable by an
|
||||
`ax_*` semantic action (Action / EditableText / Value / Focusable) with **no coordinates**. This is
|
||||
the headline result: **semantic-first is fully sufficient for actionability on GTK4/Wayland**, so
|
||||
geometry barely matters for *driving* the UI (only for overlays/annotation or the rare non-semantic
|
||||
control). ⚠️ Still to measure: **Qt, Electron/Chromium, Firefox** (not installed in the base) — the
|
||||
real coverage ceiling is toolkit-dependent (risk table §6).
|
||||
2. **AT-SPI's SCREEN extents are origin-blind — but the *system* geometry is fully known.** The
|
||||
AT-SPI *client* API can't report a window's on-screen origin on Wayland (by design). But this is
|
||||
**not a real limitation here**, because our entire display/input path is host-owned virtio in a
|
||||
single coordinate space: the VZ virtio-gpu scanout is 1920×1200, Mutter's current mode is 1920×1200
|
||||
@ scale 1.0, the host captures that scanout as a 1920×1200 JPEG, and the USB pointer targets those
|
||||
same pixels 1:1. So the compositor's window coordinates ARE screenshot pixels and pointer targets;
|
||||
we just have to read them from the compositor (which knows) rather than the AT-SPI client (which
|
||||
doesn't). That scale-1 invariant is pinned in provisioning so it can't drift.
|
||||
|
||||
Dumping gnome-text-editor's raw AT-SPI frame values shows the client-side blindness (initially I
|
||||
misread the `scr=win` summary as "usable" — wrong):
|
||||
- the **window root reports `SCREEN = {x:0, y:0, w:700, h:520}`** — the size is right but the
|
||||
on-screen **origin is lost** (Mutter doesn't place a 700×520 window at 0,0; the client can't know
|
||||
its surface position, so `getExtents(SCREEN)` falls back to a 0,0 origin);
|
||||
- child nodes are **inconsistent** — e.g. `screen={x:0,y:0,…}` while `window={x:0,y:46,…}` (the
|
||||
screen coords dropped the offset).
|
||||
|
||||
The **WINDOW-relative** extents ARE correct and self-consistent (`window={0,46,…}`). So the only
|
||||
missing piece is the **window's true on-screen origin**, which only the compositor knows. **This
|
||||
confirms the Phase 2 premise:** a focused-window Mutter helper (window origin) fused with AT-SPI's
|
||||
window-relative extents yields true global coordinates. Lesson: trust raw frame values, not the
|
||||
`scr=win` summary heuristic (which counted "screen≠window" as good even when the difference is
|
||||
garbage). Background/unfocused windows still to measure.
|
||||
|
||||
**✅ Phase 2 closed this gap (validated).** The `guest/mutter-geometry-helper/` GNOME Shell extension
|
||||
publishes the compositor's true window frame over `org.nucleic.Geometry`; the agent correlates the
|
||||
AT-SPI app to that window by title and reports `frame = origin + getExtents(WINDOW)`. On a rebuilt
|
||||
26.04 clone the extension returned gnome-text-editor at global `(610, 356, 700, 520)`, and the agent's
|
||||
`ax_dump` frames went from the broken `{x:0,y:0}` to **true global `{x:610,y:356}`** (header) /
|
||||
`{x:610,y:402}` (content). The coordinate fallback (host clicks a non-semantic element's global frame
|
||||
center) rides on these now-accurate coordinates.
|
||||
3. **Walk latency is small.** Full walk: client apps ~0.1 s; gnome-shell's ~1.4k-node tree ~0.8 s.
|
||||
Phase 4 cache/diff work isn't urgent for these; re-measure on a browser (largest tree).
|
||||
4. **Mutter SDK input/capture smoke — not done** (the host-side USB pointer + virtio-gpu capture remain
|
||||
the injection/capture path regardless; only relevant if Phase 2 needs compositor input).
|
||||
|
||||
---
|
||||
|
||||
## Exit criteria (Phase 0)
|
||||
|
||||
- [ ] Semantic-action coverage % known for each app in the set.
|
||||
- [ ] Known whether `get_extents(SCREEN)` is usable or broken on the target compositor.
|
||||
- [ ] Known whether **background-window** geometry is even needed for our workloads (or focused-window
|
||||
only suffices → Phase 3 may be unnecessary).
|
||||
- [ ] Full-tree walk latency recorded per app (sizes Phase 4 perf work).
|
||||
- [ ] Compositor decision recorded (GNOME/Mutter vs labwc vs KWin spot-check — §9 Q4).
|
||||
- [x] Semantic-action coverage % known — **100%** on GTK4 (text-editor, console) + gnome-shell. Qt /
|
||||
Electron / browser still to measure (not in the base image).
|
||||
- [x] Known whether `get_extents(SCREEN)` is usable or broken — **BROKEN** (window on-screen origin
|
||||
lost; reported at 0,0). WINDOW-relative extents are correct. → focused-window Mutter helper needed.
|
||||
- [ ] Known whether **background-window** geometry is needed / works (only focused measured so far).
|
||||
- [x] Full-tree walk latency recorded — client apps ~0.1 s, gnome-shell ~0.8 s. Browser still to measure.
|
||||
- [x] Compositor decision recorded — **GNOME/Mutter, Ubuntu 26.04 LTS** (§9 Q4).
|
||||
|
||||
---
|
||||
|
||||
@@ -117,12 +162,8 @@ WINDOW-relative extents (the two Wayland-broken-geometry symptoms)._
|
||||
**Still open (Phase 0 measurement proper):** run `scripts/a11y-probe.py` against GUI apps from within
|
||||
the session a11y bus to fill the coverage/geometry tables above.
|
||||
|
||||
- **Phase 1 end-to-end confirmed + an early geometry signal.** Driving `ax_dump` over vsock 2036 (host →
|
||||
Rust a11y agent → AT-SPI) on a live 26.04 clone returned the `gnome-shell` tree with the macOS node
|
||||
schema. Notably the `frame`s came back with **real screen-looking coordinates** (top panel
|
||||
`{x:0,y:0,w:1920,h:1200}`; nested nodes at plausible absolute x/y from `Component.GetExtents(SCREEN)`).
|
||||
If the probe run confirms this across GTK4/Qt apps, **AT-SPI SCREEN geometry may be usable on GNOME
|
||||
50/Mutter** — which would shrink or remove the need for Phase 2/3's compositor-sourced geometry. This
|
||||
contradicts the "Wayland geometry is broken" premise and is the single most important thing for the
|
||||
probe run to pin down. (Caveat: `gnome-shell` is itself the compositor, so its own geometry being
|
||||
correct doesn't prove client apps get correct SCREEN extents — measure real app windows.)
|
||||
- **Phase 1 end-to-end confirmed.** Driving `ax_dump` over vsock 2036 (host → Rust a11y agent → AT-SPI)
|
||||
on a live 26.04 clone returned the `gnome-shell` tree with the macOS node schema (refs, roles, titles,
|
||||
actions, children, frames). This validates the full semantic path. (The `frame`s on gnome-shell looked
|
||||
plausible, but gnome-shell is the compositor; the follow-up per-app frame inspection — §Findings 2 —
|
||||
showed **client-app SCREEN geometry is actually broken**.)
|
||||
|
||||
@@ -110,6 +110,18 @@ trait Text {
|
||||
fn get_text(&self, start_offset: i32, end_offset: i32) -> zbus::Result<String>;
|
||||
}
|
||||
|
||||
/// The Nucleic geometry helper (a GNOME Shell extension, docs/LINUX_VM_SEMANTIC_AGENT.md Phase 2) —
|
||||
/// the compositor's source of truth for the focused window's global origin, which AT-SPI can't provide
|
||||
/// on Wayland. Lives on the SESSION bus (where gnome-shell publishes it), not the a11y bus.
|
||||
#[zbus::proxy(
|
||||
interface = "org.nucleic.Geometry",
|
||||
default_service = "org.nucleic.Geometry",
|
||||
default_path = "/org/nucleic/Geometry"
|
||||
)]
|
||||
trait Geometry {
|
||||
fn get_focused_window_frame(&self) -> zbus::Result<(i32, i32, i32, i32, String)>;
|
||||
}
|
||||
|
||||
/// An AT-SPI object address: the app's unique D-Bus name + the object path.
|
||||
type ObjRef = (String, String);
|
||||
|
||||
@@ -131,6 +143,9 @@ struct DumpState {
|
||||
conn: zbus::Connection,
|
||||
refs: Vec<ObjRef>, // index i → ref "e{i+1}"
|
||||
budget: i32,
|
||||
/// The focused window's global origin from the compositor (Phase 2). When present, node frames are
|
||||
/// `origin + getExtents(WINDOW)` (true global); when absent, we fall back to `getExtents(SCREEN)`.
|
||||
origin: Option<(i32, i32)>,
|
||||
}
|
||||
|
||||
impl DumpState {
|
||||
@@ -197,7 +212,7 @@ impl DumpState {
|
||||
}
|
||||
|
||||
if has_component {
|
||||
if let Some(frame) = read_frame(&self.conn, dest, path).await {
|
||||
if let Some(frame) = read_frame(&self.conn, dest, path, self.origin).await {
|
||||
node.insert("frame".into(), frame);
|
||||
}
|
||||
}
|
||||
@@ -267,11 +282,22 @@ async fn read_value(
|
||||
None
|
||||
}
|
||||
|
||||
/// The element's on-screen frame. Prefer SCREEN extents; fall back to WINDOW-relative (Wayland often
|
||||
/// can't provide real SCREEN geometry — Phase 2/3 sources it from the compositor). Omitted if zero.
|
||||
async fn read_frame(conn: &zbus::Connection, dest: &str, path: &str) -> Option<Value> {
|
||||
/// The element's global on-screen frame. AT-SPI's `getExtents(SCREEN)` loses the window origin on
|
||||
/// Wayland (docs/M_A11Y_RESULTS.md), so when the compositor gave us the focused window's `origin`
|
||||
/// (Phase 2) we compute the true global frame as `origin + getExtents(WINDOW)`. Without the helper we
|
||||
/// fall back to SCREEN (best-effort, likely origin-less) then WINDOW-relative. Omitted if zero-size.
|
||||
async fn read_frame(
|
||||
conn: &zbus::Connection, dest: &str, path: &str, origin: Option<(i32, i32)>,
|
||||
) -> Option<Value> {
|
||||
let p = ComponentProxy::builder(conn)
|
||||
.destination(dest.to_owned()).ok()?.path(path.to_owned()).ok()?.build().await.ok()?;
|
||||
if let Some((ox, oy)) = origin {
|
||||
if let Ok((x, y, w, h)) = p.get_extents(1).await {
|
||||
if w > 0 && h > 0 {
|
||||
return Some(json!({"x": ox + x, "y": oy + y, "w": w, "h": h}));
|
||||
}
|
||||
}
|
||||
}
|
||||
for coord in [0u32, 1u32] {
|
||||
if let Ok((x, y, w, h)) = p.get_extents(coord).await {
|
||||
if w > 0 && h > 0 {
|
||||
@@ -329,12 +355,70 @@ async fn find_active_app(conn: &zbus::Connection) -> Result<ObjRef, String> {
|
||||
|
||||
// ---- op dispatch ----------------------------------------------------------------------------------
|
||||
|
||||
/// Per-connection state: the a11y-bus connection (shared) + the last dump's ref registry.
|
||||
/// Per-connection state: the a11y-bus connection + the session-bus connection (for the compositor
|
||||
/// geometry helper) + the last dump's ref registry.
|
||||
struct Session {
|
||||
conn: zbus::Connection,
|
||||
conn: zbus::Connection, // AT-SPI a11y bus
|
||||
session: zbus::Connection, // session bus (org.nucleic.Geometry lives here)
|
||||
refs: Vec<ObjRef>,
|
||||
}
|
||||
|
||||
impl Session {
|
||||
/// The active window's global origin + title from the compositor geometry helper, or `None` if the
|
||||
/// helper isn't present (extension not installed/enabled). The title lets `ax_dump` pick the
|
||||
/// matching AT-SPI app (so the dump + geometry agree even without keyboard focus).
|
||||
async fn focused_window(&self) -> Option<(i32, i32, String)> {
|
||||
let g = GeometryProxy::new(&self.session).await.ok()?;
|
||||
let (x, y, w, h, title) = g.get_focused_window_frame().await.ok()?;
|
||||
if w > 0 && h > 0 {
|
||||
Some((x, y, title))
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Just the origin (for the coordinate-fallback in `ax_action`).
|
||||
async fn focused_origin(&self) -> Option<(i32, i32)> {
|
||||
self.focused_window().await.map(|(x, y, _)| (x, y))
|
||||
}
|
||||
}
|
||||
|
||||
/// Find the application whose top-level window's title matches `title` (from the compositor helper), so
|
||||
/// the dump targets the same window the helper reports the geometry for — independent of AT-SPI's
|
||||
/// focus state. Returns the application's object ref.
|
||||
async fn find_app_by_window_title(conn: &zbus::Connection, title: &str) -> Option<ObjRef> {
|
||||
let root = accessible(conn, A11Y_ROOT_DEST, A11Y_ROOT_PATH).await.ok()?;
|
||||
let n = root.child_count().await.ok()?;
|
||||
for i in 0..n {
|
||||
let (adest, apath) = match root.get_child_at_index(i).await {
|
||||
Ok(v) => v,
|
||||
Err(_) => continue,
|
||||
};
|
||||
let apath = apath.as_str().to_owned();
|
||||
if adest.is_empty() {
|
||||
continue;
|
||||
}
|
||||
if let Ok(app) = accessible(conn, &adest, &apath).await {
|
||||
let wn = app.child_count().await.unwrap_or(0);
|
||||
for j in 0..wn {
|
||||
if let Ok((wd, wp)) = app.get_child_at_index(j).await {
|
||||
let wd = if wd.is_empty() { adest.clone() } else { wd };
|
||||
if let Ok(win) = accessible(conn, &wd, wp.as_str()).await {
|
||||
if let Ok(name) = win.name().await {
|
||||
if !name.is_empty()
|
||||
&& (name == title || title.contains(&name) || name.contains(title))
|
||||
{
|
||||
return Some((adest, apath));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn err(msg: impl Into<String>) -> Value {
|
||||
json!({"ok": false, "error": msg.into()})
|
||||
}
|
||||
@@ -356,11 +440,32 @@ impl Session {
|
||||
}
|
||||
|
||||
async fn op_dump(&mut self) -> Value {
|
||||
let (dest, path) = match find_active_app(&self.conn).await {
|
||||
Ok(a) => a,
|
||||
Err(why) => return err(format!("no accessible application to dump: {why}")),
|
||||
// Prefer the compositor helper's active window: it gives both the true global origin AND the
|
||||
// title, so we dump the exact window it measured — correct even with no AT-SPI focus. Fall back
|
||||
// to AT-SPI's own active-app heuristic (and no origin) when the helper isn't present.
|
||||
let (dest, path, origin) = match self.focused_window().await {
|
||||
Some((ox, oy, title))
|
||||
if !title.is_empty() =>
|
||||
{
|
||||
match find_app_by_window_title(&self.conn, &title).await {
|
||||
Some((d, p)) => (d, p, Some((ox, oy))),
|
||||
None => match find_active_app(&self.conn).await {
|
||||
Ok((d, p)) => (d, p, None),
|
||||
Err(why) => return err(format!("no accessible application to dump: {why}")),
|
||||
},
|
||||
}
|
||||
}
|
||||
_ => match find_active_app(&self.conn).await {
|
||||
Ok((d, p)) => (d, p, None),
|
||||
Err(why) => return err(format!("no accessible application to dump: {why}")),
|
||||
},
|
||||
};
|
||||
let mut st = DumpState {
|
||||
conn: self.conn.clone(),
|
||||
refs: Vec::new(),
|
||||
budget: NODE_BUDGET,
|
||||
origin,
|
||||
};
|
||||
let mut st = DumpState { conn: self.conn.clone(), refs: Vec::new(), budget: NODE_BUDGET };
|
||||
let tree = st.build_node(&dest, &path, 0).await;
|
||||
let app = accessible(&self.conn, &dest, &path).await.ok();
|
||||
let app_name = match app {
|
||||
@@ -392,24 +497,42 @@ impl Session {
|
||||
Err(e) => return e,
|
||||
};
|
||||
let want = req.get("action").and_then(|v| v.as_str()).filter(|s| !s.is_empty());
|
||||
let p = match ActionProxy::builder(&self.conn)
|
||||
.destination(dest).and_then(|b| b.path(path))
|
||||
// Gather this element's semantic actions (empty if it has no Action interface at all).
|
||||
let actions = match ActionProxy::builder(&self.conn)
|
||||
.destination(dest.clone())
|
||||
.and_then(|b| b.path(path.clone()))
|
||||
{
|
||||
Ok(b) => match b.build().await {
|
||||
Ok(p) => p,
|
||||
Err(_) => return err("this element has no Action interface — try ax_focus, or set_value"),
|
||||
Ok(p) => (p.get_actions().await.unwrap_or_default(), Some(p)),
|
||||
Err(_) => (Vec::new(), None),
|
||||
},
|
||||
Err(e) => return err(format!("bad ref: {e}")),
|
||||
};
|
||||
let actions = p.get_actions().await.unwrap_or_default();
|
||||
let (names, proxy) = actions;
|
||||
|
||||
// No semantic action available (and none was named) → coordinate fallback: hand the host the
|
||||
// element's true global frame so it can click the center via the surface pointer. This is the
|
||||
// rare path — Phase 0 measured ~100% semantic-action coverage on GTK.
|
||||
if names.is_empty() && want.is_none() {
|
||||
let origin = self.focused_origin().await;
|
||||
return match read_frame(&self.conn, &dest, &path, origin).await {
|
||||
Some(frame) => json!({
|
||||
"ok": true, "fallback": "click", "frame": frame,
|
||||
"note": "no semantic action; click the frame center"
|
||||
}),
|
||||
None => err("this element has no semantic action and no geometry to click"),
|
||||
};
|
||||
}
|
||||
|
||||
let proxy = proxy.unwrap(); // names non-empty ⇒ proxy present
|
||||
let index = match want {
|
||||
Some(name) => match actions.iter().position(|(n, _, _)| n.eq_ignore_ascii_case(name)) {
|
||||
Some(name) => match names.iter().position(|(n, _, _)| n.eq_ignore_ascii_case(name)) {
|
||||
Some(i) => i as i32,
|
||||
None => return err(format!("no action named '{name}' on this element")),
|
||||
},
|
||||
None => 0, // the primary/default action (macOS AXPress equivalent)
|
||||
};
|
||||
match p.do_action(index).await {
|
||||
match proxy.do_action(index).await {
|
||||
Ok(true) => json!({"ok": true}),
|
||||
Ok(false) => json!({"ok": true, "note": "the toolkit reported the action did not run; re-dump to verify"}),
|
||||
Err(e) => err(format!("performing the action failed: {e}")),
|
||||
@@ -464,10 +587,10 @@ impl Session {
|
||||
|
||||
// ---- connection + vsock server -------------------------------------------------------------------
|
||||
|
||||
async fn handle_connection(stream: VsockStream, a11y: zbus::Connection) {
|
||||
async fn handle_connection(stream: VsockStream, a11y: zbus::Connection, sbus: zbus::Connection) {
|
||||
let (rd, mut wr) = tokio::io::split(stream);
|
||||
let mut lines = BufReader::new(rd).lines();
|
||||
let mut session = Session { conn: a11y, refs: Vec::new() };
|
||||
let mut session = Session { conn: a11y, session: sbus, refs: Vec::new() };
|
||||
while let Ok(Some(line)) = lines.next_line().await {
|
||||
if line.trim().is_empty() {
|
||||
continue;
|
||||
@@ -485,12 +608,14 @@ async fn handle_connection(stream: VsockStream, a11y: zbus::Connection) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Connect to the session's AT-SPI a11y bus (org.a11y.Bus → GetAddress → the a11y bus).
|
||||
async fn connect_a11y() -> zbus::Result<zbus::Connection> {
|
||||
/// Connect to the session bus and, via it, the AT-SPI a11y bus (org.a11y.Bus → GetAddress → the a11y
|
||||
/// bus). Returns both: the session bus (for the org.nucleic.Geometry helper) and the a11y bus.
|
||||
async fn connect_buses() -> zbus::Result<(zbus::Connection, zbus::Connection)> {
|
||||
let session = zbus::Connection::session().await?;
|
||||
let bus = A11yBusProxy::new(&session).await?;
|
||||
let address = bus.get_address().await?;
|
||||
zbus::connection::Builder::address(address.as_str())?.build().await
|
||||
let a11y = zbus::connection::Builder::address(address.as_str())?.build().await?;
|
||||
Ok((session, a11y))
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
@@ -500,7 +625,7 @@ async fn main() {
|
||||
.and_then(|s| s.parse::<u32>().ok())
|
||||
.unwrap_or(DEFAULT_PORT);
|
||||
|
||||
let a11y = match connect_a11y().await {
|
||||
let (sbus, a11y) = match connect_buses().await {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
eprintln!("[nucleic-a11y-agent] FATAL: could not reach the AT-SPI a11y bus: {e}");
|
||||
@@ -508,9 +633,8 @@ async fn main() {
|
||||
std::process::exit(1);
|
||||
}
|
||||
};
|
||||
// Shared behind an Arc<Mutex<>> only to clone cheaply per connection; zbus::Connection is itself
|
||||
// cheaply clonable, so we just clone it.
|
||||
let a11y = Arc::new(Mutex::new(a11y));
|
||||
// zbus::Connection is cheaply clonable; the Mutex just guards the shared handles across accepts.
|
||||
let buses = Arc::new(Mutex::new((sbus, a11y)));
|
||||
|
||||
let listener = match VsockListener::bind(VsockAddr::new(VMADDR_CID_ANY, port)) {
|
||||
Ok(l) => l,
|
||||
@@ -524,8 +648,11 @@ async fn main() {
|
||||
loop {
|
||||
match listener.accept().await {
|
||||
Ok((stream, _addr)) => {
|
||||
let conn = a11y.lock().await.clone();
|
||||
tokio::spawn(handle_connection(stream, conn));
|
||||
let (sbus, a11y) = {
|
||||
let g = buses.lock().await;
|
||||
(g.0.clone(), g.1.clone())
|
||||
};
|
||||
tokio::spawn(handle_connection(stream, a11y, sbus));
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("[nucleic-a11y-agent] accept failed: {e}");
|
||||
|
||||
Reference in New Issue
Block a user