362 lines
24 KiB
HTML
362 lines
24 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<title>AX-based computer use for macOS VMs — Nucleic Support</title>
|
|
<meta name="description" content="Set up the optional in-guest Accessibility agent for Nucleic's macOS VMs: semantic control by UI element, framebuffer-independent screenshots, and the one-time SIP step." />
|
|
<meta name="theme-color" content="#faf7f3" />
|
|
<meta name="robots" content="index, follow" />
|
|
|
|
<!-- atom mark, reused as favicon -->
|
|
<link rel="icon" href="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><g fill='none' stroke-width='7'><ellipse cx='50' cy='50' rx='17' ry='40' transform='rotate(80 50 50)' stroke='%237AA7DD'/><ellipse cx='50' cy='50' rx='17' ry='40' transform='rotate(20 50 50)' stroke='%23DF619B'/><ellipse cx='50' cy='50' rx='17' ry='40' transform='rotate(140 50 50)' stroke='%23C5826F'/></g></svg>" />
|
|
|
|
<link rel="preload" href="../fonts/space-grotesk.woff2" as="font" type="font/woff2" crossorigin />
|
|
<link rel="preload" href="../fonts/jetbrains-mono.woff2" as="font" type="font/woff2" crossorigin />
|
|
<style>
|
|
@font-face{font-family:'Space Grotesk';font-style:normal;font-weight:400 700;font-display:swap;src:url(../fonts/space-grotesk.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}
|
|
@font-face{font-family:'JetBrains Mono';font-style:normal;font-weight:400 500;font-display:swap;src:url(../fonts/jetbrains-mono.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}
|
|
</style>
|
|
|
|
<style>
|
|
:root{
|
|
--ink:#181520;--ink-soft:#5b5560;--ink-faint:#8b8590;
|
|
--paper:#faf7f3;--panel:#fffdfb;--panel-2:#f3ede6;--line:#e6ded4;--line-2:#efe9e1;
|
|
--blue:#5e9bd8;--pink:#db5f97;--magenta:#c264ad;--rust:#c47e69;--green:#5fa775;--amber:#d7a23f;
|
|
--sans:"Space Grotesk", -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
|
|
--mono:"JetBrains Mono", ui-monospace, SFMono-Regular, "SF Mono", Menlo, monospace;
|
|
--maxw:1080px;--prose:760px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{scroll-behavior:smooth}
|
|
body{
|
|
margin:0;overflow-x:clip;font-family:var(--sans);color:var(--ink);background:var(--paper);
|
|
background-image:radial-gradient(var(--line) 1px, transparent 1px);
|
|
background-size:26px 26px;background-position:-13px -13px;
|
|
-webkit-font-smoothing:antialiased;line-height:1.5;
|
|
}
|
|
a{color:inherit}
|
|
.wrap{max-width:var(--maxw);margin:0 auto;padding:0 24px}
|
|
.mono{font-family:var(--mono)}
|
|
.kicker{font-family:var(--mono);font-size:11.5px;letter-spacing:.16em;text-transform:uppercase;color:var(--ink-faint)}
|
|
|
|
/* ---------- top bar ---------- */
|
|
header{position:sticky;top:0;z-index:20;backdrop-filter:saturate(1.1) blur(8px);background:color-mix(in srgb, var(--paper) 82%, transparent);border-bottom:1px solid var(--line-2)}
|
|
.bar{display:flex;align-items:center;justify-content:space-between;height:62px}
|
|
.brand{display:flex;align-items:center;gap:11px;font-weight:600;letter-spacing:-.01em;font-size:17px;text-decoration:none;color:var(--ink)}
|
|
.brand .mk{width:26px;height:26px;display:block}
|
|
.nav{display:flex;align-items:center;gap:22px;font-size:14px;color:var(--ink-soft)}
|
|
.nav a{text-decoration:none}
|
|
.nav .lk:hover{color:var(--ink)}
|
|
.btn{font-family:var(--sans);font-size:14px;font-weight:500;text-decoration:none;cursor:pointer;border:0;padding:9px 16px;border-radius:10px;color:#fff;background:var(--ink);transition:transform .12s ease, opacity .12s ease;white-space:nowrap;-webkit-tap-highlight-color:transparent}
|
|
.btn:link,.btn:visited,.btn:hover,.btn:focus,.btn:active{color:#fff}
|
|
.btn:hover{opacity:.9;transform:translateY(-1px)}
|
|
@media (max-width:600px){ .nav .lk{display:none} }
|
|
|
|
/* ---------- doc header ---------- */
|
|
.crumbs{max-width:var(--prose);margin:0 auto;padding:26px 0 0;font-family:var(--mono);font-size:12px;color:var(--ink-faint);letter-spacing:.02em}
|
|
.crumbs a{text-decoration:none;color:var(--ink-soft)}
|
|
.crumbs a:hover{color:var(--ink)}
|
|
.crumbs .sep{color:var(--line);margin:0 8px}
|
|
.doc-hero{padding:16px 0 22px}
|
|
.doc-hero .wrap{max-width:var(--prose)}
|
|
.doc-hero h1{font-size:clamp(30px,4.6vw,44px);line-height:1.05;letter-spacing:-.03em;font-weight:600;margin:12px 0 0}
|
|
.doc-hero h1 .em{background:linear-gradient(100deg,var(--rust),var(--pink) 45%,var(--magenta) 70%,var(--blue));-webkit-background-clip:text;background-clip:text;color:transparent}
|
|
.doc-hero .lede{max-width:var(--prose);margin:16px 0 0;font-size:17px;color:var(--ink-soft);line-height:1.62}
|
|
|
|
/* ---------- on this page ---------- */
|
|
.toc{max-width:var(--prose);margin:22px auto 0;border:1px solid var(--line);border-radius:14px;background:var(--panel);padding:16px 20px}
|
|
.toc h2{margin:0 0 10px;font-size:11px;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-faint);font-weight:600;font-family:var(--mono)}
|
|
.toc ol{margin:0;padding-left:0;list-style:none;counter-reset:toc;display:grid;gap:7px}
|
|
.toc li{counter-increment:toc;position:relative;padding-left:28px;font-size:14.5px}
|
|
.toc li::before{content:counter(toc,decimal-leading-zero);position:absolute;left:0;top:1px;font-family:var(--mono);font-size:11px;color:var(--ink-faint)}
|
|
.toc a{text-decoration:none;color:var(--ink-soft)}
|
|
.toc a:hover{color:var(--ink)}
|
|
|
|
/* ---------- prose ---------- */
|
|
main{padding:10px 0 30px}
|
|
.prose{max-width:var(--prose);margin:0 auto}
|
|
.prose section{padding:28px 0;border-top:1px solid var(--line-2)}
|
|
.prose h2{font-size:22px;font-weight:600;letter-spacing:-.02em;margin:0 0 6px;scroll-margin-top:80px}
|
|
.prose h2 .n{font-family:var(--mono);font-size:13px;color:var(--ink-faint);margin-right:10px;font-weight:500}
|
|
.prose h3{font-size:16px;font-weight:600;letter-spacing:-.01em;margin:22px 0 6px;color:var(--ink)}
|
|
.prose p{margin:12px 0;color:var(--ink-soft);font-size:15.5px;line-height:1.65}
|
|
.prose ul{margin:12px 0;padding-left:22px;color:var(--ink-soft);font-size:15.5px;line-height:1.65}
|
|
.prose li{margin:7px 0}
|
|
.prose strong{color:var(--ink);font-weight:600}
|
|
.prose a{color:var(--ink);text-decoration:underline;text-decoration-color:var(--line);text-underline-offset:3px}
|
|
.prose a:hover{text-decoration-color:var(--magenta)}
|
|
.prose code{font-family:var(--mono);font-size:.86em;background:var(--panel-2);border:1px solid var(--line-2);border-radius:5px;padding:1px 6px}
|
|
.prose .lede{font-size:17px;color:var(--ink);line-height:1.6}
|
|
.kbd{font-family:var(--mono);font-size:.82em;background:var(--panel);border:1px solid var(--line);border-bottom-width:2px;border-radius:6px;padding:1px 7px;color:var(--ink);white-space:nowrap}
|
|
.ui{font-weight:600;color:var(--ink)}
|
|
|
|
/* two-path table */
|
|
.ftable{width:100%;border-collapse:collapse;margin:16px 0;font-size:14px}
|
|
.ftable th,.ftable td{text-align:left;padding:10px 12px;border-bottom:1px solid var(--line-2);vertical-align:top}
|
|
.ftable th{font-size:11px;letter-spacing:.08em;text-transform:uppercase;color:var(--ink-faint);font-weight:600}
|
|
.ftable td:first-child{font-weight:600;color:var(--ink);white-space:nowrap}
|
|
.ftable td:last-child{color:var(--ink-soft)}
|
|
|
|
/* callouts */
|
|
.callout{border-left:3px solid var(--line);border-radius:0 10px 10px 0;padding:14px 16px;margin:18px 0;font-size:14.5px;color:var(--ink-soft);line-height:1.6}
|
|
.callout strong{color:var(--ink)}
|
|
.callout code{font-family:var(--mono);font-size:.86em;background:var(--panel-2);border:1px solid var(--line-2);border-radius:5px;padding:1px 6px}
|
|
.callout.note{border-color:color-mix(in srgb,var(--blue) 60%,var(--line));background:color-mix(in srgb,var(--blue) 6%,var(--panel))}
|
|
.callout.tip{border-color:color-mix(in srgb,var(--green) 60%,var(--line));background:color-mix(in srgb,var(--green) 6%,var(--panel))}
|
|
.callout.warn{border-color:color-mix(in srgb,var(--amber) 70%,var(--line));background:color-mix(in srgb,var(--amber) 8%,var(--panel))}
|
|
.callout .lbl{display:block;font-size:11px;letter-spacing:.1em;text-transform:uppercase;font-family:var(--mono);margin-bottom:4px}
|
|
.callout.note .lbl{color:var(--blue)}
|
|
.callout.tip .lbl{color:var(--green)}
|
|
.callout.warn .lbl{color:color-mix(in srgb,var(--amber) 80%,var(--ink))}
|
|
|
|
/* numbered steps */
|
|
.steps{counter-reset:step;list-style:none;padding:0;margin:16px 0}
|
|
.steps>li{counter-increment:step;position:relative;padding:0 0 20px 46px;margin:0}
|
|
.steps>li::before{content:counter(step);position:absolute;left:0;top:-2px;width:30px;height:30px;border-radius:9px;background:var(--ink);color:#fff;font-family:var(--mono);font-size:13px;display:flex;align-items:center;justify-content:center}
|
|
.steps>li:not(:last-child)::after{content:"";position:absolute;left:14px;top:32px;bottom:6px;width:2px;background:var(--line)}
|
|
.steps>li h3{margin:4px 0 4px;font-size:16px}
|
|
.steps>li p{margin:6px 0}
|
|
|
|
/* code block */
|
|
pre{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:14px 16px;overflow-x:auto;margin:14px 0}
|
|
pre code{font-family:var(--mono);font-size:13px;color:var(--ink);line-height:1.7;background:none;border:0;padding:0}
|
|
pre code .c{color:var(--ink-faint)}
|
|
|
|
/* next / related */
|
|
.related{max-width:var(--prose);margin:0 auto;padding:26px 0 0;border-top:1px solid var(--line-2)}
|
|
.related h2{font-size:12px;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-faint);font-weight:600;font-family:var(--mono);margin:0 0 12px}
|
|
.related a{display:flex;align-items:center;gap:9px;text-decoration:none;color:var(--ink);border:1px solid var(--line);border-radius:12px;padding:13px 16px;margin:8px 0;font-weight:500;font-size:15px;transition:border-color .14s ease}
|
|
.related a:hover{border-color:color-mix(in srgb,var(--magenta) 45%,var(--line))}
|
|
.related a .arw{margin-left:auto;color:var(--ink-faint)}
|
|
|
|
footer{border-top:1px solid var(--line-2);padding:30px 0 50px;margin-top:24px}
|
|
.foot{display:flex;align-items:center;justify-content:space-between;gap:18px;flex-wrap:wrap;color:var(--ink-faint);font-size:13px}
|
|
.foot .mono{font-size:12px}
|
|
.foot-links{display:flex;gap:18px;flex-wrap:wrap}
|
|
.foot-links a{text-decoration:none;color:var(--ink-soft)}
|
|
.foot-links a:hover{color:var(--ink)}
|
|
.dotsep{color:var(--line)}
|
|
</style>
|
|
</head>
|
|
<body>
|
|
|
|
<!-- ===== top bar ===== -->
|
|
<header>
|
|
<div class="wrap bar">
|
|
<a class="brand" href="/">
|
|
<svg class="mk" viewBox="0 0 100 100" aria-hidden="true">
|
|
<g fill="none" stroke-width="7">
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(80 50 50)" stroke="#7AA7DD"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(20 50 50)" stroke="#DF619B"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(140 50 50)" stroke="#C5826F"/>
|
|
</g>
|
|
</svg>
|
|
Nucleic
|
|
</a>
|
|
<nav class="nav">
|
|
<a class="lk" href="/support.html">Support</a>
|
|
<a class="btn" href="/">Home</a>
|
|
</nav>
|
|
</div>
|
|
</header>
|
|
|
|
<div class="wrap">
|
|
<nav class="crumbs" aria-label="Breadcrumb">
|
|
<a href="/support.html">Support</a><span class="sep">/</span>
|
|
<a href="/support/virtual-machines.html">Virtual Machines</a><span class="sep">/</span>
|
|
AX-based computer use
|
|
</nav>
|
|
</div>
|
|
|
|
<!-- ===== title ===== -->
|
|
<section class="doc-hero">
|
|
<div class="wrap">
|
|
<div class="kicker">Virtual Machines · Advanced</div>
|
|
<h1>Setting up <span class="em">AX-based computer use</span></h1>
|
|
<p class="lede">By default, an agent drives a macOS VM host-side — screenshots and clicks with no setup. The optional in-guest agent adds <strong>semantic</strong> control: act on a button by identity instead of by pixel, and keep working when a guest's screenshots come back blank. This guide sets it up.</p>
|
|
</div>
|
|
</section>
|
|
|
|
<div class="wrap">
|
|
<nav class="toc" aria-label="On this page">
|
|
<h2>On this page</h2>
|
|
<ol>
|
|
<li><a href="#two-paths">Two ways to drive a VM</a></li>
|
|
<li><a href="#when">When you actually need this</a></li>
|
|
<li><a href="#what">What gets installed</a></li>
|
|
<li><a href="#before">Before you start</a></li>
|
|
<li><a href="#enable">Turn it on and build the base</a></li>
|
|
<li><a href="#sip">The one-time SIP step</a></li>
|
|
<li><a href="#verify">Verify it's working</a></li>
|
|
<li><a href="#model">How an agent uses it</a></li>
|
|
</ol>
|
|
</nav>
|
|
</div>
|
|
|
|
<main>
|
|
<div class="wrap prose">
|
|
|
|
<section id="two-paths">
|
|
<h2><span class="n">01</span>Two ways to drive a VM</h2>
|
|
<p class="lede">Nucleic can let an agent see and operate a macOS VM's screen through the <code>mac_vm_computer</code> tool. There are two paths, and they stack — the semantic path is an add-on to the default, not a replacement.</p>
|
|
<table class="ftable">
|
|
<thead><tr><th>Path</th><th>What it is</th></tr></thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>Host-side<br>(default)</td>
|
|
<td>Screenshots are captured from the VM's virtual display and input is injected into its virtual keyboard and mouse — all <strong>host-side</strong>, in the Virtualization framework. No software runs inside the guest, no permissions are granted, and SIP stays on. This is <a href="/support/macos-vm-computer-use.html">computer use for macOS VMs</a>, and it needs no setup.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Semantic AX<br>(this guide)</td>
|
|
<td>An in-guest agent (<code>NucleicVMAgent</code>) exposes the guest's <strong>accessibility tree</strong> — every control's role, title, value, and frame — so an agent can act on an element <em>by identity</em>. It also captures screenshots from inside the guest, which keeps working when the host-side framebuffer is blank. This is the only path that needs the guest to grant permissions.</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
<div class="callout note"><span class="lbl">Good to know</span>Turning on the semantic agent doesn't take anything away. The agent still advertises the ordinary screenshot-and-click actions; it just <em>adds</em> the semantic ones on top.</div>
|
|
</section>
|
|
|
|
<section id="when">
|
|
<h2><span class="n">02</span>When you actually need this</h2>
|
|
<p>The default host-side path is enough for most GUI work. Reach for the semantic agent when one of these applies:</p>
|
|
<ul>
|
|
<li><strong>You want robust, semantic control.</strong> Instead of guessing pixel coordinates and clicking blind, the agent reads the accessibility tree and acts on a control by its identity — the way professional macOS UI automation works. It's far less fragile than pixel targeting.</li>
|
|
<li><strong>Your guest is macOS 26 and app windows show up blank in screenshots.</strong> On a macOS 26 (Tahoe) guest, a Virtualization-framework framebuffer bug can blank rendered app windows in pixel screenshots. The accessibility tree is produced from the app's semantic model, independent of what's on screen — so semantic control keeps working when the pixels don't.</li>
|
|
<li><strong>You're debugging a Mac or iOS app's UI</strong> and want the full accessibility tree — roles, values, focus state — the way a developer expects to inspect it.</li>
|
|
</ul>
|
|
<div class="callout tip"><span class="lbl">Rule of thumb</span>If your guest is macOS 27 and screenshots look fine, the default path is simpler and needs no setup. Enable the semantic agent when you specifically want by-identity control or you're on a macOS 26 guest.</div>
|
|
</section>
|
|
|
|
<section id="what">
|
|
<h2><span class="n">03</span>What gets installed</h2>
|
|
<p>Enabling the semantic agent adds one component to the golden base image: a small, signed app bundle called <strong><code>NucleicVMAgent</code></strong>. Inside each VM it runs as a per-user login agent in the guest's desktop session and listens on a private <code>vsock</code> channel for commands from your Mac. It drives the guest through three native macOS APIs:</p>
|
|
<ul>
|
|
<li><strong>Accessibility API</strong> — reads the accessibility tree and performs actions on elements (press a button, set a field's value, move focus).</li>
|
|
<li><strong>ScreenCaptureKit</strong> — captures screenshots from inside the guest.</li>
|
|
<li><strong>CGEvent</strong> — synthesizes raw mouse, keyboard, and scroll input.</li>
|
|
</ul>
|
|
<p>Because those APIs are privacy-sensitive, the guest account must grant the agent three permissions — <strong>Accessibility</strong>, <strong>Screen Recording</strong>, and input (post-event) rights. Nucleic writes these grants for you during provisioning; the only wrinkle is that on some guests it needs System Integrity Protection (SIP) turned off once to do so, which is the manual step below.</p>
|
|
</section>
|
|
|
|
<section id="before">
|
|
<h2><span class="n">04</span>Before you start</h2>
|
|
<p>Make sure the basics are in place first:</p>
|
|
<ul>
|
|
<li><strong>Apple silicon Mac running macOS 26 or later.</strong> Running macOS guests requires Apple silicon.</li>
|
|
<li><strong>The macOS VM service is on.</strong> In <span class="ui">Settings ▸ Virtual Machines</span>, turn on <span class="ui">Enable macOS VM service</span>. See <a href="/support/virtual-machines.html">the overview</a> for requirements and disk space.</li>
|
|
<li><strong>Computer use is on.</strong> In the same section, turn on <span class="ui">Enable computer use (screen + mouse/keyboard)</span>. The semantic-agent toggle only appears once computer use is enabled.</li>
|
|
</ul>
|
|
<div class="callout warn"><span class="lbl">Heads up</span>The semantic agent is baked into the base image at build time. If you already built a base <em>without</em> it, you'll rebuild the base once after enabling the toggle — that's covered below.</div>
|
|
</section>
|
|
|
|
<section id="enable">
|
|
<h2><span class="n">05</span>Turn it on and build the base</h2>
|
|
<ol class="steps">
|
|
<li>
|
|
<h3>Enable the semantic agent</h3>
|
|
<p>In <span class="ui">Settings ▸ Virtual Machines</span>, with computer use already on, turn on <span class="ui">Semantic AX agent (advanced — needs SIP disabled once)</span>.</p>
|
|
</li>
|
|
<li>
|
|
<h3>Build (or rebuild) the base image</h3>
|
|
<p>Click <span class="ui">Build base image</span> — or <span class="ui">Rebuild / re-provision base image</span> if you already have one. This installs macOS into the golden base, provisions the dev toolchain, and — because the agent is now enabled — installs <code>NucleicVMAgent</code> and attempts to write its permission grants. See <a href="/support/macos-vm-base-image.html">Building the base image</a> for the full walkthrough.</p>
|
|
</li>
|
|
<li>
|
|
<h3>Check readiness</h3>
|
|
<p>When the build finishes, the base-image status tells you where you stand. If it reads <span class="ui">Ready — builds + computer use + semantic AX.</span>, you're done — skip to <a href="#verify">Verify</a>. If it still shows a one-time SIP step, continue below.</p>
|
|
</li>
|
|
</ol>
|
|
<div class="callout note"><span class="lbl">macOS 27 host + guest</span>On a macOS 27 host with a macOS 27 guest, provisioning is fully automated — the permission grants land during the one-click build and there's <strong>no SIP step</strong>. The manual step below is only needed on older guests.</div>
|
|
</section>
|
|
|
|
<section id="sip">
|
|
<h2><span class="n">06</span>The one-time SIP step</h2>
|
|
<p>On a macOS 26 guest, writing the agent's permission grants requires System Integrity Protection to be disabled in the guest once. Nucleic can't script this — turning SIP off can only be done from recoveryOS — so it's a short manual pass. Nucleic shows these exact steps in-app when they're needed:</p>
|
|
<ol class="steps">
|
|
<li>
|
|
<h3>Boot the base into Recovery</h3>
|
|
<p>In the base-image status, click <span class="ui">Boot base in Recovery</span>. Nucleic boots the golden base into recoveryOS in a window.</p>
|
|
</li>
|
|
<li>
|
|
<h3>Disable SIP from the guest's Terminal</h3>
|
|
<p>In the VM, open <span class="ui">Utilities ▸ Terminal</span> and run:</p>
|
|
<pre><code>csrutil disable</code></pre>
|
|
</li>
|
|
<li>
|
|
<h3>Reboot, then shut the VM down</h3>
|
|
<p>Reboot the guest so the change takes effect, then shut the VM down and close the window.</p>
|
|
</li>
|
|
<li>
|
|
<h3>Build the base again</h3>
|
|
<p>Back in <span class="ui">Settings ▸ Virtual Machines</span>, click <span class="ui">Build base image</span> once more. Provisioning now detects that SIP is off and writes the agent's permission grants. That's the last step — the base is agent-ready.</p>
|
|
</li>
|
|
</ol>
|
|
<div class="callout tip"><span class="lbl">This affects the VM only</span>SIP is disabled <em>inside the disposable guest</em>, not on your Mac. Your host's security is untouched — the default host-side computer-use path never needs SIP at all.</div>
|
|
</section>
|
|
|
|
<section id="verify">
|
|
<h2><span class="n">07</span>Verify it's working</h2>
|
|
<p>The base-image status line is the quickest signal:</p>
|
|
<ul>
|
|
<li><span class="ui">Ready — builds + computer use + semantic AX.</span> — the agent is installed and fully granted. You're set.</li>
|
|
<li><span class="ui">Ready — builds + computer use.</span> — builds and host-side computer use work, but the semantic agent isn't ready yet. Re-check that the SIP step completed, then rebuild.</li>
|
|
</ul>
|
|
<p>If an agent gets blank screenshots or clicks that don't register once it's running, the usual causes are:</p>
|
|
<ul>
|
|
<li><strong>SIP still on in the guest.</strong> Confirm with <code>csrutil status</code> in the guest — it should say disabled.</li>
|
|
<li><strong>Grants didn't land.</strong> Rebuild the base after the SIP step so provisioning can write them.</li>
|
|
<li><strong>The guest isn't logged in.</strong> The agent runs in the desktop session, so the guest account must be logged in (not sitting at the login window). The base is configured to auto-log-in, so this normally just works.</li>
|
|
</ul>
|
|
</section>
|
|
|
|
<section id="model">
|
|
<h2><span class="n">08</span>How an agent uses it</h2>
|
|
<p>Once the semantic agent is present, the <code>mac_vm_computer</code> tool advertises both pixel actions (screenshot, click, type, key, scroll) and semantic ones. The typical loop is <strong>dump the tree, then act by identity</strong>:</p>
|
|
<pre><code><span class="c"># read the frontmost app's accessibility tree</span>
|
|
ax_dump → { role:"AXButton", title:"Build", ref:"e17", … }
|
|
|
|
<span class="c"># act on that element directly — no pixel guessing</span>
|
|
ax_press ref=e17</code></pre>
|
|
<p>Guidance the agent follows:</p>
|
|
<ul>
|
|
<li><strong>On a macOS 26 guest, prefer <code>ax_dump</code> + act-by-ref</strong> — pixel screenshots may be blank, and the tree keeps working.</li>
|
|
<li><strong>Fall back to screenshot + pixel click</strong> when a control has no accessibility action or isn't in the tree.</li>
|
|
<li><strong>On a macOS 27 guest, either path works</strong> — semantic is more robust, screenshot-first is simpler.</li>
|
|
</ul>
|
|
<p>You don't drive any of this by hand — this is what the agent does through the tool. Enabling the semantic path just makes the reliable, by-identity actions available to it.</p>
|
|
</section>
|
|
|
|
</div>
|
|
|
|
<div class="wrap related">
|
|
<h2>Related guides</h2>
|
|
<a href="/support/macos-vm-computer-use.html">Computer use for macOS VMs<span class="arw" aria-hidden="true">→</span></a>
|
|
<a href="/support/macos-vm-base-image.html">Building the base image<span class="arw" aria-hidden="true">→</span></a>
|
|
<a href="/support/virtual-machines.html">macOS virtual machines: overview<span class="arw" aria-hidden="true">→</span></a>
|
|
</div>
|
|
</main>
|
|
|
|
<footer>
|
|
<div class="wrap foot">
|
|
<div class="brand" style="font-size:15px">
|
|
<svg class="mk" viewBox="0 0 100 100" aria-hidden="true">
|
|
<g fill="none" stroke-width="7">
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(80 50 50)" stroke="#7AA7DD"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(20 50 50)" stroke="#DF619B"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(140 50 50)" stroke="#C5826F"/>
|
|
</g>
|
|
</svg>
|
|
Nucleic
|
|
</div>
|
|
<div class="foot-links">
|
|
<a href="/">Home</a>
|
|
<a href="/support.html">Support</a>
|
|
<a href="/privacy.html">Privacy</a>
|
|
<a href="/terms.html">Terms</a>
|
|
</div>
|
|
<div class="mono">© 2026 Andrew Blakeslee Moore · Made with Nucleic</div>
|
|
</div>
|
|
</footer>
|
|
|
|
</body>
|
|
</html>
|