Nucleic
Virtual Machines · Advanced

Setting up AX-based computer use

By default, an agent drives a macOS VM from the host — screenshots and clicks, no setup required. The optional in-guest agent adds semantic control: act on a button by identity, not by pixel, and inspect an app's full accessibility tree. On macOS 27, setup is fully automated. Here's how to turn it on.

01Two ways to drive a VM

In Nucleic, an agent sees and operates a macOS VM's screen through the mac_vm_computer tool. There are two paths, and they stack — the semantic path adds to the default. It doesn't replace it.

PathWhat it is
Host-side
(default)		 Screenshots come from the VM's virtual display, and input is injected into its virtual keyboard and mouse — all host-side, in the Virtualization framework. Nothing runs inside the guest, and the guest grants no permissions. This is computer use for macOS VMs, and it works with no setup.
Semantic AX
(this guide)		 An in-guest agent (NucleicVMAgent) exposes the guest's accessibility tree — every control's role, title, value, and frame — so an agent can act on an element by identity. It also captures screenshots from inside the guest. This is the only path where the guest grants permissions, and on macOS 27, provisioning handles that for you.
Good to knowTurning on the semantic agent takes nothing away. The agent still advertises the ordinary screenshot-and-click actions — it simply adds the semantic ones on top.

02When to use it

The default host-side path handles most GUI work. Turn to the semantic agent when either of these applies:

  • You want robust, semantic control. Instead of guessing pixel coordinates and clicking blind, the agent reads the accessibility tree and acts on a control by its identity — the way professional macOS UI automation works. It's far less fragile than pixel targeting.
  • You're debugging a Mac or iOS app's UI and want the full accessibility tree — roles, values, focus state — the way a developer expects to inspect it.
Rule of thumbThe default path is simpler and needs no setup, so start there for most GUI work. Enable the semantic agent when you specifically want robust, by-identity control — or you're inspecting an app's accessibility tree.

03What gets installed

Enabling the semantic agent adds one component to the golden base image: a small, signed app bundle called NucleicVMAgent. In each VM, it runs as a per-user login agent in the guest's desktop session, listening on a private vsock channel for commands from your Mac. It drives the guest through three native macOS APIs:

  • Accessibility API — reads the accessibility tree and performs actions on elements (press a button, set a field's value, move focus).
  • ScreenCaptureKit — captures screenshots from inside the guest.
  • CGEvent — synthesizes raw mouse, keyboard, and scroll input.

Because those APIs are privacy-sensitive, the guest account must grant the agent three permissions — Accessibility, Screen Recording, and input (post-event) rights. On macOS 27, Nucleic writes these grants automatically during the one-click base build. There's nothing to do by hand.

04Before you start

First, make sure the basics are in place:

  • An Apple silicon Mac running macOS 27 or later. Running macOS guests requires Apple silicon.
  • The macOS engine is on. In Settings ▸ Virtual Machines, under Virtualization engine, turn on macOS. See the overview for requirements and disk space.
  • Kernel-level use is on. Under Computer use, turn on Kernel-level use. The Semantic understanding toggle only appears once kernel-level use is enabled.
Heads upThe semantic agent is baked into the base image at build time. If you already built a base without it, you'll need to rebuild once after turning on the toggle — that's covered below.

05Turn it on and build the base

  1. Enable the semantic agent

    In Settings ▸ Virtual Machines, with kernel-level use already on, turn on Semantic understanding.

  2. Build (or rebuild) the base image

    Click Build base image — or Rebuild / re-provision base image if you already have one. This installs macOS into the golden base, provisions the dev toolchain, and — now that the agent is enabled — installs NucleicVMAgent and attempts to write its permission grants. See Building the base image for the full walkthrough.

  3. Check readiness

    When the build finishes, the base-image status reads Ready — builds + computer use + semantic AX. That's it — the permission grants landed automatically during the build. Head to Verify.

Fully automated on macOS 27On macOS 27, provisioning is fully automated — the agent's permission grants land during the one-click base build. Nothing to configure by hand.

06Verify it's working

The base-image status line tells you at a glance:

  • Ready — builds + computer use + semantic AX. — the agent is installed and fully granted. You're set.
  • Ready — builds + computer use. — builds and host-side computer use work, but the semantic agent isn't ready yet. Rebuild the base with the semantic agent toggle on.

If screenshots come back blank or clicks don't register once it's running, check two things:

  • The grants didn't land. Rebuild the base with the semantic agent enabled so provisioning can write them.
  • The guest isn't logged in. The agent runs in the desktop session, so the guest account must be logged in — not sitting at the login window. The base is configured to log in automatically, so this normally just works.

07How an agent uses it

With the semantic agent in place, the mac_vm_computer tool advertises both pixel actions (screenshot, click, type, key, scroll) and semantic ones. The typical loop: dump the tree, then act by identity.

# read the frontmost app's accessibility tree
ax_dump  → { role:"AXButton", title:"Build", ref:"e17", … }

# act on that element directly — no pixel guessing
ax_press  ref=e17

The agent follows this guidance:

  • Prefer ax_dump + act-by-ref for robust control — acting on an element by identity is far less fragile than targeting pixels.
  • Fall back to screenshot + pixel click when a control has no accessibility action or isn't in the tree.
  • Either path works — semantic is more robust, screenshot-first is simpler.

You never drive any of this by hand — the agent does it through the tool. Enabling the semantic path simply puts the reliable, by-identity actions at its disposal.

Related guides

Computer use for macOS VMs Building the base image macOS virtual machines: overview