Nucleic
Virtual Machines · Advanced

Setting up AX-based computer use

By default, an agent drives a macOS VM host-side — screenshots and clicks with no setup. The optional in-guest agent adds semantic control: act on a button by identity instead of by pixel, and inspect an app's full accessibility tree. On macOS 27 it's fully automated. This guide sets it up.

01Two ways to drive a VM

Nucleic can let an agent see and operate a macOS VM's screen through the mac_vm_computer tool. There are two paths, and they stack — the semantic path is an add-on to the default, not a replacement.

PathWhat it is
Host-side
(default)		 Screenshots are captured from the VM's virtual display and input is injected into its virtual keyboard and mouse — all host-side, in the Virtualization framework. No software runs inside the guest and no permissions are granted. This is computer use for macOS VMs, and it needs no setup.
Semantic AX
(this guide)		 An in-guest agent (NucleicVMAgent) exposes the guest's accessibility tree — every control's role, title, value, and frame — so an agent can act on an element by identity. It also captures screenshots from inside the guest. This is the only path that needs the guest to grant permissions, which provisioning handles automatically on macOS 27.
Good to knowTurning on the semantic agent doesn't take anything away. The agent still advertises the ordinary screenshot-and-click actions; it just adds the semantic ones on top.

02When you actually need this

The default host-side path is enough for most GUI work. Reach for the semantic agent when one of these applies:

  • You want robust, semantic control. Instead of guessing pixel coordinates and clicking blind, the agent reads the accessibility tree and acts on a control by its identity — the way professional macOS UI automation works. It's far less fragile than pixel targeting.
  • You're debugging a Mac or iOS app's UI and want the full accessibility tree — roles, values, focus state — the way a developer expects to inspect it.
Rule of thumbThe default path is simpler and needs no setup, so it's the right default for most GUI work. Enable the semantic agent when you specifically want robust, by-identity control or you're inspecting an app's accessibility tree.

03What gets installed

Enabling the semantic agent adds one component to the golden base image: a small, signed app bundle called NucleicVMAgent. Inside each VM it runs as a per-user login agent in the guest's desktop session and listens on a private vsock channel for commands from your Mac. It drives the guest through three native macOS APIs:

  • Accessibility API — reads the accessibility tree and performs actions on elements (press a button, set a field's value, move focus).
  • ScreenCaptureKit — captures screenshots from inside the guest.
  • CGEvent — synthesizes raw mouse, keyboard, and scroll input.

Because those APIs are privacy-sensitive, the guest account must grant the agent three permissions — Accessibility, Screen Recording, and input (post-event) rights. On macOS 27, Nucleic writes these grants for you automatically during the one-click base build — there's nothing to do by hand.

04Before you start

Make sure the basics are in place first:

  • Apple silicon Mac running macOS 27 or later. Running macOS guests requires Apple silicon.
  • The macOS VM service is on. In Settings ▸ Virtual Machines, turn on Enable macOS VM service. See the overview for requirements and disk space.
  • Computer use is on. In the same section, turn on Enable computer use (screen + mouse/keyboard). The semantic-agent toggle only appears once computer use is enabled.
Heads upThe semantic agent is baked into the base image at build time. If you already built a base without it, you'll rebuild the base once after enabling the toggle — that's covered below.

05Turn it on and build the base

  1. Enable the semantic agent

    In Settings ▸ Virtual Machines, with computer use already on, turn on Semantic AX agent (advanced).

  2. Build (or rebuild) the base image

    Click Build base image — or Rebuild / re-provision base image if you already have one. This installs macOS into the golden base, provisions the dev toolchain, and — because the agent is now enabled — installs NucleicVMAgent and attempts to write its permission grants. See Building the base image for the full walkthrough.

  3. Check readiness

    When the build finishes, the base-image status reads Ready — builds + computer use + semantic AX. and you're done — the permission grants landed automatically during the build. Head to Verify.

Fully automated on macOS 27On macOS 27, provisioning is fully automated — the agent's permission grants land during the one-click base build. There's nothing to configure by hand.

06Verify it's working

The base-image status line is the quickest signal:

  • Ready — builds + computer use + semantic AX. — the agent is installed and fully granted. You're set.
  • Ready — builds + computer use. — builds and host-side computer use work, but the semantic agent isn't ready yet. Rebuild the base with the semantic agent toggle on.

If an agent gets blank screenshots or clicks that don't register once it's running, the usual causes are:

  • Grants didn't land. Rebuild the base with the semantic agent enabled so provisioning can write them.
  • The guest isn't logged in. The agent runs in the desktop session, so the guest account must be logged in (not sitting at the login window). The base is configured to auto-log-in, so this normally just works.

07How an agent uses it

Once the semantic agent is present, the mac_vm_computer tool advertises both pixel actions (screenshot, click, type, key, scroll) and semantic ones. The typical loop is dump the tree, then act by identity:

# read the frontmost app's accessibility tree
ax_dump  → { role:"AXButton", title:"Build", ref:"e17", … }

# act on that element directly — no pixel guessing
ax_press  ref=e17

Guidance the agent follows:

  • Prefer ax_dump + act-by-ref for robust control — acting on an element by identity is far less fragile than targeting pixels.
  • Fall back to screenshot + pixel click when a control has no accessibility action or isn't in the tree.
  • Either path works — semantic is more robust, screenshot-first is simpler.

You don't drive any of this by hand — this is what the agent does through the tool. Enabling the semantic path just makes the reliable, by-identity actions available to it.

Related guides

Computer use for macOS VMs Building the base image macOS virtual machines: overview