# Computer use for macOS VMs — Nucleic Support
Let a Nucleic agent see and drive a macOS VM's screen — clicks, keys, and scrolling — through the mac_vm_computer tool. The default path runs host-side with no in-guest setup.

Virtual Machines

# Computer use for macOS VMs

Beyond running shell commands, an agent can see and operate a macOS VM's screen — clicking, typing, and scrolling through its GUI. The default path needs no setup at all.

## 01What computer use is

Computer use lets an agent see and drive a macOS VM's GUI through the `mac_vm_computer` tool — think of the VM as a Mac dev simulator the agent can look at and operate.

It's the visual counterpart to `mac_vm_exec`, which runs headless shell commands inside the VM. Where `mac_vm_exec` is for scripting and builds, computer use is for anything with a screen. Reach for it when the agent needs to:

- Look at or click through an app it just built, to confirm the UI actually behaves.
- Drive Xcode or the Simulator UI — the parts you can't reach from the command line.
- Check how something renders — a layout, a web page, a window — by taking an actual screenshot of the VM's display.

## 02Why it needs no setup

In the default path, screen capture and input both happen host-side, inside Apple's Virtualization framework. Screenshots come from the VM's virtual display, and clicks and keystrokes are injected into its virtual keyboard and mouse. The agent never reaches into the guest to do any of this — it works the display and input devices from the outside.

That design is why there's nothing to install and nothing to configure inside the VM:

- Nothing runs inside the guest. No helper app, no login agent — capture and input live in the framework on your Mac.
- The guest grants no permissions. No Accessibility, no Screen Recording — because no in-guest software is asking for them.
- The guest's security posture stays untouched. No in-guest changes are made to capture the screen or inject input.

Because it depends only on the virtual display and input devices, the default path works on any installed base image — there's no special build variant to produce first.

## 03Turning it on

In Settings ▸ Virtual Machines, turn on Enable computer use (screen + mouse/keyboard). Sessions then expose the `mac_vm_computer` tool to the agent.

The first call to the tool boots that session's VM, which takes a minute or two; later calls reuse the already-running VM, so they're fast. You don't launch anything by hand — the agent invokes the tool and the session's VM spins up on demand.

Two things need to be in place first:

- The macOS VM service is on. See macOS virtual machines: overview for requirements and how to enable it.
- A base image is built. Computer use runs against an installed VM, so you need a built base first — see Building the base image.

## 04What the agent can do

Through the `mac_vm_computer` tool, an agent can:

- Take a screenshot of the VM's screen.
- Move and click the mouse, including press-and-drag.
- Scroll with a real scroll wheel.
- Type text, with modifier keys for shortcuts.
- Launch apps in the guest.

These compose into a simple screenshot → decide → act loop: the agent captures the screen, chooses an action, performs it, then captures the screen again to see what changed — the same way a person watches a display and reacts to it.

## 05When to add the semantic agent

The default pixel path is enough for most GUI work. There's also an optional in-guest semantic AX agent you can layer on top. Add it when you want to:

- Act on a UI element by identity — press a specific button by its role and title — rather than guessing and clicking pixel coordinates.
- Inspect an app's accessibility tree — roles, values, and focus state — when debugging a Mac or iOS app's UI.

The two paths differ mainly in setup versus capability:

Path | Trade-off
--- | ---
Host-side pixel (default) | Zero setup. Screenshot-and-click through the virtual display and input devices — no in-guest software and no permissions.
Semantic AX (optional) | Adds by-identity control and lets an agent inspect the guest's accessibility tree — at the cost of an in-guest agent baked into the base image, provisioned automatically on macOS 27.

If you want by-identity controlThe semantic path is an add-on to the default, not a replacement — the ordinary screenshot-and-click actions still work. See Setting up AX-based computer use for what gets installed.

## Related guides

Setting up AX-based computer use→ macOS virtual machines: overview→ Building the base image→