288 lines
18 KiB
HTML
288 lines
18 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<title>Computer use: the two kinds — Nucleic Support</title>
|
|
<meta name="description" content="The two kinds of computer use in Nucleic virtual machines — kernel-level use and semantic understanding — what each does, when to use them, and how they compare." />
|
|
<meta name="theme-color" content="#faf7f3" />
|
|
<meta name="robots" content="index, follow" />
|
|
|
|
<!-- atom mark, reused as favicon -->
|
|
<link rel="icon" href="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><g fill='none' stroke-width='7'><ellipse cx='50' cy='50' rx='17' ry='40' transform='rotate(80 50 50)' stroke='%237AA7DD'/><ellipse cx='50' cy='50' rx='17' ry='40' transform='rotate(20 50 50)' stroke='%23DF619B'/><ellipse cx='50' cy='50' rx='17' ry='40' transform='rotate(140 50 50)' stroke='%23C5826F'/></g></svg>" />
|
|
|
|
<link rel="preload" href="../fonts/space-grotesk.woff2" as="font" type="font/woff2" crossorigin />
|
|
<link rel="preload" href="../fonts/jetbrains-mono.woff2" as="font" type="font/woff2" crossorigin />
|
|
<style>
|
|
@font-face{font-family:'Space Grotesk';font-style:normal;font-weight:400 700;font-display:swap;src:url(../fonts/space-grotesk.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}
|
|
@font-face{font-family:'JetBrains Mono';font-style:normal;font-weight:400 500;font-display:swap;src:url(../fonts/jetbrains-mono.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}
|
|
</style>
|
|
|
|
<style>
|
|
:root{
|
|
--ink:#181520;--ink-soft:#5b5560;--ink-faint:#8b8590;
|
|
--paper:#faf7f3;--panel:#fffdfb;--panel-2:#f3ede6;--line:#e6ded4;--line-2:#efe9e1;
|
|
--blue:#5e9bd8;--pink:#db5f97;--magenta:#c264ad;--rust:#c47e69;--green:#5fa775;--amber:#d7a23f;
|
|
--sans:"Space Grotesk", -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
|
|
--mono:"JetBrains Mono", ui-monospace, SFMono-Regular, "SF Mono", Menlo, monospace;
|
|
--maxw:1080px;--prose:760px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{scroll-behavior:smooth}
|
|
body{
|
|
margin:0;overflow-x:clip;font-family:var(--sans);color:var(--ink);background:var(--paper);
|
|
background-image:radial-gradient(var(--line) 1px, transparent 1px);
|
|
background-size:26px 26px;background-position:-13px -13px;
|
|
-webkit-font-smoothing:antialiased;line-height:1.5;
|
|
}
|
|
a{color:inherit}
|
|
.wrap{max-width:var(--maxw);margin:0 auto;padding:0 24px}
|
|
.mono{font-family:var(--mono)}
|
|
.kicker{font-family:var(--mono);font-size:11.5px;letter-spacing:.16em;text-transform:uppercase;color:var(--ink-faint)}
|
|
|
|
/* ---------- top bar ---------- */
|
|
header{position:sticky;top:0;z-index:20;backdrop-filter:saturate(1.1) blur(8px);background:color-mix(in srgb, var(--paper) 82%, transparent);border-bottom:1px solid var(--line-2)}
|
|
.bar{display:flex;align-items:center;justify-content:space-between;height:62px}
|
|
.brand{display:flex;align-items:center;gap:11px;font-weight:600;letter-spacing:-.01em;font-size:17px;text-decoration:none;color:var(--ink)}
|
|
.brand .mk{width:26px;height:26px;display:block}
|
|
.nav{display:flex;align-items:center;gap:22px;font-size:14px;color:var(--ink-soft)}
|
|
.nav a{text-decoration:none}
|
|
.nav .lk:hover{color:var(--ink)}
|
|
.btn{font-family:var(--sans);font-size:14px;font-weight:500;text-decoration:none;cursor:pointer;border:0;padding:9px 16px;border-radius:10px;color:#fff;background:var(--ink);transition:transform .12s ease, opacity .12s ease;white-space:nowrap;-webkit-tap-highlight-color:transparent}
|
|
.btn:link,.btn:visited,.btn:hover,.btn:focus,.btn:active{color:#fff}
|
|
.btn:hover{opacity:.9;transform:translateY(-1px)}
|
|
@media (max-width:600px){ .nav .lk{display:none} }
|
|
|
|
/* ---------- doc header ---------- */
|
|
.crumbs{max-width:var(--prose);margin:0 auto;padding:26px 0 0;font-family:var(--mono);font-size:12px;color:var(--ink-faint);letter-spacing:.02em}
|
|
.crumbs a{text-decoration:none;color:var(--ink-soft)}
|
|
.crumbs a:hover{color:var(--ink)}
|
|
.crumbs .sep{color:var(--line);margin:0 8px}
|
|
.doc-hero{padding:16px 0 22px}
|
|
.doc-hero .wrap{max-width:var(--prose)}
|
|
.doc-hero h1{font-size:clamp(30px,4.6vw,44px);line-height:1.05;letter-spacing:-.03em;font-weight:600;margin:12px 0 0}
|
|
.doc-hero h1 .em{background:linear-gradient(100deg,var(--rust),var(--pink) 45%,var(--magenta) 70%,var(--blue));-webkit-background-clip:text;background-clip:text;color:transparent}
|
|
.doc-hero .lede{max-width:var(--prose);margin:16px 0 0;font-size:17px;color:var(--ink-soft);line-height:1.62}
|
|
|
|
/* ---------- on this page ---------- */
|
|
.toc{max-width:var(--prose);margin:22px auto 0;border:1px solid var(--line);border-radius:14px;background:var(--panel);padding:16px 20px}
|
|
.toc h2{margin:0 0 10px;font-size:11px;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-faint);font-weight:600;font-family:var(--mono)}
|
|
.toc ol{margin:0;padding-left:0;list-style:none;counter-reset:toc;display:grid;gap:7px}
|
|
.toc li{counter-increment:toc;position:relative;padding-left:28px;font-size:14.5px}
|
|
.toc li::before{content:counter(toc,decimal-leading-zero);position:absolute;left:0;top:1px;font-family:var(--mono);font-size:11px;color:var(--ink-faint)}
|
|
.toc a{text-decoration:none;color:var(--ink-soft)}
|
|
.toc a:hover{color:var(--ink)}
|
|
|
|
/* ---------- prose ---------- */
|
|
main{padding:10px 0 30px}
|
|
.prose{max-width:var(--prose);margin:0 auto}
|
|
.prose section{padding:28px 0;border-top:1px solid var(--line-2)}
|
|
.prose h2{font-size:22px;font-weight:600;letter-spacing:-.02em;margin:0 0 6px;scroll-margin-top:80px}
|
|
.prose h2 .n{font-family:var(--mono);font-size:13px;color:var(--ink-faint);margin-right:10px;font-weight:500}
|
|
.prose h3{font-size:16px;font-weight:600;letter-spacing:-.01em;margin:22px 0 6px;color:var(--ink)}
|
|
.prose p{margin:12px 0;color:var(--ink-soft);font-size:15.5px;line-height:1.65}
|
|
.prose ul{margin:12px 0;padding-left:22px;color:var(--ink-soft);font-size:15.5px;line-height:1.65}
|
|
.prose li{margin:7px 0}
|
|
.prose strong{color:var(--ink);font-weight:600}
|
|
.prose a{color:var(--ink);text-decoration:underline;text-decoration-color:var(--line);text-underline-offset:3px}
|
|
.prose a:hover{text-decoration-color:var(--magenta)}
|
|
.prose code{font-family:var(--mono);font-size:.86em;background:var(--panel-2);border:1px solid var(--line-2);border-radius:5px;padding:1px 6px}
|
|
.prose .lede{font-size:17px;color:var(--ink);line-height:1.6}
|
|
.kbd{font-family:var(--mono);font-size:.82em;background:var(--panel);border:1px solid var(--line);border-bottom-width:2px;border-radius:6px;padding:1px 7px;color:var(--ink);white-space:nowrap}
|
|
.ui{font-weight:600;color:var(--ink)}
|
|
|
|
/* two-path table */
|
|
.ftable{width:100%;border-collapse:collapse;margin:16px 0;font-size:14px}
|
|
.ftable th,.ftable td{text-align:left;padding:10px 12px;border-bottom:1px solid var(--line-2);vertical-align:top}
|
|
.ftable th{font-size:11px;letter-spacing:.08em;text-transform:uppercase;color:var(--ink-faint);font-weight:600}
|
|
.ftable td:first-child{font-weight:600;color:var(--ink);white-space:nowrap}
|
|
.ftable td:last-child{color:var(--ink-soft)}
|
|
|
|
/* callouts */
|
|
.callout{border-left:3px solid var(--line);border-radius:0 10px 10px 0;padding:14px 16px;margin:18px 0;font-size:14.5px;color:var(--ink-soft);line-height:1.6}
|
|
.callout strong{color:var(--ink)}
|
|
.callout code{font-family:var(--mono);font-size:.86em;background:var(--panel-2);border:1px solid var(--line-2);border-radius:5px;padding:1px 6px}
|
|
.callout.note{border-color:color-mix(in srgb,var(--blue) 60%,var(--line));background:color-mix(in srgb,var(--blue) 6%,var(--panel))}
|
|
.callout.tip{border-color:color-mix(in srgb,var(--green) 60%,var(--line));background:color-mix(in srgb,var(--green) 6%,var(--panel))}
|
|
.callout.warn{border-color:color-mix(in srgb,var(--amber) 70%,var(--line));background:color-mix(in srgb,var(--amber) 8%,var(--panel))}
|
|
.callout .lbl{display:block;font-size:11px;letter-spacing:.1em;text-transform:uppercase;font-family:var(--mono);margin-bottom:4px}
|
|
.callout.note .lbl{color:var(--blue)}
|
|
.callout.tip .lbl{color:var(--green)}
|
|
.callout.warn .lbl{color:color-mix(in srgb,var(--amber) 80%,var(--ink))}
|
|
|
|
/* numbered steps */
|
|
.steps{counter-reset:step;list-style:none;padding:0;margin:16px 0}
|
|
.steps>li{counter-increment:step;position:relative;padding:0 0 20px 46px;margin:0}
|
|
.steps>li::before{content:counter(step);position:absolute;left:0;top:-2px;width:30px;height:30px;border-radius:9px;background:var(--ink);color:#fff;font-family:var(--mono);font-size:13px;display:flex;align-items:center;justify-content:center}
|
|
.steps>li:not(:last-child)::after{content:"";position:absolute;left:14px;top:32px;bottom:6px;width:2px;background:var(--line)}
|
|
.steps>li h3{margin:4px 0 4px;font-size:16px}
|
|
.steps>li p{margin:6px 0}
|
|
|
|
/* code block */
|
|
pre{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:14px 16px;overflow-x:auto;margin:14px 0}
|
|
pre code{font-family:var(--mono);font-size:13px;color:var(--ink);line-height:1.7;background:none;border:0;padding:0}
|
|
pre code .c{color:var(--ink-faint)}
|
|
|
|
/* next / related */
|
|
.related{max-width:var(--prose);margin:0 auto;padding:26px 0 0;border-top:1px solid var(--line-2)}
|
|
.related h2{font-size:12px;letter-spacing:.12em;text-transform:uppercase;color:var(--ink-faint);font-weight:600;font-family:var(--mono);margin:0 0 12px}
|
|
.related a{display:flex;align-items:center;gap:9px;text-decoration:none;color:var(--ink);border:1px solid var(--line);border-radius:12px;padding:13px 16px;margin:8px 0;font-weight:500;font-size:15px;transition:border-color .14s ease}
|
|
.related a:hover{border-color:color-mix(in srgb,var(--magenta) 45%,var(--line))}
|
|
.related a .arw{margin-left:auto;color:var(--ink-faint)}
|
|
|
|
footer{border-top:1px solid var(--line-2);padding:30px 0 50px;margin-top:24px}
|
|
.foot{display:flex;align-items:center;justify-content:space-between;gap:18px;flex-wrap:wrap;color:var(--ink-faint);font-size:13px}
|
|
.foot .mono{font-size:12px}
|
|
.foot-links{display:flex;gap:18px;flex-wrap:wrap}
|
|
.foot-links a{text-decoration:none;color:var(--ink-soft)}
|
|
.foot-links a:hover{color:var(--ink)}
|
|
.dotsep{color:var(--line)}
|
|
</style>
|
|
</head>
|
|
<body>
|
|
|
|
<!-- ===== top bar ===== -->
|
|
<header>
|
|
<div class="wrap bar">
|
|
<a class="brand" href="/">
|
|
<svg class="mk" viewBox="0 0 100 100" aria-hidden="true">
|
|
<g fill="none" stroke-width="7">
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(80 50 50)" stroke="#7AA7DD"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(20 50 50)" stroke="#DF619B"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(140 50 50)" stroke="#C5826F"/>
|
|
</g>
|
|
</svg>
|
|
Nucleic
|
|
</a>
|
|
<nav class="nav">
|
|
<a class="btn" href="/">Home</a>
|
|
</nav>
|
|
</div>
|
|
</header>
|
|
|
|
<div class="wrap">
|
|
<nav class="crumbs" aria-label="Breadcrumb">
|
|
<a href="/support.html">Support</a><span class="sep">/</span>
|
|
<a href="/support/virtual-machines.html">Virtual Machines</a><span class="sep">/</span>
|
|
Computer use
|
|
</nav>
|
|
</div>
|
|
|
|
<!-- ===== title ===== -->
|
|
<section class="doc-hero">
|
|
<div class="wrap">
|
|
<div class="kicker">Virtual Machines</div>
|
|
<h1>Computer use: the <span class="em">two kinds</span></h1>
|
|
<p class="lede">Beyond running commands, an agent can see and operate a virtual machine's screen. Nucleic offers this in two forms — <strong>kernel-level use</strong> and <strong>semantic understanding</strong> — that stack: the second builds on the first. This guide explains what each does and when to reach for it.</p>
|
|
</div>
|
|
</section>
|
|
|
|
<div class="wrap">
|
|
<nav class="toc" aria-label="On this page">
|
|
<h2>On this page</h2>
|
|
<ol>
|
|
<li><a href="#what">What computer use is</a></li>
|
|
<li><a href="#kernel">Kernel-level use</a></li>
|
|
<li><a href="#semantic">Semantic understanding</a></li>
|
|
<li><a href="#compare">Which to use</a></li>
|
|
<li><a href="#enable">Turning them on</a></li>
|
|
</ol>
|
|
</nav>
|
|
</div>
|
|
|
|
<main>
|
|
<div class="wrap prose">
|
|
|
|
<section id="what">
|
|
<h2><span class="n">01</span>What computer use is</h2>
|
|
<p class="lede">Computer use lets an agent <strong>see</strong> and drive a virtual machine's screen — clicking, typing, and scrolling through its interface — rather than only running commands.</p>
|
|
<p>It's the visual counterpart to running commands in the VM. Where commands are for scripting and builds, computer use is for anything with a screen. Reach for it when the agent needs to:</p>
|
|
<ul>
|
|
<li>Look at or click through an app it just built, to confirm the interface actually behaves.</li>
|
|
<li>Drive a tool's window — the parts you can't reach from the command line.</li>
|
|
<li>Check how something renders — a layout, a web page, a window — by taking an actual screenshot.</li>
|
|
</ul>
|
|
<p>Nucleic offers computer use in two forms. They <strong>stack</strong>: semantic understanding is an add-on to kernel-level use, not a replacement for it. Both are available for macOS and Linux virtual machines alike.</p>
|
|
</section>
|
|
|
|
<section id="kernel">
|
|
<h2><span class="n">02</span>Kernel-level use</h2>
|
|
<p>Kernel-level use is the foundation. The agent works the machine <strong>from the outside</strong>, the way a person at the keyboard would: it takes a screenshot of the screen, then moves the pointer, clicks, scrolls, and types.</p>
|
|
<p>Through it, an agent can:</p>
|
|
<ul>
|
|
<li><strong>Take a screenshot</strong> of the screen.</li>
|
|
<li><strong>Move and click the pointer</strong>, including press-and-drag.</li>
|
|
<li><strong>Scroll</strong> and <strong>type text</strong>, with modifier keys for shortcuts.</li>
|
|
<li><strong>Launch apps</strong> in the machine.</li>
|
|
</ul>
|
|
<p>These compose into a simple <strong>screenshot → decide → act</strong> loop: capture the screen, choose an action, perform it, then capture again to see what changed. It works quietly in the background — no window opens on your Mac, and it never captures your own mouse or keyboard. There's nothing to set up: it's ready as soon as a base image is built.</p>
|
|
</section>
|
|
|
|
<section id="semantic">
|
|
<h2><span class="n">03</span>Semantic understanding</h2>
|
|
<p>Semantic understanding layers on top of kernel-level use to give the agent a <strong>structured view of the interface</strong> — the on-screen controls as named elements rather than just pixels. With it, the agent can:</p>
|
|
<ul>
|
|
<li><strong>Act on a control by identity</strong> — press a specific button by its name — instead of guessing and clicking coordinates.</li>
|
|
<li><strong>Read the interface's structure</strong> — the controls present, their values, and what's focused — when it needs to understand a screen precisely.</li>
|
|
</ul>
|
|
<p>This makes GUI automation far more reliable, since the agent targets controls by what they are rather than where they happen to sit. The trade-off is that reading and reasoning over the interface's structure <strong>increases token usage</strong>, so it's worth turning on when you specifically want robust, by-identity control.</p>
|
|
</section>
|
|
|
|
<section id="compare">
|
|
<h2><span class="n">04</span>Which to use</h2>
|
|
<table class="ftable">
|
|
<thead><tr><th>Kind</th><th>What you get</th></tr></thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>Kernel-level use</td>
|
|
<td><strong>See and drive the screen</strong> with screenshots, clicks, and typing. Works in the background with no setup. The right default for most GUI work.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Semantic understanding</td>
|
|
<td>Adds a <strong>structured, by-identity</strong> view of the interface for more robust automation — at the cost of higher token usage. An add-on to kernel-level use, not a replacement.</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
<div class="callout tip"><span class="lbl">Rule of thumb</span>Start with kernel-level use. Turn on semantic understanding when you want the agent to act on controls by identity or to read an interface precisely.</div>
|
|
</section>
|
|
|
|
<section id="enable">
|
|
<h2><span class="n">05</span>Turning them on</h2>
|
|
<p>In <span class="ui">Settings ▸ Virtual Machines</span>, under <span class="ui">Computer use</span>, turn on <span class="ui">Kernel-level use</span>. Once it's on, <span class="ui">Semantic understanding</span> appears beneath it — turn that on too if you want the structured view. Both apply to whichever virtual machines you've enabled.</p>
|
|
<p>Computer use runs against a built virtual machine, so you'll need a base image first — see <a href="/support/virtual-machines.html">Virtual machines: overview</a>.</p>
|
|
</section>
|
|
|
|
</div>
|
|
|
|
<div class="wrap related">
|
|
<h2>Related guides</h2>
|
|
<a href="/support/virtual-machines.html">Virtual machines: overview<span class="arw" aria-hidden="true">→</span></a>
|
|
<a href="/support/macos-vm-base-image.html">Building the base image<span class="arw" aria-hidden="true">→</span></a>
|
|
</div>
|
|
</main>
|
|
|
|
<footer>
|
|
<div class="wrap foot">
|
|
<div class="brand" style="font-size:15px">
|
|
<svg class="mk" viewBox="0 0 100 100" aria-hidden="true">
|
|
<g fill="none" stroke-width="7">
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(80 50 50)" stroke="#7AA7DD"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(20 50 50)" stroke="#DF619B"/>
|
|
<ellipse cx="50" cy="50" rx="17" ry="40" transform="rotate(140 50 50)" stroke="#C5826F"/>
|
|
</g>
|
|
</svg>
|
|
Nucleic
|
|
</div>
|
|
<div class="foot-links">
|
|
<a href="/">Home</a>
|
|
<a href="/support.html">Support</a>
|
|
<a href="/privacy.html">Privacy</a>
|
|
<a href="/terms.html">Terms</a>
|
|
</div>
|
|
<div class="mono">© 2026 Andrew Blakeslee Moore · Made with Nucleic</div>
|
|
</div>
|
|
</footer>
|
|
|
|
</body>
|
|
</html>
|