Files
nucleic/docs/MACOS_VM_PROVISIONING_REPAIR_PLAN.md
T
2026-07-25 20:54:52 -07:00

1059 lines
73 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# macOS VM base-image provisioning repair
**Status:** the three-pass sequence is wired, but the mandatory MDM pass does not complete on a live
guest — see "Enrollment-pass repair" below, which is the active workstream.
**Opened:** 2026-07-23
## Objective
Make the macOS 27 golden-base build genuinely unattended: no unrecognized GUI permission sheets,
no blind keystrokes into whichever dialog happens to have focus, and no base marked ready until a
production-shaped clone can boot, mount its workspace, and execute through the in-guest agent.
The existing lightweight MDM is the policy plane for permissions macOS allows device management to
grant. Host-side display capture and virtual HID remain the default computer-use path, so guest
Screen Recording and Accessibility are not prerequisites for an ordinary usable base.
## Findings at investigation start
The implementation checkpoints below record which of these findings have since been resolved.
### 1. MDM is implemented but disconnected
`MacVMEngine+MDM.swift` contains Mode A enrollment and Mode B profile-push helpers, while
`NucleicMDM` contains a CA, enrollment profile, HTTPS server, command queue, and tests. The
production base build never calls those helpers. It runs the shell provisioner and then answers the
Network Volumes dialog with a separate OCR/click pass.
The `MacVMBaseStatus.mdmEnrolled` and `profilesInstalled` fields therefore describe planned state,
not a readiness invariant. Recipe v7 can be stamped while both remain false.
### 2. Provisioning creates policy prompts before policy exists
The bootstrap enters through Apple's VirtioFS automount, then remounts the same directory at a
custom path and executes the full provisioner there. The custom mount is treated as a network
volume, so the build touches a protected resource before any Network Volumes policy is installed.
The later consent pass repairs only `NucleicVMAgent` access. It is exact-English OCR for `Allow`,
cannot establish which dialog owns that button, and is fragile when more than one sheet is visible.
### 3. GUI fallback handling is not a state machine
When Command Line Tools are absent from the Software Update catalog, the guest launches
`xcode-select --install` and writes a sentinel. While the sentinel exists, the host sends Return
every three seconds. Return may accept the CLT sheet, an unrelated privacy sheet, or any other
focused control. Provisioning must recognize the expected UI state or stop; it must never treat all
dialogs as equivalent.
### 4. The MDM integration is still spike-grade
- UAMDM approval opens Device Management and sends one Return; Install, authentication, and
verification are placeholders.
- The built-in FDA and notification profiles are not copied into the shipped app.
- The CA, MDM configuration, device identity, and installed profile identifiers are not persisted
per base.
- Enrollment orchestration uses registry-based `run(name:)`, but maintenance boots hold a direct
`MacVMInstance`.
- A drained command queue may contain errored commands, and enrollment returning false is not
currently fatal.
- CA trust, guest-to-host TLS reachability, initial no-APNs command drain, real `mdmclient`
signatures, and headless UAMDM approval still need a live macOS 27 validation pass.
### 5. macOS 27 narrows the policy scope
Classic MDM profiles remain appropriate for Full Disk Access, Network Volumes, and targeted
notification settings. Classic PPPC Accessibility grants are removed in macOS 27 in favor of the
declarative `com.apple.configuration.app-settings` privacy model. Screen Recording cannot be
silently granted.
This does not block the base: pixel computer use is host-side and needs no guest TCC. Semantic AX
remains optional until the lightweight MDM supports Declarative Device Management.
## Implementation plan
### P0 — deterministic bootstrap
- [x] Package the built-in MDM policy profiles in `Resources/macvm`.
- [x] Add Network Volumes to the VM-agent PPPC policy.
- [x] Resolve packaged policy assets through one tested engine helper.
- [x] Stop blindly confirming the graphical CLT installer.
- [x] Keep the unattended path headless; if CLT cannot be installed headlessly, fail with an
actionable diagnostic rather than wait behind an unknown sheet.
- [x] Do not execute the full provisioner from a custom network-volume mount before policy exists.
- [x] Record screenshots, recognized text, frontmost UI state, and TCC attribution when an unexpected
modal blocks progress.
### P1 — production Mode A enrollment
- [x] Split the base build into:
1. declarative account creation and minimal exec-agent install through the automount;
2. MDM enrollment and required policy installation;
3. full toolchain provisioning after policy is active.
- [x] Refactor enrollment to operate on a direct `MacVMInstance`.
- [x] Persist the MDM CA, configuration, device identity, and profile identifiers beside the base
metadata with credential-appropriate permissions.
- [x] Replace the UAMDM placeholder with a bounded, state-aware Device Management flow that enters
the generated guest password only in the expected authentication sheet.
- [x] Treat enrollment false, command errors, missing profile identifiers, or failed functional
probes as fatal.
- [x] Make re-provisioning update policy without attempting a second enrollment.
- [x] Require `mdmEnrolled`, required profiles, agent readiness, and a production-shaped workspace
mount before stamping the provisioning recipe.
- [x] Remove `MacVMEngine+ConsentGrant.swift`'s Network Volumes click path entirely. Base publication
performs a bounded validation-only custom-mount touch after MDM policy installation.
### P2 — live macOS 27 validation
- [ ] Validate local-CA trust without a SecurityAgent prompt.
- [ ] Validate guest-to-host gateway/TLS reachability and check whether the listener introduces host
Local Network or incoming-connection prompts.
- [ ] Validate enrollment's initial command drain without an MDM APNs push certificate.
- [ ] Capture real `mdmclient` request signatures, then enable strict signature verification.
- [ ] Verify reboot, clone, reprovision, and app-update behavior.
- [ ] Test a non-English guest and reject any automation dependent on English-only button matching.
### P3 — optional semantic AX policy
- [ ] Implement Declarative Device Management enablement and declaration endpoints.
- [ ] Deliver macOS 27 app-settings privacy defaults for Accessibility/PostEvent only when semantic AX
is enabled.
- [ ] Keep Screen Recording and semantic AX out of ordinary base readiness.
## Release gate
A base is releasable only when a clean macOS 27 build proves:
- no unrecognized GUI sheets from restore through shutdown;
- required MDM enrollment and exact profile identifiers survive reboot;
- the first production-shaped workspace mount succeeds without a prompt;
- VM-agent FDA works without a Data Access Blocked notification;
- Tips/BTM noise is suppressed without disabling arbitrary app notifications;
- re-provisioning performs no duplicate enrollment;
- a fresh clone boots, mounts, executes, and shuts down unattended.
## Implementation checkpoint — bounded UAMDM approval
- The single blind Return has been replaced with an OCR-driven state machine. It acts only on the
exact Nucleic enrollment row, an exact Install control in Nucleic/Device Management context, or a
SecurityAgent sheet containing both expected system-modification phrases, the expected account,
an exact Password label, and an exact submit control.
- Credentials are submitted at most once. Repeated, foreign, incomplete, or non-English screens
receive no speculative input and eventually fail the bounded operation.
- Failure writes a private `Diagnostics/MDM` bundle beside the base: the last JPEG, recognized text,
`lsappinfo front` state, and the recent `com.apple.TCC` log tail. Password contents are never
included.
- Re-provisioning now branches on the guest's actual enrollment status: enrolled bases use the
direct-instance policy push/verification path and never reopen the enrollment profile. An enrolled
guest without `mdm-state.json` fails as orphaned before any replacement CA can be created.
- The state labels and field geometry still require the P2 macOS 27 live-validation pass. The
production path is enabled fail-closed: an unrecognized screen receives no input and blocks recipe
publication with a diagnostic rather than falling back to a permissive action.
## Implementation checkpoint — persisted Mode A state
- `mdm-state.json` now retains a versioned CA, configuration, device certificate/private key,
PKCS#12 password, and the profile identifiers verified in the guest. It is mode `0600`, remains in
the golden-base directory, and is deliberately excluded by the explicit clone copier.
- Mode A accepts a direct maintenance `MacVMInstance` and uses the existing bounded vsock/nash exec
channel instead of requiring the base to appear in the ordinary session registry.
- A drained MDM queue is no longer sufficient: every UUID from the current policy batch must report
`Acknowledged`, enrollment must report User Approved, and the guest's XML profile inventory must
contain the exact top-level identifiers before they are persisted.
- The production provisioning call site is enabled unconditionally in recipe v8. Its remaining
uncertainty is confined to the P2 live macOS 27 checks below; failures stop publication and retain
diagnostics rather than falling back to speculative input or optimistic status.
## Implementation checkpoint — production Mode A activation
- Recipe v8 runs three ordered writable-base boots: (1) declarative account creation plus only nash,
durable sudo, auto-login, and `NucleicVMAgent`; (2) mandatory MDM enrollment/policy verification;
(3) the complete toolchain, apps, packages, and optional SIP-off semantic-AX work. No protected
custom mount or heavyweight provisioning occurs before policy exists.
- The Mode A boot attaches only the `nucleic-provision` automount (`recreateHostPath: false`), so it
can stage enrollment artifacts without touching the protected custom-mount path before MDM policy.
- The listener bind address is discovered from the running guest's default route. The response must
be exactly one ordinary, non-loopback IPv4 address; whitespace-separated values, shell fragments,
unspecified, loopback, multicast, malformed, and IPv6 values fail closed.
- The pass uses the direct `MacVMInstance` to enroll a clean base or update an already-enrolled base,
persists only verified state, and then shuts the writable base down cleanly.
- Before queuing policy, the host derives the designated requirement from the exact staged agent and
rewrites both PPPC service entries. Release Developer ID and ad-hoc development builds therefore
authorize the binary actually installed; packaging now fails if the mandatory agent cannot build.
- Base publication no longer invokes the interactive Network Volumes Allow-click fallback. Its final
custom-mount touch is validation-only: if the MDM PPPC payload is acknowledged but non-functional,
the prompt remains unanswered, the probe fails boundedly, and the recipe stamp is withheld.
- The recipe cannot be stamped until MDM reports User Approved, every built-in policy command is
acknowledged, the guest inventory contains both exact profile identifiers, metadata records the
account/agent/toolchain/MDM invariants, and the production-shaped workspace/agent probe passes.
## Implementation checkpoint — first live provisioning retry
- The first live retry exposed an unexplained `Shutdown NOW` immediately after the visible bootstrap.
That message is the intended phase-boundary shutdown; the actual failure occurred earlier and was
hidden because the guest log lived only in a temporary staging directory.
- Minimal-agent launch readiness now uses the final `launchctl print` state. A concurrent automatic
`/Library/LaunchAgents` load can make an explicit `launchctl bootstrap` return nonzero even though
the service is healthy; that benign race no longer rejects the base.
- The new preflight had also promoted `sysadminctl -autologin set` from the old provisioner's
best-effort refresh into a fatal command. Declarative macOS 27 first boot already requests
auto-login, and `sysadminctl` may return nonzero when reapplying existing state. It is best-effort
again; the following boot's bounded agent probe remains the authoritative readiness gate.
- The captured live error showed the next synchronous blocker: the packaged app could not derive the
staged agent's designated requirement by spawning and scraping `/usr/bin/codesign`, so Mode A
aborted before its VM boot. Requirement extraction now uses Security.framework directly
(`SecStaticCodeCreateWithPath``SecCodeCopyDesignatedRequirement`
`SecRequirementCopyString`), eliminating the subprocess and stderr-format dependency.
- Both minimal and full passes label their shutdown outcome, include a bounded guest-log tail in the
surfaced error, and copy `STATUS` plus `provision.log` with private permissions into
`Diagnostics/Provisioning` before deleting the staging directory. No password or staged binary is
retained.
- A second macOS 27 live run is still required to establish the next release-gate result. The focused
macOS Swift test invocation was requested but not approved; materialized bootstrap scripts pass
`bash -n` in the Linux build environment.
## Verification notes
- The two existing policy files and generated minimal-agent LaunchAgent parse as plists.
- `scripts/provision-macos-guest.sh`, `scripts/build-macos-base.sh`, `scripts/package-app.sh`, and the
materialized minimal-agent bootstrap pass `bash -n`.
- Focused unit coverage now pins the three-pass boundary, PPPC requirement rewriting, gateway parser,
UAMDM classifier, persisted-status/recipe gate, clone admission, and production-shaped mount probe.
- The Swift suites still require an approved macOS toolchain run. The MDM protocol's synthetic TLS
drain does not substitute for the P2 real macOS guest/`mdmclient` release gate.
---
# Enrollment-pass repair
**Opened:** 2026-07-23 (second investigation)
**Symptom:** OS setup completes and the base is left account-provisioned with a reachable agent, but
the toolchain pass never starts. Phase 2 (`mdmPolicyPass`) throws, and its `catch` in
`provisionAndFinalize` is fail-closed, so phase 3 is never reached.
## Findings
Grouped by whether they are unambiguous from the source alone or depend on live macOS behavior that
has not yet been reproduced here.
### Blockers in the enrollment pass
1. **`mdm-enroll.sh` runs `open` as root.** `MacVMEngine+MDM.swift` invokes the whole script with
`sudo /bin/bash`, and `MDMServerController.enrollScript()` ends with `open "$PROFILE"` under
`set -euo pipefail`. Either `open` returns non-zero and `set -e` aborts before
`MDM_ENROLL_STAGED` is echoed, or it succeeds and registers the downloaded profile for **root**,
so System Settings running as the agent account has nothing to approve. Only the `/etc/hosts`
write and `security add-trusted-cert` need root.
2. **The Device Management pane is opened with a pre-Ventura identifier.**
`x-apple.systempreferences:com.apple.preferences.configurationprofiles` is the legacy `.prefPane`
id; System Settings on macOS 13+ uses `com.apple.Profiles-Settings.extension`. An unrecognized
anchor leaves Settings on its default pane, so the classifier never sees the enrollment row.
3. **The approval loop's budget is ~20 seconds.** `for _ in 0..<16` with a 1.25 s sleep has to cover
System Settings cold launch, locating the row, the Install sheet, the authentication sheet, and
the profile actually installing plus `mdmclient`'s `Authenticate`/`TokenUpdate`. `approvalTimeout`
(120 s) is not applied to this loop at all — only to `waitForDrain`.
4. **The MDM pass never brings the surface on-screen.** `MacVMSurfaceHost` documents that on-screen
rendering stays live "even when programmatic framebuffer captures come back blank (a known recent-
guest quirk)". The minimal and full passes both honor `baseObserverRequested`; `mdmPolicyPass`
only calls `attach`. Blank captures burn the entire (already too short) budget and surface as
"stopped on an unrecognized screen" with an empty `recognizedText` list.
5. **The authentication classifier requires text the sheet does not render.**
`uamdmApprovalAction` requires `contains(expectedUser)` — the account *short* name — in addition
to both SecurityAgent phrases. The modern sheet for an admin console user shows the icon, the
phrases, and a Password field, with no account name. `.authenticate` is therefore never returned
and the password is never typed.
6. **`.installed` can fire as a false positive.** A stray "Installed" anywhere in the capture, next
to the enrollment profile's display name, terminates the approval loop as successful. The drain
then times out at 120 s with no indication that the GUI step silently no-op'd.
### Protocol bugs that would break enrollment even with a working GUI
7. **The profile advertises a capability the server does not implement.**
`MDMEnrollmentProfile` sets `ServerCapabilities: ["com.apple.mdm.per-user-connections"]`, which
tells macOS to open the per-user channel and send user-scoped check-ins.
8. **Any check-in message type outside the hardcoded three returns HTTP 400.**
`MDMCheckInMessage.parse` throws `unsupported` for anything but `Authenticate`/`TokenUpdate`/
`CheckOut`, and `handleCheckIn` maps that to `400`. Real enrollment also sends
`SetBootstrapToken`/`GetBootstrapToken` and — because of (7) — `UserAuthenticate`. A 400 on the
check-in channel is a hard enrollment failure.
9. **Every certificate the CA issues carries serial 2.** `MDMCertificateAuthority.nextSerial` is a
`let` on a struct and never advances. The device identity and the TLS server leaf share
issuer+serial, and `issueServerCertificate` runs on *every* `NucleicMDMServer.start()` with a
fresh key but the same serial, each pushed through `SecPKCS12Import`. Keychain certificate
uniqueness is (type, issuer, serial), so a repeat run can fail the import and throw before the
pass ever touches the guest. The existing `testUniqueSerials` documents the collision rather than
catching it.
### Correctness and gating
10. **`mdmPolicyPass` has no login recovery and misattributes its failure.** The agent's LaunchAgent
is `LimitLoadToSessionType: Aqua`, so a guest that did not auto-login never loads it; the minimal
pass deliberately treats `sysadminctl -autologin set` as best-effort. The resulting error is
`MacVMError.appInstallFailed("the base image's in-guest agent never answered…")`, which reads as
an app-install failure inside "mandatory MDM enrollment/policy verification failed".
11. **The profile inventory check can produce a false negative.**
`missingInstalledProfileIdentifiers` drains `profiles show -output stdout-xml` through
`drainLines`, whose 64 KB cap silently drops later lines with no truncation marker.
12. **`mdm-state.json` is written before enrollment is proven.** `loadOrCreateMDMController` persists
a fresh CA immediately, so every failed attempt leaves a file that satisfies the stamp gate's
`fileExists` check.
13. **Enroll-step stdout is discarded from the failure message** — the enroll script writes all of its
progress there, and only `stderr.suffix(400)` is surfaced.
14. **`defer { Task { await server.stop() } }`** detaches listener teardown from the function's
lifetime, so the port can still be bound when a retry starts.
15. **`exact("Install…")`** expects U+2026; Vision commonly returns three periods.
Not a defect: `lowest(_:)` is correct. Vision's `boundingBox` is bottom-left origin, so the visually
lowest box has the smallest `midY`, which is what `min { $0.midY < $1.midY }` returns.
### Confidence
(3), (4), (6), (9), (11), (12), (13), (14) are unambiguous from the source. (1), (2), (5), (8) are
read off the code plus documented macOS behavior and have not been reproduced against a live guest —
Workstream 0 exists to make the next run settle them with evidence.
## Sequencing rationale
Live macOS 27 runs are the scarce resource: a full rebuild is ~40 minutes and currently yields one
bit of information. Observability therefore lands first, so one run tells us everything. The GUI and
protocol fixes then land together rather than serially — a working GUI flow still ends in a 400
mid-handshake without Workstream 2, and the protocol is unreachable without Workstream 1.
## Workstream 0 — make the next run diagnosable
- [x] 0.1 Persist the MDM protocol transcript beside the base (`Diagnostics/MDM/`): method, path,
`MessageType`/`Status`, `CommandUUID`, response code, and the leading bytes of any body that
fails to parse. Distinguishes "device never connected" from "device connected and we 400'd it".
- [x] 0.2 Capture every approval-loop iteration (screenshot + recognized text), not only the last
frame, so a blank framebuffer is visually obvious and a wording mismatch is readable.
- [x] 0.3 Include the guest enroll step's stdout in its failure message.
- [x] 0.4 Make `awaitAgentReady(onInstance:)` throw `.agentUnavailable` with a caller-supplied label
instead of `.appInstallFailed`.
- [x] 0.5 Behind a debug default, keep the guest running with the observer visible when the MDM pass
fails, so it can be inspected live instead of only post-mortem.
- [x] 0.6 A fast re-run entry point for the MDM pass alone — no restore, no phases 1/3. Turns a
40-minute cycle into roughly three minutes; the highest-leverage item in the plan.
### Implementation checkpoint — Workstream 0
- `MacVMEngine.MDMDiagnosticsRecorder` (`MacVMEngine+MDMDiagnostics.swift`) collects one attempt under
`<base>/Diagnostics/MDM/run-<label>-<token>/`: a `transcript.log` interleaving the server's protocol
lines with the orchestration's own step notes, per-iteration approval frames, and `approval.json`.
It replaces the previous single-JPEG record, retains the newest five runs, and is best-effort
throughout — losing evidence must never be why a base build fails.
- Frame images are budgeted (40 per run) and written only when they can carry new information: the
first frame, a changed classified state, a frame we acted on, and any frame that recognized nothing.
Recognized text is always recorded. A capture the surface cannot produce at all is now its own
frame rather than a silent `continue`.
- The approval failure distinguishes "every capture recognized nothing" (the guest framebuffer is
blank to the host surface) from "the screen rendered and we did not recognize it". Those have
opposite fixes and previously produced the same sentence.
- `NucleicMDMServer` logs every request before interpreting it, and logs a bounded, single-line
preview of any body that fails to parse. Whether the guest reached the listener at all was not
recoverable from the old per-message logging, which only fired after a successful parse.
- The MDM pass now honors `baseObserverRequested` like the other two provisioning boots; it was the
only invisible one, and the one most likely to need watching.
- `MacVMSettings.holdBaseOnMDMFailure` (`defaults write … -bool YES`) keeps a failed pass's guest
running, on-screen, and still attached to its staged share. `MacVMEngine.heldDiagnosticInstances`
retains the instance (its only owner) and pins `baseIsBusy`, so nothing clones a live-mutating disk.
- **Deviation from the plan as written:** 0.6 landed as
`MacVMEngine.revalidateBaseMDMPolicy()` / `MacVMManager.revalidateBaseMDMPolicy()` rather than as an
extension to `Sources/mdm-spike`. The spike is a headless CLI with no AppKit `MacVMSurfaceHost`, so
it cannot drive the UAMDM approval at all — the very step under repair. Running the loop inside the
app exercises the identical Mode A path (same surface, same direct-instance exec channel, same
verification). It deliberately does not stamp the provisioning recipe: the toolchain and readiness
passes have not run, so success means "enrollment works", not "this base is publishable".
- Unrelated pre-existing failure fixed while verifying: `baseBundleKeepsMDMCredentialsHostSideBesideMetadata`
compared `mdmStateURL.deletingLastPathComponent()` (a directory URL) against a non-directory `root`
URL. The product code was correct; the assertion now compares paths.
- Verification: `swift build` clean across all targets; `MacVM` suites 139/139 and `NucleicMDMTests`
34/34 pass, including new coverage for the body preview, the guest-output formatters, and the
approval state labels.
### First live run against Workstream 0 (2026-07-23)
The instrumented run settled finding (1) and §8.0 as the *first* blocker, exactly as the tooling was
meant to. The transcript beside the base showed the server binding, then:
```
mdm-enroll.sh exit -1
stdout:
[mdm-enroll] discovering vmnet gateway…
[mdm-enroll] pinned nucleic-mdm.host -> 192.168.64.1
stderr:
… mac_vm_exec command exceeded 1m with no exit and was terminated …
```
No check-in ever reached the server. The script wedged on the statement after the `/etc/hosts` pin —
`security add-trusted-cert` into the System keychain — for the full 60 s exec timeout. On macOS 27
that command modifies *admin* trust settings, which raises a SecurityAgent authorization prompt even
when run as root; with nothing to answer it, `security` blocks forever. This is `docs/MACOS_VM_MDM.md`
§8.0 confirmed, and it is why `open` (and therefore any enrollment traffic) never ran.
**First fix attempt (1.1, superseded in part):** ran the enroll as the console user with per-step
`sudo` and tried to make the trust step headless by granting `com.apple.trust-settings.admin` via
`security authorizationdb write … allow`.
### Second live run — the authorization override does not work (2026-07-23)
The next run got materially further: auto-login took (console user "Nucleic Agent"), the diagnostic
monitor showed the desktop, and the run reached the CA-trust step — but the SecurityAgent sheet **"You
are making changes to the System Certificate Trust Settings. Enter your password to continue with
'security'."** appeared anyway. The `authorizationdb write … allow` override did **not** suppress it
on macOS 27. So the prompt must be *answered*, not avoided (docs/MACOS_VM_MDM.md §8.0's stated
fallback). This required three coupled changes, since the surface loop only runs after the enroll
returns and the flow actually has *two* password sheets:
- **Enroll runs the trust step detached.** The script now pins `/etc/hosts` synchronously, then
backgrounds `security add-trusted-cert` + `open` (redirected to a share-local log so the exec
channel's stdio closes) and returns `MDM_ENROLL_STAGED` immediately, leaving the trust sheet up for
the surface. Completion and the trust exit status land in an `mdm-enroll.done` sentinel the host
reads for diagnostics. The `authorizationdb` dance is removed — it didn't work and only added noise.
- **The classifier recognizes the certificate-trust sheet** as its own `AuthKind.certificateTrust`
(distinct wording, "Update Settings" submit), separate from the profile-install sheet. The approval
loop answers **each kind once** rather than latching after a single submit — the previous single
latch would have left the second sheet unanswered.
- **The classifier no longer requires the account short name** (finding 5): the live sheets show the
full name "Nucleic Agent". The safety boundary is unchanged — a recognized intent phrase plus a
Password label plus a submit control, all in one capture, and at most one submit per kind.
Folded in while here, because the trust step's latency makes the next stages reachable in the same
run: the approval loop is now **time-bounded** by `approvalTimeout` (raised 120 → 300 s) instead of 16
iterations (finding 3), and it cycles **Device Management pane candidates** — the macOS 27
`com.apple.Profiles-Settings.extension` first, then privacy, then the legacy id (finding 2) — when
nothing actionable is on screen, recording which one surfaced the profile.
Verified: the real emitted enroll script passes `bash -n` (unprivileged `open`, detached brace group);
build clean; MacVM 140/140 and NucleicMDM 35/35, with new classifier coverage for the trust sheet, the
account-name-independent install sheet, and the two distinct auth kinds. The surface actually
answering these two prompts still needs a bootable guest — the next run.
### Third live run — OCR line-wrapping defeated the classifier (2026-07-23)
The detached-enroll change worked (`mdm-enroll.sh exit 0`), the pane candidates cycled, and — the
diagnostic bundle's final screenshot proved it — the **certificate-trust sheet was on screen the whole
time**. Yet all 153 frames classified `unknown` ("answered: no sheets"), captures were non-blank
(~420 KB), and the server logged no check-ins. The recognized text showed why: Vision emits one
observation per *rendered* line, so the sheet's sentence arrived as three separate lines —
`"You are making changes to the"` / `"System Certificate Trust"` / `"Settings."` — and the classifier's
per-line `contains("Certificate Trust Settings")` matched none of them.
**Fixed:** context phrases are now matched against **all recognized lines joined in reading order**,
not line by line, so a wrapped sentence still matches. `exact()` stays per-line (it locates the click
targets). The trust context simplified to `"Certificate Trust"` / `"System Certificate"` — fragments
that survive on a single line regardless. A regression test pins the exact wrapped lines Vision
returned on the guest, and a second test asserts a split sentence still classifies while still
requiring a Password label + submit before it is actionable.
Anticipating the next stage (never reached before, so its wording is unconfirmed): the profile-install
auth context was broadened to also accept `"install a new configuration profile"`, since macOS may
word that sheet differently from the trust sheet — the Password + submit pairing remains the safety
gate. If the real install sheet still misses, the transcript will now carry its exact lines to pin.
Verified: build clean; MacVM (142) and NucleicMDM (35) suites green; the seven UAMDM classifier tests
include the wrapped-line regressions. The flow reaching and completing the profile install is the next
run.
### Fourth live run — the password field was clicked off the dialog edge (2026-07-23)
The trust sheet was now recognized and answered ("answered: trust"), but 164 frames stayed on
`authenticate-trust` with only one marked `acted`, and the final screenshot showed the sheet still up
with the **"Password" placeholder still visible** — i.e. the field was empty; the password was never
typed into it. The cause was geometric: `uamdmPasswordFieldPoint` clicked at `box.maxX + 0.18`, which
assumed "Password" was a *label* with the field to its right. On the macOS 27 SecurityAgent sheet
"Password" is the placeholder *inside* a full-width field, and `maxX + 0.18` (≈ x 0.65) lands past the
right edge of the narrow centered dialog (≈ x 0.440.56). So the field never focused, the password
typed into empty space, and "Update Settings" submitted an empty field.
**Fixed:** the field is now clicked at the recognized "Password" box's **center** (`clickPoint`), which
lands inside the field. A short settle pause precedes typing, and `cmd+a` goes first so a re-type
replaces rather than appends. `uamdmPasswordFieldPoint` and its test are removed.
Hardened while here, because a 40-minute rebuild is too costly to lose to one flaky keystroke: auth is
now answered up to `maxAuthAttemptsPerKind` (3) times per sheet, and only re-answered once the *same*
sheet has persisted past `reAuthDelaySeconds` (6 s) — a correct answer dismisses the sheet within a
second, so a persisting sheet is the signal that the last attempt failed. The failure summary now
reports per-kind attempt counts (`trust×3`).
Verified: build clean; MacVM (141) and NucleicMDM (35) green; the field-click test asserts the click
stays inside the dialog. The trust answer landing, and the profile-install stage, are the next run.
### Fifth live run — OCR corrupted the exact labels, prompting an architecture change (2026-07-23)
This run the trust sheet was on screen but classified `unknown` for all 123 frames again — this time
because Vision mis-read the very labels the classifier keyed on: the field placeholder "Password" came
back as **"Peseword"** and the button "Update Settings" as **"Updale Settings"**. `exact("Password")`
and `exact("Update Settings")` both missed, so the field and button could not be located even though
the sheet was plainly visible. This is the fragility the exact-label approach always carried.
Two responses — one architectural, one to the OCR itself:
**Removed the CA-trust prompt entirely (the better way).** Apple removed headless MDM-enrollment
install in macOS 11, so the *profile-install* approval is genuinely unavoidable — but the *trust*
prompt is not. The CA root is now delivered inside the enrollment profile as a
`com.apple.security.root` payload (installed first, before the MDM payload's check-in TLS), so the
single profile-Install approval establishes trust too. `security add-trusted-cert` — and its
unsuppressible SecurityAgent sheet, the wall that stalled four straight runs — is gone. The enroll
script is now trivial and synchronous (pin hosts, open profile); no backgrounding, sentinel, or marker
race. This bets that a profile-delivered root installed at System scope with admin approval is trusted
for the check-in TLS (the standard enterprise mechanism; the doc's §2.3 "not auto-trusted" caveat
appears to describe iOS's manual-trust step, which macOS lacks). If it proves wrong, the server log
will show the check-in TLS failing with no successful drain — a clean, unambiguous signal.
**Made the one remaining prompt generic and OCR-noise-tolerant** (the user's request to detect generic
Apple password requests rather than brittle exact matches):
- Detection keys on the stable phrase **"Enter your password"** plus a *located* password field — not
on a sheet-specific intent phrase or account name. This handles any Apple authorization sheet.
- The password field is found by **fuzzy match** (Levenshtein ≤ 2 to "password"), so "Peseword" and
similar OCR slips still resolve; the lowest such box on screen is the field.
- Submission is **Return** (the sheet's default button), so the button label is never OCR-matched —
removing the "Updale Settings" class of failure entirely.
- In the sealed base build the only password sheets are our own provisioning prompts and the only
secret typed is the machine's own agent password, so a generic detector is safe here; a non-English
locale still yields `unknown` rather than speculative input.
Folded in while the check-in path is now imminent: dropped the unimplemented `ServerCapabilities`
(Workstream 2.1, finding 7) and made the check-in channel ack unmodeled `MessageType`s with an empty
200 instead of 400 (Workstream 2.2, finding 8) — both would otherwise abort the handshake the profile
install is about to trigger.
Verified: emitted profile lints clean with payload order root → pkcs12 → mdm and no
`ServerCapabilities`; emitted enroll script passes `bash -n` with no trust/background steps; build
clean; MacVM (141) and NucleicMDM (37) green, including fuzzy-field and OCR-noise regressions and the
ack-don't-throw check-in test. Whether the profile-delivered root establishes trust, and the no-APNs
drain, are the next run.
### Sixth live run — profile-delivered root worked; clicks landed on the desktop (2026-07-23)
The profile-delivered root removed the trust prompt entirely: the run reached System Settings ▸ Device
Management showing **"Nucleic MDM Enrollment — Profile not installed. Double-click to review."** in a
proper window. The classifier recognized the row and double-clicked it seven times — but every click
landed on the desktop, not the row, so the profile never opened.
Root cause was a coordinate-scaling bug in the computer-use surface, exposed by the diagnostic monitor
being on-screen. `MacVMComputerSurface.guestPoint` built mouse-event locations in fixed 1920×1200
coordinates. Off-screen the HID window is a native 1920×1200, so that is correct and ordinary agent
computer-use works. But when the monitor is shown, `setWindowOnScreen` resizes the window to 1280×800
and the `VZVirtualMachineView` autoresizes to fit — so a location computed for 1920×1200 maps to the
wrong place in the smaller view, landing a row double-click on the desktop. Showing the monitor during
the MDM approval (the previous checkpoint's change) is what surfaced it.
**Fixed:** `guestPoint` now scales the event location by the view's *live* bounds, so clicks land
correctly whether the window is off-screen (1920×1200), shown (1280×800), or user-resized. It reduces
to the old identity flip when the view is 1920×1200. Applied to click, move, and drag.
Also addressed from the same report:
- **First-boot timing.** The bootstrap's first Terminal launch fired before the freshly-booted desktop
was ready, wasting the attempt (then waiting a full retry interval). `driveHIDBootstrap` now holds an
initial settle before the first launch — 20 s on the declarative first boot, 8 s otherwise — the
Spotlight/cold-launch pauses in `typeBootstrapLaunch` are a touch longer, and the launch-retry
interval dropped 40 → 25 s.
- **Sampling frequency.** The bootstrap poll dropped 3 s → 1 s, and the approval loop now samples every
~0.4 s (0.9 s right after it acts, to let the UI react) instead of 1.25 s, so an element is acted on
promptly after it appears. Pane cycling stops once the enrollment UI first appears, so a review/
install sheet that momentarily reads as `unknown` is not navigated away from. Each click's target
coordinate is now logged to the transcript to confirm the mapping on the next run.
Verified: build clean; MacVM (141) and NucleicMDM (37) green. The profile-review → Install → password
→ drain sequence, with clicks now landing correctly, is the next run.
### Seventh live run — the review sheet's button is "Enroll", and ServerCapabilities is required (2026-07-23)
The coordinate fix landed: the double-click now opened the profile-review sheet ("Are you sure you
want to install this device profile?"). Two issues from there, both confirmed on the live guest — and
the password-field autofill + Return-submit worked correctly once reached, so those fixes are proven.
- **The confirm button is "Enroll", not "Install".** The macOS 27 review sheet uses "Enroll"; the
classifier only knew "Install"/"Install…", so it never clicked. Added "Enroll" (chosen over
Ignore/Cancel) as a confirm-button candidate, and `"install this device profile"` as an additional
enrollment context. After a manual Enroll click, the flow proceeded exactly as designed.
- **`ServerCapabilities` is mandatory, not harmful — finding 7 was backwards.** After Enroll, the
install failed with "MDM payload is missing 'ServerCapabilities' key that indicates it supports a
user channel." macOS 27 *requires* the key. Restored
`ServerCapabilities: ["com.apple.mdm.per-user-connections"]` in the enrollment profile. Workstream
2.2 (ack unmodeled check-ins with an empty 200) is what makes advertising the per-user channel safe:
macOS will send user-scoped check-ins we don't fully model, and they are acked so the device channel
still drains the InstallProfile queue.
Verified: build clean; MacVM (142) and NucleicMDM (37) green, with the Enroll-button classifier test
and the ServerCapabilities-present profile test; the emitted profile again carries ServerCapabilities.
The install completing and the queue draining are the next run.
### Eighth live run — the CA cannot bootstrap itself from inside the profile (2026-07-23)
`ServerCapabilities` cleared the previous failure and the install proceeded further, then failed with
**"The certificate could not be verified (authentication error)."** This lands exactly where the
in-profile `com.apple.security.root` payload was made the *only* trust mechanism, so that bet was
wrong and **docs/MACOS_VM_MDM.md §2.3 was right**: the identity/check-in TLS is validated before the
profile's own root payload takes effect, so the CA cannot bootstrap its own validation.
**Fixed:** `security add-trusted-cert` into the System keychain is restored, and it must run *before*
the profile opens. Because it raises the unsuppressible SecurityAgent trust sheet (§8.0), it is again
launched **detached** — the script pins `/etc/hosts`, backgrounds `add-trusted-cert``open`, and
returns `MDM_ENROLL_STAGED` immediately so the surface's approval loop can answer that sheet. This is
now viable where it wasn't before: the generic password-sheet handler is *proven* on hardware (the
seventh run's install prompt autofilled and submitted correctly). The in-profile root payload is kept
as well — harmless, and it makes the trust durable in the sealed base — but the System-keychain trust
is what lets the install succeed. The trust exit status and the detached block's log are read back into
the transcript.
Two consequences handled: the pane-cycling latch no longer trips on a *password* sheet (the trust sheet
appears before the profile is queued, so latching there would stop cycling before the profile row was
ever surfaced — it now latches only on the profile row / review sheet / installed), and the auth budget
rose 3 → 5 to cover two sheets with retry headroom.
Verified: emitted enroll script passes `bash -n` with the trust step at line 43 preceding `open` at 46,
both inside the detached block; profile retains root + ServerCapabilities; build clean; MacVM (142) and
NucleicMDM (37) green, with a test asserting trust precedes open. Two sheets answered in sequence, the
install completing, and the no-APNs drain are the next run.
### Ninth live run — duplicate certificate serials, and an unsigned profile (2026-07-23)
Both password sheets were answered (trust, then install) and the Enroll click landed, but installation
still failed with "The certificate could not be verified (authentication error)". Two defects found.
**Every issued certificate carried serial number 2 (finding 9, finally fixed).** `signLeaf` stamped
`serialBytes(nextSerial)`, and `nextSerial` is a `let` on a value type that never advanced — so the
device identity baked into the profile and the MDM server's TLS leaf shared `(issuer, serialNumber)`.
X.509 identifies a certificate by exactly that pair, and Apple's keychain/trust machinery keys
certificate lookups on it, so the guest could not resolve which certificate was which. Leaves now get a
fresh 16-byte random serial (RFC 5280 §4.1.2.2; positive, minimally encoded, unique across CA reloads
in a way a persisted counter would not be). The root keeps serial 1. The old `testUniqueSerials`, which
*documented* the collision, is replaced by one that issues nine leaves and asserts nine distinct
serials.
**The profile was unsigned ("Nucleic (Not Signed)").** Strictly, the error cannot be about a *missing*
signature — an unsigned profile has no signing certificate to verify — so this is not self-evidently
the cause. But it is the right thing to fix regardless: on modern macOS, certificate payloads inside an
*unsigned* profile are installed without being granted trust, which is precisely the failure class
here, and this profile exists to deliver a CA root plus a PKCS#12 identity that the check-in TLS then
depends on. The enrollment profile is now CMS-signed (attached `SignedData`) by a CA-issued
`digitalSignature` leaf — a CA the guest already trusts by the time it opens the profile. Signing is
best-effort: it falls back to the unsigned plist rather than breaking enrollment, and
`unsignedEnrollmentProfile()` remains for inspection.
Verified beyond the unit tests, since this changes the staged artifact's format: the emitted profile is
DER `id-signedData`, `openssl pkcs7` reports the signer as `CN=Nucleic Profile Signing` issued by
`CN=Nucleic MDM Root CA`, and — the decisive check — **macOS's own `security cms -D` recovers the
plist**, which lints clean and still contains root → pkcs12 → mdm with `ServerCapabilities`.
Also this round: the trust sentinel and the detached block's log are now recorded on the *failure* path
too (previously lost, so whether `add-trusted-cert` actually succeeded was unknowable), the diagnostics
capture the guest's `ManagedClient`/`mdmclient`/`profiles` log — where the real reason an install failed
is recorded, since the on-screen alert says nothing actionable — and control clicks (enrollment row,
Enroll) retry on a bounded time cadence instead of once per distinct OCR fingerprint, which previously
granted extra clicks on incidental text noise while starving a genuinely stuck screen.
Build clean; MacVM (142) and NucleicMDM (39) green.
### Tenth live run — sheet focus, and a shared retry budget (2026-07-24)
The certificate failure is **gone**: the serial fix plus profile signing carried the flow through the
review sheet and the Enroll confirmation. It then stalled at the last step — "Device Management is
trying to enroll you in a remote management (MDM) service" — with the password never submitted. The
operator's step-by-step named both causes precisely.
**A sheet that isn't key swallows the first click.** The trust sheet "is not consistently foregrounded…
must be manually clicked on to foreground, at which point it is automatically completed" — i.e. the
typing was always correct, but the sheet was not key, and the first click on a non-key window is
consumed activating it, so the caret never landed and the keystrokes went nowhere. The answer sequence
now sends **two** clicks at the password field, 600 ms apart: the first foregrounds the sheet, the
second places the caret. Both target the same field, so neither can stray onto another control.
**The retry budget was global, not per sheet.** `maxAuthAttempts` was one counter for the whole loop, so
while the unfocused trust sheet was retried every 6 s it consumed all five attempts — leaving nothing
for the final enrollment sheet, which is exactly where the run stalled. Attempts are now tracked per
sheet, keyed by a slug derived from the sheet's own wording (`certificate-trust`, `mdm-enroll`,
`modify-settings`, `profile-install`), so each gets its own budget and the failure summary reports them
individually (`certificate-trust×5, mdm-enroll×0` would have made this obvious at a glance).
Verified: build clean; MacVM (143) and NucleicMDM (39) green, including a test that pins the exact
macOS 27 wording of each sheet to a distinct budget key and asserts two different sheets cannot collide
onto one.
### Eleventh live run — the GUI now completes; the failure is guest→server TLS (2026-07-24)
The focus and per-sheet-budget fixes worked. The transcript shows the whole GUI sequence running
unattended for the first time: `certificate-trust` answered → pane opened → enrollment row
double-clicked → **Enroll clicked**`mdm-enroll` sheet answered. Installation then failed with the
same "The certificate could not be verified (authentication error)".
The decisive observation is in the transcript's *absence*: the host listener logs
`MDM server listening on 192.168.64.1:8443` and then **never records a single request** — in this run
or any previous one. Guest→host reachability has therefore never actually been demonstrated (it is an
open P2 item). Combined with the error appearing at the moment `mdmclient` first contacts the
ServerURL, the likely failure is the **check-in TLS handshake**, not a payload import — the earlier
assumption that this was about a certificate *inside* the profile does not fit the evidence.
Rather than guess a fourth time, the guest is now asked directly. Two probes run after the trust step
has had its chance, on both the success and failure paths, and land in the transcript:
- **CA trust:** does `Nucleic MDM Root CA` actually appear in the guest's System keychain? (The trust
sentinel only reports `add-trusted-cert`'s exit status, which can be 0 while the resulting trust
setting is not what TLS evaluation consults.)
- **Reachability:** `curl` the CheckInURL from the guest, once normally and once with `--insecure`.
curl's exit code separates the possibilities precisely — 6 unresolved, 7 refused, 28 timeout, 35/60
TLS/certificate — and the verified-vs-insecure pair is the clean discriminator: *both* failing with
7/28 means the guest cannot reach the host at all (firewall / listener binding), whereas insecure
succeeding while verified fails with 60 means it reaches the server fine and distrusts our CA. The
count of `/etc/hosts` pins is reported alongside, to rule out the DNS pin.
Build clean; MacVM (143) and NucleicMDM (39) green.
### Server-side debug trace
The request-level `LogSink` could not answer the question the enrollment failure poses, because the
suspected failure produces no request: a guest whose TLS handshake fails connects, is rejected, and
disconnects, and the old logging emitted **nothing** — the server looked identical whether it was
healthy, unreachable, or refusing the client.
`MDMDebugLog` (append-only, size-rotating, 0600) now records the server's own view to
`Diagnostics/MDM/run-*/mdm-server-debug.log`, and `mdm-spike` writes the same trace beside its staged
artifacts so failures can be reproduced without a guest. Instrumented: listener lifecycle and bind
failures; the issued TLS leaf and CA root (subject/issuer/serial/validity, to compare against the
guest's trust store); `sec_identity_t` construction; **every accepted connection with its endpoint and
each state transition** — which is where a handshake failure actually surfaces; raw byte counts with
bounded previews; parsed request line, headers (the `Mdm-Signature` blob is reported by size only) and
body; signature verification; check-in message fields including the topic actually presented versus the
one configured; and queue movement on delivery and drain.
Validated end to end against the real server rather than assumed, driving it with an untrusted and then
a CA-trusted client:
```
conn id=1 ACCEPTED from=127.0.0.1:51656 — awaiting TLS handshake
conn id=1 FAILED: -9831: unknown Cert Authority — likely TLS/certificate
...
conn id=2 READY (TLS handshake complete)
http id=2 REQUEST #1 PUT /checkin headers=5 bodyBytes=160
checkin id=2 Authenticate udid=TEST-UDID-1234 topic=- (expected topic=com.apple.mgmt.External.…)
http id=2 RESPONSE 200 OK
```
`-9831` is `errSSLUnknownRootCert` — exactly what a guest that distrusts our CA produces, and exactly
what was previously invisible. This also establishes that **the server itself is healthy**: given a
client that trusts the CA, the full check-in round-trip returns 200.
The next run therefore reads as a decision tree: no `ACCEPTED` line at all ⇒ the guest never reached
the host (routing/firewall/binding); `ACCEPTED` then `FAILED: -9831` ⇒ the guest reached us but does
not trust the CA, so `add-trusted-cert` did not take effect; a full request flow ⇒ TLS is fine and the
fault is elsewhere.
### Twelfth live run — the trace answered it: a pointer/click race (2026-07-24)
The server trace read: `listener READY`, `LISTENING on 192.168.65.1:8443`, and then **no `ACCEPTED`
line at all**. The guest never opened a TCP connection, so this was never a TLS or certificate problem
in the server — the third branch of the decision tree, resolved in one run instead of by inference.
(Note the vmnet subnet had moved from `192.168.64.x` to `192.168.65.x`; the gateway discovery handled
that correctly and bound the right address.)
The transcript supplied the cause. The keystroke-landing check reported
`"certificate-trust" attempt N: password landed in field=false` five times: the CA-trust sheet never
received the password, so the CA was never trusted, so the profile's certificate payloads could not be
validated ("The certificate could not be verified"), so `mdmclient` never reached the check-in stage
and never connected. One root cause, presenting as a certificate error three layers downstream.
Why the keystrokes missed was settled by measurement rather than argument. The diagnostic frame
(1920×1200) cropped around the computed click point shows **(880, 466) lands squarely inside the
Password field** — the coordinate was right. But the sheet's "Update Settings" button renders **grey,
not blue**, and System Settings is frontmost: the click was activating the window *behind* the sheet.
**Fixed:** the pointer move is now posted as its own event, followed by a `pointerSettleMillis` (180 ms)
settle, before the button-down. Previously the move and press went out in one batch, so the guest could
process the button-down while its pointer was still at the previous location and deliver the click to
whatever was under the *old* position. That also explains the intermittency the operator observed —
"sometimes activated and completed if lucky" is exactly what a race looks like. The fix is in the
surface, so it applies to every consumer of the computer-use path, not just enrollment.
Build clean; MacVM (143) green.
### Removing the fixed-resolution assumption
Reviewing the click path surfaced a latent bug independent of the race above:
`clickPoint(forNormalizedBox:width:height:)` multiplied Vision's normalized box through a **hardcoded
1920×1200**. Vision already returns resolution-independent coordinates, and `MacVMSurfaceHost`
guarantees that captures and `.click` share one space — so the constant was both redundant and a silent
failure waiting to happen: any change to the guest panel, the view size, or the capture normalization
would misplace *every* click with no error anywhere.
Geometry is now measured from the frame being acted on. `imageSize(ofJPEG:)` reads the capture's true
dimensions from its header (no decode), and `clickPoint(forNormalizedBox:in:)` scales against that, so
the only place a resolution enters the system is derived from the same image the text was recognized
in. The old constants remain solely as a fallback for an unreadable capture and as unit-test defaults.
The placeholder matcher was loosened in the same spirit: it accepts `password`/`passcode`/`pin` with a
tolerance that scales to the word (about a quarter of its characters), rather than one fixed edit
distance that was lenient on short words and strict on long ones. It still refuses unrelated short UI
text — `Wallpaper`, `Cancel`, and `Passkeys Access for Web Browsers` are all rejected by test.
Tests pin the property that matters: the same normalized box maps proportionally onto 1920×1200,
3840×2400, and a non-16:10 2560×1440 panel, each axis scaling independently.
Build clean; MacVM (145) and NucleicMDM (39) green.
### CA trust moved to the System Settings install flow
The CLI keychain-trust step was abandoned. `security add-trusted-cert` raises a SecurityAgent sheet
owned by a root process with no GUI application: it never becomes key, and five consecutive automated
attempts typed into nothing while the sheet sat inactive — the operator had to click it by hand every
time. Rather than keep fighting for focus on a window macOS will not activate, the CA is now delivered
the ordinary interactive way: **as its own configuration profile, installed through System Settings
before the enrollment profile.** That routes the approval through the profile-install sheet, which
belongs to System Settings and which the surface already drives successfully.
This is distinct from the earlier failed attempt to carry the root *inside* the enrollment profile.
There the root was still being installed while the very payloads depending on it were validated — the
chicken-and-egg §2.3 warned about. As a separate, earlier install it is fully in place and trusted
before the enrollment profile is opened at all.
Shape of the change:
- `MDMEnrollmentProfile.makeRootProfile` builds a standalone `com.apple.security.root` profile named
"Nucleic MDM Root CA", staged as `nucleic-mdm-ca.mobileconfig`. Deliberately **unsigned** — signing
it with the CA it delivers would be circular, since the guest cannot verify that signature until this
profile is installed.
- The enroll script now only pins `/etc/hosts` and opens the **root** profile. It is fully synchronous
again: nothing in it prompts, so no backgrounding, no sentinel, no marker race.
- Enrollment runs in two phases, each a full install driven through Device Management:
root profile → confirm the CA is in the guest's System keychain → open the enrollment profile →
install it → confirm enrollment. Reachability and CA-trust probes are recorded after each phase.
- The approval loop is generalized from "approve enrollment" to `installProfileViaComputerUse`, keyed
to a `ProfileTarget` (display name + an OCR-tolerant row keyword) and a completion predicate asked of
the **guest**, not the screen. So the same proven loop installs either profile, and pixels never
decide success.
Verified on the real emitted artifacts: all four staged; the enroll script passes `bash -n`, opens only
`$ROOT_PROFILE`, and contains no CLI trust invocation; the root profile lints clean and carries exactly
one `com.apple.security.root` payload holding the CA DER; the enrollment profile is still CMS-signed
(`id-signedData`). Build clean; MacVM (145) and NucleicMDM (40) green, including new coverage that the
root profile is unsigned and single-payload and that the enroll script cannot open the enrollment
profile early.
### Twelfth live run — the decision tree resolves: everything reduces to the trust prompt
The trace answered it on first use. The guest's own probes and the server's connection log agree:
```
conn id=1 ACCEPTED from=192.168.64.64:49152 — awaiting TLS handshake
conn id=1 FAILED: -9831: unknown Cert Authority
conn id=2 ACCEPTED → READY (TLS handshake complete) # the --insecure probe
http id=2 REQUEST GET /checkin → RESPONSE 400
guest → MDM server reachability: verified: rc=60 "unable to get local issuer certificate"
enroll trust sentinel absent
```
So **networking is entirely healthy** — the guest reaches `192.168.64.1:8443`, and the server answers
correctly the moment certificate validation is bypassed. The single fault is that the guest does not
trust the CA, because `add-trusted-cert` is still blocked on its SecurityAgent prompt (sentinel
absent). Every downstream symptom chased over the last several runs — "the certificate could not be
verified", the failed profile installation — is a consequence of that one prompt. This retires the
competing hypotheses (payload certificates, profile signing, serial collisions, reachability) as causes
of *this* failure, though the fixes made along the way were independently correct.
Measured against the raw 1920×1200 capture, the click target is right: the Password field spans
x≈851910, y≈455478 (centre ≈ 880, 466) and the loop clicked (882, 467). So this is **not** a
targeting error. It is a focus/input race — the sheet is answered "if lucky", the operator reports it
neither foregrounds nor accepts typing once foregrounded, and five attempts reported success while the
field stayed empty for 397 frames.
**Fixed:** the answer sequence no longer fires blind. The activation click gets a much longer settle
(1.2 s) before the caret click, `cmd+a` is dropped (the field is empty, and it is a no-op at best in a
non-key window), and — the substantive change — the loop now **verifies the password actually landed
before submitting**: a secure text field replaces its placeholder with bullets once it holds text, so
the placeholder still being recognised is direct evidence the keystrokes never reached the guest. Return
is sent only on confirmation; otherwise the attempt is recorded `landed=false` and retried rather than
submitting an empty field. That single line in the transcript will distinguish "keystrokes are not
reaching the guest at all" from "they land but the sheet rejects them", which no previous run could.
## Workstream 1 — enrollment-pass blockers
- [x] 1.1 Run the enroll script as the console user with per-step `sudo` (hosts pin, CA trust), so
`open` registers the profile for the console user; make the CA-trust step headless by granting
`com.apple.trust-settings.admin` non-interactively (the confirmed §8.0 hang). Log the console
user as the auto-login canary and report a non-zero `open` rather than swallowing it.
- [x] 1.2 Try an ordered list of System Settings pane identifiers (`com.apple.Profiles-Settings.extension`,
`com.apple.settings.PrivacySecurity.extension`, then the legacy id), advancing when nothing is
recognized, and record which worked. Mirrors the existing `nudgeMDMCheckIn` candidate pattern.
- [x] 1.3 Bound the approval loop by `approvalTimeout` (raised to 300 s) rather than 16 iterations,
republish per-state progress, and treat an unchanged fingerprint for ~60 s as its own failure.
- [ ] 1.4 Force the surface on-screen for the duration of the approval loop and restore the prior
state afterward; honor `baseObserverRequested` like the other passes; fail fast on N consecutive
captures with zero recognized lines.
- [x] 1.5 Drop `contains(expectedUser)` from the authentication classifier and accept the known
phrasings. The safety property stays what it already is: credentials submitted at most once,
with both a Password label and a submit control required in the same capture. Accept `Install...`
alongside `Install…`.
- [x] 1.6 Make guest state the success signal: OCR drives actions only, and the loop terminates on
`verifyEnrolled`. Tighten `verifyEnrolled` so Mode A requires *User Approved* rather than also
accepting a bare `MDM enrollment: Yes`.
## Workstream 2 — MDM protocol conformance
- [~] 2.1 ~~Remove `ServerCapabilities` from the enrollment profile.~~ **Reversed by the seventh live
run:** macOS 27 *requires* the key (install fails without it). It is kept; Workstream 2.2 makes
advertising the per-user channel safe by acking the unmodeled check-ins it triggers.
- [x] 2.2 Never 400 the check-in channel: add an `other(type:udid:)` case, ack unknown types with an
empty 200, and reserve 400 for a body that is not a property list (logging its prefix).
- [x] 2.3 Regression-test both; they are Linux-runnable in `NucleicMDMTests`.
## Workstream 3 — CA and TLS identity correctness
- [ ] 3.1 Issue leaves with a fresh 16-byte random serial (RFC 5280 §4.1.2.2 permits 20 octets).
Guarantees uniqueness across issuance and reloads without making the struct mutable or changing
the persisted shape; `Persisted.nextSerial` stays for compatibility and becomes vestigial.
Replace `testUniqueSerials` with one that asserts distinctness.
- [ ] 3.2 Import the TLS identity into a temporary per-start keychain and delete it in `stop()`,
instead of accumulating identities in the default keychain.
- [ ] 3.3 Await `server.stop()` on both the success and throw paths.
## Workstream 4 — verification and gating
- [ ] 4.1 Check the profile inventory in the guest (one `PRESENT:<identifier>` line per hit) instead
of draining a large XML document through a silently truncating cap.
- [ ] 4.2 Persist MDM state only after verification, and strengthen the stamp gate from `fileExists`
to "state parses and its `profileIdentifiers` match the required set".
- [ ] 4.3 Recover from a missing auto-login: have the minimal pass read back the effective auto-login
user and fail there, and let the MDM pass answer a *recognized* login window once. Same
fail-closed discipline — an unrecognized screen receives no input.
## Workstream 5 — escape hatch
`MacVMSurfaceHost.presentOperatorAssist(name:visible:)` already exists and is unused here. If the
UAMDM flow still cannot be driven unattended after Workstreams 14, the approval loop's terminal
state becomes "present the assist window and wait, bounded, for a human to click Install" rather than
failing the build. Base builds are rare and already operator-initiated, so a one-click hand-off is a
better outcome than a failed build, and it decouples shipping a working base from winning the OCR
fight. Wire it opt-in first (`nucleic.macvm.mdmOperatorAssist`), then decide from live evidence
whether it becomes the default. The SIP-off lane (provisioner Phase 7c/d) remains the deeper
fallback, but it flips the base's security posture and is a last resort.
## Test plan
Linux-runnable (`NucleicMDMTests`), no toolchain gate:
- unknown check-in `MessageType` → 200 empty (2.2)
- the enrollment profile carries no `ServerCapabilities` (2.1)
- two leaves from one CA have distinct serials; the root is serial 1 (3.1)
- `enrollScript()` invokes `open` unprivileged and `sudo`s only the hosts pin and CA trust (1.1)
- the classifier accepts an auth sheet lacking the account name, and a bare "Installed" does not
terminate the loop (1.5, 1.6)
- `verifyEnrolled` rejects `MDM enrollment: Yes` without *User Approved* (1.6)
- the inventory command emits one `PRESENT:` line per installed profile (4.1)
Host-side (`NucleicCoreTests`, needs an approved macOS run):
- back-to-back `MDMTLSIdentity.serverIdentity` calls both succeed (3.2)
- pane-candidate ordering and the guest inventory command shape (1.2, 4.1)
Live macOS 27: the P2 checklist above, run through the `mdm-spike` loop (0.6) until it passes twice
consecutively, then once end to end through `buildBaseImage`.
---
# Queue-drain repair (third investigation)
**Opened:** 2026-07-26
**Symptom:** enrollment completes (`Authenticate` signed + 200 OK, User Approved), then the two queued
`InstallProfile` commands never drain: no `TokenUpdate`, no `/server` poll, timeout at queue-drain.
## Diagnosis (evidence, not inference)
1. **Why there is no `TokenUpdate`:** apsd never obtains a push token because it never dials a courier.
The guest log (`run-enroll-968ae158`, ~line 1217) names the gate: `APSBAAClientIdentityProvider
failed to obtain a BAA cert` ← MobileActivation `"Failed to create reference key"`
`errSecInteractionNotAllowed`, `<sepk:* kid=0000000000000000>: unable to generate key`, then
`"Told not to connect after fetching server bag"`. macOS 27's apsd requires a SEP-backed **BAA
client identity** before opening a socket; SEP keygen fails in the guest, so apsd deliberately
stays offline — zero packets to 17/8, a rising failure counter, and no error surfaced anywhere
louder than debug logging. Networking was never the problem.
2. **Why the nudges were inert:** `launchctl kickstart` on an already-running on-demand daemon is a
no-op; `profiles renew -type enrollment` is ADE-only (repurposed on macOS 27); `mdmclient` query
verbs answer locally. Twelve "successful" nudges in `run-enroll-22d72c27` produced zero
connections. There is no supported command that makes an enrolled Mac poll on demand.
3. **The design premise was false.** "`mdmclient` polls immediately after `TokenUpdate`, so pre-queued
commands drain with no APNs" (old §2.4) is not the protocol: the command channel drains on a
push — or on the two triggers below. (§2.4 is corrected in `docs/MACOS_VM_MDM.md`.)
4. **Two drain triggers need no APNs and no push certificate:**
- **Login-time sync** (Apple-documented): at login, `mdmclient` blocks on an
`Idle`/`Acknowledged`/`Error` exchange with the server. The guest auto-logs-in ⇒ a **reboot
with the server still listening** is a supported, deterministic drain.
- **Cold daemon start** (proven live, `run-push-e3d6a047` 2026-07-26 00:1500:17): one
`launchctl kickstart` of the *stopped* `com.apple.mdmclient.daemon``Idle` poll 2 m 13 s
later → both `InstallProfile`s delivered, **Acknowledged, and verified installed**. The held
base guest already carries both policy profiles.
5. **Why BAA keygen fails — the mechanism, after the deep dig (2026-07-26).** Three facts compose:
- **VZ guests categorically cannot mint SEP-resident keys, on every macOS version.** The Sequoia+
"identity derived from the host's Secure Enclave" feature is an attestation identity blob
delivered via an exclave (the `AppleVirtualPlatformIdentity` plugin) for Apple-Account flows —
it is *not* a virtual SEP, and `kSecAttrTokenIDSecureEnclave` keygen stays unavailable
(khronokernel: the VP kexts stub SEP attestation with errors; Eclectic Light on Sequoia/Tahoe
VMs). So MobileActivation's "reference key" creation fails in *any* VZ guest — nothing Nucleic
did causes `sepk kid=0 / errSecInteractionNotAllowed`. (The same error chain even appears
sporadically on physical Macs — Apple forums thread 758028, macOS 14.5 ADE — so it is a
keystore-refusal signature, not a VM oddity.)
- **macOS ≤26 apsd did not require that key.** Tahoe (26.0/26.2) VZ guests have full working
iCloud — which rides on APNs — despite the identical SEP limitation. The courier identity there
comes from the classic Albert-provisioned client certificate.
- **macOS 27 beta 4 apsd consults `APSBAAClientIdentityProvider` and refuses to connect without
it** ("identity to connect? NO → Told not to connect after fetching server bag"). The symbol
has zero public mentions — this identity path is new in 27 (or newly mandatory, possibly
server-bag-gated), and our failure appears to be **unreported**.
Everything else was individually exonerated: solo macOS VM confirmed (the second VZ process was
a Linux guest — no Mac machine identifier, so no identity rotation), valid serial, healthy
networking, identity plugin attached, and the account comes from Apple's own
`VZMacGuestProvisioningOptions` (which feeds Setup Assistant's first-user flow; no public link
exists between SecureToken/ownership gaps and -25308). The concurrent-duplicate-MID rotation
rule remains a latent fleet hazard for session clones — D.3's concern — but did not cause this.
Actions: file Apple Feedback/DTS with the BAA chain and a minimal repro (D.7) — Apple fixed both
adjacent 27-era VM radars (163294564 serials, 179068335 installer); retest each beta; fresh
per-clone `MachineIdentifier`s remain a non-option (install is personalized to the base ECID,
VirtualBuddy #238).
## Workstreams
- [ ] D.1 **Unblock now:** release the held diagnostic guest, then re-run the MDM pass
(`revalidateBaseMDMPolicy`). The enrolled base takes `.update``pushProfiles` — the exact
path that already succeeded. Expect apsd to keep failing (see 5) — the drain does not need it;
capture the apsd/BAA lines anyway as the evidence bundle for the Apple feedback report.
- [x] D.2 **Make the Mode A drain deterministic** — done (`MacVMEngine+MDM.swift`). `enrollBaseInMDM`
now calls `drainQueueForModeA`, which escalates: (1) 90 s quiet wait for a natural check-in;
(2) `coldStartMDMClient` (`launchctl kickstart -k … || kickstart …`) + 150 s wait — the proven
cheap trigger; (3) **reboot the guest with the server still listening** (`shutdownBaseGuest`
`instance.start()``awaitAgentReady`), whose auto-login runs the login-time Idle sync, then a
420 s `drainRebootDrainSeconds` wait. `nudgeMDMCheckIn` and the `waitForDrain` nudge cadence are
deleted; `waitForDrain` is now a pure poll. Mode B (`pushProfiles`) uses a single
`coldStartMDMClient` (a live clone must not be rebooted under a user). Builds clean.
- [ ] D.3 **Solo-run invariant:** the MDM pass (and any base-mutating boot) refuses to start — with a
named error — while another macOS VM is live, so the base can never boot onto a rotated
identity mid-build.
- [ ] D.4 **Diagnose-by-default:** add `mobileactivationd` to the guest-log predicate; on drain
failure, surface the apsd BAA/courier lines in the failure message instead of a bare
"N still queued".
- [ ] D.5 **Deferred / optional APNs restoration:** if D.1 shows 5(b), file Apple feedback and retest
each beta; a macOS 26 IPSW pin (`restoreImagePath`) is the fallback lever if 27 stays broken.
A real push certificate (mdmcert.download; Topic must then come from the cert) is needed only
if Apple-initiated push ever becomes a requirement — the reboot/cold-start triggers make it
unnecessary for both Mode A and Mode B today.
- [ ] D.6 **Docs:** §2.4 corrected (done); close debug-log open question with these facts (done);
strike the debug log's parting suggestion to GUI-install the policy profiles — PPPC and
notification-settings payloads are inert unless MDM-delivered, so that fallback never worked.
- [ ] D.7 **Minimal repro + in-guest confirmation (closes the residual doubt).** Two cheap
experiments, both independent of the app's held state:
1. *Near-pristine guest:* fresh IPSW install into a temp bundle, started with only Apple's
`VZMacGuestProvisioningOptions` and `enablesRemoteLogin = true`; ssh in and run
`sudo apsctl status` (production block), `log stream` on `com.apple.apsd` +
`mobileactivationd`, `sysadminctl -secureTokenStatus <user>`, and
`diskutil apfs listUsers /`. The BAA failure reproducing on a guest Nucleic never scripted
is the definitive "it's macOS, not us" — and the artifact IS the Apple Feedback repro.
2. *Current base (after release):* same ladder, plus confirm the account's
SecureToken/ownership state — closing the last "our provisioning" suspect empirically
rather than by argument.
## Rollout
No on-disk format changes, so the provisioning recipe stays at **v8** and existing bases need no
forced rebuild beyond what v8 already triggers. On landing, close the relevant
`docs/MACOS_VM_MDM.md` §8 spikes with facts from the transcript rather than assumptions.