Files
nucleic/docs/MACOS_VM_MDM_DEBUG_LOG.md
T
2026-07-25 20:10:17 -07:00

67 KiB
Raw Blame History

macOS VM MDM provisioning — debugging log

A chronological record of the effort to make the mandatory MDM enrollment pass of the macOS 27 golden-base build complete unattended. The design and the current specification live in docs/MACOS_VM_MDM.md and docs/MACOS_VM_PROVISIONING_REPAIR_PLAN.md; this file is the narrative of what was tried, what each live run revealed, what was fixed, and — deliberately called out — what was tried and turned out to be wrong.

Status at last entry: enrollment works. Run 25 reached MDM enrollment: Yes (User Approved) with mdmclient completing a TLS handshake against our private-CA server and sending a signed Authenticate (200 OK). That settles the two questions this log carried longest:

  • Certificate pinning works. ServerURLPinningCertificateUUIDs / CheckInURLPinningCertificateUUIDs against a root carried only inside the profile — no system trust required for the check-in.
  • CA trust resolves itself. Post-enrollment the guest reports in System keychain=1, trusted for ssl=yes, for basic=yes, and verified curl returns 400 instead of rc=60. A root delivered by an MDM is auto-trusted, exactly as Apple documents. The five closed routes under run 18 were all attempts to win trust before enrollment — a fight that never needed to be had. The ACME / public-CA fallback is not needed and should not be built.

The one remaining fault is the no-APNs command drain: Authenticate arrives, TokenUpdate never follows, the device never polls /server, and the two queued policy profiles (PPPC/FDA + notification suppression) time out still queued. Mode A never nudged; Mode B always did. Fixed in run 25's change; whether nudging alone suffices is the open question. Answered 2026-07-26 (final entry below): nudging a running daemon can never suffice; the drain works via a cold daemon start or the login-time sync, and the missing TokenUpdate traces to apsd's BAA/SEP identity failing in the VZ guest. See the "Queue-drain repair (third investigation)" section of docs/MACOS_VM_PROVISIONING_REPAIR_PLAN.md.

The approval loop was reading nothing at all. Three consecutive runs recorded 485/553/566 frames with <nothing recognized> on every one, while the saved frames are sharp, fully legible screenshots — try? handler.perform(...) was discarding a Vision failure, so "the recognizer cannot run" and "the screen has no text on it" were the same value. That is what produced the endless Settings-pane cycling and the never-clicked profile row. The error is now surfaced and gated by a self-test; the capture and matching were also hardened. See the last two entries below.


Historical note on -323: it was never a CA-trust problem. Run 22 traced it to our own PKCS#12: macOS's configuration-profile certificate plugin imports identities through SecItemImport (legacy CDSA — nm -u /usr/libexec/mdmclient lists _SecItemImport, not SecPKCS12Import), and that importer accepts only the RFC 7292 v1 algorithms. PKCS12.swift was emitting PBES2/AES-256 with a SHA-256 MAC, so the identity payload failed to parse and the plugin reported the generic certificate error. It is fixed (pbeWithSHAAnd3-KeyTripleDES-CBC + SHA-1 MAC) and gated by tests.

The five closed CA-trust routes under run 18 remain factually correct as observations, but the conclusion drawn from them — "the remaining fix is a certificate from a CA it already trusts" — was wrong, and it cost several runs plus a planned ACME/public-CA detour. Two mechanisms make that whole line of work unnecessary, and run 25 confirmed both live: pinning (ServerURLPinningCertificateUUIDs) carries the check-in without the trust store, and enrollment itself then promotes the root to full system trust.


How this was approached

The scarce resource is a live macOS 27 base build (~40 min for a full build; a few minutes for the MDM-pass-only re-run added in Workstream 0). So the method was:

  1. Instrument first. Before chasing fixes, make one run yield a complete diagnosis. A per-attempt diagnostics bundle lands beside the base at …/macvms/base/Diagnostics/MDM/run-enroll-<id>/: an orchestration transcript.log, per-iteration approval screenshots + recognized OCR text, an approval.json, and (added later) an mdm-server-debug.log.
  2. Fix one confirmed blocker per run. Read the diagnostics, fix the proven cause, avoid speculative bundles of changes. Each run then reveals the next blocker precisely.
  3. Verify off-hardware where possible. The pure pieces (CA, PKCS#12, profile, protocol, classifier geometry) are unit-tested on the host; the enroll script is bash -n'd and emitted via mdm-spike; the profile is linted and round-tripped through security cms -D / plutil.

This worked, but the log below also shows its limit: several fixes were made on plausible inference rather than evidence, and the live guest contradicted them. Those are the reversals.


Summary of live runs

# Symptom Root cause Outcome
1 Toolchain never starts; OS setup completes mdm-enroll.sh timed out Located to the CA-trust step (next)
2 Enroll wedged after /etc/hosts pin security add-trusted-cert blocks on an unanswerable SecurityAgent prompt Made trust step detached + headless attempt
3 Prompt still appears authorizationdb … allow does not suppress it on macOS 27 Answer the prompt via computer-use instead
4 153 frames all unknown while the sheet is visible OCR line-wrapping: "Certificate Trust Settings" spans 3 observations Match context on joined reading-order text
5 Sheet recognized, never dismissed Password-field click landed outside the dialog (maxX+0.18 offset) Click the field's box center
6 Trust sheet answered but field stays empty OCR mangled "Password""Peseword", "Update Settings""Updale Settings" Generic sheet detection, fuzzy field, submit with Return
7 Double-click hits the desktop Click coords fixed at 1920×1200 while the shown monitor window was 1280×800 Scale click by the view's live bounds
8 Enroll review sheet not confirmed Its button is "Enroll", not "Install" Add Enroll as a confirm candidate
9 "Profile installation failed — missing ServerCapabilities" (my reversal) I'd removed a required key Restored ServerCapabilities
10 "The certificate could not be verified" CA can't bootstrap itself from inside the profile it signs Restored add-trusted-cert (detached)
11 Same cert error, after answering two sheets Every issued cert had serial 2 (collision); profile unsigned Random serials; CMS-sign the profile
12 Final "enroll you" sheet stalls Focus race + a single shared retry budget Two-click focus; per-sheet budgets
13 Same cert error again Server trace: no connection ever arrives; keystrokes never land (landed=false) Pointer/click race → move-then-settle-then-press
(design) fixed-resolution click math Latent: works only at 1920×1200 Derive geometry from the captured frame
(design) CLI trust sheet can't be focused at all Root CLI dialog never becomes key Deliver CA as a profile via System Settings (superseded)
14 Root CA profile installs, pass still stalls Two: the loop only asked "am I done?" on .unknown, which an installed row never reaches; and a manually installed root has not been auto-trusted for TLS since Ventura Poll the guest every frame; move trust to the minimal-agent boot
15 Boot 1 stops with CA=0 authorizationdb write … allow silently no-ops on macOS 27 — read back, the rule is unchanged Write /Library/Security/Trust Settings/Admin.plist directly
16 Boot 1 freezes at "trying add-trusted-cert anyway" add-trusted-cert blocks on its SecurityAgent sheet; closing stdin does nothing (it is a GUI dialog, not a terminal read) Command removed for good + every security call bounded
17 Admin.plist written correctly, still CSSMERR_TP_NOT_TRUSTED Either trustd only reads trust settings at start-up (stale verdict), or the path is legacy on macOS 27 Report CAFILE separately from CA; let boot 2's pre-flight decide
18 Pre-flight ssl=0 cms=1 tls=0 on a fresh boot The path is legacy: macOS 27 ignores a hand-written Admin.plist even across a reboot. Local-CA trust is exhausted Stop needing system trust — pin the CA in the profile
19 Row never clicked; manual install → "certificate could not be verified" Two: the .installed hint matched " setting" inside "System Settings"; and the profile's own CMS signature chains to an untrusted CA (no connection was ever made, so not TLS) Whole-line subtitle match + "not installed" veto; stage the profile unsigned
20 PREFLIGHT cms=0 — never reached the install Self-inflicted: run 19 unsigned the profile but the gate still required security cms -D, which cannot decode a plain plist Accept plist or cms; pin the coupling with a plutil -lint test; hold the guest on failure by default
21 GUI fully worked, then ConfigProfilePluginDomain -323 with no connection ever made Not yet known — but proven to be payload validation, so not TLS, not pinning, not the CMS signature (the profile was unsigned and failed identically) Capture log show --info --debug; classify .installFailed first; preserve staged artifacts
22 (no new run — run 21's artifacts + host experiments) Our PKCS#12. The profile plugin imports via legacy SecItemImport, which rejects a SHA-256 MacData (-25264) and a PBES2 key bag; only SecPKCS12Import — the one API we had ever tested against — accepts them Emit pbeWithSHAAnd3-KeyTripleDES-CBC + SHA-1 MAC; assert both importers in tests; probe the identity in the pre-flight; widen the log predicate
23 PREFLIGHT … identity=fail, "Unknown format in import" — build blocked at 14 s The new probe, not the p12. security import picks its parser from the filename extension; the probe wrote to an extensionless mktemp file. The staged blob was correct (pbeWithSHA1And3-KeyTripleDES-CBC, MAC verified) and imported fine one directory over as .p12 Name the temp file identity.p12; classify on exit status when the message is unrecognised, so silence can never fail a build
24 Enroll sheet sat through 9 clicks over 4 min, pointer on the right pixel Single click on a sheet that isn't key is consumed activating it. .authenticate had used two clicks for this exact reason since run 12; .install never did Give .install the same two-click activate-then-press
25 Enrolled (User Approved), then drain timed out with 2 command(s) still queued No APNs ⇒ no TokenUpdate ⇒ the device never polls /server. nudgeMDMCheckIn existed but was wired only into Mode B; Mode A waited passively Nudge on a 25 s cadence from inside waitForDrain, first nudge immediate; add apsd to the log predicate (current)

Run numbers are approximate to the screenshots shared; the reversals (#9, #10, and the profile-root attempt) are the important cautionary entries.


Chronological detail

Workstream 0 — diagnostics (before any fix)

Built the evidence layer so a failed run explains itself:

  • MDMDiagnosticsRecorder writes one directory per attempt: transcript, per-frame screenshots + recognized text, approval.json.
  • Every approval frame records its classified state and whether an action fired, so "the sheet was visible but we did nothing" becomes visible.
  • The MDM server log sink is routed into the transcript; awaitAgentReady failure was re-typed from .appInstallFailed to .agentUnavailable (the old label sent the first investigation to the wrong file); a debug default holds a failed guest running for inspection.
  • revalidateBaseMDMPolicy() re-runs only the MDM pass against an already-built base — the ~3-minute iteration loop that made all subsequent debugging affordable.

Run 1 — enroll times out

Transcript: server bound, then mdm-enroll.sh exit -1 after 60 s, stdout ending at the /etc/hosts pin. The next statement — security add-trusted-cert — wedged. No check-ins ever reached the server.

Run 23 — the trust prompt is unanswerable and unsuppressible

add-trusted-cert into the System keychain raises a SecurityAgent "System Certificate Trust Settings" sheet even as root. First attempt: grant com.apple.trust-settings.admin via security authorizationdb write … allow and run headless. Wrong — the prompt still appeared. So the prompt must be answered by the computer-use surface, which meant backgrounding the trust step so the exec channel returns and the approval loop can run.

Run 4 — the sheet is visible but every frame is unknown

The trust sheet was on screen for 153 frames, all classified unknown. Vision emits one observation per rendered line, so "System Certificate Trust Settings" came back as three lines and a per-line contains("Certificate Trust Settings") matched none. Fix: match context phrases against all lines joined in reading order; keep exact matching only for click targets.

Run 5 — sheet recognized, click misses the field

The field was clicked at box.maxX + 0.18, a heuristic assuming "Password" was a label to the left. On macOS 27 it is the placeholder inside a full-width field, so the offset landed past the dialog's right edge. Fix: click the center of the "Password" box.

Run 6 — field empty despite a correct click

The placeholder stayed visible (field empty), and OCR had read "Password""Peseword" and the button "Update Settings""Updale Settings". Keying on exact labels was the core fragility. Fix: detect any Apple auth sheet by the stable phrase "Enter your password" + a fuzzily located field; submit with Return so the button label is never matched.

Run 7 — double-click lands on the desktop

Clicks were computed in a fixed 1920×1200 space, but when the diagnostic monitor is shown the window is resized to 1280×800 and the VM view autoresizes. Fix: guestPoint scales the event location by the view's live bounds.

Run 8 — the review sheet's button is "Enroll"

The macOS 27 profile-review sheet confirms with Enroll, not "Install". Added it as a confirm candidate.

Run 9 — REVERSAL: ServerCapabilities is required

"Profile installation failed — MDM payload is missing 'ServerCapabilities' key…". Two runs earlier I had removed ServerCapabilities on the theory (old finding #7) that advertising the per-user channel would draw check-ins we don't implement. That was backwards: macOS 27 requires the key. Restored it, and relied on the Workstream-2.2 change (ack unmodeled check-ins with an empty 200) to make the per-user channel harmless.

Run 10 — REVERSAL: the in-profile root can't bootstrap trust

"The certificate could not be verified." I had removed add-trusted-cert and bet on carrying the CA root inside the enrollment profile. Wrong — the identity/check-in TLS is validated before that payload takes effect (exactly docs/MACOS_VM_MDM.md §2.3). Restored add-trusted-cert, detached so the surface can answer its sheet.

Run 11 — duplicate serials + unsigned profile

Both password sheets answered, Enroll clicked, still the cert error. Two real defects:

  • Every issued certificate carried serial 2. signLeaf stamped nextSerial, a let on a value type that never advanced — so the device identity and the TLS leaf shared (issuer, serial), which is exactly how the keychain identifies a cert. This was finding #9 from the very first investigation, flagged and then not fixed for nine runs — the biggest process miss here. Fixed to a random 16-byte serial per leaf.
  • The profile was unsigned ("Nucleic — Not Signed"). Certificate payloads in an unsigned profile are installed without being trusted. CMS-signed the enrollment profile with a CA-issued leaf; verified security cms -D recovers the exact plist.

Run 12 — focus race and a shared retry budget

The whole GUI sequence ran unattended for the first time (trust sheet → Enroll → enrollment sheet), then stalled on the final sheet. Two causes: a SecurityAgent sheet often isn't key, and the first click on a non-key window is consumed activating it (so the caret never lands); and maxAuthAttempts was a single global counter, so retrying the unfocused first sheet burned the budget the last sheet needed. Fixes: two clicks (foreground, then place caret) and per-sheet budgets keyed to each sheet's wording.

Run 13 — the server trace answers it: nothing ever connects

Added a dense, file-backed server trace (MDMDebugLog): listener lifecycle, every accepted connection and its state transitions, raw bytes, parsed requests, signatures, queue movement. Proven against the real server — an untrusted client shows conn FAILED: -9831 unknown Cert Authority, a trusted one shows a full 200 round-trip.

The run's trace: listener LISTENING on 192.168.65.1:8443, then no ACCEPTED line at all — the guest never opened a connection. So this was never a server/TLS fault. The transcript showed "certificate-trust" attempt N: password landed in field=false five times: keystrokes never reached the sheet → CA never trusted → the profile's certs couldn't validate → mdmclient never connected. One root cause presenting as a "certificate" error three layers up. A cropped diagnostic frame confirmed the click point (880,466) was inside the field, but the sheet was inactive and System Settings was frontmost. Cause: the pointer-move and button-press were posted back-to-back, so the guest could press while its pointer was still at the previous location. Fix: post the move, settle ~180 ms, then press. (This also explains the operator's "sometimes works if lucky".)

Design fix — remove the fixed-resolution assumption

Independent latent bug surfaced while reviewing the click path: clickPoint multiplied Vision's normalized box through a hardcoded 1920×1200. Now geometry is measured from the captured frame (imageSize(ofJPEG:)), so clicks stay correct at any panel size. The placeholder matcher was also loosened to password/passcode/pin with word-scaled tolerance.

Run 14 — REVERSAL: a separate root profile is not trusted either

The two-phase flow ran, and the screenshot settled it: the CA root profile installed correctly ("Nucleic MDM Root CA — 1 setting" under Device ▸ Device Management) and the pass still did not proceed. Two independent defects, one in the automation and one in the design:

1. The approval loop could not recognize its own success. installProfileViaComputerUse asked the guest whether it was finished only inside case .unknown. But an installed profile leaves its row on screen, and that row keeps matching rowBoxes() — so the classifier returned .openEnrollmentProfile forever and .unknown was never reached. .installed could not rescue it either: it required the literal OCR text "Installed", and macOS 27 renders an installed profile as its name over a "1 setting" subtitle. The loop re-clicked the row until maxControlClicks, idled to the 300 s deadline, and threw — with the profile sitting installed on screen the whole time. A unit test asserted the wrong wording, which is why this survived review.

Fixed: the guest predicate is polled on a cadence on every frame, whatever the screen shows; .installed no longer returns success on its own but forces an immediate re-ask; the "n settings" subtitle is recognized. Pixels can now only ever say "ask the guest again".

2. The design's premise was false. The run-13 pivot assumed "a profile-delivered root installed at System scope with admin approval is trusted". It is not, and not because of the ordering that run 10 was blamed on. Since macOS Ventura, a root delivered by a manually installed profile is deliberately not auto-trusted for TLS — Apple DTS: "this is a deliberate change to bring macOS more in line with iOS… you have to manually trust a user-installed root certificate." The certificate installs with SSL trust "no value specified". Auto-trust survives only for roots delivered by an MDM. So the separate-profile route could never have worked, and would have failed at the next step with the same "certificate could not be verified" as run 10 — a fourth reversal, caught by reading Apple's documentation rather than by burning another run.

Compounding it, caIsTrusted — the completion predicate for that phase — ran security find-certificate, which proves presence in a keychain, not trust. It would have returned true for exactly the guests that cannot enrol. probeCATrusted had the same flaw while its doc comment claimed to answer precisely the question it did not ask.

Current change — trust in the minimal-agent boot, verified by macOS

CA trust moves one boot earlier, into the minimal-agent pass, and stops being a GUI problem:

  • The com.apple.trust-settings.admin authorization right is relaxed for the length of that boot, security add-trusted-cert -d -r trustRoot runs, and the right is restored. The relaxation is read back before it is relied onauthorizationdb write itself needs system.right.write, so it can silently no-op, which is the most likely reason this route was recorded as "doesn't work" the first time it was tried (nothing in that attempt verified the write took effect).
  • Success is the guest's own evaluation (security verify-cert -p ssl), never a command's exit status. The boot reports CA=1/CA=0 in its STATUS readback and the build fails there, naming the cause, rather than three layers later as an unactionable alert. A root that verifies for basic but not ssl is called out separately — that is the exact Ventura symptom and needs a different fix.
  • The standalone root profile is no longer staged or opened; Phase 1 is gone. Before the enrollment profile is opened, preflightEnrollment asks macOS the three questions that decide the outcome — root trusted for SSL, profile signature verifies, check-in TLS handshake completes — so a broken enrollment is named on the host instead of inferred from a screenshot.
  • enrollmentProfile() no longer falls back to an unsigned plist when signing fails. That fallback was justified by "an unsigned profile still installs", but an unsigned enrollment profile is exactly what macOS refuses with an authentication error, so it could only turn a precise host-side bug into an unattributable guest failure.
  • Diagnostics moved outside the base bundle (<root>/Diagnostics/<bundle>/…). Inside it, they were destroyed by the next clean rebuild or "Delete & reinstall" — so the evidence for run 13 was deleted by the first step taken to investigate it, and this log is all that survived.
  • revalidateBaseMDMPolicy() finally has a caller (Settings ▸ Virtual Machines ▸ ⋯ ▸ Re-run MDM enrollment pass). It was written as the ~3-minute iteration loop the whole repair depends on and had been unreachable from the app the entire time.

Live run pending.

Superseded — CA via System Settings, not the CLI

Even with the pointer-settle fix, the CLI trust sheet is owned by a root process with no GUI app, so it never becomes key and could not be driven. Rather than keep fighting focus on a window macOS will not activate, the CA is now delivered the normal interactive way: a standalone com.apple.security.root configuration profile ("Nucleic MDM Root CA"), installed through System Settings before the enrollment profile. The approval loop was generalized to installProfileViaComputerUse, keyed to a target profile and a completion predicate asked of the guest (not the pixels), so the same proven loop installs the root profile then the enrollment profile. This also sidesteps the run-10 chicken-and-egg: as a separate, earlier install the CA is fully trusted before the enrollment profile is opened.

Unit-verified (root profile unsigned + single-payload, enroll script opens only the root profile, no CLI trust, enrollment profile still CMS-signed). Live run pending.


What is now known to be TRUE (confirmed on the live guest)

  • Declarative first boot + auto-login work; the console user is the agent account.
  • The whole Device Management GUI flow can be driven unattended: pane navigation, profile-row double-click, the "Enroll" review confirmation, and the password sheet (autofill + Return).
  • ServerCapabilities is mandatory on macOS 27.
  • The CA root cannot be trusted from inside the enrollment profile it validates.
  • The MDM server itself is healthy: a CA-trusting client completes a check-in with HTTP 200.
  • The remaining wall is CA trust — specifically, focus/activation of the trust dialog — which is why it was moved to the profile-install flow.

Reversals, kept on the record

  1. Removed ServerCapabilities (it is required).
  2. Made the in-profile root the sole trust mechanism (the CA can't bootstrap itself).
  3. Detached CLI add-trusted-cert and tried to drive its sheet (the dialog can't be focused; superseded by the profile approach).
  4. Delivered the CA as a separate profile installed before the enrollment profile, on the theory that the run-10 failure was one of ordering. It was not: since Ventura no manually installed profile's root is auto-trusted for TLS, so separateness changes nothing. This one was caught by reading Apple's documentation instead of by another live run — which is the cheaper habit.

The lesson, recorded plainly: changes made on inference about macOS behavior were wrong more than once. Finding #9 (duplicate serials) also sat unfixed for nine runs despite being flagged on day one. The durable corrections were (a) always ask the guest, not the screen (completion predicates, reachability and CA-trust probes, keystroke-landing checks), and (b) instrument the layer that is actually silent (the server connection trace).

Run 15 — the authorizationdb recipe does not work on macOS 27

First run of the boot-1 trust step. It failed, and — for the first time in this log — it failed where the fault was, with the evidence already written down. STATUS:

CODE=0 SIP=0 APP=1 LA=1 TCC=0 CA=0

The agent installed fine; only trust failed, and the guest log says exactly how:

✗ CA trust: could not relax com.apple.trust-settings.admin. The rule still reads:
    <array><string>entitled</string><string>authenticate-admin</string></array>
✗ CA trust: the root does not verify at all; no admin-domain trust was recorded.
── security verify-cert -p ssl ──   Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
── security dump-trust-settings -d ── SecTrustSettingsCopyCertificates: No Trust Settings were found.

So security authorizationdb write com.apple.trust-settings.admin allow silently no-ops on macOS 27 — the recipe every CI write-up gives, and the one this log had already recorded as "tried, didn't work" without ever checking why. The read-back is what turned that into a fact: the write returns without complaint and the rule is unchanged. (authorizationdb write needs system.right.write, which root alone does not satisfy.)

Note what this run did not cost: no GUI stall, no 300 s timeout, no guesswork from a screenshot, and the diagnostics survived the failure because they no longer live inside the base bundle. The build stopping at boot 1 is the intended behaviour — an untrusted CA cannot enrol, so continuing would only have converted a named cause into the old unattributable alert.

One gap the run exposed in the step itself: add-trusted-cert was skipped because the relaxation hadn't landed, so it is still unknown whether root alone satisfies the k-of-n 1 rule. It now runs unconditionally (with stdin closed, so a prompt fails fast rather than wedging the boot as the old enroll script did), and the ladder no longer gates any attempt on the previous one's success.

Run 16 — add-trusted-cert wedges the boot, for the third time

Run 15's fix made the trust command run unconditionally (previously it was skipped when the authorization relaxation failed), on the reasoning that the rule is k-of-n 1 and root might satisfy it on its own. It does not. The build stopped dead with the progress line frozen at

right NOT relaxed (write no-opped); trying add-trusted-cert anyway

security add-trusted-cert blocked on its SecurityAgent sheet until the 900 s boot timeout. Closing stdin — which run 15 added specifically to prevent this — does nothing, because the sheet is a GUI authorization dialog, not a terminal read.

That is the third time this one command has wedged the build (runs 2, 3, 16). It is now removed, and a unit test fails the build if it or authorizationdb reappears in the bootstrap. The general rule that came out of this: nothing in the provisioning bootstrap may be able to block indefinitely. The guest has no way to answer a prompt, so an unbounded command converts a diagnosable failure into a silent fifteen-minute stall. Every security invocation now runs under a run_bounded deadline, and the emitted script is linted with bash -n from a unit test — a mis-escaped continuation in a Swift multi-line string otherwise costs a full boot to discover.

Run 17 — the file is written correctly and macOS ignores it (in this boot)

No stall this time — the bounded commands did their job and the boot completed. Admin.plist landed exactly as designed:

-rw-r--r--@ 1 root  wheel  724  /Library/Security/Trust Settings/Admin.plist
  "trustList" => { "91CAAB33236EFADF68665EE15C04F0DF2E624306" => {
      issuerName … modDate … serialNumber 0x01
      trustSettings => [ { "kSecTrustSettingsResult" => 1 } ] } }
  "trustVersion" => 1

…and Security did not see it:

Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
SecTrustSettingsCopyCertificates: No Trust Settings were found.

So the encoder is fine and the write is fine; the question is whether macOS 27 reads that path at all. Two live hypotheses, and they need different answers:

  1. Timing. Trust settings are read when trustd starts. killall trustd may not be equivalent to a boot, and this boot ends in a shutdown anyway — so the verdict taken in boot 1 may simply be stale, and the guest may trust the root the moment it comes back up.
  2. The path is legacy. /Library/Security/Trust Settings/Admin.plist is a decade-old location; if macOS 27 keeps the admin domain elsewhere, the file is inert no matter when it is read.

The next run separates them for free. Boot 1 now reports two facts instead of one — CAFILE (the trust domain is on disk and the CA is in the System keychain, i.e. everything this boot controls) and CA (whether the running system already honours it) — and the build gates on CAFILE, not on a verdict that may be stale. Boot 2's pre-flight, on a freshly booted system, is the authoritative answer, and it fails in seconds before any GUI if trust genuinely did not take.

The run also added a check worth having on its own: whether add-certificates actually imported the CA. A trust setting that references a certificate absent from the keychain is inert, and that failure is indistinguishable from a rejected format without asking.

If hypothesis 2 is the true one, the local-CA approach is finished — profile delivery, the authorization relaxation, add-trusted-cert, and the file will all have been closed — and the answer is a certificate from a CA macOS already trusts.

Superseded — trust as a file, since no API will do it

With every authorization-gated route closed, the remaining one is the file the API would have written: /Library/Security/Trust Settings/Admin.plist. That directory is root:wheel and carries no SIP restricted flag on macOS 27, and trustd reads the admin trust domain from it — so writing it directly establishes the same trust without calling an authorization-gated API at all. Paired with security add-certificates, which imports into the System keychain without prompting.

MDMAdminTrustSettings builds that file host-side from the CA: keyed by the certificate's SHA-1 fingerprint (upper-case hex), with issuerName, serialNumber, modDate, and a single kSecTrustSettingsResult = 1 (ResultTrustRoot) and no policy scoping — the shape add-trusted-cert -r trustRoot produces. The layout is pinned by a unit test, because a wrong key name fails silently in the guest: trustd ignores the entry rather than erroring.

The boot now runs both attempts in order and lets the guest's own verify-cert -p ssl decide, and dumps plutil -p of the installed file alongside dump-trust-settings -d when neither works.

Run 18 — the local CA is finished: Admin.plist is ignored across a reboot

The reboot hypothesis is dead, and this run says so cleanly. Boot 1 wrote the trust domain and passed its CAFILE gate; boot 2 came up on a freshly booted system and the pre-flight reported:

PREFLIGHT ssl=0 cms=1 tls=0
── verify-cert -p ssl ──  Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
── curl (verified) ──     http=000
── curl (insecure) ──     http=400
── /etc/hosts pin ──      192.168.64.1 mdm.nucleic.internal

and the server trace agrees exactly:

conn id=1 ACCEPTED from=192.168.64.72:49152 — FAILED: -9831: unknown Cert Authority
conn id=3 REQUEST GET /checkin → RESPONSE 400        # the --insecure probe

Everything except trust is healthy: the new mdm.nucleic.internal name resolves through the /etc/hosts pin, the route to the vmnet gateway works, and the server answers the moment validation is bypassed. Every route to trusting our own CA on macOS 27 is now closed:

Route Outcome
Root payload in the enrollment profile Not auto-trusted since Ventura (run 10)
Standalone root profile, installed first Same — separateness changes nothing (run 14)
authorizationdb write … allow Silently no-ops; rule unchanged on read-back (run 15)
security add-trusted-cert Blocks forever on an undriveable SecurityAgent sheet (runs 2, 3, 16)
Hand-written /Library/Security/Trust Settings/Admin.plist Written correctly, ignored — before and after a reboot (runs 17, 18)

The remaining answer is the one this log has been circling since run 14: a certificate from a CA macOS already trusts. That fixes the check-in TLS and the profile signature together, with SIP on and no bootstrap to win.

Two secondary findings from the same run, both now fixed:

  • cms=1 while ssl=0 proves security cms -D does not require a trusted chain. It was documented as "proves the profile is signed by a chain this guest can build" — it does not. It is a well-formedness check, and ssl is the only trust verdict.
  • The tls probe was measuring the wrong trust store. macOS's /usr/bin/curl is LibreSSL-backed and validates against /etc/ssl/cert.pem, not the keychain, so it can fail on a chain the system trusts — gating on it would block a working guest for a reason mdmclient does not share. The gate is now ssl + cms, with curl demoted to a reachability signal and nscurl (NSURLSession — the same evaluation mdmclient performs) added as the meaningful end-to-end check.
  • A successful boot 1 preserved no diagnostics at all, so the run whose trust step actually mattered left no record. Its log is now kept on the success path too.

Current change — stop needing system trust: pin the CA in the profile

Eighteen runs were spent trying to get a self-issued root into the macOS trust store. The question was wrong. Apple provides a mechanism for exactly this situation, and the enrollment profile simply never used it:

CheckInURLPinningCertificateUUIDs = [<root payload UUID>]
ServerURLPinningCertificateUUIDs  = [<root payload UUID>]
PinningRevocationCheckRequired    = false

Both keys take the PayloadUUID of a certificate payload carried in the same profile, and name the anchor mdmclient evaluates the check-in and command URLs against — "the UUID of a certificate to use when evaluating trust to the '…/connect/' URLs of MDM servers" (Apple's device-management reference). Available since macOS 10.13, so it has been there the whole time. Pointing them at the root payload the profile already carries means the guest validates our server against the CA it installs from this profile, instead of against a trust store we cannot write to.

The root is pinned rather than the TLS leaf deliberately: the leaf is reissued on every server.start() with a fresh random serial, and a Mode B push reissues it again, so a leaf pin would break the moment the server restarted. PinningRevocationCheckRequired stays at its default false because a private CA publishes no CRL/OCSP responder — requiring a positive revocation answer would fail every connection.

Two consequences elsewhere:

  • The pre-flight had to stop gating on ssl. It required the CA to be in the system trust store — precisely the condition this design no longer needs, so the gate would have rejected the very configuration that works. It now gates on reach (can the guest open a connection at all) and cms (is the profile well-formed), and reports ssl/tls as advisory. No CLI probe can exercise the pinned path — pinning is evaluated by mdmclient against a payload inside the profile — so the pre-flight now checks prerequisites rather than trying to predict the verdict.
  • Boot 1's CA-trust step is no longer a gate. It is kept best-effort (cheap, harmless, and it puts the certificate in the keychain), but its verdict is logged, never enforced.

Live run pending. This is the first change in the series that removes the failing requirement instead of trying to satisfy it.

Run 19 — pinning is not the blocker; the profile's own signature is

First run with certificate pinning. Two independent failures, and the split between them is the useful part.

1. A regression I introduced. The loop never clicked the profile row. Twenty consecutive frames recorded state=installed while the OCR text on those same frames read Nucleic MDM Enrollment ⏐ Protile not instsled. Deuble-elick to review. Cause: the .installed hint added in run 14 matched the bare substring " setting" — which occurs inside "System Settings" in the menu bar of every screen this loop inspects. So the hint fired always, the loop spent its budget re-asking the guest "are you enrolled yet?", and never double-clicked. Fixed by matching the subtitle as a whole line (3 settings) and vetoing on an explicit "not installed", both regression-tested against the exact failing frame.

2. The real finding, from the manual continuation. Driven by hand past the row, the install failed with "Profile installation failed. The certificate could not be verified (authentication error)." The server trace is decisive about what that is not:

03:37:44  conn id=1 ACCEPTED → FAILED: -9831            # pre-flight: curl, verified
03:37:44  conn id=2 ACCEPTED → READY → GET /checkin → 400   # pre-flight: curl -k
(nothing further)

The manual install happened at ~03:40 and produced no connection at all. So the profile was rejected before any network contact, which rules out the check-in TLS — and therefore rules out pinning as the cause. What macOS could not verify was the profile's own CMS signing chain, which terminates at a CA the guest cannot be made to trust.

So the profile is now staged unsigned. That inverts an assumption this log carried for several runs — that "an unsigned enrollment profile is exactly what macOS refuses with an authentication error", which was the stated reason for removing the unsigned fallback. The evidence says the opposite: unsigned is the ordinary case (macOS flags it "Not Signed" and installs, as every self-service .mobileconfig does), while signed-by-an-untrusted-CA is what hard-fails. Signing only helps when the signer chains to something the guest already trusts — and if that were available, none of this subsystem's difficulty would exist. Device authentication is unaffected: check-ins still carry Mdm-Signature, verified host-side against our CA.

Pinning remains unproven either way — nothing reached the point of a check-in. The next run tests it for the first time.

Run 20 — the build failed at its own gate

PREFLIGHT reach=1 cms=0 ssl=0 tls=0
── cms -D ──  security: failed to add data to decoder: UNKNOWN (-8183)

Self-inflicted, and it never reached the install. Run 19 made the enrollment profile unsigned and left the pre-flight gating on security cms -D — a tool that by definition cannot decode a plain plist. Two coupled things were changed and only one was reasoned about, so the gate rejected exactly what the change was meant to produce.

Fixed by accepting either form and reporting which was found (profile=plist / profile=cms), so the check survives the profile changing shape again. The coupling is now covered by a test that runs the real plutil -lint against the actual staged artifact rather than asserting a shape twice.

holdBaseOnMDMFailure now defaults to ON. A failed pass stopped and destroyed its guest immediately, so the screen that could explain the failure was gone before it could be looked at, and an operator wanting to drive the remaining step by hand had nothing left to drive. The base is unusable after a failed pass regardless (the stamp is withheld), so the held VM slot had no competing use.

Note what this run does not tell us: nothing has yet reached the profile install, so both the unsigned-profile change (run 19) and certificate pinning (run 18) remain completely untested.

Observability added before run 21 — no more inference

Runs 1420 were each diagnosed partly by inference, and three of the inferences were wrong. The missing evidence was always the same shape: macOS knew why it failed and nobody asked it. What a run now records, at every milestone and before every failure it can propagate:

  • captureGuestMDMStateprofiles status -type enrollment, the installed-profile inventory, the staged share listing, the frontmost app, and — the load-bearing part — the last 5 minutes of the ManagedClient / mdmclient / profiles / System Settings unified log. macOS records the real reason a profile install failed there (which payload, which validation step); the on-screen alert only ever says "the certificate could not be verified". That log was previously captured only on approval-loop timeout, so the common case — a visible failure alert — recorded nothing. Run 19's cause had to be inferred from the absence of a server connection; it should have been read. Called at: before-install, profile-installed, and every failure path (pre-flight, open-profile, enrollment-approval, queue-drain, command-acknowledgement, enrollment-verification, profile-inventory). Each is read-only, || true-guarded, and bounded at 60 s, so it can never itself fail or hang a pass.
  • .installFailed is now a classified state. The failure alert is matched first, because it sits on top of the Device Management pane and every later rule still matches the row underneath — which is why the loop previously kept "working" against a dead screen for 300 s and then reported a generic timeout. It now stops immediately, captures, and throws an error naming the on-screen text.
  • The staged artifacts are preserved into staged/ beside the transcript on failure. Regenerating a profile to inspect it produces different UUIDs, a different identity, and a different signature — it is not the same file, so it cannot answer questions about the one that failed.
  • FAILED at <phase> is written to the transcript, so the phase is greppable rather than deduced from which step notes are absent.
  • holdBaseOnMDMFailure defaults to ON, so the guest survives the failure it caused.

Run 21 — the GUI works; macOS names the error; the cause is narrowed to one thing

The instrumentation paid for itself. The GUI automation completed for the first time — row double-clicked, Enroll clicked, password landed in field=true — and then macOS rejected the profile. Its own words, from the captured ManagedClient log:

ProfilesSettingsExt [ERROR] [CE] Profile installation
(Nucleic MDM Enrollment (xyz.blakeslee.nucleic.mdm.enroll:8F054B9C-…))
==> Error Domain=ConfigProfilePluginDomain Code=-323
    "The certificate could not be verified (authentication error)."

What this rules out, by evidence rather than inference:

  • Not the profile signature. The pre-flight recorded profile=plist — the profile was unsigned this time and failed identically. Run 19's conclusion that the CMS signature was the cause was wrong.
  • Not the check-in TLS, so not pinning. The server trace shows only the pre-flight's two curl probes and the post-failure reachability probe. mdmclient never opened a connection. The rejection happens during payload validation, before any network contact.
  • Not malformed artifacts, and not clock skew. The preserved staged/ copy was analysed directly: the PKCS#12 MAC verifies (SHA-256, 2048 iterations; shrouded keybag PBES2 / PBKDF2-SHA256 / AES-256-CBC), the device identity is CN=Nucleic macOS Guest with clientAuth EKU and CA:FALSE, its AKI matches the CA's SKI, and openssl verify -CAfile ca.pem dev-cert.pem returns OK. CA notBefore 03:46:23Z, install at 04:49Z — guest and host clocks agree.

So nothing is broken; the chain is correct given the CA as an anchor, and the CA is not a trusted anchor. The remaining question is precisely which payload's certificate macOS refused to verify — and the capture named the error without naming the payload, because log show omits info- and debug-level messages unless asked. That is now fixed (--info --debug, plus a filtered errors-only section), which is the only change made from this run.

Do not iterate this on full builds. revalidateBaseMDMPolicy() re-runs the MDM pass alone against the existing base — minutes, not the ~15 a rebuild costs — and it is reachable at Settings ▸ Virtual Machines ▸ ⋯ ▸ Re-run MDM enrollment pass. Boot 1 already succeeds, so the base satisfies its preconditions.

Run 22 — the answer: our PKCS#12 used algorithms Apple's profile importer cannot read

No new guest run. Run 21's own artifacts (run-enroll-a68d06cb) plus host-side experiments were enough, because the question "which payload?" turned out to be answerable without another boot.

What run 21's evidence already ruled in. The GUI worked (password landed in field=true), macOS authenticated the admin, then rejected the profile 1.0 s later at 21:49:18.996 — and mdm-server-debug.log shows no connection between the pre-flight probe at 21:48:15 and the post-failure probe at 21:49:34. So the rejection is payload validation, before any network contact. That excludes the check-in TLS, pinning, and the CA's trust status in one stroke.

The mechanism. macOS's configuration-profile certificate plugin does not use the modern importer:

nm -u /usr/libexec/mdmclient                          → _SecItemImport
nm -u .../ManagedClient.app/Contents/MacOS/ManagedClient → _SecItemImport

SecItemImport is the legacy CDSA-backed API. (Apple DTS, forum thread 810723: "In macOS 15 we updated the system to work with modern PKCS#12 crypto algorithms" — that landed in SecPKCS12Import; of the older path, "it tends to use CDSA code paths, which means it misses out on a lot of the new stuff.") Measured on macOS 27.0 against the exact bytes staged by run 21:

MacData shrouded key bag SecItemImport SecPKCS12Import
SHA-256 PBES2/AES-256 (ours) -25264 errSecPkcs12VerifyFailure ok
SHA-256 PBE-SHA1-3DES -25264 errSecPkcs12VerifyFailure
SHA-1 PBES2/AES-256 "Unknown format in import" ok
SHA-1 PBE-SHA1-3DES ok — 1 identity ok
SHA-1 unencrypted key bag ok — 1 certificate, key dropped

Independent corroboration: trailofbits/algo#14483 reports this exact -323 string on a root-CA + PKCS#12 profile with CSSM Exception: -25264 MAC verification failed during PKCS12 import in the same log; their fix was adding -legacy to openssl pkcs12.

Why eleven runs missed it. The only Apple API this codebase ever pointed at PKCS12.export output was SecPKCS12Import, in MDMTLSIdentity.serverIdentity() — for the server TLS identity, not the device one. It accepts modern crypto, so mdm-server-debug.log line #11, sec_identity_t created (PKCS#12 round-trip through SecPKCS12Import ok), was a genuine green light for the wrong importer. The device p12 was never parsed by any Apple API host-side, and the pre-flight had no identity probe at all. Compounding it, the guest-state log predicate named System Settings but not ProfilesSettingsExt/ProfileHelper/securityd/trustd, so the CSSM line that names the cause was filtered out — the -323 line survived only because ProfilesSettingsExt happens to log under the ManagedClient subsystem.

Fixed.

  • PKCS12.swift: shrouded key bag → pbeWithSHAAnd3-KeyTripleDES-CBC (PKCS#12 KDF/SHA-1, id=1 key / id=2 IV), MacData → HMAC-SHA-1 (id=3). Cert bag stays plaintext — measured as sufficient. An unencrypted key bag is not an escape hatch: the importer drops the key and yields a bare certificate, which would leave IdentityCertificateUUID resolving to no identity.
  • New TripleDES.swift: swift-crypto ships no 3DES. Of the v1 PBE algorithms only 3DES can also be read back by a stock OpenSSL 3 (RC4/RC2 need the legacy provider), which is what keeps our own interop test runnable. Pinned by the two FIPS 46-3 known-answer vectors.
  • PKCS12Tests: KDF known answers cross-checked against openssl's own MacData and key/IV; the wire OIDs asserted directly (and PBES2/SHA-256 asserted absent); openssl pkcs12 -info -nodes must recover the private key; and — the gate whose absence caused all of this — both Apple importers must return a full identity.
  • preflightEnrollment: extracts the staged payload with plutil and runs it through security import into a throwaway keychain, reporting identity=ok|certonly|fail|skip. fail and certonly are gates: they turn a five-minute GUI failure ending in an opaque alert into an instant, named one. skip (unextractable, e.g. a CMS-signed profile) stays advisory.
  • The guest-state log predicate gained ProfileHelper, ProfilesSettingsExt, securityd, trustd.

Not yet verified end to end. The profiles install bisection on a live macOS 27 guest could not be run — the base image was mid-reprovision — so the proof is at the importer level (through the exact symbol mdmclient links) rather than through the plugin itself. The next real run is the test.

Current change — the OCR itself was the unreliable part (2026-07-25)

Enrollment works, but not repeatably: across runs the same screens produced different classifications, and the failures always traced back to a misread label rather than to a wrong decision made on a correct one. Runs 5, 21 and 25 all recorded the same shape — "Password" → "Peseword", "Update Settings" → "Updale Settings", unknown on a sheet that is plainly on screen. Each was patched locally (fuzzy field match, Return-to-submit); the recognizer and the frames it reads were never examined. They are the cause.

The capture was half the guest's resolution, then JPEG-compressed. makeGraphics gives the guest a 3840×2400 panel at 220 PPI — a 1920×1200-pt desktop rendered @ 2×. The host surface's capture() normalized every grab to 1920×1200 and encoded it at JPEG quality 0.7, because that is the computer-use screenshot contract. For a model looking at a screenshot that is right; for OCR it destroys the signal — macOS body labels render ~13 pt, so they reached Vision ~13 px tall with ringing on every antialiased edge. That is precisely the regime where the recognizer starts substituting glyphs, and it explains why the damage was always isolated characters in short labels.

Worse, the resolution depended on the host's display. bitmapImageRepForCachingDisplay allocates at the view's backing scale, so a 1920×1200-pt view yielded the guest's native pixels only on a Retina Mac; on a 1× host every grab was already a 2:1 downsample before anything else touched it. The pass's reliability was a function of which monitor the developer was plugged into.

Fixed at the source, in three layers.

  • captureForRecognition — a second surface entry point returning the framebuffer at native resolution, losslessly (PNG), with no normalization. capture() is untouched, so the computer-use coordinate contract is unchanged. The off-screen HID window is now sized 3840 / backingScale points, so the grab is exactly the guest's native 3840×2400 on a 1× and a 2× host alike — the host display is out of the equation. Input coordinates are unaffected: guestPoint already mapped the 1920×1200 input space through the view's live bounds.
  • The recognizer is configured for this vocabularyen-US, language correction ON, and customWords from a new approvalLexicon (the words these screens are actually made of). Vision is also asked for its top 3 candidates per line instead of 1, and the reading carrying the most known words wins. This never invents text; it only picks between alternates Vision already computed. (Correction had been disabled to stop it "correcting" UI labels — but with no customWords that merely removed its only defence against glyph confusion, and "Peseword" happened anyway.)
  • Every match in the classifier is fuzzy — this was the real fragility. passwordFieldBox was fuzzy because run 5 forced it to be; everything else still compared exactly (text == "Enroll", contains("Enrollment")). One wrong glyph in a six-letter button meant .unknown, and the loop spent its deadline re-navigating panes past a perfectly good review sheet. All matching now runs through ocrFold (lowercase, merge I/l/1 and O/0, punctuation → separator) plus a bounded edit distance — roughly one damaged glyph in eight.

Wrong turn caught by a test, worth recording. The first version of fuzzyContains was a plain approximate-substring search. It made "Nucleic MDM Enrollment Installed" read as "not installed" for a single insertion — so the negation guard that protects the .installed classification fired on a profile that had installed. Phrase matching is now anchored to word boundaries, which real UI phrases always respect. Fuzziness that swallows the wrong control is worse than no fuzziness.

Two latent bugs fixed on the way. Clicks were scaled by the capture's pixel size, which was only correct while captures happened to equal the input space — with a native-resolution capture that would have doubled every coordinate off the screen; they now map through the fixed surfaceInputSize. And the diagnostics recorder now writes the capture's pixel dimensions into transcript.log (and transcodes frames to bounded JPEGs, so full-resolution evidence doesn't fill the bundle), because a downsampled capture and a changed macOS wording were previously the same line in the log.

Verified: build clean; full suite green. New coverage for mangled button/row text, the word-boundary false positive, fold merges, candidate selection, and the input-space click mapping.

The actual fault — the recognizer was returning nothing, and try? hid it (2026-07-26)

Everything above is real and worth keeping, but it was not why the pass was failing. The reported symptom — "it cycles Privacy & Security forever, never double-clicks the profile, never presses Enroll, never types the password" — was diagnosed from the run artifacts rather than from the code, and they say something much simpler.

run-enroll-4f35f90e/transcript.log, 566 frames:

frame 001 state=unknown capture=1537078B 1280×800px text: <nothing recognized>
frame 002 state=unknown capture=1540834B 1280×800px text: <nothing recognized>
…
approval failed: every approval capture recognized no text — the guest framebuffer appears blank

Every frame. No text, ever. Not one. And frame-001.jpg — the exact bytes handed to the recognizer — is a sharp, fully legible desktop: menu bar, Calendar widget, Weather widget, Dock. Run the same file through the same VNRecognizeTextRequest outside the app and it returns 15 observations in 180 ms. The framebuffer was never the problem; the recognizer was returning [].

It could return [] for only one reason: guard (try? handler.perform([request])) != nil else { return [] }. A recognizer that could not run at all was indistinguishable from a screen with no text on it. Downstream, that is a perfectly coherent machine doing exactly what it should with nothing to go on: every frame classifies .unknown, .unknown cycles deviceManagementPaneCandidates on its 20 s cadence (which is the "Privacy & Security over and over" the user watched), the row is never seen so it is never double-clicked, Enroll is never seen so it is never pressed, the password sheet is never seen so it is never answered. Two earlier runs (968ae158, 822d75b5, 485 and 553 frames) show the identical signature, so this predates the resolution work entirely.

Timing corroborates it independently. Frames are 524 ms apart against a 400 ms sleep — ~124 ms for capture and recognition. .accurate recognition of a 1920×1200 frame measures 170290 ms on this machine on its own. Vision was not running; it was failing fast.

The root cause of the throw is not yet proven — --info/--debug log entries are memory-only and had already aged out. The strongest candidate is Vision's one-time model compile: the first .accurate request in a process with cold models took 45 seconds here (0.3 s on every subsequent run, cached system-wide), and a cold-model failure is consistent with an immediate throw.

What is fixed, regardless of which error it turns out to be:

  • The error is no longer swallowed. recognizeTextLines returns a TextRecognition carrying a failure string; the loop writes it to the transcript once per distinct failure. This is the change that matters most — the missing one cost a live run and an entire investigation.
  • A self-test gates the loop. Before the deadline is even taken, verifyTextRecognition() renders a synthetic image containing known words and requires the recognizer to read them back. If it can't, the pass fails immediately with textRecognitionUnavailable naming the reason, instead of five silent minutes ending in a wrong accusation. It also pays the cold model load off the clock, so the 45 s first-request cost can no longer eat a sixth of the approval budget.
  • The capture is decoded explicitly (CGImageSourceCreateImageAtIndexVNImageRequestHandler(cgImage:)). ImageIO is proven to work in-process — the same capture's dimensions are read through it for the transcript — so this separates "could not decode" from "could not recognize" instead of merging them into one opaque throw, and drops a redundant decode.
  • .fast is tried when .accurate throws. A degraded reading of the Enroll button beats five minutes of .unknown.
  • The lying failure message is gone. "Recognized nothing" no longer claims the framebuffer is blank. It now distinguishes no-capture, captured-but-unreadable (naming the byte count, the pixel dimensions and the recognizer's own error), and read-but-not-understood.
  • landed= cannot be faked by a broken recognizer. The password-verification step inferred "the placeholder is gone, so the text landed" from an empty recognition — which a failing recognizer satisfies trivially. It now requires recognition to have actually succeeded.

The lesson worth keeping. Every prior entry in this file that ends in "the classifier missed the label" assumed the recognizer had run. Nothing asserted it. The new recognizerActuallyReadsTextFromACapture test — synthesize an image with known words, require them back, require the pipeline to classify them as install — takes 0.3 s and would have caught this before a 40-minute base build did.

Verified: build clean; full suite green (1375 + 292 + 74 + 20, up 4). Whether the recognizer now stays healthy across a real base build is the next live run — and if it does not, the transcript will say so in its own words instead of blaming the guest.

Open questions for the next run

  • Does trustd honour a hand-written Admin.plist after a reboot? Answered by run 18: no.
  • Does add-trusted-cert succeed as root without the relaxation? Answered by run 16: no — it hangs on its authorization sheet. The command is removed and test-guarded against return.
  • Which payload's certificate does macOS refuse to verify? Answered by run 22: the com.apple.security.pkcs12 identity — but for none of the three reasons listed here. It was not a chain-trust question at all: SecItemImport could not parse the blob, because the MacData was SHA-256 and the key bag was PBES2. Note that all three candidates presupposed a trust failure, which is why enumerating them did not help; what finally settled it was asking which API does the import (nm -u) and then running the real bytes through it. com.apple.security.scep remains deferred and is not needed — the pre-baked identity is fine once it is encoded in a format Apple can read.
  • Pinning is still unexercised. Answered by run 25: it works. mdmclient completed the TLS handshake and delivered a signed Authenticate against a root carried only inside the profile.
  • If pinning does not work, the fallback is a publicly trusted certificate (ACME). Not needed — do not build it. Pinning carries the check-in, and enrollment then promotes the root to full system trust (trusted for ssl=yes post-install, where every pre-enrollment route had failed). The .internal name can stay.
  • Does the profile's own CMS signature matter? No. Run 19 concluded "yes, and it was fatal"; run 21 refuted that (the profile was unsigned and failed identically, profile=plist), and run 22 found the real cause. Unsigned is the ordinary case: macOS flags "Not Signed" and installs. Apple documents signing as optional and even permits a self-signed signer, so this line of inquiry is closed — it was a wrong inference kept as an answer for two runs.
  • With trust established, does the enrollment profile install in one approval (no CA phase)? Yes — run 25: one row double-click, one Enroll sheet, one password sheet, done.
  • With the pointer-settle fix, does landed=true hold consistently? Yes in runs 21 and 25.
  • Does the no-APNs TokenUpdate drain the queued FDA + notification profiles? No — run 25: it never arrives. Authenticate is delivered and acked, then nothing: no TokenUpdate, no /server poll, both InstallProfile commands still queued at the 5-minute timeout.

The one open question. Is nudging enough to make an enrolled-but-tokenless device fetch commands? Run 25 wired nudgeMDMCheckIn (profiles renew -type enrollment, then a mdmclient.daemon kickstart) into waitForDrain on a 25 s cadence — the same lever Mode B has always used. What to read next run:

  • Does each drain nudge N (M queued) line in the transcript produce a matching conn … ACCEPTED in mdm-server-debug.log? If yes, the device is polling and the queue should move.
  • apsd is now in the log predicate. If it shows APNs registration failing, the missing TokenUpdate is explained, and the question becomes whether macOS will serve commands with no push token at all.
  • If it will not, do not reach for APNs. The two policy profiles are ordinary configuration profiles; they can be installed the same GUI way the enrollment profile now is, with the MDM channel used only for what genuinely needs it. ← WRONG, struck 2026-07-26: PPPC and com.apple.notificationsettings payloads are silently ignored unless delivered by an enrolled MDM (this file's own §2.1 in docs/MACOS_VM_MDM.md). A GUI install of these two profiles can never work; the MDM channel is the only delivery path with SIP on.

The drain, answered — apsd's BAA identity, and the two triggers that work (2026-07-26)

Third investigation, run entirely from the existing evidence plus documentation research; no new build was burned. Three findings close this file's last open question:

  • Why TokenUpdate never comes. The run-25-era capture (run-enroll-968ae158, ~line 1217) had already recorded apsd's own explanation, unread: APSBAAClientIdentityProvider failed to obtain a BAA cert ← MobileActivation "Failed to create reference key"errSecInteractionNotAllowed, <sepk:* kid=0000000000000000>: unable to generate key"Told not to connect after fetching server bag". macOS 27's apsd requires a SEP-backed attestation identity before it will open a courier socket, and that keygen fails in the VZ guest — so it chooses not to connect: zero packets, a rising Consecutive failures counter, and nothing above debug severity. Every network-side diagnosis (reachability, 5223, the bag fetch, certificate chains) was chasing a socket apsd never intended to open. Two candidate causes, discriminated by one solo run — see the repair plan's third-investigation section (concurrent same-MachineIdentifier VMs force an identity rotation per Apple's docs; or a 27 b4 AppleVirtualPlatform regression).
  • The nudges were no-ops, not failures. launchctl kickstart on a running on-demand daemon does nothing, and immediately post-enrollment the daemon is always running — which is why twelve "ok" nudges moved nothing. profiles renew -type enrollment is ADE-only. There is no on-demand poll verb; the protocol drains on a push or on the triggers below.
  • The queue drains without APNs — proven, twice over. (1) run-push-e3d6a047 (00:1500:17): one kickstart of the stopped daemon on the enrolled guest → Idle poll to /server 2 m 13 s later → both InstallProfiles delivered, Acknowledged, and verified installed. The held base guest carries both policy profiles now. (2) Apple documents a login-time blocking Idle/Acknowledged/Error sync — the guest auto-logs-in, so a reboot with the server still listening is a supported, deterministic drain trigger for Mode A.

The fix plan (repair plan, workstreams D.1D.6): re-run the pass solo against the already-enrolled base (it takes the .updatepushProfiles path that just succeeded), replace the nudge cadence with quiet-wait → reboot → longer drainTimeout, enforce a no-other-VMs invariant during the MDM pass, and surface apsd/BAA state on any future drain failure.