44 KiB
macOS VM MDM provisioning — debugging log
A chronological record of the effort to make the mandatory MDM enrollment pass of the macOS 27
golden-base build complete unattended. The design and the current specification live in
docs/MACOS_VM_MDM.md and docs/MACOS_VM_PROVISIONING_REPAIR_PLAN.md; this file is the narrative of
what was tried, what each live run revealed, what was fixed, and — deliberately called out — what was
tried and turned out to be wrong.
Status at last entry: the local CA is finished. All five routes to trusting a self-issued root
on macOS 27 have been tried and closed by live evidence (see the table under run 18): both profile
deliveries, the authorization-database relaxation, add-trusted-cert, and a hand-written admin trust
domain — the last ignored even across a full reboot. Everything else in the pipeline is now proven
healthy: boot 1 completes, the mdm.nucleic.internal pin resolves, the guest reaches the server, and
the server answers a --insecure probe with a correct 400. The single remaining fault is that the
guest will not trust our CA, and the remaining fix is a certificate from a CA it already trusts.
How this was approached
The scarce resource is a live macOS 27 base build (~40 min for a full build; a few minutes for the MDM-pass-only re-run added in Workstream 0). So the method was:
- Instrument first. Before chasing fixes, make one run yield a complete diagnosis. A per-attempt
diagnostics bundle lands beside the base at
…/macvms/base/Diagnostics/MDM/run-enroll-<id>/: an orchestrationtranscript.log, per-iteration approval screenshots + recognized OCR text, anapproval.json, and (added later) anmdm-server-debug.log. - Fix one confirmed blocker per run. Read the diagnostics, fix the proven cause, avoid speculative bundles of changes. Each run then reveals the next blocker precisely.
- Verify off-hardware where possible. The pure pieces (CA, PKCS#12, profile, protocol, classifier
geometry) are unit-tested on the host; the enroll script is
bash -n'd and emitted viamdm-spike; the profile is linted and round-tripped throughsecurity cms -D/plutil.
This worked, but the log below also shows its limit: several fixes were made on plausible inference rather than evidence, and the live guest contradicted them. Those are the reversals.
Summary of live runs
| # | Symptom | Root cause | Outcome |
|---|---|---|---|
| 1 | Toolchain never starts; OS setup completes | mdm-enroll.sh timed out |
Located to the CA-trust step (next) |
| 2 | Enroll wedged after /etc/hosts pin |
security add-trusted-cert blocks on an unanswerable SecurityAgent prompt |
Made trust step detached + headless attempt |
| 3 | Prompt still appears | authorizationdb … allow does not suppress it on macOS 27 |
Answer the prompt via computer-use instead |
| 4 | 153 frames all unknown while the sheet is visible |
OCR line-wrapping: "Certificate Trust Settings" spans 3 observations |
Match context on joined reading-order text |
| 5 | Sheet recognized, never dismissed | Password-field click landed outside the dialog (maxX+0.18 offset) |
Click the field's box center |
| 6 | Trust sheet answered but field stays empty | OCR mangled "Password"→"Peseword", "Update Settings"→"Updale Settings" |
Generic sheet detection, fuzzy field, submit with Return |
| 7 | Double-click hits the desktop | Click coords fixed at 1920×1200 while the shown monitor window was 1280×800 | Scale click by the view's live bounds |
| 8 | Enroll review sheet not confirmed | Its button is "Enroll", not "Install" | Add Enroll as a confirm candidate |
| 9 | "Profile installation failed — missing ServerCapabilities" | (my reversal) I'd removed a required key | Restored ServerCapabilities |
| 10 | "The certificate could not be verified" | CA can't bootstrap itself from inside the profile it signs | Restored add-trusted-cert (detached) |
| 11 | Same cert error, after answering two sheets | Every issued cert had serial 2 (collision); profile unsigned | Random serials; CMS-sign the profile |
| 12 | Final "enroll you" sheet stalls | Focus race + a single shared retry budget | Two-click focus; per-sheet budgets |
| 13 | Same cert error again | Server trace: no connection ever arrives; keystrokes never land (landed=false) |
Pointer/click race → move-then-settle-then-press |
| — | (design) fixed-resolution click math | Latent: works only at 1920×1200 | Derive geometry from the captured frame |
| — | (design) CLI trust sheet can't be focused at all | Root CLI dialog never becomes key | Deliver CA as a profile via System Settings (superseded) |
| 14 | Root CA profile installs, pass still stalls | Two: the loop only asked "am I done?" on .unknown, which an installed row never reaches; and a manually installed root has not been auto-trusted for TLS since Ventura |
Poll the guest every frame; move trust to the minimal-agent boot |
| 15 | Boot 1 stops with CA=0 |
authorizationdb write … allow silently no-ops on macOS 27 — read back, the rule is unchanged |
Write /Library/Security/Trust Settings/Admin.plist directly |
| 16 | Boot 1 freezes at "trying add-trusted-cert anyway" | add-trusted-cert blocks on its SecurityAgent sheet; closing stdin does nothing (it is a GUI dialog, not a terminal read) |
Command removed for good + every security call bounded |
| 17 | Admin.plist written correctly, still CSSMERR_TP_NOT_TRUSTED |
Either trustd only reads trust settings at start-up (stale verdict), or the path is legacy on macOS 27 | Report CAFILE separately from CA; let boot 2's pre-flight decide |
| 18 | Pre-flight ssl=0 cms=1 tls=0 on a fresh boot |
The path is legacy: macOS 27 ignores a hand-written Admin.plist even across a reboot. Local-CA trust is exhausted |
Stop needing system trust — pin the CA in the profile |
| 19 | Row never clicked; manual install → "certificate could not be verified" | Two: the .installed hint matched " setting" inside "System Settings"; and the profile's own CMS signature chains to an untrusted CA (no connection was ever made, so not TLS) |
Whole-line subtitle match + "not installed" veto; stage the profile unsigned |
| 20 | PREFLIGHT cms=0 — never reached the install |
Self-inflicted: run 19 unsigned the profile but the gate still required security cms -D, which cannot decode a plain plist |
Accept plist or cms; pin the coupling with a plutil -lint test; hold the guest on failure by default (current) |
Run numbers are approximate to the screenshots shared; the reversals (#9, #10, and the profile-root attempt) are the important cautionary entries.
Chronological detail
Workstream 0 — diagnostics (before any fix)
Built the evidence layer so a failed run explains itself:
MDMDiagnosticsRecorderwrites one directory per attempt: transcript, per-frame screenshots + recognized text,approval.json.- Every approval frame records its classified state and whether an action fired, so "the sheet was visible but we did nothing" becomes visible.
- The MDM server log sink is routed into the transcript;
awaitAgentReadyfailure was re-typed from.appInstallFailedto.agentUnavailable(the old label sent the first investigation to the wrong file); a debug default holds a failed guest running for inspection. revalidateBaseMDMPolicy()re-runs only the MDM pass against an already-built base — the ~3-minute iteration loop that made all subsequent debugging affordable.
Run 1 — enroll times out
Transcript: server bound, then mdm-enroll.sh exit -1 after 60 s, stdout ending at the /etc/hosts
pin. The next statement — security add-trusted-cert — wedged. No check-ins ever reached the server.
Run 2–3 — the trust prompt is unanswerable and unsuppressible
add-trusted-cert into the System keychain raises a SecurityAgent "System Certificate Trust Settings"
sheet even as root. First attempt: grant com.apple.trust-settings.admin via
security authorizationdb write … allow and run headless. Wrong — the prompt still appeared. So
the prompt must be answered by the computer-use surface, which meant backgrounding the trust step so
the exec channel returns and the approval loop can run.
Run 4 — the sheet is visible but every frame is unknown
The trust sheet was on screen for 153 frames, all classified unknown. Vision emits one observation
per rendered line, so "System Certificate Trust Settings" came back as three lines and a per-line
contains("Certificate Trust Settings") matched none. Fix: match context phrases against all lines
joined in reading order; keep exact matching only for click targets.
Run 5 — sheet recognized, click misses the field
The field was clicked at box.maxX + 0.18, a heuristic assuming "Password" was a label to the left.
On macOS 27 it is the placeholder inside a full-width field, so the offset landed past the dialog's
right edge. Fix: click the center of the "Password" box.
Run 6 — field empty despite a correct click
The placeholder stayed visible (field empty), and OCR had read "Password"→"Peseword" and the
button "Update Settings"→"Updale Settings". Keying on exact labels was the core fragility. Fix:
detect any Apple auth sheet by the stable phrase "Enter your password" + a fuzzily located
field; submit with Return so the button label is never matched.
Run 7 — double-click lands on the desktop
Clicks were computed in a fixed 1920×1200 space, but when the diagnostic monitor is shown the window
is resized to 1280×800 and the VM view autoresizes. Fix: guestPoint scales the event location by the
view's live bounds.
Run 8 — the review sheet's button is "Enroll"
The macOS 27 profile-review sheet confirms with Enroll, not "Install". Added it as a confirm candidate.
Run 9 — REVERSAL: ServerCapabilities is required
"Profile installation failed — MDM payload is missing 'ServerCapabilities' key…". Two runs earlier I
had removed ServerCapabilities on the theory (old finding #7) that advertising the per-user
channel would draw check-ins we don't implement. That was backwards: macOS 27 requires the key.
Restored it, and relied on the Workstream-2.2 change (ack unmodeled check-ins with an empty 200) to
make the per-user channel harmless.
Run 10 — REVERSAL: the in-profile root can't bootstrap trust
"The certificate could not be verified." I had removed add-trusted-cert and bet on carrying the CA
root inside the enrollment profile. Wrong — the identity/check-in TLS is validated before that
payload takes effect (exactly docs/MACOS_VM_MDM.md §2.3). Restored add-trusted-cert, detached so
the surface can answer its sheet.
Run 11 — duplicate serials + unsigned profile
Both password sheets answered, Enroll clicked, still the cert error. Two real defects:
- Every issued certificate carried serial 2.
signLeafstampednextSerial, aleton a value type that never advanced — so the device identity and the TLS leaf shared(issuer, serial), which is exactly how the keychain identifies a cert. This was finding #9 from the very first investigation, flagged and then not fixed for nine runs — the biggest process miss here. Fixed to a random 16-byte serial per leaf. - The profile was unsigned ("Nucleic — Not Signed"). Certificate payloads in an unsigned profile
are installed without being trusted. CMS-signed the enrollment profile with a CA-issued leaf;
verified
security cms -Drecovers the exact plist.
Run 12 — focus race and a shared retry budget
The whole GUI sequence ran unattended for the first time (trust sheet → Enroll → enrollment sheet),
then stalled on the final sheet. Two causes: a SecurityAgent sheet often isn't key, and the first
click on a non-key window is consumed activating it (so the caret never lands); and maxAuthAttempts
was a single global counter, so retrying the unfocused first sheet burned the budget the last sheet
needed. Fixes: two clicks (foreground, then place caret) and per-sheet budgets keyed to each
sheet's wording.
Run 13 — the server trace answers it: nothing ever connects
Added a dense, file-backed server trace (MDMDebugLog): listener lifecycle, every accepted
connection and its state transitions, raw bytes, parsed requests, signatures, queue movement. Proven
against the real server — an untrusted client shows conn FAILED: -9831 unknown Cert Authority, a
trusted one shows a full 200 round-trip.
The run's trace: listener LISTENING on 192.168.65.1:8443, then no ACCEPTED line at all — the
guest never opened a connection. So this was never a server/TLS fault. The transcript showed
"certificate-trust" attempt N: password landed in field=false five times: keystrokes never reached
the sheet → CA never trusted → the profile's certs couldn't validate → mdmclient never connected. One
root cause presenting as a "certificate" error three layers up. A cropped diagnostic frame confirmed
the click point (880,466) was inside the field, but the sheet was inactive and System Settings
was frontmost. Cause: the pointer-move and button-press were posted back-to-back, so the guest could
press while its pointer was still at the previous location. Fix: post the move, settle ~180 ms,
then press. (This also explains the operator's "sometimes works if lucky".)
Design fix — remove the fixed-resolution assumption
Independent latent bug surfaced while reviewing the click path: clickPoint multiplied Vision's
normalized box through a hardcoded 1920×1200. Now geometry is measured from the captured frame
(imageSize(ofJPEG:)), so clicks stay correct at any panel size. The placeholder matcher was also
loosened to password/passcode/pin with word-scaled tolerance.
Run 14 — REVERSAL: a separate root profile is not trusted either
The two-phase flow ran, and the screenshot settled it: the CA root profile installed correctly ("Nucleic MDM Root CA — 1 setting" under Device ▸ Device Management) and the pass still did not proceed. Two independent defects, one in the automation and one in the design:
1. The approval loop could not recognize its own success. installProfileViaComputerUse asked the
guest whether it was finished only inside case .unknown. But an installed profile leaves its row
on screen, and that row keeps matching rowBoxes() — so the classifier returned
.openEnrollmentProfile forever and .unknown was never reached. .installed could not rescue it
either: it required the literal OCR text "Installed", and macOS 27 renders an installed profile as
its name over a "1 setting" subtitle. The loop re-clicked the row until maxControlClicks, idled
to the 300 s deadline, and threw — with the profile sitting installed on screen the whole time. A unit
test asserted the wrong wording, which is why this survived review.
Fixed: the guest predicate is polled on a cadence on every frame, whatever the screen shows;
.installed no longer returns success on its own but forces an immediate re-ask; the "n settings"
subtitle is recognized. Pixels can now only ever say "ask the guest again".
2. The design's premise was false. The run-13 pivot assumed "a profile-delivered root installed at System scope with admin approval is trusted". It is not, and not because of the ordering that run 10 was blamed on. Since macOS Ventura, a root delivered by a manually installed profile is deliberately not auto-trusted for TLS — Apple DTS: "this is a deliberate change to bring macOS more in line with iOS… you have to manually trust a user-installed root certificate." The certificate installs with SSL trust "no value specified". Auto-trust survives only for roots delivered by an MDM. So the separate-profile route could never have worked, and would have failed at the next step with the same "certificate could not be verified" as run 10 — a fourth reversal, caught by reading Apple's documentation rather than by burning another run.
Compounding it, caIsTrusted — the completion predicate for that phase — ran
security find-certificate, which proves presence in a keychain, not trust. It would have
returned true for exactly the guests that cannot enrol. probeCATrusted had the same flaw while its
doc comment claimed to answer precisely the question it did not ask.
Current change — trust in the minimal-agent boot, verified by macOS
CA trust moves one boot earlier, into the minimal-agent pass, and stops being a GUI problem:
- The
com.apple.trust-settings.adminauthorization right is relaxed for the length of that boot,security add-trusted-cert -d -r trustRootruns, and the right is restored. The relaxation is read back before it is relied on —authorizationdb writeitself needssystem.right.write, so it can silently no-op, which is the most likely reason this route was recorded as "doesn't work" the first time it was tried (nothing in that attempt verified the write took effect). - Success is the guest's own evaluation (
security verify-cert -p ssl), never a command's exit status. The boot reportsCA=1/CA=0in its STATUS readback and the build fails there, naming the cause, rather than three layers later as an unactionable alert. A root that verifies forbasicbut notsslis called out separately — that is the exact Ventura symptom and needs a different fix. - The standalone root profile is no longer staged or opened; Phase 1 is gone. Before the enrollment
profile is opened,
preflightEnrollmentasks macOS the three questions that decide the outcome — root trusted for SSL, profile signature verifies, check-in TLS handshake completes — so a broken enrollment is named on the host instead of inferred from a screenshot. enrollmentProfile()no longer falls back to an unsigned plist when signing fails. That fallback was justified by "an unsigned profile still installs", but an unsigned enrollment profile is exactly what macOS refuses with an authentication error, so it could only turn a precise host-side bug into an unattributable guest failure.- Diagnostics moved outside the base bundle (
<root>/Diagnostics/<bundle>/…). Inside it, they were destroyed by the next clean rebuild or "Delete & reinstall" — so the evidence for run 13 was deleted by the first step taken to investigate it, and this log is all that survived. revalidateBaseMDMPolicy()finally has a caller (Settings ▸ Virtual Machines ▸ ⋯ ▸ Re-run MDM enrollment pass). It was written as the ~3-minute iteration loop the whole repair depends on and had been unreachable from the app the entire time.
Live run pending.
Superseded — CA via System Settings, not the CLI
Even with the pointer-settle fix, the CLI trust sheet is owned by a root process with no GUI app,
so it never becomes key and could not be driven. Rather than keep fighting focus on a window macOS will
not activate, the CA is now delivered the normal interactive way: a standalone com.apple.security.root
configuration profile ("Nucleic MDM Root CA"), installed through System Settings before the
enrollment profile. The approval loop was generalized to installProfileViaComputerUse, keyed to a
target profile and a completion predicate asked of the guest (not the pixels), so the same proven loop
installs the root profile then the enrollment profile. This also sidesteps the run-10 chicken-and-egg:
as a separate, earlier install the CA is fully trusted before the enrollment profile is opened.
Unit-verified (root profile unsigned + single-payload, enroll script opens only the root profile, no CLI trust, enrollment profile still CMS-signed). Live run pending.
What is now known to be TRUE (confirmed on the live guest)
- Declarative first boot + auto-login work; the console user is the agent account.
- The whole Device Management GUI flow can be driven unattended: pane navigation, profile-row double-click, the "Enroll" review confirmation, and the password sheet (autofill + Return).
ServerCapabilitiesis mandatory on macOS 27.- The CA root cannot be trusted from inside the enrollment profile it validates.
- The MDM server itself is healthy: a CA-trusting client completes a check-in with HTTP 200.
- The remaining wall is CA trust — specifically, focus/activation of the trust dialog — which is why it was moved to the profile-install flow.
Reversals, kept on the record
- Removed
ServerCapabilities(it is required). - Made the in-profile root the sole trust mechanism (the CA can't bootstrap itself).
- Detached CLI
add-trusted-certand tried to drive its sheet (the dialog can't be focused; superseded by the profile approach). - Delivered the CA as a separate profile installed before the enrollment profile, on the theory that the run-10 failure was one of ordering. It was not: since Ventura no manually installed profile's root is auto-trusted for TLS, so separateness changes nothing. This one was caught by reading Apple's documentation instead of by another live run — which is the cheaper habit.
The lesson, recorded plainly: changes made on inference about macOS behavior were wrong more than once. Finding #9 (duplicate serials) also sat unfixed for nine runs despite being flagged on day one. The durable corrections were (a) always ask the guest, not the screen (completion predicates, reachability and CA-trust probes, keystroke-landing checks), and (b) instrument the layer that is actually silent (the server connection trace).
Run 15 — the authorizationdb recipe does not work on macOS 27
First run of the boot-1 trust step. It failed, and — for the first time in this log — it failed where the fault was, with the evidence already written down. STATUS:
CODE=0 SIP=0 APP=1 LA=1 TCC=0 CA=0
The agent installed fine; only trust failed, and the guest log says exactly how:
✗ CA trust: could not relax com.apple.trust-settings.admin. The rule still reads:
<array><string>entitled</string><string>authenticate-admin</string></array>
✗ CA trust: the root does not verify at all; no admin-domain trust was recorded.
── security verify-cert -p ssl ── Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
── security dump-trust-settings -d ── SecTrustSettingsCopyCertificates: No Trust Settings were found.
So security authorizationdb write com.apple.trust-settings.admin allow silently no-ops on macOS
27 — the recipe every CI write-up gives, and the one this log had already recorded as "tried, didn't
work" without ever checking why. The read-back is what turned that into a fact: the write returns
without complaint and the rule is unchanged. (authorizationdb write needs system.right.write,
which root alone does not satisfy.)
Note what this run did not cost: no GUI stall, no 300 s timeout, no guesswork from a screenshot, and the diagnostics survived the failure because they no longer live inside the base bundle. The build stopping at boot 1 is the intended behaviour — an untrusted CA cannot enrol, so continuing would only have converted a named cause into the old unattributable alert.
One gap the run exposed in the step itself: add-trusted-cert was skipped because the relaxation
hadn't landed, so it is still unknown whether root alone satisfies the k-of-n 1 rule. It now runs
unconditionally (with stdin closed, so a prompt fails fast rather than wedging the boot as the old
enroll script did), and the ladder no longer gates any attempt on the previous one's success.
Run 16 — add-trusted-cert wedges the boot, for the third time
Run 15's fix made the trust command run unconditionally (previously it was skipped when the
authorization relaxation failed), on the reasoning that the rule is k-of-n 1 and root might satisfy
it on its own. It does not. The build stopped dead with the progress line frozen at
right NOT relaxed (write no-opped); trying add-trusted-cert anyway
security add-trusted-cert blocked on its SecurityAgent sheet until the 900 s boot timeout. Closing
stdin — which run 15 added specifically to prevent this — does nothing, because the sheet is a GUI
authorization dialog, not a terminal read.
That is the third time this one command has wedged the build (runs 2, 3, 16). It is now removed, and a
unit test fails the build if it or authorizationdb reappears in the bootstrap. The general rule
that came out of this: nothing in the provisioning bootstrap may be able to block indefinitely. The
guest has no way to answer a prompt, so an unbounded command converts a diagnosable failure into a
silent fifteen-minute stall. Every security invocation now runs under a run_bounded deadline, and
the emitted script is linted with bash -n from a unit test — a mis-escaped continuation in a Swift
multi-line string otherwise costs a full boot to discover.
Run 17 — the file is written correctly and macOS ignores it (in this boot)
No stall this time — the bounded commands did their job and the boot completed. Admin.plist landed
exactly as designed:
-rw-r--r--@ 1 root wheel 724 /Library/Security/Trust Settings/Admin.plist
"trustList" => { "91CAAB33236EFADF68665EE15C04F0DF2E624306" => {
issuerName … modDate … serialNumber 0x01
trustSettings => [ { "kSecTrustSettingsResult" => 1 } ] } }
"trustVersion" => 1
…and Security did not see it:
Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
SecTrustSettingsCopyCertificates: No Trust Settings were found.
So the encoder is fine and the write is fine; the question is whether macOS 27 reads that path at all. Two live hypotheses, and they need different answers:
- Timing. Trust settings are read when
trustdstarts.killall trustdmay not be equivalent to a boot, and this boot ends in a shutdown anyway — so the verdict taken in boot 1 may simply be stale, and the guest may trust the root the moment it comes back up. - The path is legacy.
/Library/Security/Trust Settings/Admin.plistis a decade-old location; if macOS 27 keeps the admin domain elsewhere, the file is inert no matter when it is read.
The next run separates them for free. Boot 1 now reports two facts instead of one — CAFILE (the
trust domain is on disk and the CA is in the System keychain, i.e. everything this boot controls)
and CA (whether the running system already honours it) — and the build gates on CAFILE, not on a
verdict that may be stale. Boot 2's pre-flight, on a freshly booted system, is the authoritative
answer, and it fails in seconds before any GUI if trust genuinely did not take.
The run also added a check worth having on its own: whether add-certificates actually imported the
CA. A trust setting that references a certificate absent from the keychain is inert, and that failure
is indistinguishable from a rejected format without asking.
If hypothesis 2 is the true one, the local-CA approach is finished — profile delivery, the
authorization relaxation, add-trusted-cert, and the file will all have been closed — and the answer
is a certificate from a CA macOS already trusts.
Superseded — trust as a file, since no API will do it
With every authorization-gated route closed, the remaining one is the file the API would have written:
/Library/Security/Trust Settings/Admin.plist. That directory is root:wheel and carries no SIP
restricted flag on macOS 27, and trustd reads the admin trust domain from it — so writing it
directly establishes the same trust without calling an authorization-gated API at all. Paired with
security add-certificates, which imports into the System keychain without prompting.
MDMAdminTrustSettings builds that file host-side from the CA: keyed by the certificate's SHA-1
fingerprint (upper-case hex), with issuerName, serialNumber, modDate, and a single
kSecTrustSettingsResult = 1 (ResultTrustRoot) and no policy scoping — the shape
add-trusted-cert -r trustRoot produces. The layout is pinned by a unit test, because a wrong key
name fails silently in the guest: trustd ignores the entry rather than erroring.
The boot now runs both attempts in order and lets the guest's own verify-cert -p ssl decide, and
dumps plutil -p of the installed file alongside dump-trust-settings -d when neither works.
Run 18 — the local CA is finished: Admin.plist is ignored across a reboot
The reboot hypothesis is dead, and this run says so cleanly. Boot 1 wrote the trust domain and passed
its CAFILE gate; boot 2 came up on a freshly booted system and the pre-flight reported:
PREFLIGHT ssl=0 cms=1 tls=0
── verify-cert -p ssl ── Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
── curl (verified) ── http=000
── curl (insecure) ── http=400
── /etc/hosts pin ── 192.168.64.1 mdm.nucleic.internal
and the server trace agrees exactly:
conn id=1 ACCEPTED from=192.168.64.72:49152 — FAILED: -9831: unknown Cert Authority
conn id=3 REQUEST GET /checkin → RESPONSE 400 # the --insecure probe
Everything except trust is healthy: the new mdm.nucleic.internal name resolves through the
/etc/hosts pin, the route to the vmnet gateway works, and the server answers the moment validation
is bypassed. Every route to trusting our own CA on macOS 27 is now closed:
| Route | Outcome |
|---|---|
| Root payload in the enrollment profile | Not auto-trusted since Ventura (run 10) |
| Standalone root profile, installed first | Same — separateness changes nothing (run 14) |
authorizationdb write … allow |
Silently no-ops; rule unchanged on read-back (run 15) |
security add-trusted-cert |
Blocks forever on an undriveable SecurityAgent sheet (runs 2, 3, 16) |
Hand-written /Library/Security/Trust Settings/Admin.plist |
Written correctly, ignored — before and after a reboot (runs 17, 18) |
The remaining answer is the one this log has been circling since run 14: a certificate from a CA macOS already trusts. That fixes the check-in TLS and the profile signature together, with SIP on and no bootstrap to win.
Two secondary findings from the same run, both now fixed:
cms=1whilessl=0provessecurity cms -Ddoes not require a trusted chain. It was documented as "proves the profile is signed by a chain this guest can build" — it does not. It is a well-formedness check, andsslis the only trust verdict.- The
tlsprobe was measuring the wrong trust store. macOS's/usr/bin/curlis LibreSSL-backed and validates against/etc/ssl/cert.pem, not the keychain, so it can fail on a chain the system trusts — gating on it would block a working guest for a reasonmdmclientdoes not share. The gate is nowssl+cms, withcurldemoted to a reachability signal andnscurl(NSURLSession — the same evaluationmdmclientperforms) added as the meaningful end-to-end check. - A successful boot 1 preserved no diagnostics at all, so the run whose trust step actually mattered left no record. Its log is now kept on the success path too.
Current change — stop needing system trust: pin the CA in the profile
Eighteen runs were spent trying to get a self-issued root into the macOS trust store. The question was wrong. Apple provides a mechanism for exactly this situation, and the enrollment profile simply never used it:
CheckInURLPinningCertificateUUIDs = [<root payload UUID>]
ServerURLPinningCertificateUUIDs = [<root payload UUID>]
PinningRevocationCheckRequired = false
Both keys take the PayloadUUID of a certificate payload carried in the same profile, and name the
anchor mdmclient evaluates the check-in and command URLs against — "the UUID of a certificate to use
when evaluating trust to the '…/connect/' URLs of MDM servers" (Apple's device-management reference).
Available since macOS 10.13, so it has been there the whole time. Pointing them at the root
payload the profile already carries means the guest validates our server against the CA it installs
from this profile, instead of against a trust store we cannot write to.
The root is pinned rather than the TLS leaf deliberately: the leaf is reissued on every
server.start() with a fresh random serial, and a Mode B push reissues it again, so a leaf pin would
break the moment the server restarted. PinningRevocationCheckRequired stays at its default false
because a private CA publishes no CRL/OCSP responder — requiring a positive revocation answer would
fail every connection.
Two consequences elsewhere:
- The pre-flight had to stop gating on
ssl. It required the CA to be in the system trust store — precisely the condition this design no longer needs, so the gate would have rejected the very configuration that works. It now gates onreach(can the guest open a connection at all) andcms(is the profile well-formed), and reportsssl/tlsas advisory. No CLI probe can exercise the pinned path — pinning is evaluated bymdmclientagainst a payload inside the profile — so the pre-flight now checks prerequisites rather than trying to predict the verdict. - Boot 1's CA-trust step is no longer a gate. It is kept best-effort (cheap, harmless, and it puts the certificate in the keychain), but its verdict is logged, never enforced.
Live run pending. This is the first change in the series that removes the failing requirement instead of trying to satisfy it.
Run 19 — pinning is not the blocker; the profile's own signature is
First run with certificate pinning. Two independent failures, and the split between them is the useful part.
1. A regression I introduced. The loop never clicked the profile row. Twenty consecutive frames
recorded state=installed while the OCR text on those same frames read
Nucleic MDM Enrollment ⏐ Protile not instsled. Deuble-elick to review. Cause: the .installed hint
added in run 14 matched the bare substring " setting" — which occurs inside "System Settings" in
the menu bar of every screen this loop inspects. So the hint fired always, the loop spent its budget
re-asking the guest "are you enrolled yet?", and never double-clicked. Fixed by matching the subtitle
as a whole line (3 settings) and vetoing on an explicit "not installed", both regression-tested
against the exact failing frame.
2. The real finding, from the manual continuation. Driven by hand past the row, the install failed with "Profile installation failed. The certificate could not be verified (authentication error)." The server trace is decisive about what that is not:
03:37:44 conn id=1 ACCEPTED → FAILED: -9831 # pre-flight: curl, verified
03:37:44 conn id=2 ACCEPTED → READY → GET /checkin → 400 # pre-flight: curl -k
(nothing further)
The manual install happened at ~03:40 and produced no connection at all. So the profile was rejected before any network contact, which rules out the check-in TLS — and therefore rules out pinning as the cause. What macOS could not verify was the profile's own CMS signing chain, which terminates at a CA the guest cannot be made to trust.
So the profile is now staged unsigned. That inverts an assumption this log carried for several
runs — that "an unsigned enrollment profile is exactly what macOS refuses with an authentication
error", which was the stated reason for removing the unsigned fallback. The evidence says the
opposite: unsigned is the ordinary case (macOS flags it "Not Signed" and installs, as every
self-service .mobileconfig does), while signed-by-an-untrusted-CA is what hard-fails. Signing only
helps when the signer chains to something the guest already trusts — and if that were available, none
of this subsystem's difficulty would exist. Device authentication is unaffected: check-ins still carry
Mdm-Signature, verified host-side against our CA.
Pinning remains unproven either way — nothing reached the point of a check-in. The next run tests it for the first time.
Run 20 — the build failed at its own gate
PREFLIGHT reach=1 cms=0 ssl=0 tls=0
── cms -D ── security: failed to add data to decoder: UNKNOWN (-8183)
Self-inflicted, and it never reached the install. Run 19 made the enrollment profile unsigned and
left the pre-flight gating on security cms -D — a tool that by definition cannot decode a plain
plist. Two coupled things were changed and only one was reasoned about, so the gate rejected exactly
what the change was meant to produce.
Fixed by accepting either form and reporting which was found (profile=plist / profile=cms), so the
check survives the profile changing shape again. The coupling is now covered by a test that runs the
real plutil -lint against the actual staged artifact rather than asserting a shape twice.
holdBaseOnMDMFailure now defaults to ON. A failed pass stopped and destroyed its guest
immediately, so the screen that could explain the failure was gone before it could be looked at, and
an operator wanting to drive the remaining step by hand had nothing left to drive. The base is
unusable after a failed pass regardless (the stamp is withheld), so the held VM slot had no competing
use.
Note what this run does not tell us: nothing has yet reached the profile install, so both the unsigned-profile change (run 19) and certificate pinning (run 18) remain completely untested.
Observability added before run 21 — no more inference
Runs 14–20 were each diagnosed partly by inference, and three of the inferences were wrong. The missing evidence was always the same shape: macOS knew why it failed and nobody asked it. What a run now records, at every milestone and before every failure it can propagate:
captureGuestMDMState—profiles status -type enrollment, the installed-profile inventory, the staged share listing, the frontmost app, and — the load-bearing part — the last 5 minutes of theManagedClient/mdmclient/profiles/ System Settings unified log. macOS records the real reason a profile install failed there (which payload, which validation step); the on-screen alert only ever says "the certificate could not be verified". That log was previously captured only on approval-loop timeout, so the common case — a visible failure alert — recorded nothing. Run 19's cause had to be inferred from the absence of a server connection; it should have been read. Called at:before-install,profile-installed, and every failure path (pre-flight,open-profile,enrollment-approval,queue-drain,command-acknowledgement,enrollment-verification,profile-inventory). Each is read-only,|| true-guarded, and bounded at 60 s, so it can never itself fail or hang a pass..installFailedis now a classified state. The failure alert is matched first, because it sits on top of the Device Management pane and every later rule still matches the row underneath — which is why the loop previously kept "working" against a dead screen for 300 s and then reported a generic timeout. It now stops immediately, captures, and throws an error naming the on-screen text.- The staged artifacts are preserved into
staged/beside the transcript on failure. Regenerating a profile to inspect it produces different UUIDs, a different identity, and a different signature — it is not the same file, so it cannot answer questions about the one that failed. FAILED at <phase>is written to the transcript, so the phase is greppable rather than deduced from which step notes are absent.holdBaseOnMDMFailuredefaults to ON, so the guest survives the failure it caused.
Run 21 — the GUI works; macOS names the error; the cause is narrowed to one thing
The instrumentation paid for itself. The GUI automation completed for the first time — row
double-clicked, Enroll clicked, password landed in field=true — and then macOS rejected the
profile. Its own words, from the captured ManagedClient log:
ProfilesSettingsExt [ERROR] [CE] Profile installation
(Nucleic MDM Enrollment (xyz.blakeslee.nucleic.mdm.enroll:8F054B9C-…))
==> Error Domain=ConfigProfilePluginDomain Code=-323
"The certificate could not be verified (authentication error)."
What this rules out, by evidence rather than inference:
- Not the profile signature. The pre-flight recorded
profile=plist— the profile was unsigned this time and failed identically. Run 19's conclusion that the CMS signature was the cause was wrong. - Not the check-in TLS, so not pinning. The server trace shows only the pre-flight's two curl
probes and the post-failure reachability probe.
mdmclientnever opened a connection. The rejection happens during payload validation, before any network contact. - Not malformed artifacts, and not clock skew. The preserved
staged/copy was analysed directly: the PKCS#12 MAC verifies (SHA-256, 2048 iterations; shrouded keybag PBES2 / PBKDF2-SHA256 / AES-256-CBC), the device identity isCN=Nucleic macOS GuestwithclientAuthEKU andCA:FALSE, its AKI matches the CA's SKI, andopenssl verify -CAfile ca.pem dev-cert.pemreturns OK. CAnotBefore03:46:23Z, install at 04:49Z — guest and host clocks agree.
So nothing is broken; the chain is correct given the CA as an anchor, and the CA is not a trusted
anchor. The remaining question is precisely which payload's certificate macOS refused to verify —
and the capture named the error without naming the payload, because log show omits info- and
debug-level messages unless asked. That is now fixed (--info --debug, plus a filtered
errors-only section), which is the only change made from this run.
Do not iterate this on full builds. revalidateBaseMDMPolicy() re-runs the MDM pass alone against
the existing base — minutes, not the ~15 a rebuild costs — and it is reachable at
Settings ▸ Virtual Machines ▸ ⋯ ▸ Re-run MDM enrollment pass. Boot 1 already succeeds, so the base
satisfies its preconditions.
Open questions for the next run
-
Does trustd honour a hand-writtenAnswered by run 18: no.Admin.plistafter a reboot? -
DoesAnswered by run 16: no — it hangs on its authorization sheet. The command is removed and test-guarded against return.add-trusted-certsucceed as root without the relaxation? -
Which payload's certificate does macOS refuse to verify? The single open question. The verbose ManagedClient capture should name it. The candidates, and what each would imply:
- the
com.apple.security.pkcs12identity — installing a pre-baked identity whose chain ends at an untrusted anchor. If so, the documented alternative is acom.apple.security.sceppayload plus a/scependpoint, which is how real MDMs issue an identity from a private CA (deferred atdocs/MACOS_VM_MDM.md§"Identity mechanism" precisely because it is ~2000 lines of ASN.1). - the
com.apple.security.rootpayload — macOS refusing to install a root from a profile in this context at all. - the
com.apple.mdmpayload validatingIdentityCertificateUUIDagainst a trusted anchor.
Do not guess between these. The log will say.
- the
-
Pinning is still unexercised — nothing has reached a check-in — so it remains neither proven nor disproven.
-
If pinning does not work, the fallback is a publicly trusted certificate for a real name (ACME through
cloud/nucleic-edge) — but note.internalcould not be used for that, since no public CA issues for a private-use TLD, so the MDM name would have to move to a real zone. -
Does the profile's own CMS signature matter?Answered by run 19: yes, and it was fatal. The profile is now unsigned. Watch for it installing after a "Not Signed" warning rather than failing. -
With trust established, does the enrollment profile install in one approval (no CA phase)?
-
With the pointer-settle fix, does
landed=truehold consistently for the profile-install password sheet? -
Once the guest connects: does the no-APNs
TokenUpdatedrain the queued FDA + notification profiles (the last unexercised step)?