Nucleic MDM — a host-side device-management plane for the macOS guests

Status: Mode A is wired into production base recipe v8 and validated live on macOS 27: a guest reaches MDM enrollment: Yes (User Approved) with mdmclient completing a pinned TLS handshake to the local server and delivering a signed Authenticate. Certificate pinning and the pre-baked PKCS#12 identity are both confirmed working; post-enrollment the CA is fully system-trusted. The one step still failing is the no-APNs command drain — TokenUpdate never arrives, so queued InstallProfile commands are not fetched (debug log run 25). The server and Mode A/Mode B plumbing live in the pure-Swift NucleicMDM target, and clones share one logical device identity. Base publication now requires User-Approved enrollment, exact policy acknowledgements/inventory, and a production-shaped functional permission probe. Mode B remains implemented but is not a base-readiness dependency.

Identity mechanism — revised 2026-07-12 (was: SCEP). Implementation surfaced that swift-certificates ships no CMS EnvelopedData and no customizable SignedData (only @_spi(CMS) sign/verify), and there is no PKCS#12 serializer in the graph — so a spec-correct SCEP PKIOperation would be ~2000 lines of hand-rolled ASN.1 that can only be round-trip-tested against my own encoder here (real interop needs a live mdmclient, i.e. a hardware spike). We instead ship a pre-baked PKCS#12 device identity embedded in the enrollment profile (com.apple.security.pkcs12): the host generates the device keypair, the CA issues its cert, and the profile carries the PKCS12+Password. This is also hand-rolled ASN.1 (RFC 7292: a pbeWithSHAAnd3-KeyTripleDES-CBC shrouded key bag plus a SHA-1 PKCS#12 v1 MAC — those obsolete algorithms are mandatory, not a default; see Sources/NucleicMDM/PKCS12.swift and run 22 of the debug log), but it is self-contained and verifiable on-host with openssl pkcs12 / security import, needs no /scep endpoint, and makes Mode A + the notification fix work end to end. SCEP moves to the P0 hardware spike (§8): if a future use case needs in-guest key generation, add a com.apple.security.scep payload + /scep endpoint behind the same MDMDeviceIdentity seam. Trade-off: the device private key travels inside the staged profile (host→guest over virtiofs only, never off-host), which is acceptable for self-owned guests.

1. Why MDM (and why now)

The notification investigation surfaced a wall: on the sealed base SIP is on and there is no MDM, and that combination makes three whole classes of guest configuration unreachable from any shell — even root:

PPPC / TCC grants (e.g. Full Disk Access for NucleicVMAgent) — the system TCC.db is unreadable and unwritable with SIP on, and tccutil can only reset.
Per-app notification policy — no supported CLI; the store is TCC-protected.
Many other privacy, security, and restriction settings.

There are exactly three ways to set those on macOS: (a) SIP off + direct sqlite3/file writes (a per-base recoveryOS csrutil disable step; already scaffolded in provision-macos-guest.sh Phase 7c/d), (b) manual GUI toggles baked into the base, or (c) MDM. Only MDM is declarative, SIP-compatible, and scriptable at scale. Crucially:

MDM-delivered PPPC is honored with SIP enabled. This is the property that makes MDM the right long-term plane: we keep SIP on (the base's current, safer posture) and still grant the agent FDA, disable Tips notifications, etc. — all without touching the notification framework a tester's app relies on.

Honest scope limit: MDM does not retire the SIP-off lane. Screen Recording (kTCCServiceScreenCapture) can never be silently granted by any PPPC payload — Apple deliberately requires a user click (MDM can only allow a standard user to flip the toggle). The in-guest agent's ScreenCaptureKit path therefore still needs either the SIP-off TCC.db write (Phase 7c/d) or a GUI-driven toggle at base-build time. MDM complements those lanes (FDA, notifications, Accessibility pre-approval, restrictions); it does not replace them for Screen Recording.

2. Apple realities to design around

PPPC and com.apple.notificationsettings payloads require MDM. A manually installed .mobileconfig is silently ignored for these payload types. So the two profiles in scripts/macos-notification-fix/ are inert until delivered by an enrolled MDM.
Enrollment itself is a GUI flow, not a CLI or "one click." Since macOS 11, profiles install cannot install an MDM enrollment profile. The real flow the computer-use surface must drive: stage/open the .mobileconfig (queues it as a downloaded profile) → System Settings ▸ General ▸ Device Management ▸ double-click the pending profile → Install… → admin password → the profile installs and the enrollment becomes User-Approved MDM (UAMDM). UAMDM is sufficient for PPPC + notification payloads — full ADE/supervision is not required. This is several deterministic HID steps during the base build, all driveable by the existing MacVMSurfaceHost input path (the same surface that already types the bootstrap launch into Terminal). Verify with profiles status -type enrollment (must report User Approved).
TLS trust is solved by PINNING, not by trusting the CA. The enrollment profile carries CheckInURLPinningCertificateUUIDs and ServerURLPinningCertificateUUIDs, each naming the PayloadUUID of the com.apple.security.root payload in the same profile (macOS 10.13+). So mdmclient evaluates the check-in and command URLs against the CA the profile itself installs, and the guest's system trust store is never consulted. The root is pinned rather than the TLS leaf because the leaf is reissued on every server.start(). PinningRevocationCheckRequired stays false: a private CA publishes no CRL/OCSP responder.

Confirmed live (debug log run 25): mdmclient/1911.0.1 completed the TLS handshake and delivered a signed Authenticate with the CA absent from the system trust store at connect time. And once enrolled the root becomes system-trusted (trusted for ssl=yes), because a root delivered by an MDM is auto-trusted — so pinning is what bootstraps the very first check-in, after which ordinary trust takes over.

This replaced a long and entirely unsuccessful effort to put the root into the system trust store, kept below because the dead ends are worth not repeating — all five routes are closed on macOS 27 (see docs/MACOS_VM_MDM_DEBUG_LOG.md, runs 10–18):
- A root payload in a manually installed profile is not auto-trusted. Deliberate since Ventura ("to bring macOS more in line with iOS" — Apple DTS): the certificate installs, but its SSL trust reads "no value specified". Auto-trust survives only for roots delivered by an MDM — the thing we are enrolling into. This kills carrying the root inside the enrollment profile and installing it as a separate profile first.
- security add-trusted-cert raises an unanswerable SecurityAgent sheet. Since Big Sur an admin-domain trust change needs interactive admin authorization; root is not sufficient. The sheet is owned by a root CLI with no GUI app, never becomes key, and blocks until the boot times out.
- security authorizationdb write com.apple.trust-settings.admin allow silently no-ops. The recipe every CI write-up gives; reading the rule back shows it unchanged (the write itself needs system.right.write).
- A hand-written /Library/Security/Trust Settings/Admin.plist is ignored. The path is not SIP-protected and the file lands correctly, but Security does not read it — dump-trust-settings -d reports an empty domain, before and after a reboot.
Because pinning removes the requirement, preflightEnrollment gates on prerequisites rather than trust: can the guest reach the server at all (curl -k), and does the staged profile decode. It reports the system-trust verdict too, but an ssl=0 is expected and is not a failure.
APNs is the one hard Apple gate — and the drain triggers that actually work without it are now known. (Corrected 2026-07-26 — the original premise here, "mdmclient performs an immediate command poll after TokenUpdate, so pre-queued commands drain in an initial burst with no APNs at all", is false: Apple's protocol drains the command channel on a push, and the NanoMDM/MicroMDM record shows queued commands sit until one arrives. Worse, in VZ guests on macOS 27 beta 4 TokenUpdate itself never happens, because apsd requires a SEP-backed BAA client identity before dialing couriers and that keygen fails in the guest — see the debug log's third investigation.) What is actually available with no push certificate:
- The login-time sync. Apple documents (Managing devices and users in macOS) that during login mdmclient blocks on an Idle/Acknowledged/ Error sequence against the server — a device-initiated drain, no push. The guest auto-logs-in, so rebooting the enrolled guest while the server stays listening is a supported drain trigger. This covers Mode A.
- A cold mdmclient daemon start. Proven live (run-push-e3d6a047): launchctl kickstart of the stopped daemon on an enrolled guest produced an Idle poll ~2 min later that drained and acknowledged both queued InstallProfiles — with no push token in existence. A kickstart of an already-running daemon is a no-op, which is why nudging on a cadence right after enrollment did nothing.
- What can never work: profiles renew -type enrollment (ADE-only / repurposed on macOS 27), mdmclient query verbs (answer locally), and any expectation that the device polls spontaneously after Authenticate.
- A real APNs push additionally requires an MDM push certificate (mdmcert.download → identity.apple.com; the Topic then must be the cert's UID, not our random com.apple.mgmt.External.<uuid>) and a working apsd. Both are optional so long as the reboot/cold-start triggers hold.
Cloned identity — decided: one shared logical device. Every session clone is a block-level copy of one sealed base, so all clones share the base's MDM enrollment identity/cert. The server treats them as one logical device: identical ephemeral machines, identical policy. Mode A needs no runtime MDM contact at all; Mode B nudges are addressed to a specific running VM over its own vsock connection, so there is no ambiguity per interaction. Per-clone re-enrollment was rejected: it is not cheap — re-enrolling means running the GUI approval flow again (there is no headless path, see #2), which is slow and fragile for ephemeral VMs and buys nothing in a single-host, self-owned fleet.

3. How this rides on the existing architecture

Nucleic already has every transport MDM needs:

Host-driven base build over a VirtioFS automount (MacVMEngine+Provision.swift, +Provision27.swift): the first boot stages only nash, the native agent, its LaunchAgent, and account material at /Volumes/My Shared Files/nucleic-provision. Mode A uses a second automounted share for the CA root, enrollment profile, and enroll script. The complete provisioner runs on a third boot only after policy is verified; no pre-policy custom mount is attached.
Host-side computer-use surface (MacVMSurfaceHost, framebuffer + injected HID): → drives the System Settings UAMDM enrollment flow during the build, unattended (it already types the bootstrap launch into Terminal today).
vsock control plane to the in-guest agent (MacVMAgentClient, port 2035; docs/MACOS_VM_NATIVE_AGENT.md §3–4): → the APNs replacement. It reuses the agent's existing streaming exec op (MacVMEngine.run(name:…), passwordless sudo) to make mdmclient connect — no new vsock op, so the signed guest agent needs no rebuild. MacVMEngine.coldStartMDMClient cold-starts the mdmclient daemon (Mode B, and Mode A's cheap first try); Mode A escalates to a guest reboot, whose auto-login session runs a login-time Idle sync against the still-listening server — Apple's documented push-free drain (see MacVMEngine.drainQueueForModeA).
NAT networking (guest → host gateway reachable): → carries the MDM HTTPS protocol itself, which must be TCP/HTTP. The vmnet subnet is not guaranteed stable, and mdmclient rejects an IP-literal ServerURL, so the enrollment profile embeds a name, never an address: the server's TLS cert carries a stable DNS SAN (mdm.nucleic.internal), and the in-guest enroll script discovers its default gateway and pins mdm.nucleic.internal → <gateway> in /etc/hosts (root-writable with SIP on) before enrolling. The name sits under .internal — reserved by ICANN in 2024 for private use and guaranteed never delegated — so a guest that loses its /etc/hosts pin fails to resolve rather than reaching a stranger's server. (It previously sat under .host, a real gTLD Nucleic does not own. .local would be worse still: macOS routes it through mDNSResponder, where /etc/hosts entries are unreliable.) Other private names can share the *.nucleic.internal space.
MacVMBaseStatus flags (provisioned, sipDisabled, agentInstalled, axAgentReady — MacVMSpec.swift): mdmEnrolled and profilesInstalled are written host-side only after acknowledgements and exact guest inventory verification. They, persisted MDM identity, and a functional custom-mount probe gate the recipe stamp and clone admission.

4. Architecture

        ┌─────────────────────────── HOST (NucleicCore, Swift) ────────────────────────────┐
        │  NucleicMDMServer  (NucleicMDM target, #if canImport(Network))                    │
        │   • HTTPS listener (NWListener + TLS; loopback + NAT-facing) speaking:            │
        │       /checkin  — MDM check-in: Authenticate, TokenUpdate, CheckOut               │
        │       /server   — MDM command channel: Idle → command plist → Acknowledged        │
        │   • MDMCertificateAuthority: local RSA CA (persisted per base); issues the TLS    │
        │     server identity (SAN mdm.nucleic.internal) and the device identity cert       │
        │   • PKCS#12 device identity baked into the enrollment profile (no /scep endpoint) │
        │   • Per-device command queue (InstallProfile / RemoveProfile / …)                 │
        │   • NO APNs: commands drain at enrollment; live nudges come via vsock             │
        └───────▲───────────────────────────────────────────────────────────▲──────────────┘
                │ virtiofs (stage CA cert + enrollment profile + enroll.sh)  │ vsock :2035
                │ NAT/HTTPS (check-in/command; mdm.nucleic.internal→gateway) │ mdmclient nudge
        ┌───────┴───────────────────────── GUEST (macOS 27) ─────────────────┴──────────────┐
        │  enroll step (root): trust CA root → pin /etc/hosts → open enrollment profile     │
        │  computer-use surface: System Settings ▸ Device Management ▸ Install ▸ password   │
        │  mdmclient (Apple)  ── installs baked PKCS#12 identity; UAMDM after approval ──    │
        │  NucleicVMAgent     ── exec-channel nudge → mdmclient checks in and polls ──       │
        └───────────────────────────────────────────────────────────────────────────────────┘

Protocol specifics (what the server must actually implement)

Enrollment profile (generated per base by MDMEnrollmentProfile): a PKCS#12 payload (com.apple.security.pkcs12: the host-generated device identity PKCS12 blob + its random Password) + an MDM payload (com.apple.mdm: ServerURL, CheckInURL, Topic = com.apple.mgmt.External.<uuid>, IdentityCertificateUUID → the PKCS#12 payload, AccessRights = 8191, SignMessage = true, CheckOutWhenRemoved = true).
PKCS#12 identity (MDMDeviceIdentity + PKCS12): the CA issues a clientAuth leaf for the host-generated device key; PKCS12 (RFC 7292) wraps key+cert — pbeWithSHAAnd3-KeyTripleDES-CBC shrouded key bag, plaintext cert bag, PKCS#12 v1 MAC (HMAC-SHA-1). Those algorithms are load-bearing: the configuration-profile certificate plugin imports through SecItemImport (legacy CDSA), which rejects PBES2/AES and a SHA-256 MAC — the failure surfaces as ConfigProfilePluginDomain -323 "The certificate could not be verified (authentication error)" and blocked enrollment for eleven runs. PKCS12Tests pins the wire OIDs and asserts both Apple importers accept the blob; preflightEnrollment re-checks it in the guest with security import before the GUI pass. Crypto is cross-platform (swift-crypto HMAC/SHA-1 plus the local TripleDES, since swift-crypto ships no 3DES), ASN.1 via swift-asn1.
Device auth: SignMessage=true makes every check-in/command request carry an Mdm-Signature header (detached CMS over the body, signed with the baked identity). The server verifies it against the local CA — this is the auth boundary instead of TLS client certs (awkward with NWListener).
PPPC identity binding: the checked-in profile is a template. Immediately before Mode A queues it, the engine runs codesign -d -r- against the exact NucleicVMAgent.app staged into this base and rewrites the FDA and Network Volumes CodeRequirement values. This keeps packaged Developer ID builds strict while allowing an ad-hoc development build to authorize its actual installed binary.
Command queue semantics: after TokenUpdate, the device PUTs an Idle status to /server; the server replies with the next queued command plist; the device replies Acknowledged/Error and re-polls; an empty 200 ends the loop. Queue the InstallProfile commands before driving enrollment so the initial burst drains them with no push.

Two operating modes

Mode A — build-time, baked (covers the notification fix; ship first). During the base build: stand up the MDM server, queue InstallProfile for the FDA + notifications profiles, run the in-guest enroll step (trust CA, pin hosts, open profile), drive the System Settings approval via the computer-use surface, let mdmclient drain the queue at enrollment (no APNs), verify (profiles status -type enrollment + installed-profile list → STATUS tokens), then seal. Every clone inherits the installed profiles from the snapshot — no runtime MDM contact required.
Mode B — runtime management of live clones (plumbed now, validated later). For pushing/removing profiles on a running session clone: the host starts the MDM server, queues a command, and sends a vsock mdm_checkin nudge; the agent (passwordless sudo) triggers mdmclient to check in and drain it. The exact nudge command is a spike (§8.2) — the guest op tries a candidate list (mdmclient check-in verbs, launchctl kickstart -k on the mdmclient daemon, profiles renew -type enrollment) and reports which worked. Note the clone's vmnet subnet may differ from build time, so a Mode B session must re-pin /etc/hosts (same gateway-discovery logic, via the vsock exec channel).

5. Integration points (concrete)

Concern	Where
MDM server + CA + PKCS#12 identity + protocol + queue	new `NucleicMDM` target — done
Spike/ops harness (P0, manual validation)	`Sources/mdm-spike` executable — done
Host-side Mode A / Mode B orchestration	`MacVMEngine+MDM.swift` + `MacVMEngine+MDMPolicyPass.swift` — Mode A production-wired; enrollment validated live, command drain outstanding
Guest-side enroll (trust CA, pin hosts, open profile)	staged `mdm-enroll.sh` (generated by `MDMServerController.enrollScript`) — done
Mode A/B drain trigger	`coldStartMDMClient` (cold daemon start) + Mode A reboot→login-sync via `drainQueueForModeA` — done (no new vsock op)
Base status	`MacVMBaseStatus.mdmEnrolled` / `.profilesInstalled` — done
Provisioning call site + bounded UAMDM state machine	recipe-v8 three-pass sequence + `approveEnrollmentViaComputerUse` — enabled fail-closed; hardware validation pending (§8.3)
First payloads	`scripts/macos-notification-fix/*.mobileconfig`

6. First use case — the notification fix

Delivered as Mode A payloads, already authored and plutil-clean:

nucleic-vmagent-fda.mobileconfig — Full Disk Access and Network Volumes for xyz.blakeslee.nucleic.vmagent. Its checked-in Developer ID requirement is a template rewritten at staging to the exact installed agent requirement → removes Data Access Blocked and the VirtioFS permission prompt while keeping SIP on.
nucleic-suppress-notifications.mobileconfig — disables notifications for com.apple.tips and com.apple.BTMNotificationAgent (plus the System Settings/Control Center belt) → removes Tips and Multiple Extensions Added, leaving all other apps' notifications intact.

Plus the non-MDM hygiene already done: the killall NotificationCenter sweeper is removed from provisioning (ineffective + it would dismiss a tester's own notifications), and the base must still be sealed with any pending alert dismissed so the snapshot doesn't replay it.

7. Phased plan

P0 — spike harness (mdm-spike). The smallest standalone driver of the real server: starts NucleicMDMServer with a fresh CA, emits the enrollment profile + CA cert + enroll script to a directory, queues the two profiles, and logs every protocol exchange. Used to validate the load-bearing assumptions (§8) against one throwaway guest — and reusable later as an ops/debug tool.
P1 — Mode A in provisioning (implemented). Minimal agent boot, host MDM server + persisted CA/identity, automounted enrollment staging, bounded UAMDM, exact policy acknowledgement/inventory, full post-policy provisioning, and validation-only production mount probe. Recipe v8 migrates all engine-built bases and refuses incomplete clone sources.
P2 — Mode B runtime. vsock mdm_checkin op; host API to push/remove profiles on a live clone (queue + nudge + re-pin hosts).
P3 — broaden. Declarative Device Management (DDM) status subscriptions, restriction/config payloads, an operator UI for "base policy," optional ADE supervision, optional real APNs if a use case needs Apple-initiated push.

8. Open questions / spikes to validate (on real hardware)

CA trust bootstrap on macOS 27 — does security add-trusted-cert -d … -k /Library/Keychains/System.keychain still work headless as root, or does it raise a SecurityAgent prompt (answerable by the computer-use surface)?
No-APNs enrollment drain on macOS 27 — reliable, or timing-sensitive? (Also: does TokenUpdate complete if APNs is unreachable?)
vsock-triggered mdmclient check-in — which trigger from the candidate list is stable enough to depend on?
UAMDM without a human — the System Settings enrollment flow driven fully by the computer-use surface: exact click/keyboard sequence on macOS 27, and whether the OS gates it further.
SCEP interop — mdmclient's SCEP client against our CMS implementation (CACaps honored? AES chosen? nonce/transactionID echo correct?).
Trust boundary — an MDM that can install arbitrary profiles is a powerful primitive: the CA private key and SCEP challenge live with the base bundle (host-only); the server binds loopback + the vmnet-facing address only, and authenticates devices via Mdm-Signature chained to our CA. Revisit before any use beyond this host's own guests.

9. Alternatives considered

Lane A (SIP-off build): cheapest for the notification fix (FDA = one line in Phase 7d), but requires a per-base recoveryOS csrutil disable, flips the base to SIP-off, and doesn't generalize to declarative policy. (Still required if the in-guest agent needs Screen Recording — see §1's scope limit.)
Lane B (GUI baked into base): zero infra, but manual per-build clicks and no general capability.
Full APNs MDM: the "correct" enterprise form, but the Apple push-cert gate and ongoing cert upkeep buy nothing over the vsock-triggered design for guests we already own end-to-end.
NanoMDM (Go) embedded: battle-tested protocol handling, but adds a Go binary + cross-process lifecycle + its own storage to a Swift-native host app. Rejected in favor of a pure-Swift server modeled on MCPApprovalServer.
Pre-baked PKCS#12 identity instead of SCEP: simpler (no CMS/SCEP), but the device private key would travel inside the staged profile; SCEP keeps key generation in-guest and the profile secret-free. Decided: SCEP.

The chosen path — host-side Swift MDM + SCEP + vsock-as-push, SIP stays on — gets the targeted notification fix and a general management plane, without the Apple push-cert dependency, at the cost of the CMS/SCEP implementation and the §8 spikes.

26 KiB Raw Blame History Unescape Escape