blakeslee-research/gitea-macos-vm-orchestrator
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge nucleic/vivid-glass-urchin-xoym into main

Browse Source
This commit is contained in:
abkslm
2026-08-07 15:41:01 -07:00
parent 6b0c01b74b
commit 902bea5091
11 changed files with 450 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DESIGN.md
+51 -7
View File
@@ -21,8 +21,9 @@ Three constraints shape everything below:
`VZError.virtualMachineLimitExceeded` from `start()`. Concurrency is therefore
2, permanently, and the config value is clamped rather than trusted.
2. **Virtualization needs a GUI session and a signed bundle.** The daemon runs as
a LaunchAgent in a logged-in user session, from inside an ad-hoc-signed `.app`
carrying `com.apple.security.virtualization`.
a LaunchAgent in a logged-in user session, from inside a signed `.app` carrying
`com.apple.security.virtualization` — Developer ID when a certificate is
available, ad-hoc otherwise (see "Verified facts", item 10).
3. **Gitea decides which job a runner claims, not us.** We supply capacity; the
server matches. Trying to pin a specific job to a specific VM would mean
reimplementing Gitea's matching rules, and would be wrong the moment they
@@ -45,7 +46,7 @@ Three constraints shape everything below:
┌──────────────────────────────── Host (Apple Silicon Mac, macOS 26+) ─────────────────────────────┐
│ │
│ LaunchAgent (user session, auto-login, login.keychain unlocked) │
│ └── GiteaMacosRunner.app (ad-hoc signed, com.apple.security.virtualization, LSUIElement) │
│ └── GiteaMacosRunner.app (signed, com.apple.security.virtualization, LSUIElement) │
│ │ │
│ │ NSApplication(.prohibited).run() ── main thread, required by Virtualization │
│ │ │
@@ -382,7 +383,8 @@ disposable and isolated, not on the job being constrained inside it.
and nothing else.
* Networking is **NAT**, not bridged. Guests can reach the LAN and Gitea, but are
not first-class hosts on it. Bridged networking would require the restricted
`com.apple.vm.networking` entitlement, which ad-hoc signing cannot grant — a
`com.apple.vm.networking` entitlement, which needs an Apple-approved
provisioning profile and which ad-hoc signing cannot grant at all — a
constraint that happens to align with what we want anyway.
* **SSH host keys are not verified.** The peer is a VM this process booted
moments ago on a link no other machine shares; pinning would break on every
@@ -527,9 +529,10 @@ timeout. The `--manual-setup` fallback is out of v1 scope (§4).
**10. Headless Virtualization requires an `NSApplication` run loop with
`.prohibited` activation policy, inside a signed `.app` bundle** carrying
`com.apple.security.virtualization`. Ad-hoc signing (`codesign -s -`) suffices.
Bridged networking would additionally need a restricted entitlement; NAT does
not.
`com.apple.security.virtualization`. That entitlement is unrestricted: ad-hoc
signing (`codesign -s -`) grants it, and a Developer ID certificate grants it
with no provisioning profile. Bridged networking would additionally need a
restricted entitlement; NAT does not.
→ *Consequence:* `CommandDaemon` starts `NSApplication` and runs the orchestrator
in a detached `Task`. This applies to **every** command that starts a VM, not
just the daemon: `vm boot`, `image build`, and `image provision` all go through
@@ -550,6 +553,24 @@ downgrade that only surfaces as a failed VM start. And `bundle` must copy
`.app` without them is a working binary with a broken `image build`,
`service install`, and `config init`.
**10a. Which signature is used decides whether the app's code identity is stable
across rebuilds.** A Developer ID signature's designated requirement is anchored
to the team (`… and certificate leaf[subject.OU] = <TEAM_ID>`), so every build
is the same program to macOS. An ad-hoc signature has no anchor, so identity
falls back to the main executable's Mach-O UUID, which the linker regenerates on
essentially every link.
→ *Consequence:* this is not cosmetic, because macOS Local Network privacy is
keyed on exactly that UUID (Fact 16). Under ad-hoc signing a grant is
withdrawn by the next `make install`; under Developer ID it persists. `make sign`
therefore selects a `Developer ID Application` identity matching `TEAM_ID` when
the keychain has one and falls back to ad-hoc with a warning when it does not —
the fallback is required because CI builds inside a throwaway guest with neither
keychain nor certificate. The Developer ID path also passes `--options runtime
--timestamp`, so the bundle is notarizable later without re-signing; notarization
itself is skipped, since it governs distribution to other Macs and this app is
built and installed in place. `Doctor.checkCodeSignature` reports which path was
taken and warns on ad-hoc.
**11. macOS 15+ requires an unlocked `login.keychain` to start a VM.**
→ *Consequence:* the service **must** be a LaunchAgent in the auto-logged-in
user's session, never a LaunchDaemon (which has no session and no unlocked
@@ -589,3 +610,26 @@ changing the MAC address or ECID.**
prohibition collides directly with the per-slot MAC scheme from Fact 12, so
adopting it would require per-slot saved states and a careful look at DHCP lease
reuse — not a drop-in optimization.
**16. macOS 15+ Local Network privacy can block host→guest connections, and
there is no reliable way to grant it interactively here.** Per
[TN3179](https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy)
it is not TCC — the check is a Network Extension packet filter, so it is absent
from `TCC.db`, cannot be queried, cannot be reset, and it "uses your main
executable UUID as part of its implementation". A denial returns `EHOSTUNREACH`
(errno 65), indistinguishable from a genuinely unreachable host. Three things
then conspire against the interactive grant: a LaunchAgent has no UI to show the
prompt in; a run started from a shell is attributed to the **responsible
process**, so the prompt and the System Settings row belong to Terminal rather
than to this app, and granting it to Terminal does not carry to the agent; and
under ad-hoc signing the UUID keying (Fact 10a) withdraws the grant on the next
rebuild.
→ *Consequence:* the deterministic fix is the subnet allowlist
(`com.apple.network.local-network`, keys `AllowedEthernetLocalNetworkAddresses`
and `AllowedWiFiLocalNetworkAddresses`), which is keyed on the network rather
than the app and is read at boot — so it needs a reboot, not a service restart.
`Doctor.checkLocalNetwork` reads those keys and requires coverage of
`192.168.64.0/18`, not a single /24, because the NAT subnet is chosen at runtime
and slides to the next free /24. `SSHExec.localNetworkHint` appends the same
guidance to connection failures, since errno 65 gives the operator nothing to go
on by itself.
docs/setup.md
+80 -19
View File
@@ -162,8 +162,9 @@ SSH server, so the build appears to hang. See
Virtualization.framework refuses to run unless the calling binary carries the
`com.apple.security.virtualization` entitlement, and entitlements are only honoured on a signed
binary inside a proper `.app` bundle. Ad-hoc signing (`codesign -s -`) satisfies this — **no paid
Apple developer account is needed.**
binary inside a proper `.app` bundle. That entitlement is not restricted, so ad-hoc signing
(`codesign -s -`) satisfies it — **the runner works with no Apple developer account.** A Developer
ID certificate buys something different and worth having; see [Code signing](#code-signing) below.
```sh
git clone <this repo> && cd gitea-macos-runner
@@ -176,7 +177,7 @@ make install
| --- | --- |
| `make build` | `swift build -c release --arch arm64` |
| `make bundle` | Assemble `GiteaMacosRunner.app` around the binary: `Contents/MacOS/gitea-macos-runner`, `Contents/Info.plist`, and `Contents/Resources/` (`provision.sh`, `launchd.plist.template`, `config.example.json`) |
| `make sign` | `codesign --sign - --entitlements …` (ad-hoc) and print the resulting entitlements |
| `make sign` | Sign the bundle with the virtualization entitlement — Developer ID when a matching certificate is in the keychain, ad-hoc otherwise — then print the entitlements and the resulting identity |
| `make all` | `build` + `bundle` + `sign`. The default target. |
| `make install` | Runs `all`, copies the app to `~/Applications`, and symlinks the CLI to `/usr/local/bin/gitea-macos-runner` |
| `make dev` | Debug build + bundle + sign, for fast iteration. Does **not** install. |
@@ -192,6 +193,58 @@ code looks in `Contents/Resources` first and only then falls back to
repo-relative paths, so an installed `.app` missing them is a working binary
with three broken commands.
### Code signing
`make sign` picks its identity automatically:
| Keychain state | What you get |
| --- | --- |
| A `Developer ID Application` certificate whose team matches `TEAM_ID` | Developer ID signature, hardened runtime (`--options runtime`), trusted timestamp (`--timestamp`) |
| No matching certificate | Ad-hoc signature (`codesign --sign -`), with a warning |
Both produce a bundle that boots VMs — the virtualization entitlement is not restricted, and needs
no provisioning profile on either path. What differs is **code identity stability**, and that is the
whole reason to prefer Developer ID:
- A Developer ID signature carries a designated requirement anchored to your team
(`… and certificate leaf[subject.OU] = L7UDTQ6F5W`). Every subsequent build satisfies it, so macOS
recognises rebuild after rebuild as *the same program*.
- An ad-hoc signature has no anchor, so the system falls back to the main executable's Mach-O UUID —
which the linker regenerates on essentially every link. Each `make install` presents a program
macOS has never seen before.
The practical consequence is Local Network privacy (§2.6): per
[TN3179](https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy)
the grant "uses your main executable UUID as part of its implementation", so under ad-hoc signing it
is silently withdrawn by the next rebuild. Under Developer ID it survives.
The team is baked into the `Makefile` as a default; override it for your own certificate:
```sh
make install TEAM_ID=ABCDE12345 # your Developer ID team
make install TEAM_ID= # force ad-hoc even if a certificate exists
```
Confirm what actually landed — `doctor`'s `code identity` check reports it, or ask `codesign`:
```sh
codesign -dvv ~/Applications/GiteaMacosRunner.app
# Identifier=xyz.blakeslee.gitea-macos-vm-orchestrator
# CodeDirectory v=20500 … flags=0x10000(runtime)
# Authority=Developer ID Application: Your Name (ABCDE12345)
# TeamIdentifier=ABCDE12345
```
`TeamIdentifier=not set` and `flags=0x2(adhoc)` mean the ad-hoc path was taken.
Two things this deliberately does **not** do. The bundle is not **notarized**: notarization matters
for software distributed to other Macs, where Gatekeeper checks the quarantine bit, and this app is
built and installed in place. `spctl -a` therefore reports `rejected: Unnotarized Developer ID`,
which is expected and does not stop anything here. Signing does require network access for
`--timestamp`, so an offline host falls back to ad-hoc. And the entitlements list stays minimal:
`com.apple.vm.networking` — needed only for bridged networking, and genuinely restricted — is not
requested. See [DESIGN.md](DESIGN.md).
If `/usr/local/bin` isn't writable, `make install` says so and prints the `sudo ln -sf …` command to
run yourself.
@@ -420,25 +473,31 @@ a **warning** when an allowlist exists but does not. Both keys are documented by
[TN3179](https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy)
and are the workaround [Tart's FAQ](https://tart.run/faq/) recommends for the same problem.
**Approving interactively instead.** The app cannot be pre-approved: it appears under **System
Settings → Privacy & Security → Local Network** only *after* it has actually attempted a connection
to a guest. An empty list is expected on a fresh install and does not mean anything is broken. To
create the entry and answer the prompt, run one boot by hand from a Terminal in the GUI session:
**Why not just approve the prompt?** Because on this host there is usually nothing able to show it,
and when something does, it is attributed to the wrong program.
```sh
gitea-macos-runner vm boot --image default
```
An app appears under **System Settings → Privacy & Security → Local Network** only *after* it has
attempted a connection to a guest, so an empty list on a fresh install is expected and means
nothing is broken. It cannot be pre-approved. But the two obvious ways to trigger the prompt both
miss:
and click **Allow**. Do not wait for the LaunchAgent to hit it — a background agent has no way to
answer the prompt.
- **From the LaunchAgent.** A background agent has no UI, so the prompt has nowhere to appear. The
connection is simply denied, and it surfaces as `No route to host` (errno 65) — not as a
permission error.
- **By hand from a Terminal**, e.g. `gitea-macos-runner vm boot --image default`. macOS assigns the
privacy decision to the *responsible process*, and a binary exec'd from a shell is Terminal's
responsibility, not its own. So both the prompt and the Settings row belong to **Terminal**, and
approving it there does not carry over to the LaunchAgent. (If you are hunting for a row that
seems missing, look for Terminal rather than for "Gitea macOS Runner".)
> **Caveat with ad-hoc signing.** An interactive grant is not durable for this project's ad-hoc
> signed bundle. Local Network privacy does not use TCC; per TN3179 it "uses your main executable
> UUID as part of its implementation", and the linker mints a fresh `LC_UUID` on essentially every
> rebuild. So `make install` after a code change is liable to present as a new app that must be
> approved again — and macOS offers no way to reset a Local Network decision back to undetermined,
> so the stale entries accumulate. This is why the allowlist above, which is keyed on the subnet
> rather than on the app, is the recommendation for an unattended machine.
> **And under ad-hoc signing it is not durable anyway.** Local Network privacy does not use TCC; per
> TN3179 it "uses your main executable UUID as part of its implementation", and the linker mints a
> fresh `LC_UUID` on essentially every rebuild. So `make install` after a code change presents as a
> new app that must be approved again — and macOS offers no way to reset a Local Network decision
> back to undetermined, so stale entries accumulate. Signing with a Developer ID certificate fixes
> the churn (see [Code signing](#code-signing)); the subnet allowlist above, keyed on the network
> rather than on the app, sidesteps the whole mechanism and is the recommendation for an unattended
> machine either way.
---
@@ -463,6 +522,7 @@ The checks, in order:
| `host capability` | Apple Silicon, and host macOS ≥ 26 |
| `Virtualization.framework` | `VZVirtualMachine.isSupported` |
| `virtualization entitlement` | `com.apple.security.virtualization` on the *running* executable — this is the check that catches running from `.build/` instead of the signed `.app` |
| `code identity` | The bundle's signature. Passes naming the identifier, team, and hardened runtime; warns on an ad-hoc signature, because that is what makes Local Network grants evaporate on every rebuild ([Code signing](#code-signing)) |
| `configuration` | The config file loads, parses, and passes validation |
| `free disk space` | Free space on the `storeDir` volume against `storage.minFreeDiskGB` |
| `login.keychain unlocked` | `security show-keychain-info login.keychain` |
@@ -470,6 +530,7 @@ The checks, in order:
| `registration token` | A static token resolves, or one can be fetched when `fetchRegistrationTokenViaAPI` is on |
| `runner download url` | The `gitea-runner` release asset is reachable |
| `token file permissions` | Warns — not fails — when a token file is group- or world-readable |
| `guest ssh` | Reachability of the most recent guest lease, when there is one. Warns on a timeout, which is most often a stale 24-hour lease for a guest that is already gone |
| `local network access` | Passes when a subnet allowlist covers `192.168.64.0/18`; warns when an allowlist exists but is scoped too narrowly; otherwise an informational note about the macOS 15+ Local Network prompt (§2.6) |
If the config file is missing or invalid, the host checks still run and the rest
docs/troubleshooting.md
+48 -21
View File
@@ -24,7 +24,7 @@ gitea-macos-runner service status
| VM won't start; entitlement / `com.apple.security.virtualization` error | Running an unsigned binary, or one outside the signed `.app` bundle | `make sign` (or re-run `make install`); invoke the installed bundle, never `.build/release/…` |
| `virtualMachineLimitExceeded` at boot | macOS allows at most **2** concurrent macOS VMs | Set `scheduler.maxConcurrentVMs` ≤ 2; kill stray VMs from earlier runs |
| VM boots but never gets an IP | DHCP lease not yet written, or Local Network privacy denial (macOS 15+) | Check `/var/db/dhcpd_leases`; grant Local Network permission or pre-authorize the subnet |
| Runner not listed under Privacy & Security → Local Network | Expected — the list is populated only after the app first attempts a local connection; it cannot be pre-approved | Boot one VM by hand from a GUI Terminal to create the entry, or (better on CI) allowlist the subnet with `defaults write com.apple.network.local-network` |
| Runner not listed under Privacy & Security → Local Network | Expected — the list is populated only after the app first attempts a local connection, and a run started from a shell is attributed to Terminal, not to the app | Allowlist the subnet with `defaults write com.apple.network.local-network` and reboot; the interactive grant does not reach the LaunchAgent |
| SSH times out on a freshly built image | Guest macOS < 27, so provisioning options were ignored and Setup Assistant is waiting | Rebuild the image from a macOS **27+** IPSW |
| VMs boot in a loop; every teardown says `reason=cancelled` and nothing is logged between the lease and the teardown | `scheduler.bootTimeoutSeconds` is below the guest's *worst-case* boot on a contended host, so each clone is killed while still starting — and each replacement makes the next one slower | Raise `scheduler.bootTimeoutSeconds` (default 900) and reduce the number of concurrent guests; see [The daemon boots VMs forever](#the-daemon-boots-vms-forever-and-every-teardown-says-reasoncancelled) |
| `ssh failed: cannot connect … No route to host) (errno: 65)` part-way through provisioning | macOS 15+ Local Network privacy blocking the app — the grant is keyed on the executable's UUID, so `make install` withdraws it | Allowlist the subnet (`192.168.64.0/18`) and **reboot**; see [SSH fails with "No route to host" mid-run](#ssh-fails-with-no-route-to-host-errno-65-mid-run) |
@@ -63,8 +63,9 @@ codesign -d --entitlements - ~/Applications/GiteaMacosRunner.app # verify
The output must list `com.apple.security.virtualization`. Then confirm the command you're running
resolves to the installed bundle's binary — `which -a gitea-macos-runner` should point at
`~/Applications/GiteaMacosRunner.app/Contents/MacOS/gitea-macos-runner`, not at
`.build/release/gitea-macos-runner`. Ad-hoc signing is sufficient; you do not need a paid developer
account.
`.build/release/gitea-macos-runner`. Ad-hoc signing is sufficient for *this* error; you do not need
a developer account to start VMs. (A Developer ID certificate solves a different problem — Local
Network grants evaporating on rebuild. See [setup.md](setup.md#code-signing).)
---
@@ -120,9 +121,10 @@ prompt**: a LaunchAgent that was never granted permission (or was denied) cannot
see [Local Network: the app is not listed in System Settings](#local-network-the-app-is-not-listed-in-system-settings)
for why the interactive grant is not.
3. Or grant it interactively: run `gitea-macos-runner vm boot --image default` from a Terminal in
the GUI session and click **Allow**. The app is not listed under **System Settings → Privacy &
Security → Local Network** until it has made that first attempt.
Do not reach for the interactive grant instead. Booting a VM by hand from a Terminal makes the
prompt (and the **System Settings → Privacy & Security → Local Network** row) belong to
*Terminal* rather than to this app, and approving it there does not carry over to the
LaunchAgent.
---
@@ -142,17 +144,24 @@ privacy controls, Local Network privacy is not stored in TCC — per
the checks live "deep in the networking stack" as a Network Extension packet filter, so the
permission is absent from `TCC.db` and `tccutil reset` does not apply to it.
**Fix — interactive.** Make the app connect once, from a GUI session where a human can answer:
**Why you will probably never see the app's own row.** macOS assigns a privacy decision to the
*responsible process*, not necessarily to the binary that opened the socket. Launching the runner
the obvious way —
```sh
gitea-macos-runner vm boot --image default
```
Click **Allow**. The entry now exists and can be toggled later. Do not wait for the LaunchAgent to
trigger it; a background agent cannot answer the prompt, so it simply fails to reach the guest.
— execs the bundle's binary from a shell, so the system holds **Terminal** responsible. Both the
prompt and the Settings row belong to Terminal, and allowing it there does **not** carry over to the
LaunchAgent, which is the process that actually needs it. Meanwhile the LaunchAgent itself has no UI
to show a prompt in, so from it the connection is denied outright and surfaces as `No route to host`
(errno 65) rather than as a permission error. Between the two, there is no reliable way to grant
this interactively on an unattended host.
**Fix — deterministic, and what to use on a CI box.** Allowlist the VM subnet instead. It is keyed
on the network rather than on the app, so no prompt is involved and nothing needs redoing:
**Fix — deterministic, and what to use on any host running the LaunchAgent.** Allowlist the VM
subnet instead. It is keyed on the network rather than on the app, so no prompt is involved, and
nothing needs redoing:
```sh
sudo defaults write com.apple.network.local-network \
@@ -165,12 +174,18 @@ Reboot afterwards. `doctor` then reports `local network access` as a **pass**. B
Apple's, documented in TN3179; the same pair is what [Tart's FAQ](https://tart.run/faq/) recommends
for this exact problem on CI hosts.
> **Why the interactive grant does not stick here.** This project ships an **ad-hoc signed** bundle
> (`codesign --sign -`), and TN3179 notes that "local network privacy uses your main executable UUID
> as part of its implementation". The linker writes a new `LC_UUID` on essentially every rebuild, so
> a rebuilt-and-reinstalled runner can read as a *different* program and prompt again — while the
> old row lingers, since macOS provides no way to reset a Local Network decision to undetermined.
> Expect duplicate entries after a few upgrades. The subnet allowlist avoids all of this.
> **And an ad-hoc signed bundle cannot hold the grant anyway.** TN3179 notes that "local network
> privacy uses your main executable UUID as part of its implementation". An ad-hoc signature has no
> team anchor, so that UUID *is* the app's identity — and the linker writes a new `LC_UUID` on
> essentially every rebuild, so a rebuilt-and-reinstalled runner reads as a *different* program and
> prompts again, while the old row lingers (macOS provides no way to reset a Local Network decision
> to undetermined). Expect duplicate entries after a few upgrades.
>
> Signing with a **Developer ID Application** certificate fixes the churn: its designated
> requirement is anchored to your team, so every build is recognised as the same program. `make sign`
> uses one automatically when it is in the keychain — check with `codesign -dvv` or `doctor`'s
> `code identity` line, and see [setup.md](setup.md#code-signing). The subnet allowlist avoids the
> mechanism entirely and is still the right answer for an unattended host.
---
@@ -591,23 +606,35 @@ error: provisioning failed: could not upload provision.sh from …/provision.sh:
ssh failed: cannot connect to 192.168.65.232:22: … No route to host) (errno: 65)
```
**First, check whether it is actually failing.** A single errno 65 at `attempt=1 elapsed=0s`,
seconds after `guest leased address`, is normal and not worth chasing. `waitForSSH` logs its first
probe unconditionally and then heartbeats every 30 s, and that first probe usually lands before the
host has an ARP entry for the guest — which is also `EHOSTUNREACH`. What matters is whether the
message *repeats* at `elapsed=30s`, `60s`, `90s`, … If it does not, boot is proceeding normally. If
it does, read on.
**Cause.** On macOS 15 and newer, an app that has not been granted Local Network access does not get
a "permission denied": the packet filter answers **`EHOSTUNREACH` — errno 65, "No route to host"**,
which is indistinguishable from a guest that is genuinely off the network. Guests here live on a
host-private NAT link that is reachable whenever the VM is up, so on this path errno 65 is far more
often the privacy filter than a routing problem.
host-private NAT link that is reachable whenever the VM is up, so on this path a *persistent* errno
65 is far more often the privacy filter than a routing problem.
Two details make it look intermittent rather than like a permission problem:
- **The grant is keyed on the executable's UUID.** Per
- **Under an ad-hoc signature the grant is keyed on the executable's UUID.** Per
[TN3179](https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy),
Local Network privacy "uses your main executable UUID as part of its implementation", and the
linker mints a fresh `LC_UUID` on essentially every build. A `make install` after a code change
therefore presents a program macOS has never seen, whose permission is undetermined again — even
though the binary you ran ten minutes ago worked.
though the binary you ran ten minutes ago worked. A Developer ID signature anchors identity to the
team instead and does not drift; `doctor`'s `code identity` check tells you which you have, and
[setup.md](setup.md#code-signing) covers switching.
- **Processes started over SSH are exempt.** Running the same command through `ssh you@host …`
succeeds while running it from a GUI Terminal fails. A remote-shell success proves nothing about
the interactive path.
- **A shell-launched run is attributed to Terminal.** macOS charges the privacy decision to the
responsible process, so the app's own identity is not what is being evaluated when you launch it
by hand — and a grant given to Terminal does nothing for the LaunchAgent.
**Fix.** Allowlist the subnet — it is keyed on the network, not on the app, so no rebuild can
withdraw it and no prompt has to be answered: