blakeslee-research/gitea-macos-vm-orchestrator
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
11019d498b0c29b7b970d8fb92f2e71913e59cd7
gitea-macos-vm-orchestrator/docs/troubleshooting.md
T

13 KiB
Raw Blame History

Troubleshooting

Start with gitea-macos-runner doctor — it catches most misconfiguration before you go symptom-hunting. Then find your symptom below.

Useful log commands throughout:

# Daemon logs, live
log stream --predicate 'process == "gitea-macos-runner"' --info

# Daemon logs, last hour
log show --predicate 'process == "gitea-macos-runner"' --info --last 1h

gitea-macos-runner service status

Quick reference

Symptom Cause Fix
VM won't start; entitlement / com.apple.security.virtualization error Running an unsigned binary, or one outside the signed .app bundle make sign (or re-run make install); invoke the installed bundle, never .build/release/…
virtualMachineLimitExceeded at boot macOS allows at most 2 concurrent macOS VMs Set scheduler.maxConcurrentVMs ≤ 2; kill stray VMs from earlier runs
VM boots but never gets an IP DHCP lease not yet written, or Local Network privacy denial (macOS 15+) Check /var/db/dhcpd_leases; grant Local Network permission or pre-authorize the subnet
SSH times out on a freshly built image Guest macOS < 27, so provisioning options were ignored and Setup Assistant is waiting Rebuild the image from a macOS 27+ IPSW
SecKeyCreateRandomKey / "Interaction is not allowed" login.keychain is locked — no GUI session Run as a LaunchAgent in an unlocked GUI session; enable auto-login
Job stays queued forever Label mismatch, or the daemon isn't running/reaching Gitea Use bare label names in runs-on; match runner.labels; check daemon logs
actions/checkout fails instantly Node.js missing from the guest image gitea-macos-runner image provision <name>
Runner rows piling up in the Gitea UI VMs killed uncleanly; registrations orphaned Reconcile loop cleans them; force it by restarting the daemon; delete manually if needed
Disk filling up Copy-on-write clones grow as jobs write Raise storage.minFreeDiskGB; delete stale clones in storeDir/vms
image build appears to hang during install Normal — macOS install is slow Wait. Do not stop the VM mid-install; if you did, delete the image and rebuild

VM won't start — entitlement error

Symptom. Any VM operation fails immediately with an error naming com.apple.security.virtualization, or a generic "operation not permitted" from Virtualization.framework.

Cause. Virtualization.framework checks the entitlement on the calling binary, and entitlements are only honoured on a signed binary inside a proper .app bundle. The raw product of swift build has neither.

Fix.

make install       # build → bundle → sign → install (make sign alone re-signs in place)
codesign -d --entitlements - ~/Applications/GiteaMacosRunner.app   # verify

The output must list com.apple.security.virtualization. Then confirm the command you're running resolves to the installed bundle's binary — which -a gitea-macos-runner should point at ~/Applications/GiteaMacosRunner.app/Contents/MacOS/gitea-macos-runner, not at .build/release/gitea-macos-runner. Ad-hoc signing is sufficient; you do not need a paid developer account.

virtualMachineLimitExceeded

Symptom. The first VM boots fine; a second or third fails with virtualMachineLimitExceeded.

Cause. macOS permits two concurrent macOS guests per host. This is an Apple kernel and licensing limit, not a resource constraint — more RAM will not raise it.

Fix. Set scheduler.maxConcurrentVMs to 2 or less. If you're already at 2 and still hitting the limit, a VM from a previous run is still alive — check for stray processes and for leftover directories under storeDir/vms, then restart the daemon so it starts from a clean state.

To handle more macOS jobs in parallel, add another Mac.

VM starts but never gets an IP

Symptom. The VM boots (you can see it progress if you use vm boot), but the daemon reports that it could not resolve the guest address, or gives up at scheduler.bootTimeoutSeconds.

Cause. The daemon resolves the guest's NAT address from the host's DHCP lease file, which is only written once the guest requests a lease — several seconds after the boot screen appears. If the address never appears at all, the usual culprit on macOS 15+ is the Local Network privacy prompt: a LaunchAgent that was never granted permission (or was denied) cannot talk to the guest.

Fix.

  1. Check the lease file after the guest has been up for ~30 seconds:

    cat /var/db/dhcpd_leases

    An entry with a recent lease timestamp and the guest's MAC means networking is fine and the problem is timing — raise scheduler.bootTimeoutSeconds.

  2. No entry at all: check System Settings → Privacy & Security → Local Network and enable the runner. If it isn't listed, trigger the prompt interactively from a Terminal in the GUI session:

    gitea-macos-runner vm boot

    and click Allow.

  3. Or pre-authorize the VM subnet, then reboot:

    sudo defaults write com.apple.network.local-network \
  AllowedEthernetLocalNetworkAddresses -array "192.168.0.0/16"

    Match the range to what your host's NAT actually hands out.

SSH times out on a freshly built image

Symptom. The VM boots and gets an IP, but SSH never connects. Attaching a display to the guest shows Setup Assistant — the region/Apple ID welcome flow — rather than a login window.

Cause. The image builder uses VZMacGuestProvisioningOptions to create the admin account, enable Remote Login, and skip Setup Assistant. That API requires macOS 27 or newer in the guest as well as the host. An older guest silently ignores the options: no error, no account, no SSH server — it just sits at first-run setup forever.

Fix. Rebuild the base image from a macOS 27+ IPSW:

gitea-macos-runner image delete default
gitea-macos-runner image build --ipsw ~/Downloads/UniversalMac_27.0_XXXXX_Restore.ipsw

Verify the host is also 27+ (sw_vers). There is no way to make a pre-27 guest work unattended with this builder.

SecKeyCreateRandomKey / "Interaction is not allowed"

Symptom. The daemon starts but fails during VM setup with a Security-framework error mentioning SecKeyCreateRandomKey, errSecInteractionNotAllowed, or "Interaction is not allowed". Often it works when you run the daemon by hand in Terminal and fails under launchd.

Cause. The login.keychain is locked. macOS 15+ requires it unlocked for key operations the VM lifecycle performs, and it is only unlocked inside a live, logged-in GUI session. A LaunchDaemon, an SSH-only session, or a Mac sitting at the login window all fail this.

Fix.

  1. Confirm the service is installed as a LaunchAgent, not a LaunchDaemon: gitea-macos-runner service install does the right thing; a hand-written plist in /Library/LaunchDaemons does not.

  2. Ensure the runner user is actually logged in with the desktop loaded. Enable auto-login: System Settings → Users & Groups → Automatically log in as.

  3. Prevent the machine from returning to a locked state:

    sudo pmset -a sleep 0 disablesleep 1

    and disable "Require password after screen saver begins" for the runner user.

Connecting over Screen Sharing to a Mac at the login window does not unlock login.keychain for launchd's session — auto-login is the reliable answer on a dedicated CI Mac.

Job stays queued and no VM boots

Symptom. The workflow shows as queued in Gitea indefinitely. Nothing appears in the daemon logs about it.

Causes and fixes, in the order worth checking:

  1. Label mismatch. runs-on must use bare label names (macos-arm64), and every label listed must appear in the host config's runner.labels. The :host suffix used at registration is runner-side only and must never appear in workflow YAML. A single typo produces exactly this symptom with no error anywhere.

  2. Daemon not running or not polling.

    gitea-macos-runner service status
log show --predicate 'process == "gitea-macos-runner"' --info --last 15m

    You should see a poll every scheduler.pollIntervalSeconds.

  3. Gitea too old. The queued-jobs API with the labels field requires Gitea ≥ 1.25. On an older instance the daemon can never see jobs. doctor reports the server version.

  4. Admin PAT wrong or under-scoped. The token must belong to a site admin and carry read:admin + write:admin. Test it:

    curl -H "Authorization: token $TOKEN" \
  "https://gitea.example.com/api/v1/admin/actions/jobs?status=queued"

    A 403 means scope or admin status; a 404 usually means the Gitea version predates the endpoint.

  5. A VM booted but its runner never came online. Then the job is queued and you see VM activity in the logs. Look at registration failures — most often an invalid or invalidated registration token (see below).

  6. Job expired. Gitea abandons a job after ABANDONED_JOB_TIMEOUT (default 24h). If the daemon was down longer than that, the job is gone; re-run it.

Registration fails with an invalid token

Registration tokens are reusable, but creating a new token for a scope invalidates the previous one. If someone clicked "create new registration token" in the Gitea UI, the token in your registrationTokenFile is now dead. Re-seed GITEA_RUNNER_REGISTRATION_TOKEN on the server (and restart Gitea), or switch to fetchRegistrationTokenViaAPI: true. See setup.md §1.3.

actions/checkout fails instantly

Symptom. The job starts, the runner connects, and the very first step fails immediately — typically a spawn error naming node, or an unhelpful non-zero exit before any output.

Cause. Gitea Actions' JavaScript actions (actions/checkout and most of the ecosystem) run by spawning node inside the guest. Node.js is required in the image, and if provisioning was interrupted it may be absent.

Fix. Confirm, then reprovision:

gitea-macos-runner vm boot
ssh admin@<guest-ip> 'node --version && git --version'

gitea-macos-runner image provision default

If git is also missing, the provisioning step failed early — check the build log and rerun provisioning.

Runner rows piling up in the Gitea UI

Symptom. Site Administration → Actions → Runners accumulates offline macos-vm-… entries.

Cause. Gitea deletes an ephemeral registration when its job completes normally. A VM that is killed uncleanly — daemon crash, host power loss, jobTimeoutMinutes kill — never reaches that point, so the row is orphaned.

Fix. Usually nothing: the daemon's reconcile loop sweeps orphaned registrations every scheduler.reconcileIntervalSeconds (default 300) via DELETE /api/v1/admin/actions/runners/{id}, plus a daily midnight sweep. Orphans should clear within a few minutes.

If they persist, the daemon's admin PAT probably lacks write:admin — check the logs for delete failures. To clear them by hand, delete the rows in the Gitea UI; they are inert (offline registrations that have already been spent cannot receive jobs).

Disk filling up

Symptom. Free space falls steadily; the daemon starts refusing to launch VMs, citing storage.minFreeDiskGB.

Cause. Each VM is an APFS copy-on-write clone of the base image. The clone is free at creation but grows as the job writes — dependency caches, build outputs, Xcode's derived data. Clones from uncleanly-killed VMs are not reclaimed automatically.

Fix.

# See what's there.
du -sh ~/Library/Application\ Support/gitea-macos-runner/*
ls -la ~/Library/Application\ Support/gitea-macos-runner/vms

# With the daemon stopped, remove stale clones.
gitea-macos-runner service uninstall     # or stop the daemon
rm -rf ~/Library/Application\ Support/gitea-macos-runner/vms/<stale-clone>
gitea-macos-runner service install

Only delete entries under vms/ — images/ holds the base images you'd otherwise have to rebuild. Longer term: raise storage.minFreeDiskGB so the guard trips earlier, delete unused base images with image delete, and remember an Xcode image needs 140 GB+ of headroom, more with two concurrent clones diverging.

image build hangs at install

Symptom. image build sits for a long time at the macOS install phase with little visible progress.

Cause. Usually none — installing macOS from an IPSW genuinely takes a long time (tens of minutes, longer on slower storage). The install phase is largely silent.

Fix. Wait, and do not stop the VM mid-install. Interrupting the installer leaves the disk image in an undefined state; the resulting image may boot and then fail in confusing ways later. There is no resume.

If you did interrupt it, or the build genuinely failed:

gitea-macos-runner image delete <name>
gitea-macos-runner image build --ipsw <path> --name <name>

Before rebuilding, verify the IPSW is complete and matches your host architecture (Apple Silicon) and version requirement (macOS 27+ for unattended provisioning), and that you have enough free disk for the IPSW plus the target disk size.

Reference in New Issue View Git Blame Copy Permalink