339 lines
14 KiB
Markdown
339 lines
14 KiB
Markdown
# Troubleshooting
|
|
|
|
Start with `gitea-macos-runner doctor` — it catches most misconfiguration before you go
|
|
symptom-hunting. Then find your symptom below.
|
|
|
|
Useful log commands throughout:
|
|
|
|
```sh
|
|
# Daemon logs, live
|
|
log stream --predicate 'process == "gitea-macos-runner"' --info
|
|
|
|
# Daemon logs, last hour
|
|
log show --predicate 'process == "gitea-macos-runner"' --info --last 1h
|
|
|
|
gitea-macos-runner service status
|
|
```
|
|
|
|
---
|
|
|
|
## Quick reference
|
|
|
|
| Symptom | Cause | Fix |
|
|
| --- | --- | --- |
|
|
| VM won't start; entitlement / `com.apple.security.virtualization` error | Running an unsigned binary, or one outside the signed `.app` bundle | `make sign` (or re-run `make install`); invoke the installed bundle, never `.build/release/…` |
|
|
| `virtualMachineLimitExceeded` at boot | macOS allows at most **2** concurrent macOS VMs | Set `scheduler.maxConcurrentVMs` ≤ 2; kill stray VMs from earlier runs |
|
|
| VM boots but never gets an IP | DHCP lease not yet written, or Local Network privacy denial (macOS 15+) | Check `/var/db/dhcpd_leases`; grant Local Network permission or pre-authorize the subnet |
|
|
| SSH times out on a freshly built image | Guest macOS < 27, so provisioning options were ignored and Setup Assistant is waiting | Rebuild the image from a macOS **27+** IPSW |
|
|
| `SecKeyCreateRandomKey` / "Interaction is not allowed" | `login.keychain` is locked — no GUI session | Run as a LaunchAgent in an unlocked GUI session; enable auto-login |
|
|
| Job stays queued forever | Label mismatch, or the daemon isn't running/reaching Gitea | Use bare label names in `runs-on`; match `runner.labels`; check daemon logs |
|
|
| `actions/checkout` fails instantly | Node.js missing from the guest image | `gitea-macos-runner image provision <name>` |
|
|
| Runner rows piling up in the Gitea UI | VMs killed uncleanly; registrations orphaned | Reconcile loop cleans them; force it by restarting the daemon; delete manually if needed |
|
|
| `doctor` says `runner download url → 403` but the URL works in a browser | Old build: the check used `HEAD`, and the presigned redirect target is signed per method | Upgrade — the check now uses a ranged `GET`. If it persists, the asset really is missing |
|
|
| Disk filling up | Copy-on-write clones grow as jobs write | Raise `storage.minFreeDiskGB`; delete stale clones in `storeDir/vms` |
|
|
| `image build` appears to hang during install | Normal — macOS install is slow | Wait. **Do not stop the VM mid-install**; if you did, delete the image and rebuild |
|
|
|
|
---
|
|
|
|
## VM won't start — entitlement error
|
|
|
|
**Symptom.** Any VM operation fails immediately with an error naming
|
|
`com.apple.security.virtualization`, or a generic "operation not permitted" from
|
|
Virtualization.framework.
|
|
|
|
**Cause.** Virtualization.framework checks the entitlement on the calling binary, and entitlements
|
|
are only honoured on a signed binary inside a proper `.app` bundle. The raw product of
|
|
`swift build` has neither.
|
|
|
|
**Fix.**
|
|
|
|
```sh
|
|
make install # build → bundle → sign → install (make sign alone re-signs in place)
|
|
codesign -d --entitlements - ~/Applications/GiteaMacosRunner.app # verify
|
|
```
|
|
|
|
The output must list `com.apple.security.virtualization`. Then confirm the command you're running
|
|
resolves to the installed bundle's binary — `which -a gitea-macos-runner` should point at
|
|
`~/Applications/GiteaMacosRunner.app/Contents/MacOS/gitea-macos-runner`, not at
|
|
`.build/release/gitea-macos-runner`. Ad-hoc signing is sufficient; you do not need a paid developer
|
|
account.
|
|
|
|
---
|
|
|
|
## `virtualMachineLimitExceeded`
|
|
|
|
**Symptom.** The first VM boots fine; a second or third fails with `virtualMachineLimitExceeded`.
|
|
|
|
**Cause.** macOS permits **two** concurrent macOS guests per host. This is an Apple kernel and
|
|
licensing limit, not a resource constraint — more RAM will not raise it.
|
|
|
|
**Fix.** Set `scheduler.maxConcurrentVMs` to 2 or less. If you're already at 2 and still hitting
|
|
the limit, a VM from a previous run is still alive — check for stray processes and for leftover
|
|
directories under `storeDir/vms`, then restart the daemon so it starts from a clean state.
|
|
|
|
To handle more macOS jobs in parallel, add another Mac.
|
|
|
|
---
|
|
|
|
## VM starts but never gets an IP
|
|
|
|
**Symptom.** The VM boots (you can see it progress if you use `vm boot`), but the daemon reports
|
|
that it could not resolve the guest address, or gives up at `scheduler.bootTimeoutSeconds`.
|
|
|
|
**Cause.** The daemon resolves the guest's NAT address from the host's DHCP lease file, which is
|
|
only written once the guest requests a lease — several seconds after the boot screen appears. If the
|
|
address never appears at all, the usual culprit on macOS 15+ is the **Local Network privacy
|
|
prompt**: a LaunchAgent that was never granted permission (or was denied) cannot talk to the guest.
|
|
|
|
**Fix.**
|
|
|
|
1. Check the lease file after the guest has been up for ~30 seconds:
|
|
|
|
```sh
|
|
cat /var/db/dhcpd_leases
|
|
```
|
|
|
|
An entry with a recent `lease` timestamp and the guest's MAC means networking is fine and the
|
|
problem is timing — raise `scheduler.bootTimeoutSeconds`.
|
|
|
|
2. No entry at all: check **System Settings → Privacy & Security → Local Network** and enable the
|
|
runner. If it isn't listed, trigger the prompt interactively from a Terminal in the GUI session:
|
|
|
|
```sh
|
|
gitea-macos-runner vm boot
|
|
```
|
|
|
|
and click **Allow**.
|
|
|
|
3. Or pre-authorize the VM subnet, then reboot:
|
|
|
|
```sh
|
|
sudo defaults write com.apple.network.local-network \
|
|
AllowedEthernetLocalNetworkAddresses -array "192.168.0.0/16"
|
|
```
|
|
|
|
Match the range to what your host's NAT actually hands out.
|
|
|
|
---
|
|
|
|
## SSH times out on a freshly built image
|
|
|
|
**Symptom.** The VM boots and gets an IP, but SSH never connects. Attaching a display to the guest
|
|
shows **Setup Assistant** — the region/Apple ID welcome flow — rather than a login window.
|
|
|
|
**Cause.** The image builder uses `VZMacGuestProvisioningOptions` to create the admin account, enable
|
|
Remote Login, and skip Setup Assistant. That API requires **macOS 27 or newer in the guest as well
|
|
as the host**. An older guest **silently ignores** the options: no error, no account, no SSH server
|
|
— it just sits at first-run setup forever.
|
|
|
|
**Fix.** Rebuild the base image from a macOS 27+ IPSW:
|
|
|
|
```sh
|
|
gitea-macos-runner image delete default
|
|
gitea-macos-runner image build --ipsw ~/Downloads/UniversalMac_27.0_XXXXX_Restore.ipsw
|
|
```
|
|
|
|
Verify the host is also 27+ (`sw_vers`). There is no way to make a pre-27 guest work unattended
|
|
with this builder.
|
|
|
|
---
|
|
|
|
## `SecKeyCreateRandomKey` / "Interaction is not allowed"
|
|
|
|
**Symptom.** The daemon starts but fails during VM setup with a Security-framework error mentioning
|
|
`SecKeyCreateRandomKey`, `errSecInteractionNotAllowed`, or "Interaction is not allowed". Often it
|
|
works when you run the daemon by hand in Terminal and fails under launchd.
|
|
|
|
**Cause.** The **`login.keychain` is locked.** macOS 15+ requires it unlocked for key operations the
|
|
VM lifecycle performs, and it is only unlocked inside a live, logged-in GUI session. A LaunchDaemon,
|
|
an SSH-only session, or a Mac sitting at the login window all fail this.
|
|
|
|
**Fix.**
|
|
|
|
1. Confirm the service is installed as a **LaunchAgent**, not a LaunchDaemon:
|
|
`gitea-macos-runner service install` does the right thing; a hand-written plist in
|
|
`/Library/LaunchDaemons` does not.
|
|
2. Ensure the runner user is actually logged in with the desktop loaded. Enable auto-login:
|
|
**System Settings → Users & Groups → Automatically log in as**.
|
|
3. Prevent the machine from returning to a locked state:
|
|
|
|
```sh
|
|
sudo pmset -a sleep 0 disablesleep 1
|
|
```
|
|
|
|
and disable "Require password after screen saver begins" for the runner user.
|
|
|
|
Connecting over Screen Sharing to a Mac at the login window does not unlock `login.keychain` for
|
|
launchd's session — auto-login is the reliable answer on a dedicated CI Mac.
|
|
|
|
---
|
|
|
|
## Job stays queued and no VM boots
|
|
|
|
**Symptom.** The workflow shows as queued in Gitea indefinitely. Nothing appears in the daemon logs
|
|
about it.
|
|
|
|
**Causes and fixes, in the order worth checking:**
|
|
|
|
1. **Label mismatch.** `runs-on` must use **bare label names** (`macos-arm64`), and every label
|
|
listed must appear in the host config's `runner.labels`. The `:host` suffix used at registration
|
|
is runner-side only and must never appear in workflow YAML. A single typo produces exactly this
|
|
symptom with no error anywhere.
|
|
|
|
2. **Daemon not running or not polling.**
|
|
|
|
```sh
|
|
gitea-macos-runner service status
|
|
log show --predicate 'process == "gitea-macos-runner"' --info --last 15m
|
|
```
|
|
|
|
You should see a poll every `scheduler.pollIntervalSeconds`.
|
|
|
|
3. **Gitea too old.** The queued-jobs API with the `labels` field requires **Gitea ≥ 1.25**. On an
|
|
older instance the daemon can never see jobs. `doctor` reports the server version.
|
|
|
|
4. **Admin PAT wrong or under-scoped.** The token must belong to a **site admin** and carry
|
|
`read:admin` + `write:admin`. Test it:
|
|
|
|
```sh
|
|
curl -H "Authorization: token $TOKEN" \
|
|
"https://gitea.example.com/api/v1/admin/actions/jobs?status=queued"
|
|
```
|
|
|
|
A 403 means scope or admin status; a 404 usually means the Gitea version predates the endpoint.
|
|
|
|
5. **A VM booted but its runner never came online.** Then the job is queued *and* you see VM
|
|
activity in the logs. Look at registration failures — most often an invalid or invalidated
|
|
registration token (see below).
|
|
|
|
6. **Job expired.** Gitea abandons a job after `ABANDONED_JOB_TIMEOUT` (default 24h). If the daemon
|
|
was down longer than that, the job is gone; re-run it.
|
|
|
|
### Registration fails with an invalid token
|
|
|
|
Registration tokens are reusable, but **creating a new token for a scope invalidates the previous
|
|
one**. If someone clicked "create new registration token" in the Gitea UI, the token in your
|
|
`registrationTokenFile` is now dead. Re-seed
|
|
`GITEA_RUNNER_REGISTRATION_TOKEN` on the server (and restart Gitea), or switch to
|
|
`fetchRegistrationTokenViaAPI: true`. See
|
|
[setup.md §1.3](setup.md#13-choose-a-registration-token-strategy).
|
|
|
|
---
|
|
|
|
## `actions/checkout` fails instantly
|
|
|
|
**Symptom.** The job starts, the runner connects, and the very first step fails immediately —
|
|
typically a spawn error naming `node`, or an unhelpful non-zero exit before any output.
|
|
|
|
**Cause.** Gitea Actions' JavaScript actions (`actions/checkout` and most of the ecosystem) run by
|
|
spawning `node` inside the guest. **Node.js is required in the image**, and if provisioning was
|
|
interrupted it may be absent.
|
|
|
|
**Fix.** Confirm, then reprovision:
|
|
|
|
```sh
|
|
gitea-macos-runner vm boot
|
|
ssh admin@<guest-ip> 'node --version && git --version'
|
|
|
|
gitea-macos-runner image provision default
|
|
```
|
|
|
|
If `git` is also missing, the provisioning step failed early — check the build log and rerun
|
|
provisioning.
|
|
|
|
---
|
|
|
|
## Runner rows piling up in the Gitea UI
|
|
|
|
**Symptom.** **Site Administration → Actions → Runners** accumulates offline `macos-vm-…` entries.
|
|
|
|
**Cause.** Gitea deletes an ephemeral registration when its job completes normally. A VM that is
|
|
killed uncleanly — daemon crash, host power loss, `jobTimeoutMinutes` kill — never reaches that
|
|
point, so the row is orphaned.
|
|
|
|
**Fix.** Usually nothing: the daemon's reconcile loop sweeps orphaned registrations every
|
|
`scheduler.reconcileIntervalSeconds` (default 300) via
|
|
`DELETE /api/v1/admin/actions/runners/{id}`, plus a daily midnight sweep. Orphans should clear
|
|
within a few minutes.
|
|
|
|
If they persist, the daemon's admin PAT probably lacks `write:admin` — check the logs for delete
|
|
failures. To clear them by hand, delete the rows in the Gitea UI; they are inert (offline
|
|
registrations that have already been spent cannot receive jobs).
|
|
|
|
---
|
|
|
|
## `doctor` warns "runner download url → 403"
|
|
|
|
**Symptom.** `doctor` reports the runner download URL as a 403, but pasting the same URL into a
|
|
browser downloads the binary fine:
|
|
|
|
```
|
|
! runner download url https://gitea.com/gitea/runner/releases/download/v3.0.2/… → 403
|
|
```
|
|
|
|
**Cause.** gitea.com does not serve release assets itself. It answers with a `303 See Other`
|
|
pointing at a presigned object-storage URL, and that signature covers the HTTP method of the
|
|
request that minted it. A `HEAD` probe therefore gets a HEAD-signed URL — and then the HTTP client,
|
|
following the 303, rewrites the method to `GET` (RFC 7231 §6.4.4) and replays the signed URL with
|
|
the one verb it was not signed for. The store answers `403 SignatureDoesNotMatch`. Nothing is
|
|
actually wrong with the asset.
|
|
|
|
**Fix.** Upgrade — the check now probes with a one-byte ranged `GET` (`Range: bytes=0-0`), the same
|
|
verb `image provision` uses for the real download, and falls back to a `HEAD` only if the range is
|
|
rejected. If you still see a 403 after upgrading, the asset really is gone: check `runner.version`
|
|
and `runner.runnerDownloadURL` for a `darwin-arm64` build.
|
|
|
|
---
|
|
|
|
## Disk filling up
|
|
|
|
**Symptom.** Free space falls steadily; the daemon starts refusing to launch VMs, citing
|
|
`storage.minFreeDiskGB`.
|
|
|
|
**Cause.** Each VM is an APFS copy-on-write clone of the base image. The clone is free at creation
|
|
but **grows as the job writes** — dependency caches, build outputs, Xcode's derived data. Clones
|
|
from uncleanly-killed VMs are not reclaimed automatically.
|
|
|
|
**Fix.**
|
|
|
|
```sh
|
|
# See what's there.
|
|
du -sh ~/Library/Application\ Support/gitea-macos-runner/*
|
|
ls -la ~/Library/Application\ Support/gitea-macos-runner/vms
|
|
|
|
# With the daemon stopped, remove stale clones.
|
|
gitea-macos-runner service uninstall # or stop the daemon
|
|
rm -rf ~/Library/Application\ Support/gitea-macos-runner/vms/<stale-clone>
|
|
gitea-macos-runner service install
|
|
```
|
|
|
|
Only delete entries under `vms/` — `images/` holds the base images you'd otherwise have to rebuild.
|
|
Longer term: raise `storage.minFreeDiskGB` so the guard trips earlier, delete unused base images
|
|
with `image delete`, and remember an Xcode image needs 140 GB+ of headroom, more with two
|
|
concurrent clones diverging.
|
|
|
|
---
|
|
|
|
## `image build` hangs at install
|
|
|
|
**Symptom.** `image build` sits for a long time at the macOS install phase with little visible
|
|
progress.
|
|
|
|
**Cause.** Usually none — installing macOS from an IPSW genuinely takes a long time (tens of
|
|
minutes, longer on slower storage). The install phase is largely silent.
|
|
|
|
**Fix.** **Wait, and do not stop the VM mid-install.** Interrupting the installer leaves the disk
|
|
image in an undefined state; the resulting image may boot and then fail in confusing ways later.
|
|
There is no resume.
|
|
|
|
If you did interrupt it, or the build genuinely failed:
|
|
|
|
```sh
|
|
gitea-macos-runner image delete <name>
|
|
gitea-macos-runner image build --ipsw <path> --name <name>
|
|
```
|
|
|
|
Before rebuilding, verify the IPSW is complete and matches your host architecture (Apple Silicon)
|
|
and version requirement (macOS 27+ for unattended provisioning), and that you have enough free disk
|
|
for the IPSW plus the target disk size.
|