nucleic/docs/OBSERVABILITY_AND_TESTING.md

Nucleic — Observability & Testing Strategy (v0)

Two halves of one backbone. The adapters concentrate the unknowns (ADAPTERS §0 confidence key), the runtime is concurrent and event-driven (RUNTIME), and the product handles proprietary source code. So observability is local-first and redaction-aware, and testing is built so the captured native streams that help debug a session are the same fixtures that pin the golden tests.

Status: design draft.

---

Part A — Observability

A.1 Invariants

1. Local-first. All logs, traces, metrics, and stream captures stay on the device by default. Nothing leaves the machine without explicit, per-action consent.
2. Sensitive-by-default redaction. Prompts, file contents, diffs, tool inputs/outputs, and transcripts are sensitive. They are redacted/hashed in logs unless the user explicitly enables verbose diagnostics for a specific session.
3. Structural metadata is freely loggable. IDs, seq, status transitions, event types, durations, token counts, exit codes — never content.
4. One correlation key. Everything is tagged (sessionID, seq) so a log line, a trace span, and a transcript event line up.

A.2 Structured logging (OSLog / unified logging)

Apple's unified logging is a near-perfect fit because privacy is a first-class field:

let log = Logger(subsystem: "com.nucleic.app", category: "session")
log.info("event \(event.kind.tag, privacy: .public) seq=\(seq, privacy: .public) \
          session=\(sessionID.short, privacy: .public)")
log.debug("tool input \(input.json, privacy: .private)")   // redacted in logs/sysdiagnose

• Categories: session, adapter.claude, adapter.codex, worktree, sync, approval, process, db.
• Privacy discipline: structural fields .public; anything derived from user content .private (redacted unless the user opts a session into verbose). Enforced by a lint rule (A.6) so a stray .public on content fails CI.
• Levels: debug (per-event, off in release), info (lifecycle/state transitions), error (recoverable), fault (process death, DB write failure, handshake failure).

A.3 Tracing (signposts)

os_signpost intervals over the pipeline (RUNTIME §2) so the cost of each hop is measurable in Instruments:

Span	Begin → End
ingest	adapter emits event → fan-out complete
transcript.append	write begin → fsync/return
approval.roundtrip	approvalRequested → approvalResolved (the human-latency metric)
integrate	strategy chosen → result
sync.broadcast	event in → all subscribers flushed

A.4 Metrics (in-app diagnostics, no external sink by default)

// counters
events_total{category}              unknown_events_total{backend}   // ← drift alarm
adapter_errors_total{backend,kind}  approvals_total{decision}
process_restarts_total{backend}     sync_frames_total{dir}
// histograms
approval_latency_seconds            turn_duration_seconds   integrate_duration_seconds
// gauges
sessions_live{status}               worktrees_count{project}  worktree_bytes{project}
sync_clients_connected              transcript_bytes{session}
// per-session rollup
tokens{dir}  cost_usd (Claude only)  turns

Surfaced in an in-app Diagnostics panel (and the per-session Log tab, UX §3.3). No Prometheus/OTel exporter ships by default; an opt-in local OTel endpoint is a later toggle for power users — never on by default.

A.5 Native-stream capture (the debug↔fixture bridge)

Per session, an optional raw capture records the exact native bytes (Claude NDJSON / Codex JSON-RPC) and the normalized AgentEvent[] side by side:

sessions/<id>/capture/native.ndjson   sessions/<id>/capture/normalized.json

• Off by default (it's the most sensitive artifact — full content). User toggles per session, or globally for adapter development.
• This is the single most valuable debugging artifact for adapter drift, and with one redaction-scrub step it becomes a test fixture (Part B). The capture format == the fixture format, deliberately.
• Drift signal: any line that decodes to AgentEvent.raw (unknown native shape) increments unknown_events_total and, in dev builds, surfaces a non-fatal diagnostic — the early warning that a CLI upgrade changed the wire format.

A.6 Crash & error reporting

• Opt-in only. Default off; when enabled, reports carry stack + structural metadata, with all .private fields scrubbed and a final allowlist pass (no paths, prompts, or diffs).
• CLI version + adapter confidence-tag of the last event are attached — so a crash report immediately says "Codex 0.41, last event was a 🟡 item shape."

A.7 Health & redaction in the sync path

• Sync exposes connection state, per-client lag (broadcast backpressure), handshake failures, and relay presence in the Diagnostics panel.
• Reminder (SYNC §6): push payloads carry title + reason only — content never traverses the push path.

---

Part B — Testing

B.1 The pyramid (and what makes it hermetic)

        ▲  Tier 3  Live smoke (real claude/codex)   — opt-in, nightly/manual, costs tokens
       ╱ ╲ Tier 2  Adapter contract via FAKE CLIs   — every commit, no tools/network
      ╱   ╲ Tier 1  Component w/ fakes (Controller, Sync, Worktree) — every commit
     ╱─────╲ Tier 0 Unit (pure logic)               — every commit, milliseconds

The product is hermetically testable end-to-end without claude/codex installed, because adapters talk to fake CLI binaries that replay fixtures (B.4). The real tools appear only in Tier 3.

B.2 Determinism prerequisites

Core logic takes an injected Clock and RandomNumberGenerator (slugs, IDs, timestamps). No ambient Date()/random() in the event/transcript/worktree paths — same constraint as a resumable workflow, and it makes golden tests stable.

B.3 Tier 0 — unit (pure)

Fixture-driven and property-based:

• Event normalization — recorded native line → expected AgentEvent (golden equality).
• Decision mapping — every Decision → exact native reply bytes per backend (ADAPTERS §1.2/§2.4).
• Risk classification — (toolName,input) → Risk table (e.g. Bash(rm -rf)→destructive, Read→readOnly), with adversarial inputs.
• State machine — event sequences → Session.status transitions (RUNTIME §5 / WORKTREE §3).
• Naming — slug/branch generation + collision dedupe; idempotent, filesystem-safe.
• Diff parsing — git status --porcelain=v2 / diff output → DiffStat (rename/copy cases).
• Properties: seq strictly monotonic; transcript dedupe idempotent on (sessionID,seq); replay invariant — replaying the JSONL reconstructs byte-identical UI state to live ingest (validates the "transcript is source of truth" claim, RUNTIME §6).

B.4 Tier 2 — adapter contract via fake CLIs (the keystone)

Two stub executables speak the real wire protocols off recorded fixtures:

• fake-claude — emits a fixture's NDJSON on stdout; for the approval fixture it actually performs the MCP HTTP tools/call(approve) against the adapter's in-process server and asserts on the reply — so the approval bridge (ADAPTERS §1.2) is exercised for real, over real stdio + HTTP, with no claude and no cost.
• fake-codex — speaks JSON-RPC over stdio: replays initialize/thread/turn notifications and issues real requestApproval server→client requests, asserting our reply.

These run through the production ProcessHost + adapter code. They catch line-framing bugs, partial-message assembly, JSON-RPC correlation, and approval-suspension deadlocks that pure unit tests can't. Each ADAPTERS 🔴/🟡 item maps to a fixture here.

Tier 1 (component) sits below this in the diagram but is introduced after, because it uses a simpler FakeBackend (next), not the wire-level fakes.

B.5 Tier 1 — component with fakes

A FakeBackend: AgentBackend emits scripted AgentEvents and parks approvals — used to test everything above the adapter without any subprocess:

• SessionController — single-writer ordering, status derivation, fan-out to N subscribers, injected synthetic events (setup logs, interrupts) interleave correctly.
• ApprovalCoordinator — suspend/resolve, first-responder-wins (two racers, one wins, other gets alreadyResolved), always_rule short-circuit, abandon-on-kill.
• TranscriptWriter/Reader — append→read round-trip, resume from offset, corruption fallback.
• SyncServer — two in-process SecureChannel peers: subscribe/sinceSeq catch-up, verbosity coalescing, reconnect + resync idempotency, scope enforcement.
• WorktreeManager — against real temp git repos (hermetic, created per test): create/ diff/finalize/integrate(merge|rebase|squash)/conflict/discard, and reconcile after a simulated crash (kill mid-run, relaunch, assert orphan handling).

B.6 Security tests (SecureChannel)

First-class, given E2EE:

• Noise handshake test vectors (known-answer) for XX (pairing) and IK (reconnect).
• Negative: MITM with a swapped static key → handshake fails; wrong pairing PSK → fails.
• Replay/nonce: replayed frame rejected; nonce monotonic.
• Revocation: removed device → next handshake unauthorized.
• Relay confidentiality property: a man-in-the-middle relay sees only ciphertext + sizes (assert no plaintext substring of a known secret appears in routed frames).

B.7 Concurrency tests

• Build the whole package under Swift 6 strict concurrency (warnings = errors).
• Thread Sanitizer suite running many parallel FakeBackend sessions.
• Isolation property: crash/throw in one session's stream never perturbs siblings (validates RUNTIME open-Q #4 / the M2 parallel milestone).

B.8 UI tests

• SwiftUI snapshot tests of key views from canned state: dashboard at 0/1/20 sessions, approval bar per Risk and per BackendCapabilities (modify-and-allow shown/hidden), conflict view, provisioning view.
• A few end-to-end XCUITest flows on the headline paths: create → approve → integrate.

B.9 Tier 3 — live smoke (opt-in)

Gated behind a flag + secrets; nightly + manual, never on PR:

• Run real claude/codex on a tiny committed fixture repo through the full stack; assert high-level invariants (a turn completes, an approval round-trips, a diff appears, integrate merges).
• Re-records fixtures for the pinned CLI versions → the diff against committed fixtures is the drift report. A changed wire format shows up here first, as a failing nightly, with the new capture ready to promote.

B.10 CI matrix

Tier	When	Needs
0 Unit	every push/PR	nothing
1 Component	every push/PR	git in the runner
2 Adapter (fake CLIs)	every push/PR	the fake stubs (built in-repo)
Security / Concurrency / UI snapshot	every push/PR	TSAN job + macOS runner
3 Live smoke	nightly + manual dispatch	claude/codex installed, API creds in secrets, token budget

Also in CI: swift-format + the privacy-lint (A.2) that fails on .public over content-derived values, and a strict-concurrency build gate.

---

C. The loop that ties it together

real CLI run ──capture (A.5)──▶ native.ndjson + normalized.json
                                      │ redaction-scrub + commit
                                      ▼
                              fixtures/<backend>/<version>/*
                                      │
         ┌────────────────────────────┼───────────────────────────┐
         ▼                            ▼                            ▼
   Tier 0 golden            Tier 2 fake-CLI replay        Tier 3 re-record → drift diff

A debugging capture today is a regression fixture tomorrow; a CLI upgrade that breaks the wire format trips unknown_events_total in the app and the nightly drift diff in CI — the same signal from two directions.

D. Open questions

1. Capture redaction depth — is structural-only scrubbing enough to commit a fixture, or do we need synthetic-repo runs so fixtures contain no real proprietary content at all? (Leaning: dedicated throwaway fixture repos for anything committed.)
2. Snapshot test stability — SwiftUI snapshot flakiness across macOS versions; per-OS baselines vs. tolerance, or prefer view-model assertions over pixels.
3. Live-tier budget — token cost ceiling per nightly run and which backends/models are in the matrix.
4. Opt-in telemetry shape — if/when a hosted error/metrics sink exists, the consent model and the exact allowlist of fields permitted to leave the device.