# Nucleic — Observability & Testing Strategy (v0)

Two halves of one backbone. The adapters concentrate the unknowns (ADAPTERS §0 confidence
key), the runtime is concurrent and event-driven (RUNTIME), and the product handles
**proprietary source code**. So observability is local-first and redaction-aware, and testing
is built so the *captured* native streams that help debug a session are the same *fixtures*
that pin the golden tests.

**Status:** design draft.

---

## Part A — Observability

### A.1 Invariants

1. **Local-first.** All logs, traces, metrics, and stream captures stay on the device by
   default. Nothing leaves the machine without explicit, per-action consent.
2. **Sensitive-by-default redaction.** Prompts, file contents, diffs, tool inputs/outputs, and
   transcripts are *sensitive*. They are redacted/hashed in logs unless the user explicitly
   enables verbose diagnostics for a specific session.
3. **Structural metadata is freely loggable.** IDs, `seq`, status transitions, event *types*,
   durations, token counts, exit codes — never content.
4. **One correlation key.** Everything is tagged `(sessionID, seq)` so a log line, a trace
   span, and a transcript event line up.

### A.2 Structured logging (OSLog / unified logging)

Apple's unified logging is a near-perfect fit because **privacy is a first-class field**:

```swift
let log = Logger(subsystem: "com.nucleic.app", category: "session")
log.info("event \(event.kind.tag, privacy: .public) seq=\(seq, privacy: .public) \
          session=\(sessionID.short, privacy: .public)")
log.debug("tool input \(input.json, privacy: .private)")   // redacted in logs/sysdiagnose
```

- **Categories:** `session`, `adapter.claude`, `adapter.codex`, `worktree`, `sync`,
  `approval`, `process`, `db`.
- **Privacy discipline:** structural fields `.public`; anything derived from user content
  `.private` (redacted unless the user opts a session into verbose). Enforced by a lint rule
  (A.6) so a stray `.public` on content fails CI.
- **Levels:** `debug` (per-event, off in release), `info` (lifecycle/state transitions),
  `error` (recoverable), `fault` (process death, DB write failure, handshake failure).

### A.3 Tracing (signposts)

`os_signpost` intervals over the pipeline (RUNTIME §2) so the cost of each hop is measurable in
Instruments:

| Span | Begin → End |
| --- | --- |
| `ingest` | adapter emits event → fan-out complete |
| `transcript.append` | write begin → fsync/return |
| `approval.roundtrip` | `approvalRequested` → `approvalResolved` (the human-latency metric) |
| `integrate` | strategy chosen → result |
| `sync.broadcast` | event in → all subscribers flushed |

### A.4 Metrics (in-app diagnostics, no external sink by default)

```swift
// counters
events_total{category}              unknown_events_total{backend}   // ← drift alarm
adapter_errors_total{backend,kind}  approvals_total{decision}
process_restarts_total{backend}     sync_frames_total{dir}
// histograms
approval_latency_seconds            turn_duration_seconds   integrate_duration_seconds
// gauges
sessions_live{status}               worktrees_count{project}  worktree_bytes{project}
sync_clients_connected              transcript_bytes{session}
// per-session rollup
tokens{dir}  cost_usd (Claude only)  turns
```

Surfaced in an in-app **Diagnostics panel** (and the per-session **Log tab**, UX §3.3). No
Prometheus/OTel exporter ships by default; an *opt-in* local OTel endpoint is a later toggle
for power users — never on by default.

### A.5 Native-stream capture (the debug↔fixture bridge)

Per session, an optional **raw capture** records the exact native bytes (Claude NDJSON / Codex
JSON-RPC) and the normalized `AgentEvent[]` side by side:

```
sessions/<id>/capture/native.ndjson   sessions/<id>/capture/normalized.json
```

- **Off by default** (it's the most sensitive artifact — full content). User toggles per
  session, or globally for adapter development.
- **This is the single most valuable debugging artifact** for adapter drift, and with one
  redaction-scrub step it becomes a test fixture (Part B). The capture format == the fixture
  format, deliberately.
- **Drift signal:** any line that decodes to `AgentEvent.raw` (unknown native shape) increments
  `unknown_events_total` and, in dev builds, surfaces a non-fatal diagnostic — the early
  warning that a CLI upgrade changed the wire format.

### A.6 Crash & error reporting

- **Opt-in only.** Default off; when enabled, reports carry stack + structural metadata, with
  all `.private` fields scrubbed and a final allowlist pass (no paths, prompts, or diffs).
- CLI version + adapter confidence-tag of the last event are attached — so a crash report
  immediately says "Codex 0.41, last event was a 🟡 item shape."

### A.7 Health & redaction in the sync path

- Sync exposes connection state, per-client lag (broadcast backpressure), handshake
  failures, and relay presence in the Diagnostics panel.
- Reminder (SYNC §6): push payloads carry title + reason only — content never traverses the
  push path.

---

## Part B — Testing

### B.1 The pyramid (and what makes it hermetic)

```
        ▲  Tier 3  Live smoke (real claude/codex)   — opt-in, nightly/manual, costs tokens
       ╱ ╲ Tier 2  Adapter contract via FAKE CLIs   — every commit, no tools/network
      ╱   ╲ Tier 1  Component w/ fakes (Controller, Sync, Worktree) — every commit
     ╱─────╲ Tier 0 Unit (pure logic)               — every commit, milliseconds
```

The product is hermetically testable end-to-end **without** `claude`/`codex` installed,
because adapters talk to *fake CLI binaries* that replay fixtures (B.4). The real tools appear
only in Tier 3.

### B.2 Determinism prerequisites

Core logic takes an injected `Clock` and `RandomNumberGenerator` (slugs, IDs, timestamps).
No ambient `Date()`/`random()` in the event/transcript/worktree paths — same constraint as a
resumable workflow, and it makes golden tests stable.

### B.3 Tier 0 — unit (pure)

Fixture-driven and property-based:
- **Event normalization** — recorded native line → expected `AgentEvent` (golden equality).
- **Decision mapping** — every `Decision` → exact native reply bytes per backend (ADAPTERS
  §1.2/§2.4).
- **Risk classification** — `(toolName,input)` → `Risk` table (e.g. `Bash(rm -rf)`→destructive,
  `Read`→readOnly), with adversarial inputs.
- **State machine** — event sequences → `Session.status` transitions (RUNTIME §5 / WORKTREE §3).
- **Naming** — slug/branch generation + collision dedupe; idempotent, filesystem-safe.
- **Diff parsing** — `git status --porcelain=v2` / diff output → `DiffStat` (rename/copy cases).
- **Properties:** `seq` strictly monotonic; transcript dedupe idempotent on `(sessionID,seq)`;
  **replay invariant** — replaying the JSONL reconstructs byte-identical UI state to live
  ingest (validates the "transcript is source of truth" claim, RUNTIME §6).

### B.4 Tier 2 — adapter contract via fake CLIs (the keystone)

Two stub executables speak the **real wire protocols** off recorded fixtures:

- **`fake-claude`** — emits a fixture's NDJSON on stdout; for the approval fixture it *actually
  performs the MCP HTTP `tools/call(approve)`* against the adapter's in-process server and
  asserts on the reply — so the approval bridge (ADAPTERS §1.2) is exercised for real, over
  real stdio + HTTP, with no `claude` and no cost.
- **`fake-codex`** — speaks JSON-RPC over stdio: replays `initialize`/thread/turn
  notifications and issues real `requestApproval` server→client requests, asserting our reply.

These run through the *production* `ProcessHost` + adapter code. They catch line-framing bugs,
partial-message assembly, JSON-RPC correlation, and approval-suspension deadlocks that pure
unit tests can't. Each ADAPTERS 🔴/🟡 item maps to a fixture here.

> Tier 1 (component) sits *below* this in the diagram but is introduced after, because it uses
> a simpler `FakeBackend` (next), not the wire-level fakes.

### B.5 Tier 1 — component with fakes

A `FakeBackend: AgentBackend` emits scripted `AgentEvent`s and parks approvals — used to test
everything above the adapter without any subprocess:

- **SessionController** — single-writer ordering, status derivation, fan-out to N subscribers,
  injected synthetic events (setup logs, interrupts) interleave correctly.
- **ApprovalCoordinator** — suspend/resolve, **first-responder-wins** (two racers, one wins,
  other gets `alreadyResolved`), `always_rule` short-circuit, abandon-on-kill.
- **TranscriptWriter/Reader** — append→read round-trip, resume from offset, corruption fallback.
- **SyncServer** — two in-process `SecureChannel` peers: subscribe/`sinceSeq` catch-up,
  verbosity coalescing, reconnect + resync idempotency, scope enforcement.
- **WorktreeManager** — against **real temp git repos** (hermetic, created per test): create/
  diff/finalize/integrate(merge|rebase|squash)/conflict/discard, and `reconcile` after a
  simulated crash (kill mid-run, relaunch, assert orphan handling).

### B.6 Security tests (SecureChannel)

First-class, given E2EE:
- Noise handshake **test vectors** (known-answer) for XX (pairing) and IK (reconnect).
- **Negative:** MITM with a swapped static key → handshake fails; wrong pairing PSK → fails.
- **Replay/nonce:** replayed frame rejected; nonce monotonic.
- **Revocation:** removed device → next handshake `unauthorized`.
- **Relay confidentiality property:** a man-in-the-middle relay sees only ciphertext + sizes
  (assert no plaintext substring of a known secret appears in routed frames).

### B.7 Concurrency tests

- Build the whole package under **Swift 6 strict concurrency** (warnings = errors).
- **Thread Sanitizer** suite running many parallel `FakeBackend` sessions.
- **Isolation property:** crash/throw in one session's stream never perturbs siblings
  (validates RUNTIME open-Q #4 / the M2 parallel milestone).

### B.8 UI tests

- SwiftUI **snapshot tests** of key views from canned state: dashboard at 0/1/20 sessions,
  approval bar per `Risk` and per `BackendCapabilities` (modify-and-allow shown/hidden),
  conflict view, provisioning view.
- A few end-to-end **XCUITest** flows on the headline paths: create → approve → integrate.

### B.9 Tier 3 — live smoke (opt-in)

Gated behind a flag + secrets; **nightly + manual**, never on PR:
- Run **real** `claude`/`codex` on a tiny committed fixture repo through the full stack;
  assert high-level invariants (a turn completes, an approval round-trips, a diff appears,
  integrate merges).
- **Re-records fixtures** for the pinned CLI versions → the diff against committed fixtures is
  the **drift report**. A changed wire format shows up here first, as a failing nightly, with
  the new capture ready to promote.

### B.10 CI matrix

| Tier | When | Needs |
| --- | --- | --- |
| 0 Unit | every push/PR | nothing |
| 1 Component | every push/PR | git in the runner |
| 2 Adapter (fake CLIs) | every push/PR | the fake stubs (built in-repo) |
| Security / Concurrency / UI snapshot | every push/PR | TSAN job + macOS runner |
| 3 Live smoke | nightly + manual dispatch | `claude`/`codex` installed, API creds in secrets, token budget |

Also in CI: `swift-format` + the **privacy-lint** (A.2) that fails on `.public` over
content-derived values, and a strict-concurrency build gate.

---

## C. The loop that ties it together

```
real CLI run ──capture (A.5)──▶ native.ndjson + normalized.json
                                      │ redaction-scrub + commit
                                      ▼
                              fixtures/<backend>/<version>/*
                                      │
         ┌────────────────────────────┼───────────────────────────┐
         ▼                            ▼                            ▼
   Tier 0 golden            Tier 2 fake-CLI replay        Tier 3 re-record → drift diff
```

A debugging capture today is a regression fixture tomorrow; a CLI upgrade that breaks the wire
format trips `unknown_events_total` in the app *and* the nightly drift diff in CI — the same
signal from two directions.

## D. Open questions

1. **Capture redaction depth** — is structural-only scrubbing enough to commit a fixture, or do
   we need synthetic-repo runs so fixtures contain no real proprietary content at all?
   (Leaning: dedicated throwaway fixture repos for anything committed.)
2. **Snapshot test stability** — SwiftUI snapshot flakiness across macOS versions; per-OS
   baselines vs. tolerance, or prefer view-model assertions over pixels.
3. **Live-tier budget** — token cost ceiling per nightly run and which backends/models are in
   the matrix.
4. **Opt-in telemetry shape** — if/when a hosted error/metrics sink exists, the consent model
   and the exact allowlist of fields permitted to leave the device.
