Nucleic Control: spawn watchdog + reliable Stop + stdio diagnostics

Containerized control sessions could hang with no output ("Working…" forever): the agent's exec
stdio over the vminitd vsock channel intermittently failed to carry bytes, so claude ran and
reached the approval server but its stdin/stdout never connected — it idled in interactive
stream-json mode and never exited. Forensics (nucleic.sqlite + per-session claude-home MCP logs)
showed every stdio layer byte-identical to a working state, i.e. a flaky framework race, not a
regression in our code. Make the failure recoverable and visible instead of an eternal spinner:

- ClaudeCodeBackend: a 60s spawn watchdog on containerized runs — no first stdout → emit a
  recoverable error (with guest stderr + container probe), SIGKILL the wedged process, and finish
  the run errored, instead of awaiting stdoutLines forever.
- Stop reliability: ProcessHandle.forceCloseStreams() (ContainerizedProcessHandle finishes its
  line streams host-side; default no-op for the host pipe handle), wired into every kill
  escalation (terminate/interruptThenKill/killGroupAfter) so a force-killed run always settles
  even when the guest wait/stdio RPC wedges — the real cause of "Stop is inconsistent".
- Cap MCP_TIMEOUT (connection) to 60s so an unreachable approval server can't wedge startup for
  ~24.8 days; MCP_TOOL_TIMEOUT stays unbounded for human-answered approvals.
- LinuxProcess.setupIO logs which stdio stream fails to connect (os.Logger, com.nucleic /
  container-io) so a stall pinpoints the failing stream. Vendored patch #3.
- Tests: stop-escalation force-close, responsive-process no-op, MCP timeout asymmetry.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Nucleic
2026-06-25 17:37:46 -07:00
co-authored by Claude Opus 4.8
parent f8b200128c
commit 3d02a0a2f7
6 changed files with 246 additions and 20 deletions
@@ -65,18 +65,35 @@ public actor ClaudeCodeBackend: AgentBackend {
}
}
/// Gated MCP calls (`approve`, `AskUserQuestion`, `check_conflict`, `host_exec`)
/// SUSPEND until a human responds which may be many minutes if they step away.
/// The CLI's default MCP tool-call timeout would otherwise abandon the suspended
/// call and let the model proceed *without* an answer (observed as questions/approvals
/// "timing out"). Pin both MCP timeouts to Node's max delay (~24.8 days, the largest
/// `setTimeout` accepts before it wraps) so a call blocked on user input effectively
/// never times out. An explicit `RunSpec.extraEnv` still wins.
/// MCP timeouts for the spawned CLI. Two knobs with deliberately different values:
///
/// * `MCP_TOOL_TIMEOUT` per *tool call*. Our gated calls (`approve`, `AskUserQuestion`,
/// `check_conflict`, `host_exec`) SUSPEND until a human responds, which may be many minutes
/// if they step away. The CLI's default would abandon the suspended call and let the model
/// proceed *without* an answer (observed as questions/approvals "timing out"), so this stays
/// pinned to Node's max delay (~24.8 days, the largest `setTimeout` accepts before it wraps).
///
/// * `MCP_TIMEOUT` the initial server *connection/handshake*, which involves no human. Pinning
/// it as high as the tool timeout meant an unreachable approval server would wedge startup for
/// ~24.8 days instead of failing fast, so it's bounded to a generous-but-finite 60s. The
/// approval server is in-process and already listening before the child execs, so a real
/// connection completes in milliseconds; 60s only ever bounds a genuine failure.
///
/// An explicit `RunSpec.extraEnv` still wins.
static let mcpTimeoutEnv: [String: String] = [
"MCP_TIMEOUT": "2147483647",
"MCP_TIMEOUT": "60000",
"MCP_TOOL_TIMEOUT": "2147483647",
]
/// How long a *containerized* run may produce NO stdout before it's declared a stalled spawn.
/// The first event (`system/init`) is emitted within seconds of launch, before any model call,
/// so a containerized run silent this long has hit a stdio-transport failure: the agent is
/// running (it reached the approval server) but its stdout/stdin isn't getting through the
/// guesthost vsock relay. Generous enough never to false-fire on a slow cold container start;
/// host runs are exempt (a host pipe either works at once or the spawn throws). See
/// ``reportStalledSpawnIfSilent(stderrTail:)``.
static let firstOutputTimeoutNanos: UInt64 = 60 * 1_000_000_000
private let configuration: Configuration
private let processHost: ProcessHost
/// The per-backend approval server used for host runs and per-session sandbox containers.
@@ -111,6 +128,13 @@ public actor ClaudeCodeBackend: AgentBackend {
private var interruptRequested = false
private var terminating = false
private var sawRunFinished = false
/// True once the agent's first stdout line reaches the decode loop. The containerized spawn
/// watchdog disarms on this; while it's false past the deadline the run is a stalled spawn.
private var sawFirstOutput = false
/// Set when the spawn watchdog fired (no first output in time), so the post-loop exit handling
/// settles the run as a stalled-spawn error and doesn't also report the watchdog's own SIGKILL
/// as an abnormal/OOM exit.
private var spawnStalled = false
/// Whether the current run is in auto-approve mode (set per run). Edit tools are routed
/// through us even in auto mode for a conflict pre-check; on no-conflict they auto-allow.
private var autoApprove = false
@@ -214,6 +238,8 @@ public actor ClaudeCodeBackend: AgentBackend {
self.sessionID = run.sessionID
self.sawRunFinished = false
self.interruptRequested = false
self.sawFirstOutput = false
self.spawnStalled = false
self.autoApprove = run.autoApprove
do {
@@ -431,9 +457,25 @@ public actor ClaudeCodeBackend: AgentBackend {
// 5. Optional raw native capture (OBSERVABILITY A.5).
let capture = try makeCaptureFile()
// 5b. Spawn watchdog (containerized runs only). The first event lands within seconds of
// launch, so if NO stdout reaches the decode loop within `firstOutputTimeoutNanos`, the
// agent started but its stdio transport stalled surface a retryable error and kill the
// wedged process so the run settles, instead of spinning "Working" forever. Disarmed by
// the first decoded line below; cancelled unconditionally after the loop.
let watchdog: Task<Void, Never>? = (run.container != nil)
? Task { [weak self] in
try? await Task.sleep(nanoseconds: Self.firstOutputTimeoutNanos)
await self?.reportStalledSpawnIfSilent(stderrTail: stderrTail)
}
: nil
// 6. Decode loop the single consuming task of the native stream.
let decoder = ClaudeStreamDecoder()
for try await line in handle.stdoutLines {
if !sawFirstOutput {
sawFirstOutput = true
watchdog?.cancel() // output flows the spawn isn't stalled
}
capture?.append(line: line)
for decoded in decoder.decode(line: line) {
if case .sessionStarted(let started) = decoded.kind {
@@ -461,13 +503,16 @@ public actor ClaudeCodeBackend: AgentBackend {
// 7. Exit handling: if the CLI died without a result event, synthesize
// the run outcome (ADAPTERS §1.5).
watchdog?.cancel()
let exitCode = await handle.wait()
stderrTask.cancel()
capture?.close()
if !sawRunFinished {
// A nonzero exit is only an error if we didn't ask the process to stop
// (interrupt or session shutdown both end it deliberately).
if exitCode != 0 && !interruptRequested && !terminating {
// (interrupt or session shutdown both end it deliberately). A stalled spawn already
// emitted its own error and SIGKILLed the process, so don't also report that kill as
// an abnormal/OOM exit.
if exitCode != 0 && !interruptRequested && !terminating && !spawnStalled {
let message: String
// A SIGKILL inside the sandbox is ambiguous (real OOM vs. the whole container
// going down vs. another process) so probe the container now and report what
@@ -486,7 +531,8 @@ public actor ClaudeCodeBackend: AgentBackend {
emit(.error(AgentError(recoverable: false, message: message)), nativeType: nil)
}
let outcome: RunFinished.Outcome =
interruptRequested ? .interrupted
spawnStalled ? .errored
: interruptRequested ? .interrupted
: (terminating || exitCode == 0) ? .completed : .errored
emit(.runFinished(RunFinished(outcome: outcome)), nativeType: nil)
}
@@ -505,9 +551,11 @@ public actor ClaudeCodeBackend: AgentBackend {
// session settles to `awaitingInput` (resumable) instead of being thrown
// as an error, which `SessionController.consume` would turn into a terminal
// `.error` and lock the composer.
if interruptRequested || terminating {
if interruptRequested || terminating || spawnStalled {
if !sawRunFinished {
let outcome: RunFinished.Outcome = interruptRequested ? .interrupted : .completed
let outcome: RunFinished.Outcome =
spawnStalled ? .errored
: interruptRequested ? .interrupted : .completed
emit(.runFinished(RunFinished(outcome: outcome)), nativeType: nil)
}
await approvals.cancelOutstanding(reason: "Run interrupted")
@@ -537,6 +585,39 @@ public actor ClaudeCodeBackend: AgentBackend {
self.handle = nil
}
/// Spawn-watchdog handler (containerized runs). Fires `firstOutputTimeoutNanos` after launch; if
/// the agent STILL hasn't produced any stdout, the spawn has stalled the agent is running (it
/// reached the approval server) but its stdout/stdin isn't crossing the guesthost vsock relay,
/// so it sits idle forever and the UI spins "Working" with nothing to show. Convert that dead
/// end into an actionable, *recoverable* error (the session stays resumable) and SIGKILL the
/// wedged process group so its stdio closes, the decode loop ends, and the run settles. No-op if
/// output has since arrived or the run is already interrupting/terminating/finished.
private func reportStalledSpawnIfSilent(stderrTail: StderrTail) async {
guard !sawFirstOutput, !sawRunFinished, !interruptRequested, !terminating,
let handle
else { return }
spawnStalled = true
var message =
"The agent started inside the container but produced no output within "
+ "\(Self.firstOutputTimeoutNanos / 1_000_000_000)s. It reached the approval server, so "
+ "it is running, but its output never streamed back — the container's stdio transport "
+ "stalled. This is usually transient; run the turn again."
let tail = stderrTail.joined()
if !tail.isEmpty { message += "\n\nLast container stderr:\n\(tail)" }
if let name = activeContainerName, let containerManager {
let diagnosis = await containerManager.diagnoseContainerKill(name: name)
message += diagnosis.containerRunning
? "\n\n(The container is still running — this is a stdio relay stall, not an "
+ "out-of-memory kill.)"
: "\n\n(The container is no longer reachable — it may have stopped or crashed.)"
}
emit(.error(AgentError(recoverable: true, message: message)), nativeType: nil)
// Force the wedged process group down, then finish the line streams host-side so the decode
// loop ends (EOF) and the run settles even if the guest's wait/stdio RPC is itself stalled.
handle.sendSignal(SIGKILL)
handle.forceCloseStreams()
}
// MARK: - Approval bridge
private func handleApprovalCall(_ call: MCPApprovalServer.ApprovalCall) async
@@ -19,6 +19,8 @@ public final class ContainerizedProcessHandle: ProcessHandle, @unchecked Sendabl
private let process: LinuxProcess
private let stdin: StreamReader
private let stdoutWriter: StreamWriter
private let stderrWriter: StreamWriter
private let exitTask: Task<Int32, Never>
/// The guest process id. Best-effort: the framework exposes it once started.
@@ -38,6 +40,8 @@ public final class ContainerizedProcessHandle: ProcessHandle, @unchecked Sendabl
) {
self.process = process
self.stdin = stdin
self.stdoutWriter = stdoutWriter
self.stderrWriter = stderrWriter
self.stdoutLines = stdoutLines
self.stderrLines = stderrLines
// `LinuxProcess.wait()` resolves once the guest process exits; cache the code so the backend
@@ -86,6 +90,21 @@ public final class ContainerizedProcessHandle: ProcessHandle, @unchecked Sendabl
public func wait() async -> Int32 {
await exitTask.value
}
/// Finish the stdout/stderr line streams NOW, without waiting for the guest's wait RPC.
///
/// Normally the streams finish when `exitTask`'s `process.wait()` resolves and closes the writers.
/// But that wait rides the same guesthost vsock agent channel as the stdio relay, so if that
/// channel stalls (the bug behind "no output"), `wait()` can hang and the writers never close
/// leaving the backend's `for await stdoutLines` decode loop stuck forever, so a Stop kills the
/// process yet the session never settles. Calling this right after a force-kill closes the writers
/// host-side, ending the decode loop so the run always settles. Idempotent: the writers' `close()`
/// finishes each `AsyncThrowingStream` once, and any later guest write is yielded into an
/// already-finished stream (a no-op).
public func forceCloseStreams() {
try? stdoutWriter.close()
try? stderrWriter.close()
}
}
/// stdout/stderr sink handed to a `LinuxProcessConfiguration`. The framework calls `write(_:)` with
+22 -1
View File
@@ -36,6 +36,18 @@ public protocol ProcessHandle: Sendable {
func closeStdin()
func sendSignal(_ sig: Int32)
func wait() async -> Int32
/// Finish the stdout/stderr line streams immediately, independent of the process's exit, so a
/// consumer blocked on `for await stdoutLines` is released. Only meaningful when the streams can
/// outlive a killed process (the containerized handle, whose stream closure rides a guest RPC
/// that can stall); the host pipe handle EOFs reliably on exit, so its default is a no-op.
func forceCloseStreams()
}
extension ProcessHandle {
/// Default: nothing to force. The host `ChildProcess` reads OS pipes that EOF the moment the
/// process dies, so the line streams always finish on their own. ``ContainerizedProcessHandle``
/// overrides this because its streams close only when a guest wait RPC resolves.
public func forceCloseStreams() {}
}
/// One-shot claim guard: the first caller to `claim()` wins, the rest get `false`. Lets two racing
@@ -77,7 +89,10 @@ extension ProcessHandle {
sendSignal(SIGKILL)
// SIGKILL is uncatchable, so the process exits and `wait()` resolves; still bound it so a
// broken wait can never hang teardown.
_ = await waitForExit(within: graceNanos)
if await waitForExit(within: graceNanos) { return }
// Killed, but `wait()` still hasn't resolved the guest wait/stdio RPC is wedged. Finish the
// line streams ourselves so a consumer blocked on `stdoutLines` is released and teardown ends.
forceCloseStreams()
}
/// Best-effort escalation for an interactive Stop: SIGINT now (let the agent end its turn
@@ -90,6 +105,10 @@ extension ProcessHandle {
Task { [self] in
if await waitForExit(within: graceNanos) { return }
sendSignal(SIGKILL)
// If even the kill doesn't settle `wait()` (wedged guest RPC), release any consumer
// blocked on the line streams so Stop always takes effect.
if await waitForExit(within: graceNanos) { return }
forceCloseStreams()
}
}
@@ -102,6 +121,8 @@ extension ProcessHandle {
Task { [self] in
if await waitForExit(within: graceNanos) { return }
sendSignal(SIGKILL)
if await waitForExit(within: graceNanos) { return }
forceCloseStreams()
}
}
}
@@ -25,4 +25,84 @@ import Testing
#expect(lines.first?.allSatisfy { $0 == UInt8(ascii: "A") } == true)
#expect(lines.last.map { String(decoding: $0, as: UTF8.self) } == "done")
}
/// Stop must always settle, even when the process's `wait()` never resolves the case behind the
/// "inconsistent Stop" on containerized sessions, where the guest wait/stdio RPC can wedge so the
/// line streams never EOF and the decode loop hangs. `terminate` escalates SIGTERM SIGKILL and,
/// when the kill still doesn't resolve `wait()`, force-closes the streams so a consumer blocked on
/// `stdoutLines` is released.
@Test(.timeLimit(.minutes(1))) func terminateForceClosesStreamsWhenWaitNeverResolves() async {
let handle = StallingHandle()
let grace: UInt64 = 50_000_000 // 50ms
await handle.terminate(graceNanos: grace)
#expect(handle.didForceClose)
#expect(handle.deliveredSignals.contains(SIGTERM))
#expect(handle.deliveredSignals.contains(SIGKILL))
// The streams must be finished a consumer iterating them now completes instead of hanging.
var produced = 0
do { for try await _ in handle.stdoutLines { produced += 1 } } catch {}
#expect(produced == 0)
}
/// A responsive process that exits on the first signal must NOT be force-killed or have its
/// streams force-closed the escalation is only for a wedged wait.
@Test(.timeLimit(.minutes(1))) func terminateLeavesAResponsiveProcessAlone() async {
let handle = StallingHandle(exitsAfterFirstSignal: true)
await handle.terminate(graceNanos: 1_000_000_000)
#expect(handle.deliveredSignals == [SIGTERM]) // exited on SIGTERM; no SIGKILL escalation
#expect(!handle.didForceClose)
}
/// The MCP *connection* timeout is bounded (so an unreachable approval server can't wedge startup
/// for ~24.8 days) while the *tool* timeout stays effectively unbounded (human-answered approvals
/// may take many minutes). Locks in the asymmetry against an accidental re-pin of both to max.
@Test func mcpConnectionTimeoutIsBoundedButToolTimeoutIsNot() {
#expect(ClaudeCodeBackend.mcpTimeoutEnv["MCP_TIMEOUT"] == "60000")
#expect(ClaudeCodeBackend.mcpTimeoutEnv["MCP_TOOL_TIMEOUT"] == "2147483647")
}
}
/// A `ProcessHandle` whose `wait()` never resolves (unless told to exit on the first signal), used to
/// drive the Stop-escalation paths deterministically. Records the signals it received and whether its
/// streams were force-closed.
private final class StallingHandle: ProcessHandle, @unchecked Sendable {
let stdoutLines: AsyncThrowingStream<Data, Error>
let stderrLines: AsyncThrowingStream<Data, Error>
private let outCont: AsyncThrowingStream<Data, Error>.Continuation
private let errCont: AsyncThrowingStream<Data, Error>.Continuation
private let lock = NSLock()
private var signals: [Int32] = []
private var forceClosed = false
private let exitsAfterFirstSignal: Bool
init(exitsAfterFirstSignal: Bool = false) {
self.exitsAfterFirstSignal = exitsAfterFirstSignal
(stdoutLines, outCont) = AsyncThrowingStream.makeStream()
(stderrLines, errCont) = AsyncThrowingStream.makeStream()
}
var deliveredSignals: [Int32] { lock.withLock { signals } }
var didForceClose: Bool { lock.withLock { forceClosed } }
var processID: Int32 { 4242 }
func writeLine(_ data: Data) throws {}
func closeStdin() {}
func sendSignal(_ sig: Int32) { lock.withLock { signals.append(sig) } }
func wait() async -> Int32 {
if exitsAfterFirstSignal {
while lock.withLock({ signals.isEmpty }) {
try? await Task.sleep(nanoseconds: 5_000_000)
}
return 0
}
while true { try? await Task.sleep(nanoseconds: 50_000_000) } // never exits
}
func forceCloseStreams() {
lock.withLock { forceClosed = true }
outCont.finish()
errCont.finish()
}
}
+13 -4
View File
@@ -29,7 +29,15 @@ in-tree means the patch can't be lost to a dependency re-resolve.
which let a forked child survive a Stop in a long-lived shared container. Marked with
`[Nucleic vendored patch]`; used by `ContainerizedProcessHandle.sendSignal` in NucleicCore.
3. **Trimmed for footprint (no behavior change).** `Tests/`, `docs/`, `examples/`, and `images/`
3. **`Sources/Containerization/LinuxProcess.swift` — stdio-connection diagnostics (log-only).**
`setupIO` logs (`os.Logger`, subsystem `com.nucleic`, category `container-io`) when a *configured*
stdio stream's guest side never connects — which leaves its host `FileHandle` nil, so the relay /
readability handler is never wired and the agent's stdin is never delivered (it hangs) or its
stdout is never read (the "no output, just a spinner" symptom in Nucleic Control containers).
Behavior is unchanged; it only surfaces the failing stream. Marked `[Nucleic vendored patch]`
(the `import os`, the `nucleicIOLog` static, and the per-stream check in `setupIO`).
4. **Trimmed for footprint (no behavior change).** `Tests/`, `docs/`, `examples/`, and `images/`
were dropped, and the corresponding `.testTarget(...)` entries removed from `Package.swift`. The
library/executable targets we build are untouched.
@@ -38,9 +46,10 @@ in-tree means the patch can't be lost to a dependency re-resolve.
1. `git clone` upstream (or copy `.build/checkouts/containerization` after bumping the URL pin
temporarily), check out the desired commit.
2. `rsync -a --exclude=.git --exclude=.build --exclude=.swiftpm --exclude=Tests/ --exclude=docs/ \
--exclude=examples/ --exclude=images/ <upstream>/ third_party/containerization/`
--exclude=images/ <upstream>/ third_party/containerization/`
3. Remove the `.testTarget(...)` blocks from `third_party/containerization/Package.swift`.
4. Re-apply patch #1 (the `vmExtensions` field + the `vmConfig.extensions = …` forward) and patch
#2 (`LinuxProcess.killProcessGroup(_:)`). Grep for `[Nucleic vendored patch]` to find every site.
4. Re-apply patch #1 (the `vmExtensions` field + the `vmConfig.extensions = …` forward), patch #2
(`LinuxProcess.killProcessGroup(_:)`), and patch #3 (the `setupIO` stdio-connection log + its
`import os` / `nucleicIOLog`). Grep for `[Nucleic vendored patch]` to find every site.
5. Update the commit hash above and in the root `Package.swift` comment.
6. `swift build` and run the balloon tests.
@@ -21,10 +21,14 @@ import ContainerizationOS
import Foundation
import Logging
import Synchronization
import os // [Nucleic vendored patch] stdio-connection diagnostics
/// `LinuxProcess` represents a Linux process and is used to
/// setup and control the full lifecycle for the process.
public final class LinuxProcess: Sendable {
/// [Nucleic vendored patch] Diagnostic log for stdio stream-connection failures (see `setupIO`).
static let nucleicIOLog = os.Logger(subsystem: "com.nucleic", category: "container-io")
/// The ID of the process. This is purely metadata for the caller.
public let id: String
@@ -97,7 +101,7 @@ public final class LinuxProcess: Sendable {
private let agent: any VirtualMachineAgent
private let vm: any VirtualMachineInstance
private let ociRuntimePath: String?
private let logger: Logger?
private let logger: Logging.Logger? // [Nucleic vendored patch] disambiguated from os.Logger
private let onDelete: (@Sendable () async -> Void)?
init(
@@ -108,7 +112,7 @@ public final class LinuxProcess: Sendable {
ociRuntimePath: String?,
agent: any VirtualMachineAgent,
vm: any VirtualMachineInstance,
logger: Logger?,
logger: Logging.Logger?, // [Nucleic vendored patch] disambiguated from os.Logger
onDelete: (@Sendable () async -> Void)? = nil
) {
self.id = id
@@ -146,6 +150,18 @@ extension LinuxProcess {
}
}
// [Nucleic vendored patch] Diagnostics: a configured stdio stream whose guest side never
// connected leaves its host FileHandle `nil`, so the relay / readability handler below is
// never wired the agent's stdin is then never delivered (it hangs waiting for input) or
// its stdout is never read ("no output, just a spinner"). Log that specific failure (Console
// / `log show`, subsystem com.nucleic, category container-io) so a stall pinpoints the stream
// instead of proceeding silently. Log-only; behavior is unchanged.
let configured = [self.ioSetup.stdin != nil, self.ioSetup.stdout != nil, self.ioSetup.stderr != nil]
for (index, label) in [(0, "stdin"), (1, "stdout"), (2, "stderr")] where configured[index] && handles[index] == nil {
Self.nucleicIOLog.error(
"setupIO[\(self.id, privacy: .public)]: \(label, privacy: .public) stream never connected from the guest — agent stdio will stall")
}
// Note: stdin relay is started separately via startStdinRelay() after
// the process has started, to avoid a deadlock where closeStdin is
// called before the process is consuming from the pipe.