blakeslee-research/containerization
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Deep-sweep fixes: cooperative-pool starvation, stdio tail loss, epoll integrity, UI pins

Browse Source 
Host — the two remaining app-wide stall mechanisms plus main-thread pins
found by mining all nine hang reports:
- LinuxProcess.startStdinRelay wrote to a BLOCKING stdin fd on a
  width-limited cooperative-pool thread, non-cancellably; wedged guests
  starved the whole concurrency runtime (decode loops, watchdogs — an
  app-wide freeze surviving the reconcile fix). Writes now offload to a
  per-process GCD queue (vendored patch #18).
- TranscriptWriter (actor) did blocking write/fsync on the cooperative
  pool; it now runs on its own DispatchSerialQueue executor.
- UserMessageBubble's truncation probe typeset entire pasted-log-sized
  messages through CoreText per layout pass (100% main-thread pins in
  the 07-21 hang reports); certainly-long messages now skip the probe
  and render a prefix while collapsed.
- toolGroupSignature JSON-encoded every tool input in the transcript up
  to 12.5x/s on the MainActor; now a structural hash. The summary pass
  is trailing-throttled to 0.4s, and flatItems joins streaming chunks
  once instead of re-copying the prefix per delta.
- StatusFeedFetcher.parseDate allocated three formatters per call (86%
  of a pool thread in the 07-26 report); now shared statics.

Guest (vminitd) — teardown data loss and epoll registration hazards:
- IOPair no longer closes on a bare EPOLLHUP with a backpressure flush
  in flight (dropped the CLI's final output line); EPOLLOUT finishes the
  flush, then EOF closes loss-free. ManagedProcess.setExit closes only
  stdin, letting stdout/stderr self-close on EOF, with an 8s grace pass
  (patch #16).
- Epoll events carry a registration generation; the supervisor ignores
  stale events for recycled fd numbers. registerFd refuses EEXIST
  instead of clobbering the existing handler. TerminalIO's stdin relay
  writes a dup of the terminal fd so its backpressure registration
  can't collide with the stdout relay's (patch #17).
- VsockProxy flushes bytes parked toward the surviving peer on hangup,
  closes the dialing socket on a failed backend connect, and
  StandardIO/TerminalIO clean up partially-created pairs on setup
  failure (patch #16).

Full suite: 1451+292+74+20 tests, two failures — both pre-existing
environmental (MacVM base image absent on this machine; a load-flaky
liveness test that passes 3/3 in isolation).

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
abkslm
2026-07-28 15:35:35 -07:00
co-authored by Claude Fable 5
parent 6f950d859b
commit 17e84c9573
10 changed files with 358 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vminitd/Sources/VminitdCore/IOCloser.swift
+24
View File
@@ -16,6 +16,9 @@
#if os(Linux)
import ContainerizationOS
import Foundation
protocol IOCloser: Sendable {
var fileDescriptor: Int32 { get }
@@ -34,4 +37,25 @@ struct UnownedIOCloser: IOCloser {
func close() throws {}
}
/// [Nucleic vendored patch] Owns a `dup(2)` of another descriptor. Epoll keys registrations
/// on the fd NUMBER, so two IOPairs sharing one underlying file (TerminalIO: the stdout
/// relay reads the terminal fd, the stdin relay writes it) collide the moment the stdin
/// side needs an EPOLLOUT backpressure registration on the number the stdout side already
/// registered — which now fails fast (EEXIST) and killed the stdin relay. A dup shares the
/// open file description but has its own number, so each relay registers independently; the
/// description stays valid until every descriptor over it is closed.
struct DupIOCloser: IOCloser {
let fileDescriptor: Int32
init(duplicating fd: Int32) throws {
let duplicate = dup(fd)
guard duplicate != -1 else { throw POSIXError.fromErrno() }
self.fileDescriptor = duplicate
}
func close() throws {
guard Foundation.close(fileDescriptor) == 0 else { throw POSIXError.fromErrno() }
}
}
#endif
vminitd/Sources/VminitdCore/IOPair.swift
+10 -1
View File
@@ -180,7 +180,16 @@ final class IOPair: Sendable {
if mask.isHangup && !mask.readyToRead {
self.logger?.debug("received EPOLLHUP with no EPOLLIN")
if !ignoreHup {
// [Nucleic vendored patch] Never close on a bare HUP while a backpressure
// flush is in flight: the writer closing right after its final burst was
// stashed in `pending` (destination momentarily full) used to hit this
// close — whose single best-effort flush pass EAGAINed — and DROP the tail
// of the stream (the CLI's final result line). With pending outstanding,
// just return: the destination's EPOLLOUT edge flushes the backlog, the
// pump then reads the drained closed-writer pipe, observes EOF, and closes
// loss-free. If the destination dies instead, its own error/EPOLLERR wakes
// the write handler, whose failed flush closes the pair — no orphan.
if !ignoreHup && io.pending.isEmpty && !io.writeFdRegistered {
io.close(logger: self.logger)
}
return
vminitd/Sources/VminitdCore/ManagedProcess.swift
+25 -3
View File
@@ -27,7 +27,9 @@ import Synchronization
final class ManagedProcess: ContainerProcess, Sendable {
// swiftlint: disable type_name
protocol IO {
// [Nucleic vendored patch] Sendable so the exit path's delayed grace-close can capture
// the IO existential; both conformers (StandardIO, TerminalIO) already are.
protocol IO: Sendable {
func attach(pid: Int32, fd: Int32) throws
func start(process: inout Command) throws
func resize(size: Terminal.Size) throws
@@ -332,10 +334,30 @@ extension ManagedProcess {
let exitStatus = ContainerExitStatus(exitCode: status, exitedAt: Date.now)
state.exitStatus = exitStatus
// [Nucleic vendored patch] Close only stdin here. Force-closing ALL stdio at
// exit raced the relay: SIGCHLD is serviced before the poller thread drains the
// pipe's final EPOLLIN edge, and the old `io.close()` drain silently dropped
// whatever the destination wouldn't take (`drain` ignores short writes) — the
// CLI's final output line, truncated at process exit. The stdout/stderr write
// ends inside vminitd were already closed by `closeAfterExec`, so the child's
// exit delivers EOF/HUP to each relay, which self-closes loss-free (the relay
// never closes with a backpressure flush in flight). The delayed full close is
// a backstop for a grandchild that inherited the pipes and never exits, or a
// host that never drains — it must never fire on a healthy teardown path.
do {
try state.io.close()
try state.io.closeStdin()
} catch {
self.log.error("failed to close I/O for process: \(error)")
self.log.error("failed to close stdin for exited process: \(error)")
}
let io = state.io
let log = self.log
DispatchQueue.global(qos: .utility).asyncAfter(deadline: .now() + 8) {
// Idempotent: a relay already self-closed on EOF makes this a no-op.
do {
try io.close()
} catch {
log.error("failed to close I/O for exited process (grace pass): \(error)")
}
}
for waiter in state.waiters {
vminitd/Sources/VminitdCore/ProcessSupervisor.swift
+36 -7
View File
@@ -23,7 +23,18 @@ import Synchronization
final class ProcessSupervisor: Sendable {
private let poller: Epoll
private let handlers = Mutex<[Int32: @Sendable (Epoll.Mask) -> Void]>([:])
/// [Nucleic vendored patch] Handler table keyed by fd, each entry stamped with the
/// registration generation echoed back through epoll (`Epoll.Event.generation`).
/// Dispatch compares the event's generation against the live entry's, so an event
/// queued for a CLOSED registration of a recycled fd number can never fire the new
/// registration's handler (a stale EPOLLHUP used to be able to tear down a brand-new
/// healthy connection that inherited the number mid-batch).
private struct HandlerTable {
var nextGeneration: UInt32 = 1
var entries: [Int32: (generation: UInt32, handler: @Sendable (Epoll.Mask) -> Void)] = [:]
}
private let handlers = Mutex<HandlerTable>(HandlerTable())
private let queue: DispatchQueue
// `DispatchSourceSignal` is thread-safe.
@@ -58,8 +69,13 @@ final class ProcessSupervisor: Sendable {
return
}
for event in events {
let handler = self.handlers.withLock { $0[event.fd] }
handler?(event.mask)
// [Nucleic vendored patch] Dispatch only when the event belongs to the
// CURRENT registration of this fd number — an earlier handler in this
// batch may have closed the fd and something else re-registered the
// recycled number already (see HandlerTable).
let entry = self.handlers.withLock { $0.entries[event.fd] }
guard let entry, entry.generation == event.generation else { continue }
entry.handler(event.mask)
}
}
}
@@ -70,23 +86,36 @@ final class ProcessSupervisor: Sendable {
///
/// The handler is stored before the fd is added to epoll, ensuring no
/// events are missed.
///
/// [Nucleic vendored patch] Refuses (EEXIST) an fd that is already registered rather
/// than clobbering its handler: the old overwrite-then-fail-EEXIST path destroyed the
/// existing registration's handler AND removed the map entry, leaving the fd armed in
/// epoll with no handler — a silently dead relay (the TerminalIO shared-fd case).
func registerFd(
_ fd: Int32,
mask: Epoll.Mask = [.input, .output],
handler: @escaping @Sendable (Epoll.Mask) -> Void
) throws {
self.handlers.withLock { $0[fd] = handler }
let generation: UInt32 = try self.handlers.withLock { table in
guard table.entries[fd] == nil else { throw POSIXError(.EEXIST) }
let generation = table.nextGeneration
// 0 is reserved for the poller's internal eventFD registration.
table.nextGeneration = table.nextGeneration &+ 1
if table.nextGeneration == 0 { table.nextGeneration = 1 }
table.entries[fd] = (generation, handler)
return generation
}
do {
try self.poller.add(fd, mask: mask)
try self.poller.add(fd, mask: mask, generation: generation)
} catch {
self.handlers.withLock { _ = $0.removeValue(forKey: fd) }
self.handlers.withLock { _ = $0.entries.removeValue(forKey: fd) }
throw error
}
}
/// Remove a file descriptor from epoll monitoring and discard its handler.
func unregisterFd(_ fd: Int32) throws {
self.handlers.withLock { _ = $0.removeValue(forKey: fd) }
self.handlers.withLock { _ = $0.entries.removeValue(forKey: fd) }
try self.poller.delete(fd)
}
vminitd/Sources/VminitdCore/StandardIO.swift
+66 -58
View File
@@ -50,74 +50,82 @@ final class StandardIO: ManagedProcess.IO & Sendable {
func attach(pid: Int32, fd: Int32) throws {}
func start(process: inout Command) throws {
// [Nucleic vendored patch] All-or-nothing: a failure partway (a vsock connect or a
// relay registration throwing) used to discard the IO object with earlier pairs
// LIVE — the supervisor's handler map retains a registered IOPair forever, so a
// dead exec left a relay pumping host stdin into a pipe no child would ever read,
// plus its socket, pipe fds, and epoll slot. Dial each socket safely, and on any
// failure close every pair created so far before rethrowing.
try self.state.withLock {
if let stdinPort = self.hostStdio.stdin {
let inPipe = Pipe()
process.stdin = inPipe.fileHandleForReading
$0.stdinPipe = inPipe
let type = VsockType(
port: stdinPort,
cid: VsockType.hostCID
)
let stdinSocket = try Socket(type: type, closeOnDeinit: false)
try stdinSocket.connect()
let pair = IOPair(
readFrom: stdinSocket,
writeTo: inPipe.fileHandleForWriting,
reason: "StandardIO stdin",
logger: log
)
$0.stdin = pair
try pair.relay()
func dialHost(port: UInt32) throws -> Socket {
let type = VsockType(port: port, cid: VsockType.hostCID)
let socket = try Socket(type: type, closeOnDeinit: false)
do {
try socket.connect()
} catch {
try? socket.close()
throw error
}
return socket
}
do {
if let stdinPort = self.hostStdio.stdin {
let inPipe = Pipe()
process.stdin = inPipe.fileHandleForReading
$0.stdinPipe = inPipe
if let stdoutPort = self.hostStdio.stdout {
let outPipe = Pipe()
process.stdout = outPipe.fileHandleForWriting
$0.stdoutPipe = outPipe
let pair = IOPair(
readFrom: try dialHost(port: stdinPort),
writeTo: inPipe.fileHandleForWriting,
reason: "StandardIO stdin",
logger: log
)
$0.stdin = pair
let type = VsockType(
port: stdoutPort,
cid: VsockType.hostCID
)
let stdoutSocket = try Socket(type: type, closeOnDeinit: false)
try stdoutSocket.connect()
try pair.relay()
}
let pair = IOPair(
readFrom: outPipe.fileHandleForReading,
writeTo: stdoutSocket,
reason: "StandardIO stdout",
logger: log
)
$0.stdout = pair
if let stdoutPort = self.hostStdio.stdout {
let outPipe = Pipe()
process.stdout = outPipe.fileHandleForWriting
$0.stdoutPipe = outPipe
try pair.relay()
}
let pair = IOPair(
readFrom: outPipe.fileHandleForReading,
writeTo: try dialHost(port: stdoutPort),
reason: "StandardIO stdout",
logger: log
)
$0.stdout = pair
if let stderrPort = self.hostStdio.stderr {
let errPipe = Pipe()
process.stderr = errPipe.fileHandleForWriting
$0.stderrPipe = errPipe
try pair.relay()
}
let type = VsockType(
port: stderrPort,
cid: VsockType.hostCID
)
let stderrSocket = try Socket(type: type, closeOnDeinit: false)
try stderrSocket.connect()
if let stderrPort = self.hostStdio.stderr {
let errPipe = Pipe()
process.stderr = errPipe.fileHandleForWriting
$0.stderrPipe = errPipe
let pair = IOPair(
readFrom: errPipe.fileHandleForReading,
writeTo: stderrSocket,
reason: "StandardIO stderr",
logger: log
)
$0.stderr = pair
let pair = IOPair(
readFrom: errPipe.fileHandleForReading,
writeTo: try dialHost(port: stderrPort),
reason: "StandardIO stderr",
logger: log
)
$0.stderr = pair
try pair.relay()
try pair.relay()
}
} catch {
// IOPair.close is idempotent and closes both of a pair's fds; pairs whose
// relay registration never happened are closed the same way.
$0.stdin?.close()
$0.stdin = nil
$0.stdout?.close()
$0.stdout = nil
$0.stderr?.close()
$0.stderr = nil
throw error
}
}
}
vminitd/Sources/VminitdCore/TerminalIO.swift
+66 -25
View File
@@ -59,24 +59,45 @@ final class TerminalIO: ManagedProcess.IO & Sendable {
process.stdout = nil
process.stderr = nil
if let stdinPort = self.hostStdio.stdin {
let type = VsockType(
port: stdinPort,
cid: VsockType.hostCID
)
let stdinSocket = try Socket(type: type, closeOnDeinit: false)
try stdinSocket.connect()
$0.stdinSocket = stdinSocket
}
// [Nucleic vendored patch] Close whatever connected on a partial failure —
// these sockets are closeOnDeinit: false, so a discarded IO object would leak
// the earlier fd in PID-1.
do {
if let stdinPort = self.hostStdio.stdin {
let type = VsockType(
port: stdinPort,
cid: VsockType.hostCID
)
let stdinSocket = try Socket(type: type, closeOnDeinit: false)
do {
try stdinSocket.connect()
} catch {
try? stdinSocket.close()
throw error
}
$0.stdinSocket = stdinSocket
}
if let stdoutPort = self.hostStdio.stdout {
let type = VsockType(
port: stdoutPort,
cid: VsockType.hostCID
)
let stdoutSocket = try Socket(type: type, closeOnDeinit: false)
try stdoutSocket.connect()
$0.stdoutSocket = stdoutSocket
if let stdoutPort = self.hostStdio.stdout {
let type = VsockType(
port: stdoutPort,
cid: VsockType.hostCID
)
let stdoutSocket = try Socket(type: type, closeOnDeinit: false)
do {
try stdoutSocket.connect()
} catch {
try? stdoutSocket.close()
throw error
}
$0.stdoutSocket = stdoutSocket
}
} catch {
if let stdinSocket = $0.stdinSocket {
try? stdinSocket.close()
$0.stdinSocket = nil
}
throw error
}
}
}
@@ -98,13 +119,23 @@ final class TerminalIO: ManagedProcess.IO & Sendable {
$0.parent = term
if let stdinSocket = $0.stdinSocket {
// [Nucleic vendored patch] The stdin relay's destination is a dup of the
// terminal fd, NOT the terminal fd itself: both relays sharing one number
// meant the stdin side's EPOLLOUT backpressure registration collided with
// the stdout side's read registration (see DupIOCloser) — one bulk paste
// permanently killed the terminal's stdout relay.
let pair = IOPair(
readFrom: stdinSocket,
writeTo: UnownedIOCloser(term),
writeTo: try DupIOCloser(duplicating: term.fileDescriptor),
reason: "TerminalIO stdin",
logger: log
)
try pair.relay(ignoreHup: true)
do {
try pair.relay(ignoreHup: true)
} catch {
pair.close()
throw error
}
$0.stdin = pair
}
@@ -115,7 +146,17 @@ final class TerminalIO: ManagedProcess.IO & Sendable {
reason: "TerminalIO stdout",
logger: log
)
try pair.relay(ignoreHup: true)
do {
try pair.relay(ignoreHup: true)
} catch {
// [Nucleic vendored patch] The stdin pair (and its relay) is already
// live; a discarded IO object would leave it registered and pumping
// forever (the supervisor's handler map retains it).
pair.close()
$0.stdin?.close()
$0.stdin = nil
throw error
}
$0.stdout = pair
}
}
@@ -123,11 +164,11 @@ final class TerminalIO: ManagedProcess.IO & Sendable {
func close() throws {
self.state.withLock {
// stdout must close before stdin because both IOPairs share the
// Terminal fd. stdout registered that fd with epoll (as its read
// source) and needs to unregister it while the fd is still valid.
// stdin closes the Terminal as its write destination, which would
// invalidate the fd before stdout can unregister.
// stdout closes first: it registered the Terminal fd with epoll (as its read
// source) and unregisters it while the fd is still valid. The stdin pair's
// write destination is its own dup of the terminal (see attach), so its
// close-time flush stays valid regardless of ordering — the shared open file
// description outlives the stdout side's close until the dup closes too.
if let stdout = $0.stdout {
stdout.close()
$0.stdout = nil
vminitd/Sources/VminitdCore/VsockProxy.swift
+24 -1
View File
@@ -242,7 +242,16 @@ extension VsockProxy {
)
}
try relayTo.connect()
// [Nucleic vendored patch] A failed backend connect must close the socket it
// was dialing from: `closeOnDeinit` is false, so throwing out of here (the
// caller closes only `conn`) leaked one PID-1 fd per attempt while the
// backend was down — sustained control-plane churn walked toward EMFILE.
do {
try relayTo.connect()
} catch {
try? relayTo.close()
throw error
}
// [Nucleic vendored patch] BOTH fds must be non-blocking BEFORE either is
// registered. `Epoll.add` sets O_NONBLOCK only at registration time, and the first
@@ -365,6 +374,15 @@ extension VsockProxy {
}
if mask.isHangup {
// Full hangup of the client. Before tearing down, make one best-effort
// pass toward the SURVIVING peer: bytes already read off the client may
// be parked in toServer's pipe (its earlier flush EAGAINed), and the
// server is still healthy — dropping them loses a delivered-to-us
// control message. One pass only (no spin risk: a still-full server
// just returns and cleanup proceeds).
if !toServerDone {
_ = apply(Self.relayStep(&toServer, description: "client:hangup:toServer", log: self.log))
}
toServerDone = true
toClientDone = true
} else if mask.isRemoteHangup && !toServerDone {
@@ -404,6 +422,11 @@ extension VsockProxy {
}
if mask.isHangup {
// Mirror of the client handler: flush bytes parked toward the
// surviving client before teardown.
if !toClientDone {
_ = apply(Self.relayStep(&toClient, description: "server:hangup:toClient", log: self.log))
}
toServerDone = true
toClientDone = true
} else if mask.isRemoteHangup && !toClientDone {