Async child-process management for Rust + tokio with a kernel-backed no-orphan guarantee: every process you start — and everything it spawns — lives in a kill-on-drop container (a Windows Job Object, a Linux cgroup v2, or a POSIX process group), so no descendant ever outlives your program.

Beyond spawning a subprocess: run-and-capture, line streaming, interactive stdin, real pseudo-terminal (PTY) sessions, shell-free pipelines, readiness probes, timeouts & cancellation, supervision with restart/backoff, and a mockable runner seam for subprocess-free tests.

cargo add processkit

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let version = Command::new("cargo").arg("--version").run().await?;
    println!("{version}");
    Ok(())
}

Continue: find a recipe · learn the command model · browse the API

Why processkit?

std::process and tokio::process reach (at most) the direct child. The processes it spawned — a build tool's compiler children, the real payload behind a wrapper (cmd /c …, sh -c …), a test's helper servers — can survive a timeout, a panic, or a dropped future and keep running as orphans.

processkit puts every child inside the operating system's own containment primitive. Teardown is one operation over the whole process tree, not a best-effort signal to one PID:

• Nothing escapes silently. Dropping a run or group reaps every descendant; the active Mechanism tells you which OS guarantee is in effect.
• Terminal when you need one. The opt-in pty feature gives terminal-only tools a real controlling terminal while keeping the same containment and kill-on-drop lifecycle.
• Honest outcomes. A non-zero exit remains inspectable data until you ask for success; timeouts, cancellation, spawn failures, and resource limits stay distinct in the typed error model.
• Async and testable. Streaming, pipelines, readiness, and supervision are tokio-native, while the ProcessRunner seam replaces real processes with scripted or record/replay doubles in tests.

3.0 highlights

• PTY support is here. With the opt-in pty feature, Command::use_pty() gives terminal-only tools a real controlling terminal: openpty on Unix and CreatePseudoConsole (ConPTY) on Windows. The terminal has one merged output stream, supports interactive input plus initial/live window sizing, and stays inside the same kill-on-drop containment as a piped run. Start with the PTY dialog and window-size sections, then check the platform matrix.
• Errors are split by responsibility. Error is now a pointer-sized wrapper around Box<ErrorReason>; its existing accessors remain the convenient path, reason() exposes the structured variant, and ErrorKind is the coarse total classification for routing. See Errors for the model and Upgrading for the 2.x migration.
• Lifecycle streaming is end to end. The renamed events() stream carries ProcessEvent::Started → ordered output → ProcessEvent::Exited, while Pipeline::start() and Supervisor::start() expose live sessions for observing and stopping multi-stage or restarted workloads. Continue with lifecycle events, live pipelines, and live supervision.
• Tree shutdown and ownership are explicit. ProcessGroup::stop() returns a ShutdownReport, update_limits() changes resource caps on a live group, and spawn_detached() is a separate opt-in for the exceptional child that must outlive its launcher. See observable teardown, live limits, and detached children.
• Batch streaming and capture are production-ready. Completion-order output_stream fan-out no longer holds fast results behind slow work; byte-accurate raw tees expose the undecoded stream; and CapturePolicy redacts or reshapes retained lines before they enter a result. See results as they finish, raw output, and redaction at capture.
• The host is inspectable before and after launch. host_containment() reports the effective containment mechanism without spawning; process_info() / process_is_alive() provide reuse-safe PID identity checks; and the new metrics feature publishes secret-safe counters and histograms. See Platform support, process identity, and Observability.

The complete release record is in the v3.0.0 release.

Guides

New to the crate? Start with the Cookbook — short task-to-snippet recipes for everything the crate does — then read Running commands end to end (it's the vocabulary every other guide builds on). Reach for the rest as the need arises, and keep Platform support handy before you ship: it collects every per-OS caveat in one place.

Feature flags

Every flag is additive and only gates visibility — the kill-on-drop tree guarantee is unconditional in every configuration.

[dependencies]
processkit = { version = "3", features = ["limits"] }

The 60-second tour

use processkit::{Command, ProcessGroup, Stdin};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // One-shot: capture everything. A non-zero exit is data, not an Err.
    let head = Command::new("git").args(["rev-parse", "HEAD"]).output_string().await?;
    println!("HEAD = {}", head.stdout().trim());

    // Success-checking: non-zero exit / timeout / signal-kill become typed errors.
    let version = Command::new("cargo").arg("--version").run().await?;

    // Stdin, timeout, streaming, pipelines, supervision … see the guides.
    let sorted = Command::new("sort")
        .stdin(Stdin::from_string("b\na\n"))
        .timeout(std::time::Duration::from_secs(5))
        .run()
        .await?;

    // Containment: anything spawned through a group dies with it.
    let group = ProcessGroup::new()?;
    let _server = group.start(&Command::new("dev-server")).await?;
    drop(group); // the server — and everything *it* spawned — is reaped

    let _ = (version, sorted);
    Ok(())
}

API reference

The rustdoc on docs.rs is the authoritative per-item reference; these guides are the narrative layer on top — they explain how the pieces compose, with the platform fine print collected in Platform support.

These examples are compiler-checked. Every fenced Rust block across these guides and the root README.md is compiled (and, unless annotated no_run or ignore, actually run) as an ordinary doctest by cargo test --all-features (as CI does) — a signature change that stops matching a guide's snippet fails CI instead of silently lying to a reader. The hidden harness only builds under --all-features, so a plain cargo test with the default features does not exercise this check. See src/doc_examples.rs for the (test-only, hidden) harness.

Cookbook

‹ docs index

Task-oriented recipes: find the thing you're trying to do, copy the snippet, follow the link when you need the fine print. Every snippet assumes a tokio runtime and use processkit::Command; unless shown otherwise.

• Run a command and get its output
• Inspect a failure instead of erroring
• Ask a yes/no question
• Accept non-zero exit codes as success
• Bound a run with a timeout
• Let a tool clean up on timeout
• Show a useful error message
• Check a tool is installed without running it
• Feed the child's stdin
• Stream output as it arrives
• Talk to an interactive child
• Run a tool that requires a terminal
• Answer an unterminated PTY prompt
• Read in-place progress from an agent CLI
• Avoid PTY full-duplex deadlocks
• Test PTY behavior without a terminal
• Driving ssh
• Pipe commands without a shell
• Start a server and wait until it's ready
• Tear down several children as a unit
• React to whichever child exits first
• Sandbox an untrusted tool
• Keep a crash-prone service running
• Retry a flaky command
• Cancel runs on shutdown
• Measure what a run cost
• Contain a process you didn't spawn
• Test code that runs processes — without processes
• Test streaming code — without processes
• Wrap a CLI tool behind a typed API

Run a command and get its output

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let head = Command::new("git").args(["rev-parse", "HEAD"]).run().await?;
    Ok(())
}

run() requires a zero exit and returns stdout with trailing whitespace trimmed; a non-zero exit, spawn failure, or timeout is a typed Error. For a one-liner without the builder: processkit::run("git", ["rev-parse", "HEAD"]).

Fine print: Running commands → consuming verbs.

Inspect a failure instead of erroring

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("git").args(["merge", "topic"]).output_string().await?;
    if !result.is_success() {
        eprintln!("merge exited {:?}: {}", result.code(), result.stderr());
    }
    Ok(())
}

output_string() (and output_bytes() for raw bytes) treats the exit code as data — Err means the run couldn't happen at all. Call result.ensure_success()? later to convert a stored failure into the same typed error run() would have produced.

Fine print: Running commands → results and errors.

Ask a yes/no question

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let dirty = !Command::new("git").args(["diff", "--quiet"]).probe().await?;
    Ok(())
}

probe() maps exit 0 → true, exit 1 → false, and anything else to an error — the git diff --quiet / grep -q convention without manual code matching.

Accept non-zero exit codes as success

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // `grep` exits 1 when it finds no match — not a failure for this call.
    let found = Command::new("grep")
        .args(["needle", "haystack.txt"])
        .ok_codes([0, 1])
        .output_string()
        .await?;
    let matched = found.code() == Some(0); // 0 = matched, 1 = no match (both "success")
    Ok(())
}

ok_codes widens what the checking verbs (run/run_unit) and is_success/ensure_success treat as success — for tools whose non-zero exit is a normal result (grep 1 = no match, diff 1 = differs, rsync's code families). It does not change exit_code (always the raw code) or probe (always the 0/1 convention). An empty set is ignored, so the default stays exit 0.

Bound a run with a timeout

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("slow-tool")
        .timeout(Duration::from_secs(30))
        .output_string()
        .await?;
    if result.timed_out() {
        eprintln!("gave up after 30s; partial output: {}", result.stdout());
    }
    Ok(())
}

At the deadline ProcessKit attempts whole-tree teardown. Once terminal state is confirmed, capture verbs keep the timeout as data (timed_out(), partial output kept) and success-checking verbs (run, exit_code) surface ErrorReason::Timeout. A rejected kill/escalation/reap that leaves the tree potentially live is ErrorReason::Teardown instead.

Let a tool clean up on timeout

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("dev-server")
        .timeout(Duration::from_secs(30))
        .timeout_grace(Duration::from_secs(5)) // SIGTERM, wait up to 5s, then SIGKILL
        .output_string()
        .await?;
    Ok(())
}

timeout_grace turns the hard deadline kill into a graceful one: SIGTERM (or the signal from timeout_signal, with the process-control feature), up to the grace window to exit, then SIGKILL. A signal-handling child exits early; timed_out() stays true. Windows has no signal tier — the deadline kills atomically.

Fine print: Timeouts → graceful timeout.

Fine print: Timeouts, retries & cancellation.

Show a useful error message

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    if let Err(e) = Command::new("git").args(["merge", "topic"]).run().await {
        eprintln!("merge failed: {}", e.diagnostic().unwrap_or("(no output)"));
    }
    Ok(())
}

Error::diagnostic() picks the most explanatory captured text — stderr, falling back to stdout (git writes CONFLICT … there) — so callers don't re-implement the same heuristic.

Check a tool is installed without running it

use processkit::Command;

fn main() {
    // The crate-level shortcut for a bare tool:
    match processkit::which("git") {
        Ok(path) => println!("git is at {}", path.display()),
        Err(e) if e.is_not_found() => eprintln!("git is not installed"),
        Err(e) => eprintln!("could not resolve git: {e}"),
    }

    // On a builder it honors that command's `prefer_local` / env, resolving
    // exactly what a real run would launch:
    let _ = Command::new("eslint")
        .prefer_local("./node_modules/.bin")
        .resolve_program();
}

which / Command::resolve_program() locate a program without spawning it — a side-effect-free doctor / preflight check ("is the tool installed?") for a friendly up-front error. Resolution reuses the crate's own launch-path logic (the same PATH/PATHEXT/execute-bit and prefer_local handling a real run uses), so a hit is exactly what would be launched and a miss is exactly the ErrorReason::NotFound (is_not_found()) a run would raise. It is synchronous — no tokio runtime needed. A wrapped tool's client offers the same via CliClient::resolve_program().

Fine print: Running commands → preflight.

Feed the child's stdin

use processkit::Command;
use processkit::Stdin;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // A string you already have:
    let sorted = Command::new("sort")
        .stdin(Stdin::from_string("banana\napple\n"))
        .run()
        .await?;

    // …or any async source: a reader (file, socket) or a stream of lines.
    let from_file = Stdin::from_reader(tokio::fs::File::open("input.txt").await?);
    let from_chan = Stdin::from_lines(tokio_stream::iter(vec!["one".to_owned()]));
    let _ = (sorted, from_file, from_chan);
    Ok(())
}

One-shot sources (from_reader/from_lines) feed a single run; re-running the same Command afterwards fails loud (an ErrorReason::Io at launch, D10) instead of silently seeing empty stdin. For a conversation, see the next recipe but one.

Fine print: Running commands → standard input.

Stream output as it arrives

use processkit::Command;
use processkit::Finished;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    use processkit::prelude::StreamExt; // re-exported; provides `.next()`

    let mut run = Command::new("cargo").args(["build", "--verbose"]).start().await?;
    let mut lines = run.stdout_lines()?;
    while let Some(line) = lines.next().await {
        println!("build: {line}");
    }
    let Finished { outcome, stderr, .. } = run.finish().await?; // outcome + buffered stderr
    Ok(())
}

No waiting for exit, no full-output buffering; stderr is drained in the background so the child can't block. A timeout on the command bounds the stream itself. Prefer a callback? .on_stdout_line(|l| …) runs one per line while any capture verb drives the run.

Fine print: Streaming & interactive I/O.

Forward a child's exact bytes (binary passthrough)

use processkit::Command;
use tokio::fs::File;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exact = File::create("archive.tar").await?;
    let outcome = Command::new("git")
        .args(["archive", "HEAD"]) // binary tar on stdout
        .stdout_raw_tee(exact)     // exact bytes, before any decoding
        .start()
        .await?
        .drain()                   // stream through, retain nothing
        .await?;
    println!("done: {outcome:?}");
    Ok(())
}

stdout_raw_tee / stderr_raw_tee write each chunk exactly as read from the pipe — no decoding, no CRLF rewrite, no line splitting, no lost tail — so non-UTF-8 output (git archive, tar -cz -, ffmpeg … -) survives byte for byte. It runs alongside the decoded sinks (stdout_lines, on_stdout_line, stdout_tee), so you can forward the raw stream and react to decoded lines at once. For the whole output in memory instead of streamed through, output_bytes() returns the exact bytes directly.

Fine print: Streaming → byte-accurate raw output.

Talk to an interactive child

use processkit::Command;
use processkit::prelude::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut run = Command::new("bc").keep_stdin_open().start().await?;
    let mut stdin = run.take_stdin().expect("stdin was kept open");
    stdin.write_line("2 + 2").await?;
    stdin.finish().await?; // EOF — bc exits

    let mut answers = run.stdout_lines()?;
    while let Some(answer) = answers.next().await {
        println!("{answer}");
    }
    Ok(())
}

keep_stdin_open() hands you an async writer instead of closing stdin at spawn; interleave writes with reads for request/response tools. Its writer methods return std::io::Result (idiomatic for a writer) — convert with .map_err(processkit::ErrorReason::Io)? in a processkit::Result function, or use Box<dyn std::error::Error>.

Fine print: Streaming & interactive I/O → interactive stdin.

Run a tool that requires a terminal

Some tools change behavior behind isatty() or refuse to start without a controlling terminal. Enable the pty feature, opt this run into PTY mode, and read its output through the usual capture verbs:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("terminal-only-tool")
        .arg("--status")
        .use_pty()
        .pty_size(120, 30)
        .output_string()
        .await?;

    // A PTY has one merged output stream. Everything the child wrote to its
    // terminal is captured as stdout; the separate stderr result is empty.
    println!("{}", result.stdout());
    assert!(result.stderr().is_empty());
    Ok(())
}

use_pty() changes the transport, not the lifecycle: timeout, cancellation, containment, and kill-on-drop behave as for a piped child. It is a minimal terminal session rather than a terminal emulator; set pty_size when wrapping and layout matter.

Fine print: Streaming → PTY window size and live resize · Platform support → PTY mode.

Answer an unterminated PTY prompt

Terminal prompts commonly end in ": " rather than a newline. Use wait_for_output, not wait_for_line, to match that live partial tail; then answer over the same PTY master:

use processkit::Command;
use std::time::Duration;

async fn authenticate(secret: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut run = Command::new("sudo")
        .args(["-k", "-v"])
        .use_pty()
        .keep_stdin_open()
        .start()
        .await?;

    run.wait_for_output(
        |tail| tail.contains("Password:"),
        Duration::from_secs(10),
    )
    .await?;

    let mut stdin = run.take_stdin().expect("PTY stdin was kept open");
    stdin.write_line(secret).await?;
    run.finish().await?;
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    authenticate("secret supplied by a protected source").await
}

On Unix, processkit disables terminal echo for its PTY slave, so the secret is not copied into merged output. That echo-off guarantee is Unix-only: ConPTY has no portable per-session equivalent, so do not retain or log the merged transcript when sending a secret on Windows. wait_for_output is byte-driven and works for ordinary prompts on both platforms; it does not need a newline or a signal from the child.

Fine print: Streaming → prompt-aware waiting · Running commands → interactive auth / TTY.

Read in-place progress from an agent CLI

Agent CLIs often require a terminal, redraw progress with bare carriage returns, and decorate it with VT/ANSI control sequences. Frame each redraw as a line and sanitize the captured text before inspecting it:

use processkit::prelude::StreamExt;
use processkit::{Command, LineTerminator};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut run = Command::new("agent")
        .arg("run")
        .use_pty()
        // PTY output is the logical stdout stream. `line_terminator(...)`
        // is the equivalent whole-command spelling.
        .stdout_line_terminator(LineTerminator::CarriageReturn)
        .stdout_sanitize_vt()
        .start()
        .await?;

    let mut frames = run.stdout_lines()?;
    while let Some(frame) = frames.next().await {
        println!("agent: {frame}");
    }
    run.finish().await?;
    Ok(())
}

PTY mode already defaults to carriage-return-aware framing; spelling it out is useful when this behavior is part of your wrapper's contract. Use line_terminator and sanitize_vt to configure both logical streams when the same builder may also run without PTY. VT sanitization is destructive and affects captured text only; exact-byte output and raw tees remain untouched.

Fine print: Streaming → PTY output hygiene · Untrusted children → content transforms.

Avoid PTY full-duplex deadlocks

A PTY master, like a pair of pipes, has finite kernel buffers. Do not await a large write while leaving output unread: the child may fill its output buffer, stop reading input, and leave both sides parked. Drain and write concurrently:

use processkit::prelude::StreamExt;
use processkit::Command;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let payload = vec![b'x'; 4 * 1024 * 1024];
    let mut run = Command::new("terminal-filter")
        .use_pty()
        .keep_stdin_open()
        .start()
        .await?;
    let mut stdin = run.take_stdin().expect("PTY stdin was kept open");
    let mut output = run.stdout_lines()?;

    let write = async move {
        stdin.write(&payload).await?;
        stdin.finish().await
    };
    let drain = async move {
        while let Some(line) = output.next().await {
            println!("{line}");
        }
        Ok::<(), std::io::Error>(())
    };

    tokio::try_join!(write, drain)?;
    run.finish().await?;
    Ok(())
}

Also remember that a PTY has no independent stderr channel. Both child file descriptors arrive through stdout_lines/on_stdout_line; on_stderr_line is not delivered, stderr_tee has nothing to write, and the finished result's stderr is empty. Keep both destinations StdioMode::Piped: PTY mode rejects Inherit, Null, and file redirects for either output descriptor before spawn. If stream identity or another destination matters, use ordinary pipes instead of PTY.

Fine print: Streaming → interactive stdin · Running commands → interactive auth / TTY.

Test PTY behavior without a terminal

The ScriptedRunner PTY variant is selected by putting use_pty() on the scripted command. It hermetically models the merged master and PTY line framing, so CI needs neither a real terminal nor a platform-specific helper:

use processkit::prelude::StreamExt;
use processkit::testing::{Reply, ScriptedRunner};
use processkit::{Command, LineTerminator, ProcessRunner};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let runner = ScriptedRunner::new().fallback(
        Reply::ok("working 10%\rworking 100%\r")
            .with_stderr("final diagnostic"),
    );
    let command = Command::new("agent")
        .use_pty()
        .line_terminator(LineTerminator::CarriageReturn);

    let mut run = runner.start(&command).await?;
    let mut merged = run.stdout_lines()?;
    let mut seen = Vec::new();
    while let Some(frame) = merged.next().await {
        seen.push(frame);
    }
    let finished = run.finish().await?;

    assert!(seen.iter().any(|line| line == "working 100%"));
    assert!(seen.iter().any(|line| line == "final diagnostic"));
    assert!(finished.stderr.is_empty());
    Ok(())
}

For dialogs, script Reply::dialog("Password: ", "accepted\n"); it emits the unterminated prompt, waits for take_stdin() to receive an answer, then emits the continuation. This exercises wait_for_output without subprocess timing.

Fine print: Testing → scripted streaming · Streaming → prompt-aware waiting.

Driving ssh

ssh is the one tool worth its own recipe. It often demands a terminal (password/passphrase prompts), and — alone among the tools you'll launch — it spawns work on another host, past anything kill-on-drop can reach. Two things to get right: how you authenticate, and what "contained" does and doesn't cover.

Prefer the non-interactive path — key auth, BatchMode=yes, ordinary pipes. With key-based auth and BatchMode=yes, ssh never prompts, so a plain run captures the remote command's output like any local one:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("ssh")
        .args(["-o", "BatchMode=yes", "deploy@host", "systemctl is-active app"])
        .output_string()
        .await?;

    match result.code() {
        // 255 is ssh's OWN failure (host unreachable, auth/BatchMode rejected) —
        // NOT the remote command's exit; don't read it as the remote result.
        Some(255) => eprintln!("ssh could not connect: {}", result.stderr()),
        // Any other code came straight through from the remote command.
        Some(code) => println!("remote `systemctl is-active` exited {code}"),
        None => eprintln!("ssh was signalled or timed out on this side"),
    }
    Ok(())
}

Exit code 255 is ssh's own "connection/auth failed", not the remote command's code. ssh forwards the remote command's real exit status as its own — except 255, which it reserves for its own failures (host unreachable, auth rejected, BatchMode with no usable key). A caller that cares about the remote side must special-case 255 before trusting the code, as above. The one residual ambiguity ssh can't resolve for you: a remote command that itself exits 255 looks identical to a connection failure.

The remote command line is parsed by the remote shell — the crate's shell-free guarantee stops at the local process. processkit runs the local ssh with no shell: its argv is passed literally, with no interpolation and no injection surface (the crate's shell-free property). But ssh host "some command" hands that command string to a login shell on the far end, which word-splits and expands it like any shell line. So any value you interpolate into a remote command needs manual quoting/escaping (or build it so no untrusted value reaches the remote shell at all) — the injection surface the local API removes reappears on the remote side.

Password / passphrase / host-key prompts need a PTY. When key-auth isn't an option, use_pty() (the pty feature) gives ssh the terminal it insists on; drive the prompt over the merged master with keep_stdin_open + take_stdin:

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut run = Command::new("ssh")
        .args(["deploy@host", "systemctl status app"])
        .use_pty()          // real terminal (needs the `pty` feature) so ssh prompts
        .keep_stdin_open()
        .start()
        .await?;

    // Match the live, unterminated prompt tail; no newline is required.
    run.wait_for_output(
        |tail| tail.contains("passphrase"),
        Duration::from_secs(10),
    )
    .await?;
    let mut stdin = run.take_stdin().expect("stdin kept open");
    stdin.write_line("s3cr3t-passphrase").await?;

    // stdout and stderr are MERGED onto the one master in PTY mode — read the
    // rest of the exchange from run.stdout_lines().
    Ok(())
}

wait_for_output sees the un-terminated prompt tail that wait_for_line deliberately cannot. The same platform echo caveat as the general PTY prompt recipe applies: echo is disabled by processkit on Unix, but not portably controllable through ConPTY.

The containment boundary stops at the local ssh client. kill-on-drop, timeout, and cancellation reap the local process tree — here, the ssh client itself. They do not reach across the connection: dropping the handle (or a timeout, or a panic) severs the local ssh client, but the command it started on the remote host can keep running, orphaned. The crate's whole-tree guarantee is about your machine's process tree; an ssh hop is a boundary it cannot cross. Contain the remote side on the remote side:

• ssh -tt forces a remote pseudo-terminal, so when the connection drops the remote session receives SIGHUP — which ends many (not all) remote programs.
• A server-side deadline — ssh host 'timeout 300 long-job', or the service's own timeout/idle limit — bounds the remote work regardless of the local client's fate.
• For anything that must be torn down reliably, make the remote command own its own lifecycle (a unit/scope with a deadline, a job the remote scheduler can kill), not the ssh client's liveness.

Fine print: Running commands → interactive auth · Running untrusted children → what not to rely on.

Pipe commands without a shell

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let authors = Command::new("git").args(["log", "--format=%an"])
        .pipe(Command::new("sort"))
        .pipe(Command::new("uniq").arg("-c"))
        .output_string()
        .await?;
    Ok(())
}

Native pipes — no shell string, no quoting, no injection surface. The outcome is pipefail: stdout comes from the last stage, the reported failure from the first stage that didn't exit cleanly. All stages share one kill-on-drop group. The | operator is equivalent sugar: (a | b | c).output_string().

For a consumer that legitimately stops reading early — the | head -1 shape, where the producer's broken-pipe death (its next write fails once the downstream closes, or SIGPIPE where the OS delivers it) is expected — mark the producer unchecked_in_pipe() so that death doesn't fail the chain:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let first = (Command::new("seq").args(["1", "1000000"]).unchecked_in_pipe()
        | Command::new("head").args(["-n", "1"]))
        .run()
        .await?;
    Ok(())
}

Fine print: Pipelines → unchecked stages.

Start a server and wait until it's ready

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut server = Command::new("my-server").args(["--port", "8080"]).start().await?;

    // Pick the probe that matches how the server announces readiness:
    server.wait_for_line(|l| l.contains("listening"), Duration::from_secs(10)).await?;
    // server.wait_for_port("127.0.0.1:8080".parse().unwrap(), Duration::from_secs(10)).await?;
    // server.wait_for_http("127.0.0.1:8080".parse().unwrap(), "/healthz", |s| (200..300).contains(&s), Duration::from_secs(10)).await?;
    // server.wait_for_socket("/tmp/my-server.sock", Duration::from_secs(10)).await?; // Unix only
    // server.wait_for_pipe("my-server", Duration::from_secs(10)).await?; // Windows only
    // server.wait_for(|| async { https_or_body_health().await }, Duration::from_secs(10)).await?;

    // …use the server; dropping `server` kills its whole tree.
    Ok(())
}

A probe that can't succeed fails fast with ErrorReason::NotReady and never kills the child — you decide what happens next. No more sleep(2) and hoping. wait_for_http is intentionally plain HTTP and status-only; use the generic predicate with your own client when TLS, redirects, or response bodies matter.

Fine print: Streaming & interactive I/O → readiness probes.

Tear down several children as a unit

use processkit::Command;
use processkit::ProcessGroup;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _db = group.start(&Command::new("dev-db")).await?;
    let _api = group.start(&Command::new("dev-api")).await?;

    // Either: graceful — SIGTERM, bounded wait, optional SIGKILL escalation…
    group.shutdown().await?;
    // …or just drop(group): hard kill-on-drop of everything, grandchildren included.
    Ok(())
}

The group is the unit of fate: a panic or early return anywhere reaps every member. Configure the grace window via ProcessGroupOptions.

Fine print: Process groups.

React to whichever child exits first

use processkit::Command;
use processkit::{ProcessGroup, wait_any};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let mut a = group.start(&Command::new("worker-a")).await?;
    let mut b = group.start(&Command::new("worker-b")).await?;

    let (idx, outcome) = wait_any(&mut [&mut a, &mut b]).await?;
    println!("worker #{idx} exited first with {outcome:?}");
    // `a` and `b` are only borrowed — the loser is still usable here.
    Ok(())
}

Fine print: Streaming & interactive I/O → racing children.

Sandbox an untrusted tool

use processkit::Command;
use processkit::{ProcessGroup, ProcessGroupOptions};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Cap the whole tree (requires the `limits` feature; Windows Job / Linux cgroup):
    let group = ProcessGroup::with_options(
        ProcessGroupOptions::default()
            .max_memory(512 * 1024 * 1024)
            .max_processes(64)
            .cpu_quota(0.5),
    )?;

    let result = group
        .start(
            &Command::new("untrusted-tool")
                .inherit_env(["PATH"]) // allow-list: everything else is cleared
                .timeout(std::time::Duration::from_secs(60)),
        )
        .await?
        .output_string()
        .await?;
    Ok(())
}

Unenforceable limits are a hard ErrorReason::ResourceLimit, never a silently unbounded group. On Unix, add .uid(…)/.gid(…) to drop privileges (note the cgroup-mechanism caveat in the guide).

Fine print: Process groups → resource limits · Running commands → privileges.

Keep a crash-prone service running

use processkit::Command;
use processkit::{RestartPolicy, Supervisor};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("my-service"))
        .restart(RestartPolicy::OnCrash)
        .max_restarts(5)
        .backoff(Duration::from_millis(200), 2.0)
        .storm_pause(Duration::from_secs(15)) // crash-loop guard (off by default)
        .run()
        .await?;
    println!(
        "stopped after {} restarts ({} storm pauses): {:?}",
        outcome.restarts, outcome.storm_pauses, outcome.stopped
    );
    Ok(())
}

Exponential backoff with jitter by default; stop_when(…) ends supervision on a condition; .with_runner(&group) keeps every incarnation inside one shared kill-on-drop group. storm_pause arms the failure-storm guard: failures feed a decaying score, and past the threshold the supervisor takes one collective pause instead of hammering restarts — "fails rarely" and "crash-looping" stop being the same case.

Fine print: Supervision, failure storms.

Retry a flaky command

use processkit::Command;
use processkit::ErrorReason;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let fetched = Command::new("git")
        .args(["fetch", "--quiet"])
        .timeout(Duration::from_secs(10))
        .retry(3, Duration::from_millis(200), |e| {
            matches!(e.reason(), ErrorReason::Timeout { .. })
                || e.diagnostic().is_some_and(|m| m.contains("Could not resolve host"))
        })
        .run()
        .await?;
    Ok(())
}

The classifier sees the typed error and decides whether this failure is worth another attempt; each attempt is a fresh process. retry replays a run to success — for keeping a process alive, use a Supervisor (previous recipe).

Fine print: Timeouts, retries & cancellation → retry.

Cancel runs on shutdown

use processkit::Command;
use processkit::CancellationToken;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let token = CancellationToken::new();

    let job = tokio::spawn({
        let token = token.child_token();
        async move { Command::new("long-job").cancel_on(token).run().await }
    });

    // On Ctrl-C / shutdown signal / sibling failure:
    token.cancel(); // kills the tree; the run resolves to ErrorReason::Cancelled
    let outcome = job.await; // Err(ErrorReason::Cancelled { .. }) inside
    Ok(())
}

Cancellation is always an error (the run was abandoned, there is no result), beats a simultaneous timeout, and is terminal for retry and Supervisor alike.

For a typed wrapper whose commands never cross your code, set the token once on the client — every command it builds carries it:

#![allow(unused)]
fn main() {
use processkit::{CancellationToken, CliClient};

let token = CancellationToken::new();
let gh = CliClient::new("gh").default_cancel_on(token.child_token());
// token.cancel() → every in-flight command of THIS client dies.
}

Fine print: Timeouts, retries & cancellation → cancellation, client-level default.

Measure what a run cost

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // One run, summarized (requires the opt-in `stats` feature):
    let profile = Command::new("crunch").start().await?.profile(Duration::from_millis(100)).await?;
    println!("outcome={:?} took={:?} peak_rss={:?} avg_cpu_cores={:?}",
        profile.outcome, profile.duration, profile.peak_memory_bytes, profile.avg_cpu_cores());
    Ok(())
}

For a live series over a whole group, group.sample_stats(every) yields a Stream of snapshots. CPU/memory need a real container (Windows Job / Linux cgroup); elsewhere you still get process counts.

Fine print: Process groups → stats · Streaming → profiling.

Contain a process you didn't spawn

use processkit::{Error, ErrorReason, ProcessGroup};

fn main() -> processkit::Result<()> {
    // `tokio`/`std` calls return `io::Error`, which the crate does NOT auto-convert
    // into `processkit::Error` (there is no blanket `From<io::Error>`, by design) —
    // map it explicitly, or use a `Box<dyn std::error::Error>` / `anyhow` return in
    // your own code so both error types `?` freely.
    let child = tokio::process::Command::new("legacy-launcher")
        .spawn()
        .map_err(|e| Error::from(ErrorReason::Io(e)))?;

    let group = ProcessGroup::new()?; // `adopt` is part of `process-control` (default-on)
    group.adopt(&child)?;            // from now on the group's teardown covers it
    Ok(())
}

Adoption is best-effort by mechanism — on Windows/cgroup the whole running tree joins; on the POSIX process-group backends an exec'd child is contained individually (its future forks too, where it could be re-grouped). The guide spells out exactly what each mechanism can promise.

Only a pid to go on? When the process is not yours to hold a Child for — an outside supervisor started it, the number came from a pidfile, or the caller is an FFI binding that cannot build a Child at all — use adopt_external(pid):

use processkit::ProcessGroup;

fn main() -> processkit::Result<()> {
    let pid: u32 = std::env::var("HELPER_PID").unwrap().parse().unwrap();
    let group = ProcessGroup::new()?;
    group.adopt_external(pid)?; // the crate takes its own identity anchor here
    // The group's teardown now covers it — but nothing here will ever reap it,
    // and no exit status for it appears in this API.
    Ok(())
}

The number is an address, not a handle: the crate anchors on the process the pid names at that moment (the process object on Windows, cgroup membership on Linux, the start-time token on the POSIX process-group backends), so a later recycle of the number is not signalled. It cannot check the window before the call, so look the pid up as late as you can. FreeBSD and the other BSDs have no start-time reader and return Unsupported rather than tracking a bare number.

Adopting a process another supervisor already contains has a side effect that outlives the group, in opposite directions per platform: on Windows this group's job is nested under the job the process was already in (so that outer job then reaches this group's members, later-started ones included — and an adoption into a group that already has its own members may simply be refused, so adopt before you start); on Linux cgroup v2 the process is taken out of its previous cgroup, which disables that supervisor's teardown and limits for it.

Fine print: Process groups → adopt · Adopting by pid · Platform support.

Test code that runs processes — without processes

use processkit::Command;
use processkit::testing::{Reply, ScriptedRunner};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Your code takes any `R: ProcessRunner`; in tests, hand it a script.
    // Rules match on a prefix of the *program name followed by its arguments*
    // (the first element is the program):
    let runner = ScriptedRunner::new()
        .on(["git", "rev-parse"], Reply::ok("abc123\n"))
        .on(["git", "push"], Reply::fail(128, "remote: permission denied"))
        .fallback(Reply::ok(""));

    // my_deploy(&runner).await? — no subprocess, fully deterministic.
    Ok(())
}

RecordingRunner wraps any runner and captures every Invocation for assertions; MockRunner (feature mock) gives mockall expectations; and the record feature's RecordReplayRunner records real runs into a JSON cassette once and replays them hermetically in CI.

Fine print: Testing your code.

Test streaming code — without processes

use processkit::{Command, Outcome, ProcessRunner, Finished};
use processkit::testing::{Reply, ScriptedRunner};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let runner = ScriptedRunner::new()
        .on(["gh", "run", "watch"], Reply::lines(["queued", "in_progress", "completed"])
            .with_line_delay(Duration::from_millis(50))); // paced delivery

    let mut run = runner.start(&Command::new("gh").args(["run", "watch", "123"])).await?;
    run.wait_for_line(|l| l.contains("completed"), Duration::from_secs(5)).await?;
    let Finished { outcome, .. } = run.finish().await?;
    assert_eq!(outcome, Outcome::Exited(0));
    Ok(())
}

A scripted start() feeds the canned lines through the same pump machinery a real child uses, so stdout_lines, the readiness probes, and finish behave identically — and with_line_delay is deterministic under #[tokio::test(start_paused = true)]. Canned output also replays through on_stdout_line/on_stderr_line handlers on the bulk verbs, so progress-reporting paths test hermetically too.

Fine print: Testing → scripted streaming.

Wrap a CLI tool behind a typed API

#![allow(unused)]
fn main() {
use processkit::{cli_client, ProcessRunner, Result};

cli_client!(pub struct Git => "git");

impl<R: ProcessRunner> Git<R> {
    pub async fn current_branch(&self) -> Result<String> {
        // A verb takes the args directly (D7); pass a built `command(..)` only
        // when you need to customize it (per-call timeout, stdin, …).
        self.core.run(["branch", "--show-current"]).await
    }
    pub async fn is_clean(&self) -> Result<bool> {
        self.core.probe(["diff", "--quiet"]).await
    }
}
}

The generated struct carries a runner and per-client defaults (default_timeout, default_env); your methods are just argument lists and parsers — and because the runner is injectable, the whole wrapper is testable with the previous recipe's ScriptedRunner.

Deep guide: Building typed CLI clients · Testing doubles and cassettes.

Running commands

‹ docs index

Command is the entry point of the runner layer: a builder describing what to run and how, plus a family of consuming verbs that decide what you get back. Every one-shot verb spawns the child into a fresh, private kill-on-drop process group, so an early return, panic, or dropped future can never leak a process tree.

• Program, arguments, working directory
• Resolving a locally-installed tool: prefer_local
• Environment
• Standard input
• Redirecting output directly to a file
• Output handling
• Timeouts and retries
• Privileges and spawn flags
• Consuming verbs
• Results and errors

Program, arguments, working directory

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("git")
        .arg("log")                          // one at a time…
        .args(["--oneline", "-n", "10"])     // …or in bulk
        .current_dir("/path/to/repo")        // run there
        .run()
        .await?;
    Ok(())
}

Arguments are passed as an array — there is no shell between you and the child, so there is no quoting, no word-splitting, and no injection surface. (When you actually want a | b | c, use a pipeline, which connects the stages in-process instead of invoking a shell.)

The program name reaches the OS verbatim — two deliberate non-goals (conveniences some libraries layer on, e.g. duct): a bare name is resolved on PATH by the OS, never rewritten to ./name; and current_dir does not re-anchor a relative program path against the new directory — whether Command::new("./tool").current_dir(dir) resolves tool relative to dir is the platform's behavior (Unix: yes; Windows: the parent's directory may win). Pass absolute program paths when combining the two.

For quick one-liners the free functions skip the builder:

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let version = processkit::run("cargo", ["--version"]).await?;       // trimmed stdout, success required
    let result  = processkit::output_string("git", ["status", "-s"]).await?;   // full ProcessResult
    Ok(())
}

Resolving a locally-installed tool: prefer_local

prefer_local adds a directory to check before the system PATH when resolving a bare-name program for this one run — for a project's own node_modules/.bin, a target/debug build, or a vendored toolchain, without hand-rolling a PATH override:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("eslint")
        .prefer_local("./node_modules/.bin")
        .arg("src/")
        .output_string()
        .await?;
    Ok(())
}

Resolution order. Repeated calls accumulate, in priority order: the directory from the first call is probed first, then the second, and so on, with the system PATH tried last as the final fallback:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    Command::new("tool")
        .prefer_local("./vendor/bin")   // checked first
        .prefer_local("./target/debug") // checked second
        .run()                          // then the system PATH
        .await?;
    Ok(())
}

Resolution reuses the exact same PATHEXT-aware lookup as the PATH search (the same internal probe_dir helper — not a separate implementation), so a .exe/.cmd/.bat on Windows is found under a prefer_local directory exactly as it would be on PATH.

Only a bare name is affected. If the program passed to Command::new is a path — absolute, or relative with a separator ("./tool", "../bin/x") — prefer_local has no effect at all: the existing contract that such a program is never looked up on PATH (or here) is unchanged.

Interaction with PATH/inherit_env/env. prefer_local only changes where the parent looks to resolve the program for this one launch. It does not rewrite or extend the PATH the child sees in its own environment — that is governed entirely by env/inherit_env/env_clear, as usual. When the program is found under a prefer_local directory, the child is simply spawned via that resolved absolute path instead of the bare name; a grandchild the program itself spawns does not inherit this reach — only the one program named in this Command benefits.

Interaction with current_dir. A relative prefer_local directory (as in the examples above) is probed against the process's actual current directory, never against whatever is set via current_dir on the same Command. The resolved match is then always turned into an absolute path before being handed to the OS, so it can't later be reinterpreted against the child's working directory once current_dir is set — unlike a relative-path program passed straight to Command::new, which is subject to that footgun (see Program, arguments, working directory above).

Diagnostics. If resolution fails everywhere, ErrorReason::NotFound's searched field includes the prefer_local directories too — first, in priority order, ahead of the PATH directories — so the diagnostic never hides that they were checked.

Preflight: resolve a program without running it

Sometimes you want to know whether an external tool is available before you run it — a doctor check at startup, a friendly "is git installed?" error up front — with no side effects. resolve_program locates a command's program and returns its absolute path without spawning anything:

use processkit::Command;

fn main() -> processkit::Result<()> {
    // `which` is the crate-level shortcut for a bare tool.
    let git = processkit::which("git")?;   // Ok(/usr/bin/git) or Err(NotFound)
    println!("git lives at {}", git.display());

    // On a builder it honors that command's own `prefer_local` and env, so it
    // resolves exactly what a real run of that command would launch.
    let eslint = Command::new("eslint")
        .prefer_local("./node_modules/.bin")
        .resolve_program()?;
    println!("eslint lives at {}", eslint.display());
    Ok(())
}

No divergence from a real run. Resolution reuses the crate's own launch-path logic — the same PATH/PATHEXT/execute-bit resolution and prefer_local handling a spawn performs, not a second copy — so a resolve_program hit is exactly the executable a run would launch, and a miss is exactly the ErrorReason::NotFound (with the same searched diagnostic and is_not_found() classification) a run would raise. A command that relocates the child's PATH (env/env_remove of PATH, env_clear, inherit_env) is resolved against that effective child PATH, so preflight still matches the spawn.

It is a synchronous, cheap filesystem probe (a few stats) — no async runtime is required, and no process is ever started. Contrast probe(), which runs the tool to read its exit code; resolve_program only locates it.

fn main() {
    match processkit::which("definitely-not-installed") {
        Ok(path) => println!("found: {}", path.display()),
        Err(e) if e.is_not_found() => eprintln!("tool not installed"),
        Err(e) => eprintln!("resolution error: {e}"),
    }
}

For a tool wrapped behind a CliClient, CliClient::resolve_program() does the same for the client's program, honoring its env defaults. The dedicated typed CLI clients guide covers wrapper structure, shared defaults, parsing, and hermetic tests end to end.

Environment

Four builders compose, applied in a fixed order at spawn:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    Command::new("worker")
        .env("RUST_LOG", "debug")        // set one variable
        .env_remove("GIT_DIR")           // unset one inherited variable
        .run().await?;

    // Unix: choose a multicall mode without changing executable resolution.
    Command::new("busybox").arg0("httpd").arg("-f").run().await?;

    // Allow-list mode: clear everything, copy only the named parent variables.
    Command::new("sandboxed-tool")
        .inherit_env(["PATH", "HOME", "LANG"])
        .env("MODE", "ci")               // explicit env/env_remove still apply on top
        .run().await?;

    // Scorched earth: the child starts with an empty environment.
    Command::new("hermetic-tool").env_clear().run().await?;
    Ok(())
}

inherit_env is the sandboxing middle ground: it implies env_clear, then copies the listed variables from the parent at each spawn (so a retry sees fresh values), and repeated calls accumulate names. A name the parent doesn't have is skipped, not set to empty.

Standard input

By default stdin is closed at spawn — the child reads EOF immediately and can never hang waiting for input. Everything else is opt-in via stdin(Stdin::…):

use processkit::{Command, Stdin};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let sorted = Command::new("sort")
        .stdin(Stdin::from_iter_lines(["banana", "apple", "cherry"]))
        .run()
        .await?;
    assert_eq!(sorted, "apple\nbanana\ncherry");
    Ok(())
}

The payload is written on a background task (so a large input can't deadlock against the child's output) and the pipe is dropped afterwards to signal EOF. The two one-shot sources are consumed by their first run: a retried or cloned command reusing them fails loud the second time — re-running a consumed from_reader/from_lines source is an ErrorReason::Io (InvalidInput) at launch (D10), not a silent empty stdin. Prefer the reusable sources when a command may run more than once.

For conversational, request/response stdin — write a line, read the answer, repeat — use keep_stdin_open() and the streaming API instead: see Streaming & interactive I/O.

Inheriting the parent's stdin: inherit_stdin()

inherit_stdin() hands the child the parent's own standard input — it reads directly from whatever this process's stdin is (a terminal, a file, a pipe) rather than from a crate-managed pipe. It is the stdin counterpart of stdout(StdioMode::Inherit) / stderr(StdioMode::Inherit): the child shares the parent stream instead of the crate mediating it.

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // `git commit` opens $EDITOR on the parent's terminal; the child talks to the
    // real tty directly. stdout/stderr are still captured as usual.
    Command::new("git").arg("commit").inherit_stdin().run().await?;
    Ok(())
}

Reach for it when a child must talk to the real terminal — git commit opening $EDITOR, a tool prompting for a password or a yes/no — or to forward the parent's piped stdin straight through. This covers the common non-tty-negotiating interactive cases without the crate having to pump bytes; a tool that truly demands a tty (not just inherited stdin) instead wants use_pty (the pty feature). Because the child reads the parent's stdin directly, the crate neither feeds nor captures that input, and take_stdin() returns None (as for a non-keep_stdin_open run). Capturing and streaming the child's output is unaffected.

Why a dedicated verb rather than a Stdin::Inherit source or a mode enum. For stdout/stderr the three StdioMode variants map cleanly onto one setter, but stdin's "piped" case is not modeless — it needs a payload (which source? what bytes?), already expressed by stdin(Stdin::…), and its "null" case is Stdin::empty(). Folding inheritance into that same stdin(Stdin) field would make "inherit and a source" collapse to silent last-write-wins, impossible to flag. A separate inherit_stdin() keeps the two intents in distinct fields so an incompatible pairing is a detectable, rejectable error instead.

Accordingly, inherit_stdin() is mutually exclusive with either way the crate would otherwise drive stdin — a configured stdin(Stdin::…) source (including an explicit Stdin::empty()) or keep_stdin_open()'s interactive pipe. Setting inherit_stdin() together with one of those is a contradiction (feed the child a source and let it read the terminal?), so it is refused at the launch boundary with a typed ErrorReason::Io (InvalidInput) — the same failure mode as re-running a consumed one-shot source — rather than silently letting one win. Drop the other stdin knob to resolve it. The refusal is enforced on the same launch seam the hermetic test doubles route through, so a ScriptedRunner rejects the conflict exactly as a live run does.

Redirecting output directly to a file

stdout_file(path) and stderr_file(path) give the child a file descriptor at spawn (Stdio::from(File)). They do not tee through a parent task: no output is line-pumped, decoded, or retained in memory, and the child can keep writing if the parent exits suddenly.

The plain builder creates the file when needed and truncates it for that spawn. Use the explicit *_file_truncate spelling when choosing a mode in a conditional expression, or *_file_append to preserve existing contents and append each new child incarnation:

use processkit::{Command, Supervisor};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let service = Command::new("my-service")
        .stdout_file_append("service.log") // every Supervisor incarnation shares this log
        .stderr_file_append("service.log");

    Supervisor::new(service).run().await?;
    Ok(())
}

This is the low-overhead choice for a service under Supervisor that writes its own log: append mode accumulates every restart in one file, while truncate mode starts a fresh log for each spawn. The command's stdout(StdioMode::…) / stderr(StdioMode::…) setters are last-wins and clear a prior file destination.

A redirected stdout is deliberately not piped. output_string, output_bytes, stdout_lines, and events therefore reject it just as they reject Inherit/Null; call start().await?.wait().await? (or supervise the command) when only its exit outcome matters. Stderr may be redirected independently; it does not prevent stdout capture.

Output handling

Encodings

Output is decoded line by line, UTF-8 by default (invalid bytes become U+FFFD, never an error). Legacy-encoding tools can override per stream:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("legacy-tool")
        .encoding(encoding_rs::SHIFT_JIS)          // both streams…
        // .stdout_encoding(…) / .stderr_encoding(…) // …or each its own
        .output_string()
        .await?;
    Ok(())
}

(processkit::prelude::Encoding re-exports encoding_rs::Encoding, so any of its encodings works — the single-byte and ASCII-compatible multibyte ones (WINDOWS_1252, GBK, SHIFT_JIS, …) and the non-ASCII-compatible ones (UTF_16LE/UTF_16BE): output is fed through one persistent decoder and split on decoded newlines, so a 0x0A byte inside a UTF-16 code unit is not mistaken for a line break. A leading byte-order mark of the chosen encoding is stripped once at the stream start.)

Buffer policies — bounding memory on chatty children

Captured lines are held in memory; a multi-gigabyte log would normally grow the buffer to match. output_buffer bounds retention (the pipe is always fully drained, so the child never blocks):

use processkit::{Command, OutputBufferPolicy, OverflowMode};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let tail = Command::new("verbose-build")
        .output_buffer(OutputBufferPolicy::bounded(1_000)) // keep the newest 1000 lines
        .output_string()
        .await?;

    // …or keep the head instead of the tail:
    let head_policy = OutputBufferPolicy::bounded(1_000).with_overflow(OverflowMode::DropNewest);
    Ok(())
}

DropOldest (the default) keeps a rolling tail; DropNewest freezes the head — a contiguous prefix of the output: the first line that doesn't fit seals the head, so every later line is dropped too (even a shorter one that would still fit), never leaving a set that skipped a dropped line and kept a later one. bounded(0) retains nothing — useful when a line handler (below) is the real consumer. Under a line cap, dropped or not, every line still feeds the handlers and the line counters.

The line cap alone does not bound memory — one enormous newline-free "line" (base64 -w0) is held whole. Add with_max_bytes to cap the retained bytes too (either ceiling, or both); the byte cap also bounds the pump's in-flight assembly buffer, so a never-terminated flood can't exhaust memory. One consequence: a line whose own length exceeds the byte cap can't be assembled, so it is dropped whole — counted, but not delivered to a per-line handler or stdout_tee (don't set a byte cap if a tee must see arbitrarily long lines):

#![allow(unused)]
fn main() {
use processkit::{Command, OutputBufferPolicy};
let policy = OutputBufferPolicy::unbounded().with_max_bytes(8 << 20); // 8 MiB ring
let strict = OutputBufferPolicy::fail_loud(10_000).with_max_bytes(8 << 20); // error on either
}

fail_loud makes the ceiling error instead of dropping: the run fails with ErrorReason::OutputTooLarge once the cumulative output (lines or bytes) crosses the cap — even when a streaming consumer is draining lines as they arrive. It bounds memory, not wall-time, so pair it with timeout against a flooding child.

Even under a drop policy (DropOldest/DropNewest), the checking verbs that hand back stdout as if complete — run, parse, try_parse, output_json — refuse silently-truncated output (B12): if the policy dropped lines they fail with ErrorReason::OutputTooLarge rather than feed a parser a truncated tail. The lenient capture verbs (output_string / output_bytes) are unaffected — they return the partial result with truncated() set for you to inspect.

Line handlers — tee output as it arrives

on_stdout_line / on_stderr_line run a callback on each decoded line in addition to capture or streaming — logging, progress bars, metrics:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("cargo")
        .args(["build", "--release"])
        .on_stderr_line(|line| eprintln!("[build] {line}"))
        .output_string()
        .await?;
    Ok(())
}

The handler runs on the read pump — keep it cheap. The contract is forgiving and precisely specified:

• A panicking handler does not poison the run. The panic is caught, the handler is disabled for the rest of the run (surfaced as a tracing warn when that feature is on), and pumping continues — the final result still carries every line. You can safely re-export this callback seam to your own users without auditing their closures.
• Ordering: invocations are FIFO within a stream; there is no ordering between stdout and stderr handlers (two independent pumps). On the consuming verbs, all handler calls happen-before the awaited future resolves — finalize a progress bar the moment the call returns. (One documented exception: a leaked pipe held open past the child's death is cut off after a bounded teardown grace.)
• Handlers are hermetically testable: ScriptedRunner replays canned output through them — see Testing → scripting replies.

For a ready-made tee to an async sink — a file, socket, or any [tokio::io::AsyncWrite] — reach for stdout_tee / stderr_tee instead of hand-writing a handler. Each decoded line is written to the sink (plus a \n) as it is produced, awaited on the pump so a slow sink applies backpressure (the pump slows, the pipe fills, the child blocks) rather than blocking the runtime; a write error disables the tee with a tracing warn instead of being swallowed. It runs independently of on_stdout_line — set both and both fire per line.

Timeouts and retries

use processkit::{Command, ErrorReason};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("flaky-network-tool")
        .timeout(Duration::from_secs(30))                 // kill the tree at the deadline
        .retry(3, Duration::from_millis(200), |e| {       // up to 3 attempts total
            matches!(e.reason(), ErrorReason::Timeout { .. })            // …but only retry timeouts
        })
        .run()
        .await?;
    Ok(())
}

• timeout attempts whole-tree teardown at the deadline. After terminal state is confirmed, capturing verbs keep the expiry as data (ProcessResult::timed_out) and success-checking verbs raise ErrorReason::Timeout; an unconfirmed kill/escalation/reap is ErrorReason::Teardown instead. The full decision table lives in Timeouts, retries & cancellation.
• retry applies to the success-checking verbs only — run, run_unit, exit_code, probe, checked, parse, try_parse, and (with json) output_json (each runs through the retry loop). The classifier sees the typed error and decides. Cancelled and Teardown are terminal even when that classifier accepts them. The non-erroring output_string/output_bytes paths never retry, and neither does first_line (its stream search is single-attempt).

Privileges and spawn flags

Spawn-time controls for sandboxing and service launch:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Unix: drop privileges (uid + gid + supplementary groups) and detach.
    Command::new("worker")
        .gid(1000)            // applied before uid (a gid change needs privilege)
        .groups([1000])       // replace the inherited (often root's) supplementary groups
        .uid(1000)            // dropped last
        .setsid()             // new session: survives the controlling terminal
        .run().await?;

    // Windows: no console window flashing up from a GUI app.
    Command::new("helper").create_no_window().run().await?;

    // Hardening: take the direct child down even if THIS process is SIGKILLed
    // (Drop never runs). Windows has this for free; Linux arms PDEATHSIG.
    Command::new("worker").kill_on_parent_death().start().await?;
    Ok(())
}

uid / gid / groups / setsid / arg0 are POSIX-only — on Windows the run fails with ErrorReason::Unsupported rather than silently skipping a privilege drop. A correct drop sets all three of uid/gid/groups: dropping the uid alone leaves the child holding the parent's (often root's) supplementary groups. arg0 changes only the value delivered as the child's first argument: lookup, preflight, containment, and spawn errors still use Command::program(). create_no_window is a harmless no-op outside Windows. kill_on_parent_death is best-effort by design: guaranteed on Windows (regardless of the knob), direct-child-only on Linux, unavailable on macOS/BSD — the graceful-exit guarantee via Drop holds everywhere either way. When the owner dies abruptly the kernel has no portable Unix way to take the whole tree down, so Command::kill_on_parent_death_scope() reports the honest reach as a ParentDeathCleanup — WholeTree (Windows), DirectChildOnly (Linux), or Unsupported (macOS/BSD). A wrapper can surface that scope instead of overpromising a whole-tree cleanup:

#![allow(unused)]
fn main() {
use processkit::{Command, ParentDeathCleanup};

match Command::kill_on_parent_death_scope() {
    ParentDeathCleanup::WholeTree => { /* the whole tree dies with the owner */ }
    ParentDeathCleanup::DirectChildOnly => { /* only the direct child; grandchildren survive */ }
    ParentDeathCleanup::Unsupported => { /* no abrupt-death cleanup on this platform */ }
    _ => {}
}
}

Containment is preserved in every combination; the platform fine print (the Linux cgroup × uid interaction, setsid × process-group coordination, the pdeathsig thread caveat) is collected in Platform support.

Scheduling, umask, and per-process rlimits

Spawn-time knobs reuse the same seams as the builders above — Unix pre_exec and Windows' suspended-child configuration — for background/batch children that shouldn't starve the foreground, and for controlling the permissions of files a child creates:

use processkit::{Command, IoPriority, Priority, RlimitResource};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Run at a lower CPU-scheduling priority — supported on BOTH platforms.
    Command::new("batch-job")
        .priority(Priority::BelowNormal)
        .run().await?;

    // Linux + Windows: keep a noisy worker on logical CPUs 2 and 3.
    Command::new("compiler")
        .cpu_affinity([2, 3])
        .run().await?;

    // Linux only: yield disk time to foreground users.
    Command::new("indexer")
        .io_priority(IoPriority::BestEffort(7))
        .run().await?;

    // Unix only: files this child creates get 0644/0755 instead of 0666/0777.
    Command::new("worker").umask(0o022).run().await?;

    // Unix only: disable core dumps and cap this child's open descriptors.
    Command::new("secret-worker")
        .rlimit(RlimitResource::Core, 0, 0)
        .rlimit(RlimitResource::NoFile, 256, 256)
        .run().await?;
    Ok(())
}

priority maps onto nice/setpriority on Unix and a priority class on Windows (Idle/BelowNormal/Normal/AboveNormal/High); unlike the privilege builders, every variant is supported on both platforms, so this knob never yields ErrorReason::Unsupported. One caveat: lowering nice below its inherited value on Unix — raising priority via Priority::AboveNormal/High, or even requesting Priority::Normal under a positively-niced parent (e.g. a niced CI/batch launcher) — needs CAP_SYS_NICE/root; without it the OS rejects the change and the spawn fails loud (ErrorReason::Spawn), never silently downgrading to a lower priority.

cpu_affinity accepts logical CPU indices. Linux applies a cpu_set_t with sched_setaffinity before exec; Windows calls SetProcessAffinityMask after race-free Job assignment while the child is still suspended (the ConPTY launch does the same), then resumes it. Descendants inherit the mask, though a child with sufficient rights may later change its own. Empty or unrepresentable sets fail before user code runs; an OS-rejected processor fails the spawn. macOS/BSD return ErrorReason::Unsupported. The Windows API is one processor-group mask, so indices are limited to the native mask width. spawn_detached refuses the knob; on Windows to_tokio_command does too because a raw command cannot carry the required post-spawn configuration seam.

io_priority is Linux-only: it calls ioprio_set(2) in pre_exec before the program starts. BestEffort(7) is the lowest normal Linux I/O priority; smaller data values are more urgent, while Idle runs only when the device is otherwise idle. RealTime can starve other users and normally needs CAP_SYS_ADMIN; a rejected request fails as ErrorReason::Spawn. On Windows, macOS/BSD, and other Unix targets, requesting I/O priority fails with ErrorReason::Unsupported rather than silently inheriting the caller's I/O priority. It is also refused by spawn_detached, whose owner-independent launch contract cannot honor it.

umask is Unix-only — like setsid/groups, requesting it on Windows fails with ErrorReason::Unsupported rather than being silently ignored.

Interactive auth / TTY. By default processkit wires pipes, not a pseudo-terminal, so a tool that demands a tty — an ssh/sudo password prompt, some credential helpers, an isatty()-gated agentic CLI — won't get one. Two ways to satisfy them:

• Non-interactive (no PTY needed). Prefer this when possible: key-based auth, ssh -o BatchMode=yes, GIT_SSH_COMMAND / GIT_TERMINAL_PROMPT=0, or feed a known answer over interactive stdin. Conversational tools that read stdin without needing a tty work today via keep_stdin_open + stdout_lines.

• PTY mode (use_pty, the pty feature). For a tool that truly requires a controlling terminal, Command::use_pty() launches it under a real pseudo-terminal — openpty on Unix, CreatePseudoConsole (ConPTY) on Windows — so isatty() reports a terminal. This is a minimal single-master-fd mode, not a terminal emulator, with four things to know:

  • stdout and stderr are merged onto the one master, so in this mode the on_stderr_line / stderr_tee split collapses and ProcessResult::stderr is empty — the whole output arrives through piped logical stdout. Both destinations must remain StdioMode::Piped: stdout or stderr Inherit/Null and *_file* redirects are rejected before a file is opened or a child is spawned, because the merged terminal cannot honor a separate per-descriptor destination.
  • Interactive input runs over the same master (keep_stdin_open + take_stdin); on Unix terminal echo is disabled so a written password is not echoed back into the merged output (the ConPTY has no portable per-write echo control, so that is Unix-only).
  • The child receives COLUMNS/LINES matching the initial PTY size (80×24, or pty_size(cols, rows)). A zero axis is rejected before spawn on every platform; Windows also rejects either axis above i16::MAX rather than clamping ConPTY's signed COORD, while Unix accepts the full remaining u16 range. Unix also defaults TERM=xterm-256color; Windows relies on ConPTY's console/VT APIs and does not synthesize TERM. Explicit env(...) or env_remove(...) calls for any of these names win.
  • Containment is unchanged — the PTY child lives in the same job/cgroup/process group, so whole-tree kill-on-drop, timeouts, and cancellation behave exactly as for a piped run.

  Scenario recipes live in the cookbook: run an isatty()-requiring tool, wait for an unterminated prompt and answer, consume agent-CLI progress, avoid full-duplex deadlock, and test the PTY contract hermetically. The streaming guide owns the detailed PTY I/O mechanics and output-hygiene knobs. For the end-to-end ssh case specifically — including the boundary where kill-on-drop stops at the local client — see Driving ssh.

  The historical defer/design is recorded in decisions/permissions-privileges-pty-network.md §4.

Consuming verbs

Typed JSON and NDJSON

With the additive json feature, output_json::<T>() runs to an accepted exit and deserializes the complete stdout. It is available on Command, ProcessRunnerExt, and CliClient, so a typed wrapper keeps the same verb when its runner changes from a real process to ScriptedRunner:

#![allow(unused)]
fn main() {
use processkit::Command;
use serde::Deserialize;

#[derive(Deserialize)]
struct Release {
    tag_name: String,
}

async fn example() -> processkit::Result<()> {
let release: Release = Command::new("gh")
    .args(["release", "view", "--json", "tagName"])
    .output_json()
    .await?;
println!("{}", release.tag_name);
Ok(())
}
}

For NDJSON, start the process and take a typed line stream. Each item is its own Result<T>: a malformed line is reported with its one-based line/column and zero-based byte offset, then the stream continues. Empty lines are errors rather than being silently skipped.

#![allow(unused)]
fn main() {
use processkit::prelude::StreamExt;
use processkit::Command;
use serde::Deserialize;

#[derive(Deserialize)]
struct Message {
    reason: String,
}

async fn example() -> processkit::Result<()> {
let mut process = Command::new("cargo")
    .args(["check", "--message-format=json"])
    .start()
    .await?;
let mut messages = process.stdout_json_lines::<Message>()?;
while let Some(message) = messages.next().await {
    println!("{}", message?.reason);
}
let finished = process.finish().await?;
assert_eq!(finished.outcome.code(), Some(0));
Ok(())
}
}

Both verbs reject incomplete data instead of pretending it is valid: output_json fails before parsing when a bounded capture was truncated, while the NDJSON stream inherits stdout_lines' fail-loud overflow and timeout contracts. JSON parse errors use ErrorReason::Parse; their child-controlled fragment is capped to 160 input bytes and control-escaped even in the public message field. NDJSON offsets refer to ProcessKit's decoded, \n-normalized stdout.

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // probe(): the exit code as a boolean.
    let clean = Command::new("git").args(["diff", "--quiet"]).probe().await?;

    // first_line(): stop as soon as the interesting line appears.
    let first_match = Command::new("git")
        .args(["log", "--oneline"])
        .first_line(|l| l.contains("fix:"))
        .await?;
    Ok(())
}

first_line returns Ok(None) when stdout closes without a match, and kills the (private-group) child once it has its answer — you never wait out a long log for one line. A cancel_on token that fires while the search is still running surfaces as ErrorReason::Cancelled, so a readiness probe with a shutdown token can't misread token-driven teardown as "the line never appeared" — while a run that genuinely ends with no match still reports Ok(None), even if the token happens to fire an instant later.

Results and errors

The capturing verbs hand back a ProcessResult:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("git").args(["merge", "feature"]).output_string().await?;

    result.code();         // Option<i32> — None = killed (timeout/signal), no code
    result.signal();       // Option<i32> — the signal number (Unix), else None
    result.is_success();   // code in ok_codes (default {0})
    result.timed_out();    // an absolute or inactivity timeout expired
    result.inactivity_timed_out(); // specifically: stdout/stderr went quiet
    result.outcome();      // the explicit disposition behind the accessors above
    result.stdout();       // &str (or &[u8] from output_bytes)
    result.stderr();       // &str
    result.combined();     // stdout + stderr concatenated
    result.diagnostic();   // stderr if non-empty, else stdout — the human-facing line
                           // (git/jj put "CONFLICT …" on stdout!)
    result.configured_timeout(); // Option<Duration> — the timeout this run was launched with
    result.ok_codes();     // &[i32] — the accepted exit codes ({0} by default)

    // Opt into erroring whenever you're ready:
    let ok = result.ensure_success()?; // Exit / Timeout / Signalled (signal-kill) as typed errors
    Ok(())
}

When the exact disposition matters, match on Outcome instead of mentally decoding the code()/timed_out() pair:

use processkit::Outcome;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = processkit::Command::new("git").args(["merge", "feature"]).output_string().await?;
    match result.outcome() {
        Outcome::Exited(0) => println!("clean"),
        Outcome::Exited(code) => println!("failed with {code}"),
        Outcome::Signalled(signal) => println!("killed by signal {signal:?}"),
        Outcome::TimedOut => println!("hit its deadline"),
        Outcome::InactivityTimedOut => println!("stopped producing output"),
        _ => {} // non_exhaustive: future dispositions
    }
    Ok(())
}

For a single query you usually don't need the match (and its #[non_exhaustive] wildcard): Outcome carries the same code() / signal() / timed_out() / inactivity_timed_out() accessors as ProcessResult, so a bare Outcome (from RunningProcess::wait or Finished::outcome) answers directly — outcome.code(), outcome.signal(), outcome.timed_out(). There is no Outcome::is_success (success is ok_codes-aware — use ProcessResult::is_success).

The error enum is structured and #[non_exhaustive]:

Error::diagnostic() returns the most useful human-facing line out of a failure that captured output — Exit, and (D12) Timeout / Signalled (the partial streams of a hung-then-killed or crashed tool). Each of those variants' one-line Display also appends a bounded excerpt of that diagnostic (the last non-empty line, capped at 200 bytes), so a bare eprintln!("{e}") reads `git` exited with code 2: fatal: boom — actionable in a log line without dumping multi-KiB streams into it.

Escape hatch: a platform knob the crate doesn't model

Command exposes typed builders for the OS knobs that carry their weight — priority, cpu_affinity, io_priority, umask, create_no_window, windows_graceful_ctrl_break, run_as (uid/gid), parent_death, and so on. New real needs are added the same way: as a typed verb, so the command stays inspectable (its Debug, its Clone, and — with the record feature — the cassette it serialises to all stay truthful). There is deliberately no before_spawn-style hook that stores an opaque closure to mutate the raw command per launch: a stored mutator is invisible to Debug/Clone and would make a recorded cassette lie — it would replay the command as written while a different, mutated command actually ran. (The full reasoning is in decisions/before-spawn-hook-2026-07.md.)

When you genuinely hit a platform knob the crate has no verb for — a niche creation flag, your own pre_exec — the honest escape is to lower the builder to a raw tokio::process::Command and spawn it into a ProcessGroup:

use processkit::{Command, ProcessGroup};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Build the OS command exactly as ProcessKit would: program resolution,
    // environment, working directory, scheduling/umask knobs, capture-wired stdio.
    let raw = Command::new("odd-tool")
        .args(["--serve"])
        .to_tokio_command()?;

    // `raw` is a plain `tokio::process::Command`. Set the platform knob the
    // builder has no typed verb for here — e.g. a raw creation flag on Windows
    // or your own `pre_exec` on Unix.

    // Spawn INTO a group so containment still holds (see below).
    let group = ProcessGroup::new()?;
    let mut child = group.spawn(raw)?;
    let status = child.wait().await?; // the bare tokio Child is yours to drive
    let _ = status;
    Ok(())
}

What survives, and what you give up. to_tokio_command() carries over everything the builder resolves at the OS level (program/args/cwd, the layered environment, the Unix priority/umask/privilege-drop/setsid pre_exec hooks (including Linux-only io_priority and cpu_affinity), Windows creation flags, and stdio wired for capture). Windows cpu_affinity is the deliberate exception: it needs a live, still-suspended process handle, so lowering such a command fails with Unsupported instead of dropping the request. Spawning the result through ProcessGroup::spawn still enrolls the child in the group's Job/cgroup/process-group, so containment is preserved — kill-on-drop and the group-level teardown verbs still reach it. What you leave behind is the high-level machinery that lives above the OS command: the async output pump and capture, the ProcessResult/RunningProcess verbs, and the per-run timeout/cancel_on/timeout_grace/windows_graceful_ctrl_break wiring. You own the bare tokio::process::Child — draining its pipes and reaping it are your job. (On Windows, spawn re-sets creation flags for a race-free job assignment, so a creation flag left on the raw command is overwritten there; prefer the typed create_no_window on a high-level launch path — see Process groups for the full raw-spawn contract.)

Next: Streaming & interactive I/O · Timeouts, retries & cancellation · Process groups

Building typed CLI clients

CliClient is the reusable middle layer between a raw Command and a domain-specific wrapper such as Git, Jj, or Gh. It owns the program, an injectable ProcessRunner, and defaults shared by every invocation. Your wrapper owns the public vocabulary and parsing rules.

Use CliClient when several operations target the same executable, share timeout/environment/retry policy, or need hermetic tests. For one or two direct calls, Command is simpler and loses no capability.

Scaffold a wrapper

cli_client! creates a generic wrapper whose production runner is JobRunner and whose tests can inject any ProcessRunner. The generated core field is module-private: implement the tool-specific methods beside the macro invocation.

#![allow(unused)]
fn main() {
use processkit::{cli_client, Error, ProcessRunner, Result};
use std::path::Path;

cli_client!(
    /// A small typed wrapper around Git.
    pub struct Git => "git"
);

impl<R: ProcessRunner> Git<R> {
    pub async fn head(&self, repo: &Path) -> Result<String> {
        self.core
            .run(self.core.command_in(repo, ["rev-parse", "HEAD"]))
            .await
    }

    pub async fn is_clean(&self, repo: &Path) -> Result<bool> {
        self.core
            .probe(self.core.command_in(repo, ["diff", "--quiet"]))
            .await
    }

    pub async fn branches(&self, repo: &Path) -> Result<Vec<String>> {
        self.core
            .try_parse(
                self.core
                    .command_in(repo, ["branch", "--format=%(refname:short)"]),
                |stdout| {
                    let branches: Vec<String> = stdout.lines().map(str::to_owned).collect();
                    if branches.is_empty() {
                        Err(Error::parse("git", "no branches returned"))
                    } else {
                        Ok(branches)
                    }
                },
            )
            .await
    }

    pub async fn version_json(&self) -> Result<serde_json::Value> {
        self.core.output_json(["version", "--json"]).await
    }
}
}

The macro supplies Git::new(), Default, Git::with_runner(runner), and builders for the client defaults. A hand-written struct containing CliClient<R> is equally supported when you need a different layout or want to expose the core deliberately.

Build commands and choose verbs

command(args) builds program <args>; command_in(dir, args) adds a working directory. Most verbs also accept an argument list directly, so build a Command only for a per-call override:

#![allow(unused)]
fn main() {
use processkit::CliClient;
use std::time::Duration;

async fn example() -> processkit::Result<()> {
let git = CliClient::new("git").default_timeout(Duration::from_secs(30));

let version = git.run(["--version"]).await?;
let status = git
    .output_string(git.command(["status", "--short"]).timeout(Duration::from_secs(5)))
    .await?;
assert!(status.is_success());
let _ = version;
Ok(())
}
}

The verbs are the same vocabulary used by Command and ProcessRunnerExt:

parse and try_parse require an accepted exit and reject truncated capture before invoking the parser. try_parse should return Error::parse (or another typed processkit error) when output is malformed. With the json feature, output_json supplies that policy for complete JSON documents; its parse failure is ErrorReason::Parse with a bounded fragment and location.

Client defaults and precedence

Defaults are gap-filled into every command. An explicit per-command value wins.

#![allow(unused)]
fn main() {
use processkit::{CliClient, RetryPolicy};
use std::time::Duration;

let gh = CliClient::new("gh")
    .default_timeout(Duration::from_secs(30))
    .default_env("GH_PAGER", "cat")
    .default_env_remove("GH_PROMPT_DISABLED")
    .default_env_fn("REQUEST_ID", || "request-42")
    .default_retry(
        RetryPolicy::new().max_retries(2),
        |error| error.is_transient(),
    );

// The explicit 5 s timeout wins; the env and retry defaults are still filled in.
let command = gh.command(["api", "user"]).timeout(Duration::from_secs(5));
let _ = command;
}

• default_timeout, default_cancel_on, and default_retry apply wherever the command did not set its own value.
• Static default_env / default_env_remove values override a dynamic resolver for the same key. Per-command env / env_remove override both.
• default_env_fn runs synchronously once when a command is built. A retry or second run of that already-built command reuses the baked value. Use it for cheap per-operation values, not a token that must change for every retry.
• Retry only operations safe to replay, and classify typed errors instead of retrying every failure. Non-erroring capture verbs do not retry.

Defaults also apply when a verb receives a ready-made Command, but that command keeps its own program. Prefer argument lists or client.command(...) unless deliberately grafting one client's defaults onto another executable.

Error boundaries

The wrapper should preserve processkit's typed taxonomy:

• launch and resolution failures remain NotFound, Spawn, Unsupported, or Io;
• a checking verb maps an unaccepted exit to Exit and a run deadline to Timeout;
• parser failures belong in Parse, never a fabricated exit code;
• checked is intentionally lenient about truncated capture, while run, parse, try_parse, and output_json fail loud with OutputTooLarge.

This keeps callers able to branch on error.reason(), is_not_found(), is_timeout(), or is_transient() without parsing strings. See Errors for the complete taxonomy and Timeouts, retries & cancellation for replay safety.

Test the wrapper without subprocesses

ScriptedRunner matches program plus argv and returns deterministic replies. The wrapper code and parsers are the production code; only process execution is replaced.

#![allow(unused)]
fn main() {
use processkit::testing::{Reply, ScriptedRunner};
use processkit::{cli_client, ProcessRunner, Result};

cli_client!(pub struct Git => "git");

impl<R: ProcessRunner> Git<R> {
    pub async fn current_branch(&self) -> Result<String> {
        self.core.run(["branch", "--show-current"]).await
    }
}

#[tokio::test]
async fn current_branch_is_trimmed() {
    let git = Git::with_runner(
        ScriptedRunner::new()
            .on(["git", "branch", "--show-current"], Reply::ok("main\n")),
    );
    assert_eq!(git.current_branch().await.unwrap(), "main");
}
}

For a response too awkward to maintain by hand, record it once and replay the cassette in tests. RecordReplayRunner still implements ProcessRunner, so the wrapper needs no alternate code path:

#![allow(unused)]
fn main() {
use processkit::testing::RecordReplayRunner;
use processkit::CliClient;

async fn replay() -> processkit::Result<String> {
let runner = RecordReplayRunner::replay("fixtures/git.json")?;
let git = CliClient::with_runner("git", runner);
git.run(["rev-parse", "HEAD"]).await
}
}

Use replay in normal tests; keep recording as an explicit fixture-refresh step. The full cassette security and matching contract is in Testing your code.

Worked wrappers

• The .NET version shows a larger typed surface and error translation around a real CLI.
• The Python wrapper shows how the same runner/client seam maps across a language boundary.
• Cookbook: wrap a CLI tool is the short recipe; this chapter is the design and testing guide behind it.

Running many at once

‹ docs index

The batch helpers cover two distinct shapes of concurrent process work:

• output_all / output_all_bytes start a collection of Commands while retaining a fixed number of live runs, then hand back every result in input order once the whole batch finishes. output_stream / output_stream_bytes run the same bounded fan-out but yield each result the moment it finishes.
• wait_any / wait_all observe a fixed collection of RunningProcess handles that you started earlier.

They provide bounded fan-out and joins, not a scheduler or worker-pool API. Timeouts, retries, streaming, cancellation, and containment remain the normal Command and ProcessGroup primitives.

• Bounded fan-out
• Results as they finish
• Containment scope
• Racing and joining handles
• Timeouts and output
• Common batch shapes

Bounded fan-out

output_all(commands, concurrency, runner) consumes any iterator of commands and returns one result per input command, in input order. At most concurrency runs are live at once; 0 is treated as 1, and an empty input returns an empty Vec.

It is collect-all: a launch or I/O failure fills that command's Err slot, and the remaining commands still run. A non-zero exit is not an Err; it is an Ok(ProcessResult) whose code() or is_success() you inspect. Fold completed results after the batch rather than expecting the first failed command to stop it:

use processkit::{Command, JobRunner, output_all};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let conversions = (0..200)
        .map(|n| Command::new("convert").args([format!("input-{n}.png"), format!("output-{n}.webp")]));
    let results = output_all(conversions, 8, &JobRunner).await;
    let failed = results
        .iter()
        .filter(|result| !matches!(result, Ok(output) if output.is_success()))
        .count();
    println!("{failed} conversions failed");
    Ok(())
}

output_all_bytes has identical scheduling and error semantics, but each ProcessResult contains Vec<u8> stdout. Use it when stdout is an artifact, such as an archive, image, or git cat-file object:

use processkit::{Command, JobRunner, output_all_bytes};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let commands = [
        Command::new("git").args(["cat-file", "blob", "HEAD:logo.png"]),
        Command::new("git").args(["cat-file", "blob", "HEAD:icon.png"]),
    ];
    for blob in output_all_bytes(commands, 2, &JobRunner).await {
        println!("captured {} bytes", blob?.stdout().len());
    }
    Ok(())
}

Like one-shot capture verbs, output_all* waits for and captures every command's output. The vector appears only after the entire batch finishes; dropping its future returns no partial vector. See Timeouts and cancellation when cancellation is part of the design.

Results as they finish

output_stream(commands, concurrency, runner) is the streaming sibling of output_all: the same bounded fan-out — the same concurrency cap, the same per-command error semantics, the same containment — but instead of one Vec at the very end it returns a Stream that yields each result the instant that command finishes. Each item is an (input index, result) pair, so a result is still traceable to the command that produced it even though items arrive in completion order rather than input order. Drive it with StreamExt::next (the crate re-exports StreamExt from processkit::prelude):

use processkit::{Command, JobRunner, output_stream};
use processkit::prelude::StreamExt;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let conversions = (0..200)
        .map(|n| Command::new("convert").args([format!("input-{n}.png"), format!("output-{n}.webp")]));
    let runner = JobRunner;
    let mut results = output_stream(conversions, 8, &runner);
    while let Some((index, result)) = results.next().await {
        // Handle each conversion as soon as it lands, without waiting for the
        // slowest one in the batch.
        match result {
            Ok(output) if output.is_success() => {}
            _ => eprintln!("conversion #{index} failed"),
        }
    }
    Ok(())
}

examples/batch_stream.rs is a self-contained completion-order demo that launches child copies of itself.

Reach for output_stream over output_all when either of these matters:

• First result early. A fast command is handed back immediately instead of waiting behind the slowest in the batch — useful for progress reporting, or to start downstream work on the first artifact.
• Partial results survive cancellation. Every result you have already pulled from the stream is yours; dropping the stream mid-fan-out keeps them. This is the gap output_all cannot cover — its Vec materializes only at the end, so dropping its future discards even the commands that had already completed.

Cancellation and containment work exactly as for output_all. Dropping the stream drops the in-flight command futures: with &JobRunner (an own group per run) that kills every still-live process tree with no orphans; with a shared &group those children live until you tear the group down. Commands still waiting for a concurrency slot are dropped without ever being spawned — a queued command runs nothing until it is scheduled, so cancelling the fan-out cancels them for free.

output_stream_bytes is the raw-bytes twin (each ProcessResult carries Vec<u8> stdout), with identical scheduling, ordering, and cancellation — the streaming counterpart of output_all_bytes. In fact output_all is exactly output_stream collected back into input order: both are the same fan-out engine, so their concurrency and no-short-circuit guarantees cannot drift apart.

Containment scope

The runner argument controls containment, not only testability.

Use &JobRunner for independent conversions. Use a shared group when every process must disappear together:

use processkit::{Command, ProcessGroup, output_all};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let checks = [
        Command::new("check-shard").arg("a"),
        Command::new("check-shard").arg("b"),
    ];
    let results = output_all(checks, 2, &group).await;
    assert_eq!(results.len(), 2);
    group.shutdown().await?;
    Ok(())
}

Do not choose a shared group merely because the commands began together. It couples their fate: cancelling output_all leaves shared-group children under the group's lifetime, while each JobRunner run owns a private group. See Process groups for the teardown model.

Racing and joining handles

Use wait_any and wait_all when you need live handles — for readiness, incremental stdin, or a decision before workers finish — rather than only final captured output. Both functions borrow their handles, so those handles remain usable afterwards.

wait_any returns the index and outcome of the first child to exit. A mirror race can then stop its loser by shutting down the shared group:

use processkit::{Command, ProcessGroup, wait_any};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let mut primary = group.start(&Command::new("fetch-from").arg("primary")).await?;
    let mut mirror = group.start(&Command::new("fetch-from").arg("mirror")).await?;
    let (winner, outcome) = wait_any(&mut [&mut primary, &mut mirror]).await?;
    println!("mirror #{winner} finished with {outcome:?}");
    group.shutdown().await?;
    Ok(())
}

wait_all joins a fixed worker set and returns every outcome in slice order:

use processkit::{Command, ProcessGroup, wait_all};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let mut workers = Vec::new();
    for shard in ["a", "b", "c", "d"] {
        workers.push(group.start(&Command::new("quiet-worker").arg(shard)).await?);
    }
    let mut borrowed: Vec<_> = workers.iter_mut().collect();
    let outcomes = wait_all(&mut borrowed).await?;
    println!("joined {} workers: {outcomes:?}", outcomes.len());
    Ok(())
}

wait_any rejects an empty slice because no child can win; wait_all accepts one and returns an empty vector. A non-zero exit or signal is an Outcome, not an error. wait_all can return the first cancellation, stdin, or reaping error; its remaining handles are still waitable.

Timeouts and output

wait_any and wait_all intentionally add neither of these features:

1. Per-process timeout. Put Command::timeout on each command before start, or wrap the complete wait in tokio::time::timeout when one deadline belongs to the collection.
2. Output pumping. A chatty child can fill stdout or stderr and block before exit. Drain it with stdout_lines / stderr_lines, a line handler, or a tee; use capture-oriented output_all* when incremental output is unnecessary.

An individual deadline belongs to its command:

#![allow(unused)]
fn main() {
use processkit::Command;
use std::time::Duration;

let worker = Command::new("worker").timeout(Duration::from_secs(30));
}

One collection deadline is explicit instead:

#![allow(unused)]
fn main() {
use processkit::{Command, ProcessGroup, wait_all};
use std::time::Duration;
async fn example() -> processkit::Result<()> {
let group = ProcessGroup::new()?;
let mut worker = group.start(&Command::new("quiet-worker")).await?;
let outcome = tokio::time::timeout(
    Duration::from_secs(60),
    wait_all(&mut [&mut worker]),
).await;
let _ = outcome;
Ok(())
}
}

The outer timeout only stops waiting. Choose containment deliberately, then explicitly shut down or retain children according to your policy. For streaming patterns, see Streaming & interactive I/O.

Common batch shapes

• Bounded file conversion: generate one Command per file and pass the iterator to output_all with a cap chosen for CPU, memory, file descriptors, and the external tool's limits. Fold all results to report the full failure set.
• Mirror race: start one RunningProcess per endpoint in a shared ProcessGroup, call wait_any, preserve the winner as needed, then shut down the group to stop losers.
• Fixed worker join: start known quiet or independently-drained workers, retain their handles in a Vec, and pass mutable references to wait_all for ordered outcomes.

For single-command recipes, return to the Cookbook.

Comparative benchmarks

processkit adds process-tree containment, decoded line streaming, bounded capture, and a consistent async API around child processes. Those guarantees have a cost, so this repository keeps an end-to-end comparison against the plain Tokio and standard-library APIs. The benchmark is intended to answer "what does the convenience and safety layer cost for this workload?", not to declare one API universally faster.

On Windows there is a second question the end-to-end comparison cannot answer: creating a process costs tens of milliseconds all by itself on a machine with real-time antivirus, so an absolute "start took N ms" number attributes nothing. benches/win_spawn_phases.rs answers it by splitting the fixed start cost into phases; see Where a Windows start goes.

What is measured

benches/compare.rs runs the same real passthrough child (cat on Unix, cmd /c findstr on Windows) for every contender. Payload construction is outside the timed sections where possible. Each sample includes child spawn, pipe setup, stdin/stdout transfer, output handling, and process wait. The children are intentionally trivial: they only echo stdin to stdout, so the measurement is about process handling rather than application work.

The recorded run used an Intel Core i9-12900H CPU with 20 logical processors, Windows 11 Enterprise (build 10.0.26200) with Defender real-time protection enabled, and Rust 1.93.0. The machine had 514 live processes and 7,761 live threads while the numbers were taken — a figure that matters for the start-cost attribution below, and that a quiet CI runner will not reproduce. Criterion was configured in benches/compare.rs::configure with sample_size=20, warm_up_time=10s, and measurement_time=5s for each series. The table reports the Criterion mean and standard deviation; throughput is derived from the fixed workload size (it is not a separately configured Criterion throughput measurement). Deltas use the std_process result in the same scenario as the baseline. Absolute values are machine-dependent: CPU, OS process-spawn cost, antivirus, scheduler load, filesystem state, Rust toolchain, and available Tokio runtime threads can all change them. Compare contenders only within the same invocation and retain the command and environment with any published result.

One Windows-specific measurement hazard is handled inside the benchmark rather than left to the reader. A host with behavioural monitoring throttles a process that abruptly starts creating children and only settles tens of seconds later, which is longer than Criterion's per-series warm-up; without intervention that whole penalty lands on whichever series runs first, and it was observed making the first contender read two to four times its value from a later run while every other series stayed stable. benches/compare.rs::prime_process_creation therefore spawns real children, recording nothing, until batch times stop falling, and only then lets the first series start. On a host that does not throttle it costs a few seconds.

Measured results

These measurements were produced by cargo bench --bench compare on the host described above. Times are end-to-end and shown as mean +/- standard deviation. The throughput units are runs/s for small capture, MiB/s for the 1,032,000-byte stream, and child processes/s for the 16-child fan-out.

Where a Windows start goes

The small-capture delta above is a few milliseconds on a host where creating the child itself costs about 50. Interpreting that requires knowing which part of a start belongs to ProcessKit and which to the operating system, so benches/win_spawn_phases.rs measures the phases directly. It uses a fixed absolute-path child (cmd /c exit) so no series pays for a program lookup, and it is configured with sample_size=50, warm_up_time=15s, and measurement_time=20s.

os_spawn_plain is the floor: CreateProcess and wait, nothing else. The os_spawn_suspended_resume_* series add ProcessKit's CREATE_SUSPENDED and resume cycle to it, and the containment_sequence_* series add Job Object creation and assignment on top of that, mirroring sys::windows::Job::{new, spawn} step for step. Within each pair, ..._snapshot and ..._direct differ only in how the launcher finds the primary thread of the suspended child it must release: the documented system-wide TH32CS_SNAPTHREAD ToolHelp snapshot, or the per-process ntdll!NtGetNextThread walk ProcessKit uses (falling back to the snapshot when that entry point is unavailable).

The primitives behind those phases, measured with no child process involved so that a spawn outlier cannot hide them:

The two tables reconcile. os_spawn_plain (22.527 ms) plus thread_snapshot_walk (73.990 ms) plus job_object_create (0.004 ms) is 96.521 ms against a measured containment_sequence_snapshot of 97.695 ms, leaving 1.2 ms for the assignment, the per-thread open and resume, and the suspended spawn itself. The same sum with direct_thread_walk instead is 22.540 ms against a measured 22.661 ms.

Read together:

• Creating the Job Object is free. At 4.5 microseconds it is four orders of magnitude below the CreateProcess it contains. Whole-tree containment on Windows is not what makes a start expensive, and no amount of deferring or pooling the container could pay for itself.
• Finding the suspended child's primary thread was the whole cost. The only documented pid-to-thread mapping on Windows is a snapshot that is system-wide — the process-id argument is ignored for thread lists — so it materialises all 7,761 threads on the machine to locate the one thread that was just created. At 74 ms that was 3.3x the cost of the CreateProcess it followed, and it dominated everything else in the start path put together. Asking the same question per-process instead answers it in 8.6 microseconds, about 8,600 times faster, and collapses the whole containment sequence to within noise of a plain unguarded spawn (+0.6%).
• Program lookup is the remaining ProcessKit-specific cost, and it is under the caller's control. Resolving a bare name across PATH x PATHEXT — which ProcessKit does so that a launch spawns exactly what the spawn-free Command::resolve_program preflight reports, including a .cmd/.bat the OS's own .exe-only search would never find — cost 5.4 ms against 37 microseconds for a program named by an absolute path, on a PATH with 77 entries and a PATHEXT with 14 extensions. That is the processkit versus processkit_resolved_program gap in the table above. A caller starting the same program repeatedly can resolve it once with Command::resolve_program and pass the resulting path.
• Serialising the spawn call itself is not the bottleneck. ProcessKit holds one process-global lock across every child creation, so an ordinary spawn cannot observe the ConPTY path's temporary process-global standard handles. Running the same 16 spawns with and without that lock measured 131.893 +/- 24.799 ms parallel against 150.864 +/- 62.608 ms serialised. That +14.4% sits inside the serialised series' own standard deviation, and a repeat run put the same pair at -1.2%; the two runs disagree in sign, so no reliable penalty is resolved at this noise level. Windows serialises process creation internally in any case. What it does rule out is the lock as an explanation for a slow fan-out — that was the same per-spawn thread snapshot, paid once per child.

Running the comparison

Run the complete local suite with:

just bench-compare

The equivalent Cargo command is:

cargo bench --bench compare

The Windows start-cost attribution is a separate bench:

just bench-win-phases

cargo bench --bench win_spawn_phases

Criterion's normal filtering and measurement options remain available, for example cargo bench --bench compare -- stream_large_stdout. Both benchmarks spawn real children and are deliberately not part of the ordinary CI test gate; use them when comparing a change, a platform, or a toolchain. win_spawn_phases is Windows-specific and prints an explanatory message elsewhere.

Reading the result

The comparison is meaningful only when the workloads and semantics match:

• processkit measures its normal private process-group setup. This is the price of its unconditional kill-on-drop tree guarantee, and is not present in the plain baselines. On Windows the phase table above shows what that actually costs.
• The small-capture group includes a processkit_resolved_program series using an absolute program path and a processkit_discard_stdout series using stdout(StdioMode::Null) + start().wait(). Compare each with processkit in the same group to see the crate's program-lookup and capture/pump costs separately; StdioMode::Null is a discard path, so it is not an output-equivalent replacement for capture.
• The capture case compares a decoded ProcessResult<String> with byte output from the plain APIs. The benchmark uses ASCII payloads so decoding does not change the transferred content; the APIs still have deliberately different result and failure semantics.
• The streaming case consumes every line before waiting for the child. This prevents a full stdout pipe from turning the benchmark into a deadlock and measures the live line-delivery path rather than only a bulk read.
• The fan-out case gives each API its normal concurrency primitive: Tokio tasks for async contenders and scoped standard threads for std. It measures the whole batch, not a synthetic per-child loop.

Conclusions

Using a 10% delta as a practical negligible-overhead threshold, the small capture (+3.7%) and concurrent fan-out (+9.2%) were negligible relative to the direct std_process API in this run, and large streaming (+16.9%) was not. The plain Tokio contender stayed within 5% of the standard-library baseline in all three scenarios. Small capture with an absolute program path (-1.1%) was indistinguishable from the baseline, which places the crate's remaining short-run overhead in program resolution rather than in containment.

For a short-lived Windows child, expect ProcessKit's own fixed start cost to be a few milliseconds against a CreateProcess that costs tens of them on an antivirus-equipped host — dominated by the PATH/PATHEXT lookup when the program is named by a bare name, and essentially nothing when it is not. If you are diagnosing a slow start, measure the difference against a plain spawn of the same command on the same machine, as spawn_capture_small does, before attributing it to this crate: the absolute number is mostly the operating system's.

The remaining streaming cost is the trade-off for ProcessKit's decoded, bounded-capture line delivery. Choose ProcessKit when containment, streaming, bounded capture, and typed outcomes matter; choose a plain process API when minimum overhead on a bulk transfer is the priority and those guarantees are not required. These are single-machine measurements, so they describe this workload and environment, not a universal performance ranking. For this Windows run, std_process is the direct CreateProcess-based reference; Unix users should rerun the benchmark before treating it as a comparison with their platform's fork/exec path.

If a change improves one scenario and regresses another, report the scenario and its Criterion output rather than collapsing the results into one score. Containment, streaming, and typed outcome handling are features; a faster plain spawn is not automatically a better substitute for them.

Process groups

‹ docs index

A ProcessGroup ties the lifetime of a whole child-process tree to a Rust value: every process spawned into the group — and everything those processes spawn — is killed when the group is dropped. An exiting, panicking, or ?-returning owner never leaks subprocesses; the kernel object enforcing this (Job Object / cgroup / POSIX process group) catches even grandchildren you never knew about. (Killing grandchildren is the problem duct.py's gotchas list files under "currently unsolved" for pipe-based designs — kernel containment is the solution, and the reason this crate exists.)

• Creating a group
• Putting processes in
• Tearing down: drop, terminate, shutdown
• Signalling the whole tree
• Asking whether a soft stop is available
• Suspending and resuming
• Listing members
• Resource limits
• Stats and sampling

Creating a group

use processkit::{ProcessGroup, ProcessGroupOptions};
use std::time::Duration;

fn main() -> processkit::Result<()> {
    // Defaults: 2s graceful-shutdown grace, escalate to SIGKILL.
    let group = ProcessGroup::new()?;

    // Tuned:
    let group = ProcessGroup::with_options(
        ProcessGroupOptions::default()
            .shutdown_timeout(Duration::from_secs(10))
            .escalate_to_kill(true),
    )?;

    // Which kernel mechanism is actually containing the tree?
    println!("{:?}", group.mechanism()); // JobObject | CgroupV2 | ProcessGroup
    Ok(())
}

mechanism() reports what you actually got: CgroupV2 quietly falls back to ProcessGroup on Linux hosts without cgroup delegation (see Platform support).

You rarely create a group explicitly for one-shot runs: every Command::run()-style call makes a private group automatically. Reach for an explicit group when several children should share one fate, or when you need the group verbs below.

Putting processes in

Four doors, in order of preference:

use processkit::{Command, ProcessGroup};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let group = ProcessGroup::new()?;

    // 1. start(): the full Command experience (capture, streaming, timeouts) in a
    //    SHARED group. The handle does not own the group — dropping the handle
    //    kills that child, dropping the group kills everyone.
    let server = group.start(&Command::new("dev-server")).await?;

    // 2. spawn(): the raw escape hatch for a tokio::process::Command you already
    //    have. You get the bare Child back; pipes and reaping are your problem.
    //    spawn() takes the command BY VALUE (reuse would stack pre-exec hooks).
    let raw = tokio::process::Command::new("background-helper");
    let child = group.spawn(raw)?;

    // 3. adopt(): contain a child that was spawned OUTSIDE the group, while you
    //    still hold its Child handle (and stay responsible for reaping it).
    let external = tokio::process::Command::new("legacy-launcher").spawn()?;
    group.adopt(&external)?;

    // 4. adopt_external(): contain a process you have only a pid for — one an
    //    outside supervisor started, or one you forked but never handed over.
    //    Nothing here will ever reap it; the group can only signal it.
    let pid_from_elsewhere: u32 = 4321;
    let _ = group.adopt_external(pid_from_elsewhere);
    let _ = (server, child);
    Ok(())
}

Both adoption doors move only the named process: descendants it already has keep their old containment (future forks are captured — on Windows/cgroup). A few sharp edges worth knowing:

• A child that already exited but has not been reaped (no wait() yet — a zombie whose pid/handle is still valid) is a successful no-op: there is nothing left to contain, so adopt returns Ok on the containment backends.
• A child that already exited and was reaped (wait()ed) has no pid left — adopt returns an error rather than silently tracking nothing.
• On the POSIX process-group mechanism, a child that has already exec'd can't be re-grouped (POSIX forbids it), so it is tracked individually: the child itself is signalled/killed with the group, but its future forks are not. The caller keeps the Child handle and is responsible for reaping.

Adopting by pid (adopt_external)

adopt needs a live tokio::process::Child, which a non-Rust consumer cannot construct at all. adopt_external(pid) takes the one identifier such a caller does have — and treats it as an address, not as a handle, because the OS may give a reaped process's number to an unrelated one:

• the crate captures an identity anchor of its own for the process the number currently names, during the call: the process object behind an OpenProcess on Windows, kernel cgroup membership (plus a /proc start-time read on either side of the write) on Linux cgroup v2, the start-time token on the POSIX process-group backends. From then on the group's probes, signals and teardown are bound to that, so a process that recycles the number afterwards is not a member and is not signalled;
• a number recycled during the call itself is detected by that closing read and reported as an error — but the two unix mechanisms are left in different states, which the error message spells out. The process-group backends have nothing to undo (the entry they made is identity-gated and is pruned unsignalled). Linux cgroup v2 has already migrated a task, so the call tries to move the number back out into the cgroup this group's own directory lives in; where a host refuses that move, the process holding the number stays a member of this group and this group's teardown will kill it. Windows cannot reach this state — it uses the number once, for OpenProcess;
• adoption is not neutral for containment the process already has, and the direction differs per mechanism: on Windows the process keeps its existing job and this group's job becomes a child of it (so the outer job's terminate/close then reaches this group's members, including ones started later, and its limits bind them — and whether the assign succeeds at all depends on this group's own state, so adopt before you start); on Linux cgroup v2 the process loses its previous cgroup, since v2 membership is exclusive and nothing can restore it; on the process-group backends nothing is taken away. See platform support for the full note;
• what no crate can check is the window before the call — whether the pid still named the process you meant when you looked it up. Look it up as late as you can, and use process_is_alive(pid, start_time) to re-check an instance later;
• nothing here ever reaps it. No exit status for an adopted-by-pid process appears anywhere in this API; the group can signal it and list it, and that is all. Reaping stays with whoever is its actual parent — this process, an outside supervisor, or init;
• FreeBSD and the other BSDs return Unsupported. No start-time reader is wired up there, so there is no anchor to capture, and the crate refuses rather than tracking a bare number it would later SIGKILL. adopt is unaffected — the Child you hold un-reaped is what keeps its number from being recycled.

pid == 0 and this process's own pid are refused everywhere: both would point the group's own teardown at the caller.

Tearing down: drop, terminate, shutdown

use processkit::{Command, ProcessGroup, ProcessGroupOptions};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::with_options(
        ProcessGroupOptions::default()
            .shutdown_timeout(Duration::from_secs(5))
            .escalate_to_kill(true),
    )?;
    let _service = group.start(&Command::new("my-service")).await?;

    // SIGTERM, give it 5s to flush and exit, SIGKILL stragglers:
    group.shutdown().await?;
    Ok(())
}

A child that handles SIGTERM ends the grace early — shutdown returns as soon as the tree is empty, not after the full timeout. One subtlety: the liveness probe sees an exited-but-unreaped child (a zombie) as alive on the process-group backends, so keep wait()ing your handles concurrently if you want the early return. Drop can't await, which is why the graceful tier lives in this async method — dropping without calling it performs only the hard kill.

Windows: the graceful soft tier (WM_CLOSE, opt-in CTRL_BREAK)

A Windows shutdown has no POSIX SIGTERM, but it still tries to trigger a clean exit before the atomic Job Object kill. For a windowed child (Electron app, desktop tool, windowed service) this is automatic: WM_CLOSE is posted (never sent, so a hung window can't block us) to every top-level window a live member owns, then the same signal → wait → escalate ladder runs — the child gets the shutdown_timeout to flush and exit, else TerminateJobObject. A console child has no window, so opt in per child with Command::windows_graceful_ctrl_break(): the direct child is spawned in its own console process group (CREATE_NEW_PROCESS_GROUP), and shutdown then sends it GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, pid), waits the shutdown_timeout, and TerminateJobObjects any survivor — the very same signal → wait → escalate ladder as Unix, so a console child that handles CTRL_BREAK shuts down softly.

use processkit::{Command, ProcessGroup, ProcessGroupOptions};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::with_options(
        ProcessGroupOptions::default().shutdown_timeout(Duration::from_secs(5)),
    )?;
    // CTRL_BREAK is sent on shutdown; a console child gets 5s to exit, else kill.
    let _service = group
        .start(&Command::new("my-service").windows_graceful_ctrl_break())
        .await?;
    group.shutdown().await?;
    Ok(())
}

The CTRL_BREAK opt-in is console-only: a child spawned create_no_window or DETACHED_PROCESS does not share this process's console, so it never receives the event and rides the grace to the TerminateJobObject fallback. Only the direct child is addressed by CTRL_BREAK — an adopted child is not — and the event is CTRL_BREAK, not CTRL_C (a new process group disables CTRL_C). The automatic WM_CLOSE path is the complement: it reaches any live member that owns a top-level window (including forked descendants and adopted children), needs no console and no opt-in, but only a member that actually has a window. A member with neither a window nor the console opt-in is hard-killed promptly at the deadline. Off Windows the builder is a no-op.

Observing the teardown: stop and ShutdownReport

Requires the default-on process-control feature (the report carries a Signal).

shutdown/shutdown_ref are fire-and-forget — they report only success or an error. When you own your own end-of-run race (a timeout ⨯ Ctrl-C ⨯ control-socket race, not a Command::timeout) you usually want to stop the instant the tree is empty rather than always spend the whole grace, and to report the tier the kernel observed rather than what you tried. ProcessGroup::stop(grace, escalate) is that verb: the same SIGTERM / CTRL_BREAK / WM_CLOSE → wait → escalate ladder, taking grace/escalate explicitly and returning a ShutdownReport.

use processkit::{Command, ProcessGroup};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _service = group.start(&Command::new("my-service")).await?;

    // SIGTERM, up to 5s to drain, then SIGKILL survivors — and report what happened.
    let report = group.stop(Duration::from_secs(5), true).await?;
    if report.drained_within_grace() {
        println!("clean exit in {:?}", report.elapsed());
    } else if report.escalated() {
        let survivors = report.members_after().unwrap_or(0);
        eprintln!("hard-killed {survivors} survivor(s) after the grace");
    }
    if let Some(sig) = report.attempted_signal() {
        println!("attempted soft signal: {sig:?}");
    }
    Ok(())
}

The report is honest per platform. members_before/members_after count the same member set as members — the whole tree on the Job Object / cgroup mechanisms, the tracked group leaders on the process-group fallback (where an unreaped zombie still counts, so reap your handles for a true members_after). The soft_signal() verdict is three-way — SoftSignal::Sent(sig), SoftSignal::Failed(sig), or SoftSignal::Unsupported; the last arises only on a windowless Windows Job Object with no console-CTRL leader (every Unix mechanism always has a real SIGTERM tier). stop(Duration::ZERO, true) is the "kill and wait" path: it hard-kills at once (a zero grace waits not at all) and reports what was still live — where bare kill_all returns as soon as the kill is issued. shutdown/shutdown_ref are unchanged; stop is purely additive.

ShutdownReport is the teardown facts as a typed value after the teardown returns. For the same transitions live — stamped the instant each one happens — enable the tracing feature: the teardown driver narrates soft_signal → grace_started → drained | escalated | spared (each in a stable phase field) on the processkit target, for every graceful path (stop, shutdown, a run-level timeout_grace, a supervisor's graceful stop). The two are one seam read two ways (both derive from the same driver outcome); neither can influence the teardown, and neither carries argv/env.

Deliberately detaching a child (spawn_detached)

This inverts the crate's headline guarantee — on purpose. Everything above is about keeping a tree contained so nothing escapes. Command::spawn_detached is the crate's one deliberate escape hatch for the opposite need: a child that must outlive its launcher — daemonizing, a nohup-style long-lived helper, a handoff to a process you want to keep running after this one exits.

use processkit::Command;

fn main() -> processkit::Result<()> {
// Launch a helper that survives this process. Its stdout goes to a file — never
// a pipe, which would deadlock the child once nothing is left to drain it.
let child = Command::new("my-daemon")
    .arg("--serve")
    .stdout_file("/var/log/my-daemon.log")
    .spawn_detached()?;
println!("detached daemon pid = {}", child.pid());
// Dropping `child` does NOT kill the daemon — the crate is done with it.
Ok(())
}

For a safe runnable demonstration whose detached child exits by itself, see examples/detached.rs.

What it does, and what it deliberately does not:

• Detach at birth. Unix — a new session (setsid), no controlling terminal. Windows — the child is not assigned to this crate's Job Object. It is not made to break away from a Job Object / cgroup the host already put your process in (a CI runner, a systemd scope, this crate's own supervisor): that would be hostile to whoever set up the host containment. So a detached child escapes this crate's per-run containment, not a broader host one it inherits.
• A separate, non-interchangeable type. You get a DetachedChild carrying only the pid — no public kill, wait, timeout, capture, or control/teardown APIs — because it is no longer contained. Dropping it does nothing to the child. On Unix, a private background reaper owns the detached Child and collects its exit status so it does not become a zombie; that bookkeeping is not a public wait or control API. Windows and other non-Unix targets use their normal process-handle cleanup instead. (Left as a bare code span, not a docs.rs link: this type ships in the next release, so a docs.rs URL would 404 until then.)
• stdio is null, or a file — never a pipe. With no owner left to drain it, a pipe would deadlock the child the moment its buffer fills. stdout/stderr are null by default; the only alternative is a file redirect (stdout_file / stderr_file). stdin is always null.
• Incompatible knobs are refused loudly. A Command carrying a timeout, capture wiring (on_stdout_line/tees/capture_policy), an interactive stdin (keep_stdin_open/inherit_stdin/a stdin source), retry, cancel_on, kill_on_parent_death (its exact opposite), windows_graceful_ctrl_break, cpu_affinity, or Linux io_priority is rejected with a typed ErrorReason::Unsupported naming it — never silently ignored. Program/args/env/working-directory and the privilege-drop knobs (uid/gid/groups/umask/priority) are honored.

Reach for spawn_detached only when you truly want a child to outlive its launcher. For everything else, start/run/output_* keep the child contained.

Signalling the whole tree

signal/suspend/resume/members/adopt — this section and the two below — require the default-on process-control feature. The teardown verbs above are core and always present.

use processkit::{Command, ProcessGroup, Signal};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _server = group.start(&Command::new("my-server")).await?;

    group.signal(Signal::Hup)?;        // "reload your configuration"
    group.signal(Signal::Usr1)?;       // whatever the tool defines
    group.signal(Signal::Other(34))?;  // raw signal number escape hatch
    Ok(())
}

Signal::Kill always takes the same atomic whole-tree kill path as kill_all (cgroup.kill / PROC_REAP_KILL / killpg / job terminate), so it cannot miss a process forked mid-broadcast. Other signals are a per-member broadcast — best-effort against a tree that is forking at that exact moment. On Windows, Signal::Int/Signal::Term do not wait or escalate (they only trigger a soft close — contrast the graceful shutdown, which then waits the grace and escalates). An empty group accepts any deliverable signal trivially — except Windows Int/Term, which report Unsupported on an empty group (no member, hence no console or windowed target to soft-close). On every Unix mechanism a real send failure is surfaced as an Err rather than swallowed — an EINVAL (an out-of-range Other(n)) always, and an EPERM against a live, non-zombie member (a sudo/setuid child that rejects the signal, or a seccomp/container restriction). The process-group mechanism (macOS/the other BSDs, Linux-without-cgroup) reaches the same verdict as the cgroup one by checking the target's run state after an EPERM, so a harmless zombie-only EPERM — and, on the bare BSDs where no state reader exists, every EPERM — stays swallowed. The FreeBSD reaper makes that discrimination too, from the kernel's own zombie flag on the member PROC_REAP_KILL names as the failing one — so unlike the bare-BSD process-group path it does surface a live member's EPERM. An ESRCH race (the member already exited) is still success. Signal::Other(0) is the POSIX existence probe: it returns Ok having delivered nothing (a live target was reached, not signalled) — and because that probe never takes a delivery path (FreeBSD routes it back through the process group, which has no state reader on any BSD but macOS), the EPERM rule above does not reach it everywhere: on FreeBSD and the bare BSDs a live target that rejects even the null signal still answers Ok, where Linux and macOS surface the EPERM. suspend/resume on the process-group mechanism now use the same honest delivery verdict for SIGSTOP/SIGCONT: a live-member EPERM surfaces as an Err, while ESRCH, zombie-only EPERM, an empty group, and BSD-without-state-reader EPERM remain Ok. The FreeBSD reaper applies that verdict to freezing and thawing too — SIGSTOP/SIGCONT ride the same PROC_REAP_KILL classification as any other signal there. The older-kernel cgroup per-process fallback reports its SIGSTOP/SIGCONT failures the same way.

Asking whether a soft stop is available

Before you fire a soft stop (signal(Signal::Term) / Signal::Int), you can ask the group whether one will actually reach anything — soft_stop_scope() returns a SoftStopScope capability report, so a caller cancelling a run on its own schedule (a UI Cancel, a control-socket command, a timeout it owns) can decide up front whether to attempt a graceful stop and can tell its user the real reach, instead of firing a signal, catching ErrorReason::Unsupported, and reverse-engineering the scope. It is the group-axis sibling of Command::kill_on_parent_death_scope() -> ParentDeathCleanup, but read from the group's live membership (not fixed per platform) and side-effect-free — it delivers no signal, posts no WM_CLOSE, spawns nothing, and does not mutate the group.

use processkit::{Command, ProcessGroup, Signal, SoftStopScope};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _server = group
        .start(&Command::new("my-server").windows_graceful_ctrl_break())
        .await?;

    // Decide BEFORE attempting — no ErrorReason::Unsupported to parse back.
    match group.soft_stop_scope() {
        SoftStopScope::WholeTree | SoftStopScope::OptInMembers => {
            group.signal(Signal::Term)?; // a soft stop will reach a member
        }
        SoftStopScope::Unsupported => {
            group.kill_all()?; // no soft tier here — go straight to the hard kill
        }
        other => eprintln!("unknown soft-stop scope: {}", other.name()),
    }
    Ok(())
}

Consistent with signal by construction: it reads the very same live-membership primitives signal(Int/Term) acts on, so what it reports matches what a real soft stop would then reach. It describes the soft tier only — the unconditional hard kill (Signal::Kill, kill_all, dropping the group) always tears the whole tree down regardless. SoftStopScope is #[non_exhaustive] and carries a stable name() / from_name() machine identifier (whole_tree / opt_in_members / none), like the other reporting enums.

Suspending and resuming

Freeze a tree (to snapshot it, to starve a runaway while you investigate, to pause background work), then thaw it:

use processkit::{Command, ProcessGroup};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _cruncher = group.start(&Command::new("cpu-hog")).await?;

    group.suspend()?;   // the whole tree stops consuming CPU
    // … inspect, snapshot, wait for the user …
    group.resume()?;
    Ok(())
}

Per-platform machinery — and its visible differences:

Two caveats that bite in practice:

• Spawning into a suspended group diverges. Under the cgroup mechanism a child spawned or adopted while the group is frozen starts frozen — and start() may never return until resume (the forked child joins the cgroup before exec, so it can freeze before completing the spawn handshake). Windows and the pgroup backends freeze only members present at the call. Rule of thumb: resume before starting new work.
• A suspended tree can still be hard-killed (drop / kill_all / Signal::Kill all act on frozen processes), but a graceful shutdown starts with a SIGTERM the frozen tree can't act on — it would wait out the whole grace. Resume first for a clean shutdown.

Listing members

use processkit::{Command, ProcessGroup};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _a = group.start(&Command::new("worker-a")).await?;
    let _b = group.start(&Command::new("worker-b")).await?;

    let pids: Vec<u32> = group.members()?;
    println!("live members: {pids:?}");
    Ok(())
}

What "members" means depends on the mechanism: Windows and Linux-cgroup list the whole tree (every descendant pid); the POSIX process-group backends list the tracked group leaders (one pid per started/adopted child) — their descendants are contained but not enumerated. An exited child still counts until it is reaped. The snapshot is point-in-time: a tree that is forking races it.

To wait on members rather than list them, race the handles with wait_any.

Enriched snapshot: members_info

When bare pids aren't enough — a diagnostic members_snapshot event, a process-tree view — members_info returns the same member set as members, but each pid comes wrapped in a MemberInfo carrying best-effort parent pid, image name, and start time:

use processkit::{Command, ProcessGroup};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _a = group.start(&Command::new("worker-a")).await?;

    for m in group.members_info()? {
        println!(
            "pid={} ppid={:?} exe={:?} start={:?}",
            m.pid(),
            m.ppid(),
            m.exe_name(),
            m.start_time(),
        );
    }
    Ok(())
}

The fields are read where the platform can report them and are None otherwise — never a fabricated value. Windows and Linux (both the cgroup and /proc fallback paths) and macOS fill all four; on the bare BSDs only the pid is reported and the rest are None. start_time is an opaque identity anchor (its unit and epoch differ per platform), not a wall-clock timestamp — its use is pairing with the pid to tell a recycled number apart from the original process, not display. The raw command line is deliberately never included on any platform: it routinely carries secrets, and redaction is the consumer's policy to own.

Same point-in-time contract as members, with one addition: if a member exits between its pid being enumerated and its metadata being read, that pid is skipped rather than reported with fabricated fields — a single vanished member never fails the whole call.

Identifying a process by pid (outside a group)

Sometimes the pid you care about is not a member of any group you own — a pid saved to disk between runs, a launch registry checking whether the owner of a crash-surviving entry is still alive, an e2e probe watching a process from outside its container. For that, the crate publishes the same identity query as a free-standing function (needs process-control):

fn main() -> processkit::Result<()> {
let pid = 4321;

// Look up an arbitrary pid — the standalone twin of `members_info`, returning the
// same best-effort `MemberInfo` (parent pid, image name, start time).
match processkit::process_info(pid)? {
    Some(info) => println!(
        "pid={} ppid={:?} exe={:?} start={:?}",
        info.pid(), info.ppid(), info.exe_name(), info.start_time(),
    ),
    None => println!("pid {pid} is not running"),
}
Ok(())
}

process_info returns three distinct outcomes, and the distinction is the point:

• Ok(Some(info)) — the process exists; fields are best-effort Option exactly as in members_info.
• Ok(None) — the pid names no process: an honest negative, the "it's gone" answer a liveness check wants.
• Err — the process may well exist, but you couldn't inspect it (no permission — a Windows protected/System process, a Linux hidepid mount, a macOS restricted process — or an OS read error). Never read this as "dead." That is the whole reason it is an error rather than Ok(None).

It reads no argv/environment, on any platform — the same "never argv/env" stance MemberInfo documents.

Reuse-safe liveness: process_is_alive

Because the OS reuses pid numbers, "is pid N still alive?" is the wrong question for a saved pid: a stranger may have recycled the number after your process exited. Pair the pid with the start-time token and ask instead "is the same process still running?":

fn main() -> processkit::Result<()> {
// Earlier: record identity.
let pid = 4321;
let saved_start = processkit::process_info(pid)?.and_then(|i| i.start_time());

// Later (perhaps after a restart): is that same process still alive?
if processkit::process_is_alive(pid, saved_start)? {
    println!("the original process {pid} is still alive");
} else {
    println!("process {pid} is gone (exited, or its number was recycled)");
}
Ok(())
}

process_is_alive is Ok(true) only when the process exists and its current start time matches the saved one; a different start time on the same number (the number was recycled) reads as Ok(false), and so does a nonexistent pid. A permission Err propagates just like process_info — again, never "dead". The start time is an opaque identity anchor (unit/epoch differ per platform), used only for this pairing, never displayed. Where the platform reports no start time (the bare BSDs, where start_time() is None), the check degrades to bare-pid liveness — exactly the number-only check you'd otherwise write by hand, no weaker, and never a false "dead".

Resource limits

Requires the limits feature. Caps are a property of the group, set at creation (and adjustable later — see Updating a live group) and enforced by the same kernel object that contains the tree:

use processkit::{Command, ProcessGroup, ProcessGroupOptions};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::with_options(
        ProcessGroupOptions::default()
            .max_memory(512 * 1024 * 1024) // bytes, whole tree
            .max_processes(64)             // fork-bomb ceiling
            .cpu_quota(0.5),               // half of one core
    )?;
    let _sandboxed = group.start(&Command::new("untrusted-tool")).await?;
    Ok(())
}

cpu_quota is a fraction of a single core (2.0 = two cores). Limits need a real container; when a requested cap can't be enforced — no Job Object/cgroup, or a Linux cgroup whose controllers can't be enabled — with_options returns ErrorReason::ResourceLimit { kind, reason, detail } instead of handing back a silently-unbounded group: kind names the limit (max_memory/max_processes/cpu_quota), reason says whether the value was simply invalid, no mechanism with whole-tree resource accounting exists here (Unsupported — the pgroup mechanisms, which have no whole-tree container at all, and the FreeBSD reaper, which contains a tree without accounting for it), or a mechanism exists but rejected this request (Unenforceable) — branch on these instead of parsing detail. On Linux this needs the process to run at the real cgroup-v2 root: the crate enables the controllers in this process's own cgroup, which cgroup v2's "no internal processes" rule allows only for the real hierarchy root — not a cgroup-namespace root (so an ordinary container fails too), not under systemd — and the crate doesn't migrate your process. See the limits prerequisites in Platform support. The uid()-drop interaction lives under its Caveats.

Updating a live group

ProcessGroup::update_limits(ResourceLimits) re-applies a fresh set of caps to an already-running group — without recreating the container or restarting its children — for adaptive resource management (tighten a slumping batch's memory, widen a long-lived worker pool's CPU quota):

use processkit::{ProcessGroup, ResourceLimits};

fn main() -> processkit::Result<()> {
let mut group = ProcessGroup::new()?;

// Later, adapt the caps on the already-running group:
let mut limits = ResourceLimits::default();
limits.max_memory = Some(256 * 1024 * 1024); // tighten to 256 MiB
limits.cpu_quota = Some(2.0);                 // widen CPU to two cores
group.update_limits(limits)?; // max_processes left None → that cap is lifted
Ok(())
}

The new value is a full replacement, not a merge: an axis left None is lifted back to unbounded — it does not keep its previous cap — so always describe the complete desired state (start from ResourceLimits::default() and set the axes you want capped). On Windows the live Job Object's caps are reissued; on Linux cgroup v2 the memory.max / pids.max / cpu.max files are rewritten (a removed axis written back to max). It routes through the same live container the tree-control verbs use, so the same platform matrix and ErrorReason::ResourceLimit { kind, reason, detail } classification apply — a process-group mechanism (macOS/the other BSDs, the Linux fallback) and the FreeBSD reaper refuse any requested cap with Unsupported rather than silently dropping it, while lifting all caps there is a trivial success.

A failure is not a rollback. The caps are written axis by axis — on Windows one Job Object call for the memory and process caps and a second for the CPU cap, on cgroup v2 memory.max, then pids.max, then cpu.max — and nothing about that is transactional. A call that fails part-way can leave the container carrying a mix of old and new caps, including an axis the request meant to lift that has already been lifted; only the group's reflected options (what Debug shows) stay on the previous set, and the error doesn't say how far the write got. Re-issue the complete desired set (a full replacement, so retrying is idempotent) or tear the group down. An Invalid value is the one case guaranteed to change nothing: it is rejected before the OS is touched. Evidence stays honest across all of this — every axis a request names joins the group's sticky cap record whether the call succeeds or fails, so an axis that did land before the failure is still read from the kernel's counters by limit_evidence() rather than reported NotTripped with nothing behind it.

Did the cap actually fire? (limit_evidence)

The caps above answer "may this tree use more?". They don't, by themselves, tell you afterwards whether one of them stopped something — and a plain exit status can't either: a child OOM-killed under max_memory and a child that crashed on its own both surface as an ordinary non-zero exit (a SIGKILL on Unix). stats reports peak and cumulative samples, which is a measurement, not a verdict.

ProcessGroup::limit_evidence() closes that gap. It returns a LimitEvidence report carrying one LimitVerdict per axis, read from the kernel/OS container the crate owns:

use processkit::{Command, LimitVerdict, ProcessGroup, ProcessGroupOptions};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::with_options(
        ProcessGroupOptions::default().max_memory(512 * 1024 * 1024),
    )?;
    let outcome = group.start(&Command::new("untrusted-tool")).await?
        .output_string().await?;

    if !outcome.is_success() {
        match group.limit_evidence().memory() {
            LimitVerdict::Tripped => eprintln!("killed by its memory cap"),
            LimitVerdict::NotTripped => eprintln!("the tool failed on its own"),
            // `LimitVerdict` is `#[non_exhaustive]`; treat anything new the way
            // you treat `Unknown` — as "no answer", never as a "no".
            _ => eprintln!("this platform can't say"),
        }
    }
    Ok(())
}

Two different questions. ErrorReason::ResourceLimit is admission: "the cap you asked for could not be applied" (Invalid / Unsupported / Unenforceable). with_options returns it instead of running anything at all — it hands back no group, so there is nothing left to ask. update_limits returns it against an already-running tree, where it undoes nothing that already landed (see A failure is not a rollback above). limit_evidence is the other side: did a cap on this axis then fire? Nothing about the error's behaviour changes — but on a live group the two can meet on the same axis: after a failed update_limits the error says the requested set could not be applied whole, while the evidence still answers what actually fired, read from the counters rather than assumed away.

Three-valued on purpose, and never a guess. Tripped is returned only on authoritative kernel/OS evidence recorded by this group's own container. NotTripped means the evidence says it did not fire — or that the axis never carried a cap, so nothing could. Unknown means no evidence is available, and is deliberately not folded into a "no". Exit codes and signals are never consulted: they cannot separate a cap-driven kill from a self-inflicted one.

What "fired" means differs by axis, because the OS's own behaviour does: memory means the container hit its own cap and the kernel had to OOM inside it; processes means a fork was refused; CPU means the quota throttled the tree at least once — a CPU cap slows work rather than stopping it, so a Tripped CPU verdict reads as "the quota bound this workload", not "the quota broke it".

The memory axis keys on cgroup v2's oom counter and deliberately not on oom_kill, which the kernel documents as processes of this cgroup killed by any OOM killer — a host-wide out-of-memory kill would otherwise be reported as "your cap killed it".

A NotTripped memory verdict on a host with swap deserves one caveat, and it is the kernel's, not the report's: memory.max caps memory, not memory + swap. Where swap is available to the tree (memory.swap.max defaults to max, and the crate sets no swap cap), the kernel may page a hog out instead of OOM-killing it — the cap engages, nothing dies, and NotTripped is the truthful answer. If you need "over the cap means death", take swap off the tree externally (memory.swap.max, or a swapless host/container — which is what most containers already are).

Windows is a measured negative, not an oversight. A Job Object enforces all three caps but preserves nothing about them afterwards: the active-process cap refuses the offending process without ever counting it as a member (the job accounting's "terminated because of a limit violation" tally is measurably unmoved by a real violation), the memory cap fails a commit rather than killing and is surfaced only as a live IO-completion-port notification, and the CPU hard cap throttles with no counter at all. Reading any of them would mean attaching a completion port and a drain thread to every group — new machinery on the containment object itself, purely for reporting — which this crate does not do. Inferring from PeakJobMemoryUsed or an exit code is refused as a guess, so a capped axis reports Unknown there. (On the process-group mechanism the answer is Unknown too, for a different reason: it has no accounting to read — and it refuses to carry a cap in the first place.)

Read it before the group goes away. The evidence lives in the container, so call limit_evidence() while the group is still alive; dropping it (or the consuming shutdown()) removes the cgroup / closes the job handle and takes the counters with it. Reading is free of side effects and repeatable: it sends no signal, kills nothing, writes nothing, and cannot perturb teardown or kill-on-drop whenever you call it. The counters are cumulative and are not reset by reading, by a teardown, or by update_limits — an axis whose cap was later lifted still reports that it fired while the cap was in force, and so does an axis named by an update_limits call that failed (see above: that call is not a rollback, so the axis may have been applied before the failure). An axis that never carried a cap is answered without touching the OS at all, so a group created without caps performs no evidence I/O whatsoever.

(LimitEvidence and LimitVerdict are left as bare code spans, not docs.rs links: they ship in the next release, so a docs.rs URL would 404 until then.)

Stats and sampling

Requires the opt-in stats feature (features = ["stats"], or limits).

use processkit::prelude::StreamExt;
use processkit::{Command, ProcessGroup};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let _worker = group.start(&Command::new("worker")).await?;

    // Point-in-time:
    let snap = group.stats()?;
    println!(
        "procs={} cpu={:?} peak_rss={:?}",
        snap.active_process_count, snap.total_cpu_time, snap.peak_memory_bytes,
    );
    println!(
        "io_read={:?} io_write={:?} peak_procs={:?}",
        snap.io_read_bytes, snap.io_write_bytes, snap.peak_process_count,
    );

    // …or a series: first sample immediate, then every 250ms; missed ticks are
    // skipped; the stream ends when the group can no longer report.
    let mut samples = group.sample_stats(Duration::from_millis(250));
    while let Some(s) = samples.next().await {
        println!("rss now: {:?}", s.peak_memory_bytes);
    }
    Ok(())
}

CPU time and peak memory are available where the kernel accounts for the whole tree (Windows, Linux cgroup); the process-group backends report the member count only — the Option fields stay None. The sampler borrows the group, so it can neither outlive it nor keep it (and the kill-on-drop guarantee) alive. For a single run's end-to-end summary, see profile.

What each measurement is, and where it comes from

None never means zero anywhere in this snapshot: it means this mechanism does not account for that. Which fields carry a number depends on the mechanism, and on two different kinds of source — a counter the container itself keeps (whole-tree and cumulative, exited members included), or a sum over the members that are live right now (a member that exits takes its share out of the next snapshot):

Three caveats are worth reading before comparing numbers across hosts:

• The I/O counters measure different traffic per platform. Windows counts the bytes the job's read/write operations moved against any target — file, pipe, device. A cgroup's io.stat counts what reached the block layer, so a read served from the page cache or traffic over a pipe or socket is not in it, and a write is counted when the kernel writes the page back — possibly after the member that dirtied it exited, and not yet at all in a snapshot taken before that. A short write-and-exit run can therefore report fewer bytes than it handed to write(2).
• A cgroup reports these only where the controller is enabled for the group's cgroup (in its parent's cgroup.subtree_control). processkit enables exactly the controllers a requested resource cap needs — memory, pids, cpu — and never io, so on a host that has not enabled io itself the byte counters are honestly None rather than a zero.
• peak_process_count is a peak of what the pids controller counts, which is tasks: every thread of a multi-threaded member counts towards it, so it equals a process count only while the members are single-threaded. Windows reports None here rather than a substitute — a job's ActiveProcesses is how many are in it now and TotalProcesses how many ever were, and neither is a peak.

An owning, 'static sampler

Because sample_stats borrows the group, its StatsSampler is tied to that borrow — it can't be moved into a [tokio::spawn]ed task or handed across an FFI boundary, both of which need a 'static value. When the group already lives behind a shared [Arc] (a long-lived service, a supervisor, an FFI wrapper), reach instead for OwnedStatsSampler, the owning twin — the sampling analogue of how shutdown_ref is the non-consuming twin of shutdown:

use std::sync::Arc;
use std::time::Duration;

use processkit::prelude::StreamExt;
use processkit::{Command, OwnedStatsSampler, ProcessGroup};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = Arc::new(ProcessGroup::new()?);
    let _worker = group.start(&Command::new("worker")).await?;

    // `Send + 'static`: build it from `&Arc<…>` (the caller keeps the `Arc`) and
    // move it into a task. Same cadence as `sample_stats` — first sample
    // immediate, then one per interval, missed ticks skipped.
    let mut samples = OwnedStatsSampler::new(&group, Duration::from_millis(250));
    tokio::spawn(async move {
        while let Some(s) = samples.next().await {
            println!("rss now: {:?}", s.peak_memory_bytes);
        }
        // The series ended: either the container can no longer report, or every
        // `Arc` to the group was dropped and the tree is gone.
    });
    Ok(())
}

It holds the group only weakly, so — exactly like the borrowing StatsSampler — it never keeps the group or its kill-on-drop guarantee alive; a sampler left running in a detached task can't pin a tree that should have been torn down. That makes its behaviour when the group goes away well-defined: the stream ends — yields None, and stays ended (it is fused) — on the first tick that can't produce a snapshot, whether because the container was torn down (a failed stats(), same as the borrowing sampler) or because every strong Arc to the group was released while the sampler ran (the weak handle no longer upgrades). It never silently repeats the last snapshot and never leaves the task awaiting a tick that will never come.

Next: Streaming & interactive I/O · Platform support · Supervision

Streaming & interactive I/O

‹ docs index

The one-shot verbs in Running commands buffer the whole output. For long-running or conversational children, Command::start() returns a live RunningProcess you drive yourself: stream stdout as it arrives, write stdin incrementally, probe for readiness, race several children, or profile a run.

The same live-handle model extends to multi-stage chains: Pipeline::start() returns a PipelineSession — the multi-stage analogue of RunningProcess — that streams the last stage's stdout, waits for a readiness line, and folds the pipefail outcome at finish(), all while the whole chain is bounded and torn down as a unit. See Pipelines → streaming a live chain.

• Lifecycle
• Streaming stdout
• The full lifecycle as one stream (events())
• Interactive stdin
• Readiness probes
• Prompt-aware waiting (wait_for_output)
• Racing children with wait_any
• Per-run telemetry

Lifecycle

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut run = Command::new("dev-server").start().await?;

    run.pid();        // Option<u32> — None once the child is reaped
    run.elapsed();    // time since spawn

    // Consume the handle exactly one way:
    //   output_string() / output_bytes()  → capture everything (same as the one-shot verbs)
    //   wait()                            → just the Outcome; output is discarded
    //   drain()                           → like wait(), but honors output_buffer's byte cap; feeds tees, retains nothing
    //   finish()                 → after streaming stdout (below)
    //   profile(every)                    → resource samples; output discarded, like wait() (stats feature)
    let outcome = run.wait().await?;   // Outcome: Exited(code) / Signalled(sig) / TimedOut
    Ok(())
}

wait() vs drain()

Both wait for exit while draining stdout/stderr so the child never blocks on a full pipe, and both return the same Outcome. They differ only in the in-flight memory bound:

• wait() ignores output_buffer and pins a large fixed internal cap. Reach for it when you just want the exit outcome.
• drain() honors the configured output_buffer byte cap (with_max_bytes) for the in-flight bound, retaining nothing. Reach for it when the output is already going where you want it — a stdout_tee/stderr_tee writing to a file, or an on_stdout_line/on_stderr_line handler — and you want held memory bounded by your configured limit rather than the child's output size. Those sinks still see every line that fits the cap; a line longer than the byte cap is skipped for every sink alike (counted only via the truncation signal), and an unbounded output_buffer falls back to the same fixed floor wait uses.

use processkit::{Command, OutputBufferPolicy};
use tokio::fs::File;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Tee a noisy build to a file, keep only ~1 MiB in flight, capture nothing.
    let log = File::create("build.log").await?;
    let outcome = Command::new("cargo")
        .args(["build", "--release"])
        .output_buffer(OutputBufferPolicy::unbounded().with_max_bytes(1 << 20))
        .stdout_tee(log)
        .start()
        .await?
        .drain()
        .await?;
    println!("build finished: {outcome:?}");
    Ok(())
}

start() puts the child in a private group the handle owns: dropping the RunningProcess kills the whole tree, exactly like dropping a one-shot run's future. The shared-group variant — group.start(&cmd) — gives the same handle but the group controls the tree's fate (see Process groups).

There is also an explicit run.start_kill() for "stop it now, I'll wait() for the code myself".

Streaming stdout

stdout_lines() yields decoded lines as the child produces them — no waiting for exit, no full-output buffering. StreamExt (in processkit::prelude, re-exported from tokio-stream) provides .next():

use processkit::prelude::StreamExt;
use processkit::{Command, Finished, Outcome};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut run = Command::new("cargo")
        .args(["build", "--release"])
        .start()
        .await?;

    let mut lines = run.stdout_lines()?;
    while let Some(line) = lines.next().await {
        println!("build: {line}");
    }

    // The stream ended (stdout closed). Collect the outcome and stderr —
    // stderr was drained in the background the whole time, so a noisy child
    // could never block on a full pipe.
    let Finished { outcome, stderr, .. } = run.finish().await?;
    if outcome != Outcome::Exited(0) {
        eprintln!("build failed ({outcome:?}):\n{stderr}");
    }
    Ok(())
}

Things to know:

• Call stdout_lines() once. It is fallible: a second stdout_lines / events call (stdout is consumed once), or a non-piped stdout (StdioMode::Inherit/Null or Command::stdout_file*), returns Err rather than a silently-empty stream.
• The command's timeout bounds the stream: at the deadline the tree (own-group handle) or the direct child (shared-group handle) is killed, the pipes close, and the stream ends — a streamed run can't hang past its deadline. A cancel_on token ends it the same way; the following finish then reports ErrorReason::Cancelled. Details in Timeouts & cancellation.
• Line counters tick live: run.stdout_line_count() / stderr_line_count() are cheap progress gauges even while you stream.
• Byte counters tick live: run.stdout_bytes_seen() / stderr_bytes_seen() count raw bytes read from each pipe before decoding or line splitting. They are monotonic and include bytes discarded by the buffer policy, including oversized lines, and remain stable after the pump completes. A stream that is not pumped (a file redirect or StdioMode::Null/Inherit) reports 0 honestly.
• The buffer policy and line handlers apply to streamed runs too — a handler sees each line on the pump, in addition to your loop.
• The whole streaming surface is hermetically testable: a ScriptedRunner's start() returns a handle whose canned lines flow through the same pump machinery — stdout_lines, the readiness probes, and finish behave identically with no subprocess. See Testing → scripted streaming.

Carriage-return progress output

Tools like curl, pip, and apt redraw a progress bar in place with a carriage return (\rProgress: 50%\rProgress: 100%) and emit no \n until the very end. By default the pump splits on \n only, so that whole sequence is a single, ever-growing line: nothing streams live, and under a byte cap the one over-cap line is dropped whole.

Set line_terminator(LineTerminator::CarriageReturn) to treat a bare \r as a line terminator too. Each carriage-return frame then arrives as its own line — live, one at a time:

use processkit::prelude::StreamExt;
use processkit::{Command, LineTerminator};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut run = Command::new("pip")
        .args(["install", "big-package"])
        // Frame the stream you actually consume: `stdout_lines()` surfaces
        // stdout only (stderr is drained in the background and discarded), so
        // put the CR framing on stdout. If a tool draws its bar on stderr,
        // set `line_terminator(..)` for both streams and read `events()`.
        .stdout_line_terminator(LineTerminator::CarriageReturn)
        .start()
        .await?;

    let mut lines = run.stdout_lines()?;
    while let Some(frame) = lines.next().await {
        // Overwrite your own display with the latest frame.
        print!("\r{frame}");
    }
    run.finish().await?;
    Ok(())
}

The chosen framing is one shared definition of a line for every sink: the streaming verbs, the on_stdout_line/on_stderr_line handlers, a stdout_tee/stderr_tee (which writes each frame followed by \n), and output_string all see the same per-frame lines. A \r\n pair stays a single terminator (no empty line between them), so ordinary CRLF text reads identically to the default; only a \r not followed by a \n splits a frame. The OutputBufferPolicy byte cap now bounds an individual runaway frame — a frame whose content exceeds the cap is skipped as it streams (never assembled whole) — rather than dropping the whole progress stream. Use stdout_line_terminator / stderr_line_terminator for one stream, or line_terminator for both.

Byte-accurate raw output (stdout_raw_tee)

Every path above hands you decoded text: bytes go through encoding_rs, lines are split on the terminator, and CRLF is normalized. That is exactly right for text logic — but a transparent wrapper needs the child's bytes unaltered. Decoding mangles four things a passthrough must preserve: non-UTF-8 stdout (binary from git archive, tar -cz -, ffmpeg … -) becomes U+FFFD; CRLF is rewritten and a missing final newline is fabricated; an unterminated prompt (Password: ) sits in the decode buffer until EOF and reads as a hang; and a line past the byte cap vanishes from the transcript entirely.

stdout_raw_tee(writer) / stderr_raw_tee(writer) are the byte plane, orthogonal to stdout_tee. Each chunk is written to writer exactly as read from the pipe — before decoding, before line splitting — so it is byte-for-byte the child's output, in order: non-UTF-8 bytes survive, CRLF is untouched, the tail is never padded, an unterminated chunk arrives the instant it is read, and even a policy-dropped line is teed whole. It is strictly additive — the decoded line path (capture, on_*_line, stdout_tee, the buffer policy) is unchanged, and both tees can run at once, each seeing its own view. The write is awaited on the capture pump, so a slow raw sink applies the same backpressure as the line tee (no unbounded in-flight buffer), and it is flushed at stream end.

use processkit::Command;
use tokio::fs::File;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Transparent passthrough: forward the child's *exact* stdout bytes to a file
    // (binary-safe — no decoding, no CRLF rewrite, no lost tail), retaining
    // nothing in memory. `drain` streams the bytes through to the tee and returns
    // just the classified exit outcome.
    let exact = File::create("archive.tar").await?;
    let outcome = Command::new("git")
        .args(["archive", "HEAD"]) // writes a binary tar to stdout
        .stdout_raw_tee(exact)
        .start()
        .await?
        .drain()
        .await?;
    println!("archive written: {outcome:?}");
    Ok(())
}

The raw tee fires from the line/streaming verbs (output_string, start + stdout_lines / events, wait / drain). It is a no-op under stdout(Inherit) / stdout(Null) / a stdout_file redirect — none run a capture pump — and under output_bytes, whose own return value already is the exact raw stdout. Reach for it when you need the raw bytes alongside decoded lines.

Redaction at capture (capture_policy)

Every plane above either observes a line (on_stdout_line, stdout_tee) or returns it verbatim (stdout_raw_tee, output_bytes). None can change what is retained: a secret a child echoes — a passphrase prompt from an agent CLI, a --token re-printed in a diagnostic — lands in the captured ProcessResult word for word. capture_policy is the seam that shapes the backlog before the line settles:

use std::borrow::Cow;
use processkit::{Command, CapturePolicy, OutputStream};

// A typed, named policy — introspectable in `Command`'s `Debug`, not an opaque
// closure. It rewrites secrets as each line is captured.
struct RedactTokens;
impl CapturePolicy for RedactTokens {
    fn name(&self) -> &str { "redact-tokens" }
    fn on_capture<'a>(&self, _stream: OutputStream, line: &'a str) -> Cow<'a, str> {
        if let Some(i) = line.find("token=") {
            Cow::Owned(format!("{}token=[REDACTED]", &line[..i]))
        } else {
            Cow::Borrowed(line) // unchanged: retained verbatim, no allocation
        }
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let out = Command::new("agent-cli")
        .arg("login")
        .capture_policy(RedactTokens)
        .output_string()
        .await?;
    // `out.stdout()` — and a streamed `stdout_lines` / `events` — carry
    // the redacted text; the raw secret never reached the buffer.
    assert!(!out.stdout().contains("token=hunter2"));
    Ok(())
}

Return Cow::Borrowed(line) to keep a line unchanged (no allocation), a Cow::Owned to rewrite it, or an empty string to blank the content while keeping the line's slot and the exact line counter. The policy is handed the OutputStream each line came from, so one implementation can treat stdout and stderr differently. This completes the crate's secret-hygiene story — a cassette stores env names only, Debug redacts env values, and capture_policy scrubs secrets a child prints — see Running untrusted children.

Scope. The seam shapes the capture backlog only — output_string / ProcessResult and the streaming verbs. The observing on_*_line handlers, the decoded stdout_tee / stderr_tee, the byte-plane stdout_raw_tee, and the raw stdout of output_bytes are independent and see the line un-redacted; if you also tee to a log, redact in that sink too. A line past an OutputBufferPolicy byte cap is never assembled, so — like the handlers/tee — it never reaches the policy. How much is retained stays OutputBufferPolicy's job (the two compose: the policy shapes each retained line's content, the buffer policy bounds how many survive). A policy that panics fails closed — the offending line is blanked, never leaked raw.

The full lifecycle as one stream (events())

stdout_lines() gives you stdout; events() gives you the child's whole lifecycle as one ordered, typed stream — the single asynchronous source a TUI, dashboard, or supervisor wants, instead of stitching together separate channels for "started", output, and "exited":

Started { pid } → interleaved Stdout / Stderr lines → Exited(outcome)

• ProcessEvent::Started { pid } leads the stream, emitted as soon as the pid is known (before any output). pid is None for a scripted double.
• ProcessEvent::Stdout(line) / Stderr(line) carry each decoded OutputLine (read line.text()), the two streams polled fairly so neither starves the other.
• ProcessEvent::Exited(outcome) ends the stream, carrying the run's Outcome — the same value finish() reports.

The enum is #[non_exhaustive]: a match needs a _ arm, and future kinds (e.g. the graceful-teardown transitions) can be added without a breaking change. ProcessEvent::name() gives a stable "started" / "stdout" / "stderr" / "exited" tag for logging or serialization.

Drive the stream and the finisher together. The terminal Exited is delivered when the run is reaped — which is what finish() (or wait()) does. So poll the stream and that finisher concurrently (e.g. with tokio::join!), rather than draining the stream to completion first and only then calling finish() — the stream would park waiting for an Exited no one is yet producing. After the stream ends, finish()'s stderr is empty because stderr was delivered to you as Stderr events.

use processkit::prelude::StreamExt;
use processkit::{Command, ProcessEvent};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut run = Command::new("deploy").start().await?;

    let mut events = run.events()?;
    let render = async {
        while let Some(ev) = events.next().await {
            match ev {
                ProcessEvent::Started { pid } => eprintln!("started {pid:?}"),
                ProcessEvent::Stdout(line) => println!("{}", line.text()),
                ProcessEvent::Stderr(line) => eprintln!("! {}", line.text()),
                ProcessEvent::Exited(outcome) => eprintln!("exited {outcome:?}"),
                _ => {} // non_exhaustive: future event kinds
            }
        }
    };
    // Poll the event stream and the reaping `finish()` together.
    let (_, finished) = tokio::join!(render, run.finish());
println!("run finished: {:?}", finished?.outcome);
Ok(())
}

Run the same pattern without external dependencies in examples/lifecycle_events.rs.

Interactive stdin

Conversational tools — write a request, read the response, repeat. Keep stdin open with keep_stdin_open(), take the writer with take_stdin():

use processkit::prelude::StreamExt;
use processkit::{Command, Finished, Outcome};

// `ProcessStdin`'s writer methods return `std::io::Result`; `Box<dyn Error>`
// mixes them with the crate's `Result` (or `.map_err(processkit::ErrorReason::Io)?`).
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // `bc` evaluates each stdin line and prints the result.
    let mut run = Command::new("bc").keep_stdin_open().start().await?;
    let mut stdin = run.take_stdin().expect("stdin was kept open");
    let mut answers = run.stdout_lines()?;

    stdin.write_line("2 + 2").await?;             // sends a line + Enter, flushed
    println!("= {}", answers.next().await.unwrap());

    stdin.write_line("6 * 7").await?;
    println!("= {}", answers.next().await.unwrap());

    stdin.finish().await?;                        // send EOF — bc exits
    let Finished { outcome, .. } = run.finish().await?;
    assert_eq!(outcome, Outcome::Exited(0));
    Ok(())
}

ProcessStdin offers write(&[u8]), write_line(&str) (terminal Enter + flush), flush(), and finish() (EOF). Dropping the writer requests the same EOF: while the PTY session and its async runtime remain live, the backend retains that request through temporary input backpressure and delivers it after the child resumes reading. finish() makes delivery explicit and awaitable, including its I/O error; dropping has no error channel. Dropping the whole RunningProcess tears the child tree down, so neither EOF form promises delivery after the session/runtime is already irreversibly ending. write_line sends \n to a pipe or Unix PTY and \r to a Windows ConPTY, where a lone LF is Ctrl-J rather than the Enter key; use write when you need byte-exact input.

Avoid the full-duplex deadlock. A child's stdout pipe has a finite OS buffer; once it fills, the child blocks writing stdout until something reads it. If you push a large interactive stdin while nothing drains the child's stdout, the child stops reading stdin (blocked on stdout), your write parks waiting for stdin buffer space, and neither side progresses. The bc example above is safe because it interleaves one write with one read; when you both feed a sizable stdin and the child produces output, drain stdout_lines from one task while writing stdin from another. The same rule applies to a PTY's single, full-duplex master: awaiting one large write while no sink reads merged output can deadlock. (The non-interactive Stdin::from_* sources are safe — the crate writes them on a background task that runs concurrently with the output pumps.)

For one-directional streamed input (a channel, a file tail) you don't need interactivity — give the command Stdin::from_lines(stream) / Stdin::from_reader(reader) and let the background writer feed it; see the stdin source table.

PTY dialog: wait for a prompt, then answer

A terminal prompt is usually an un-terminated tail (Password: ), not a line. Combine use_pty, keep_stdin_open, and wait_for_output to drive an expect-style exchange without guessing when the child is ready:

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut run = Command::new("terminal-login")
        .use_pty()
        .pty_size(100, 30)
        .keep_stdin_open()
        .start()
        .await?;

    run.wait_for_output(
        |tail| tail.ends_with("Password: "),
        Duration::from_secs(10),
    )
    .await?;
    run.take_stdin()
        .expect("PTY stdin was kept open")
        .write_line("secret supplied by a protected source")
        .await?;

    run.finish().await?;
    Ok(())
}

The self-contained examples/pty_dialog.rs also demonstrates live resize; run it with --features pty.

On Unix, the PTY slave has echo disabled so the answer is not copied into the merged capture. That guarantee is Unix-only; ConPTY offers no portable equivalent, so a Windows caller sending secrets should avoid retaining or logging the transcript. The prompt wait itself is byte-driven on both platforms. Signal-driven terminal behavior is separate: on Unix, use_pty creates a controlling terminal so terminal signals such as the SIGWINCH raised by resize_pty reach the child as described below.

PTY output destination

A PTY has one merged output master, exposed as piped logical stdout. Keep both output destinations at their default StdioMode::Piped: capture, stdout_lines, stdout handlers, and stdout tees then receive the merged bytes, while ProcessResult::stderr stays empty. There is no independent stderr destination after the terminal has merged the child descriptors.

Accordingly, use_pty rejects stdout Inherit, Null, stdout_file/stdout_file_append, and any separate stderr Inherit, Null, stderr_file/stderr_file_append destination with ErrorReason::Unsupported. The check runs before opening, creating, or truncating a redirect file and before spawning the child on both Unix and Windows. This makes the unsupported combinations fail visibly instead of creating an unused file or secretly draining the real PTY master to a discard sink. Use ordinary pipe mode when those per-descriptor destinations are needed.

PTY window size and live resize

Under use_pty the child runs on a real pseudo-terminal, and terminal-aware tools care about its size: it drives line wrapping, TUI/progress layout, and pager behavior. Set the initial geometry with pty_size(cols, rows) (default 80×24), and change it on a running session with RunningProcess::resize_pty(cols, rows) — the way you propagate a host window resize down to the child. At spawn, COLUMNS/LINES match that initial geometry (and Unix defaults TERM=xterm-256color); explicit env/env_remove operations override those values:

use processkit::prelude::StreamExt;
use processkit::Command;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open the terminal at 120×40 instead of the 80×24 default.
    let mut run = Command::new("htop").use_pty().pty_size(120, 40).start().await?;
    let mut screen = run.stdout_lines()?;

    // …later, the host window grows — tell the child so it re-renders.
    run.resize_pty(160, 50)?;

    while let Some(line) = screen.next().await {
        // render `line`…
        let _ = line;
        break;
    }
    Ok(())
}

Dimensions must be non-zero. Windows ConPTY represents each axis in a signed 16-bit COORD, so 1..=32767 is accepted there and larger values fail with ErrorReason::Io (InvalidInput) before a child is spawned or a live terminal is resized. Unix winsize uses u16 fields and accepts the full non-zero range through 65535. Values are never clamped: a successful spawn therefore keeps the applied terminal geometry and synthesized COLUMNS/LINES aligned, while explicit environment operations retain their documented precedence.

resize_pty works while you still hold the handle — typically interleaved with driving an owned stdout_lines()/events() stream and the take_stdin() writer of a live session. It returns ErrorReason::Unsupported — never a panic or a silent no-op — on a run that is not use_pty (there is no terminal to size) or once the child has exited. Platform delivery differs: on Unix the resize (TIOCSWINSZ) raises SIGWINCH on the child immediately; on Windows ResizePseudoConsole has no signal, so a console client observes the new size on its next console query and conhost may reflow a little later. On a non-use_pty command pty_size is a documented no-op (nothing to size). See platform support. An invalid resize is checked before the backend call and leaves the live PTY usable for a later valid resize. Because a running process's environment is immutable, live resize does not rewrite COLUMNS/LINES; applications learn the new size through the platform resize mechanism.

PTY output hygiene: line framing and VT sanitization

A PTY child writes like a terminal, which two things about use_pty handle so a line-oriented consumer gets sensible output — one automatic, one opt-in:

• Framing is \r-aware by default under use_pty. Terminal tools draw progress by redrawing a line in place with a bare \r (no \n until the end). Under the default Newline framing that whole progress stream is one ever-growing line that only surfaces at EOF; so use_pty makes the effective default line_terminator CarriageReturn, and each redraw becomes its own frame/line live. This only changes where lines split (a \r\n still counts as one terminator), so it is the safe default for the mode. An explicit line_terminator(...) — even Newline — always wins, so you can pin the framing if you need to.
• Escape sanitization is opt-in. Agentic CLIs spray VT/ANSI escapes (colors, cursor moves, alternate-screen switches, OSC window-title/hyperlink codes) into their merged output, so output_string, wait_for_line/first_line, and the streaming verbs otherwise carry \x1b[31m…-mucked strings. Turn on Command::sanitize_vt() to strip those sequences (and lone control codes, keeping tabs) from the capture backlog. It is opt-in because it is destructive — it removes bytes from the captured output — unlike the non-destructive framing default.

use processkit::prelude::StreamExt;
use processkit::{Command, LineTerminator};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // A PTY agent CLI: make the progress-frame contract explicit and opt into
    // de-escaping, so every captured line is readable text.
    let mut run = Command::new("agent")
        .use_pty()
        .pty_size(120, 30)
        .line_terminator(LineTerminator::CarriageReturn)
        .sanitize_vt()
        .start()
        .await?;
    let mut lines = run.stdout_lines()?;
    while let Some(line) = lines.next().await {
        // `line` is a clean, de-escaped frame — safe for a `contains(...)` probe.
        let _ = line;
        break;
    }
    Ok(())
}

Because a PTY merges both child descriptors into its logical stdout, the whole-command setters above are the clearest spelling. For a wrapper that can also run over ordinary pipes, the per-stream variants let each real pipe keep its own policy:

use processkit::{Command, LineTerminator};

fn command_for_pipes() -> Command {
    Command::new("agent")
        .stdout_line_terminator(LineTerminator::CarriageReturn)
        .stderr_line_terminator(LineTerminator::CarriageReturn)
        .stdout_sanitize_vt()
        .stderr_sanitize_vt()
}

fn command_for_pty() -> Command {
    Command::new("agent")
        .use_pty()
        .pty_size(120, 30)
        .line_terminator(LineTerminator::CarriageReturn)
        .sanitize_vt()
}

fn main() {
    let _ = (command_for_pipes(), command_for_pty());
}

In PTY mode, stderr_line_terminator and stderr_sanitize_vt have no separate stream to act on: the child stderr bytes have already joined stdout on the master. Likewise, on_stderr_line is not delivered and the final stderr capture is empty. Use the stdout/whole-command settings for PTY output, or use pipes when preserving stream identity is required.

Sanitization shapes only the capture backlog — exactly the boundary capture_policy draws. The per-line handlers (on_stdout_line), the decoded stdout_tee, the byte-plane stdout_raw_tee, and output_bytes are independent and keep seeing the raw, escape-laden bytes; if you also tee to a log and want it clean, sanitize in that sink. When combined with capture_policy, sanitization runs first, so a secret-scrubbing policy matches on already-cleaned text rather than a token a color escape could split mid-word. Set it per stream with stdout_sanitize_vt() / stderr_sanitize_vt() when only one stream needs it. See platform support for the per-platform PTY table.

Readiness probes

"Start a server, then use it" needs ready, not merely started. Readiness probes replace the arbitrary sleep, each bounded by its own deadline:

async fn health_check() -> bool { true }
use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut run = Command::new("my-server").start().await?;

    // 1. A line on stdout (returns the matching line):
    let banner = run
        .wait_for_line(|l| l.contains("listening on"), Duration::from_secs(10))
        .await?;

    // 2. A pidfile or readiness sentinel appearing:
    run.wait_for_path("run/my-server.ready", Duration::from_secs(10))
        .await?;

    // 3. A TCP port accepting connections:
    run.wait_for_port("127.0.0.1:8080".parse().unwrap(), Duration::from_secs(10))
        .await?;

    // 4. A plain HTTP endpoint returning an expected status:
    run.wait_for_http(
        "127.0.0.1:8080".parse().unwrap(),
        "/healthz",
        |status| (200..300).contains(&status),
        Duration::from_secs(10),
    )
    .await?;

    // 5. A Unix domain socket accepting connections (Unix only):
    run.wait_for_socket("/tmp/my-server.sock", Duration::from_secs(10))
        .await?;

    // 6. A Windows named pipe accepting clients (Windows only):
    run.wait_for_pipe("my-service", Duration::from_secs(10)).await?;

    // 7. Any async predicate (an HTTPS/body or metadata check, …):
    run.wait_for(|| async { health_check().await }, Duration::from_secs(10))
        .await?;

    // ready — use the server…
    let _ = banner;
    Ok(())
}

If a tool writes its banner to stderr, use the matching counterpart instead:

let banner = run
    .wait_for_stderr_line(|line| line.contains("backend ready"), Duration::from_secs(10))
    .await?;

Choose one consuming line probe for a handle: while it hands you the selected stream, the other stream is already being drained in the background for liveness and later capture.

Probe semantics, deliberately uniform:

• A probe that can't pass within its deadline fails with ErrorReason::NotReady — distinct from ErrorReason::Timeout, which is the run's own deadline.
• A probe also fails fast once readiness can no longer happen: the child exits, or (for wait_for_line) its stdout closes — no waiting out a 30s deadline on a dead server.
• A failed probe never kills the child. You decide: retry, log and continue, or tear down.
• All probes background-drain stdout/stderr while they poll, so a child with a large startup burst can't stall in write() on a full OS pipe buffer. wait_for_line and wait_for_stderr_line consume the selected stream up to (and including) the match — continue with finish (whose stderr omits lines already consumed by wait_for_stderr_line). wait_for_path / wait_for_port / wait_for_http / wait_for_socket / wait_for_pipe / wait_for drain the same way but never hand any of it back mid-probe; wait / output_string afterward still see the full captured output, but output_bytes or a fresh stdout_lines / events call do not compose with any readiness probe that started a line pump (same as calling wait_for_line first).
• wait_for_socket uses a real connection attempt, not just a socket-file existence check, and returns ErrorReason::Unsupported immediately on platforms without AF_UNIX (including Windows).
• wait_for_path is deliberately existence-only: a file or directory counts. Use wait_for with tokio::fs::metadata when readiness also depends on type, size, permissions, or other metadata.
• wait_for_pipe accepts a bare name or a fully-qualified \\.\pipe\... path. It returns ErrorReason::Unsupported off Windows. A busy pipe counts as ready: all server instances being occupied still proves the endpoint is serving.
• wait_for_http sends a minimal HTTP/1.1 GET and reads only a bounded status line. It is plain HTTP only: no TLS, redirect following, response bodies, or authentication policy. Redirects count only if your status predicate accepts them; use wait_for with your own HTTP client for HTTPS or richer checks.

Prompt-aware waiting (wait_for_output)

wait_for_line only ever sees complete lines — text with its terminator. An interactive prompt is the opposite: Password: , passphrase: , (y/N) , a REPL >>> are all written without a trailing newline and then blocked on, waiting for you to answer. Such a prompt never becomes a line, so wait_for_line (and stdout_lines) cannot see it until the stream ends — the "wait for the prompt, then answer it" dialog can't be expressed line by line.

wait_for_output(predicate, within) closes that gap. Its stderr counterpart is wait_for_stderr_output; both are expect-style primitive (in the spirit of rexpect): it matches the child's current un-terminated output tail — the partial line the pump has decoded but not yet split — and hands it back so you can answer over take_stdin(). PTY is the motivating case (a merged terminal stream is full of un-terminated prompts), but it is not PTY-specific — a plain piped run benefits too (e.g. a progress meter that rewrites one line without a newline).

Unlike the line probes, these methods do not consume their selected stream and are repeatable: a whole session can be a sequence of wait_for_output → answer turns, one per prompt.

use processkit::{Command, ProcessRunner};
use processkit::testing::{Reply, ScriptedRunner};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // A hermetic stand-in for a tool that prompts, reads a secret, then
    // continues — no real subprocess. `Reply::dialog` emits the prompt, waits
    // for the answer over stdin, then emits the continuation.
    let runner = ScriptedRunner::new()
        .fallback(Reply::dialog("Password: ", "granted, welcome> "));

    let mut run = runner
        .start(&Command::new("login").keep_stdin_open())
        .await?;

    // Wait for the un-terminated `Password: ` prompt (a whole line never arrives).
    let prompt = run
        .wait_for_output(|tail| tail.ends_with("Password: "), Duration::from_secs(5))
        .await?;
    assert!(prompt.contains("Password:"));

    // Answer it, then wait for the next (also un-terminated) prompt.
    run.take_stdin().expect("interactive stdin").write_line("s3cret").await?;
    let cont = run
        .wait_for_output(|tail| tail.contains("welcome>"), Duration::from_secs(5))
        .await?;
    assert!(cont.contains("welcome>"));

    run.finish().await?;
    Ok(())
}

Semantics, and how it differs from wait_for_line:

• Un-terminated tail, not lines. wait_for_output matches the live partial line; once the child terminates it with a newline it becomes a complete line (seen by wait_for_line / stdout_lines, not here). So answer a prompt before waiting for the next, and reach for wait_for_line when you want whole lines.
• Non-consuming and repeatable. It only peeks while the background pump keeps draining stdout under your OutputBufferPolicy; the handle stays usable for take_stdin, further wait_for_output calls, and finish. wait_for_line takes the stdout stream and so is one-shot.
• Raw, not redacted. The predicate (and the returned fragment) see the tail before any capture_policy redaction (see Redaction at capture) — the same observation category as handler / tee / raw_tee / output_bytes, not the redacted backlog wait_for_line and ProcessResult draw from. A partial line can't be run through a per-line redactor, and a prompt is a synchronization token you match verbatim. The retained/finished output stays redacted independently — just match on prompts, not on secret-bearing partial text.
• Same probe deadline. It fails with ErrorReason::NotReady when within elapses (or stdout closes) with no match, never killing the child or arming the run's timeout watchdog — exactly like the readiness probes above.
• Choose the actual pipe. Use wait_for_stderr_line / wait_for_stderr_output when the tool reports readiness on stderr. Both keep stdout draining in the background. A PTY exposes one merged stream, so its stdout methods observe output originally written to either child stream.

Racing children with wait_any

The free function wait_any races several running processes and reports whichever exits first — the natural primitive for "restart whatever died" or "first answer wins":

use processkit::{Command, ProcessGroup, wait_any};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;
    let mut a = group.start(&Command::new("replica-a")).await?;
    let mut b = group.start(&Command::new("replica-b")).await?;

    let (index, outcome) = wait_any(&mut [&mut a, &mut b]).await?;
    println!("contender #{index} exited first with {outcome:?}");

    // Only borrows: the loser is still usable.
    let survivor = if index == 0 { &mut b } else { &mut a };
    Ok(())
}

wait_any takes &mut borrows, applies no timeout of its own (wrap it in tokio::time::timeout to bound the race), and does no output pumping — drain chatty children first or give them bounded buffer policies.

Per-run telemetry

With the opt-in stats feature, a running child reports its own resource usage, and profile() turns a whole run into a summary:

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let run = Command::new("crunch").start().await?;
    run.cpu_time();          // Option<Duration> — user+kernel so far
    run.peak_memory_bytes(); // Option<u64>

    // …or capture + sample on an interval until exit:
    let profile = Command::new("crunch")
        .start().await?
        .profile(Duration::from_millis(100))
        .await?;

    println!(
        "outcome={:?} wall={:?} cpu={:?} peak_rss={:?} avg_cpu_cores={:?} ({} samples)",
        profile.outcome,            // Exited(code) / Signalled(sig) / TimedOut
        profile.duration,
        profile.cpu_time,
        profile.peak_memory_bytes,
        profile.avg_cpu_cores(),    // cpu / wall — e.g. Some(1.7) ≈ 1.7 cores busy
        profile.samples,
    );
    Ok(())
}

These read the child process itself (not a whole tree — that's ProcessGroup::stats), and availability follows the platform: full CPU/memory on Windows and Linux, None where the kernel doesn't account per-process cheaply — see Platform support.

That boundary is why RunProfile carries no whole-tree counters — no I/O bytes, no peak process count — even though ProcessGroupStats does. Those come from the containment object, and a run has no whole-tree scope of its own to report them under: started into a shared ProcessGroup, it shares that container with every other run in it, and a container's counters cannot be split into per-run shares. When you want a run's tree, give it a group of its own and read ProcessGroup::stats — the same numbers, named as the group's.

Lifecycle narration (tracing)

With the opt-in tracing feature, a run narrates its lifecycle on the processkit target — spawn (pid, mechanism), the graceful-teardown transitions (soft_signal → grace_started → drained / escalated / spared, each in a stable phase field), and exited — so a subscriber can stamp each transition the instant the layer that observed it crossed it, and map it to an event of your own protocol. The teardown transitions are narrated the same way for a streaming handle's own shutdown(grace) as for a whole-group ProcessGroup::stop, so you get one uniform timeline. It is observation only (never a control surface) and never carries argv/env; for the same teardown facts as a typed value returned after the fact, see ShutdownReport.

Next: Pipelines · Timeouts, retries & cancellation · Supervision

Pipelines

‹ docs index

a | b | c without a shell. Each stage's stdout feeds the next stage's stdin through an in-process relay (a tokio::io::copy task per boundary) — there is no shell string anywhere, so no quoting rules, no word splitting, no injection surface. Each stage spawns into its own kill-on-drop process group sub-group, so a per-stage Command::timeout tears down that stage's whole subtree (grandchildren of a forking sh -c … included); a chain-wide teardown fans the kill across every stage's sub-group, so the chain still lives and dies as a unit. (The relay is an implementation detail, not a kernel splice: a producer whose consumer exits early stops on a broken pipe when the relay's next write fails, rather than instantly via SIGPIPE.)

• Building and running
• Semantics: pipefail and the ends
• Merging a stage's stderr into the pipe
• Unchecked stages
• Timeouts
• Streaming a live chain
• Re-running a pipeline

Building and running

Command::pipe(next) starts a Pipeline; chain more stages with Pipeline::pipe; drive it with output_string() or run():

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // git log --format=%an | sort | uniq -c
    let authors = Command::new("git").args(["log", "--format=%an"])
        .pipe(Command::new("sort"))
        .pipe(Command::new("uniq").arg("-c"))
        .run()                         // require every stage to succeed
        .await?;
    println!("{authors}");
    Ok(())
}

The verbs mirror Command's, each operating on the pipefail outcome:

Err from output_string itself means a stage couldn't be started or driven at all (spawn failure, broken plumbing) — never a mere non-zero exit.

The first_line probe is deliberately not a buffering pipeline verb: those verbs consume the last stage in full to fold the pipefail outcome. To capture the first matching line of a finished chain, add a | head -n1 (Unix) / grep -m1 / findstr stage and capture. To instead read a chain that keeps running — wait for a banner line, then stream the rest — use Pipeline::start(), which gives a chain the same live streaming surface a single Command::start() gives a process.

The | operator is sugar for the same thing — a | b | c ≡ a.pipe(b).pipe(c). Parenthesize the chain before a terminal verb, since method calls bind tighter than |:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let authors = (Command::new("git").args(["log", "--format=%an"])
        | Command::new("sort")
        | Command::new("uniq").arg("-c"))
        .run()
        .await?;
    Ok(())
}

Semantics: pipefail and the ends

The outcome is pipefail, like set -o pipefail in a shell:

• stdout is always the last stage's output — that's what the chain produced.
• code, stderr, and the reported program come from the culprit stage: the leftmost stage that didn't exit cleanly (non-zero, signal-killed, or timed out), but preferring a real failure over a downstream SIGPIPE victim — a stage killed only because a later stage closed the pipe early. If every failure is such a broken-pipe victim, the leftmost one wins; when every stage succeeded, the last stage speaks.

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("cat").arg("data.txt")
        .pipe(Command::new("grep").arg("ERROR"))      // suppose grep exits 2 (bad pattern)
        .pipe(Command::new("wc").arg("-l"))
        .output_string()
        .await?;

    // Diagnostics point at grep — the first unclean stage — while stdout is
    // whatever wc managed to print:
    assert_eq!(result.code(), Some(2));
    println!("blamed: {}", result.ensure_success().unwrap_err()); // names `grep`
    Ok(())
}

Failure tears the chain down proactively. The moment a stage ends with a checked failure (a non-zero exit outside its ok_codes, a signal kill, or its own per-stage timeout), every stage's sub-group is torn down at once — the failure does not wait to trickle out through closing pipes. This matters for a quiet sibling that would otherwise hang: an upstream producer that never writes never dies of a broken pipe, so under a purely passive teardown a downstream failure could be held open indefinitely by that silent producer. Now the failure surfaces immediately, and the killed siblings are treated as victims (like a downstream SIGPIPE death) — the stage that actually failed keeps the blame. The one death that does not trigger this is an unchecked_in_pipe() stage's: its unclean exit is forgiven, so it leaves the rest of the chain running. (A stuck stage that never fails — a healthy producer that simply never finishes — is still bounded only by Pipeline::timeout or cancellation.)

The ends of the chain behave like a single Command:

• The first stage's configured stdin source is honored — feed the whole pipeline from a string, file, or stream.
• Inner stages read from the pipe, full stop: any stdin source or keep_stdin_open configured on them is overridden.
• Inner stages likewise write to the pipe, full stop — see below.
• The last stage's stdout is the chain's own output and is honored as configured when that destination is supported (stdout(Null), stdout_file(..) and friends behave as they do on a standalone non-PTY command).
• Inner stages' stderr is captured per-stage for pipefail diagnostics; only the last stage's stdout reaches you. A stage explicitly marked with merge_stderr_in_pipe() is the exception described below.

use processkit::{Command, Stdin};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let unique_count = Command::new("sort")
        .stdin(Stdin::from_iter_lines(["b", "a", "b", "c"]))
        .pipe(Command::new("uniq"))
        .pipe(Command::new("wc").arg("-l"))
        .run()
        .await?;
    assert_eq!(unique_count.trim(), "3");
    Ok(())
}

A non-final stage's stdout belongs to the pipe

Because the link between two stages is unconditional, the chain — not the stage — owns a non-final stage's stdout. If such a stage carries stdout(StdioMode::Null), stdout(StdioMode::Inherit), or a file redirect, the pipe wins and that configuration goes inert for the run:

• the stage's bytes go to the next stage and nowhere else — not to /dev/null, not to the parent's terminal, not to the file;
• a configured redirect file is neither created nor truncated, so a log a previous run left there is left intact;
• the stage's stdout observers (on_stdout_line, stdout_tee, stdout_raw_tee) do not fire — as they do not on any non-final stage, whose stdout goes to the next stage rather than through a pump that could run them;
• stderr is untouched: it keeps its own configuration and stays available for pipefail diagnostics.

This is the same precedence inner-stage stdin has had all along, applied to the other end of the same relay, and it is what keeps a Command that carries a redirect for standalone use reusable as a pipeline stage. The alternative — treating the combination as a configuration error — would reject such a stage before spawn; what neither does is quietly drop the producer's output and start the next stage on an empty stdin.

The one non-final stdout configuration that is rejected rather than overridden is use_pty(): a PTY master carries a merged terminal stream that cannot feed a later stage at all, so the chain fails before spawn with ErrorReason::Unsupported. The final stage may use a PTY only with its supported merged Piped stdout and Piped stderr destination; Inherit, Null, and file redirects are rejected. The pipeline checks every PTY stage's destination before it starts any stage, so an invalid final-stage destination cannot run an upstream command or open a redirect file first. A non-final PTY retains the more specific pipeline-topology error because it cannot provide the next stage's stdin pipe at all.

Merging a stage's stderr into the pipe

Mark a non-final stage with merge_stderr_in_pipe() for the shell-free equivalent of command 2>&1 | next:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Feed both compiler diagnostics and ordinary output to the filter.
    let matches = Command::new("cargo").args(["check", "--message-format=short"])
        .merge_stderr_in_pipe()
        .pipe(Command::new("grep").arg("warning"))
        .output_string()
        .await?;
    println!("{}", matches.stdout());
    Ok(())
}

The child receives two cloned handles to the same anonymous-pipe writer for stdout and stderr. Their bytes therefore enter one OS pipe in the order the child writes them; processkit does not merge two independently-read streams in userspace. The normal pipeline boundary still relays that one reader into the next stage's stdin, as described at the top of this page.

The marker is opt-in per stage and only applies when that stage has a downstream neighbor. It is a no-op on a standalone command and on the final pipeline stage. For an affected stage, the downstream pipe overrides its configured stdout and stderr destinations. The shared anonymous-pipe wiring is supported on Unix and Windows; activating it on another target fails before spawn with ErrorReason::Unsupported.

This changes pipefail diagnostics deliberately: the merged stage no longer has a separate stderr capture. If it is blamed for a failure, the result's stderr is empty; its diagnostic bytes may instead be present in the final stdout after flowing through the remaining stages. Leave the marker off when retaining the culprit stage's dedicated stderr is more important than filtering the combined stream.

Unchecked stages

Strict pipefail has one classic false positive: a consumer that legitimately stops reading early. In producer | head -1 the consumer exits 0 after one line and closes the pipe; the producer then stops on a broken pipe — its next write fails once the relay's downstream is gone (a broken-pipe write error, or SIGPIPE where the OS delivers it) — a perfectly normal death that strict pipefail would blame the chain for. Mark that stage unchecked_in_pipe():

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // seq 1 1000000 | head -1 — the producer's broken-pipe death is expected.
    let first = (Command::new("seq").args(["1", "1000000"]).unchecked_in_pipe()
        | Command::new("head").args(["-n", "1"]))
        .run()
        .await?;
    assert_eq!(first.trim(), "1");
    Ok(())
}

The rules (a design borrowed from duct's unchecked() — the idea, not the code):

• An unchecked stage's unclean exit — a non-zero code, a broken-pipe write failure (or SIGPIPE where the OS delivers it) from a consumer that closed early, or its own per-stage timeout kill — is skipped when the chain decides what to report.
• A checked failure always trumps an unchecked one, regardless of position: unchecked never shields another stage's real failure.
• A chain whose only failures are unchecked reports success with the last stage's stdout and its real exit code preserved (not a fabricated 0 — the accepted-code set is widened to include it). The carve-out is for an exit status only: a last stage killed by a signal or its own timeout is not a status to forgive and still surfaces as the failure.
• unchecked forgives exit status only — never a whole-chain Pipeline::timeout, and it has no effect on a Command run outside a pipeline (a single run's status is already plain data in its ProcessResult).

Timeouts

Two scopes, deliberately distinct:

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("producer")
        .timeout(Duration::from_secs(10))      // per-STAGE: kills just `producer`
        .pipe(Command::new("consumer"))
        .timeout(Duration::from_secs(30))      // whole-CHAIN: Pipeline::timeout
        .output_string()
        .await?;
    Ok(())
}

• Pipeline::timeout bounds the whole chain: at the deadline teardown is attempted for every stage sub-group and, when confirmed, the result reports timed_out. The result keeps best-effort stdout and stderr already captured by the final stage before teardown, subject to the same buffer/truncation policy as a normal capture. A successful fan-out kill is followed by a bounded wait for every stage-owned handle to observe and reap its child. If any group kill is rejected or a stage does not reach that terminal disposition within the teardown bound, the call fails closed with ErrorReason::Teardown (TeardownCause::Timeout) and retains that prefix instead of claiming a timed-out result over a potentially-live stage.
• A per-stage Command::timeout kills that stage's whole subtree — its own sub-group, grandchildren of a forking sh -c … included, not just its direct child. Every stage is evaluated by the same pipefail rule (D14): a stage that hit its own deadline — inner or last — surfaces on run() as that stage's ErrorReason::Timeout, reporting that stage's own deadline (not the chain's, and never 0ns).

Cancellation has two forms. Pipeline::cancel_on(token) is the chain-level control: the token gap-fills into every stage that doesn't already carry its own Command::cancel_on (an explicit per-stage token is left intact), so firing it tears the whole chain down and the run resolves to ErrorReason::Cancelled after confirmed teardown; a refused terminal step is ErrorReason::Teardown. (A cancel_on token on an individual stage Command also cancels that stage and errors the pipeline, but the pipeline-level builder is the clearer authority.) See Timeouts & cancellation.

Cancellation finalization collects the stages' bounded terminal dispositions, including siblings that settle after the first Cancelled error. If any sibling reports an unconfirmed kill/escalation/reap, its ErrorReason::Teardown deterministically outranks ordinary cancellation, stdin, and output-pump errors; the first teardown error observed in completion order supplies the retained OS source. A failed fallback group kill also keeps the cause latched when proactive teardown first fired: stage-local or chain cancellation reports TeardownCause::Cancellation, while an earlier stage failure remains TeardownCause::PipelineFailure even if a token fires during the drain grace. The same terminal-confirmation bound follows a successful fallback fan-out kill, so buffered collection cannot wait forever for an unreaped sibling.

Streaming a live chain

The verbs above buffer the whole run. For a long-lived chain you read from rather than wait out — journalctl -f | grep ERROR, tail -F access.log | jq — Pipeline::start() returns a PipelineSession: the multi-stage analogue of the RunningProcess a single Command::start() gives you. It streams the last stage's stdout as it arrives while every inner stage drains in the background, then folds the same pipefail outcome at finish().

use processkit::prelude::StreamExt;
use processkit::{Command, Finished, Outcome};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // journalctl -f | grep --line-buffered ERROR
    let mut session = Command::new("journalctl").arg("-f")
        .pipe(Command::new("grep").args(["--line-buffered", "ERROR"]))
        .start()
        .await?;

    let mut lines = session.stdout_lines()?;   // the *last* stage's stdout, live
    while let Some(line) = lines.next().await {
        println!("error: {line}");
        // …break out when you've seen enough, then tear the chain down…
        break;
    }
    drop(lines);

    // Fold the pipefail outcome. The last stage's stdout was already streamed,
    // so — like RunningProcess::finish — none is re-bundled; `outcome` and
    // `stderr` come from the pipefail-attributed (culprit) stage.
    session.start_kill()?;                      // stop the whole chain now
    let Finished { outcome, stderr, .. } = session.finish().await?;
    if outcome != Outcome::Exited(0) {
        eprintln!("chain ended {outcome:?}: {stderr}");
    }
    Ok(())
}

The session mirrors RunningProcess:

• stdout_lines() / events() — the last stage's stdout (lines) or its full lifecycle (Started → interleaved stdout+stderr → Exited), each consume-once (a second take is a loud Err, never a silently-empty stream).
• wait_for_line(pred, within) — wait for a readiness banner on that stream without tearing the chain down (an ErrorReason::NotReady on timeout, like the single-process probe).
• finish() — the streaming analogue of output_string(): the pipefail-attributed stage's outcome and its own stderr in a Finished (no stdout — you already streamed it). A chain-wide Pipeline::timeout that elapsed reports Outcome::TimedOut; a Pipeline::cancel_on that fired while the chain was still running surfaces as ErrorReason::Cancelled — exactly as in the buffering verbs. After a timeout/cancellation/proactive fan-out kill, finish() boundedly waits for every stage-owned handle to confirm its child was reaped. A rejected kill or expired confirmation is ErrorReason::Teardown and takes priority over the initiating disposition. As for a single Command::cancel_on, that disposition is first-observation-wins: every stage is observed while the session is live (each inner stage by its background drain, the last stage by the session's own watcher), so a token fired after the chain had already ended does not rewrite its real outcome into Cancelled, even when finish() is called afterwards.
• start_kill() and kill-on-drop — stop the whole chain now, or drop the session and every stage's tree dies. start_kill() attempts every stage group and returns ErrorReason::Teardown with TeardownCause::ExplicitKill when any group rejects the operation; kill-on-drop remains the backstop.

Whole-chain teardown still applies while streaming: any stage's checked failure proactively tears the chain down (so a quiet upstream can't hold a failed live chain open), and the chain-wide timeout / cancellation bounds the session regardless of which stage is slow. That includes the last stage, and without waiting for finish() — a standing watcher observes it for a terminal outcome, so holding an unfinished session is not a way for a failed chain's upstream to keep running. The teardown is prompt rather than instantaneous: it lands within one probe interval of the last stage's failure plus the short drain grace. A last stage that exits cleanly deliberately fires no teardown — a chain whose stages all succeed is not a failed chain.

Re-running a pipeline

A Pipeline is Clone and re-runnable — stages are re-cloned per run. The one caveat is inherited from Command: a one-shot stdin source on the first stage (Stdin::from_reader / from_lines) is consumed by the first run; re-running then fails loud (an ErrorReason::Io at launch, D10) rather than silently feeding empty stdin. Use the reusable sources (from_string / from_bytes / from_iter_lines / from_file) when a chain runs more than once.

Next: Timeouts, retries & cancellation · Running commands · Process groups

Timeouts, retries & cancellation

‹ docs index

Three ways a run ends early, with three different philosophies:

• a timeout is data — the deadline was part of the run's contract, so its expiry is captured in the result (and only the success-checking verbs turn it into an error);

• a retry is a policy — the success-checking verbs replay the run while your classifier says the failure is transient;

• a cancellation is an abandonment — the caller changed its mind, so every path reports an error; there is no result worth inspecting. (The abandonment is about the outcome, not the manner: the goodbye can be made soft with cancel_grace, and it is still always an error.)

• Timeouts

• Retries

• Cancellation

• Precedence and interactions

Timeouts

Command::timeout(d) kills the whole process tree at the absolute deadline — not just the direct child, so a wrapper script's grandchildren die too.

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Captured: inspect the flag yourself.
    let result = Command::new("slow-tool")
        .timeout(Duration::from_secs(5))
        .output_string()
        .await?;
    if result.timed_out() {
        println!("partial output before the kill: {}", result.stdout());
    }

    // Raised: the checking verbs convert the flag into a typed error.
    let err = Command::new("slow-tool")
        .timeout(Duration::from_secs(5))
        .run()
        .await
        .unwrap_err();
    assert!(matches!(err.reason(), processkit::ErrorReason::Timeout { .. }));
    Ok(())
}

Where each verb lands:

Every row below assumes the terminal kill/escalation/reap was confirmed. If the OS rejects that teardown, the consuming path fails closed with ErrorReason::Teardown { cause, operation, source, .. } instead of reporting a successful timeout disposition over a potentially-live child/tree. Captured stdout/stderr remains attached as a bounded best-effort prefix.

Output-inactivity watchdog

Command::inactivity_timeout(d) kills a single run when neither stdout nor stderr has produced bytes for d. Its clock starts at spawn and resets on every successful read from either stream, including merged PTY output:

use processkit::{Command, Outcome};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
let result = Command::new("build-tool")
    .timeout(Duration::from_secs(30 * 60))
    .inactivity_timeout(Duration::from_secs(5 * 60))
    .output_string()
    .await?;

if result.outcome() == Outcome::InactivityTimedOut {
    eprintln!("build produced no output for five minutes");
}
Ok(())
}

The first watchdog to fire wins. Both use the same whole-tree teardown and honor timeout_grace / timeout_signal, but their results remain distinct: Outcome::TimedOut means the absolute runtime expired; Outcome::InactivityTimedOut means the output went quiet. timed_out() is true for either; inactivity_timed_out() identifies the latter. Checking verbs turn both into ErrorReason::Timeout after confirmed teardown; its inactivity field carries the distinction and its timeout field is the window that fired. A failed kill/escalation/reap is ErrorReason::Teardown with TeardownCause::InactivityTimeout or TeardownCause::Timeout instead.

This first iteration applies to individual command runs, including capture, streaming, first-line consumption, and PTY mode. Readiness probes drain output and therefore refresh the activity clock, but do not arm either command watchdog themselves. Pipeline-wide and supervisor-wide inactivity policies are separate concerns; commands they launch still keep their own configured watchdog.

Two distinct deadline families to keep apart:

• Command::timeout / Command::inactivity_timeout — the run's own contracts, this section.
• The readiness probes' within parameter — gives ErrorReason::NotReady and never kills the child.

Graceful timeout

By default a watchdog hard-kills at once. Add timeout_grace(d) to give the tree a chance to clean up: when it fires the tree is sent SIGTERM (or the signal chosen with timeout_signal, which needs the process-control feature), allowed up to the grace window to exit, then SIGKILLed — the same SIGTERM → wait → SIGKILL tier as ProcessGroup::shutdown. A signal-handling child that exits ends the grace early.

use processkit::Command;
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let result = Command::new("slow-tool")
        .timeout(Duration::from_secs(30))
        .timeout_grace(Duration::from_secs(5)) // SIGTERM, wait up to 5s, then SIGKILL
        .output_string()
        .await?;
    Ok(())
}

timed_out() is true regardless of whether the child exited on the signal or was SIGKILLed after the grace — the deadline is what fired. Windows has no signal tier by default: timeout_grace is accepted but the deadline kills the job atomically. To get a real soft-shutdown window there, add windows_graceful_ctrl_break(): the direct child is spawned in its own console process group and, at the deadline, sent a console CTRL_BREAK before the grace window, then TerminateJobObject'd if it hasn't exited — the same timeout_grace window now actually meaning something on Windows. It works only for a child that shares this process's console (a create_no_window / DETACHED_PROCESS child never receives the event and rides the grace to the hard kill), and sends CTRL_BREAK rather than the Unix timeout_signal. See Process groups → Windows opt-in.

The explicit RunningProcess::shutdown(grace) verb (stop a started handle on demand) composes with a Command::timeout: its own SIGTERM → grace → SIGKILL is the single teardown (it does not also fire the run's timeout teardown), and if the deadline has already elapsed when you call shutdown, the outcome is reported as Outcome::TimedOut — the grace you pass governs the teardown timing.

Observing the grace window

With the tracing feature on, the teardown driver narrates each grace-window transition live on the processkit target — soft_signal (the SIGTERM / CTRL_BREAK was issued), grace_started (the wait began, with grace_ms), and then one of drained (the tree exited in time), escalated (the grace elapsed and the tree was hard-killed), or spared (a non-escalating stop left survivors) — each carried in a stable phase field and stamped by your subscriber at the instant it happens. This is the same soft-signal → grace → drain/kill ladder every graceful path drives (timeout_grace, cancel_grace, RunningProcess::shutdown, ProcessGroup::stop), so you get one uniform timeline whichever verb fired it. For the same facts after the teardown returns — as a typed value rather than log events — reach for ProcessGroup::stop's ShutdownReport. Neither is a control surface: both are observation only, and never carry argv/env.

Retries

retry(max_attempts, backoff, classifier) replays a failed run — up to max_attempts total attempts, sleeping backoff between tries, retrying only while the classifier accepts the error:

use processkit::{Command, ErrorReason};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("curl")
        .args(["-fsS", "https://example.com/api"])
        .timeout(Duration::from_secs(10))
        .retry(3, Duration::from_millis(250), |e| {
            // transient: network timeouts and curl's "couldn't connect" (7)
            matches!(e.reason(), ErrorReason::Timeout { .. })
                || matches!(e.reason(), ErrorReason::Exit { code: 7, .. })
        })
        .run()
        .await?;
    Ok(())
}

Ground rules:

• Retries apply to the success-checking paths only (run, exit_code, probe, ProcessRunnerExt::checked — and everything built on them, e.g. CliClient). The non-erroring output_string capture never retries: it didn't fail.
• The classifier sees the typed error — match on variants, codes, even the captured stderr.
• Each attempt re-runs the same Command — so a command whose stdin is a one-shot source (table), consumed by the first run, is not retried at all: the first attempt's error is returned as-is, since a second attempt could only replay empty stdin. Use a reusable stdin source if a stdin-bearing command must retry. (A one-shot source re-run outside the retry loop — a Supervisor incarnation, a pipeline re-run — instead fails loud with an ErrorReason::Io (InvalidInput) at launch.)
• Cancelled and Teardown errors are never retried, classifier or not. A cancelled token stays cancelled; an unconfirmed teardown may have left the failed attempt live, so another attempt could duplicate it.

For "keep it alive" (restart a service whenever it exits) rather than "replay this one operation", use a Supervisor — same backoff shape, different loop condition.

Cancellation

Hand any command a CancellationToken (re-exported at the crate root); cancelling the token tears the run's tree down and makes every consuming path report ErrorReason::Cancelled once terminal teardown is confirmed. If the OS rejects a required kill/escalation/reap, the path reports ErrorReason::Teardown with TeardownCause::Cancellation instead:

use processkit::{CancellationToken, Command};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let shutdown = CancellationToken::new();

    // Wire the same parent token into many jobs via child tokens:
    let job = tokio::spawn({
        let token = shutdown.child_token();
        async move {
            Command::new("long-export").cancel_on(token).run().await
        }
    });

    // Ctrl-C handler, sibling failure, UI button, …
    shutdown.cancel();

    assert!(matches!(
        job.await.unwrap().map_err(|e| e.into_reason()),
        Err(processkit::ErrorReason::Cancelled { .. })
    ));
    Ok(())
}

The contract, path by path:

Graceful cancellation

By default a cancellation hard-kills at once — the mirror image of a bare watchdog. Add cancel_grace(d) to give the tree a chance to clean up: when the token fires the tree is sent SIGTERM (or the signal chosen with cancel_signal, which needs the process-control feature), allowed up to the grace window to exit, then SIGKILLed. This is the exact same SIGTERM → wait → SIGKILL ladder as timeout_grace — the same driver, the same phase observability — just fired by the token instead of the deadline. A signal-handling child that exits ends the grace early.

use processkit::{CancellationToken, Command};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let shutdown = CancellationToken::new();

    let job = tokio::spawn({
        let token = shutdown.child_token();
        async move {
            Command::new("long-export")
                .cancel_on(token)
                // SIGTERM, wait up to 5s for a clean shutdown, then SIGKILL
                .cancel_grace(Duration::from_secs(5))
                .run()
                .await
        }
    });

    shutdown.cancel(); // Ctrl-C, sibling failure, …

    // Still an error — only the manner of the teardown changed.
    assert!(matches!(
        job.await.unwrap().map_err(|e| e.into_reason()),
        Err(processkit::ErrorReason::Cancelled { .. })
    ));
    Ok(())
}

This is the knob for the recommended "one shared token for the whole app" shutdown pattern: without it, a Ctrl-C on the parent SIGKILLs every child outright, with no chance to flush state, finish a transaction, or remove a pidfile. RunningProcess::shutdown(grace) already covered that for handles you started by hand; cancel_grace extends it to the bulk verbs, the streamed runs, and the Supervisor — everything the token reaches.

Ground rules:

• Opt-in, and inert by default. Without cancel_grace every cancellation path behaves exactly as it always has (immediate hard kill). It also does nothing without a token.
• The confirmed outcome never changes. Whether the child exited on the soft signal or was killed after the grace, every consuming path reports ErrorReason::Cancelled. A refused escalation/reap is not confirmation and reports ErrorReason::Teardown instead. Both are terminal under retry.
• Independent of the timeout knobs. cancel_grace/cancel_signal do not read (and are not filled in by) timeout_grace/timeout_signal, so a command can say farewell differently for "the caller changed its mind" than for "the deadline expired". cancel_signal defaults to SIGTERM, like timeout_signal.
• Same teardown scope as any cancellation. An own-group run tears down the whole tree; a shared-ProcessGroup run reaches only its direct child (its grandchildren remain the documented shared-group gap).
• Windows has no POSIX signal tier — exactly as for timeout_grace, the soft tier is the best-effort WM_CLOSE post plus the opt-in windows_graceful_ctrl_break() console event; a tree with neither is killed atomically and grace goes unused.
• Cancel racing the deadline. Cancellation still wins the race (the outcome is Cancelled, never TimedOut). Which teardown runs follows the cancellation policy: without cancel_grace that tie hard-kills, as it always did; with cancel_grace it takes the graceful ladder, because that is what the caller asked cancellation to do.

Client-level default

A typed wrapper built on CliClient usually constructs and consumes its Commands internally — there is no place to chain a per-call cancel_on. Set the token once on the client; every command it builds carries it:

#![allow(unused)]
fn main() {
use processkit::{CancellationToken, CliClient};

let token = CancellationToken::new();
let gh = CliClient::new("gh").default_cancel_on(token.child_token());
// ... controller cancels `token` → every in-flight command of THIS client
// dies (whole tree), surfacing ErrorReason::Cancelled to the awaiting call.
}

Clients are cheap — scope cancellation by building one client per cancellable scope with its own (child) token, instead of threading tokens through call signatures. cli_client!-generated wrappers re-emit the builder, so Git::new().default_cancel_on(t) works for downstream crates too.

Precedence: a per-command cancel_on chained on a built command replaces the client default (explicit beats default, like a per-command timeout after default_timeout). To honor both sources, wire it explicitly — CancellationToken has no built-in merge: derive a child of the default (let c = default.child_token()), hand the command cancel_on(c.clone()), and have the second source call c.cancel(). Or simpler: build a dedicated client per scope.

Precedence and interactions

Timeout vs. cancellation. A timeout is captured; a cancellation is always an error. When both land on the same run, cancellation wins — you asked the run to stop mattering, so no result is synthesized:

use processkit::{CancellationToken, Command};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let token = CancellationToken::new();
    token.cancel();

    let err = Command::new("tool")
        .timeout(Duration::from_millis(1))   // would have been a Timeout…
        .cancel_on(token)                    // …but cancellation takes priority
        .run()
        .await
        .unwrap_err();
    assert!(matches!(err.reason(), processkit::ErrorReason::Cancelled { .. }));
    Ok(())
}

Teardown failure vs. every initiating disposition. The timeout/cancellation ordering above applies only after terminal teardown is confirmed. A non-routine kill, graceful escalation, or reap failure takes priority over timeout, inactivity timeout, cancellation, pipeline failure, stdin failure, and pump failure. ErrorReason::Teardown preserves the initiating TeardownCause, the original io::Error, and any prefix the bounded output drain already captured; callers must assume the child/tree may still be live.

Which knob for which job:

Next: Supervision · Streaming & interactive I/O · Running commands

Errors

‹ docs index

One structured type covers every failure mode — spawn, exit, timeout, cancellation, IO — so a caller pattern-matches on a typed variant instead of parsing strings. Since 3.0, Error is a pointer-sized wrapper — a Box<ErrorReason> — that keeps Result<T, Error> small on the pervasive run path; the variants live on the re-exported [ErrorReason] enum, reached through err.reason() (or moved out with err.into_reason()). The read accessors (code(), is_timeout(), diagnostic(), …) and Display/Debug work on Error directly, unchanged. Several variants look alike at a glance but carry different contracts; this guide is the one page that lays them all out side by side: which variant fires from where, how to classify it, and what to do about it.

Upgrading from 2.x: a direct match err { Error::Exit { .. } => … } becomes match err.reason() { ErrorReason::Exit { .. } => … }. See Upgrading.

• Variant reference
• Variants that look alike but aren't
• Classifiers
• Total classification: kind()
• Stable machine identifiers
• Matching under #[non_exhaustive]
• Errors and retries
• Errors and supervision

Variant reference

Variants that look alike but aren't

• Timeout is captured, Cancelled is always an error. output_string/ output_bytes return Ok with timed_out() == true on a deadline — the caller decides whether that counts as failure. A cancellation, by contrast, reports Err(ErrorReason::Cancelled) on every consuming path, streaming included, because it's a deliberate caller action, not run data. When a run both hits its deadline and gets cancelled, cancellation wins (checked first). See Precedence and interactions.
• Teardown outranks the disposition that requested it. Timeout, inactivity timeout, and Cancelled are reported only after terminal teardown is confirmed (or an already-exited race is confirmed by reap). If a kill/escalation/reap fails, Teardown carries the initiating TeardownCause, original io::Error, and the bounded output prefix instead of claiming a potentially-live tree was handled. It also outranks stdin and pump errors from the same finalization window.
• NotReady is not Timeout. Command::timeout is the run's own contract and kills the tree; a readiness probe's within deadline is a separate clock layered on top of an already-running child, and giving up on it never kills anything. is_timeout() is false for NotReady.
• NotFound vs. Spawn. NotFound is the single representation of "program not found" — bare name or path, any platform, one variant, is_not_found() == true. Spawn is every other OS-level launch failure once the program was located (permissions, a bad cwd, a .cmd/.bat needing cmd.exe) — is_not_found() is false there, so a "not installed?" hint never fires on the wrong condition (e.g. a bad working directory).
• CassetteMiss is not is_not_found. A stale/incomplete cassette and a genuinely missing program are different failures a wrapper needs to tell apart — treating a cassette miss as "optional tool not installed" would silently hide a test-fixture bug.
• Exit vs. Signalled. Both carry captured streams and a Display diagnostic tail, but Exit has a code() and may or may not be a failure (is_success()/ok_codes decide); Signalled has no code at all — code() is None — and is always terminal.
• ResourceLimit means "the cap couldn't be applied", never "the cap fired". It is an admission error: the requested value was nonsensical (Invalid), the platform has no mechanism with whole-tree resource accounting (Unsupported), or a capable mechanism rejected this request (Unenforceable). ProcessGroup::with_options raises it before anything runs and hands back no group. ProcessGroup::update_limits raises it against an already-running group, where it does not mean "nothing changed": the caps are written axis by axis, so a part-way failure can leave a mix of old and new in force — see updating a live group. A cap that was applied and then actually stopped the tree produces no error at all — the child just exits non-zero (or dies by SIGKILL), exactly like a self-inflicted crash. For that question ask the group afterwards: ProcessGroup::limit_evidence() returns a per-axis LimitVerdict — Tripped (authoritative kernel/OS evidence that this cap fired), NotTripped, or Unknown (no evidence available on this mechanism — deliberately not folded into a "no"). Exit codes and signals are never consulted for it, precisely because they cannot tell a cap-driven kill from an ordinary failure. So: ResourceLimit on the error path, limit_evidence on the result path — two different questions, and this error's semantics are unchanged by it. They can land on the same axis on a live group, though: an axis named by a failed update_limits is still read from the counters, because that failure is not a rollback. (Left as a bare code span, not a docs.rs link: LimitVerdict ships in the next release, so a docs.rs URL would 404 until then.)

Classifiers

Total classification: kind()

The classifiers above answer one question each. When you route every failure onto your own shape — a CLI folding each disposition into a distinct process exit code, a cross-language binding raising a matching exception class, a router picking a retry policy — you want one total classification instead of a chain of is_* checks ending in "everything else". err.kind() is that: a compact [ErrorKind] with one bucket per operational disposition, derived from each variant's existing semantics (not invented), and covering every variant — present and future — through an exhaustive match inside the crate.

kind() is a routing answer, deliberately coarser than the variant — it is not a replacement for matching reason() when you need the details (the exit code, the captured streams, the timeout duration, which limit failed). It stays consistent with the point classifiers: is_not_found() ⇔ kind() == NotFound, is_permission_denied() ⇔ PermissionDenied, is_timeout() ⇔ Timeout, is_teardown() ⇔ Teardown, and so on.

ErrorKind mirrors std::io::ErrorKind: it is #[non_exhaustive] and carries an Other bucket, so a downstream match needs a catch-all arm — which is exactly what makes it forward-compatible.

#![allow(unused)]
fn main() {
use processkit::{Error, ErrorKind};

// Fold every failure onto your own exit codes — one arm per kind you care
// about, a catch-all for the rest (kinds are `#[non_exhaustive]`).
fn exit_code(err: &Error) -> i32 {
    match err.kind() {
        ErrorKind::NotFound => 127,
        ErrorKind::PermissionDenied => 126,
        ErrorKind::Timeout => 124,
        ErrorKind::Exit => 1,
        // A future kind (or one behind a feature this build doesn't enable,
        // e.g. ResourceLimit) routes here instead of breaking the build.
        _ => 70,
    }
}

let err = Error::parse("jq", "unexpected token");
assert_eq!(err.kind(), ErrorKind::Other);
assert_eq!(err.kind().name(), "other");
assert_eq!(exit_code(&err), 70);
}

Stable machine identifiers

When you publish a machine-readable contract over this crate's types — a CLI's JSONL schema, a cross-language binding, a structured log field — you need one canonical string per enum variant, not a table you hand-maintain (and that silently mislabels a new variant as "unknown"). The reporting and configuration enums carry that table for you:

The language-neutral, canonical dictionary is spec/identifiers.json. The table below explains the Rust API that produces it; both shapes describe the same contract and must change together.

The identifiers are a compatibility surface, held stable like the rest of the public API: a new variant gets a new identifier, and an existing identifier is never renamed without a major release. They are diagnostic names — a stable vocabulary, not a frozen record schema — and the opt-in report-serde feature puts that very vocabulary on the wire rather than minting a second one: it implements serde::Serialize for the crate's report types, and every enum in them serializes as its name() ({"kind": "exited", "code": 0, "signal_number": null}, never serde's derived {"Exited": 0}), so a consumer that adopted the dictionary and a consumer that serializes a report read the same strings. The single exception is the one the table above already records: Signal::Other has no curated identifier, so a Signal travels as its identifier string when curated and as a bare i32 when not — confined to the signal key, while the raw exit-signal number an Outcome carries is the separate key signal_number. That feature is Serialize-only (these values are reported, never supplied back — see the report-only enums listed below), it never puts captured output, argv or environment values on the wire, and it promises the spelling of what it emits, not a frozen field set: every report type is #[non_exhaustive], keeps its fields private, or both — grown, not frozen — so a minor release may add a key, never rename one, and a consumer must ignore unknown keys. See the crate-root report-serde section for the full rules. Mechanism and ParentDeathCleanup use the spellings downstream tools already publish (job_object/cgroup_v2/process_group, whole_tree/direct_child_only/none), so adopting them needs no migration; the FreeBSD reaper mechanism adds process_reaper in the same shape. SoftStopScope (the group-axis soft-stop reach, process-control) reuses the same whole_tree and none spellings for its shared cases, adding opt_in_members for the Windows partial-reach case. LimitVerdict (limits) is the one entry that is not an error classification at all: it is the post-run answer to "did this cap fire?" (tripped / not_tripped / unknown), while LimitKind / LimitReason describe the admission-time ResourceLimit failure — two different questions, never two spellings of one answer (see Variants that look alike but aren't), and both carry the same stability promise.

#![allow(unused)]
fn main() {
use processkit::{Mechanism, Priority};

// Forward — a stable identifier for machine-readable output:
assert_eq!(Mechanism::CgroupV2.name(), "cgroup_v2");
assert_eq!(Priority::BelowNormal.name(), "below_normal");

// Inverse (a config value, a CLI flag, a call from another language) — an
// honest `None` on an unrecognized name, never a silent default:
assert_eq!(Priority::from_name("below_normal"), Some(Priority::BelowNormal));
assert_eq!(Priority::from_name("turbo"), None);
}

The enums the table above lists as from_name exceptions report a name() but take no inverse, because they are classifications, events or fates the crate reports and never accepts back. Outcome::name() reports the disposition only (exited / signalled / timed_out) — the name alone can't carry the exit code or signal number (read those from code() / signal()), and an Outcome is always reported, never supplied. ErrorKind is the same: a total failure classification the crate hands you via Error::kind(), never one you supply back in. Read the fuller payload from the [ErrorReason] variant when a name isn't enough. ProcessEvent and SupervisionEvent similarly identify lifecycle event kinds; their payloads carry the process output, outcome, timing, or supervision detail. SoftSignal (process-control) names which fate a graceful stop's best-effort soft-signal tier met — sent / unsupported / failed — an observation made after a teardown, never a request; the Signal it concerns travels separately (ShutdownReport::attempted_signal). Signal::name() returns Option because the Other(i32) escape hatch has no curated name (it still has a from_name).

Matching under #[non_exhaustive]

Error — and several of its struct-like variants — are #[non_exhaustive]: a future release can add a variant or field without that being a breaking change, but it also means every downstream match must carry a catch-all arm.

use processkit::{Command, ErrorReason};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let err = Command::new("maybe-missing-tool").run().await.unwrap_err();

    match err.reason() {
        ErrorReason::NotFound { .. } => eprintln!("is it installed?"),
        ErrorReason::Timeout { .. } => eprintln!("hit its deadline"),
        ErrorReason::Cancelled { .. } => { /* caller-initiated, nothing to log */ }
        ErrorReason::Teardown { cause, operation, .. } => {
            eprintln!("{cause:?} teardown failed during {operation}")
        }
        ErrorReason::Exit { code, .. } => eprintln!("exited with {code}"),
        // #[non_exhaustive]: a future variant (or a today's variant behind a
        // feature this build doesn't enable, e.g. ResourceLimit) falls here.
        other => eprintln!("run failed: {other}"),
    }
    Ok(())
}

Prefer the classifiers above (is_not_found(), is_timeout(), …) over destructuring when all you need is a yes/no answer — they read the variant without you having to keep the catch-all arm in sync as fields are added.

Errors and retries

Command::retry(max_attempts, backoff, classifier) replays a failed run while your classifier accepts the typed error — the classifier is exactly this guide's variant table, read by the caller:

use processkit::{Command, Error};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let out = Command::new("curl")
        .args(["-fsS", "https://example.com/api"])
        .timeout(Duration::from_secs(10))
        .retry(3, Duration::from_millis(250), |e: &Error| {
            // Transient: our own deadline, or a spawn/IO condition a bare retry clears.
            e.is_timeout() || e.is_transient()
        })
        .run()
        .await?;
    Ok(())
}

ErrorReason::Cancelled and ErrorReason::Teardown are never retried, whatever the classifier says. A cancelled token stays cancelled; an unconfirmed teardown may have left the failed attempt live, so launching another copy would compound it. See Retries for the full ground rules (stdin re-use, which verbs retry at all).

Errors and supervision

A Supervisor restarts a crashing service rather than replaying one operation, but it faces the same "is this permanent?" question — its give_up_when(classifier) gate is the same classifiers applied to a GiveUpAttempt:

use processkit::{Command, GiveUpAttempt, Supervisor};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("maybe-typo'd-binary"))
        .give_up_when(|attempt| match attempt {
            // The child never started at all — e.g. is_not_found() for ENOENT.
            GiveUpAttempt::Failed(err) => err.is_not_found(),
            // A completed run your own domain knows is a permanent crash.
            GiveUpAttempt::Crashed(res) => res.code() == Some(78),
            _ => false,
        })
        .run()
        .await?;
    println!("stopped: {:?}", outcome.stopped);
    Ok(())
}

GiveUpAttempt::Failed(&Error) is the spawn/IO path (no ProcessResult was ever produced); GiveUpAttempt::Crashed(&ProcessResult<String>) is a completed-but-failing run. A recognized-permanent failure reports StopReason::GaveUp instead of restarting forever. ErrorReason::Cancelled is terminal here too — supervision returns Err(Cancelled) instead of restarting into a still-cancelled token, give_up_when or not.

ErrorReason::ResourceLimit and ErrorReason::Unsupported specifically are how a sandboxing request fails loud instead of silently doing less than asked — see Running untrusted children for the hardening checklist that relies on exactly that honesty.

Next: Running commands · Timeouts, retries & cancellation · Supervision · Running untrusted children

Troubleshooting

Start here when you have a symptom but do not yet know which processkit subsystem owns it. The sections below give a short diagnosis and route to the guide that owns the full contract.

Before changing code, preserve the structured evidence:

• inspect Error::reason() / Error::kind() and Error::diagnostic() instead of parsing Display text;
• record ProcessGroup::mechanism() (or use host_containment() before any group exists);
• note whether the run owns a private group or uses a shared group — RunningProcess::kills_tree_on_drop() answers that directly;
• distinguish the command's lifetime deadline from a readiness probe's within deadline.

A child survived drop

First identify what was dropped and who owns containment:

• Command::start() creates a private group. Its handle reports kills_tree_on_drop() == true; dropping it hard-kills the local tree.
• ProcessGroup::start() creates a shared-group handle. It reports false: the separately held ProcessGroup controls the tree's lifetime, so stop or drop that group (or call start_kill() when only the direct child should stop). Keeping an Arc<ProcessGroup> clone alive keeps the owner alive too.
• spawn_detached() is the deliberate exception. Dropping DetachedChild does nothing because the child was explicitly launched to outlive this process.
• A process spawned outside processkit is outside the boundary until adopt() (with its Child handle) or adopt_external() (with only its pid). Adoption moves the named process, not descendants it already created; the POSIX process-group backend can only track an already-executed adopted child individually. adopt_external additionally reports Unsupported on FreeBSD and the other BSDs, where the crate has no start-time reader to anchor the pid on — a refusal, so a process left running there was never contained.
• A group whose members die to a teardown it never ran, or whose start suddenly fails "job is full" (Windows). Adopting a pid that already belonged to another Job Object nests this crate's job under that one, so the outer job's terminate or close reaches these members (later-started ones included) and its limits bind them. The mirror image on Linux cgroup v2: adopting a process takes it out of its previous cgroup, so an outside supervisor's teardown and limits stop applying to the process it thought it held. Neither is reverted on drop — see platform support.
• adopt_external returned "pid … was recycled while it was being adopted". The number changed hands inside the call, so nothing was adopted. Read the rest of that message before retrying: on Linux cgroup v2 it also says whether the number could be moved back out of this group's cgroup, and in the case where it could not, the process now holding the number is a member of this group and will be killed by its teardown — drop or tear down that group promptly if that is not acceptable.
• Nothing reaps a process adopted by pid. No exit status for it appears anywhere in this API, so a wait-shaped answer for it has to come from whoever is its actual parent; on the POSIX process-group backends an exited one that nobody reaps stays a zombie, keeps probing alive through a graceful shutdown's whole grace, and cannot be cleared by escalate_to_kill.
• A command reached through ssh is local containment only: the local ssh client dies, but the remote command needs remote-side containment.

Then inspect the mechanism. JobObject and CgroupV2 are real tree containers. ProcessGroup is intentionally reported as the weaker fallback: a descendant that calls setsid can escape it. Also remember that a process abort or SIGKILL can skip Rust Drop. Windows still closes the Job Object and reaps the whole tree; Linux's opt-in kill_on_parent_death() reaches only the direct child, and macOS/BSD have no equivalent owner-death hook.

If an ordinary private-group child survives a normal Rust drop without one of those boundaries, capture the mechanism, platform, program shape, and a minimal reproducer — that is not an expected lifecycle outcome.

Full contracts: process ownership and adoption, deliberate detachment, containment mechanisms, and the remote-execution boundary.

NotFound, but the tool is installed

NotFound means processkit could not resolve the program under the command's actual launch rules. Check these in order:

1. A bare name is searched on the command's effective child PATH (and on Windows with PATHEXT). env_clear(), inherit_env(...), an explicit PATH, or a missing extension can make that different from the interactive shell where you verified the installation.
2. A program containing a path separator is not searched on PATH. current_dir does not portably re-anchor a relative program path; combine a child working directory with an absolute program path.
3. prefer_local(dir) affects only bare names. A relative preferred directory is resolved against the parent's real current directory, not the command's current_dir.
4. Run Command::resolve_program() (or processkit::which for a bare name). It is a spawn-free preflight using the same PATH / PATHEXT, executable-bit, environment, and prefer_local logic as the real launch.

If preflight finds the file but launch still fails, inspect whether the error is actually Spawn: permissions, a bad cwd, or a Windows .cmd/.bat that needs cmd.exe are launch failures, not missing programs.

Full contracts: program and working-directory resolution, prefer_local, spawn-free preflight, and NotFound versus Spawn.

ResourceLimit while creating a group

Do not parse the English detail. Read limit_kind() and limit_reason():

• Invalid means the requested value itself is invalid;
• Unsupported means there is no whole-tree resource accounting for that limit (no container at all, or — on FreeBSD — a reaper that contains but does not account);
• Unenforceable means a suitable mechanism exists but this process cannot apply the cap.

On Linux, processkit's per-tree limits require the process to own the real cgroup v2 hierarchy root so it can enable controllers. A container cgroup namespace root is not that root, and a normal systemd session, scope, or service is not delegated this way. Both ordinary and privileged containers therefore commonly return Unenforceable; requesting a cap fails loud instead of silently creating an unbounded group.

Inside Docker/Kubernetes, use the orchestrator's memory/CPU limits as the outer boundary. Use processkit's limits on a host where the process genuinely owns the cgroup root (typically a minimal non-systemd init on bare metal or a VM). Kill-on-drop can still fall back to a POSIX process group when no per-tree cap was requested; it is the unenforceable protection request that must fail.

Full contracts: container limits versus processkit limits, resource-limit semantics, and cgroup prerequisites.

The process runs, but no output appears

Separate transport from the child's own buffering:

• One-shot capture (output_string, run) returns output when the child exits. For live output, use start() plus stdout_lines(), a line handler, or a tee.
• A line stream emits only after a terminator. An interactive Password: or REPL prompt without \n needs wait_for_output, not wait_for_line.
• A child may block-buffer stdout when it sees a pipe. Prefer the tool's own unbuffered/line-buffered switch when available. If its behavior is genuinely isatty()-gated, enable the pty feature and use use_pty().
• StdioMode::Inherit, Null, and stdout_file* intentionally bypass the capture pump in ordinary pipe mode. A PTY instead exposes its merged stdout and stderr only as piped logical stdout, so combining use_pty with those destinations (or a separate stderr destination) is rejected before spawn; its separate stderr capture is empty on a supported PTY run.

PTY is a semantic change, not just a flushing switch: it merges streams, uses terminal line framing, and may add control sequences. Keep pipes when stream identity or exact bytes matter.

Full contracts: streaming stdout, prompt-aware waiting, interactive/TTY launch, and the PTY platform matrix.

PTY output contains ANSI/VT garbage

Terminal applications emit color, cursor movement, alternate-screen, OSC title/hyperlink, and other VT sequences. use_pty() does not emulate a screen. Opt into sanitize_vt() (or the per-stream stdout_sanitize_vt() / stderr_sanitize_vt() variants for pipe-capable wrappers) when retained text should be plain.

Sanitization is deliberately scoped to the capture backlog. Handlers, decoded tees, raw tees, and output_bytes() still see exact input; sanitize inside those sinks if they also need clean text. PTY framing is separately \r-aware by default; an explicit LineTerminator::Newline can make redraw-style progress look delayed even after escapes are removed.

Full contract: PTY output hygiene.

PTY window size is rejected

pty_size and resize_pty accept u16 arguments, but zero is not a usable PTY axis and is rejected as ErrorReason::Io (InvalidInput) on every platform. Windows ConPTY has the narrower 1..=32767 range because COORD stores signed 16-bit fields; Unix accepts every non-zero u16 value. ProcessKit rejects an out-of-range ConPTY request rather than silently clamping it, so retry with a representable size. A refused launch starts no child, and a refused live resize does not mutate or close the existing terminal.

Full contract: PTY window size and live resize.

wait or a stream never finishes

Work through the resources that can still be open:

1. Stdin is closed by default. If you opted into inherit_stdin(), the child can wait for EOF from the parent's terminal or pipe. If you called keep_stdin_open() and took the writer, call ProcessStdin::finish() or drop it when the conversation ends.
2. Do not perform a large interactive write while nothing drains output. A full stdout/PTY buffer can stop the child reading stdin while the parent's stdin write is also blocked. Read and write concurrently.
3. stdout_lines() and wait_for_line() need a complete line. For an unterminated prompt, use wait_for_output().
4. wait_any() does not pump output. Drain chatty children first (or concurrently) before racing them.
5. A still-running descendant can hold an inherited pipe open after its direct parent exits. Bound the run with Command::timeout() or inactivity_timeout(). On a shared-group handle those watchdogs stop the direct child; stop the owning group when the whole shared tree must end.

A readiness within bound is not a run bound: NotReady leaves the child alive. If giving up on readiness should end the run, explicitly shutdown() an own-group handle, start_kill() a direct child, or stop the shared group.

Full contracts: interactive stdin and full-duplex deadlock, wait versus drain, stream deadlines, and timeouts and inactivity.

Graceful shutdown does not work on Windows

Dropping a handle/group is always the hard safety net; it never waits for application cleanup. Use shutdown, stop, or timeout_grace for a graceful tier, then identify which Windows soft-close path the child can receive:

soft_stop_scope() is the side-effect-free check for the live group: OptInMembers means at least a console/window target is reachable; Unsupported means no soft target exists. The console event is CTRL_BREAK, not CTRL_C, and the child must install the corresponding handler and exit within the grace.

Full contract: Windows graceful teardown and graceful timeouts.

Timeout instead of NotReady — or vice versa

They answer different questions:

Readiness probes deliberately do not arm the command watchdog while they poll. A probe can therefore return NotReady even if the command's absolute deadline has passed; the following consuming verb (finish, wait, output_string, and so on) enforces that run deadline and can then report Timeout. Conversely, a probe whose output stream closes before a match returns NotReady immediately because readiness can no longer happen — it does not wait out the rest of within.

After NotReady, choose explicitly: keep waiting, stop the handle/group, or surface startup failure. For checking code, branch on err.reason() / is_timeout(); is_timeout() is intentionally false for NotReady.

Full contracts: readiness probes, deadline families, and look-alike errors.

Next: Errors · Streaming & interactive I/O · Platform support · Running in containers

Supervision

‹ docs index

Where retry answers "run this once, replaying on failure", a Supervisor answers the different question "keep this alive": restart a child per policy whenever it exits, with bounded restarts, exponential backoff, and jitter — a minimal runit/systemd-style keeper, platform-agnostic because it sits entirely on the ProcessRunner seam.

• The shape
• Policies: what counts as a crash
• Backoff and jitter
• Failure storms
• Liveness health checks
• Stopping
• Giving up on permanent failures
• Outcomes
• Live sessions: start()
• Supervising inside a shared group
• Errors and cancellation

The shape

use processkit::{Command, RestartPolicy, Supervisor};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("my-server").args(["--port", "8080"]))
        .restart(RestartPolicy::OnCrash)           // default
        .max_restarts(5)                           // default: unlimited
        .backoff(Duration::from_millis(200), 2.0)  // default: 200ms × 2.0
        .max_backoff(Duration::from_secs(30))      // default: 30s cap
        .jitter(true)                              // default: on
        .stop_when(|res| res.code() == Some(0))    // optional exit condition
        .run()
        .await?;

    println!(
        "ended after {} restarts, reason: {:?}, last exit: {:?}",
        outcome.restarts, outcome.stopped, outcome.final_result.code(),
    );
    Ok(())
}

Each incarnation is one full captured run of the command (so the command's own timeout, stdin, env, … all apply per run — with the usual one-shot-stdin caveat for the second run onward).

Policies: what counts as a crash

A crash is any run that is not a success (ProcessResult::is_success, which honors the command's ok_codes): an exit code outside the accepted set (default {0}), a timeout, a signal-kill, or a spawn failure. A command with ok_codes([0, 2]) that exits 2 is a success, so OnCrash treats it as clean, not a crash.

Backoff and jitter

The n-th restart (0-based) sleeps

delay(n) = min(base × factor^n, max_backoff) × jitter

with jitter drawn uniformly from [0.5, 1.5) per restart. Jitter is on by default so a fleet of supervised workers restarted by the same incident doesn't stampede back in lockstep; jitter(false) gives deterministic delays (useful in tests with a paused tokio clock). A non-finite or < 1.0 factor is treated as 1.0 — constant delay, never a shrinking one.

base=200ms, factor=2.0, cap=30s:
restart #0 → ~200ms   #1 → ~400ms   #2 → ~800ms … #7 → ~25.6s   #8+ → 30s (cap)

The exponent n is not the lifetime restart count — it resets whenever a run stays up at least as long as max_backoff. So a long-lived service that crashes now and then restarts near base each time (it demonstrated health between crashes), while only a tight loop — each incarnation shorter than the cap — climbs to the ceiling. The floor is on uptime, not exit kind: under Always a worker that exits (cleanly or not) in under max_backoff is treated as flapping and escalates, which is what stops an exit 0 spin loop from hammering at the base delay. A single jittered delay can reach up to 1.5 × max_backoff.

Failure storms

Backoff spaces individual restarts; max_restarts is a lifetime cap. Neither distinguishes a service that fails once a day from one that is suddenly crash-looping. The opt-in storm guard does (a design borrowed from Go's suture supervisor — the idea, not the code):

use processkit::{Command, Supervisor};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("worker"))
        .storm_pause(Duration::from_secs(15))     // master switch — off by default
        .failure_decay(Duration::from_secs(30))   // score half-life (default 30s)
        .failure_threshold(5.0)                   // trip point (default 5.0)
        .run()
        .await?;

    println!("storm pauses taken: {}", outcome.storm_pauses);
    Ok(())
}

Each failed run adds 1 to a score that halves every failure_decay:

score = score × 0.5^(Δt / failure_decay) + 1

• Fails rarely: the score decays back toward 1 between failures and never reaches the threshold — the guard stays out of the way.
• Failure storm: failures arrive faster than the half-life drains them, the score climbs past failure_threshold, and the supervisor takes one collective pause of storm_pause (jittered into [0.5, 1.5) like the backoff), resets the score, and resumes.

Only failures feed the score — crashes and spawn errors, not clean exits restarted under RestartPolicy::Always. The pause stacks with (runs before) the per-restart backoff, and the max_restarts budget is checked first, so a storm pause never extends an exhausted budget. Pauses taken are reported in SupervisionOutcome::storm_pauses.

Liveness health checks

A RestartPolicy only ever reacts to a child that exits. It is blind to a child that is still alive but wedged — a deadlocked server, a stuck event loop, a process that stopped serving but never dies. From the policy's view that child is "running", so it is never restarted. The opt-in health check adds the missing dimension — the analogue of systemd's WatchdogSec and a container liveness probe:

use processkit::{Command, Supervisor};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("my-server").args(["--port", "8080"]))
        .health_check(
            // Any async predicate: `true` = healthy. A port connect here; an
            // HTTP `/healthz` request, a heartbeat file, a custom check elsewhere.
            || async { tokio::net::TcpStream::connect("127.0.0.1:8080").await.is_ok() },
            Duration::from_secs(5),       // probe cadence
        )
        .health_check_failures(3)         // consecutive misses tolerated (default 3)
        .max_restarts(10)
        .run()
        .await?;

    println!("liveness kills: {}", outcome.liveness_kills);
    Ok(())
}

The probe re-runs every interval for the life of each incarnation, in the same shape as the readiness wait_for check (it takes no handle — it observes the child out-of-band, over a port/endpoint/file). The first probe fires one interval after the incarnation starts, so a booting child gets that long before liveness is judged; a healthy child then loops here untouched for its whole lifetime. A zero (or otherwise degenerate) interval is clamped to a small safe minimum rather than passed through as-is — it neither panics nor turns the probe loop into a busy-spin, and the startup-grace promise above still holds.

When the probe fails health_check_failures times in a row (any healthy probe resets the streak, so a single blip is forgiven), the wedged incarnation is force-killed — dropped, which kills it on drop under the default JobRunner (the shared-group caveat from Errors and cancellation applies) — and treated as a crash: it flows through the RestartPolicy, backoff, the storm guard and max_restarts exactly like a real crash, and its synthetic final result carries Outcome::Signalled. It does not consult stop_when (there is no cleanly-completed run to evaluate). The effective grace before a kill is roughly interval × health_check_failures.

A liveness kill counts toward the backoff escalation using how long the incarnation actually stayed up before wedging — so a service that runs healthy for a long stretch and only occasionally wedges restarts near base, while one that wedges promptly after each restart self-throttles (the same uptime floor as Backoff and jitter). Force-kills are counted in SupervisionOutcome::liveness_kills. Under RestartPolicy::Never — a single "run once, but kill it if it goes unresponsive" bound — the force-killed run ends supervision with StopReason::Unhealthy. Left unset (the default), a supervisor behaves exactly as it did before health-checking existed.

Stopping

Four gates, checked in this order after every completed run:

1. stop_when(predicate) — sees the run's ProcessResult; returning true ends supervision regardless of policy (→ StopReason::Predicate). "Exit 0 is done, anything else is a crash" is the classic: stop_when(|res| res.code() == Some(0)) under RestartPolicy::Always.
2. The policy — OnCrash stops on a clean exit (→ PolicySatisfied).
3. give_up_when(classifier) — only consulted for a crash the policy would otherwise restart; recognizing it as permanent ends supervision (→ StopReason::GaveUp). See Giving up on permanent failures.
4. max_restarts(n) — at most n restarts = n + 1 total runs; an exhausted budget reports the last result (→ RestartsExhausted). max_restarts(0) means exactly one run.

give_up_when is checked before max_restarts, so a recognized-permanent crash reports the more specific GaveUp even when the budget hasn't run out yet — and before the failure-storm guard, so giving up never pays for a storm pause it was going to end anyway.

A liveness kill is not a completed run, so it skips gate 1 (stop_when) but still runs through the policy and gates 3–4 as a crash; under RestartPolicy::Never it ends supervision with StopReason::Unhealthy.

Giving up on permanent failures

Without more, the supervisor cannot tell a transient crash from a permanent one — a command that can never succeed (a missing binary, a config error that crashes on startup, a permanently-taken port) restarts forever under the default unlimited OnCrash, throttled only by backoff. give_up_when lets you recognize the unrecoverable case and stop instead:

use processkit::{Command, GiveUpAttempt, Supervisor};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("maybe-typo'd-binary"))
        .give_up_when(|attempt| match attempt {
            // The child never even started — e.g. ENOENT for a mistyped name.
            GiveUpAttempt::Failed(err) => err.is_not_found(),
            // A completed crash your own domain knows is permanent, e.g. a
            // documented "config invalid, do not restart" exit code.
            GiveUpAttempt::Crashed(res) => res.code() == Some(78),
            _ => false,
        })
        .run()
        .await?;
    println!("stopped: {:?}", outcome.stopped);
    Ok(())
}

GiveUpAttempt distinguishes the two shapes a permanent failure can take:

• Crashed(&ProcessResult<String>) — a completed run that counts as a crash. A match reports StopReason::GaveUp in the SupervisionOutcome, same as any other stop reason.
• Failed(&Error) — the child never started at all (spawn/IO failure, e.g. ENOENT). There is no ProcessResult to report here, so a match surfaces the classified error directly as run()'s Err — the same contract an exhausted budget already has on this path (see Errors and cancellation).

Unset by default: a permanent failure restarts forever exactly as before, bounded only by max_restarts / the storm guard — adding give_up_when never changes existing behavior until you set it.

Fallible control predicates

The control predicates above are infallible — stop_when and give_up_when return bool, health_check resolves to bool. When your predicate can itself fail (an I/O probe, a classification that parses something, or — the driving case — a callback in a wrapper over a language where any callback can throw), each has a try_* twin that returns Result<bool, E> for any E: Into<Box<dyn Error + Send + Sync>>:

Ok(true) / Ok(false) behave exactly as the infallible twin's true / false. An Err, however, aborts supervision and surfaces to the caller as run()'s Err — an ErrorReason::Predicate (kind ErrorKind::Predicate) carrying your predicate's own error verbatim as its source:

use processkit::{Command, Supervisor};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("my-server"))
        // The probe may itself fail — e.g. a health endpoint whose body your
        // parser rejects. An `Err` aborts supervision (surfaced as an
        // `ErrorReason::Predicate`) instead of voting "unhealthy".
        .try_health_check(|| async { probe_health().await }, Duration::from_secs(5))
        .run()
        .await?;
    println!("stopped: {:?}", outcome.stopped);
    Ok(())
}

#[derive(Debug)] struct ProbeError;
impl std::fmt::Display for ProbeError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!(f, "probe error") }
}
impl std::error::Error for ProbeError {}
async fn probe_health() -> Result<bool, ProbeError> { Ok(true) }

The distinction is deliberate: a predicate error is never a fabricated verdict. A normal stop_when/try_stop_when match (Ok(true)) ends supervision with a benign SupervisionOutcome (StopReason::Predicate); a predicate Err ends it with a failure the caller can tell apart by err.kind() == ErrorKind::Predicate and recover the original error from err.reason()'s Predicate { source, .. }. A failing try_health_check probe still tears the live incarnation down by the same kill a liveness failure uses — so aborting on a probe error leaks no child.

Each twin is purely additive: setting try_stop_when (etc.) replaces its infallible sibling on the same slot, and the infallible builders are unchanged.

Outcomes

run() resolves to a SupervisionOutcome:

use processkit::{Command, Supervisor};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let outcome = Supervisor::new(Command::new("job")).run().await?;

    outcome.final_result;  // ProcessResult<String> of the LAST run
    outcome.restarts;      // how many restarts happened (not counting run #1)
    outcome.stopped;       // StopReason::{Predicate, PolicySatisfied, GaveUp, RestartsExhausted, Unhealthy, Stopped}
    outcome.storm_pauses;  // failure-storm pauses taken (0 unless storm_pause is set)
    outcome.liveness_kills; // incarnations force-killed by a health check (0 unless health_check is set)
    Ok(())
}

Note run() returning Ok does not mean the child succeeded — it means supervision concluded. Inspect final_result (or ensure_success() it) for the child's own verdict.

Live sessions: start()

run() owns supervision from start to finish and only hands back a SupervisionOutcome at the very end — there is no handle to watch or steer it while it runs. For a daemon / process-manager that needs a live view, call start() instead of run(). It returns a SupervisionSession, a Send handle to supervision running on a background task:

• status() — a consistent snapshot (SupervisionStatus): whether supervision is still active, the live restart count, whether a failure-storm pause is in effect right now, and the current incarnation's pid / start time (both None between incarnations, during a backoff, or for a capture-only test double). Poll it any time; it never races the loop.
• events() — take the session's typed, one-consumer SupervisionEvents stream. It reports incarnation starts and outcomes, launch-error classes, scheduled backoffs, storm pauses, health-check failures, give-up decisions, and the terminal reason. Each variant has a stable name() for structured logs and metrics. The history is bounded at 128 events; a slow consumer gets an explicit SupervisionEvent::Lagged { skipped } marker and resumes from the oldest retained event instead of growing the supervisor without limit.
• stop(grace) — stop the current child gracefully (its normal SIGTERM → wait grace → SIGKILL path, under the default own-group runner) and end supervision with StopReason::Stopped — a deliberate, honest reason distinct from a crash, an exhausted budget, a cancellation, or a stop_when match. A stop taken during a backoff (no child alive right now) cuts the sleep short and ends at once, launching no further incarnation. An incarnation with no own group to shut down (a shared-ProcessGroup or a capture-only child) is stopped by firing its cancel token instead — so give the supervised command a cancel_grace if that child, too, should get a soft signal and a window rather than an outright kill.
• wait() — await the final SupervisionOutcome, exactly what run() would have returned.

use processkit::prelude::StreamExt;
use processkit::{Command, RestartPolicy, SupervisionEvent, Supervisor};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let mut session = Supervisor::new(Command::new("my-server"))
        .restart(RestartPolicy::Always)
        .start(); // supervision runs in the background from here

    // Take the stream once and drain it independently of the session handle.
    let mut events = session.events()?;
    tokio::spawn(async move {
        while let Some(event) = events.next().await {
            match event {
                SupervisionEvent::RestartScheduled { restart, delay } => {
                    println!("restart {restart} after {delay:?}");
                }
                other => println!("supervision event: {}", other.name()),
            }
        }
    });

    // ... elsewhere: observe it live ...
    let status = session.status();
    println!("active={}, restarts={}", status.is_active(), status.restarts());

    // ... on shutdown: stop gracefully and read the final outcome ...
    let outcome = session.stop(Duration::from_secs(5)).await?;
    println!("stopped: {:?}", outcome.stopped); // StopReason::Stopped
    Ok(())
}

Live status and events are additions to the exit-driven policy and callbacks, not replacements — supervision behaves identically to run(). Only a session stop() produces StopReason::Stopped; run() never does.

Two handle contracts round it out: dropping a SupervisionSession without wait()/stop() aborts the background task (no orphaned supervision task, and under the default JobRunner the in-flight incarnation is killed on drop), and start() needs a 'static runner because supervision moves onto a spawned task — the borrowed shared-group form (with_runner(&group)) is available through run(); own the group (by value, or an Arc<ProcessGroup>) to drive a shared group from a session.

Supervising inside a shared group

The supervisor runs through any ProcessRunner. The headline production variant injects a ProcessGroup so every incarnation — and everything it spawns — lives in one kill-on-drop container:

use processkit::{Command, ProcessGroup, RestartPolicy, Supervisor};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    let group = ProcessGroup::new()?;

    let outcome = Supervisor::new(Command::new("worker"))
        .with_runner(&group)                 // &group is itself a ProcessRunner
        .restart(RestartPolicy::OnCrash)
        .max_restarts(10)
        .run()
        .await?;

    // The group outlives supervision: drop it (or shutdown) to reap any strays.
    Ok(())
}

Mind one interaction: don't supervise into a group you've suspended — under the cgroup mechanism the restarted child would start frozen (and the spawn itself can block). Resume first.

The same injection point makes supervision logic hermetically testable — script a sequence of fake results and assert the restart/stop behavior with no real process; see Testing your code.

Errors and cancellation

A run that produces no result at all (spawn/IO failure) can't be judged by stop_when (it needs a ProcessResult) — but it is visible to give_up_when as GiveUpAttempt::Failed. The policy treats it as a crash and restarts (with backoff) unless the policy is Never, give_up_when classifies it as permanent, or the budget is exhausted — any of those surfaces the error itself as run()'s Err.

A cancelled incarnation is terminal: run() returns Err(ErrorReason::Cancelled) immediately. The token never un-cancels, so a restart could only produce another instantly-cancelled run — the supervisor refuses the futile loop.

An ErrorReason::Teardown incarnation is terminal too: its child or descendants may still be alive because the required kill/escalation/reap was not confirmed, so starting a replacement would violate the supervisor's one-incarnation-at-a-time contract. The original teardown error is returned without consulting restart policy or give_up_when.

A fallible control predicate (try_stop_when / try_give_up_when / try_health_check) that returns Err is likewise terminal: run() returns Err(ErrorReason::Predicate) — kind ErrorKind::Predicate, carrying your predicate's own error verbatim as its source — rather than a fabricated stop/unhealthy verdict.

Next: Testing your code · Timeouts, retries & cancellation · Process groups

Observability

‹ docs index

The crate can narrate what it does without you instrumenting a single call site, through two independent, additive, off-by-default seams:

• tracing — one structured event per lifecycle transition (spawn, exit, timeout/cancel firing, per-phase graceful teardown, retries, supervisor restarts). Good for a human reading a live log or a distributed trace.
• metrics — aggregate counters and histograms over the same moments (how many runs, how long they took, how they ended, how often you retried or restarted). Good for a dashboard, an SLO alert, or a fleet-wide latency percentile.

Both derive everything from data the crate already computes on the run's existing path — they add no extra work, no second wall-clock read, and no new public API. With a feature off, its call sites compile out entirely; the kill-on-drop tree guarantee and every run's behavior are byte-identical either way.

The motivating consumer is an orchestrator supervising many agentic-CLI subprocesses: SLO/latency signals across a production fleet, without hand-wiring a timer around every spawn.

metrics

Enable the metrics feature and the crate emits into whatever global metrics recorder your process installs. The crate ships no exporter of its own — you pick the backend (Prometheus, OpenTelemetry, StatsD, a test recorder …) exactly as you would for your own application metrics, and the crate's series simply appear alongside yours.

# Cargo.toml
[dependencies]
processkit = { version = "…", features = ["metrics"] }
metrics = "0.24"
# …plus any metrics-compatible exporter, e.g. metrics-exporter-prometheus.

Install a recorder once at startup — before you run any commands — and that is the entire wiring:

For a dependency-free exporter example, the runnable examples/metrics.rs installs a tiny in-process recorder, launches two self-exec children, and prints the counters and histogram summaries it captured:

cargo run --example metrics --features metrics

// Any `metrics::Recorder` works; a Prometheus exporter is shown for concreteness.
use metrics_exporter_prometheus::PrometheusBuilder;

PrometheusBuilder::new()
    .install()                       // registers the global recorder + scrape endpoint
    .expect("install Prometheus recorder");

// From here on, every processkit run feeds the counters/histograms below.

With the feature on and a recorder installed, an ordinary run needs no metrics code at all:

use processkit::Command;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Emits `processkit.spawns.total`, then on completion `processkit.runs.total`
    // + `processkit.run.duration_seconds` (+ `processkit.exit_code.total` on a
    // clean exit) — no per-call-site instrumentation.
    let _ = Command::new("git")
        .args(["rev-parse", "HEAD"])
        .output_string()
        .await?;
    Ok(())
}

What is measured

The restarts/storm_pauses counters carry no program label: a supervisor drives a runner, not one fixed command, so it has no single program name.

Secret hygiene

The label values are exactly the same secret-safe facts the tracing seam logs: a program name, a containment mechanism, an outcome class, an exit code, a teardown phase. Command argv and environment values — which routinely carry tokens, passwords, and connection strings — are never emitted, in a metric name or a label. This is a structural guarantee, not a review convention: the emission layer is handed only the already-derived program name and outcome, never the Command, so there is no code path through which argv or env could reach a series. The crate carries a test that runs a real child whose argv and environment hold sentinel secrets and asserts neither ever surfaces in an emitted name or label.

Cardinality

Labels are kept to bounded dimensions so a time-series backend is not flooded with unique series. In particular the process pid — harmless on the one-shot tracing seam but effectively unbounded as an aggregation key — is deliberately not a metric label; nor is any file path or argv fragment. The one numeric label, the exit code, is bounded in practice (a process exit code is a small integer). If your programs are themselves unbounded in number (say, a distinct binary per tenant), pre-aggregate or drop the program label in your exporter's relabeling — the same discipline you would apply to any high-cardinality dimension.

tracing

The tracing feature emits one event per lifecycle transition on the processkit target — spawn/exit (with program, pid, mechanism), timeout and cancellation firing, the per-phase graceful teardown timeline (soft_signal → grace_started → drained/escalated/spared), retry attempts, and supervisor restarts and storm pauses. It follows the same secret-hygiene rule: argv and environment values are never logged. Enable it with features = ["tracing"] and subscribe with any tracing subscriber; the two seams are independent and can be enabled together or separately.

See also

• Supervision — the restart/backoff/storm machinery the restarts/storm_pauses counters instrument.
• Timeouts, retries & cancellation — where the retries counter and the timed_out/cancelled outcomes come from.
• Process groups — the containment mechanisms behind the mechanism label and the graceful-teardown phases.

Testing your code

‹ docs index

Code that shells out is miserable to test — unless the subprocess is behind a seam. In processkit that seam is one small trait. Only output_string is required; output_bytes (raw-byte stdout) and start (a live handle for streaming/probes) are defaulted, so a minimal double implements just output_string:

#[async_trait]
pub trait ProcessRunner: Send + Sync {
    async fn output_string(&self, command: &Command) -> Result<ProcessResult<String>>;
    // Defaulted (route through `start`); override for byte/streaming support:
    async fn output_bytes(&self, command: &Command) -> Result<ProcessResult<Vec<u8>>>;
    async fn start(&self, command: &Command) -> Result<RunningProcess>;
}

Production code takes a runner (generically or as &dyn ProcessRunner); tests hand it a double. Five doubles ship with the crate, plus a macro that makes whole CLI wrappers testable for free.

• The ProcessRunner seam
• Scripting replies: ScriptedRunner
• Asserting invocations: RecordingRunner
• Echoing without spawning: DryRunRunner
• Expectation-style: MockRunner
• Record/replay cassettes: RecordReplayRunner
• Wrapping a CLI tool: CliClient
• How the crate tests itself: the stress tier

The ProcessRunner seam

JobRunner is the real implementation (each run in a fresh private group); a ProcessGroup is also a runner (runs land in that shared group); and impl ProcessRunner for &R means a borrowed runner works wherever an owned one does — inject &group or &recording without giving ownership away.

Every runner — real or double — gets the convenience helpers of ProcessRunnerExt for free: run (trimmed stdout, success required), run_unit, exit_code, probe (exit code as a boolean), checked (success-checked full result), parse/try_parse (feed stdout to a closure), and — with json — output_json (typed deserialization). These are all callable on a &dyn ProcessRunner; being generic over the closure or output type, parse/try_parse/output_json/first_line simply can't be dispatched through a dyn ProcessRunnerExt object (the ext trait isn't object-safe). Retry policies work through the seam too, so a double exercises your retry handling hermetically.

The seam covers streaming as well as bulk runs: ProcessRunner::start returns a live RunningProcess, and a ScriptedRunner's start hands back a scripted handle whose canned lines flow through the same pump machinery a real child uses — stdout_lines, wait_for_line, and finish behave identically, with no subprocess (see Scripted streaming below). An output_string-only custom runner keeps compiling: start is defaulted to ErrorReason::Unsupported.

use processkit::{Command, ProcessRunner, ProcessRunnerExt, Result};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Production code: generic over the runner.
    async fn current_branch(runner: &impl ProcessRunner) -> Result<String> {
        runner
            .run(&Command::new("git").args(["branch", "--show-current"]))
            .await
    }
    Ok(())
}

Scripting replies

ScriptedRunner returns canned Replys for matched commands — the work-horse double:

use processkit::{Command, ProcessRunnerExt};
use processkit::testing::{Reply, ScriptedRunner};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    #[tokio::test]
    async fn detects_the_branch() {
        let runner = ScriptedRunner::new()
            // Match by program + argument PREFIX (element-wise; first element is
            // the program name, in registration order):
            .on(["git", "branch", "--show-current"], Reply::ok("main\n"))
            // …or by any predicate over the full Command:
            .when(
                |cmd| cmd.working_dir().is_some(),
                Reply::fail(128, "fatal: not a git repository"),
            )
            // …with an optional catch-all:
            .fallback(Reply::ok(""));

        assert_eq!(current_branch(&runner).await.unwrap(), "main");
    }
    Ok(())
}

The pieces:

• Reply::ok(stdout) — exit 0. Reply::fail(code, stderr) — non-zero with stderr. Reply::lines(["a", "b"]) — exit 0 with the lines joined (and streamed one by one on a scripted start). Reply::timeout() — a timed-out run (the checking helpers raise ErrorReason::Timeout from it, carrying the command's own configured deadline). On a scripted start it resolves immediately as timed-out; to exercise a real deadline race, use Reply::pending() + a Command::timeout. .with_stdout(text) — attach stdout to any of them (e.g. the CONFLICT … text git prints on a failing merge). .with_line_delay(d) — pace a scripted stream's lines.
• Reply::pending() — parks the call until the command's cancellation token (per-command cancel_on or the client-level default_cancel_on) fires, resolving with ErrorReason::Cancelled — so a test can prove an orchestration actually cancels a blocked call, not just that it formats a canned error — or until the command's timeout deadline elapses, resolving timed-out (Outcome::TimedOut) on the bulk verbs and start alike, like a child killed for overrunning its deadline. Whichever fires first wins. With neither a token nor a timeout it parks forever, like a hung child.
• Rules are tried in registration order; first match wins. Prefix matching is element-wise over the program name then the arguments (the first element is the program) — on(["git", "foo"]) matches git foo bar but not git foobar (and not rm foo). Use on_sequence to serve an ordered sequence of replies (each once, then the last repeats) for a fail-then-succeed scenario.
• when has a fallible twin, try_when — the predicate returns Result<bool, E> (any E: Into<Box<dyn Error + Send + Sync>>). Ok(true) / Ok(false) match exactly as when's true / false, but an Err aborts the run: the verb fails with an ErrorReason::Predicate (kind ErrorKind::Predicate) carrying the predicate's own error verbatim, and no later rule is consulted. The ScriptedRunner counterpart of the Supervisor try_* predicate twins — for exercising a wrapper whose match callback can throw.
• No match and no fallback is a loud error (ErrorReason::Spawn, not-found) — an unexpected invocation can't slip through a test silently.
• Bulk runs also replay the canned lines through the command's on_stdout_line/on_stderr_line handlers, so a wrapper's progress-reporting path is exercised without a subprocess.

Scripted streaming

ScriptedRunner::start returns a live RunningProcess backed by the canned reply instead of an OS child. The canned stdout/stderr feed the same pump machinery a real child uses, so the whole streaming surface works hermetically — stdout_lines yields the lines, wait_for_line probes them, finish reports the canned outcome and stderr:

use processkit::prelude::StreamExt;
use processkit::testing::{Reply, ScriptedRunner};
use processkit::{Command, Finished, Outcome, ProcessRunner};
use std::time::Duration;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    #[tokio::test]
    async fn server_becomes_ready() {
        let runner = ScriptedRunner::new()
            .on(["server", "serve"], Reply::lines(["booting", "listening on 8080"]));

        let mut run = runner.start(&Command::new("server").arg("serve")).await.unwrap();
        run.wait_for_line(|l| l.contains("listening"), Duration::from_secs(5))
            .await
            .unwrap(); // satisfied by the canned banner — no subprocess

        let Finished { outcome, .. } = run.finish().await.unwrap();
        assert_eq!(outcome, Outcome::Exited(0));
    }
    Ok(())
}

Reply::lines([...]) scripts the stdout lines; .with_line_delay(d) paces them (deterministic under #[tokio::test(start_paused = true)]), and the scripted run "exits" after the last line. The honest boundaries: a scripted handle has no OS identity (pid() is None, profile reports empty samples), does not compose into a real Pipeline, and does not model interactive stdin. Reply::pending() scripts a run that never exits on its own — cancel or time it out through the command's own knobs. A command timeout does bound a scripted stream (it ends at the deadline and reports Outcome::TimedOut, like a real child), but a scripted handle has no signal tier, so — like on Windows — it ignores timeout_grace and ends at once.

Asserting invocations

RecordingRunner wraps another runner and records every Invocation — what was asked — so a test asserts inputs, not just outputs:

use processkit::{Command, ProcessRunnerExt};
use processkit::testing::{RecordingRunner, Reply, ScriptedRunner};

#[tokio::main]
async fn main() -> processkit::Result<()> {
    #[tokio::test]
    async fn passes_the_right_flags() {
        let runner = RecordingRunner::new(
            ScriptedRunner::new().fallback(Reply::ok("done")),
        );

        runner
            .run(&Command::new("gh").args(["pr", "create", "--draft"]).current_dir("/repo"))
            .await
            .unwrap();

        let call = runner.only_call(); // panics unless exactly one call
        assert_eq!(call.args_str(), ["pr", "create", "--draft"]);
        assert!(call.has_flag("--draft"));
        assert_eq!(call.cwd.as_deref().map(|c| c.to_str().unwrap()), Some("/repo"));
        assert!(!call.has_stdin);
    }
    Ok(())
}

An Invocation captures the routing knobs — program, args, cwd, envs (explicit overrides, None = removal), has_stdin — not the I/O-shaping ones (timeout, encodings, buffer policy); assert those through a when predicate over the Command itself. calls() returns the full list when more than one run is expected.

Echoing without spawning: DryRunRunner

DryRunRunner never spawns a process at all: it renders each command through Command::command_line — the crate's own display quoting, not a hand-rolled shell-escaper — and hands back a synthetic successful result. It's the seam behind a tool's own --dry-run/--echo mode: wire your production code to it (instead of JobRunner) and it shows what would run instead of running it.

use processkit::{Command, ProcessRunner};
use processkit::testing::DryRunRunner;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    #[tokio::test]
    async fn dry_run_shows_the_command_without_running_it() {
        let runner = DryRunRunner::new();
        let out = runner
            .output_string(&Command::new("rm").args(["-rf", "build"]))
            .await
            .unwrap();
        assert!(out.is_success()); // synthetic — no process ever ran
        assert_eq!(runner.only_command(), "rm -rf build");
    }
    Ok(())
}

Unlike ScriptedRunner, there is nothing to script — a dry run has no real output to fake, only a command line to show — so every call unconditionally succeeds on both output_string and start: empty stdout/stderr, and an exit code drawn from the command's own ok_codes (0 by default) so is_success() and the ergonomic run/run_unit/checked/parse verbs agree it succeeded even for a command whose ok_codes excludes 0. The rendered lines are available two ways, usable together or alone:

• a collected snapshot, in the style of RecordingRunner::calls — commands() (all of them, in order) / only_command() (panics unless exactly one call was made);
• a live on_invocation(|line| …) callback, invoked with the rendered line as each call happens — e.g. printing it to the terminal immediately, in addition to (not instead of) the collected snapshot:

#![allow(unused)]
fn main() {
use processkit::testing::DryRunRunner;

let runner = DryRunRunner::new().on_invocation(|line| println!("+ {line}"));
}

Expectation-style: MockRunner

With the mock feature, mockall generates a MockRunner for expectation-style tests (call counts, argument matchers, ordered expectations) — the right tool when the interaction is the contract.

Note: MockRunner's expect_* surface is generated by mockall and is exempt from this crate's semver guarantees — it tracks the mockall dependency, not a frozen API. For a stable double, prefer ScriptedRunner (canned replies) or RecordingRunner (input assertions) above.

#![allow(unused)]
fn main() {
use processkit::testing::MockRunner;

let mut mock = MockRunner::new();
mock.expect_output_string()
    .times(1)
    .returning(|_cmd| todo!("build a Result<ProcessResult<String>>"));
}

MockRunner does not inherit the defaults. Unlike a hand-written runner (where output_bytes/start are defaulted), mockall::automock replaces every method with an expectation — so a verb that routes through start or output_bytes needs its own expect_start() / expect_output_bytes(), or the unset call panics ("no expectation"). ScriptedRunner provides the defaults and the streaming seam out of the box.

For most tests ScriptedRunner/RecordingRunner read better; reach for the mock when you need mockall's matching machinery.

Record/replay cassettes

With the record feature, RecordReplayRunner closes the loop: record real runs to a JSON cassette once, then replay them deterministically — fast, hermetic, byte-stable, no subprocess in CI:

The runnable examples/record_replay.rs does the complete round trip with a self-exec child and a temporary cassette, including a scrub hook and a guard proving replay never spawns:

cargo run --example record_replay --features record

use processkit::{Command, JobRunner, ProcessRunnerExt};
use processkit::testing::RecordReplayRunner;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Record once against the real tool (an opt-in `--record` test run, say):
    let runner = RecordReplayRunner::record("fixtures/git.json", JobRunner::new());
    let version = runner.run(&Command::new("git").arg("--version")).await?;
    runner.save()?;                                  // the error-surfacing flush
                                                     // (best-effort on drop too)

    // Replay everywhere else:
    let runner = RecordReplayRunner::replay("fixtures/git.json")?;
    assert_eq!(runner.run(&Command::new("git").arg("--version")).await?, version);
    Ok(())
}

Semantics worth knowing before you commit a cassette:

By default only env values are excluded. program, args, cwd, stdout, and stderr are stored verbatim and can carry secrets (a --password=… flag, a token echoed to output). For a fixture that can contain one, opt into a field-aware scrub hook:

#![allow(unused)]
fn main() {
use processkit::{JobRunner, ProcessRunnerExt};
use processkit::testing::{CassetteField, RecordReplayRunner};

let scrub = |field: CassetteField, text: &str| match field {
    CassetteField::Argument | CassetteField::Stdout | CassetteField::Stderr =>
        text.replace("s3cret", "<redacted>"),
    CassetteField::Cwd => text.replace("/home/alice", "<workspace>"),
    _ => text.to_owned(),
};

let recorder = RecordReplayRunner::record("fixtures/tool.json", JobRunner::new())
    .scrub_with(scrub);
// run commands, then recorder.save()?;

// Apply the same deterministic hook before replay lookup: a live secret argv
// is transformed to the same redacted key stored in the fixture.
let replayer = RecordReplayRunner::replay("fixtures/tool.json")?
    .scrub_with(scrub);
let _ = (recorder, replayer);
Ok::<(), processkit::Error>(())
}

The hook changes only the persisted entry. Record-mode callers still receive the real unsanitized stdout/stderr; replay naturally returns the scrubbed text that exists in the fixture. Use the same deterministic hook on both runners. Still review a fixture before committing it. On Unix the file is written 0600 and the write refuses to follow a symlink at the cassette path (O_NOFOLLOW, so a planted link can't redirect the secret-bearing write — it fails loud instead). On Windows the file inherits the containing directory's ACL, so restrict that directory (or use a per-user temp dir, not a world-writable shared one) for secret-bearing fixtures.

A neat trick: in tests, record against a ScriptedRunner instead of JobRunner — the whole record→save→replay round trip is then itself hermetic.

Opt-in stricter matching (cwd / selected env values)

The portable default keys on program + args + stdin only. It deliberately leaves cwd and the environment out of the key: a cassette recorded in one absolute working directory (a dev box, a tempdir) then replays cleanly from a different one (a CI workspace), and an env variable that differs between the record and replay machines but doesn't change the tool's output can't cause a spurious miss. That portability is the right default for most tools.

But some tools' output genuinely depends on where they run or on a specific environment variable — and with those out of the key, two such invocations collide on one entry: the first recording silently answers for both on replay. When that matters, opt in to a stricter key:

use processkit::{Command, JobRunner, ProcessRunnerExt};
use processkit::testing::RecordReplayRunner;

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Record: also key on the working directory and on LC_ALL's value.
    let runner = RecordReplayRunner::record("fixtures/tool.json", JobRunner::new())
        .match_on_cwd()
        .match_on_env(["LC_ALL"]);
    let cmd = Command::new("tool").current_dir("/repo").env("LC_ALL", "C");
    let _ = runner.run(&cmd).await?;
    runner.save()?;

    // Replay: set the *same* policy — it must match on both sides.
    let runner = RecordReplayRunner::replay("fixtures/tool.json")?
        .match_on_cwd()
        .match_on_env(["LC_ALL"]);
    // A differing cwd or LC_ALL value now MISSES (a loud `CassetteMiss`) instead
    // of replaying the wrong entry; the same cwd + LC_ALL hits.
    let cmd = Command::new("tool").current_dir("/repo").env("LC_ALL", "C");
    let _ = runner.run(&cmd).await?;
    Ok(())
}

Notes:

• Env values still never reach the file. match_on_env keys on an FNV digest of the selected (name, value) pairs, not the raw values — the cassette continues to store variable names only, so no env secret is written to disk even under the stricter policy. (cwd is keyed the same way; it is also already stored verbatim on the entry, as before.)
• Symmetric by contract. Set the same policy on the record and replay runner, exactly as you target the same tool. A mismatched policy simply misses (it never serves a wrong entry) — including replaying a policy-keyed cassette with no policy, or vice versa.
• Only the named variables participate. Variables you don't name stay out of the key, preserving portability for env differences that don't matter. A named variable that is set, removed (env_remove), or untouched are three distinct keys.
• On-disk format. A policy-keyed entry carries an extra opaque match_digest number; a cassette recorded without a policy omits it and is byte-identical to before but for the bumped version (now 4). Older cassettes (no match_digest) load and replay unchanged under a no-policy replayer.

Wrapping a CLI tool

CliClient is the foundation for typed wrappers around external tools (git, jj, gh, kubectl, …): it owns the program name, per-client defaults, and the runner; your wrapper contributes only commands and parsers. The cli_client! macro generates the boilerplate:

For the full design guide — default precedence, dynamic env resolution, retry/error policy, typed JSON, and worked wrappers — see Building typed CLI clients. This section focuses on the test seam.

use processkit::{cli_client, Error, ProcessRunner, Result};
use std::path::Path;
use std::time::Duration;

cli_client!(
    /// A typed `git` client.
    pub struct Git => "git"
);

impl<R: ProcessRunner> Git<R> {
    /// HEAD's commit id.
    pub async fn head(&self, repo: &Path) -> Result<String> {
        self.core.run(self.core.command_in(repo, ["rev-parse", "HEAD"])).await
    }

    /// Is the work tree clean? (exit code IS the answer)
    pub async fn is_clean(&self, repo: &Path) -> Result<bool> {
        self.core.probe(self.core.command_in(repo, ["diff", "--quiet"])).await
    }

    /// Branch list, parsed — the parser is fallible and returns the crate's
    /// `Result`, typically an `ErrorReason::Parse` naming the program.
    pub async fn branches(&self, repo: &Path) -> Result<Vec<String>> {
        self.core
            .try_parse(
                self.core.command_in(repo, ["branch", "--format=%(refname:short)"]),
                |out| {
                    let list: Vec<String> = out.lines().map(str::to_owned).collect();
                    if list.is_empty() {
                        Err(Error::parse("git", "no branches"))
                    } else {
                        Ok(list)
                    }
                },
            )
            .await
    }
}

#[tokio::main]
async fn main() -> processkit::Result<()> {
    // Production: the real runner, with per-client defaults.
    let git = Git::new().default_timeout(Duration::from_secs(30));
    let head = git.head(Path::new(".")).await?;
    Ok(())
}

The generated type is Git<R: ProcessRunner = JobRunner> with Git::new(), Git::with_runner(runner), default_timeout / default_env / default_env_remove builders, and a module-private core: CliClient<R> (reach it as self.core from the wrapper's own methods) whose helpers speak the crate-wide verb vocabulary: run (trimmed stdout), output_string (full result), run_unit (success only), exit_code, probe, plus parse (infallible) and try_parse (fallible → ErrorReason::Parse).

And the payoff — the wrapper tests hermetically with any double:

#![allow(unused)]
fn main() {
#[tokio::test]
async fn head_is_trimmed() {
    let git = Git::with_runner(
        ScriptedRunner::new().on(["git", "rev-parse", "HEAD"], Reply::ok("abc123\n")),
    );
    assert_eq!(git.head(Path::new("/repo")).await.unwrap(), "abc123");
}
}

…or with a cassette recorded against the real tool once.

How the crate tests itself: the stress tier

The doubles above are for testing your code. The crate's own suite has an extra, non-functional layer you'll only meet if you hack on processkit itself: an opt-in stress tier (tests/stress/) that exercises behaviour at scale and under multiplicity — spawn bursts, cancel storms, mass teardown, and buffer floods. It is gated behind the PROCESSKIT_STRESS env var (so the normal PR matrix compiles it for free but pays nothing) and runs on its own scheduled / on-demand CI leg (.github/workflows/stress.yml), never in the fast PR run.

Its most searching scenario is the seeded randomized-interleaving harness (tests/stress/interleave.rs). A seed deterministically generates a random combination of the public lifecycle operations — wait / start_kill / take_stdin / the streaming verbs / group signal / suspend / shutdown — over a shared ProcessGroup and children of three shapes (short-lived, long-lived, stdout-flooding), then runs them concurrently so real thread scheduling explores the orderings. After every combination it checks the invariants that must hold under any interleaving:

• no operation panics or returns an impossible Error (only the documented, operational variants);
• no live descendants or zombies survive (reusing the tier's reap helpers);
• no background pump/drain/supervision task faulted unnoticed;
• every descriptor and group is released (Job handle closed, cgroup directory removed).

The seed fixes the plan, not the scheduling: a re-run with the same seed replays the identical sequence of operations, so a failure is reproducible. The offending seed is printed on failure — re-run just that plan in a loop with PROCESSKIT_STRESS_SEED=<seed> while you debug. Run the tier locally with:

PROCESSKIT_STRESS=1 cargo test --all-features --test stress -- --test-threads=1

Next: Platform support · Supervision · Running commands

Platform support

‹ docs index

processkit supports Unix and Windows only — it requires tokio::process and OS job / process-group primitives that have no equivalent on bare targets like wasm. Building for such a target fails at compile time (a compile_error! guard, or earlier in tokio's own dependencies). Within the supported set, it treats platform support as first-class: every capability is either fully implemented, honestly partial (documented and typed), or refused with ErrorReason::Unsupported — never silently skipped. This page collects all the matrices and fine print in one place.

• CI coverage
• Containment mechanisms
• Capability matrices
• Caveats

CI coverage

.github/workflows/ci.yml's test job runs the full real-subprocess suite (--include-ignored, so kill-on-drop is actually exercised) on glibc x86_64 (ubuntu-latest), glibc aarch64 (ubuntu-24.04-arm), Windows x64 (windows-latest), Windows ARM64 (windows-11-arm), and macOS. The aarch64 leg is the only place the native Linux syscall/layout code (sys/{linux,pgroup,unix,pid_gate}.rs) actually runs on Linux/glibc-aarch64 — elsewhere aarch64 is only cargo check-compiled against aarch64-apple-darwin (Darwin, in the msrv job), never executed. A separate test-musl job runs the full suite a further time inside a real rust:alpine container — musl libc and a busybox userland, not merely a cross-compiled x86_64-unknown-linux-musl binary executed by glibc userland tools — because musl/Alpine is the de-facto standard for the container images this crate actually runs in, and its libc, signal, and userland-utility details genuinely differ from glibc's. Alpine's busybox already covers every external utility the suite spawns (sh, cat, sleep, yes, head, grep, sort, id, printf, env, seq) except one: its ps applet has no -p PID filter, so the job installs procps (procps-ng) to get one that does. gcc/musl-dev (needed to link) already ship in the base image.

Both Windows architectures run the same three feature configurations for clippy and the same real-subprocess test suite. The ARM64 leg therefore executes the crate's most platform-specific surface — Job Objects, CTRL delivery, per-thread suspend/resume, and ConPTY — under the ARM64 Windows ABI instead of only proving that x64 code compiles.

FreeBSD has two explicit tiers rather than inheriting confidence from macOS: check-freebsd cross-compiles the library and binaries for x86_64-unknown-freebsd, while test-freebsd boots a real FreeBSD 14.4 VM. The VM runs the hermetic library suite plus representative real-subprocess checks for mechanism selection, kill-on-drop, Unix signal delivery, graceful TERM shutdown, and — since FreeBSD has a containment backend of its own — the procctl reaper's resistance to a setsid escapee. This directly exercises the FreeBSD reaper backend and the process-group machinery underneath it against FreeBSD's kernel and userland without making the already broad full-suite matrix depend on an emulated VM. It is also the only place that backend runs at all: the procctl code cannot execute on any other host, so a change to it that passes check-freebsd is compiled, not tested, until this job says otherwise.

Two container-runtime quirks — unrelated to musl/Alpine itself, but specific to running any test suite inside a plain container — need working around, via the job's (and the just test-musl recipe's) --init and --cap-add=SYS_NICE options:

• No subreaper. A plain container's PID 1 is just the job's own entry process, not a real init — so a killed process's orphaned grandchildren become zombies that are never reaped and still probe alive via kill(pid, 0). That silently breaks every test that asserts a forked grandchild is actually gone after teardown. --init runs tini as PID 1, a real subreaper, which is what a properly configured container host provides. Any container's own PID 1 has the identical gap for its own orphaned grandchildren — see Running in containers → PID 1.
• CAP_SYS_NICE dropped by default. Docker excludes it from the default capability set even for a root container user, so raising scheduling priority fails EPERM regardless of uid (lowering it never needs the capability). One test exercises Priority::High (a negative nice) together with a privilege drop; --cap-add=SYS_NICE restores just that one narrow, low-blast-radius capability rather than skipping the test.

Run the same job locally with just test-musl (requires Docker); see the recipe in justfile for details.

Containment mechanisms

ProcessGroup::mechanism() reports which one you actually got:

FreeBSD is not "macOS with a different name". It is the only non-Linux unix with a real whole-tree containment primitive, so it gets its own mechanism rather than the process-group fallback the other BSDs use. Three consequences are worth knowing up front:

• A setsid escapee stays contained. The classic escape — a descendant that starts its own session, including the daemonising double-fork — leaves the process group but not the reaper's subtree. members(), signal, shutdown and kill-on-drop all still reach it.
• members() is the whole tree, one entry per process, not one per contained child (see the inspection matrix).
• Orphans re-parent to this process, not to init. That is inherent to being the reaper, and this crate takes on the duty that comes with it: a re-parented descendant that exits is waited for by the crate itself, so it does not accumulate as a zombie. Children this process forked itself are never touched by that sweep — their exit status belongs to whoever spawned them, processkit or not. Reaper status is acquired lazily (on the first ProcessGroup), never released while the process lives, and is shared harmlessly with an application that acquired it first.

To learn which mechanism you would get without creating a group — for a spawn-free preflight / host-check that must have no side effects — call host_containment(), which returns a HostContainment (mechanism(), plus soft_stop_scope(), parent_death_cleanup(), and crate_version()) from read-only checks that create no container and spawn no process (on Linux: no cgroup directory). The predicted mechanism matches a real ProcessGroup::new on the same host; two answers are best-effort, both because the query must create nothing. The Linux cgroup answer probes whether a cgroup could be created rather than creating one, and the FreeBSD answer reports ProcessReaper without acquiring reaper status (acquiring it is a real, permanent side effect on the process — available unprivileged on every supported kernel, so the prediction holds in practice). Either way a live ProcessGroup::mechanism() remains the final word. See Running in containers → Which containment mechanism you get.

On Linux the cgroup backend requires controller delegation, and resource limits specifically need this process to run at the real cgroup-v2 root. The crate creates the limit cgroup under this process's own cgroup and enables the controllers in that cgroup's subtree_control, which cgroup v2's "no internal processes" rule allows only for the real hierarchy root (the one exempt cgroup). A cgroup namespace root does not qualify — it only virtualizes the view — so an ordinary (private-cgroupns) container fails EBUSY just like a systemd session/scope/service. The crate does not migrate your process into a sub-cgroup to work around it, so in practice limits apply only at a minimal non-systemd init sitting at the real root. Without a usable cgroup it quietly falls back to ProcessGroup — unless you requested resource limits, which fail fast instead (ErrorReason::ResourceLimit), because an unapplied cap is no protection. The error's reason distinguishes the two ways this happens: LimitReason::Unsupported when no cgroup v2 is mounted at all (or on macOS/BSD, which has no whole-tree container of any kind), LimitReason::Unenforceable when cgroup v2 exists but this process isn't at the real hierarchy root (the delegation case above) or the OS otherwise rejected the request.

Capability matrices

Teardown & containment

Adopting by pid is not neutral for containment the process already has. The two ⚠️ cells above point opposite ways, and neither is undone when the group is dropped:

• Windows — a process may belong to several nested jobs (Windows 8+), so a target already in an orchestrator's or CI agent's job is not refused for that reason alone; the assign that succeeds makes this crate's job a child of that outer job. Membership of a child job is membership of every job above it, so from then on the outer job's terminate/close reaches this group's members — including ones started after the adoption — and the outer job's limits bound them. Whether the assign succeeds also depends on this group's own state: observed on Windows 11, an empty group takes such a process in, while a group that already holds a member outside that outer job's hierarchy is refused ERROR_ACCESS_DENIED. Adopt first, then start.
• Linux cgroup v2 — membership is exclusive, so the write that moves the process into this group's cgroup takes it out of the cgroup it was in: the teardown and limits of whoever contained it before stop applying. The kernel does not report what a task left behind, so nothing restores it.
• Process-group backends — nothing is taken away: containment there is tracking, not moving. The setpgid the call attempts is permitted only for a not-yet-exec'd child of this process; where it does apply, the process becomes a group leader of its own.

Windows has no POSIX signal tier, so for a windowless child with no opt-in a graceful shutdown collapses to the atomic Job kill — but it still honors escalate_to_kill: false spares the survivors (closes the Job handle without KILL_ON_JOB_CLOSE) rather than killing them, so the Windows column is "atomic kill when it kills", not an unconditional kill. That promise covers what this group does; it cannot bind a job above it. A group that adopted a process already belonging to another Job Object is nested under that job (see the adopt_external note above), and the outer job's own terminate or close still reaches these survivors.

Automatic soft tier for windowed children (Windows). Before the atomic kill, a graceful shutdown posts WM_CLOSE to every top-level window owned by a live member and then drives the same signal → wait → escalate loop the unix backends use. A windowed child (Electron app, desktop tool, windowed service) that handles WM_CLOSE can flush and exit within the grace; any survivor is then TerminateJobObject'd, the same hard fallback as before. This is automatic — no opt-in — and WM_CLOSE is posted, never sent, so a hung window can never block teardown. A windowless tree with no console opt-in still hard-kills promptly at the deadline (no grace wait is introduced for it), so its timings are unchanged.

Opt-in soft tier for console children (Windows). Command::windows_graceful_ctrl_break() gives Windows a real soft-shutdown trigger: the direct child is spawned in its own console process group (CREATE_NEW_PROCESS_GROUP), and at graceful teardown it is sent GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, pid) before the grace window — driven through the same signal → wait → escalate loop the unix backends use. A console child that handles CTRL_BREAK (many CLIs, Node, Python, Go services do) can flush and exit within the grace; any survivor is then TerminateJobObject'd, the same hard fallback as before, so containment is never weakened. Boundaries: it works only for children that share this process's console — a child spawned create_no_window / DETACHED_PROCESS (or a GUI / service parent with no console) never receives the event and simply rides the grace to the fallback kill; the event is CTRL_BREAK (not CTRL_C, which a new process group disables); and only the direct child is addressed (an adopted child is not). Off Windows the builder is a no-op — the graceful ladder already sends a real signal.

Signals & freezing

On every Unix mechanism, a signal broadcast surfaces a real send failure as an Err rather than swallowing it — an EINVAL (an out-of-range Other(n)) and an EPERM against a live, non-zombie member (a uid-changed child, or a seccomp/container restriction) — consistent with the "never silently skipped" philosophy. The process-group backend (macOS/BSD, Linux fallback) matches the cgroup verdict by checking the target's run state after an EPERM, so a harmless zombie-only EPERM — and every EPERM on the bare BSDs (no state reader) — stays swallowed; the FreeBSD reaper makes the same discrimination from the kernel's own zombie flag on the failing member, which PROC_REAP_KILL names explicitly; an ESRCH race (the member already exited) is still success, and Signal::Other(0) returns Ok having delivered nothing (the POSIX existence probe). That probe never takes a delivery path — FreeBSD routes it back through the process group, which has no state reader on any BSD but macOS — so the EPERM rule above does not extend to it there: on FreeBSD and the bare BSDs a live target that rejects even the null signal is still an Ok, where Linux and macOS surface it. suspend/resume on that process-group mechanism now apply the same honest verdict to SIGSTOP/SIGCONT: a live-member EPERM is an Err, while ESRCH, zombie-only EPERM, an empty group, and BSD-without-state-reader EPERM remain Ok. The FreeBSD reaper freezes and thaws with the same honesty — its SIGSTOP/SIGCONT go out through the very PROC_REAP_KILL classification described above, so a live member's EPERM (and an EINVAL/ECAPMODE that means the request never ran) surfaces, while a drained subtree and a zombie-only refusal stay Ok. On the cgroup mechanism the SIGSTOP/SIGCONT fallback used on pre-5.2 kernels (no cgroup.freeze) surfaces failures the same way.

soft_stop_scope() answers, before you attempt a soft Int/Term, which members it would reach — a side-effect-free SoftStopScope capability report read from the group's live membership, so a caller cancelling on its own schedule can decide up front instead of firing a signal and parsing an ErrorReason::Unsupported back. The Unix backends always reach the whole tree (WholeTree, never Unsupported); Windows reports OptInMembers when a live console-CTRL leader (windows_graceful_ctrl_break) or a windowed member exists, and Unsupported otherwise — exactly the split where signal(Int/Term) returns Ok versus ErrorReason::Unsupported. It is the group-axis sibling of kill_on_parent_death_scope() below, but read at runtime rather than fixed per platform. Gated on the process-control feature, like signal.

Inspection & accounting

members() is gated on the process-control feature; the CPU / memory / I/O / profile rows are gated on the stats feature.

The two whole-tree counter rows read a counter the container keeps, so they are cumulative — a member that already exited is still in the number — where the group CPU / peak-memory row is a per-live-member sum on the Linux cgroup backend. They are also not directly comparable across platforms: a Job Object counts bytes moved against any target (file, pipe, device), a cgroup's io.stat only what reached the block layer. The 🟡 cells are honest None on a host that has not enabled the controller for the group's cgroup — processkit enables only the controllers a requested resource cap needs (memory/pids/cpu), never io. The per-run profile row deliberately does not gain these: they are group-level facts, and RunProfile says so rather than reporting a shared group's tree under a per-run name.