Timeouts, retries & cancellation

‹ docs index

Three ways a run can end early, with three different philosophies:

• a timeout is data — the deadline was part of the run's contract, so its expiry is captured in the result, and only the success-checking verbs turn it into an error;
• a retry is a policy — the verbs replay the run while your classifier says the failure is worth another attempt;
• a cancellation is an abandonment — the caller changed its mind, so every path reports an error and there is no result worth inspecting.

The samples below run inside a task { } block and use match! / let!; from C# the same surface is await-able fluent methods. Every builder method has a pipe-friendly Command.* mirror (Command.timeout, Command.retry, Command.cancelOn), shown alongside the fluent form.

• Timeouts
• Graceful timeout
• Idle timeout
• Captured vs raised: the decision table
• Retries
• Cancellation
• Pipelines and clients
• Precedence and interactions

Timeouts

Command.Timeout(duration) (mirror: Command.timeout) kills the whole process tree at the deadline — not just the direct child, so a wrapper script's grandchildren die too. The run's Outcome becomes Outcome.TimedOut.

F#

task {
    // Captured: a non-zero exit / timeout is data on the capture verbs.
    let cmd =
        Command.create "slow-tool"
        |> Command.timeout (TimeSpan.FromSeconds 5.0)

    match! cmd.OutputStringAsync() with
    | Ok result when result.IsTimedOut ->
        // Code is None on a timeout; the partial output captured before the kill is kept.
        printfn $"timed out; partial stdout before the kill: {result.Stdout}"
    | Ok result -> printfn $"exited {result.Code}: {result.Stdout}"
    | Error err -> eprintfn $"{err.Message}"
}

C#

// Captured: a non-zero exit / timeout is data on the capture verbs.
var cmd = new Command("slow-tool")
    .Timeout(TimeSpan.FromSeconds(5));

Console.WriteLine(await cmd.OutputStringAsync() switch
{
    // Code is None on a timeout; the partial output captured before the kill is kept.
    { IsOk: true, ResultValue: { IsTimedOut: true } result } => $"timed out; partial stdout before the kill: {result.Stdout}",
    { IsOk: true, ResultValue: var result }                  => $"exited {result.Code}: {result.Stdout}",
    { IsOk: false, ErrorValue: var err }                    => err.Message,
});

The same command finished with a success-checking verb raises the deadline as a typed error instead:

F#

task {
    let cmd =
        Command.create "slow-tool"
        |> Command.timeout (TimeSpan.FromSeconds 5.0)

    match! cmd.RunAsync() with
    | Ok stdout -> printfn $"{stdout}"
    | Error(ProcessError.Timeout(program, timeout, _, _)) ->
        eprintfn $"{program} exceeded {timeout}"
    | Error err -> eprintfn $"{err.Message}"
}

C#

var cmd = new Command("slow-tool")
    .Timeout(TimeSpan.FromSeconds(5));

Console.WriteLine(await cmd.RunAsync() switch
{
    { IsOk: true, ResultValue: var stdout } => stdout,
    { IsOk: false, ErrorValue: ProcessError.Timeout { Program: var p, Timeout: var t } } => $"{p} exceeded {t}",
    { IsOk: false, ErrorValue: var err } => err.Message,
});

ProcessError.Timeout(program, timeout, stdout, stderr) carries the partial stdout/stderr captured before the kill — a hung tool's last words are still available on the error, not discarded.

Two distinct deadline families — keep them apart. Command.Timeout is the run's own contract (this guide): it kills the tree. The readiness probes' within parameter (WaitForLineAsync / WaitForPortAsync / WaitForAsync, see streaming.md) is a different deadline: it gives ProcessError.NotReady and never kills the child — the caller decides what happens next.

Graceful timeout

By default the deadline hard-kills the tree at once. Add Command.TimeoutGrace(grace) (mirror: Command.timeoutGrace) to give the tree a chance to clean up: at the deadline it is sent SIGTERM, allowed up to the grace window to exit, then SIGKILLed — the same SIGTERM → wait → SIGKILL tier as ProcessGroup.ShutdownAsync. A signal-handling child that exits ends the grace early.

F#

task {
    let cmd =
        Command.create "slow-tool"
        |> Command.timeout (TimeSpan.FromSeconds 30.0)
        |> Command.timeoutGrace (TimeSpan.FromSeconds 5.0) // SIGTERM, wait up to 5s, then SIGKILL

    let! _ = cmd.OutputStringAsync()
    ()
}

C#

var cmd = new Command("slow-tool")
    .Timeout(TimeSpan.FromSeconds(30))
    .TimeoutGrace(TimeSpan.FromSeconds(5)); // SIGTERM, wait up to 5s, then SIGKILL

await cmd.OutputStringAsync();

IsTimedOut is true regardless of whether the child exited on the signal or was SIGKILLed after the grace — the deadline is what fired. On Windows there is no signal tier: TimeoutGrace is accepted but the deadline kills the Job Object atomically, so the grace window has no effect there.

Idle timeout

Command.Timeout bounds the total run length. The other common failure is a run that is still alive but stuck — it has stopped producing output. Command.IdleTimeout(duration) (mirror: Command.idleTimeout) catches exactly that: it kills the tree when neither stdout nor stderr produces output for duration. Every chunk of output resets the deadline, so a run that keeps streaming stays alive; one that goes quiet is killed.

F#

task {
    // Kill the build if it stops printing for 30s, however long it runs overall.
    let cmd =
        Command.create "long-build"
        |> Command.idleTimeout (TimeSpan.FromSeconds 30.0)

    match! cmd.OutputStringAsync() with
    | Ok result when result.IsTimedOut -> eprintfn "stalled — no output for 30s"
    | Ok result -> printfn $"exited {result.Code}"
    | Error err -> eprintfn $"{err.Message}"
}

C#

// Kill the build if it stops printing for 30s, however long it runs overall.
var cmd = new Command("long-build")
    .IdleTimeout(TimeSpan.FromSeconds(30));

Console.WriteLine(await cmd.OutputStringAsync() switch
{
    { IsOk: true, ResultValue: { IsTimedOut: true } } => "stalled — no output for 30s",
    { IsOk: true, ResultValue: var result }           => $"exited {result.Code}",
    { IsOk: false, ErrorValue: var err }              => err.Message,
});

Key facts:

• Same honest result as Timeout. An idle kill surfaces as Outcome.TimedOut — so IsTimedOut on the capture verbs and ProcessError.Timeout on the success-checking verbs, exactly like the total timeout. At the API level the two are not distinguished (both mean "killed on a deadline"); logs tell them apart (the message names an idle kill and reports the idle window, under the same ProcessTimedOut event id).
• Byte granularity, every verb. Activity is any output read from the child, measured in bytes — so a single long line without a newline still counts as active, and it works uniformly for the buffered capture verbs, the streaming verbs, the raw OutputBytesAsync, and even the output-discarding WaitAsync/ProfileAsync. It is independent of StdoutLineCount/StderrLineCount, which stay pure line counters.
• The idle clock starts when consumption begins (the verb's exit wait), not at some earlier construction, so a handle you drive later is not killed for a gap before you started reading.
• Independent of Timeout. Set both — each fires on its own condition, whichever comes first, with a single kill and a single reported outcome (no double kill). IdleTimeout honours TimeoutGrace (SIGTERM → grace → SIGKILL) exactly as Timeout does.
• A negative duration is rejected (ArgumentOutOfRangeException); one larger than ~24.8 days is treated as no idle deadline (as with Timeout).

Like Command.Timeout, a per-stage Command.IdleTimeout cannot bound one stage of a pipeline — a pipeline captures only the last stage's output and does not monitor per-stage activity — so .Pipe rejects it with an ArgumentException rather than silently ignoring it (see Pipelines and clients).

Captured vs raised: the decision table

The same timeout lands differently depending on the verb you finish with. The capture verbs treat the deadline as data; the success-checking verbs raise it.

Streaming makes the "captured" half concrete — the deadline bounds the stream, and the outcome is readable afterwards:

F#

task {
    let cmd =
        Command.create "chatty-job"
        |> Command.timeout (TimeSpan.FromSeconds 10.0)

    match! cmd.StartAsync() with
    | Ok proc ->
        use _ = proc
        let e = proc.StdoutLinesAsync().GetAsyncEnumerator()

        try
            let mutable go = true

            while go do
                match! e.MoveNextAsync() with
                | true -> printfn $"> {e.Current}"
                | false -> go <- false // the stream ends when the deadline kills the tree
        finally
            e.DisposeAsync().AsTask().Wait()

        match! proc.FinishAsync() with
        | Ok finished when finished.Outcome.IsTimedOut -> eprintfn "killed at the deadline"
        | Ok _ -> ()
        | Error err -> eprintfn $"{err.Message}"
    | Error err -> eprintfn $"{err.Message}"
}

C#

var cmd = new Command("chatty-job")
    .Timeout(TimeSpan.FromSeconds(10));

await using var proc = (await cmd.StartAsync()).GetValueOrThrow();

await foreach (var line in proc.StdoutLinesAsync())
    Console.WriteLine($"> {line}"); // the stream ends when the deadline kills the tree

var finished = await proc.FinishAsync();
if (finished is { IsOk: true, ResultValue: { Outcome.IsTimedOut: true } })
    Console.Error.WriteLine("killed at the deadline");
else if (finished is { IsOk: false, ErrorValue: var err })
    Console.Error.WriteLine(err.Message);

Retries

Command.Retry(maxAttempts, delay, predicate) (mirror: Command.retry) replays a failed run, sleeping delay between tries, retrying only while predicate accepts the error. The predicate is a Func<ProcessError, bool> (from F#, a plain ProcessError -> bool through the module mirror).

maxAttempts is the total number of runs (the first run plus up to maxAttempts - 1 retries), so Retry 3 runs the command at most three times, and 0/1 both mean a single run — a command always runs at least once.

F#

task {
    let cmd =
        Command.create "curl"
        |> Command.args [ "-fsS"; "https://example.com/api" ]
        |> Command.timeout (TimeSpan.FromSeconds 10.0)
        |> Command.retry
            3
            (TimeSpan.FromMilliseconds 250.0)
            (fun err ->
                // transient (spawn/I/O), a timeout, or curl's "couldn't connect" (exit 7)
                ProcessError.isTransient err
                || err.IsTimeout
                || (match err with
                    | ProcessError.Exit(_, 7, _, _) -> true
                    | _ -> false))

    match! cmd.RunAsync() with
    | Ok body -> printfn $"{body}"
    | Error err -> eprintfn $"gave up: {err.Message}"
}

C#

var cmd = new Command("curl")
    .Args(["-fsS", "https://example.com/api"])
    .Timeout(TimeSpan.FromSeconds(10))
    .Retry(
        3,
        TimeSpan.FromMilliseconds(250),
        err =>
            // transient (spawn/I/O), a timeout, or curl's "couldn't connect" (exit 7)
            err.IsTransient
            || err.IsTimeout
            || err is ProcessError.Exit { Code: 7 });

Console.WriteLine(await cmd.RunAsync() switch
{
    { IsOk: true, ResultValue: var body } => body,
    { IsOk: false, ErrorValue: var err } => $"gave up: {err.Message}",
});

The two built-in classifiers are ready to drop in as predicates:

• ProcessError.isTransient (from C#, err.IsTransient) — true for Spawn and Io errors (spawn races, transient I/O blips) that may succeed on another try.
• ProcessError.isNotFound (from C#, the generated err.IsNotFound tester) — true for a program-not-found failure (usually a reason to install-then-retry rather than to blindly replay).

F#

let cmd =
    Command.create "flaky-tool"
    |> Command.retry 5 (TimeSpan.FromMilliseconds 200.0) ProcessError.isTransient

C#

var cmd = new Command("flaky-tool")
    .Retry(5, TimeSpan.FromMilliseconds(200), err => err.IsTransient);

Where retry earns its keep. Retry replays the run whenever a verb yields an Error your predicate accepts. The success-checking verbs (RunAsync / RunUnitAsync / ExitCodeAsync / ProbeAsync / ParseAsync / TryParseAsync) are where that matters: they turn a non-zero exit into ProcessError.Exit and a timeout into ProcessError.Timeout, so your classifier can act on the outcome of the run. The capture verbs (OutputStringAsync / OutputBytesAsync) keep a non-zero exit and a timeout as data — an Ok result — so a retry there can only ever fire on a genuine failure-to-run (a transient spawn or I/O error), never on an exit code or a deadline.

Two ground rules:

• The classifier sees the typed ProcessError — match on the case, on an exit code, even on the captured stderr.
• A ProcessError.Cancelled is effectively terminal: the built-in classifiers reject it, and once the run's token is cancelled the retry loop stops re-trying regardless — another attempt could only fail the same way.

For "keep a service alive whenever it exits" rather than "replay this one operation", reach for a supervision.md Supervisor — the same backoff shape, a different loop condition.

Cancellation

Hand any verb a System.Threading.CancellationToken; cancelling the token kills the run's tree and makes every consuming path report ProcessError.Cancelled. Every verb takes an optional CancellationToken (cmd.RunAsync(token), cmd.OutputStringAsync(token), …):

F#

task {
    use cts = new CancellationTokenSource()
    let job = (Command.create "long-export").RunAsync(cts.Token)

    // elsewhere — a shutdown signal, a sibling failure, a UI button:
    cts.Cancel()

    match! job with
    | Error(ProcessError.Cancelled program) -> printfn $"{program} cancelled"
    | _ -> ()
}

C#

using var cts = new CancellationTokenSource();
var job = new Command("long-export").RunAsync(cts.Token);

// elsewhere — a shutdown signal, a sibling failure, a UI button:
cts.Cancel();

if (await job is { IsOk: false, ErrorValue: ProcessError.Cancelled { Program: var p } })
    Console.WriteLine($"{p} cancelled");

Or tie a token to a command for its whole lifetime with Command.CancelOn(token) (mirror: Command.cancelOn) — it is linked in addition to any per-verb token, so either source cancels the run:

F#

let cmd = Command.create "long-export" |> Command.cancelOn shutdownToken
let! _ = cmd.RunAsync() // also cancels if shutdownToken fires

C#

var cmd = new Command("long-export").CancelOn(shutdownToken);
await cmd.RunAsync(); // also cancels if shutdownToken fires

The contract, path by path:

Unlike a timeout — whose expiry is captured as IsTimedOut — a cancellation is always an error: the run was abandoned, so there is no result to synthesize. A token cancelled before the run starts short-circuits without spawning anything.

Pipelines and clients

A whole pipeline has its own deadline and token, bounding the entire chain:

F#

task {
    let pipeline =
        (Command.create "producer")
            .Pipe(Command.create "consumer")
            .Timeout(TimeSpan.FromSeconds 30.0)   // whole-chain deadline (mirror: Pipeline.timeout)
            .CancelOn(shutdownToken)              // whole-chain token   (mirror: Pipeline.cancelOn)

    match! pipeline.OutputStringAsync() with
    | Ok result -> printfn $"timedOut={result.IsTimedOut}"
    | Error err -> eprintfn $"{err.Message}" // ProcessError.Cancelled when the token fires
}

C#

var pipeline = new Command("producer")
    .Pipe(new Command("consumer"))
    .Timeout(TimeSpan.FromSeconds(30))   // whole-chain deadline (mirror: Pipeline.timeout)
    .CancelOn(shutdownToken);            // whole-chain token   (mirror: Pipeline.cancelOn)

Console.WriteLine(await pipeline.OutputStringAsync() switch
{
    { IsOk: true, ResultValue: var result } => $"timedOut={result.IsTimedOut}",
    { IsOk: false, ErrorValue: var err }   => err.Message, // ProcessError.Cancelled when the token fires
});

Pipeline.Timeout tears the shared group down at the deadline and reports the timeout (IsTimedOut on OutputStringAsync, Error on RunAsync) — but, unlike a single command's captured timeout, there is no salvaged partial stdout to read back. A per-stage Command.Timeout cannot bound one stage of a chain — a pipeline spawns its stages directly, so a stage's own deadline never fires — so .Pipe rejects it with an ArgumentException instead of silently ignoring it. A per-stage Command.IdleTimeout is rejected the same way (a pipeline captures only the last stage's output and does not monitor per-stage activity). See pipelines.md for the full chain model.

A CliClient usually builds and consumes its Commands internally, so set the deadline and token once on the client and every command it builds carries them:

F#

let gh =
    (CliClient.create "gh")
        .WithDefaults(fun c ->
            c
                .Timeout(TimeSpan.FromSeconds 30.0)  // applied to every built command
                .CancelOn(shutdownToken))            // …controller cancels → all in-flight runs die

C#

var gh = new CliClient("gh")
    .WithDefaults(c =>
        c
            .Timeout(TimeSpan.FromSeconds(30))  // applied to every built command
            .CancelOn(shutdownToken));          // …controller cancels → all in-flight runs die

Clients are cheap — scope cancellation by building one client per cancellable scope with its own token instead of threading tokens through call signatures. See testing.md for the CliClient wrapper pattern.

Precedence and interactions

Timeout vs cancellation. A timeout is captured; a cancellation is always an error. When both land on the same run, cancellation wins — you asked the run to stop mattering, so no result is synthesized and the verb reports ProcessError.Cancelled, even on the capture verbs that would otherwise have returned an IsTimedOut result.

Which knob for which job:

Next: supervision.md · streaming.md · pipelines.md · commands.md · testing.md