Supervisor Type
Keeps a `Command` alive: runs it, classifies every exit against the `RestartPolicy` and the
`StopWhen` predicate, and restarts it after an exponential-backoff delay until supervision ends.
`Command.Retry` answers "run this once, replaying on failure"; a supervisor answers the
different question **"keep this alive"** — a minimal `runit`/`systemd`-style keeper on top of
the runner layer. The two are distinct layers: a supervised command's own `Retry` is **not**
applied per incarnation (supervision runs the bare runner), so use the supervisor's own restart
policy and backoff instead. Runs go through an `IProcessRunner` (the default `JobRunner`);
override with `WithRunner` to share a `ProcessGroup` or inject a test double.
Defaults: `OnCrash`, unlimited restarts, backoff `200ms × 2.0` capped at 30 s, jitter on,
failure-storm guard off (enable with `StormPause`; failure half-life 30 s, threshold 5.0).
**Observability while supervision runs.** `RunAsync` only reports its `SupervisionOutcome` at the
very end, which is unusable for a long-lived (potentially never-ending) supervised service — so
two callback seams, `OnRestart` and `OnStormPause`, report restarts and storm pauses *live*, as
they happen (e.g. for a health check or crash-loop alerting). Both callbacks are invoked
synchronously, from the supervision loop itself (the same async context driving `RunAsync`),
right before the corresponding delay is slept out — so a slow or blocking handler delays every
restart/pause; keep handlers quick and non-blocking. Neither callback changes
`SupervisionOutcome`'s semantics — `Restarts`/`StormPauses`/`Stopped` are unaffected and remain
the authoritative final tally; the callbacks are an additive, best-effort live view.
**Interactive supervision.** For a poll-and-control view — a live `Status` snapshot (activity,
restart count, storm-pause flag, the current child's pid/start time), a graceful `StopAsync`, and a
`Completion` task — use `StartAsync`, which returns a live `SupervisionSession` handle. `RunAsync`
is a thin wrapper over `StartAsync` + awaiting `Completion`; the `Status` snapshot *adds* to the
`OnRestart`/`OnStormPause` callbacks without replacing them.
**Event stream.** Where the callbacks push two specific transitions into your code, `Events` opts in
to the whole lifecycle as a *pull*-based `IAsyncEnumerable
Constructors
| Constructor |
Description
|
|
Supervise `command` with the default `JobRunner` (a fresh private kill-on-drop group per incarnation).
|
Instance members
| Instance member |
Description
|
|
Exponential backoff before each restart: the delay is `base × factor^n`, capped by `MaxBackoff`, where `n` is an escalation exponent that climbs by one per restart but **resets to 0 after a healthy incarnation** (one that stayed up at least as long as `MaxBackoff` and wasn't a hang killed by its timeout) — so a long-lived service that crashes occasionally restarts promptly instead of being pinned at the ceiling. `n` is not the lifetime restart count (`SupervisionOutcome.Restarts`). A `factor` below `1.0` (or non-finite) is treated as `1.0`. A negative `baseDelay` is rejected with `ArgumentOutOfRangeException`; `TimeSpan.Zero` is accepted and restarts with no backoff delay. Default: `200ms × 2.0`.
|
|
Bound (or widen) the output captured from each incarnation. The default is a bounded tail; pass `OutputBufferPolicy.Unbounded` to retain everything.
|
|
`Events` with the default capacity of 128 unread events — deep enough that an ordinary consumer never lags (one crash-restart cycle costs three events), shallow enough that a stream nobody reads still cannot grow the supervisor's memory.
|
|
Enable the **live event stream** on the sessions this supervisor starts: `StartAsync` then hands out a typed `SupervisionEvent` sequence through `SupervisionSession.EventsAsync`, reporting every incarnation start/outcome, launch-failure class, scheduled restart, storm pause, health-check verdict, give-up decision, and the terminal reason. Off by default, and purely additive: the `OnRestart`/`OnStormPause` callbacks and the `Status` snapshot keep working exactly as before, and no supervision decision, delay, or outcome depends on whether a stream is enabled or read. It is enabled here, on the builder, rather than discovered when a consumer first asks: the session must already be retaining events when its first incarnation starts, which happens as soon as `StartAsync` returns. Without this opt-in a session allocates no buffer and builds no event at all — `RunAsync` and every existing consumer pay nothing. `capacity` is the number of *unread* events the session retains (must be at least 1). Because an observer must never be able to stall a supervisor, a consumer that falls behind does not apply backpressure: the oldest unread events are dropped to make room for newer ones, and the gap is reported explicitly — see `SupervisionSession.EventsAsync` and `DroppedEventCount`.
|
|
Half-life of the failure score used by the storm guard (default: 30 s). A zero half-life keeps no history (every failure scores exactly `1.0`). A negative `decay` is rejected with `ArgumentOutOfRangeException`. No effect unless `StormPause` is set.
|
|
Failure score above which the storm guard trips (default: `5.0`). A non-finite threshold never trips. No effect unless `StormPause` is set.
|
|
Classify a crash — or a spawn/IO failure that never produced a result — as *permanent*, so the supervisor gives up instead of restarting it forever. `classifier` receives the `ProcessError` of the failed incarnation: for a crashed run (one that produced a `ProcessResult` but is not a success) that is the crash's own `ProcessResult.FailureError` projection; for a run that never produced a result at all, it is the runner's own error. This is a different seam than `StopWhen`, which classifies by *outcome* (`ProcessResult`) — `GiveUpWhen` classifies by *error kind*, independent of whether the incarnation ever ran. Not checked for a clean exit, nor for a run `StopWhen` already ended, nor for a crash the `RestartPolicy` itself would not have restarted (e.g. under `Never`) — those already stop supervision with a more specific reason. When checked, it runs *before* `MaxRestarts`: a permanent-failure verdict wins over "budget not yet exhausted". A crashed match reports `StopReason.GaveUp`; a match on a run that never produced a result has no result to report and surfaces the classified error directly as `RunAsync`'s `Error`, same as an exhausted budget on that path. Default: unset — a permanent failure restarts forever (throttled only by backoff/`MaxRestarts`/the storm guard), matching the prior behavior. If `classifier` throws, supervision ends with `Error(ProcessError.Io ...)`. The callback exception never escapes `RunAsync` or `SupervisionSession.Completion`; the error detail names `GiveUpWhen` and retains the classified error context. The session still performs normal teardown.
|
|
Multiply each backoff delay by a uniform factor in `[0.5, 1.5)` (default: **on**), so a fleet of supervised workers restarted by the same incident does not stampede back in lockstep. Disable for deterministic delays.
|
|
Enable a **predicate liveness probe**: every `interval`, evaluate `probe` and treat the *live* child as healthy when it returns `true`. After `LivenessFailures` consecutive failed attempts the supervisor gracefully stops and restarts the child, exactly like `LivenessHttp`. Off by default. `probe` is the caller's own health check (a custom RPC, a file/socket poke, a metric read); a returned `false` or a raised exception both count as a failed attempt, and the API cannot force a caller-owned `probe` to stop, so a hung probe is bounded by `LivenessTimeout` and abandoned (its late outcome safely observed) rather than pinning the monitor. A zero or negative `interval` is clamped to a safe 1 ms minimum.
|
|
How many **consecutive** failed liveness attempts trip a restart (default `3`). For HTTP and predicate probes, a single healthy attempt resets the run, so a flaky endpoint that recovers does not restart the child. For `LivenessMemory`, a healthy attempt resets the run only while the incarnation's peak is still at or below the limit; after the monotonic peak crosses it, later lower current usage remains failed. `count` must be at least `1`. No effect unless a liveness probe (`LivenessHttp`/`LivenessCheck`/`LivenessMemory`) is set.
|
|
The grace window passed to `RunningProcess.StopAsync` when a liveness failure forces a restart (default 2 s): the unresponsive child is asked to stop softly and hard-killed only if it does not exit within this window. `TimeSpan.Zero` intentionally escalates the kill immediately; a negative value is rejected. No effect unless a liveness probe is set.
|
Full Usage:
this.LivenessHttp
Parameters:
Uri
client : HttpClient
isSatisfactory : Func<HttpResponseMessage, bool>
interval : TimeSpan
Returns: Supervisor
|
Like the predicate overload, but sends requests through the caller-owned `client`. ProcessKit reuses the client across attempts and never mutates or disposes it.
|
Full Usage:
this.LivenessHttp
Parameters:
Uri
isSatisfactory : Func<HttpResponseMessage, bool>
interval : TimeSpan
Returns: Supervisor
|
Like `LivenessHttp(uri, interval)`, but uses `isSatisfactory` to decide whether a response means the child is healthy (e.g. accept only a specific health-endpoint status/body). A zero or negative `interval` is clamped to a safe 1 ms minimum.
|
Full Usage:
this.LivenessHttp
Parameters:
Uri
client : HttpClient
interval : TimeSpan
Returns: Supervisor
|
Like `LivenessHttp(uri, interval)`, but sends requests through the caller-owned `client`. ProcessKit reuses the client across attempts and never mutates or disposes it.
|
|
Enable an **HTTP liveness probe**: every `interval`, poll `uri` with an HTTP GET and treat the *live* child as healthy when the response passes the default 2xx check. After `LivenessFailures` consecutive failed attempts (default 3) the supervisor **gracefully stops** the child (with the `LivenessGrace` window) and restarts it through the ordinary policy/backoff path — closing the "alive but no longer responding" gap that `RestartPolicy` (exit-driven) and `Command.IdleTimeout` (stdout-silence-driven) miss. Off by default. The probe checks an *external* endpoint the child serves; it never reads the child's stdout/ stderr (those belong to the incarnation's own capture) and never appears in argv/env or a log. The first attempt runs one `interval` after the child starts, giving a natural startup window. Liveness needs a live child handle, so it applies only to a spawn-capable runner (the default), not a capture-only test double. A single attempt reuses the same poll/deadline core as `RunningProcess.WaitForHttpAsync`. A zero or negative `interval` is clamped to a safe 1 ms minimum so a configuration typo does not reject supervisor startup or create a hot loop. `uri` must be absolute; a relative URI throws `ArgumentException` while building the supervisor.
|
Full Usage:
this.LivenessMemory
Parameters:
int64
interval : TimeSpan
Returns: Supervisor
|
Like `LivenessMemory(maxBytes)`, but also sets the sampling interval. A zero or negative interval is clamped to the same safe 1 ms minimum as the other liveness probes.
|
|
Enable a whole-process-tree **memory liveness probe**. The supervisor samples attributable peak resident memory since the incarnation started every configured liveness interval and treats a value above `maxBytes` as a failed attempt. The peak is monotonic for that incarnation: once it crosses the limit, later lower current usage does not produce a healthy memory attempt. `LivenessFailures` therefore controls how many observations precede the restart, but cannot forgive an already-crossed peak. `maxBytes` must be positive. Whole-tree accounting requires a private Job Object or cgroup. If the active backend cannot provide an attributable metric, supervision ends with a typed `ProcessError.Unsupported` instead of silently falling back to leader-only or shared-group memory.
|
|
The per-attempt timeout for a liveness probe (default 2 s): one attempt gives the endpoint/ predicate up to this long to prove healthy before it counts as a failure. `TimeSpan.Zero` is a meaningful fail-fast timeout: the attempt is immediately `NotReady` without invoking the probe; a negative value is rejected. No effect unless a liveness probe is set.
|
|
Cap any single backoff delay (default: 30 s). A negative `cap` is rejected with `ArgumentOutOfRangeException` — besides being a nonsensical negative ceiling, a negative cap would make the healthy-incarnation escalation reset (`result.Duration >= MaxBackoff` in `RunAsync`) fire after *every* incarnation, so the backoff would never climb. `TimeSpan.Zero` is accepted (every backoff delay is then capped to zero — restart immediately).
|
|
Restart at most `count` times — `count + 1` total runs (default: unlimited). `count` must be non-negative (`0` means no restarts at all — a single run; a negative value is rejected with `ArgumentOutOfRangeException`).
|
|
Observe restarts live: `handler` runs synchronously, from the supervision loop, right before each restart's backoff delay is slept out — after the failed/finished incarnation, before the next one starts. Invoked on every restart (a crash, a timeout, a retried transient runner error, or a liveness-probe failure), never for the initial run. The event's `SupervisorRestartEvent.Cause` distinguishes an ordinary `Exit` restart from a `Liveness` one (a live-but-unresponsive child the probe stopped). `handler` runs on the same async context driving `RunAsync`, so keep it quick and non-blocking — a slow handler delays every restart. If it throws, supervision ends with `Error(ProcessError.Io ...)`; the raw exception never escapes `RunAsync` or `SupervisionSession.Completion`, the error detail names `OnRestart` and retains the result/error context that led to the restart. Normal session teardown still runs. Otherwise this callback is purely additive: it does not change `SupervisionOutcome.Restarts` or any other final semantics. Default: unset.
|
Full Usage:
this.OnStormPause
Parameters:
Action<SupervisorStormPauseEvent>
Returns: Supervisor
|
Observe failure-storm pauses live: `handler` runs synchronously, from the supervision loop, right before each pause is slept out — see `StormPause`. Same synchronous, keep-it-quick contract as `OnRestart`. No effect unless `StormPause` is set. Purely additive: does not change `SupervisionOutcome.StormPauses` or any other final semantics. If it throws, supervision ends with `Error(ProcessError.Io ...)`; the raw exception never escapes `RunAsync` or `SupervisionSession.Completion`, the error detail names `OnStormPause` and retains the result/error context that led to the pause. Normal session teardown still runs. Default: unset.
|
|
|
Full Usage:
this.RunAsync
Parameters:
CancellationToken
Returns: Task<Result<SupervisionOutcome, ProcessError>>
|
Supervise until the policy, the predicate, or the restart budget ends it, and report the `SupervisionOutcome`. A thin wrapper over `StartAsync` + awaiting the session's `Completion`, so its behaviour is identical to driving a `SupervisionSession` to its natural end. Returns `Error` when the *terminating* attempt failed to produce a result at all (a spawn/IO failure with no further restart allowed), or when one of the `StopWhen`, `GiveUpWhen`, `OnRestart`, or `OnStormPause` callbacks throws. A callback exception is converted to a terminal `Error(ProcessError.Io ...)`; it never escapes `RunAsync` or the session's `Completion` task, and normal teardown still runs. The error detail names the callback and retains the source context available at the failure: the completed `ProcessResult` for `StopWhen`, the classified `ProcessError` for `GiveUpWhen`, and the result/error context that led to the restart or storm pause for `OnRestart`/`OnStormPause`. A callback fault is terminal and is not retried. A spawn failure with restarts remaining counts as a crash and is retried. An incarnation cancelled via its token is terminal: supervision returns that `Cancelled` immediately, regardless of policy or budget.
|
Full Usage:
this.StartAsync
Parameters:
CancellationToken
Returns: Task<SupervisionSession>
|
Start supervising and return a live `SupervisionSession` handle — the interactive counterpart to
`RunAsync`. Supervision runs in the background from the moment this returns; poll the session's
`Status` for a live snapshot (activity, restart count, storm-pause flag, current child pid/start
time), ask it to stop gracefully with `StopAsync`, or `await` its `Completion` for the final
`SupervisionOutcome` (which is exactly what `RunAsync` would have returned).
Returns a already-resolved `Task
|
Full Usage:
this.StopWhen
Parameters:
Func<ProcessResult<string>, bool>
Returns: Supervisor
|
End supervision when `predicate` matches a completed run — checked before the `RestartPolicy` on every exit, clean or not. (It never sees a run that failed to *start*; spawn errors are classified by the policy alone.) If `predicate` throws, supervision ends with `Error(ProcessError.Io ...)`. The callback exception never escapes `RunAsync` or `SupervisionSession.Completion`; the error detail names `StopWhen` and retains the completed result context. The session still performs normal teardown.
|
|
Enable the **failure-storm guard**: when crash-restarts cluster faster than the failure score can decay, pause restarts once for `pause` (jittered per `Jitter`), then reset the score and resume. Off by default. Pauses taken are reported in `SupervisionOutcome.StormPauses`. A negative `pause` is rejected with `ArgumentOutOfRangeException`; `TimeSpan.Zero` is accepted and still counts as a storm pause (it resets the score, increments `StormPauses`, and fires `OnStormPause`) but sleeps out no real time — enabling the guard's accounting without a wait.
|
|
Run every incarnation through `runner` instead of the default `JobRunner` — e.g. a shared `ProcessGroup` runner for one kill-on-drop group, or a test double.
|
ProcessKit API Reference