Detached runs
--detach transfers a run to a detached copy of processkit-cli and returns
once that copy has provably started the child. It is intended for an
orchestrator that wants to launch now and supervise later from a different
process.
processkit-cli run \
--detach \
--run-id build-42 \
--capture-dir ./build-42-output \
--jsonl build-42.jsonl \
-- cargo build
processkit-cli inspect --run-id build-42 --json
processkit-cli wait --run-id build-42
What successful return proves
The launching invocation waits until run_started is readable in the JSONL
file. That event is written after the detached runner has:
- created the ProcessKit container;
- published the registry record and control endpoint;
- spawned the child.
After a successful return, the run is discoverable by list, addressable by
inspect / cancel / kill, and waitable by wait. A timeout in the detached
startup handshake kills the detached copy instead of leaving an unreported run
behind.
Exit-code semantics change
Foreground run forwards the child's exact exit code. Detached launch cannot:
the launching process exits while the child is still running.
| Detached launch result | Launcher exit code |
|---|---|
Run reached run_started | 0 |
| Program could not spawn | SPAWN (101) |
| Container could not be created | BACKEND (102) |
| JSONL or setup failed | SETUP (111) or the applicable reserved code |
The child's eventual result is in the terminal runner_exit event. A detached
caller must treat 0 as “started,” never as “payload succeeded.” See
Exit-code contract.
Output behavior
The detached runner owns none of the launching caller's standard handles. It
uses the same pump as --no-echo: child output is drained, but not retransmitted
to a terminal that no longer exists.
These features remain active:
--jsonl(still required);--capture-dirand--capture-max-bytes;--idle-timeout;--snapshot-interval(see below);- overall timeout, grace, environment, cwd, and resource limits.
--inherit-stdio and --inherit-stdin are rejected because there is no caller
left to provide those handles. Use --stdin-file when a detached payload needs
finite input.
Recorded tree snapshots without a live watcher
--snapshot-interval <duration> works under --detach — the launcher forwards it
to the detached copy like any other run flag — and this is the mode it was designed
for. A detached run is precisely the one nobody is watching with inspect at the
interesting moment, so the periodic members_snapshot events are the only record
of how the tree evolved between spawn and teardown.
processkit-cli run --detach \
--run-id nightly-index \
--snapshot-interval 5m \
--jsonl /var/lib/my-orchestrator/runs/nightly-index.jsonl \
-- indexer
Two consequences are specific to detaching:
- The JSONL file is the only diagnostic channel. The detached runner's
stdin/stdout/stderr are
null, so every warning the foreground runner would print reaches nobody. This is why a failed member read is recorded in the stream as amembers_snapshotwithread_error: truerather than merely warned about (see JSONL event schema). It is also why a JSONL write failure — a full disk, most plausibly — is invisible: the runner disables event logging after one unseen warning and the stream simply stops mid-run. - Detached runs are the longest ones, so size the cadence first. The stream
grows as
duration / interval; a day-scale run wants minutes, not seconds. The arithmetic and the recorded decision not to impose a ceiling are in Running commands.
Windows
The detached runner is created with DETACHED_PROCESS and owns no console.
A console child may therefore receive a new visible console from Windows. Pass
--create-no-window for a headless payload:
processkit-cli run --detach --create-no-window `
--run-id worker `
--jsonl worker.jsonl `
-- worker.exe
A detached runner cannot escape a Job Object that already contains its caller. If the outer job is kill-on-close, its owner still controls the detached run's ultimate lifetime.
Unix
The detached copy starts a new session with setsid, so terminal hang-up and
Ctrl-C from the launching session no longer target it. After the launcher
exits, the system's init process adopts the detached runner.
This changes session ownership, not ProcessKit's containment mechanism. The
run still reports cgroup_v2 or process_group, and the platform-specific
abrupt_cleanup value still applies if the detached runner itself is killed
before it can perform teardown.
Supervision pattern
Use one durable JSONL path and one stable run id per detached run:
run_id=nightly-index
events="/var/lib/my-orchestrator/runs/${run_id}.jsonl"
processkit-cli run --detach --run-id "$run_id" --jsonl "$events" -- indexer
processkit-cli wait --run-id "$run_id" --timeout 30m
The run id is an address while the runner is live. The JSONL file is the durable record after the registry entry has been removed on clean completion.
Reading the stream back — events
events closes the detach loop: it reads that JSONL stream back without an
external tailer or JSON filter, and without the caller having to look the locator
up first.
# Watch a detached run as it happens, until its terminal `runner_exit`.
processkit-cli events --run-id nightly-index --follow
# What happened, after the fact — the registry record is gone, the file is not.
processkit-cli events --file "$events"
# The runner's own bytes, for a machine (line-for-line, nothing re-serialized).
processkit-cli events --file "$events" --json
# Conformance-check a stream against this binary's embedded event schema.
processkit-cli events --file "$events" --validate
Naming the stream. --run-id resolves the locator through the registry — the
same jsonl field list --json publishes — which works while a record exists,
live or merely not yet reaped. --file reads a path directly and is the answer
once the record is gone (a clean exit deletes its own record, and prune reaps
what an abrupt death left), or for a stream this registry never knew about. The two
are mutually exclusive and exactly one is required; passing both is a USAGE (100)
error rather than a silent choice between them. When --run-id names no single
readable stream — no record, several records naming different streams, or a run
started without --jsonl — the refusal is the same CONTROL (103) verdict every
other by-run-id command gives.
Where a follow stops. --follow polls the file for growth (there is no
notification to subscribe to for either a file or a runner's death) and returns at
the first of: the terminal runner_exit event, or the registry reporting the run
over with the stream no longer growing. The second case is what an abruptly killed
runner leaves behind — it never got to write its terminal event — and it is
explained on stderr rather than passed off as a complete stream. --follow never
invents a deadline of its own: it is bounded by the run's lifetime, so a caller
that wants a wall-clock bound imposes one itself (wait --timeout alongside it, or
a bound on the whole invocation).
Read-only, like list/wait. events opens the registry read-only, never
connects to a run's control transport, and mutates nothing — so following a
production run cannot disturb, end, or even be noticed by it.
Shutdown sequence
For one detached run:
processkit-cli cancel --run-id nightly-index
processkit-cli wait --run-id nightly-index --timeout 30s
processkit-cli prune
For all runs confirmed live at one instant:
processkit-cli cancel --all
processkit-cli wait --all --timeout 30s
processkit-cli prune --dry-run
processkit-cli prune
The --all target set is a snapshot. A run registered afterward is out of
scope; repeat the sequence when the surrounding system allows concurrent new
launches.
Recovery after orchestrator restart
- Run
list --jsonto discover registry entries. - Read each durable JSONL file back with
events --file <path>(orevents --run-id <id> --followfor one still live), instead of hand-rolling a tailer: it hands out only complete lines, so a run being appended to while it is read never yields a half-written event. - Use
inspect --jsononly for entries confirmed live. - Treat stale records as evidence of abrupt runner loss, not as permission to address the recorded PID.
- Preview
prune --dry-run --json, then prune confirmed-stale entries.
See also
- Run registry — liveness and stale-entry semantics.
- Live-run control plane — control acknowledgements.
- Integration guide — adapter startup and recovery.
- Platform support — abrupt-runner-death guarantees.