Architecture

This page is for contributors: how the binding is put together, where the line between "binding" and "crate" runs, and the conventions that keep the two layers — and the three parallel views of the public API — in sync. The user- facing guides (linked from the docs home) explain what the library does; this page explains how the code that implements it is organized.

Two layers, one boundary

processkit-py is a thin PyO3 binding to the processkit Rust crate — not a reimplementation. Concretely:

┌───────────────────────────────────────────────────────────────────┐
│  Python package (src/processkit/)                                 │
│  __init__.py facade · _aio.py · _protocols.py · _types.py         │
├───────────────────────────────────────────────────────────────────┤
│  Binding crate (src/*.rs) — cdylib `_processkit`                  │
│  pyclasses/verbs, error mapping, runtime driving — thin glue only │
├───────────────────────────────────────────────────────────────────┤
│  `processkit` crate (crates.io, pinned exact version)             │
│  ALL platform logic: Windows Job Objects, Linux cgroup v2,        │
│  POSIX process groups, race-free spawn, async-throughout (tokio)  │
└───────────────────────────────────────────────────────────────────┘

Everything that decides how a process tree is actually contained and torn down on a given OS — Job Object completion ports on Windows, cgroup v2 on Linux, process-group fallbacks, the race-free spawn sequencing — lives in the processkit crate (see its own docs at docs.rs/processkit). The binding crate (src/*.rs, compiled to the cdylib _processkit) never reimplements any of that; it exists solely to:

• expose the crate's types as PyO3 pyclasses with a Python-shaped verb surface (kwargs, str/os.PathLike, sync and async pairs),
• drive the crate's async-throughout futures to completion from Python's sync and async worlds (the runtime.rs trio, below),
• map the crate's single processkit::Error onto a typed Python exception hierarchy (errors.rs's map_err, below),
• and re-export a small amount of pure-Python convenience on top (src/processkit/, further below) that composes on the compiled surface instead of touching the OS itself.

If you find yourself teaching the binding crate a new fact about an OS mechanism, that fact almost certainly belongs upstream in processkit instead — bump the pinned crate version and bind the new capability, don't duplicate it here.

The Rust module map

src/lib.rs is the #[pymodule(gil_used = false)] entry point. It declares no logic of its own beyond calling each module's register(m) — registration is delegated so that adding a new pyclass or function touches only its own module, not this central list:

#![allow(unused)]
fn main() {
mod batch;       mod cancellation; mod cli;      mod command;
mod convert;     mod errors;       mod group;    mod logging;
mod result;      mod runner;       mod running;  mod runtime;
mod supervisor;
}

This table is a map, not a promise: consult each module's own doc comment for the authoritative, current description.

gil_used = false opts the module into PEP 703 free-threaded CPython (on a free-threaded build, importing it does not force the GIL back on). This is sound only because the binding holds no unsynchronized shared state — see lib.rs's own comment for the itemized reasons (the tokio runtime is a managed singleton, exception caches use PyOnceLock, stream handles are Arc<Mutex<…>>, the stateful pyclasses that carry consumable/reconfigurable state — ProcessGroup, RunningProcess, ScriptedRunner, DryRunRunner — are #[pyclass(frozen)] with an interior std::sync::Mutex that serializes cross-thread access, and the remaining immutable pyclasses lean on PyO3's own per-object borrow checking). Keep that invariant in mind before adding any new shared mutable state to a pyclass.

The call flow: Python → crate → typed exception

Every consuming verb (output, run, exit_code, probe, start, and their a-prefixed async twins) funnels through the same shape:

Python call
  │
  ▼
PyO3 pyclass method (#[pymethods], e.g. PyCommand::output / Runner::aoutput)
  │
  ▼
crate future (processkit::Command::output_string(&cmd), etc. — async-throughout)
  │
  ▼
runtime.rs: block_on(...) [sync verbs]  or  drive_async(...) [async verbs]
  │                                            │
  │  block_on_interruptible: GIL released,      │  PyLazyFuture starts work on
  │  polls the future on a fixed tick so a       │  tokio; completion writes to a
  │  blocked Ctrl+C still raises on the main      │  socket and loop.sock_recv
  │  thread; a reentrant call from inside the     │  resolves the Python Future on
  │  runtime (e.g. a Supervisor stop_when         │  the event-loop thread
  │  thread; a reentrant call from inside the     │
  │  runtime (e.g. a Supervisor stop_when         │
  │  callback) is rejected with a clear error      │
  │  instead of panicking tokio                    │
  ▼                                            ▼
Result<T, processkit::Error>
  │
  ▼
errors.rs: map_err(error) -> PyErr   (the ONLY place a crate Error becomes a PyErr)
  │
  ▼
Typed Python exception (ProcessError subclass, or a dual-base one like
Timeout/ProcessNotFound/PermissionDenied that also inherits a builtin)

Two invariants worth internalizing when adding a new verb:

• runtime.rs is the only place a future is driven (block_on, drive_async, and the lower-level block_on_interruptible that both build on). A new verb should call one of these three, never hand-roll its own rt().block_on(...) — that's how the reentrancy guard and the Ctrl+C polling stay uniform across the whole surface.
• map_err is the only funnel from processkit::Error to PyErr. It picks the exception class from the error's own accessors (is_timeout()/is_not_found()/is_permission_denied(), falling back to a match on the enum variant for the rest) and attaches the structured fields (code, stdout, stderr, program, signal, timeout_seconds, diagnostic, output-cap counters) via setattr. A new crate error variant is covered automatically as long as it exposes the right accessor; no other module should construct a ProcessError subclass by hand from a crate error.

Conventions

• Per-module register(m). Every src/*.rs module exposes pub(crate) fn register(m: &Bound<'_, PyModule>) -> PyResult<()> that adds its own classes/functions (and, for errors.rs, the whole exception hierarchy). lib.rs only calls each register; it never lists an individual class or function itself. Adding a pyclass or function means adding it to its module's register, nothing in lib.rs.
• runner_pymethods! (runner.rs). PyO3's multiple-pymethods feature is off, so a pyclass may have only one #[pymethods] impl. Five runner pyclasses (Runner, ScriptedRunner, RecordReplayRunner, RecordingRunner, DryRunRunner) each need the identical twelve-verb surface (output/output_bytes/run/exit_code/probe/start, times their a-prefixed async twins) forwarding to the generic runner_* helper functions over ProcessRunner. The macro splices that shared block together with each type's own unique members (constructor, builders, __repr__) passed in as a token tree, so the run-verb surface has a single source of truth instead of five hand-copied blocks that could drift.
• Config struct → kwargs, not a mirror pyclass. When the crate exposes a builder/options struct (e.g. ProcessGroupOptions), the binding does not create a matching Python class for it. Instead the pyclass constructor takes the options as #[pyo3(signature = (*, field=None, ...))] keyword arguments, builds the crate's options struct from defaults, and applies only what was actually passed (see PyProcessGroup::new in group.rs). This keeps the Python surface flat (ProcessGroup(max_memory=..., cpu_quota=...)) instead of forcing callers to construct and thread through a second object.
• Sync/async verb parity (the a-prefix). Every consuming verb ships as a pair: a blocking one (output, run, start, …) and an a-prefixed asyncio one (aoutput, arun, astart, …) that accepts the identical arguments and returns the identical wrapped type. This holds across Command/Pipeline, every runner (real and test doubles), RunningProcess, ProcessGroup, Supervisor, and CliClient. A new verb should ship both halves together, wired through block_on/drive_async respectively. drive_async returns a lazy awaitable (PyLazyFuture): it schedules nothing until the first await, so an a-verb built but never awaited starts no work and, when dropped, releases what it captured (and tears down a process it already owns). Once awaited it delegates to a real asyncio.Future for cancellation. Tokio stores the completed outcome in Rust memory and wakes one shared per-loop sock_recv dispatcher; value conversion and Future resolution happen on the event-loop thread, so completion never attaches to Python from a foreign runtime thread and repeated stream steps do not allocate a socket each.
• gil_used = false. See the free-threading note above — a deliberate, narrowly-justified opt-in, not a default to imitate carelessly in a module that does need shared mutable state outside PyO3's own guarding.

The pure-Python layer (src/processkit/)

Alongside the compiled _processkit extension, a small amount of hand-written Python composes on top of it rather than adding more Rust surface:

• _aio.py — asyncio readiness helpers (wait_until, wait_for_line, wait_for_port, wait_for_path) and WaitTimeout. These compose on the already-compiled async surface (a StdoutLines iterator, a plain TCP connect) instead of bridging the crate's own probing methods, which keeps them simpler and usable against any server, not only one this package started. It also holds sample_stats(group, every), a periodic ProcessGroupStats series built directly on ProcessGroup.stats() — the crate's own StatsSampler borrows the group by lifetime and has no FFI-safe equivalent, so this is plain Python for the same reason as the readiness helpers.
• _protocols.py — the ProcessRunner / StreamingRunner Protocol classes: the typed dependency-injection seam that lets code written against "a runner" accept the real Runner, any of the test doubles, or a hand-rolled double, all checked structurally by the type checker.
• _types.py — the public type aliases (StrPath, Args, SignalName, RetryIf, ReadableBuffer, LineTerminatorName, Priority) exported so callers can annotate their own wrappers with the same vocabulary the API uses.
• __init__.py — the facade. It re-exports the compiled classes/functions from _processkit together with the pure-Python helpers above, and its __all__ list is the public surface: anything not listed there is not public, regardless of what's importable by digging into a submodule. The test-double runners (ScriptedRunner, RecordReplayRunner, RecordingRunner, DryRunRunner, Reply, Invocation) are deliberately excluded from the top-level __all__ and re-exported instead from processkit.testing, so the production surface and the testing surface stay visibly separate.

Guarding against drift: the stub/runtime/surface triangle

The compiled module (_processkit), the hand-written type stub (src/processkit/_processkit.pyi), and the package's __all__ re-exports are three independent, hand-maintained mirrors of one surface. Nothing keeps them in sync automatically — a renamed method, a new pyclass, or a dropped kwarg default can drift silently in any one of them. Two independent mechanisms catch that:

1. tests/test_api_surface.py is an AST-based drift guard, run as part of the normal test suite. It parses _processkit.pyi and compares it against the compiled module at runtime: every compiled class/function must be stubbed (and vice versa), every class's members must match (name, and property-vs-method kind), every __all__ must be sorted/unique/importable and cover every compiled export and every shim module's own __all__, the (async) context-manager dunders must be declared where promised, and every exported exception must remain a ProcessError subclass. A dedicated test (test_signature_parameters_match_the_stub) additionally compares each callable's actual parameter list (name, kind, whether it has a default) against the stub's — catching a renamed/reordered kwarg or a dropped default that the name-only checks can't see.
2. stubtest (mypy.stubtest), run in CI's typecheck job (uv run python -m mypy.stubtest processkit --ignore-disjoint-bases --allowlist stubtest-allowlist.txt), checks the stub against the compiled module from the opposite direction — signature shape (parameter names/kinds/defaults) and member existence both ways, at a level test_api_surface.py's hand-written checks don't reach. stubtest-allowlist.txt suppresses only the small set of irreducible false positives this pairing produces (PyO3's __new__-only construction vs. the stub's __init__ form, module-level Literal aliases stubtest doesn't recognize as such, and the compiled module's own auto-generated __all__) — every entry there documents why it's a false positive, not a real gap, and an unused entry fails CI (--ignore-disjoint-bases is passed but --ignore-unused-allowlist is not), so a stale suppression surfaces on its own.

When you add a new pyclass, method, property, or module-level function: add it to the #[pymethods]/#[pyfunction] in Rust, add the matching declaration to _processkit.pyi, and re-export it (top-level __init__.py for production surface, processkit/testing.py for a test double) if it's meant to be public. Run uv run pytest tests/test_api_surface.py and uv run python -m mypy.stubtest processkit --ignore-disjoint-bases --allowlist stubtest-allowlist.txt locally (both also run in CI) before opening a pull request — they will fail loudly, and specifically, if any of the three views disagree.

Rust unit tests (cargo test) vs. the Python suite (tests/)

The binding has two independent levels of test coverage, split by what they can exercise without a live Python interpreter:

• Rust #[cfg(test)] modules (src/convert.rs, src/supervisor.rs) cover the crate's pure, PyO3-free helpers — string/number parsing (parse_priority, parse_signal/parse_signal_name, parse_overflow_mode, parse_line_terminator, parse_restart_policy, stop_reason_str) and boundary-value validation (positive_duration/nonnegative_duration's NaN/infinite/negative/overflowing-Duration cases, build_output_buffer_policy's cap combinations). These are cheap to write and run per-case (every named preset, every alias, the unknown-name rejection), which the Python suite can only reach indirectly and rarely exhaustively. cargo test runs them without the extension-module feature — the crate is deliberately structured (see the [features] comment in Cargo.toml) so cargo test/cargo check work without ever linking as a Python extension; the handful of these tests that do need the GIL (e.g. parse_signal's Bound<'_, PyAny> argument) call Python::initialize() first, since nothing else brings up the interpreter in a plain test binary. cargo test runs in CI's rust-lint job alongside cargo fmt/cargo clippy.
• tests/ (pytest, uv run pytest) covers everything that needs PyO3, the GIL, or a real child process/event loop: the compiled classes' behavior (Command, Pipeline, ProcessGroup, Supervisor, CliClient, the runner doubles), the sync/async verb pairs, exception mapping, the stdout/stderr capture and tee pipeline, and the stub/runtime/surface drift guards above. This is also where a parsing helper's observable behavior through the Python-facing API is covered end-to-end (e.g. Command.priority("bogus") raising ValueError), even though the exhaustive boundary-value cases for the helper itself live in the Rust tests instead.

When adding a new pure helper to convert.rs/supervisor.rs, prefer a Rust #[cfg(test)] case for its boundary values; reach for a Python test only for behavior that's actually observable through the compiled API (an exception's type/message, a builder's resulting policy) rather than the helper's internals directly.

ProcessKit 3.2.0 surface audit

The 3.2.0 crate release adds several public Rust surfaces that are intentionally not swept into the Python binding as an automatic parity exercise. Each item needs its own contract, tests, and review before it becomes a Python API:

• SupervisionSession::events() and SupervisionEvent are a plausible future async iterator, but the Python representation of the event enum, including the bounded-channel Lagged case, must be designed first.
• ProcessGroup::limit_evidence() is useful post-run evidence, but its Tripped/NotTripped/Unknown result must remain explicitly tri-state; it must not be reduced to a boolean on platforms where the crate cannot prove the verdict.
• cancel_signal and cancel_grace require an explicit compatibility design against the binding's current cancellation and timeout behavior. They are not safe as passive builder kwargs until the soft-cancellation ordering and platform defaults are documented and tested.
• output_json is presently a likely duplicate of the binding's existing run_json()/arun_json() contract. A future proposal should prove a user visible semantic gap before adding another verb.
• report-serde is an opt-in crate feature and is not needed while the Python ShutdownReport exposes structured fields directly. Enabling it would be a dependency/feature decision, not a free API-parity improvement.
• Mechanism::ProcessReaper is platform-specific and the crate enum is non-exhaustive. Any Python-facing representation must preserve unknown future mechanisms and document the FreeBSD-only availability.
• wait_for_path and the crate's HTTP probe overlap with the binding's Python-side probes. The Python implementation remains canonical until a measured compatibility or performance gap justifies a separate migration.

This audit deliberately creates follow-up scope rather than adding public symbols here. A follow-up that changes src/*.rs, _processkit.pyi, or the top-level exports must carry its own API-surface tests and human-review gate.