Skip to content

CLI, server, and worker

@rulvar/cli is the optional ops layer. Same engine, three lifetimes: the rulvar command for a terminal, createServer for a network surface, and createWorker for background multi-process runs. All three are built strictly on the public engine API, so anything a shell does, your host application can do with the same calls; the shells exist so you do not have to write them.

Library mode is the default

Embed the engine directly for scripts, tests, and single-process apps. Reach for a shell when:

  • you want terminal ops over a journal directory (rulvar run, inspect, resume), OR
  • you expose runs over HTTP (start, watch, approve from a browser or another service), OR
  • runs must be resumed by whichever process is available, safely, across machines.

Install

bash
pnpm add @rulvar/cli

The package is ESM only and requires Node >= 22.12.0, like the rest of Rulvar. Some commands load optional companions dynamically at command time: rulvar plan needs @rulvar/planner installed, rulvar effects sweep needs @rulvar/effects, rulvar kb sweep needs @rulvar/evals, and rulvar kb inbox and rulvar kb gate need @rulvar/plan. A missing companion is a clear error on that command, never a load failure of the others, and missing is distinguished from broken: only a real module-not-found for the companion itself produces the install hint, while an installed companion that fails to load surfaces its own error with the cause preserved. The OTel exporter declares @opentelemetry/api (^1.9) as an optional peer.

One naming caveat: run the binary from a project that installs @rulvar/cli (pnpm exec rulvar ... or a package script). A bare npx rulvar in a project without it fetches the unscoped rulvar package from the registry, which is the library alias and ships no binary.

The rulvar command

The canonical grammar, with no aliases:

text
rulvar run <file|name> [--args JSON] [--store PATH] [--budget-usd N] [--profile NAME] [--strict] [--acceptance-policy POLICY]
rulvar resume <runId> [--args JSON] [--store PATH] [--registry FILE] [--dry-run] [--allow-args-change] [--strict] [--acceptance-policy POLICY]
rulvar replay <runId> [--args JSON] [--store PATH] [--registry FILE] [--assert-no-live] [--compare-output-hash]
rulvar runs ls [--store PATH]
rulvar runs audit [--store PATH] [--repair] [--no-load-repair]
rulvar inspect <runId> [--store PATH] [--candidates] [--candidate-bytes HASH]
rulvar invoice <runId> [--store PATH] [--json]
rulvar cost-audit [<runId>] [--store PATH] [--all] [--json]
rulvar effects ls <runId> [--store PATH] [--json]
rulvar effects show <runId> <intentSeq> [--store PATH]
rulvar effects sweep <runId> [--store PATH] [--single-process]
rulvar plan "<goal>" [--planning-budget-usd N] [--budget-usd N] [--allow-unbounded] [--dry-run]
rulvar preflight <file|name> [--budget-usd N] [--profile NAME] [--spawns JSON] [--json]
rulvar kb <list | inbox | gate | sweep>
CommandPurpose
runStart a workflow from a file path or a registered name, drive it to a settled outcome, exit with a code reflecting it.
resumeRebind a journal to its workflow and continue; fully replayed prefixes cost zero live calls.
replayVerify a recorded run without paying: a dry-run resume that reports replay accounting, localized determinism warnings, and the output digest comparison; the assertion flags turn the report into a gate.
runs lsList run metadata (id, status, last update, workflow, name) from the store.
runs auditCompare every run's meta row against its journal and name the divergences worker sweeps cannot see; --repair rewrites the sound ones from the journal.
inspectPrint one run's journal-derived state: entries, segments, suspensions, refused finish candidates, spend.
invoiceExport the per-dispatch reconciliation ledger: one row per billable provider call (failed and retried attempts included) with the provider response id, plus the gross/net totals; --json for the machine-readable form. See the invoice export.
cost-auditVerify the one-denominator contract on a stored run (RV1910): the roster is closed, the settle is the billing boundary, the settled fold, the invoice totals and the wire cardinality agree, and every terminal dispatch set equals its incremental provider-call rows (RV2008; journals without rows pass vacuously); exit 1 with the failing checks named, exactly what a pre-RV1904 journal reports. --all (RV2209) runs the same six checks over every run the store lists in run-id order, one summary row each (verdict, passed-of-total checks with the failing names, gross, wires), exit 1 when ANY run diverges; name a runId or pass --all, never both. When the invoice carries the orphanedReceipts lane (RV3405, paid wires the settled terminal's record set does not cover), every output form surfaces it (RV3501): the single run text prints the lane totals plus one line per receipt, the JSON shapes carry the lane verbatim under invoice, and the sweep appends an orphaned suffix to the run's row and a carrying count to its header; the lane never moves the verdict or the exit code, because an orphaned receipt is the honest double payment window of a resume, not a divergence. When the journal proves a repair was paid for (RV4002), the single-run forms print the workflow repair ledger (repairs: total N | draft | composition | semantic, one row per granted repair with its validators, sections, and priced wire; the JSON form carries the same repairs object): the fifth comparison run's judge rebuilt exactly this count from the raw transcript because no surface would answer for the workflow; journals without one, every pre-RV4002 journal among them, render byte for byte.
planAsk the planner to write a workflow script for a goal, then run it in the worker sandbox.
preflightLint the effective config and estimate the run before any provider dispatch: effective merged limits per declared spawn, the admission projection, bottleneck ordering, and exposure floors; --json for the machine-readable report. See the preflight estimator.
kbMaintain the model knowledge claim store.

Two inspect lines read what a terminal already recorded rather than re-deriving anything (RV2605). segments: partitions the journal at its settle boundaries (logical run telemetry, RV2510) and names how each segment settled and how many entries it appended, so a resumed run stops being one undifferentiated entries: N; when the journal continued past its last settle (RV1407) the count of those entries follows, because the last settled status is then not the run's last word. rejected finish candidates: lists what the declared finish contract refused (RV2507) with each row's verdict, size, hash prefix, failing validators, and blob ref when the bytes were retained; rows sharing one hash are counted as one distinct document, because the model serving the same text three times is a different failure from three genuine attempts. Both are absent when the journal records nothing of the kind. A third line is the settle's own semantic claim (RV2703): completion: prints what the last settle recorded about the work, which is the only reading available for a run that died before its acceptance policy ran or was resumed past it, since the acceptance: line below comes from the acceptance DECISION and exists only where one was rendered. Beside the child roster, inspect also prints the observed tool-budget calibration (RV3103): observed tool calls per recorded evidence entry: with the aggregate rate over the dispatches whose terminals carry both the evidence verdict and the executed-call counter, and named counts for the unpaired sides (contracts with no journaled counter, counters with no contract). The aggregate line exists only when at least one dispatch paired both sides, and a journal carrying neither prints nothing: absence stays NOT RECORDED in operator output too.

children under <scope>: is the roster the journal already holds (RV2702): how many children the orchestration admitted, how many settled and with what statuses, how many it refused admission, and the ones that settled ok below a declared evidence floor, named by the dispatch seq the orchestrator's own turns used as their handle. The live childrenAtFailure (RV2602) answers this for a consumer watching the run and dies with the process that held it, and the settle persists the completion lift only, so a post-mortem over a journal, which is all a paid run leaves behind, had no way to ask. Nothing is re-derived and no validator runs again, so journals written before either field existed read exactly as well. It is not the live roster: this reading happens after the terminal child barrier, so a child the live field called unsettled usually has a terminal here, and a missing status means the journal truly ends mid-flight. Children on branches the run ABANDONED are named too (RV2804): the provider billed that work and the orchestration kept none of it, and the roster used to present a discarded child exactly like a kept one, so "four children settled ok" counted branches the run had thrown away. The money layer has separated the two since RV1904 (grossUsd keeps abandoned spend, totalUsd does not), and this reading uses the same first-wins abandon projection the replayer disposes by, over the same journal, so it needs nothing that was not already written down.

run and resume print the same claim on stderr, beside the transport status (RV2703). status: ok has never said whether the work is done, whether the artifact passed the contract it declared, or what the children produced when nothing judged them, so the report names completion: with its degraded reasons, deliverable: (accepted or REFUSED by the declared finish contract, and whether the terminal carries an artifact at all), the count of rejected finish candidates: with the distinct documents among them, and children at failure: for a run that died before any policy judged its roster (RV2602). --strict turns those same fields into exit codes; this is the reading for a human who did not pass the flag. Every line is absent when its field is: nothing judged means no verdict, never a negative one.

Flag semantics are uniform:

  • --store PATH selects the JsonlFileStore directory (default .rulvar). Every command that opens a journal store selects it the same way. An explicit stores entry in your config's engineOptions wins over the flag.
  • --args JSON supplies workflow arguments. It appears on resume too because original run arguments are not journaled in this version: the host re-supplies them. What IS recorded at genesis is the binding (RunMeta.argsProvided plus a canonical argsHash, never the raw args), and resume verifies the re-supplied value against it before the engine starts: forgetting --args on a run started with them, adding them to a run started without them, or supplying a different value is a typed refusal, because a silently changed value changes the logical run and re-pays every args-dependent call. Runs recorded before v1.24.0 carry no binding, so a bare resume of one demands the explicit acknowledgment below. The --args value must be finite JSON (representable in canonical JCS): a numeric literal that overflows to Infinity is a typed refusal at parse time, before any store or adapter loads, because a non-canonical value would record a binding with no hash and defeat this gate. The recorded argsHash is a deterministic, unsalted digest, so it reveals args equality across runs and low-entropy args are recoverable by hashing candidates; rulvar inspect prints the full hash as a sensitive diagnostic, so treat it and the store with the same care as the journal.
  • --allow-args-change (resume only) is that acknowledgment: it overrides the args gate deliberately (resume without the genesis args, with new args, or of a legacy run whose journal predates the binding), always with a loud warning on stderr.
  • --dry-run (resume only) previews the resume without performing it: the engine replays in strict mode and the CLI prints the replay accounting (hits, misses, reruns, skipped, orphaned effect roots, invalid resolutions) plus what the run would settle as, with zero journal or meta writes and zero adapter calls. A preview that reaches work needing a live call reports the exact stopping point instead of paying for it.
  • --assert-no-live (replay only) exits 1 unless the replay was pure: zero misses and zero reruns, meaning a real resume would perform no new paid work. --compare-output-hash (replay only) exits 1 unless the replayed result's canonical (JCS) sha256 equals the outputHash the settling segment journaled on its run-settle decision; a run recorded before the digest shipped, a run that settled without a value, or a value JCS cannot serialize fails the comparison explicitly rather than passing vacuously. Together the two flags are the replay-strict gate: zero live calls and a reproduced output. replay follows the resume args binding exactly but deliberately has no --allow-args-change: changed args change the logical run, and verifying a different logical run proves nothing. Determinism warnings the re-executed body raises are printed with their localized frame either way (see runtime detection).
  • --repair (runs audit only) rewrites each divergent meta row from the journal: a row behind a journaled settle takes that settle's status, and a stranded row (terminal meta over live journal work, the fenced run state RFC's finding F1 residue) becomes sweepable again. When the store is leasable the repair takes a brief per-run lease, so a live owner makes it skip instead of racing; suspect verdicts are printed and never rewritten. Without --repair the command only reports. Either way it exits 0 only when the catalog ends consistent, so it can gate a cron probe.
  • --strict (run and resume) refuses a partial orchestration: when the settled value is an acceptance envelope whose completion is not 'complete', the command prints the degraded reasons and exits nonzero even though the run status is ok. Outcomes without an acceptance envelope are unaffected, and non ok statuses keep their ordinary exit codes. The same flag reads the claim-coverage grade (RV1702) when the envelope carries a claim-consistency meta: 'judge-failed' (nothing was judged), 'judge-declined' (RV2508: the judge was refused admission and never dispatched, so nothing was judged either) and 'critical-uncovered' (declared claims went unverified) exit nonzero, all states that previously slipped through strict as green, while 'partial' prints its counts to stderr and keeps the exit, because the bounded pass is the documented default and declaring critical anchors is the opt-in that makes the subset enforceable. 'vacuous' (RV2508: the draft cited nothing, so the configured pass verified nothing) prints and keeps the exit too, because citing nothing breaks no contract the pass declares. A stamped lowCoverage block (RV1809, the run declared a coverage floor and the pass ran under it) exits nonzero with the ratios and floors printed: "complete but under-verified by the declared floor" never reads green under strict. Ahead of every coverage grade, strict reads the deliverable verdict (RV2604): deliverableAccepted: false exits nonzero even under a complete completion, because completion answers for the CHILDREN and this field answers for the artifact, and the twenty-fifth comparison run is exactly that row (a child roster that passed, a declared finish contract that refused every synthesis, a run that settled on unvalidated output). An ABSENT verdict is left alone: no finishValidation was declared, so nothing judged anything, and a host that declares no contract is its own judge. Strict also binds the semantic verdict to the shipped document (RV3207): a coverage grade rendered over judgedStage: 'draft' while draftToFinal.rewritten reports the synthesis replaced that draft exits nonzero, because nothing semantically judged the artifact the run settled on; configure claimConsistency.stage: 'final' (or 'both') to grade the shipped document, and an unchanged draft or an absent bridge stays out of scope.
  • --acceptance-policy production (run and resume, RV4209) is the FAIL-CLOSED gate for consumers that ship the artifact, and it is a separate flag exactly because --strict keeps its documented exit 0 on partial and vacuous: the sixth comparison run settled ok under a standing waiver with three unsupported citations, and a pipeline reading strict's exit shipped it. The policy runs strict's mechanical checks first, then reads the envelope's one-word semanticTerminalVerdict (folded once at the orchestrator settle; productionAcceptable is the same exported predicate for HTTP and event consumers): a suspended run (unsettled), an ABSENT verdict (not-judged: nothing was configured, so nothing judged anything), and every verdict but clean (findings, partial, vacuous, waived, not-judged) exit nonzero with ONE stable JSON reason line on stderr ({"acceptancePolicy":"production","exit":1,"reason":...,"verdict":{...}}), so a pipeline parses the refusal instead of scraping prose. The one shipped policy name is production; anything else refuses typed.
  • --budget-usd N sets the run's dollar ceiling, immutable within a segment; only the explicit resume override changes it (see Budgets). On plan it caps the execution run of the generated workflow, consistent with run.
  • --planning-budget-usd N (plan only) caps the planning run: the planner conversation is its own paid run with its own journal, so its ceiling is separate from the execution ceiling by construction.
  • --allow-unbounded (plan only) waives the missing ceilings explicitly. plan never runs unbounded silently: without this flag, --planning-budget-usd is required, and full execution additionally requires --budget-usd.
  • --json (invoice only) prints the machine-readable InvoiceExport object instead of the line form, for piping into finance tooling. The export is self-describing: pricingBasis says per-row usd prices each call individually, rowUsdNonAdditive: false says those rows sum to the gross total (the per-request fold of a fully attributed run, RV504) while true marks an aggregate-priced remainder or legacy entry in the fold, and the additive per-row allocatedUsd column is the one whose flat sum reproduces the gross total exactly in every case.
  • --profile NAME applies a shipped run profile (fast, standard, deep, ultra): pure data bundles of effort hints, concurrency, budget defaults, and a permission preset, merged under your own options so your config always wins. The effort hints seed only routing entries your config already declares and that carry no effort of their own: an explicit effort wins, a role you do not route stays unrouted, ladder entries are untouched, and a profile never names a model. The lookup behind the flag is own-property only (RV1411): a name outside the shipped roster is the typed unknown-profile refusal, including inherited object names like toString, which the plain-object lookup used to hand back as if they were profiles (the CLI then silently accepted --profile toString as an empty profile instead of refusing it).

The CLI renders progress from the run's event stream: live TUI rendering on a TTY, plain line output otherwise. When a run suspends, the CLI resolves interactively: approvals prompt for allow or deny, awaitExternal suspensions prompt for a value. If input runs dry (EOF), the run is left suspended in the store, ready for a later rulvar resume, the HTTP server, or a queue worker.

Diagnostic output follows two rules. An error about a supplied --args value never echoes the value: the message names the failure class (invalid JSON, or a numeric overflow that defeats canonicalization) and the way out, because workflow args may carry private data and stderr routinely lands in CI logs. And every dynamic value a diagnostic line embeds (a runId, a suspension key, a provider error message, a model ref) is stripped of terminal control sequences before printing, exactly like the live progress renderer, so untrusted text cannot recolor, retitle, or rewrite the terminal it lands on.

Configuration file discovery

Commands assemble their engine from rulvar.config.mjs (or rulvar.config.js) in the working directory. The default export has four optional fields: engineOptions (anything createEngine accepts), workflows (the registry for by-name runs), kbSweep (the rulvar kb sweep matrix), and configFingerprint (the module's own configuration identity, recorded at genesis and verified on resume, RV4602). An absent config is fine; a workflow module passed to rulvar run may also carry workflow, engineOptions, workflows, and configFingerprint as named exports.

ts
// rulvar.config.mjs
import { defineWorkflow } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';

const triage = defineWorkflow({ name: 'triage' }, async (ctx) => {
  return ctx.agent('Triage the open incidents and rank them by blast radius.');
});

export default {
  engineOptions: {
    adapters: [anthropic()],
  },
  workflows: { triage },
};

With that file in place, rulvar run triage --budget-usd 2 starts the registered workflow against a JSONL journal in .rulvar.

Portable replay descriptors

A run started programmatically (an eval harness, a comparison experiment, a one-off script) records its workflow NAME in the journal, but the workflow VALUE lives in no rulvar.config.mjs, so rulvar resume and rulvar replay used to refuse it from any other checkout: the seventh comparison experiment's replay --assert-no-live died exactly there. --registry FILE (RV4602) closes the loop: the file is an ordinary module with the same named exports a workflow module carries (workflows, engineOptions, and since RV4602 configFingerprint), merged OVER the config registry for this one command. A run therefore travels as a three-part descriptor: the journal (the store directory or --store PATH), the args (--args JSON, verified against the genesis binding), and the registry module naming the workflow under its recorded name.

ts
// descriptor.mjs, beside the exported journal
import { comparisonWorkflow } from './harness.mjs';
export const workflows = { 'comparison-v3': comparisonWorkflow };
export const engineOptions = {
  /* the adapters and pricing of the original run */
};
export const configFingerprint = 'harness:v3:2026-08-21';

rulvar replay <runId> --store ./journal --registry ./descriptor.mjs --assert-no-live then verifies the recorded run offline from a clean checkout. The configFingerprint export closes the drift loop the engine already enforces: rulvar run records it at genesis (from the workflow module or the config), and a resume or replay that supplies one is verified against the genesis record STRICTLY before ownership, meta writes, or any provider call, so a descriptor for the wrong configuration refuses typed instead of replaying under drifted policy. The engine's own immutability refusals (the execution scope, the scope normalization table, an immutable-lifetime budget) stand unchanged underneath; the workflow body hash check still warns or refuses on a drifted registry entry per its declared bodyHash policy.

The plan command

rulvar plan "<goal>" is the terminal entry to the planned mode: the planner model writes a workflow script against the ctx dialect and your profile cards, the script is linted and self-repaired from structured diagnostics, compiled, and executed deterministically in the worker sandbox. --dry-run prints the accepted script without running it. The command imports @rulvar/planner dynamically, so install it alongside @rulvar/cli to use planning.

Both stages are paid runs with their own immutable ceilings, and a machine-written workflow never runs unbounded silently:

  • --planning-budget-usd N freezes as the planning run's ceiling B0 at its journal's genesis (PlanOptions.run.budgetUsd; re-planning the same goal resumes the existing journal under its recorded ceiling, see The planner). Required unless waived.
  • --budget-usd N is the execution run's ceiling (RunOptions.budgetUsd), exactly as on rulvar run. Required for full execution unless waived; combining it with --dry-run is an error, because a dry run executes nothing for it to bound.
  • --allow-unbounded waives the missing ceilings explicitly and loudly.

Planning exhaustion stops before execution starts (plan() throws its typed ScriptRejected carrying budget_exhausted), and execution exhaustion never touches the planning journal: two runs, two ceilings, two journals.

The preflight command

rulvar preflight <file|name> is the effective-config linter and dry-run estimator: it assembles exactly the options rulvar run would (the config file, the workflow module's exports, --profile, --budget-usd) but constructs no engine, opens no store, and dispatches nothing, then prints what the engine would derive. The target must resolve a workflow exactly like run, so a green preflight always describes the run you are about to pay for. See the preflight estimator for the report's semantics.

The report covers the effective merged UsageLimits per declared spawn (call over profile over engine defaults, the same merge the runtime applies), each spawn's layer-1 admission reserve and which arm of the reserve formula produced it, the admission projection over the declared wave (which spawns admit, which are denied and by what: the budget, the lifetime spawn cap, the orchestrator's maxSpawns, or an orchestrator cap its own reserve cannot fit), the per-tool and weighted-unit executed-call ceilings with the first bottleneck named, the orchestrator's effective cap and finalize reserve, and the concurrency and quota exposure floors. When the serving row carries ratesVerifiedAt, the spawn line also prints ratesVerified=<date> with its age, so the staleness of the rates behind every projected dollar is visible before any spend (see rate verification and drift); the rates verified: line of rulvar invoice answers the same question after the run.

The declared spawn wave comes from the preflight export of the config or workflow module ({ spawns?, orchestrator?, quotaRules? }, module over config), and --spawns JSON overrides it from the command line. --json emits the machine-readable PreflightReport. The exit code is the linter contract: 1 when any finding has severity error (an unrouted role, an unknown profile, a wave that admits nothing), 0 otherwise.

Knowledge-base maintenance

The kb subcommands maintain the per-project model knowledge claim store (./rulvar.models.json):

  • rulvar kb list prints the claims with full provenance.
  • rulvar kb inbox [--store PATH] aggregates the kb_propose proposals of finished runs from their run ledgers into a read-only review view, grouped by subject, task class, and polarity. Proposals expire 14 days after their run finished; the command writes nothing and authorizes no spend. Requires @rulvar/plan.
  • rulvar kb gate <runId> <entryRef> --approver NAME --ruled-out a,b,c turns one inbox proposal into a committed human-editorial claim. --approver and --ruled-out are mandatory: they form the attribution attestation, and the ruled-out vocabulary is prompt, tools, difficulty, transient-provider. Contrast evidence is optional via --contrast-run runId#seq or --contrast-eval reportId:caseId[,caseId...] (mutually exclusive), --confidence high|medium|low defaults to medium, and --store PATH selects the journal store as usual. Requires @rulvar/plan.
  • rulvar kb sweep runs the falsification matrix declared in the kbSweep section of your config: a fixed model pool (sweep volume is never authorized by proposal volume) unioned with every model carrying an active negative claim plus the re-measure queue. Optional canary probes run per pool member first and flip drifted claims stale (only when every probe settled ok: a budget-starved or transiently failed probe fingerprints differently without the model having drifted, so it never flips a claim). Requires @rulvar/evals.

A sweep multiplies paid runs, so kbSweep.budgets is required (or waive it explicitly with allowUnbounded: true): every target, judge, and canary run carries an immutable per-run ceiling, and maxTotalUsd is a debit-only envelope over the whole sweep. Each run authorizes its ceiling against the envelope BEFORE it starts, so a run that would breach it is refused before any provider call. Refusals never erase paid work: a cell whose walk was stopped partway renders as INCOMPLETE: envelope refused ... after N of M case(s) with everything already measured kept; only a cell refused before ANY work reports envelope exhausted, not measured. A target that hits its OWN ceiling reports exhausted, a case whose judge could not finish stays as evidence with the reason named, and refused canary probes render as such; every one of those shapes emits no claim, because a budget-starved measurement must not become a false weakness that blames the model for the ceiling.

ts
// rulvar.config.mjs: the kb sweep budget surface
export default {
  kbSweep: {
    committerId: 'ci-evals',
    models: [{ model: 'anthropic:claude-fable-5' }],
    cases: [/* EvalCases tagged by taskClass, built with @rulvar/evals */],
    canary: { agentType: 'probe', prompts: ['ping one', 'ping two'] },
    // Immutable per-run ceilings and the aggregate envelope. Required
    // unless you set allowUnbounded: true.
    budgets: {
      targetUsd: 0.5, // ceiling of every eval target run
      judgeUsd: 0.5, // ceiling of every judge run
      canaryUsd: 0.2, // ceiling of every canary probe run
      maxTotalUsd: 50, // hard debit-only envelope over the whole sweep
    },
  },
};

The worst-case authorized spend the command prints before its first call is canaryUsd * probes * pool + targetUsd * cases * pool, plus judgeUsd per judge call. Judge-call counts are grader behavior and unknowable upfront, so maxTotalUsd is the only guaranteed aggregate ceiling: keep it at or above that worst case for the sweep to finish, or set it lower deliberately to stop the matrix partway.

The HTTP server

createServer turns an engine and a workflow registry into a WHATWG fetch handler. It opens no socket of its own: you mount server.fetch on whatever Request/Response HTTP layer your host already runs, and your middleware owns TLS, timeouts, and authentication.

ts
import { createEngine, defineWorkflow, JsonlFileStore } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';
import { createServer } from '@rulvar/cli';

const releaseNotes = defineWorkflow({ name: 'release-notes' }, async (ctx) => {
  const draft = await ctx.agent('Draft release notes from CHANGELOG.md.');
  const verdict = await ctx.awaitExternal<{ approved: boolean }>('editor-signoff', {
    prompt: 'Approve the draft?',
  });
  return verdict.approved ? draft : null;
});

const engine = createEngine({
  adapters: [anthropic()],
  stores: { journal: new JsonlFileStore({ dir: '.rulvar' }) },
});

const server = createServer({ engine, workflows: { 'release-notes': releaseNotes } });

const response = await server.fetch(
  new Request('http://localhost/runs', {
    method: 'POST',
    body: JSON.stringify({ workflow: 'release-notes', options: { budgetUsd: 5 } }),
  }),
);
const { runId } = (await response.json()) as { runId: string };

Authentication is deliberately out of scope

The server is host-embedded, and auth belongs to host middleware. Do not expose server.fetch to an untrusted network without your own authentication layer in front of it.

Routes

MethodPathPurpose
POST/runsStart a run of a registered workflow. Body: { workflow, args?, options? } where options accepts runId, budgetUsd, name, tags, deadlineAt, and, since RV4805, the regulated posture subset of RunOptions: budgetPolicy ('immutable-lifetime' pins the genesis ceiling across every resume), maxInFlightExposureUsd, configFingerprint, scope, and scopePolicy. Answers 201 with { runId, status, workflow } and a Location header. What never enters this body stays with the host process by doctrine: authentication (host middleware, OQ-16), price tables, adapters, stores, redaction patterns, and secrets are createEngine configuration owned by the process that constructed the engine.
GET/runs/:idRun status. A run tracked in this process reports the live outcome, including the pending list of open suspensions; any other known run reports its stored metadata.
GET/runs/:id/eventsSSE event stream with Last-Event-ID reconnection.
POST/runs/:id/external/:keyResolve an awaitExternal suspension or an approval.
GET/runs/:id/costThe run's CostReport.

The server is a single-process shell: it tracks the runs it started (or resumed) in memory and serves everything else from the engine's stores (engine.stores), which is why it takes no store parameter of its own.

Server-sent events

Each SSE frame carries id: (the event's per-run telemetry seq), event: (the WorkflowEvent type), and data: (the full event JSON). Reconnect with the standard Last-Event-ID header and the server replays strictly AFTER that seq (the buffer is seq-ordered, so the resume point is a binary search, and a cursor seq the buffer does not hold simply replays everything after it): delivery is at-least-once, so deduplicate on the id. A terminal settle closes connected streams only AFTER the run's event pump has drained, so a client that keeps reading receives the complete tail, run:end included; if the pump itself failed, the close is preceded by an SSE comment saying the stream may be incomplete. Events are process-local telemetry, never run truth; a run known to the store but not live in this process answers with an empty stream that closes immediately. When maxBufferedEventsPerRun has dropped buffered events, the response carries an x-rulvar-events-dropped header with the count, and a client whose cursor lies before the retained window additionally gets a leading SSE comment naming the first retained seq; the journal remains the durable record. See Observability for the event catalog.

Every connection's pending queue is bounded, independently of the replay buffer: maxPendingEventsPerClient (default 10000) caps the frames queued in one response that the consumer has not read yet. A consumer that stops reading is closed at the bound with an SSE comment naming it; the frames already queued stay readable, and the standard Last-Event-ID reconnect resumes strictly after the last frame the client consumed. A replay longer than the bound is delivered the same way, in bounded chunks across reconnects, so pending memory per connection is O(bound) while at-least-once delivery is preserved end to end.

bash
curl -N -H 'Last-Event-ID: 42' http://localhost:8787/runs/$RUN_ID/events

Resolving approvals and external input

POST /runs/:id/external/:key is the HTTP form of RunHandle.resolveExternal. The key of an awaitExternal suspension is the key the workflow chose (editor-signoff above), and its value must validate against the schema pinned at suspension time, when one was set. An approval suspension synthesizes its key as approval:<seq> and resolves with { "decision": "allow" | "deny", "reason"?: string }; a flavor B ESCALATION shares the same key shape but resolves with its own EscalationDecision ({ "kind": "retry" | "decompose" | "cancel" | "accept", ... }), and since RV1408 the offline path applies the engine's own flavor classifier (validateDetachedResolution, exported) instead of a lookalike, so an escalation resolved on a server that never held the run takes its own payload and a wrong-shaped one is refused typed before anything is journaled, exactly as detached-live. Both appear in the pending list of the run status. For a run this server started that has settled suspended, the response's resumed: true means the server applied the durable resolution and started the run's ONE continuation segment itself; the approved tool then executes in that segment without asking again, under the same at-least-once tool window as every tool execution (see Resolving a settled run).

bash
curl -X POST http://localhost:8787/runs/$RUN_ID/external/editor-signoff \
  -d '{ "approved": true }'

Two paths serve the request:

  • Live in this process: the resolution goes through the run handle. If the run had already settled suspended and the resolution applied, the server resumes it in place, re-binding the registry workflow and the original arguments it retained; the response reports resumed: true.
  • Not live here: the server appends the resolution directly to the journal (under a lease when the store is leasable) and leaves the resume to a queue worker or a later rulvar resume. Payload validation still runs before the append, so an invalid resolution fails the request instead of poisoning the journal.

A resolution against an already-closed suspension is never an error that damages anything: the first closing entry wins, and the response reports applied: false with the superseding entry.

Cost, errors, and retention

GET /runs/:id/cost returns the exact in-process CostReport for a run that settled here (per-phase and per-agent-type attribution exists only in process). For any other run it folds the journal and prices usage through the optional priceUsd callback of createServer; without one, those usages surface in the report's unpriced list, never as a silent zero.

GET /runs/:id answers for a run this process never held too, and since RV1209 it answers with the same shape: the response carries the terminal envelope rebuilt from the journal and marked provenance: 'journal', priced through the same composed settle pins as the cost endpoint, or a typed terminalUnavailable: { reason, message } where nothing durable records a terminal. The rebuilt completion survives the restart when the settle recorded the semantic lift; what a rebuilt envelope still cannot carry (the run's own error), and why its absence is honest rather than empty, is in the terminal envelope. Every status body additionally answers the SSE capability machine-readably: capabilities: { events: boolean } says whether GET /runs/:id/events would stream this run's telemetry from THIS process (events are process-local), so a client learns it from the status instead of connecting into an immediately closing stream.

Typed engine errors map onto status codes with a { error: WireError } body: configuration and invalid-resolution errors answer 400, a held lease or a journal outside the compatibility window answers 409, anything else 500.

A segment can also fail without ever producing an outcome: the genesis ownership boot refuses a run another process owns (with zero writes), and a settlement whose durable write failed is withheld deliberately. GET /runs/:id reports those as status: "error" carrying the typed wire error, connected SSE streams close with a comment naming the failure, a late subscriber gets that comment instead of an empty stream, and the tracked run becomes eligible for retention and the settled cap like any other terminal run. Where anything was written at all, the journal stays the durable record: a withheld settlement re-settles by replay on rulvar resume without paying for a provider call.

Retention comes in two decoupled kinds, both opt-in and both evaluated when a tracked run settles terminally:

  • retention: (meta) => boolean is DURABLE retention: a true verdict applies engine.deleteRun (transcripts first, then the journal) and untracks the run. This deletes the record itself.
  • memoryRetention: (meta) => boolean releases only the tracked state (arguments, outcome, handle, SSE buffer); the journal and transcripts stay. After it, GET /runs/:id and /cost serve from the store exactly as for a run another process owns, and /events answers with the empty not-live stream.

Three bounds keep the memory of a server with long uptime finite without any predicate: maxTrackedRuns caps how many SETTLED runs stay tracked (oldest released first, live runs never counted or evicted), maxBufferedEventsPerRun caps each run's SSE replay buffer (oldest events dropped in chunks and counted; see the SSE section for how a replay marks the gap), and maxPendingEventsPerClient caps what any single SSE connection can accumulate unread (see the SSE section for the disconnect and reconnect contract). All three are validated at construction, a typed ConfigError for anything but a safe integer in the documented domain (maxTrackedRuns accepts zero, the other two are positive). The two event bounds are finite by default: absent, they resolve to the exported DEFAULT_MAX_BUFFERED_EVENTS_PER_RUN (50,000 events per run, enough that any ordinary run keeps its full replay and only long agent:stream delta torrents get windowed) and DEFAULT_MAX_PENDING_EVENTS_PER_CLIENT (10,000 frames per connection). Before v1.94.0 an absent maxBufferedEventsPerRun meant unbounded; a deployment that wants that behavior back sets an explicit huge bound (Number.MAX_SAFE_INTEGER) and accepts the memory contract that comes with it. Absent the retention options and maxTrackedRuns, tracked state persists in process memory for the server's lifetime; the per-run replay window is what keeps each tracked run's share finite.

The queue worker

createWorker runs suspended and interrupted runs in the background, safely across processes. It demands a store with the lease capability; handing it a plain JournalStore is a typed ConfigError at construction, never a silent split-brain. It also verifies the store is the same instance the engine writes (engine.stores.journal), because a fencing epoch protecting a store nobody appends to would protect nothing.

ts
import { createEngine } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';
import { SqliteStore } from '@rulvar/store-sqlite';
import { createWorker } from '@rulvar/cli';
import { workflows } from './workflows.js';

const store = new SqliteStore({ path: '/var/lib/acme/runs.db' });

const engine = createEngine({
  adapters: [anthropic()],
  stores: { journal: store },
  defaults: { workflows }, // the worker resolves workflows through this registry
});

const worker = createWorker(engine, {
  store,
  concurrency: 1,
  onError: (runId, error) => console.error(runId, error),
});
worker.start();

process.on('SIGTERM', () => {
  void worker.stop();
});

Leases and the fencing epoch

Every sweep asks the store for candidate metadata only, listRuns({ statuses: ['running', 'suspended'] }), so the poll cost tracks the resumable backlog rather than the whole historical catalog (with the retention option the sweep lists everything, because terminal metas are what retention judges; the statuses filter is advisory, and candidacy is re-checked on what comes back). Sweeps never overlap: a poll tick that fires while the previous sweep is still scanning reports zero picks instead of overlapping it. For each candidate the worker acquires a lease; a LeaseHeldError means another worker owns it, and at-least-once semantics make skipping safe. The resume itself passes the lease through ResumeOptions.lease, so every journal append of the resumed run carries the fencing epoch: a stale worker's writes are rejected by the store and never become visible, whether or not that worker noticed it lost the lease. Split-brain is excluded by construction, not by timing.

The lease is renewed at a third of its ttl (default ttl 60000 ms, exposed as DEFAULT_WORKER_TTL_MS; ttlMs must match the store's configured ttl). The match is executable now: a store exposing the optional leaseTtlMs capability (SqliteStore and PostgresStore do) is VERIFIED at construction, a mismatch is a ConfigError, and an omitted ttlMs simply adopts the store's value, so one config source drives both sides of the protocol. Both ttlMs and pollMs must be integers between 1 and 2147483647 ms, refused typed at construction: an overflow, NaN, zero, or negative cadence would collapse to Node's 1 ms interval floor and storm the store with renew and poll writes. There is no pollMs: 0 manual mode; drive sweeps directly with worker.sweep(). A failed renew cancels the run promptly instead of burning live calls whose appends would be rejected anyway.

Two more checks keep the loop honest:

  • At acquire, the journal's hashVersion is checked against the engine's compatibility window, strictly before any append: an older library never writes into a newer journal. Runs failing this (or workflow binding) are poisoned for this worker and reported through onError; they need the host, not a retry loop. See Journal compatibility.
  • A run that settles suspended is remembered with its journal length AND its generation token (RunMeta.genesis, minted at the run's fresh start and preserved across resumes), and is not re-leased until the journal grows, which is exactly what an offline resolution (from the server's external endpoint, for example) does. The generation is what tells a deleteRun and recreate of the same runId apart from the old run standing still: length can coincide, the token cannot. The same rule releases a poisoned runId when its run is deleted and recreated, and every sweep drops skip and poison entries for runIds that left the candidate set, so externally deleted runs never pin process-local state until restart.

Queue semantics are honestly at-least-once, with deduplication provided by the journal's two-phase entries: re-leasing a settled or unchanged run replays to the same outcome with zero live calls, which is the never-pay-twice invariant doing its job. Workflows resolve through the engine's defaults.workflows registry plus persisted compiled-workflow sources, never through a worker parameter; original run arguments are re-supplied per run through the optional argsFor(meta) callback.

The returned handle is small: start() begins sweeping on the poll cadence (default 1000 ms), sweep() performs one deterministic pass and returns the number of runs picked up (useful in tests and cron-shaped hosts), stop() cancels in-flight runs and releases held leases (a sweep still scanning the store is waited out first and picks nothing further, so no run slips past the cancel snapshot), and active() lists the runIds currently held. Retention mirrors the server: an opt-in retention predicate evaluated during sweeps over settled runs, applied under a briefly held lease, and never starved by load (a worker whose every concurrency slot is busy still applies retention on its sweeps).

No cross-process rate limiter

Worker processes share provider quota through the QuotaLimiter SPI (createEngine quota): point every worker's SqliteQuotaLimiter at one database file (one host), or every host's PostgresQuotaLimiter at one database and schema (a multi-host fleet), and the fleet enforces one global rate, see shared provider quotas. Per-provider concurrency caps still live in each engine and bound parallelism only. Scaling out concurrency defaults to 1 leased run per worker process, and hosts scale by adding processes, which the fencing epoch makes safe; a provider-side gateway remains the alternative when you already front your providers.

A typical multi-process deployment composes the shells over one leasable store:

The OTel exporter

toOtel projects one settled run's event stream onto an OpenTelemetry tracer. Events are consumed in seq order: span-opening events start spans, their matching closers end them, and payload-only events attach as span events on the innermost open span, following the run, phase, agent, tool, child span hierarchy. The function returns the number of spans created.

ts
import { trace } from '@opentelemetry/api';
import { toOtel } from '@rulvar/cli';

const handle = engine.run(releaseNotes, undefined);
const spanCount = await toOtel(
  { runId: handle.runId, events: handle.events, result: handle.result },
  trace.getTracer('rulvar'),
);

Pass contextApi and setSpan (the context API and trace.setSpan from @opentelemetry/api) to get real parent-child span nesting: every child span starts under a context derived from its parent, so the run > phase > agent > tool > child tree lands in the trace structure. Without them, spans come out flat but fully attributed, with the parentage riding the rulvar.* attributes (see Observability). The exporter needs only the tiny structural TracerLike surface, so it works with any SDK setup and stays out of your dependency tree until you opt in.

Deployment notes

ShellProcess modelNotes
rulvar CLIA project-local tool, one process per invocationJournal and config travel with the project directory: .rulvar for the JSONL journal, rulvar.config.mjs for engine assembly. Anyone with the directory can inspect and resume. Best for development loops and operator resolution of suspended runs.
HTTP serverEmbedded in your existing serviceMount server.fetch behind your auth middleware and reverse proxy. Use a durable store: the default InMemoryStore disables resume and is only fit for demos. Single process by design; pair it with workers so resolutions posted for non-live runs actually resume.
Queue workerOne process per concurrency slot, scaled horizontallyRun one worker per unit of provider quota under systemd or a container orchestrator; every worker builds its own engine over the same LeasableStore (for SqliteStore, the same database file or volume). Keep ttlMs equal to the store's lease ttl, and wire onError into your alerting: poisoned runs need a human.

Next steps