Durability and resume

A Rulvar process is disposable. Every effectful operation a run performs is appended to the journal through a pluggable store, so the process can die at any instant, on any machine, and the run loses at most the work that was in flight. Resuming re-executes your workflow body from the top; every call that already completed is served from the journal instead of a provider, and only the genuinely unfinished work runs live. That is the never-pay-twice invariant, and everything on this page is a consequence of it.

There is no snapshotting, no state machine to persist, and no per-step re-entry of your code. The journal entries plus the transcript blobs are the complete durable state of a run.

engine.run and engine.resume

engine.run starts a fresh run: it mints a run id (or takes yours), records the run metadata, and executes the workflow body once from top to bottom. engine.resume rebinds an existing journal to a workflow definition and executes the body again, matching calls against the journal as it goes.

import {
  createEngine,
  defineWorkflow,
  FileTranscriptStore,
  JsonlFileStore,
} from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';

const engine = createEngine({
  adapters: [anthropic()],
  stores: {
    journal: new JsonlFileStore({ dir: './runs' }),
    transcripts: new FileTranscriptStore({ dir: './runs' }),
  },
});

const review = defineWorkflow({ name: 'review' }, async (ctx, pr: number) => {
  const diff = await ctx.step('fetch-diff', () => fetchDiff(pr));
  const findings = await ctx.parallel([
    () => ctx.agent(`Review this diff for correctness:\n${diff}`, { agentType: 'reviewer' }),
    () => ctx.agent(`Review this diff for security:\n${diff}`, { agentType: 'reviewer' }),
  ]);
  return ctx.agent(`Merge these findings into one report:\n${findings.join('\n---\n')}`);
});

const handle = engine.run(review, 4242, { runId: 'review-pr-4242', budgetUsd: 10 });
await handle.result;

If the process crashes, restarts, or is redeployed, resume against the same store:

const resumed = engine.resume('review-pr-4242', review, { args: 4242 });
console.log(await resumed.preview); // { hits, misses, skipped, reruns, orphaned, invalidResolutions }
const outcome = await resumed.result;

engine.resume returns a ResumeHandle, which is a RunHandle plus a preview promise that settles with the replay accounting. A few contract points:

• Arguments are re-supplied, and the binding is recorded. Original run arguments are not journaled for in-process workflows in v1; the host passes them again via ResumeOptions.args. (Structure and identity come from the journal either way; the args feed your code, not the matcher.) What genesis DOES record is the binding: RunMeta.argsProvided and RunMeta.argsHash (sha256 over the JCS canonical form via hashRunArgs, never the raw args), preserved verbatim by every later segment. The engine does not enforce them (hosts may transform args legitimately); a host that wants the guarantee compares hashRunArgs(args) against the recorded hash before resuming, exactly what the CLI does: silently dropped or changed args change the logical run and re-pay every args-dependent call. The recorded argsHash is sensitive-derived metadata, not an opaque token: hashRunArgs is a deterministic, unsalted SHA-256, so it reveals when two runs were started with identical args and low-entropy args (a boolean, an approval flag, a role, a short id) are recoverable by hashing candidate values. The raw args are never journaled, but protect the store, rulvar inspect output, and run listings with the same access control as the journal and transcripts; the digest confers no confidentiality on the args it binds.
• Run-to-definition binding is checked. engine.run records the workflow name and a content hash of the body in the run metadata. Resuming with a workflow whose name differs is a typed ConfigError; a body-hash mismatch produces a loud warning (code RULVAR_RESUME_HASH_MISMATCH) and proceeds, because the journal itself decides replay versus live per content key. You can also omit wf entirely: the engine resolves the recorded name against the defaults.workflows registry. Hosts that treat an edited body as a different workflow can pin the binding with ResumeOptions.bodyHash: 'refuse': the same mismatch then becomes a typed ConfigError before ownership, meta writes, or any append, so a refused resume mutates nothing durable. The default 'warn' keeps the historical behavior byte for byte, and name or compiled-source mismatches stay hard errors under either value. The body-text hash cannot see CLOSURE values (two byte-identical bodies over different captured configs pin identically, RV3210), so the preferred pattern is to close over nothing and pass config through args; where something must stay closed over, declare RunOptions.configFingerprint (an opaque host string, for example a hash of the captured config) at genesis and assert it back with ResumeOptions.configFingerprint: a mismatch refuses the resume typed before ownership, meta writes, or any append, a recorded fingerprint the resume does not supply warns (RULVAR_RESUME_FINGERPRINT_UNCHECKED), and a supplied fingerprint the run never recorded warns (RULVAR_RESUME_FINGERPRINT_UNRECORDED) because absence means NOT RECORDED. The bounded execution scope binds the same way (RV4007): RunOptions.scope ({ tenant?, account?, project?, legalDomain?, region?, providerAccount?, sponsor? } since RV4205, sponsor since RV4408, at least one field, attribution only, never IAM, with one declared exception: a quota config under tenantFrom: 'scope' reads the scope's tenant into its reservations) records at genesis into RunMeta and a journaled execution_scope decision beside its canonical scopeDigest (sha256 over the normalized JCS bytes, the fixed-length join column), is immutable for the run's life (no resume door at all), rides the invoice header as executionScope plus executionScopeDigest and the export bundle via its meta, and ResumeOptions.scope asserts it back: a mismatch refuses typed before ownership, a supplied scope over a run that recorded none warns (RULVAR_RESUME_SCOPE_UNRECORDED), and the recorded identity resumes verbatim whether or not it is re-asserted. An unknown scope field DROPS silently by default (pinned, the RV4107 posture); RunOptions.scopePolicy: { unknown: 'reject' } (RV4205) refuses it typed by name instead, and the regulated floor enforces the refusal.
• Compiled workflows resume without code. For a planner-generated CompiledWorkflow the engine persisted the source in the transcript store at run start, pinned by its hash, so engine.resume(runId) rehydrates it byte-identically. This is why cross-process resume of compiled runs needs a durable transcript store such as FileTranscriptStore.

The same operation is available from the CLI, which enforces the args binding (a missing, added, or changed --args is a typed refusal unless you pass --allow-args-change) and exposes the preview as --dry-run:

rulvar resume review-pr-4242 --args '4242' --store ./runs
rulvar resume review-pr-4242 --args '4242' --store ./runs --dry-run

The default journal store is in-memory

An engine without a configured journal store uses InMemoryStore, which disables resume (with a one-time loud warning). Everything on this page assumes a durable store; see Stores.

What resume actually does

Resume is a pure function of the journal plus one forward pass of your code:

1. The journal is loaded and scanned once for hash-version compatibility, strictly before any live call, append, or budget reserve. An out-of-window journal is a typed refusal with zero side effects (see Journal compatibility).
2. Pure folds are built over the entries in append order: the abandon overlay (which branches were journaled as dropped), the resolution fold (which suspensions are closed), and the budget ledger (spend is restored from terminal entries and reserves from decision entries, never re-estimated and never double-counted).
3. The body executes from the top. Each scope keeps a forward cursor; a call whose content key matches an unconsumed entry ahead of the cursor is served from that entry. This is scoped forward-matching, and it is insertion-stable: a miss does not advance the cursor and does not extinguish future hits, so inserting a new call into your code costs exactly one live call, and everything around it still replays.

What happens to a matched entry depends on its terminal status:

Journaled status	On resume	Why
ok	replay	Completed, paid work is never re-executed.
escalated	replay	An escalation report is a completed, paid outcome; the consumer sees the identical report.
error	rerun	Failures rerun by default; opt into replaying task-class failures with memoizeOutcome: true.
limit	rerun	Same rule: memoizeOutcome: true replays the paid partial outcome instead.
cancelled	rerun	Cancellation is caller intent, not a task outcome. Aborted ctx.parallel siblings land here and rerun.
skipped (derived)	skip	Branches covered by a journaled abandon are not re-dispatched and cost nothing.
running (hanging)	redispatch	The crash interrupted it; see the two-phase section below.
suspended	wait or continue	Closed by a resolution entry if one exists, otherwise the run stays suspended.

Entries that no live call consumes (you deleted the call from your code) are silently skipped: never re-dispatched, never charged again, their payloads still addressable for audit. The resume report's orphaned list is stricter than that: it names only effects that genuinely need recovery under the per-kind pairing rules, meaning a dangling running dispatch with no terminal and a suspension with no resolution. Completed operations, decisions, plan and termination entries, and resolved suspensions never appear there, so a fully successful replay reports orphaned: []. There is no global prefix rule and no invalidation cascade; editing code between resumes costs exactly the calls whose identity changed. The identity rules themselves (content keys, scope paths, ordinals) live on the journal page.

A crash and resume walkthrough

Run the review workflow above and pull the plug at the worst moment: the step is done, the correctness reviewer finished, the security reviewer is mid-flight, and the merge agent has not started. The journal on disk looks like this:

seq  scope    kind   key      status   notes
0    (root)   step   3b7e...  running  fetch-diff dispatched
1    (root)   step   3b7e...  ok       terminal for seq 0; value: the diff
2    par:0:0  agent  8f2a...  running  correctness reviewer dispatched
3    par:0:1  agent  c41d...  running  security reviewer dispatched
4    par:0:0  agent  8f2a...  ok       terminal for seq 2; usage and servedBy recorded

     <- process dies here

Now engine.resume('review-pr-4242', review, { args: 4242 }) replays it:

1. ctx.step('fetch-diff', ...) derives the same content key, matches the completed pair (0, 1), and returns the journaled diff. The function body is not executed.
2. ctx.parallel allocates the same parallel site deterministically. Branch 0's agent call matches the completed pair (2, 4): the AgentResult is synthesized entirely from the payload, with zero adapter calls, and its usage folds into the budget ledger once, never twice.
3. Branch 1's agent call matches the hanging running entry 3. There is no terminal entry, so the work is redispatched live. If the agent had completed turns before the crash, it boots from its last turn-boundary checkpoint instead of starting over (next section).
4. The merge agent finds no candidate in its scope: an ordinary miss. It runs live and is journaled as a new entry pair.
5. The run settles; await resumed.preview reports:

{ hits: 2, misses: 1, skipped: 0, reruns: 1, orphaned: [], invalidResolutions: [] }

You paid for the interrupted reviewer's remaining turns and the merge agent. The step, the finished reviewer, and every dollar recorded before the crash are read back: the ledger is a fold over the journal, not process memory, so the resumed run's spent figures and its final cost report stay truthful, and the pre-crash spend counts against the same ceiling the run started with.

The budgetUsd ceiling survives resume through the run record

The dollar ceiling set at engine.run(...) time is recorded in the run's store metadata (RunMeta.budgetUsd) and restored on every resume: the restored pre-crash spend counts against the restored ceiling, and nothing changes the posture silently. Two degradation notes: the ceiling rides the run record rather than the content-addressed journal, so a custom store must round-trip optional RunMeta fields (the conformance kit checks this), and a journal written before the field existed resumes uncapped.

Raising a ceiling at resume time

A run that died against its own budgetUsd used to be unfinishable: the recorded ceiling governed every later segment and ResumeOptions deliberately carried no budget field, so the only way forward was a fresh run that re-paid the whole journaled prefix. ResumeOptions.run is the one explicit door through that doctrine:

const resumed = engine.resume('review-pr-4242', review, {
  args: 4242,
  run: { budgetUsd: 14, maxInFlightExposureUsd: 3 },
});

Each supplied value is validated exactly like its RunOptions counterpart and applies to the resumed segment and the run's remaining life: the segment's first meta write records it back into RunMeta, so a LATER bare resume restores the overridden posture, not the genesis one. The change is never silent: before the meta mirror flips, the segment journals a run_budget_override decision naming the recorded value, the applied value, the source, and the settled spend it was judged against (a run that started uncapped records null for the old value). A budgetUsd below the journal's settled spend refuses with a typed ConfigError before ownership, meta, or any append: such a ceiling would exhaust the segment before its first turn and read like a fresh money death. Absent fields keep the recorded values; an absent run object keeps the historical behavior byte for byte, and strictPricing deliberately stays out of the override: pricing hygiene is not a per-segment decision.

The door itself is a posture (RV3902): a run started with budgetPolicy: 'immutable-lifetime' records that posture in RunMeta, and a resume carrying ANY applying ResumeOptions.run refuses with a typed ConfigError before ownership, meta writes, or any append, raising and lowering alike. A bare resume of such a run stays an ordinary pure replay; the emergency lever for a run that must stop spending is cancel, not a ceiling edit.

Previewing a resume before paying

dryRun: true runs the same matching in replay-strict mode: the first call that would go live throws a typed JournalMissError and the run settles with that error, with zero live calls performed. A preview also performs zero store mutations by invariant: no meta write (no status flip, no segments bump), no transcript blob writes, and the journal's single append site refuses any append under replay-strict, so previewing repeatedly is always free and always safe.

const dry = engine.resume('review-pr-4242', review, { args: 4242, dryRun: true });
const report = await dry.preview; // honest hit/miss/orphan accounting, nothing paid

Use it to check what an edited workflow would cost before resuming for real (rulvar resume <runId> --dry-run prints the same accounting from the CLI). The inverse knob is invalidate: [seq, ...]: it unpins specific entries (typically failures memoized with memoizeOutcome) so this resume reruns them, for the case where an external system has recovered and you want a fresh attempt.

Suspended runs and how they resolve

Some entries do not complete; they wait. ctx.awaitExternal journals a suspended entry keyed by your key; a tool approval (an ask verdict from the permission chain) journals a suspended approval entry; the escalate tool suspends on the same machinery. When every in-flight branch of a run is blocked on suspensions, the process is free to exit: the run settles with outcome suspended and RunOutcome.pending lists the open keys.

const deploy = defineWorkflow({ name: 'deploy' }, async (ctx, service: string) => {
  const plan = await ctx.agent(`Draft a rollout plan for ${service}.`);
  const approval = await ctx.awaitExternal<{ approved: boolean }>('rollout-approval', {
    schema: {
      type: 'object',
      properties: { approved: { type: 'boolean' } },
      required: ['approved'],
      additionalProperties: false,
    },
    prompt: 'Approve the rollout plan?',
  });
  if (!approval.approved) return 'aborted';
  return ctx.agent(`Execute this rollout plan:\n${plan}`);
});

const handle = engine.run(deploy, 'billing', { runId: 'deploy-billing', budgetUsd: 5 });
const outcome = await handle.result;
// outcome.status === 'suspended'
// outcome.pending -> [{ key: 'rollout-approval', scope: '', entryRef: 2, prompt: 'Approve the rollout plan?' }]

Hours or days later, in a different process or on a different machine, resume and resolve:

const resumed = engine.resume('deploy-billing', deploy, { args: 'billing' });
const resolution = await resumed.resolveExternal('rollout-approval', { approved: true });
// resolution.applied === true
const final = await resumed.result; // continues into the execution agent

Resolving a settled run

Exactly one live execution segment owns a run at a time. The moment handle.result settles (with suspended or any other status) that segment is closed permanently: its parked branches never run again. A resolveExternal on the settled handle still works, but it appends the durable resolution through the journal fold and wakes nothing; the continuation belongs to exactly one subsequent engine.resume. The engine enforces the rule at both ends: starting a second concurrent segment of the same run in one engine throws a typed ConfigError before any side effect, and a stale writer racing the journal from an outdated tail is rejected by the store with the typed JournalOrderViolation (see Stores).

That gives you two equivalent safe orders, one per situation:

• Same process, settled handle in hand (what the CLI and the HTTP server do): resolve on the settled handle first, then resume once.

  const outcome = await handle.result; // 'suspended'
  await handle.resolveExternal(outcome.pending[0].key, { approved: true }); // durable, wakes nothing
  const resumed = engine.resume(handle.runId, deploy, { args: 'billing' }); // the ONE continuation
  await resumed.result;

• Fresh process, no handle: resume first (journaled work replays for free and the body parks again), then resolve on the RESUMED handle, which settles the parked position in place. That is the example above.

Before the settle, a live resolveExternal (from an approval:pending listener, say) still settles the waiting position in place without any resume at all.

Resolution never mutates the suspended entry. Every attempt to close a suspension, whether a live resolveExternal, an operator action in the CLI, a deadline timer, a class-level escalation decision, or an engine fallback, is itself an appended resolution entry referencing the suspended entry by sequence number. The first valid closing entry in journal order wins; later attempts are also journaled but classify as no-ops, so a second resolveExternal returns an outcome with applied: false and the reason already_resolved instead of throwing. A racing timer and a racing human can both fire; exactly one of them takes effect, deterministically, on every store and every replay.

Two more properties worth relying on:

• Validation is pinned. The schema you passed to awaitExternal is hashed into the suspended entry. A live resolution with an invalid payload throws the typed InvalidResolutionError and journals nothing; a resolution recorded while the run was not live is validated when the next resume consumes it, and an invalid one leaves the entry suspended.
• Approvals resume mid-turn. An ask verdict is journaled together with the agent's turn checkpoint, so after the approval resolves (even after a crash and a machine move) the agent continues the same turn without re-paying earlier turns and without re-running already-executed tools.

Deadlines survive resume

Two suspension flavors carry journaled deadlines: an escalation always does (its deadlineMs is required per spawn), and a tool approval does when the host opts in through permissions.approvalDeadlineMs (RV1107). Either way the deadline is journaled as deadlineAt on the suspended entry itself, not held in a process timer. On resume the engine reads it back: if the deadline has not arrived, the timer is re-armed for the remainder; if it has already passed and no closing entry exists, a timeout resolution attempt is submitted immediately, applying the configured default decision (escalations) or the typed deny (approvals). The re-armed timer is sliced against the Node timer maximum (2147483647 ms), so a remainder beyond about 24.8 days stays suspended for its full interval instead of resolving by timeout immediately. Both intervals must be positive integers no larger than the deadline ceiling, one hundred years in milliseconds (RV1204): the ceiling is not a wait bound but a date bound, guaranteeing now + interval always journals as a valid absolute date. A journaled deadlineAt that does not parse as a date refuses typed as journal corruption, at importRun intake and again before any timer arms; it never silently resolves the suspension immediately.

Suspension	Deadline	On timeout
ctx.awaitExternal	none in v1; waits until resolved	n/a
Tool approval (ask verdict)	optional, the permissions.approvalDeadlineMs opt-in (RV1107); absent config waits until resolved	denied with a typed reason (denied by timeout) through the same arbiter a live decision uses
Escalation (the escalate tool)	required, explicit per spawn	the configured default decision is applied; absent one, the report is accepted

Wall clock never decides an outcome by itself: time only influences which resolution attempts appear in the journal, and journal order decides which one wins. Two resumes of the same journal always agree.

Interrupted agents: turn-boundary checkpoints

An agent spawn is a single journal entry pair, but a long tool-using agent is not atomic in practice. With a durable transcript store the runtime writes a checkpoint of the agent's canonical history at the boundary of every turn. A crash mid-agent therefore costs at most one partial turn: resume matches the hanging running entry, boots the agent from its last checkpoint, and continues the loop. Compaction points are recorded in the checkpoint too, so a resumed agent never re-summarizes history it already compacted.

The same bound holds for the money: the partial turn after the last checkpoint is repaid live, and that single turn is the worst case. Agents in the dynamic orchestrator mode checkpoint mandatorily at every turn boundary, which is what lets a crashed orchestrate() restore its own conversation and find its children's results by content key without regenerating a single spawn decision.

Tools executed inside a turn are at-least-once: between a tool's execution and the checkpoint write there is a window where a crash forgets the execution but not its side effects. Make tools idempotent where they touch the outside world.

The boot trusts nothing it cannot decode. A checkpoint blob that does not parse, carries an unknown format byte, or has a malformed message structure is refused whole, and since RV1409 so is one whose counters are garbage: turns, toolCallsUsed, schemaAttempts, every usage field, and the compaction points must be non-negative finite numbers, because those counters seed the loop's limit arithmetic and are reported to the budget as paid spend (a negative restored turns would credit the maxTurns ceiling with turns nobody paid). The dispatch then reruns from the top, which is exactly the at-least-once floor above. Refusal is for garbage no boundary write ever produced; a checkpoint written before the usage invariants shipped still decodes, and the restore path sanitizes it as it always has.

Two tool-budget facts are journaled the moment they happen, as decision entries bound to the agent's dispatch: an adaptive tool budget extension grant, and the entry into the finalization window. Both are promises made to the model inside the conversation, and the executed-call count alone cannot always reconstruct them: a grant whose calls never ran before the crash is invisible to the count, and a later grant can move the counts back out of the window. On resume the runtime restores them from the journal, so a granted extension survives the crash instead of being silently revoked, and a replayed result's toolBudget summary reports the journal-backed fields (used, the granted cap, extensionsGranted, finalizationWindowEntered) with zero provider calls. The soft pressure notices around them stay events, never journal entries, and a run that grants nothing journals nothing new.

When a run is finished, engine.pruneRun(runId) deletes the checkpoint blobs of successfully completed attempts; completed work replays from the journal and never boots its checkpoint again. Parked, cancelled, escalated, and hanging attempts keep theirs.

At-least-once dispatch, exactly-once pay

Dispatched operations (agent, step, and child workflow entries) are two-phase: a running entry is appended at dispatch, and a terminal entry referencing it by sequence number is appended at completion. This split is what makes crash recovery honest:

• A completed pair replays exactly once. The terminal payload, usage, and cost are read back; the provider is never called.
• A hanging running entry with no terminal is redispatched at least once. The operation runs live and its terminal entry is appended against the original running entry. If the provider actually finished the first attempt but the crash beat the terminal append, you pay for that overlap; the journal guarantees you never pay for anything it recorded as complete, and the checkpoint bound above keeps the overlap to one turn for agents.
• The EVIDENCE of that overlap payment survives separately from the payment itself (RV2008/RV3405). The loop journals a receipt row for every wire call the moment it settles, so a crash inside a turn loses at most the execution, not the proof of what was paid: the invoice reports receipts of still running agents in its unsettled lane, and receipts the resumed terminal's record set does not cover in orphanedReceipts, where the redispatch paid again and the orphan is the first attempt's money. The receipt append is fire and forget by default (the loop never blocks its dispatch path on journal IO); defaults.billingReceipts: 'awaited' makes the loop await each append before the turn proceeds, so the receipt of the very wire being paid at the moment of death is the one guaranteed durable, at the cost of one journal IO await per wire call. In both postures a failed append degrades loudly to the terminal lane and never fails the run. Execution repay is unchanged either way: receipts make the at least once window ACCOUNTABLE, not narrower. The third posture goes to the other side of the wire (RV4006): defaults.billingReceipts: 'intent' journals a provider-intent decision BEFORE every dispatched wire attempt (awaited, the executor ledger's own intent-before-effect rule: a failed intent append refuses the dispatch, because a wire whose intent could not be made durable must not be able to bill), keeps receipts awaited, and turns the crash window into a NAMED fact: an intent with no receipt and no terminal coverage is a wire whose outcome this process never learned. The invoice reports every such wire in its openIntents lane (coordinates and request fingerprint, no invented dollars), rulvar cost-audit prints the lane, and a resume that finds one refuses the blind retry typed until the host passes ResumeOptions.acknowledgeOpenWireIntents: true, which the new segment journals (open_wire_intents_acknowledged), so waving the risk through is itself a durable, attributable act. Dispatch stays at-least-once with attempt binding: the intent narrows and names the unknown-outcome window, it does not close it.
• The budget ledger folds usage from terminal entries only, so redispatch cannot double-count: an interrupted attempt that never reached its terminal has no recorded usage to fold, the redispatched attempt's terminal folds exactly once, and admission reserves are restored from their decision entries rather than re-estimated.

Orphaned running entries you did not cause (for example, a call deleted from the code between resumes left its pair unconsumed) are reported in preview.orphaned and are never redispatched and never charged.

Deterministic shims (ctx.random, ctx.now, ctx.uuid) journal their single-phase records fire-and-forget through a serialized append queue, and the engine awaits the queue's flush() barrier before settling the run. A persist that fails inside that queue latches as the segment's first lost append: flush() rethrows it as a typed JournalIntegrityError, and the settle converts a would-be ok (or suspended) outcome into an error terminal, because an ok settle over a journal missing a deterministic record would replay differently than the run executed. The latch is permanent for the segment; a resume constructs a fresh queue against whatever the store actually holds, so a healed store resumes normally and re-executes from the last durable truth.

Moving a run between machines

Because the journal and the transcript blobs are the entire run state, a run moves by moving its store:

• JsonlFileStore and FileTranscriptStore keep one directory; copy it.
• SqliteStore (from @rulvar/store-sqlite) keeps one database file; copy it, or point both machines at it.
• PostgresStore (from @rulvar/store-postgres) keeps everything in a postgres schema; machines point at the database, and your postgres backup/PITR discipline is the copy story (see the runbook).

On the target machine you need the same workflow definition (same registered name; the body hash is checked and a mismatch warns loudly) and an engine whose supported hash-version window covers the journal's entries. For compiled workflows you need only the copied stores: the source travels inside the transcript store.

When two processes might touch the same journal, use a leasable store. SqliteStore (one host) and PostgresStore (across hosts) implement the lease contract: acquire hands out a fenced lease (acquiring a held lease rejects with the typed LeaseHeldError), and the lease rides every durable write, so a stale writer's mutations are rejected by the fencing epoch instead of corrupting the run.

Over a leasable store the engine enforces one genesis ownership protocol by default: EVERY execution segment holds the run's lease while it drives. A fresh engine.run and an in-process engine.resume that were not handed a lease acquire their own before their first durable write, renew it at a third of the store TTL exactly like a queue worker, and release it at settle (including a suspended settle, so the next owner can take the run). A second driver, whatever its shape (a worker sweep adopting a live fresh run, a double resume from another process, a simultaneous genesis of one explicit runId) rejects at its own boot with the typed LeaseHeldError, before any journal write, any meta write, or any provider dispatch. A crashed owner simply lets its lease expire, and the next worker sweep acquires and resumes; the at-least-once redispatch of its dangling turn happens under the NEW owner's lease, never concurrently with the old one. Nothing about this changes what is journaled: leases live beside the journal, and replay identity is byte for byte the same with or without them.

The default needs no code beyond choosing a leasable store. The explicit form is for hosts that admit runs through their own queue: acquire the lease at admission time and hand it to the engine (RunOptions.lease at genesis, ResumeOptions.lease on resume), and the engine carries it on every write but leaves acquire, renew, and release to you:

import { SqliteStore } from '@rulvar/store-sqlite';

const store = new SqliteStore({ path: './runs.db' });
const engine = createEngine({
  adapters: [anthropic()],
  stores: { journal: store, transcripts: new FileTranscriptStore({ dir: './blobs' }) },
});

const lease = await store.acquire('review-pr-4242', 'worker-7');
try {
  const resumed = engine.resume('review-pr-4242', review, { args: 4242, lease });
  await resumed.result;
} finally {
  await store.release(lease);
}

Leases carry a store-configured TTL (60 seconds by default for the SQLite store) and the holder renews at most every third of it; a worker that dies simply lets its lease expire, and the next worker acquires and resumes. If a live segment loses its lease (a paused process resumed past the TTL), the engine cancels the run promptly instead of burning live calls whose writes fencing would reject anyway. Even beneath fencing, the resolution fold is order-deterministic: whatever total order a store persisted, every reader derives the same outcome.

The ownership topology

The supported multi-process topology follows from the protocol: any number of HTTP or host processes starting and resuming runs, plus any number of queue workers sweeping, over ONE leasable journal store (SQLite for one host, PostgreSQL across hosts), with at most one owner per run at any instant:

• A fresh run's server process owns the run from genesis until that segment settles; a sweep that arrives mid-flight is excluded by the lease, not by timing.
• A suspended run is owned by nobody; whoever resolves or resumes it next (a server's live resume, a worker sweep) acquires ownership for exactly that segment.
• Offline resolution appends (a server writing an approval into a run it does not drive) take a brief lease around the append, exactly like rulvar runs audit --repair takes one around a repair.
• A dry-run preview (ResumeOptions.dryRun) performs zero store mutations and never acquires, so it stays available while another owner drives the run.
• createEngine({ ownership: 'none' }) opts an engine out of automatic acquisition for hosts that coordinate ownership entirely outside the engine; a caller-supplied lease always wins over both modes.

The protocol is store-generic: it rides the same acquire/renew/release contract the conformance kit checks, so it holds on PostgresStore across hosts, not only on process-local state.

Know the exact fencing boundary. The lease rides every durable mutation of an owned segment (a leased resume, and by default every fresh or resumed segment over a leasable store): every journal append (through the kernel's single append site), every RunMeta write, every transcript blob write, and the queue worker's retention deletes. What the STORE does with it depends on a declared capability. The journal side is always fenced for a leasable store. Meta and deletion are fenced when the journal store declares fencedWrites, as SqliteStore does: a superseded segment's late terminal putMeta is refused typed instead of overwriting the successor's row with a terminal status and a regressed segments counter (which could strand the run: a run whose meta looks settled is invisible to every worker sweep until an operator resumes it by runId). Transcript blobs are fenced when the transcript store declares the same capability, which the sqlite twin (store.transcripts()) does by keeping blobs in the store's own database: a stale segment still finishing a turn of the same attempt is refused typed at its checkpoint save instead of overwriting the successor's blob at the deterministic ref both share, so a later boot of that attempt (a crash resume, park or unpark, a dangling redispatch) can no longer regress to older turn state, replay paid turns, and widen the at-least-once tool window. Over the file and in-memory transcript stores (single-writer by contract, no marker) that surface stays advisory. Assert what your deployment requires with assertFencedWrites(engine.stores), treat prompt worker shutdown on lease loss as load-bearing, and see the fenced run state RFC for the audit behind the boundary. The whole promise is proven under real concurrent processes by the conformance kit's multi-process soak, which storms one store location through every fenced surface with injected stalls and diffs the final state against the serial history the epochs require; SqliteStore runs it in CI. The recovery half of the story is proven by the kit's kill-point suite: a child process is SIGKILLed around each durable write of a scripted run (the running entry, the ok terminal, the limit terminal, the run settle, the meta projection) and a fresh process resumes it, asserting the exact re-pay each bracket documents, one ok settle, and a healed meta; SqliteStore and PostgresStore run the whole table in CI.

Auditing and reconciling the meta projection

The journal is the source of truth and the meta row is a projection of it. Since the fenced run state RFC's phase 3 that is literal: every settle whose segment did durable work (or changed the recorded status) appends a run_settle decision entry to the journal BEFORE the meta write, so the run's outcome is journaled and the row is rebuildable. The write-on-change rule keeps replay byte stable: a pure-replay resume of an already settled run appends nothing.

A settle whose writes fail is acknowledged, never absorbed. A fenced store's LeaseHeldError on the settle append means a successor segment holds the lease and owns settlement (the fencing contract working), and nothing durable records THIS segment's outcome: since RV1009 the stale segment rejects handle.result with the typed SupersededError (data carries the runId and the computed runStatus; the cause is the fencing rejection), its run:end refuses green with settled: false and the distinct settledReason: 'superseded', and its meta write is skipped: the authoritative outcome is the successor's settle, never the stale computation (before RV1009 the superseded segment resolved ok silently, a green terminal no durable store wrote). A meta-only lease bounce over an already durable settle stays swallowed: the journal records the outcome, and only the projection belongs to the current holder. Any other failure of the run_settle append or the terminal meta write rejects handle.result with the typed SettlementError (stage names the write that failed; data carries the runId and the computed runStatus) instead of resolving: a caller must never act on an outcome nothing durable records. A failed settle append also skips the meta write entirely, so the projection can never run ahead of the journal. The event stream says so too (RV907): that segment's run:end carries settled: false beside its computed status, so an event-only consumer is refused the green terminal exactly like the rejected handle.result; the re-settled terminal after a healed resume carries no such field. Recovery is deterministic and free: every entry the run appended before settlement is already durable, so engine.resume replays to the same outcome without one paid provider call and re-attempts the settlement writes (a non-empty journal with no recorded settle re-settles on pure replay), and rulvar runs audit [--repair] reconciles the catalog when you would rather repair offline.

auditRun compares one run's meta row against its journal and names the divergence; auditRuns sweeps the whole catalog (it loads every journal it audits, which makes it operator tooling, not a hot path); reconcileRunMeta rewrites the divergent row from the journal where that is sound, preserving every other meta field byte for byte, with zero model calls and no workflow needed. All three are exported from @rulvar/core. Two divergence classes repair: meta-behind (the crash residue between the journal flush and the meta write, or a stale write contradicted by a journaled settle: the row takes the journaled status) and stranded (a terminal meta over live journal work, the F1 residue an unfenced store admits: the row becomes sweepable again). suspect audits (open suspensions under a completed meta, a journal with no meta row) are reported and never rewritten, because the legitimate look-alikes cannot be told apart mechanically. Operators reach the same probe as rulvar runs audit [--repair], which takes a brief per-run lease on a leasable store so a live owner is skipped, never raced, and exits 0 only when the catalog ends consistent.

One more boundary whose defaults stay with the host: everything these stores persist (journal values, transcripts, artifacts, prompts inside checkpoints) is plaintext unless you turn the data protection hooks on. createEnvelopeEncryption puts AES-256-GCM ciphertext under every persisted byte through the serialization seam, with KMS-shaped key management behind DataKeyProvider, and the masking policy applies redaction patterns at the telemetry boundary. What the library deliberately does not decide for you: enabling the hooks, custody of the master key, and the retention policy (journal redaction stays a host trade because a lossy journal corrupts replay; delete with engine.deleteRun under your own retention rules).

Next steps

• The journal explains entry identity: content keys, scope paths, ordinals, and why editing code costs only the calls you changed.
• Stores covers the shipped stores, the store contract, and the conformance kit for writing your own.
• Journal compatibility covers resuming journals written by older engine versions.
• Budgets explains the ledger that resume restores and the three-layer budget it feeds.
• Testing shows replay-strict cassettes that assert a resume performs zero live calls.

Durable admission and the run bracket

With createEngine({ admission: { scheduler } }) (plan 45, rfcs/admission.md) every non-preview run is bracketed by a durable admission ticket under the run's own identity (runId, genesis): a queued run waits for its grant, the terminal denied verdict refuses typed before any store mutation or provider dispatch, the lease renews while the run lives, and the release is part of settlement ordering. The bracket is built for the crash model of this page: a resumed segment RECOVERS its ticket by unit identity instead of minting a duplicate, a settled unit re-admits as a fresh ticket under the same identity, and a holder that dies mid-run settles conservatively through lease expiry (the scheduler refunds exactly what the fenced cover proves unused). Admission is an environmental fact exactly like the wire-level quota limiter: nothing of it is journaled, and replay never consults it.