Evals

@rulvar/evals is the quality-measurement package. It is built strictly on the public engine API: an eval case runs your workflow as an ordinary engine run, and a judge grader is an ordinary agent invocation on the same engine. Nothing bypasses the engine, and that is the point. Eval runs are journaled, budget-bounded by the same three-layer budget as production runs, and recordable at the adapter boundary, so an eval suite in CI is fully deterministic: record once, replay forever.

pnpm add @rulvar/evals

A case at a glance

An EvalCase names a workflow, its arguments, and the graders that judge the settled outcome:

import { createEngine, defineWorkflow } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';
import { goldenGrader, runEvalCase } from '@rulvar/evals';

const triage = defineWorkflow(
  { name: 'triage' },
  async (ctx, args: { report: string }) =>
    ctx.agent(`Classify this bug report as "low" or "high" severity.\n\n${args.report}`, {
      schema: {
        type: 'object',
        properties: { severity: { enum: ['low', 'high'] } },
        required: ['severity'],
      },
    }),
);

const engine = createEngine({
  adapters: [anthropic()],
  defaults: {
    routing: {
      loop: 'anthropic:claude-sonnet-5',
      // Schema-bearing agent calls resolve the extract role too.
      extract: { model: 'anthropic:claude-sonnet-5', effort: 'low' },
    },
  },
});

const result = await runEvalCase(
  engine,
  {
    workflow: triage,
    args: { report: 'Crash on startup when the config file is missing' },
    graders: [goldenGrader({ severity: 'high' })],
  },
  { budgetUsd: 0.5 },
);

console.log(result.passed, result.costUsd, result.latencyMs);

runEvalCase runs the target workflow as its own run on the engine you pass, waits for it to settle, then applies every grader. Pure graders (golden, rubric) execute host-side over the outcome; judge graders go back through the engine. The passed bit is strict: the run must settle ok and every grader must pass.

The measured EvalCaseResult reads entirely off surfaces the engine already provides; no separate measurement channel exists:

Field	Meaning
status	The target run's settle status.
passed	status === 'ok' and every grader passed.
verdicts	One GraderVerdict per grader: { grader, passed, score?, details? }.
costUsd	Target run cost plus all judge run costs (sums of CostReport.totalUsd).
judgeCostUsd	The judge-run share of costUsd.
latencyMs	Run start to run end, from the run's own event timestamps.
usage	The target run's normalized usage.
error	The typed wire error when the run did not settle ok.

RunEvalCaseOptions.budgetUsd sets the run ceiling of the target run and judgeBudgetUsd the ceiling of each judge run, so an eval that spirals is cut off exactly like any production run (see Budgets).

Graders

A grader receives a GraderContext: value (the run's structured output), outcome (the full RunOutcome, for status- and cost-aware grading), and judge(), the only channel back into the engine. It returns a GraderVerdict. A grader that throws is not absorbed: a grader that cannot grade is a defect of the suite, not a failed case, and the suite run fails loudly.

Family	Factory	Verdict	Model calls
Golden	goldenGrader(expected)	Comparison against a committed expected output; the evidence lands in details.	None
Rubric	rubricGrader(criteria, options?)	Fraction of named criteria met, reported as score; passes at passThreshold (default 1, all criteria; must be a finite fraction in [0, 1], anything else is a ConfigError at construction).	None
Judge	judgeGrader(options)	The judge model's structured verdict.	One journaled, budgeted judge run through the engine

Golden graders

goldenGrader(expected) compares RunOutcome.value against an expected output you commit next to the case. It needs the workflow to produce structured output (a schema on the final agent call, or a plain return value), which is what makes golden comparison mechanical rather than fuzzy.

Rubric graders

Rubric graders score against declared, named criteria; each criterion is a pure predicate over the output, and the per-criterion verdicts land in details:

import { rubricGrader } from '@rulvar/evals';

type Brief = { summary?: string; citations?: string[] };

const briefRubric = rubricGrader(
  [
    { name: 'has a summary', check: (v) => typeof (v as Brief | undefined)?.summary === 'string' },
    { name: 'cites two sources', check: (v) => ((v as Brief | undefined)?.citations?.length ?? 0) >= 2 },
  ],
  { passThreshold: 0.5 },
);

Judge graders

For open-ended output, judgeGrader asks a judge model for a verdict against a schema:

import { judgeGrader } from '@rulvar/evals';

const factuality = judgeGrader({
  model: { model: 'anthropic:claude-opus-4-8', effort: 'high' },
  instruction: 'Pass only when every factual claim in the answer is supported by the quoted sources.',
});

Two properties distinguish this from a typical LLM-as-judge harness:

• The judge runs through the engine itself. Each judge invocation is an ordinary journaled, budgeted agent run: it appears in the journal, it spends against judgeBudgetUsd, and it records and replays at the adapter boundary like every other call. A judge run that does not settle ok throws a typed EvalJudgeError instead of silently scoring zero.
• There is no default judge model. Judge model selection is subject to the router's role quality floors: weak defaults for judging are forbidden, and no advice can override the floors (see Model routing), so model is always explicit.

The default verdict shape is JUDGE_VERDICT_SCHEMA with its boolean passed; supply a custom schema plus a toVerdict mapper when you need a richer verdict. For fully custom judging logic, write your own Grader and call context.judge(spec) with a JudgeSpec (model, prompt, schema) directly.

Suites and the config matrix

runEvalSuite(engine, cases, options?) runs a case list sequentially, in declaration order, and aggregates passRate, totalCostUsd, and meanLatencyMs into an EvalSuiteResult alongside plannedN and completedN. Sequential execution is deliberate: it keeps journal and cassette order deterministic. Duplicate workflow names get #<ordinal> suffixes so every result row is unambiguous. Options carry the budget surface: budgetUsd is every target run's immutable ceiling, judgeBudgetUsd every judge run's, and envelope (a SpendEnvelope) is the aggregate debit-only bound each of those runs authorizes its ceiling against before starting. Refusals are monotone, never destructive: a refused TARGET stops the walk and lands as the typed refusal field with everything already measured intact, and a judge budget event (its own ceiling exhausted, or the envelope refusing it) normalizes into the owning row as incomplete: { reason: 'judge-exhausted' | 'judge-refused' } with the failing judge run's actual cost counted; such a row keeps its paid target evidence but can never count as passed. Non-budget grader errors still throw: a grader that cannot grade is a defect of the suite.

The envelope's accounting unit is integer micro-USD ($0.000001) and conservative at the boundary: the cap converts down, every debit converts up (a positive ceiling always debits at least one micro-USD), and a maxTotalUsd below one micro-USD is rejected outright, so the sum of admitted original ceilings can never exceed the cap. Amounts that are integer micro-USD up to float noise stay exact: 0.1 + 0.2 against a 0.3 envelope is a fit, not a float rejection.

runEvalMatrix runs the same cases against several engine configurations for side-by-side comparison: profile vs profile, cheap workers vs premium, reviewer on vs off. Each MatrixCell supplies a fresh engine factory, so cells stay isolated:

import { createEngine, type Engine, type ModelRef } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';
import { openai } from '@rulvar/openai';
import { runEvalMatrix } from '@rulvar/evals';

function engineWithWorkers(loop: ModelRef): Engine {
  return createEngine({
    adapters: [anthropic(), openai()],
    // Route extract at the cell's model too, so schema-bearing cases
    // measure the same member end to end.
    defaults: { routing: { loop, extract: { model: loop, effort: 'low' } } },
  });
}

const report = await runEvalMatrix(
  [
    { name: 'sonnet workers', engine: () => engineWithWorkers('anthropic:claude-sonnet-5') },
    { name: 'mini workers', engine: () => engineWithWorkers('openai:gpt-5.4-mini') },
  ],
  cases,
  { budgetUsd: 2 },
);

for (const cell of report.cells) {
  console.log(cell.cell, cell.passRate, cell.totalCostUsd, cell.meanLatencyMs);
}

Pass rate, cost, and latency per cell come from the runs' own usage and cost accounting; the harness adds no measurement of its own.

The benchmark kit

runBenchmark(engine, spec, options?) (RV-213) turns one workflow into a citable measurement series: it runs the spec's repeats sequentially, verifies every run, and reports nearest-rank percentiles over the runs that survived. The distinction it enforces is the one a hand-rolled loop silently skips: a run that FINISHED is not yet a run that counts. A run is SCORED only when it settles ok, every grader passes, and the replay-strict verification holds: a dry-run resume (zero journal or meta writes, zero adapter calls) replays it with zero misses and reruns, reproduces the journaled settle status and the run's outputHash digest, and the re-executed body raises zero workflow-provenance determinism warnings. Anything else lands in the run's record with machine-readable rejectedReasons (verification:output-diverged, verification:determinism-warning, grader:<name>, judge:refused, ...) and stays out of the series, so a workflow whose result mixes in bare Math.random() produces scored: 0 and no percentiles at all rather than clean-looking numbers that no one can reproduce. An aggregate envelope bounds the series the same way it bounds a suite, and its refusals are monotone: a refused target ends the series with the typed refusal marker and every completed repeat preserved, never a throw destroying paid evidence.

import { createEngine } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';
import { judgeGrader, rubricGrader, runBenchmark, SpendEnvelope } from '@rulvar/evals';
import { research } from './workflows/research.js';

const engine = createEngine({
  adapters: [anthropic()],
  defaults: { routing: { loop: 'anthropic:claude-sonnet-5' } },
});

const report = await runBenchmark(
  engine,
  {
    name: 'research-corpus',
    workflow: research,
    args: { corpus: 'docs/' },
    repeats: 5, // the regression protocol wants at least 5 for a citable series
    graders: [
      rubricGrader([
        { name: 'has-citations', check: (v) => Array.isArray((v as { citations?: unknown[] })?.citations) },
      ]),
      judgeGrader({ model: 'anthropic:claude-opus-4-8', instruction: 'the report must cover risks and cite sources' }),
    ],
  },
  {
    budgetUsd: 5,
    judgeBudgetUsd: 0.5,
    envelope: new SpendEnvelope(30),
    labels: { commit: 'abc1234', series: 'cold' },
  },
);

console.log(report.scored, report.wallMs?.p50, report.wallMs?.p90, report.costUsd?.p90);

The report's shape follows the honesty rules of the rest of the package:

• wallMs and costUsd are { min, p50, p90, max, mean } over SCORED runs and absent entirely when nothing scored: the kit never fabricates a series. Percentiles use the nearest-rank method (1-based, ascending, no interpolation), so a reported p90 is always a value that actually occurred. Wall time comes from each run's own run:start/run:end event timestamps; the kit reads no clock.
• Every run's full record is in runs: its runId (so anyone can re-verify with rulvar replay), status, wall, cost, usage, agentDispatches and invocations counts, grader verdicts, the complete verification verdict with both digests, and per-run metric values.
• totalCostUsd counts every target and judge run, scored or rejected: rejected work was still paid for.
• fingerprint records where the numbers came from: Node version, platform, arch, the resolved @rulvar/core and @rulvar/evals versions, the first run's start timestamp, and whatever labels the host supplies (the commit, the pricing snapshot id, the corpus hash, the cache series). The kit never shells out or guesses; identity the host does not supply is not recorded.
• options.metrics takes named extractors over each run's captured event stream ((events, outcome) => number), and each gets its own percentile series over scored runs; this is where critical-path metrics plug in without the kit hardcoding any orchestration shape.

Judging stays blind by construction: graders (including LLM judges through the shared judge channel, journaled and VCR-recordable like everything else) see the run's output and their own rubric, never a system label, run ordinal, or runId, so comparing two systems is running the same spec on two engines and comparing reports. Repeats run sequentially in ordinal order; cold-versus-warm cache comparisons are two benchmark invocations with different labels. When the second system is not a Rulvar engine at all, the kit's guarantees stop at the seam; that comparison has its own discipline, cross-system parity.

Cross-system parity

The benchmark kit compares two engines it controls. A parity run compares a Rulvar workflow against a system that is not Rulvar (an agent CLI, a rival harness), where nothing about the other arm is journaled, replayable, or blind by construction. The discipline below is the methodology the ninth comparison run validated end to end; it exists because every shortcut in it was tried by an earlier run and produced a number nobody could defend.

Freeze the inputs first. Both arms read the same bytes: a frozen manifest naming every input file with its SHA-256, written before either arm starts and shipped with the artifacts. A judged difference between arms is meaningless if the arms saw different trees, and "the repo at roughly that commit" is different trees. The manifest is also what makes a rerun a rerun instead of a new experiment.

One question, one contract. Both arms receive the same task verbatim and the same declared output contract (structure, coverage, length window). On the Rulvar arm the contract is enforced by the finish validators; on the other arm it is at least measurable after the fact. Contract compliance is scored mechanically per arm before any judge reads any prose.

Mechanical audit before judging. Every number in the comparison must be recomputed from each arm's own raw records by someone who did not produce it: the Rulvar arm from its journal (the critical path, synthesisCandidatesFromJournal, toolCalibrationFromJournal, the invoice), the other arm from its own event log. What a record set cannot answer is reported as NOT RECORDED, never as zero, and never borrowed from the arm's self-report: the ninth run's rival arm reported a total its own event stream could not reproduce, and the audit is where that surfaced. Only audited numbers enter the comparison tables.

A separate judge, honestly non-blind. Quality adjudication is a distinct invocation with pinned written instructions, run after the mechanical audit and barred from changing it. Cross-system blinding does not survive contact with the artifacts: arm identity leaks through telemetry formats, file layouts, and prose registers, and pretending otherwise upgrades adjudication into measurement. Name the judge non-blind in the report, keep its scores BESIDE the mechanical numbers, and treat a judge verdict that contradicts an audited number as a finding about the judge.

Subscription leverage. The judge, and often the rival arm itself, can run through a subscription CLI behind a thin exec adapter at zero marginal cost, with raw events captured for the audit; the metered API spend then stays on the arm being measured. This is how a full parity series stays affordable enough to repeat: the measurement is paid, the adjudication is not.

Pin the acceptance envelope before the run. Score thresholds, contract-pass requirements, and the cost and time ratios that count as a win are written into the series config before anything runs, and a run outside the envelope is a refused dossier, not a re-argued one. The subscription parity series that settled the postFanInShare targeting rule accepted its dossiers exactly this way.

The contract audit lexer

The mechanical audit step above keeps rediscovering the same two counting defects, and the seventh comparison experiment's post audit shipped both: it counted acceptance.minSpawnedChildren:4, a config property in citation clothing, as a citation occurrence, and recognized zero of the winning answer's 88 requirement ids because they were written as dash led list items (- N01 - ..., the original with an em dash separator) instead of the colon form the counter expected, while the losing candidate's colon runs and tables counted fine; both texts carried the full N48/R24/C16 sets, and the report had to recount them by hand. lexContractAudit(text, options?) (RV4603) is that recount as an exported grammar. The citation shape is the engine's own DEFAULT_CITATION_PATTERN with the citation audit's range tail semantics, fenced code is stripped by default with the same stripFencedBlocks the finish validators use, and the lexer adds the acceptance judgment a bare pattern cannot make: a citation must name a known source file extension (DEFAULT_CITATION_EXTENSIONS, overridable), and, when the host supplies the pure snapshot resolve, the first cited line must resolve; everything the pattern matched and the lexer refused lands in rejected with its reason, so a corrected count never silently disappears an occurrence. Requirement ids accept the colon, dash and table notations as one vocabulary (families defaults to N, R, C), reporting every occurrence with its form plus the DISTINCT count per family, which is the contract set size the audit compares. The result carries citationOccurrences, uniqueAnchors, perSection (the per H2 breakdown the harness tables want), requirementIds, and distinctRequirementCounts; on the seventh experiment's frozen records the lexer reproduces the corrected numbers exactly: 292 occurrences and 276 unique anchors for the winner with the one property notation rejected, 145 and 128 for the candidate, and 48/24/16 on both sides in both notations.

The claim corpus

The eighteenth comparison benchmark shipped a dossier whose three worst failures were semantic, and each rode straight past a green mechanical surface: "real models were not run" beside 125 recorded wire requests, @rulvar/plan described through a packages/planner citation, and a store default inverted in prose. A judge model can only rule on what the folds put in front of it, so the offline regression that matters is the PRECONDITION: for every named failure class, do the deterministic layers still form the pair, trigger on the run facts, prioritize the declared claim, and grade the coverage honestly?

CLAIM_CORPUS pins that precondition as data (RV1704): one adversarial case per failure class (live-fact, package-identity, inverted-default, numeric-range, negation, bounded-coverage, the nineteenth benchmark's modality-overclaim and scope-ambiguity (RV1809), the third comparison experiment's trio (RV3804): bound-conflation, a draft listing opt-in caps and unconditional guards as one mode; derived-premise, a derived figure whose premise contradicts the declared input, 2,000 slots computed from a 30 minute window where the input declares a 20 minute burst; and cost-basis, a locally estimated total printed as the provider's bill; and the fourth comparison experiment's decisive class (RV3909): stale-doctrine-echo, a draft echoing a DOCUMENTED doctrine while the pool holds the diverging source fact with both sides cited, the exact shape of the answer that echoed the retired budget-immutability wording from a guide six weeks stale into a run whose pool never carried the source side), each carrying a draft written to commit the falsehood, the pool readings or recorded fact sheet that contradict it, and the mechanical expectations. runClaimCorpus() executes every case through the same pure folds the orchestrator runs (pairDraftClaims, pairRunFactClaims, claimCoverageOf), no engine and no model, and reports per-case verdicts with the formed pairs attached; the shipped test asserts every case passes, so a change that stops forming any of these pairs fails the suite by case id instead of surfacing in the next paid benchmark.

What the corpus deliberately does not claim: that the pairs would be JUDGED correctly. The pool excerpts ride every verdict precisely so a host can hand them to a real judge and adjudicate the semantic half on their own budget: feed the drafts through the orchestrator claim-consistency pass with critical declared, or wrap the pairs in a judgeGrader rubric inside a suite. Human adjudication of a full benchmark answer stays a human step; the corpus keeps the machine layers under it from silently going blind.

Deterministic eval CI

Because target runs and judge runs both cross the ProviderAdapter seam, the VCR from @rulvar/testing applies unchanged. Record the suite once against live providers, commit the redacted cassette, and run CI hermetically:

import { createEngine } from '@rulvar/core';
import { anthropic } from '@rulvar/anthropic';
import { record, replay } from '@rulvar/testing';

// Record once, locally, against the live provider:
const recording = createEngine({
  adapters: record({ adapters: [anthropic()], cassette: './evals/triage.jsonl' }),
  defaults: {
    routing: {
      loop: 'anthropic:claude-sonnet-5',
      extract: { model: 'anthropic:claude-sonnet-5', effort: 'low' },
    },
  },
});

// Replay forever, hermetically, in CI. Routing must match the recording
// engine: the resolved model is part of every request's cassette key.
const ci = createEngine({
  adapters: replay({ cassette: './evals/triage.jsonl', onMiss: 'throw' }),
  defaults: {
    routing: {
      loop: 'anthropic:claude-sonnet-5',
      extract: { model: 'anthropic:claude-sonnet-5', effort: 'low' },
    },
  },
});

Suite, matrix, and sweep runners all execute sequentially in declaration order precisely so that cassette consumption is deterministic, and that determinism extends to identical requests: rows sharing one canonical request hash replay one per call, in recorded call order (file order only for groups recorded before v1.32.0, whose rows carry no occurrence numbers), so a recorded retry or a repeated case replays exactly as it ran. onMiss: 'throw' raises a typed VcrMissError on any unrecorded request (or one whose recorded occurrences are exhausted), so a changed prompt or a new case fails CI loudly instead of quietly going live.

The fault-injection kit

The comparison experiments left a standing list of fail-closed branches never observed live, and a branch nobody has ever driven is a claim, not a guarantee. runFaultInjection(options?) (RV811) closes the list by construction: each named scenario DELIBERATELY drives one such branch on the real engine with scripted adapters, zero provider calls and zero keys, verifies the documented typed observable, and leaves experiment-grade artifacts. The scenarios, in run order (FAULT_SCENARIO_NAMES):

• in-flight-exposure-refusal: a dispatch whose worst-case estimate does not fit maxInFlightExposureUsd is refused typed BEFORE any provider call, never a claimed budget-ceiling crossing.
• duplicate-quota-rule: two quota rules with one canonical content key are refused typed at construction, because one configuration must never admit differently per storage backend.
• torn-jsonl-tail and glued-jsonl-tail: a crash-torn trailing journal fragment is discarded with every whole record salvaged and the file repaired in place; whole records glued onto one trailing line are accepted data, both salvaged, never discarded with the fragment.
• crash-resume-settle-boundary: a journal cut immediately after an agent terminal entry resumes to ok with the settled step replayed free and only the unsettled remainder re-run.
• pricing-rotation-uncovered-tail: after a price-table rotation that drops the model, the pinned segment still prices under its own pin while the uncovered tail folds unpriced (undefined), surfaced, never a silent zero at stale rates.
• unknown-provider-id: routing to an unregistered provider id fails typed naming the id; nothing dispatches.

The RV909 scenarios turn the thirteenth experiment's fixed defects into permanent gates, so reverting any fix reports matched: false here, not only in the unit suite that shipped it:

• nan-statement-refusal: a statement whose dollars cannot be summed is refused typed at reconciliation intake, never verdict match over NaN totals with the divergence check silently disarmed.
• token-mismatch-divergence: provider-reported token counts that disagree with our recorded usage decide the reconciliation verdict even when the dollars agree; tokenComparison: 'informational' stays the declared opt-out with the mismatch still counted.
• audit-missing-field-finding (RV902 + RV1007): the documented-rates comparator (compareRates, the one the weekly audit runs) fails closed in both directions: a billable page rate the seed never declared is a named finding, and so is a seed rate the page dropped; a long-context tier only the page documents and a NaN scalar (a broken extraction) are findings too, never silent passes.
• anthropic-1h-priced: the shipped Anthropic table prices the 1h cache-write share at the documented 2x-input premium under its pinned pricingVersion, on the per-call reconciliation ledger, never the whole write count at the 5m rate.
• pause-turn-units: continuations absorbed into one dispatch settle at true wire units: the quota reservation reconciles the actual request count, the invoice row names every segment id, and a per-request statement joins the whole set all-or-nothing (a partial segment set reads partial-coverage, never no-overlap).
• pre-admission-count-refusal: a spawn the budget could never admit refuses before the countTokens egress, so the full child prompt never leaves the process.
• forced-finish-completion: a budget-capped adaptive orchestration settles ok with the honest completion envelope { result, completion: 'partial' } mirrored onto the outcome, never a bare result a consumer could execute as a full success.
• settlement-terminal-honesty: a run whose settlement write fails rejects typed and its run:end carries settled: false; the healed resume re-settles by replay with zero live calls, and the settled terminal carries no such mark.

The RV1002 scenario turns the fourteenth experiment's live-budget probe into the same kind of gate, driven on the REAL live path rather than post-hoc pricing:

• ttl-live-budget-parity: one differentiated cache write (a mid-stream usage event carrying the TTL split) debits the live ledger and settles to the SAME dollars, the run usage aggregate keeps the split it was billed under, and a ceiling set between the unsplit and split readings severs the run instead of settling ok over its own hard ceiling.
• pause-turn-real-adapter (RV1003 + RV1004): a legitimate two-segment pause_turn through the REAL Anthropic adapter and the real engine settles ok with the whole logical turn's usage (the finish confirms every per-segment mid-stream report), true wire units in the quota window and the invoice row, and an invalid pauseTurnMaxContinuations refusing typed before any wire; never a synthetic adapter with ready wire metadata.
• statement-settleable-guard (RV1005 + RV1006): a match verdict with complete coverage still reads settleable: false while a REAL usage-unknown attempt holds unattributed money, the clean twin reads settleable: true, and an export row whose own total contradicts its own component split refuses typed at reconciliation intake.
• superseded-terminal-honesty (RV1009): a fenced-out segment rejects typed (SupersededError) while its run:end refuses green under the distinct settledReason: 'superseded', and exactly one successor settles the run by replay; a terminal no durable store wrote must never read ok.
• tier-crossing-live-parity (RV1101): a long-context tier crossed by the SUM of one call that no single mid-stream slice reached debits the live ledger exactly like the settled fold (the per-call marginal meter re-prices the whole call at the crossing slice), and a ceiling set between the per-slice and tiered readings severs the run instead of settling ok over its own hard ceiling.
• benchmark-primary-preflight-parity (RV1905): the four-role benchmark's primary configuration ($6.00 ceiling, $4.50 orchestrator cap, $1.00 synthesis reserve, four workers at estCost $0.62) projects 2 of 4 seats with the synthesis hold and the per-row held terms exposed and the roster shortfall named admission-below-roster-floor, never the 5/5 green wave the live gate refused in the benchmark (RV1901).
• benchmark-recovery-root-exposure (RV1905): the recovery arm's shape on the real engine: a root turn refused by the exposure cap beside live gated children parks and completes after a hold releases (RV1902), every child terminal precedes run_settle (RV1903), and the terminal envelope and the invoice cardinality agree on the wire count (RV1904); the benchmark's recovery run terminal-failed on every one of these.

The parity-series scenarios turn the paid and subscription parity runs' crash shapes into the same kind of permanent gate:

• parity-quiescence-deadlock (RV2009): the coordination turn eats the exposure cap, every worker dies typed exposure-drained at zero provider attempts (RV2001/RV2002), the root forced-finishes partial (RV1902), and the exhausted terminal seals a one-denominator journal (RV2003); the third paid parity rerun exited mid-run with none of these.
• parity-sequential-roster-floor (RV2009): a seat-by-seat roster under an unreachable acceptance floor refuses its FIRST seat typed roster_floor with the whole-roster arithmetic journaled and zero paid children (RV2005); the parity arm paid three seats the settle verdict was bound to reject.
• parity-reserve-line-redemption (RV2101, RV2210): a coordination turn refused at spent + held synthesis reserve + proposed folds typed 'budget-floor' and the held reserve then FUNDS the synthesis the run kept it for, whose result rides the partial envelope; the scenario drives the same shape through makeOrchestratorWorkflow AND through the PlanRunner extension (orchestratePlanned) and demands the identical fold and the identical redeemed result from both, which is the DEF-7 redemption parity, verified rather than assumed.
• resume-spawn-famine (RV2201, RV2210): the kill-mid-fan-out journal resumes at the EXACT lifetime spawn cap to the finished dossier, with recovered agents re-admitted but never re-counted, no lifetime spawn cap decline journaled, and only the unsettled workers re-paid; the seventh subscription resume starved its judge and synthesis on re-counted admissions with the money reserve intact.
• validator-guidance-conflict (RV2202, RV2210, RV3801): the c3 trap finish (a live-observed claim with no artifact in its sentence) now heals in ZERO model repairs: the finish loop performs the evidence-grade prescription host side, the accepted document carries the run's own id inside the graded sentence with cited-value reading it as identity and never rejecting, and the composition safe guidance bytes are asserted on the journaled healed verdict; the third subscription run burned both repairs between the two verdicts, and the wire that repair once cost is now zero.

The report is fail closed: a scenario whose branch stops producing its documented observable reports matched: false with the observed detail quoted, and allMatched says so, instead of the list quietly becoming untested again. The report also carries requested and selected counts (RV1014), so a consumer that pins them can never watch the gate quietly shrink. With artifactsDir each scenario writes one <scenario>.json bundle (its doctrine, the observation, and every artifact: outcomes, journals, and the raw pre-repair bytes for the byte faults), the trace a review can cite exactly like an experiment's; only: [...] runs a named subset, an unknown name is a typed ConfigError, and an empty only refuses typed too: a gate that runs zero scenarios cannot report success.

Matrix sweeps across models

runSweepMatrix measures a fixed pool of models against a fixed pool of cases, one cell per (model, taskClass) pair, and turns threshold-crossing cells into eval-measured claims for ModelKnowledge:

import { FileModelKnowledgeStore } from '@rulvar/core';
import { runSweepMatrix, type SweepModel } from '@rulvar/evals';

const store = new FileModelKnowledgeStore(); // ./rulvar.models.json, git-diffable

const report = await runSweepMatrix(
  {
    models: [{ model: 'anthropic:claude-sonnet-5' }, { model: 'openai:gpt-5.4-mini' }],
    cases: [
      { taskClass: 'extraction', case: invoiceCase },
      { taskClass: 'synthesis', case: briefingCase },
    ],
  },
  {
    reportId: 'sweep-2026-07-12',
    committerId: 'ci-eval-pipeline',
    observedAt: '2026-07-12',
    engineFor: (member: SweepModel) => engineRoutedAt(member.model),
    store,
  },
);

console.log(report.cells.length, report.claims.length, report.committedVersion);

The moving parts:

• The pool is fixed and caller-declared. engineFor builds a fresh engine routed at each pool member; you own the adapters, budgets, and VCR posture, so a sweep records and replays like any engine run. effort on a SweepModel is part of the claim subject's identity.
• Thresholds gate claim emission. A cell's pass rate at or above thresholds.strength (default 0.9) emits a strength claim; at or below thresholds.weakness (default 0.5) a weakness claim; the mid-band emits nothing, because a 0.7 pass rate is uninformative. The defaults ship as SWEEP_THRESHOLD_DEFAULTS. Effective thresholds are validated before any engine, store, or envelope activity: both must be finite fractions in [0, 1] with weakness strictly below strength, and anything else is a ConfigError, because an out of range or reversed band would let a failing cell commit a false strength claim.
• The sweep is the deconfounder. The matrix is independent of your current routing, so it measures models where routing would never send them, which is what breaks self-fulfilling routing bias.
• observedAt is explicit. The sweep reads no wall clock; claim TTLs apply from the date you pass, which keeps recorded sweeps replayable.
• Budgets compose from per-run ceilings and one envelope. suite: { budgetUsd, judgeBudgetUsd } gives every target and judge run its own immutable ceiling, and envelope: new SpendEnvelope(maxTotalUsd) bounds the whole matrix: each run authorizes its ceiling against the envelope BEFORE starting (debit-only; completions, replays, and CAS retries return nothing), so pool times cases times judge-call growth cannot exceed maxTotalUsd even when falsification widens the pool. Refusals never erase paid evidence: a cell keeps every completed case (its n, caseNames, and costs) next to plannedN, flags an envelope-refused target as envelopeExhausted with refusedRunLabel, counts targets that hit their own ceiling in exhaustedRuns, targets that settled neither ok nor exhausted (a provider failure, a host cancellation, a suspension) in nonOkRuns, and unfinished judges in judgeIncompleteRuns, and names the cause in incompleteReason. Any incomplete cell emits no claim, because a measurement degraded by a budget ceiling, a provider outage, or a cancellation must never become a belief about the model.

When store is given, emitted claims commit through the eval-committer identity (below); either way the SweepReport carries every cell and every emitted MeasuredClaimInput for inspection.

The canary fingerprint

Provider model ids can silently start pointing at different weights. The registry-derived modelEpoch stamp (registry version, price-table version, caps hash; built by modelEpochOf in @rulvar/core) catches overt swaps and deprecations, but not silent alias re-pointing. The canary fingerprint is the probe that does:

import { runCanary, flipStaleOnCanaryDrift } from '@rulvar/evals';

const canary = await runCanary(
  engine,
  {
    agentType: 'canary', // a registered profile pinned to the model under probe
    prompts: [
      'List the prime numbers below 30, comma separated.',
      'Rewrite in one sentence: the cat sat on the mat because it was warm.',
    ],
  },
  { budgetUsd: 0.2 }, // each probe run's immutable ceiling
);

if (canary.allOk) {
  const drift = await flipStaleOnCanaryDrift(store, 'anthropic:claude-sonnet-5', canary.fingerprint);
  if (drift.flipped.length > 0) {
    console.warn(`model drift: ${drift.flipped.length} claims flipped to stale`);
  }
}

A fingerprint is a sha256 over the normalized outputs of the fixed probe set (normalizeCanaryOutput: NFC, trim, collapse whitespace), prefixed with the probe count so a probe-set edit never masquerades as drift; prompt order matters and enters the hash. Nothing on this path pins sampling parameters such as temperature: drift detection rests on the fixed prompts, the normalization, and exact fingerprint comparison. Probes run sequentially through the ordinary engine, one run per probe, each under the optional budgetUsd ceiling, so canary runs record and replay like everything else.

The allOk gate is load-bearing: a probe that did not settle ok, or one the envelope refused before it started (status: 'refused'; the loop keeps walking so completed probes survive), enters the fingerprint as its status marker, so a budget-starved or transiently failing probe fingerprints differently without the model having drifted. Never feed a non-allOk fingerprint to flipStaleOnCanaryDrift; the rulvar kb sweep command skips flipping on such runs automatically. (canaryFingerprint remains exported for fingerprint-only callers; runCanary is the drift-flip surface.)

Stamp sweeps with the fingerprint so drift detection has a baseline, via modelEpochFor:

import { modelEpochOf } from '@rulvar/core';

const report = await runSweepMatrix(pool, {
  // ...as above, plus:
  modelEpochFor: () => modelEpochOf({ canaryFingerprint: fresh }),
});

When a later flipStaleOnCanaryDrift sees a fresh fingerprint that differs from the one recorded on a claim, it flips the model's active eval-measured claims to stale in one CAS-rebased commit. Claims without a recorded fingerprint have no baseline and stay untouched; running it twice is an idempotent noop. In deployments that never run probes, the insurance is structural: negative eval claims expire after 30 days regardless.

Feeding ModelKnowledge

Sweep evidence is the only path by which measured quality reaches routing:

The contract, in full:

• Eval-measured claims are the only claims with metrics (passRate, n, graderId, optional cost and baseline), and the only class the knowledge card's verified layer compiles into start-tier recommendations. Human-editorial notes render as explicitly unverified and never steer a tier.
• The eval-committer identity is the only gate under which eval-measured claims commit. Runs themselves physically cannot write: the runtime holds a read-only handle to the store, so no prompt injection can forge a measurement.
• The blast radius of a false belief is clamped. A verified-layer recommendation shifts a ladder's entry tier by at most one rung, and role quality floors stay hard constraints no claim can weaken.
• Claims decay. Eval strength claims expire after 90 days, eval weakness claims after 30 (a stale negative belief is costlier, through lock-in); expiry is re-applied on every knowledge pin and every resume re-pin, so a multi-day suspension never resumes under dead beliefs.
• Standing claims get falsified. rulvar kb sweep (in @rulvar/cli, see CLI) re-tests claims through the ordinary engine, journaled, recordable, and budgeted (immutable per-run ceilings plus the maxTotalUsd envelope from kbSweep.budgets), and always includes models with active negative claims, so a model that improved gets a chance to clear its name.

runSweepMatrix with a store does the committing for you. For custom pipelines, the same two primitives are exported directly: evalMeasuredClaim builds one claim with the TTL applied per the decay table, and commitEvalMeasured commits a batch with the CAS rebase recipe (on rejection, read the fresh version again and retry; default 3 attempts). The attempts option here and on flipStaleOnCanaryDrift is a positive integer, refused as a ConfigError before the first store read: an unvalidated NaN or zero skipped the CAS loop entirely and surfaced a generic internal error instead of a typed refusal:

import { commitEvalMeasured } from '@rulvar/evals';

const version = await commitEvalMeasured(store, report.claims, {
  committerId: 'ci-eval-pipeline',
  reportId: report.reportId,
});

Every claim carries its evidence: sweep-born claims reference the sweep report and the exact case ids that produced the number, so rulvar kb list can always answer "why does the engine believe this?".

The measured-value checkpoint

runValueCheckpoint answers one question with money on the table: does the knowledge card DEMONSTRABLY improve tier and agentType selection on eval cases? It was the M12 shipping gate for the proposal loop, and it stays the standing way to re-measure the card's value after material changes. Two A/B experiments run under identical fixed pools, and the checkpoint passes only when BOTH criteria hold.

The vocabulary, bottom up:

• Rung: one concrete model of a declared ladder (a SweepModel).
• Ladder (CheckpointLadder): a named rung sequence with a startTier, the declared escalation order routing climbs.
• Cell: one (ladder, task class) pair; criterion 1 is judged per cell and pooled.
• Pool (CheckpointPool): the ladders plus evalCases, the MEASUREMENT half of the split. The claims snapshot you pass in must come from a seeding sweep over a DISJOINT case set, or the measurement is leakage.
• Arm (CheckpointArm): one side of an A/B comparison, reduced to { passRate, totalCostUsd, n }.

Criterion 1, rung selection (per cell, then pooled). The baseline arm runs every case at the ladder's default start tier; the treatment arm starts at the tier compileVerifiedLayer recommends from the snapshot's active claims (default when no recommendation). A cell passes on the exported rungRuleHolds rule: the treatment matches or beats the baseline pass rate at no more than 90 percent of its cost, OR beats it by at least 5 points at no more than its cost. Criterion 1 holds when a majority of cells pass AND the pooled aggregate passes the same rule.

Criterion 2, agentType selection (pooled). The same orchestrate-role cases run twice: with the knowledge store configured (the pinned card docks into the spawn tool description) and without it. The exported agentTypeRuleHolds rule (OQ-09 as amended 2026-07-12) passes the card-informed arm when it matches or beats the baseline pass rate at no more than 105 percent of its cost, OR beats it by at least 15 points at no more than 115 percent (the quality branch: when the baseline fails cheaply, the flat cost bar would punish the card exactly when it wins on quality). A vacuous-pass guard sits on top: two arms of total failures demonstrate nothing, so an informed arm with a zero pass rate fails criterion 2 outright.

Cost discipline: every case runs through the ordinary engine, so each arm pays real (or cassette-recorded) model calls under the per-run ceilings of suite/orchestratedSuite; the orchestrated arms usually need the larger budgetUsd because the run ceiling must host the orchestrator cap math and the finalize reserve. Budget for roughly cells x cases x 2 loop-role runs plus 2 x orchestratedCases orchestrate-role runs.

A minimal runnable shape:

import { renderCheckpointReport, runValueCheckpoint } from '@rulvar/evals';

const report = await runValueCheckpoint(
  {
    ladders: [
      {
        name: 'triage',
        startTier: 1,
        rungs: [cheapModel, midModel, strongModel],
      },
    ],
    // Each SweepCase carries its taskClass; disjoint from the seeding sweep.
    evalCases: seededPool.evalCases,
  },
  {
    snapshot: await store.snapshot(), // claims the seeding sweep committed
    observedAt: '2026-07-19',
    engineFor: (member) => engineFor(member),
    orchestrateEngineFor: (withKnowledge) => orchestrateEngine(withKnowledge),
    orchestratedCases,
    suite: { budgetUsd: 0.5 },
    orchestratedSuite: { budgetUsd: 2 },
  },
);

console.log(renderCheckpointReport(report));
console.log(report.passed ? 'the card earns its keep' : 'do not ship the card');

Reading the report: criterion1 lists every cell verdict plus the pooled arms, criterion2 shows both arms with the guard applied, and top-level passed is the AND of the two (criterion 2 counts as failed when unmeasured). A measurement artifact in either arm of a comparison (an envelope refusal, a judge budget event, a target that settled non-ok) marks the cell or criterion contaminated: the arms are not comparable, that verdict can never pass, and criterion 1 fails while contaminatedCells names the count, because an envelope drained by the baseline would otherwise leave an empty refused treatment arm that mechanically beats any baseline on cost. renderCheckpointReport prints the same as a terminal table. The four building blocks (runValueCheckpoint, renderCheckpointReport, rungRuleHolds, agentTypeRuleHolds) are all exported from the package root, so a custom pipeline can reuse the rules on arms it computed itself.

Next steps

• Testing: FakeAdapter, VCR cassettes, and replay-strict runs, the layers eval CI stands on.
• Model knowledge: the claim store, the pinned card, and the human-editorial class.
• Model routing: ladders, role quality floors, and the resolution chain that claims may only advise.
• Budgets: the three-layer budget that bounds every eval run.
• API reference: every exported symbol of @rulvar/evals, the checkpoint API included.