Rulvar API reference / @rulvar/evals
@rulvar/evals
The Rulvar eval framework: eval cases with golden outputs, rubric and judge graders that run through the engine itself, matrix sweeps across models and configurations, and the canary fingerprint. Exports runEvalSuite, runEvalMatrix, goldenGrader, rubricGrader, judgeGrader, and canaryFingerprint.
Part of Rulvar, an embeddable TypeScript engine for durable, budget-bounded multi-agent LLM workflows, where a completed LLM call is never paid for twice. Full documentation: docs.rulvar.com.
Install
bash
pnpm add -D @rulvar/evalsDocumentation
License
Classes
| Class | Description |
|---|---|
| EvalJudgeError | Thrown when a judge run does not settle ok. |
| SpendEnvelope | One envelope bounds one whole sweep invocation: share the instance across the canary loop and runSweepMatrix so canary, target, and judge runs all draw from the same remainder. |
| SweepBudgetError | Thrown when authorizing a run's ceiling would exceed the envelope. |
Interfaces
| Interface | Description |
|---|---|
| BenchmarkFingerprint | Where the numbers came from; percentiles without this are hearsay. |
| BenchmarkPercentiles | Nearest-rank percentile summary of one scored series. |
| BenchmarkReport | - |
| BenchmarkRunRecord | The full record of one benchmark run, scored or not. |
| BenchmarkSpec | One benchmark: a workflow measured over a series of repeats. |
| BenchmarkVerification | The replay-strict verification verdict of one run. |
| CanaryDriftReport | - |
| CanaryProbeSet | - |
| CanaryReport | - |
| CanaryRunOptions | - |
| CheckpointArm | - |
| CheckpointCell | - |
| CheckpointLadder | One declared checkpoint ladder: rungs are concrete pool members. |
| CheckpointPool | - |
| CheckpointReport | - |
| ClaimCorpusCase | One adversarial case: a draft, its contradicting evidence, and the mechanical expectations. |
| ClaimCorpusVerdict | One case's verdict: mechanical expectations against the folds' output. |
| ContractAuditLex | The lex of one contract audited document. |
| ContractAuditLexOptions | - |
| CriterionOneReport | - |
| CriterionTwoReport | - |
| EvalCase | One quality-measurement case. The shape is the documented interface verbatim; display names derive from the workflow name (the suite runner disambiguates duplicates by ordinal). |
| EvalCaseResult | The measured result of one EvalCase. |
| EvalCommitterOptions | - |
| EvalMatrixReport | - |
| EvalSuiteResult | Aggregate view of a suite run. |
| FaultInjectionReport | - |
| FaultScenarioArtifact | One artifact a scenario leaves, JSON or raw text. |
| FaultScenarioObservation | One machine-checkable observation of a driven branch. |
| FaultScenarioReport | - |
| GoldenGraderOptions | - |
| Grader | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| GraderContext | What a grader sees; judge() is the only channel back into the engine. |
| GraderVerdict | One grader's outcome for one case. |
| JudgeGraderOptions | - |
| JudgeSpec | A judge invocation specification. The judge runs through the engine as an ordinary journaled, budgeted invocation; model selection is subject to the router quality floors, and @rulvar/evals ships NO default judge model: weak defaults for judging are forbidden, so the model is always explicit. |
| LexedCitation | One accepted citation occurrence, in document order. |
| LexedRequirementId | One requirement id occurrence with the notation it was written in. |
| MatrixCell | One configuration under comparison. |
| MatrixCellReport | - |
| MeasuredClaimInput | - |
| OrchestratedCase | - |
| RejectedCitationSpan | One span the pattern matched and the lexer refused to count. |
| RubricCriterion | - |
| RubricGraderOptions | - |
| RunBenchmarkOptions | - |
| RunCheckpointOptions | - |
| RunEvalCaseOptions | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| RunEvalSuiteOptions | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| RunFaultInjectionOptions | - |
| RunSweepOptions | - |
| SweepCase | An eval case bound to the taskClass axis of the matrix. |
| SweepCellReport | - |
| SweepModel | One fixed pool member; effort is part of the claim subject identity. |
| SweepPool | - |
| SweepReport | - |
| SweepThresholds | The claim bands. Both effective values must be finite fractions in [0, 1] with weakness strictly below strength (so the bands are ordered and an uninformative mid band exists); runSweepMatrix rejects anything else with a ConfigError before any engine, store, or envelope activity. |
Type Aliases
| Type Alias | Description |
|---|---|
| BenchmarkMetricExtractor | A per-run metric extractor over the run's full event stream. |
| ClaimCorpusClass | The failure classes the eighteenth benchmark shipped, plus the bound classes, plus the nineteenth benchmark's pair (RV1809): 'modality-overclaim' is a mitigation stated as an unconditional guarantee, and 'scope-ambiguity' is a child-only total printed as a whole-workflow figure. The third comparison experiment validated three more (RV3804): 'bound-conflation' lists opt-in caps and unconditional guards as one mode, 'derived-premise' is a derived figure whose premise contradicts the declared input (2,000 slots computed from a 30 minute window where the input declares a 20 minute burst), and 'cost-basis' prints a locally estimated total as the provider's bill. The fourth comparison experiment named the decisive one (RV3909): 'stale-doctrine-echo' is a draft echoing a DOCUMENTED doctrine while the pool holds the diverging source fact, both sides cited; the experiment's answer echoed the retired budget-immutability wording from a guide six weeks stale, and no judge could flag it because the pool never carried the source side. |
Variables
| Variable | Description |
|---|---|
| CLAIM_CORPUS | The shipped corpus, one case per failure class, adversarial by design. |
| DEFAULT_CITATION_EXTENSIONS | Source file extensions a citation may name; lowercase, no dots. |
| DEFAULT_REQUIREMENT_FAMILIES | Requirement id families of the comparison contract (N, R, C). |
| FAULT_SCENARIO_NAMES | The scenario names in run order. |
| JUDGE_VERDICT_SCHEMA | The default judge verdict shape. |
| SWEEP_THRESHOLD_DEFAULTS | - |
Functions
| Function | Description |
|---|---|
| agentTypeRuleHolds | The OQ-09 criterion 2 rule (as amended 2026-07-12): match-or-beat at 105 percent of baseline cost, OR at least 15 points better at 115 percent (the quality branch: the baseline fails cheaply, so the flat bar tightened exactly when the card won on quality). The vacuous-pass guard stays with the caller. |
| canaryFingerprint | The fingerprint alone (the pre-v1.16.2-review surface, kept compatible). Prefer runCanary: its allOk is the drift-flip gate. |
| commitEvalMeasured | Commits measured claims through the eval-committer gate with the documented rebase recipe: on a CAS rejection, re-read current() and retry against the fresh version. Returns the committed version. |
| evalMeasuredClaim | One measured claim; claimExpiry applies the TTL from the decay table. |
| flipStaleOnCanaryDrift | Flips the model's ACTIVE eval-measured claims to stale when their recorded canary fingerprint differs from the fresh one. Claims without a recorded fingerprint have no baseline and stay untouched (the documented no-probe posture); a second run is an idempotent noop. CAS-rebased like every maintenance commit; the retries run no engine work and pay nothing. |
| goldenGrader | - |
| judgeGrader | - |
| lexContractAudit | Lexes one document under the shared contract audit grammar; see the module comment for the doctrine. Malformed options refuse typed. |
| normalizeCanaryOutput | The committed v1 normalization (OQ-06): NFC, trim, collapse whitespace. |
| renderCheckpointReport | The deterministic render for the M12 gate docs amendment. |
| rubricGrader | - |
| runBenchmark | Runs the spec's repeats sequentially and reports the verified series. Throws only for spec defects (invalid repeats, a throwing grader or extractor); everything a run does wrong lands in its record, and a target-run envelope refusal ends the series monotonically with the completed repeats preserved (report.refusal). |
| runCanary | Runs the fixed probe set through the ordinary engine. Probes run sequentially in declaration order, one run per probe, so recordings replay deterministically. Each probe run carries the optional immutable ceiling (options.budgetUsd) and authorizes it against the optional envelope before starting; an envelope refusal records the probe as 'refused' and keeps walking instead of throwing away the completed probes. A non-ok or refused probe enters the fingerprint as !status and clears allOk: callers gate drift flipping on allOk, because a budget-starved or transiently failing probe fingerprints differently without the model having drifted. |
| runClaimCorpus | Runs every corpus case through the pure folds and grades the mechanical expectations. No engine, no model, no journal: the same functions the orchestrator runs, on the same bytes. |
| runEvalCase | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| runEvalMatrix | Runs the same case list against every cell's engine, sequentially and in declaration order (deterministic cassette consumption), and reports per-cell aggregates for side-by-side comparison. |
| runEvalSuite | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| runFaultInjection | Runs the fault-injection scenarios sequentially and reports each driven branch's observation; with artifactsDir, writes one <scenario>.json bundle per scenario (the observation plus every artifact), the experiment-grade trace a review can cite. |
| rungRuleHolds | The OQ-09 cell rule (shared by the per-cell and pooled verdicts). |
| runSweepMatrix | Runs the fixed matrix sequentially in declaration order (deterministic cassette consumption), aggregates per (model, taskClass) cell, emits threshold-crossing claims, and commits them through the eval-committer identity when a store is given. |
| runValueCheckpoint | Runs the checkpoint over the fixed pool. Sequential in declaration order (deterministic cassette consumption when recorded); every cell runs baseline then treatment. |