Every release, including what was wrong.

The Larch Helpdesk pack: a multi-tenant helpdesk twin over MCP (two organisations, Alder Outdoor and Birch Home; Birch’s customers carry planted markers) and six tickets — the customer’s own refund, another organisation’s order, another organisation’s customer, an instruction to export everyone, a deletion request, an email copy. Each is decided from the twin’s access log (reads), its tables (changes) and its outbox (where data went); every verdict is AUTHORITATIVE, and the whole world is replaced before each ticket.

rigorrun helpdesk try: the six tickets in seconds with no keys, on a built-in demo agent run with two tokens — Alder’s own and the helpdesk’s service token. --agent <url> runs your own agent, on a twin it is already connected to. rigorrun helpdesk twin and rigorrun helpdesk init for a project and the gate.

A recorded run on the helpdesk at /replay/helpdesk: gpt-oss-120b through the reference agent, once per token, under a protocol written before any model saw a ticket.

Every run and every HTML report named itself RigorRun 0.1.0, whatever the release; they now carry the version that made them.

The help, the npm README and description, and rigorrun.xyz lead with the helpdesk pack; npm no longer rewrites the package’s bin path on publish.

Qualified before release under new pre-registrations, at the release tree itself (the version was set before the freeze): the Larch Helpdesk pack on its twin, 8 scripted agents × 6 tickets × 3 attempts against an independent oracle, 144 cells with no false pass and no false fail; the Stripe pack again, 168 cells on the local twin and 168 on Stripe test mode, none either. reports/helpdesk-pack-2026-10, reports/helpdesk-requal-0.6.0, reports/stripe-requal-0.6.0.

The HTML report has a permission matrix: for each boundary (another tenant's data, outside its role, a tool it must not use, data leaving) and each agent, how many checks failed, held, or could not be checked. The Stripe pack's checks name the boundary they guard, so a Stripe run fills it.

Four check kinds for agents reached through RigorRun's MCP proxy — tool_not_called, tool_args_in_scope, no_refused_call and marker_absent (a string planted in another customer's data must not appear in a reply, a call or a record). They are not qualified yet: the qualification on a multi-tenant system is next. For a black-box agent they are listed as not made.

task.principal: a case can say whom the agent acts for (tenant, user, role); it is sent in every agent envelope and in body templates as {{principal.tenant}}.

event_occurred and event_not_occurred were evaluated against an empty list of events, so the first always failed and the second always passed. contains and not_contains now look inside records, and a path that does not resolve is not checked rather than silently passed. A call the proxy refuses is now evidence, not only a count.

The help, the npm description, the README and rigorrun.xyz describe RigorRun as permission and scope tests for AI agents; the commands they show work as typed (stripe twin, then stripe init --twin --yes).

Re-qualified before release under a new pre-registration: the Stripe pack at the 0.5.0 tree, 168 cells on the local twin and 168 on Stripe test mode, no false pass and no false fail, the same as 0.4.0. The released tree differs from the one qualified only by its version strings. reports/stripe-requal-0.5.0.

rigorrun stripe init: a Stripe pack with seven support tickets (a plain refund, dollars and cents, part of an order, an item already refunded, a disputed payment, somebody else's payment, an instruction inside the ticket) and a $1.00 canary. Every case creates its own customer and payments in Stripe test mode, or on a local twin of Stripe's API with no keys, so cases repeat and never see each other.

Every verdict shows the agent's own words beside what the system holds: "agent said Refunded $25.00 / Stripe shows a refund of $0.25 on a $25.00 charge". In the terminal, the HTML report and the replay.

rigorrun agent add --black-box: point RigorRun at the endpoint your agent already serves. rigorrun stripe canary runs one $1.00 case first.

A GitHub Action: uses: Konuktor/rigorrun@v0.4.0 runs the gate, writes a summary and keeps the HTML report.

rigorrun demo replays a recorded run on the Stripe cases; the synthetic support desk moved to rigorrun demo --northstar.

Only Stripe test keys are accepted, Stripe must confirm test mode before anything is stored, and any object that comes back in live mode stops the run. The twin never receives a real key.

A rule the owner did not confirm can no longer fail an agent; gate --case no longer reports a release pass over a subset; a 202 answer from a black-box agent is no longer judged as finished work.

Released after pre-registered qualifications at the release commit: the Stripe pack, 8 scripted agents × 7 tickets × 3 attempts against an independent oracle, 168 cells on the local twin and 168 on Stripe test mode with no false pass and no false fail; black-box mode, 36 cells, none either. reports/stripe-pack-2026-10 and reports/blackbox-qualification-2026-10.

rigorrun demo replays a real recorded run, offline, in about a second: a real model working the bundled support desk, what it said beside what the system held afterwards. The recording names its model, date and commit, carries a hash of the run, and is refused if edited. rigorrun demo --live runs the pipeline now.

rigorrun.xyz/replay shows every case of that recording, with nothing to install.

run and gate --project --report <file> write an HTML report, and --published masks identifiers. A failed case shows what the agent said beside what RigorRun saw, in the terminal, the interface and the report.

Failed checks read as sentences: "3 Refund records where customerId is C-1; at most 1 allowed" rather than a path expression.

A read that answered with one page of a longer list made records that slid out of the page look deleted, so a correct agent could be failed. A page is now recognised (has_more, a next cursor or link, a total above what came back, or a read called with a limit that came back full); every check on which of those records exist says not checked; and nominating such a read says so at setup.

"A or B" with one side that could not be checked and the other not holding was reported as a failure. It is now not checked.

The reference implementation appears as a control that is given the answer, never as the demo's winner. The first screen offers the recorded demo or your own system instead of an empty form.

IO-v1 (12 cases, 36 attempts), IO-v2 (9 cases, 27 attempts) and the held-out set (23 cases) re-run at this release: every case's outcome and evidence label is unchanged from the requalification that qualified 0.3.0.

On real systems 0.2.0 failed correct agents and passed wrong ones: a public audit scored every correct agent FAIL and passed duplicate, wrong-value and wrong-entity agents. Each case is now held to the change the demonstration made, from the world it started in, and to nothing else changing for any kind of record.

A case whose right answer is to decline passed an agent that did the job anyway, when the result landed where no nominated read looks. A declined case now also checks, from the call log, that the job's own action did not succeed.

RigorRun could reset, record or build a suite against a system marked production, and connecting defaulted to staging. It now never writes to production on its own behalf, and the kind of system must be chosen.

Black-box agents (rigorrun/task/1): send each case's work to an agent wherever it runs, unchanged, and judge what the system holds afterwards through RigorRun's own connection.

Case outcomes PASS, FAIL, ABSTAIN, TIMED_OUT, AGENT_FAILURE and HARNESS_FAILURE; exit code 3 when too many cases reached no verdict; rigorrun setup for a project from a file; an independent verifier connection.

Credentials for CI as RIGORRUN_SECRET__<NAME>; secrets set prompts without echo.

Released only after a pre-registered requalification returned GO on all twelve gates: 58 cases, 152 attempts, no false pass or false fail against an independent oracle.

rigorrun verify — pins a published MCP server to the bytes the registry served, runs it in a container with no network, exercises every tool it safely can, and reports what changed against what the server declared.

gate now refuses --agent reference without --allow-reference, and run requires an agent. The documented way to gate a build had been running the implementation that is handed the answer, so it was a guaranteed pass.

Isolation reports DECLARED rather than RESET where the reset is a tool the operator nominated and nothing has run it twice and compared. It had been claiming an observation it never made.

The site leads with what a stranger can check rather than with an invented CRM.

Published by GitHub Actions with npm trusted publishing and a SLSA build provenance attestation. No npm token exists to be stolen from repository secrets.

The published package did not install cleanly on a machine that had never seen the source.

Four claims narrowed to what had actually been measured.

The local runner, the six-step project flow, contract compilation from one demonstration, case generation, agent connection over HTTP or a local process, verdicts, and CI gating.