How it works
Define your agent’s scope. Check what actually happened.
In the Helpdesk twin, check which organisation was read, which order changed and where customer data was emailed. In the Stripe pack, inspect the resulting payment state. Both illustrate the same approach: the transcript is shown beside the result, but it does not decide the verdict. Coverage depends on what the environment can actually expose.
One bundled example: Stripe refunds
- 01
Set up the Stripe pack
npx rigorrun stripe twinnpx rigorrun stripe init --twin --yesThe twin is a local stand-in for Stripe, so this path needs no keys. Leave it running, then initialise the pack in another terminal.
- 02
Point it at your agent
npx rigorrun agent add --project <id> --name my-agent --black-box <url> --claim-path messageNo code changes. RigorRun sends one ticket at a time to the endpoint your agent already serves; each ticket names the customer's email and a charge id.
- 03
Gate every change
npx rigorrun gate --project <id> --report report.htmlThe verdict comes from what Stripe holds after each ticket, not from the reply. Exit 1 stops the build when a case fails, and the report shows what happened beside what the agent said it did.
What is being built next
What the agent read and what it sent are checked today on the Larch Helpdesk twin, from the twin’s own access log and marker strings placed in another organisation’s data. Checks on the agent’s tool calls themselves are being built next.
Permission tests on your own system — two tenants, your MCP server or API — are not built yet; that path comes next.
For a system with no pack
This is the general path from v0.2 for a system with no pack: demonstrate the job, confirm the proposed rules, and run the resulting diagnostic suite against the agent. It is not an off-the-shelf custom permission pack or evidence that every API is supported.
Generated suites need a current quality assessment before the local MVP accepts a release gate. The development safeguards are not yet an npm release; existing public replays and qualification reports describe their recorded builds, not the changed MVP.
- 01
Do the job once
RigorRun reads your system before you start and again when you finish, then works out what the rules must be from what changed. It reads the shape of the data and never the names of things, which is what stops it only working on businesses it has already seen — and is also why a few things it genuinely cannot know.
A job that leaves no trace cannot be learned. If nothing observable changed, there is nothing to derive a rule from.
- 02
Rule on what it worked out
Each proposed rule shows the evidence behind it, so you disagree with the evidence rather than with a verdict. The ones it is least sure about come first. RigorRun deliberately proposes more than it expects you to keep: missing a real rule is far worse than proposing one you do not want.
Rejected rules are excluded from the suite. Keep intended boundaries explicit; rejecting a real safety rule reduces what the test can establish. Review alone does not establish suite quality.
- 03
It builds the suite
The confirmed rules become executable cases — the happy path, boundaries, missing preconditions, duplicate actions, policy violations, malformed input, tool failures, timeouts, unexpected state, and prompt injection hidden where an agent would actually meet it.
How many categories appear depends on what the job touched. A two-step job produces fewer, and RigorRun says which it could not build and why.
- 04
Your agent runs it
RigorRun hands your agent one task at a time and a URL to work through. It works however it normally works. The private half — the checks and the reference plan — is never sent to the agent, so there is nothing to read the answer from.
If your agent already speaks MCP, the integration is roughly ten lines. If RigorRun cannot start it, you can pull work with a key instead, so the agent can live behind a login.
- 05
The verdict comes from the system
After each case, RigorRun calls the reads you nominated and compares what it finds to what should be true. The agent’s own account of what it did is displayed, labelled, and never scored.
Every verdict carries how strongly it was verified and whether the cases were isolated from each other. A verdict against a real system is PARTIAL, and says so on the same line as the answer.
- 06
Establish quality, then gate the next change
A generated suite needs an explicit quality assessment in a resettable test environment. Reference replay and independent defect controls test whether the suite can distinguish expected behaviour from faults. Then rerun a current accepted suite when a prompt, model or tool changes.
Assessment makes repeated test writes. A gate must not silently start it. The local MVP refuses generated release gates with missing, stale or inadequate evidence; it does not block production actions.
About the numbers on this page
They come from one worked example: a support agent processing a single refund in a CRM. Every count is what the pipeline produces when it runs, written into the page by a script and checked in CI against a fresh run, so it cannot quietly drift.
That CRM is synthetic — an application written for this example, with invented customers and orders. The pipeline running over it is the one that ships. For results against software nobody here wrote, the evidence page is the honest place to look.