Evidence

Four servers we did not write.

Each one was fetched from the public registry, pinned to the exact bytes the registry published, and run in a container with no network, no mounts and no access to this machine. Every tool was called with arguments derived from its own schema, and the container's filesystem was read before and after to see what actually changed. Then that was compared against what the server declared about itself.

$ rigorrun verify npm:@modelcontextprotocol/server-memory@2026.8.31

Reproduce any row below with one command.

The Larch Helpdesk pack

The helpdesk pack's qualification

On the Larch Helpdesk twin — two organisations, one helpdesk — 8 scripted black-box agents (1 correct, 7 with one defect each: reading or changing the other organisation's records, disclosing them, following an injection, deleting a customer, emailing the wrong person, or claiming work it did not do) worked the pack's 6 tickets 3 times: 144 cells. An independent oracle read the twin's whole state after every ticket; there was no false pass and no false fail, and the same again at the 0.6.0 tree. Reads are decided from the twin's own access log, leaks from planted markers. It covers the twin, not a live helpdesk.

Helpdesk pack evidence Re-qualification at 0.6.0 The recorded run

And the Stripe pack

The Stripe pack's qualification

Before 0.4.0, and again at the 0.5.0 and 0.6.0 trees, the Stripe refunds pack ran 168 cells on the local twin and 168 on Stripe test mode. An independent oracle checked every verdict; there was no false pass and no false fail any time.

The separate black-box qualification ran 36 of 36 cells against an agent reached over HTTP, with no change to its code and state-only verdicts.

Stripe pack evidence Black-box evidence

What it reached

19 of 37 tools were exercised — 51%. The other 18 are named below with the reason each one was not.

That is the shape of the answer and it is worth stating plainly rather than tuning away: a general harness with no credentials and no fixtures verifies about half of a typical server's surface. A fabricated 100% would be worth less than a measured 51% with the remainder itemised.

What this does not establish

Nothing about the wider registry. 4 servers, all published by @modelcontextprotocol, all well maintained. That is the weakest possible sample.
No security claim about any of them. A server that contradicted nothing here contradicted nothing this harness can observe, on the tools it could exercise, at one digest, on one day.
No claim that unexercised tools are safe. They are unexercised.
No market validation. Our own scan is not a user.

An audit that found RigorRun wrong

In September 2026 RigorRun 0.2.0 was audited against three third-party MCP servers with a ground truth written before each run. It failed every correct agent and passed wrong ones. The cause was found and fixed, a pre-registered requalification against an independent oracle returned GO on every gate, and 0.3.0 is the fixed release. The audit, the fix and the requalification are published as they ran, including the attempt that returned NO_GO.

Read the audit and what changed

npm:@modelcontextprotocol/server-memory@2026.8.31

sha512-ljj/3S4aGjxdNSQWw6gucKKGnTLdBPWxzapyY/MT2tOVyZwvxChvevXSLPwx59nKJAlVpUVW+cOlnVRXRwiqMQ==

exit 0 verified, nothing contradicted

Tools discovered: 9
Exercised: 5 of 9
Contradicted: 0
Undetermined: 0
Isolation: RESET · 2 starts compared
Network egress: none
Server reports: memory-server 0.6.3
Base image: node@sha256:fb4cd12c85ee03686f6af5362a0b0d56d50c58a04632e6c0fb8363f609372293

Not exercised

add_observations — needs a fixture the harness will not invent
delete_entities — declared destructive, and believed rather than tried
delete_observations — declared destructive, and believed rather than tried
delete_relations — declared destructive, and believed rather than tried

npm:@modelcontextprotocol/server-filesystem@2026.8.31

sha512-kKaFkyAh6oipvc9+EAbJ552JafnMnOq5nzmzWkp1jJdBhTAAGpmIpWihUG1+rfNhmEFM98gUZDdCHCDD4v6a7Q==

exit 0 verified, nothing contradicted

Tools discovered: 14
Exercised: 2 of 14
Contradicted: 0
Undetermined: 0
Isolation: RESET · 2 starts compared
Network egress: none
Server reports: secure-filesystem-server 0.2.0
Base image: node@sha256:fb4cd12c85ee03686f6af5362a0b0d56d50c58a04632e6c0fb8363f609372293

Not exercised

directory_tree — needs a fixture the harness will not invent
get_file_info — needs a fixture the harness will not invent
list_directory — needs a fixture the harness will not invent
list_directory_with_sizes — needs a fixture the harness will not invent
read_file — needs a fixture the harness will not invent
read_media_file — needs a fixture the harness will not invent
read_text_file — needs a fixture the harness will not invent
search_files — needs a fixture the harness will not invent
create_directory — needs a fixture the harness will not invent
edit_file — declared destructive, and believed rather than tried
move_file — declared destructive, and believed rather than tried
write_file — declared destructive, and believed rather than tried

npm:@modelcontextprotocol/server-everything@2026.8.31

sha512-5U3OZh8Xq0Li4nA26l6uNvV9/1suMDuWSn+NjLZIhzinhMY6N3A4DCrxVecBGPf2PsNFKTEv5krxGPRwzxc7jQ==

exit 3 ran, established too little to mean much

Tools discovered: 13
Exercised: 11 of 13
Contradicted: 0
Undetermined: 2
Isolation: RESET · 2 starts compared
Network egress: none
Server reports: mcp-servers/everything 2.0.0
Base image: node@sha256:fb4cd12c85ee03686f6af5362a0b0d56d50c58a04632e6c0fb8363f609372293

Not exercised

gzip-file-as-resource — needs a fixture the harness will not invent
simulate-research-query — needs a fixture the harness will not invent

npm:@modelcontextprotocol/server-sequential-thinking@2026.8.31

sha512-/EMH++1j2e+VmHwpibw1IUPHxA0tcQY09Yo5/OIPwW/8irEKLAufJNGSZQhCK18n8TUvV6WwZyQOuaDKxe/CeQ==

exit 0 verified, nothing contradicted

Tools discovered: 1
Exercised: 1 of 1
Contradicted: 0
Undetermined: 0
Isolation: RESET · 2 starts compared
Network egress: none
Server reports: sequential-thinking-server 2026.8.31
Base image: node@sha256:fb4cd12c85ee03686f6af5362a0b0d56d50c58a04632e6c0fb8363f609372293

What the harness itself cannot do

Carried on every record it writes, not only on this page.

The container runtime daemon runs as root. RigorRun reduces blast radius but does not claim to prevent a container escape; an escape reaches the host.
Where a state reading came only from the server’s own tools, it is corroboration and not proof: the server that may be misdeclaring a tool is the one answering the read.
State is read from the container’s writable mounts, which is complete over durable storage and blind to state a process holds only in memory.

Produced by node scripts/build-evidence.mjs at commit ee547ad on 9 September 2026, using RigorRun 0.2.0-dev and Docker 28.5.2+dfsg4. Took 115s. Re-run it and compare.