ARGUS·LAB
Console
repo

Open beta — 33 live engine runs free (3 welcome + 30 beta pack), no card. Every run starts with the canonical payment gate: fail it and the run is refunded (net 0) — you learn for free whether your payment layer stalls even against a well-behaved counterparty.

Dashboard

adversarial testing lab — AI agents that move money, and the rails they settle on

We break AI agents that move money.
We sign our findings, not our opinions.

Argus runs pre-registered, black-box batteries against buyer and merchant agents — and the payment path they rely on: x402 payment flows, credit and settlement handling, idempotency guarantees. An agent is only as solvent as the code that signs for it. Every verdict is a sealed record in a public hash chain that anyone can verify in one command.

Explore the registryCommission a roundDashboard
sealed records, hash-chained
18
sealed records, hash-chained
systems under test
6
systems under test
pre-registered probes
60+
pre-registered probes
findings retracted
0
findings retracted

the registry

A chain of seals, not a list of claims

Each node is an Ed25519-signed record chained to the hash of its predecessor: finding or battery, verdicts, findings, hash-pinned evidence. Select a seal to inspect it.

latest seal

registry intact · 18 records · verify-registry.mjs ↗

argus-seal-00162026-11-01npm: cloudflare/agents@0.32.0 + @circle-fin/x402-batching@4.0.0 + @coinbase/cdp-sdk@1.60.0 (T3 battery chain v2, offline-by-default)

T3 battery chain v2, SUT-3 — @coinbase/cdp-sdk@1.60.0 guardrails offline battery: 9 PASS / 2 FAIL; CDP-unique defenses verified (timeout cap, malformed-requirement rejection, allowedPayees); FAIL: W-CAP2 dust, W-NO-CONTROLS default-unsafe

verdicts

SUT-3: MIXED (9 PASS / 2 FAIL)

chain 3c428d07 → 069e2e05 ✓ · Ed25519 · prev_record_hash → chain head

latest correction argus-correction-0017 · CORRECTION-0012-01 — seal #0012 fixture label erratum: probe_start.control_under_test stale template default; build/install receipt added ↗

hash-pinned evidence (1)

t3-battery-export-hashes.json

$ git clone github.com/igor-vii/Argus-Agent-Test-Lab && node tools/registry/verify-registry.mjs → registry intact: 18 records, chain head 069e2e05…

attestations/records — hash chain explorer
argus-correction-00172026-11-08argus-seal-0012 (SUT: x402-foundation/x402 PR #3920, @x402/core 2.28.0+pr3920@e993bef)

CORRECTION-0012-01 — seal #0012 fixture label erratum (S-BS-06 cycle 3): stale control_under_test default; build/install receipt added

Correction sealing the reviewer's evidence-gap note on the S-BS-06 cycle-3 battery (seal #0012): the fixture label probe_start.control_under_test carried a stale template default (built from @4232cf6) while the same fixture's pull_request.head and tarball hash record the true binding - @x402/core 2.28.0+pr3920@e993bef. A build/install receipt (docs/block-b/fixtures/S-BS-06/receipts/c3-build-receipt.json) now ties checkout, tarball and installed path/version in one reader-verifiable file, and future batteries derive the label from the recorded pull_request.head so it can never lag the build pin again. Label-only: seal #0012 verdicts (17/19 PASS; A10a/A10b identical-offer continuity FAIL) are unaffected - the identical-offer window is filed upstream as x402#4198.

verdicts

Label erratum CORRECTION-0012-01: applied — receipt-bound, append-onlySeal #0012 verdicts after correction: unaffected — 17/19 PASS; A10a/A10b FAIL

1 hash-pinned evidence file

findings

— none recorded on this record —

record #0017 ↗x402#4198 (identical-offer window) ↗

$ node tools/registry/verify-registry.mjs → registry intact: 18 records, chain head 069e2e05626c3f3e… records ↗

method

Four rules that make a verdict worth something

The lab's credibility is structural: it comes from what a probe must go through before anyone is allowed to call it a failure.

01

Black-box by construction

The SUT is always the published artifact — npm dist, exact pin, unmodified. No source access, no vendor cooperation required. If it can't be shown against the artifact users actually install, it isn't shown.

02

Pre-registered before the run

Required behaviors and verdict rules are fixed in committed matrix files before a single probe fires. No post-hoc reshaping of what a failure means — the rig can't drift toward the conclusion.

03

Offline by default

All traffic on 127.0.0.1 against adversarial stands and honest mock backends modeled from the vendor's own contracts, each decision citing a vendor source line. No vendor-operated service is touched without written authorization.

04

Sealed and hash-chained

Every record is signed with Ed25519 and chained to the previous record's hash. Anyone can verify the whole chain from the public repo in one command. Seals attest run validity and findings completeness — not bug-freeness.

verdict taxonomy

FAIL
Vendor-side failure under the probe — the SDK or platform loses money or integrity under the tested fault.
HOLD
The SDK rejected or limited the fault — a genuine guard fired where the battery expected one.
PASS
The platform held beyond the documented scenarios.
OBSERVED
Observed behavior matches the vendor's published disclosure — an honest-posture result, not a defect.
DX
Developer-experience finding: no direct money-loss path, but the tooling hides a hazard the runtime itself flags.

cases

Every verdict, public and linked

Six fronts so far — from a live merchant agent to the SDKs and rails that sign and settle on their behalf. Each card links back to sealed records and public threads; nothing on this page exists outside the registry.

x402-foundation

x402 reference client + PR #3920 core · @x402/core 2.28.0

A11 closedA10c/d PASSA10a/b open

Issue #3866 rounds 2–4 + PR #3920 cycles 1–3: A11 verified closed, conflicting-offer invariant holds, normalization delta verified.

  • Origin finding (#0000): the reference client signs any server-declared 402 amount.
  • Round 2–4 engagement on issue #3866 with control batteries and delta probes.
  • Cycle-2 at PR head b1a4d8ce: non-atomic amount rejection verified closed; binding race (signer ×2) reproduces.
  • Cycle-3 at e993bef (17/19 PASS): conflicting-offer races A10c/A10d PASS — one commitment + one pre-signer rejection; case-variant recipient normalization A12/A13 PASS.
  • Author accepted the cycle-3 results; remaining gate: upstream maintainer integration review.
seals#0000#0003#0004#0008#0012
Issue #3866 PR #3920

Upstream engagement · cycles 1–3

BlockRunAI

@blockrun/nano-client · 0.8.0 (npm exact pin)

3 FAIL3 HOLD

A signature over the attacker's payTo, ×50 with no cap, and a README claim its own advice breaks.

  • F1: shape-only payTo check signs authorization over attacker-controlled recipient.
  • F2: no default spend cap — ×50 of quoted price signed on default config.
  • F3: lost-response dead-end; the README's own recovery advice routes into a double charge.
  • Earlier report #15 closed as not planned; the sealed battery answers the closure premise.
seals#0006
Issue #18 Closure #15

Public · awaiting vendor

Nevermined

@nevermined-io/payments · 1.13.0 (npm exact pin)

7 FAIL1 DX3 HOLD

Seven failures in credit and settlement handling — including a double charge on retry.

  • F1: lost-response retry double-charges on the default token; no idempotency key, no settlement query.
  • F2: single-use guard is per-process and reactive — concurrent race delivers a free service.
  • F3: any settle failure except one special case delivers the resource unpaid — with a receipt.
  • The class was already flagged in the vendor's own tracker (#374/#367, open since June); the battery makes it sealed and reproducible.
seals#0007
nevermined-io/payments

Sealed · publication ready

Inngest

inngest + @inngest/test · 4.26.0 / 1.0.0 (npm dist)

0 FAIL2 DX

Thirteen probes, zero failures: behavior matches the vendor's own disclosure on every probe.

  • Commit-then-crash window verified live against the documented A3 model, including supervisor-restart variant.
  • Event-ID dedup holds under a 12-way concurrent race; prescribed mitigations verified effective.
  • Two developer-experience findings in the vendor's test engine; triaged upstream (Linear EXE-2315).
  • Lab close-out with seal #0005 published to the issue thread on closure.
seals#0005
Issue #5091

Closed · close-out published

tripl3tr3s

mcp-tool-idempotency · 4d665ae → v0.2.0

3 fixed10/10 verified

Reported privately, fixed publicly, verified independently — credited in SECURITY.md.

  • Three durability windows: lost response after commit, crash after commit, cross-instance race.
  • Coordinated disclosure before publication; maintainer shipped owner-gated claim leases.
  • Ten-window battery on v0.2.0: 10/10 PASS, suite 91/91 at tag.
seals#0009
mcp-tool-idempotency

Fixed · verified · credited

AgentMiddleware

amw-paypal-agent-demo · live demo (Block D)

REPRODUCED1 high finding

The lab's first live round — and the clean-machine reproduction that set the standard.

  • S10/S11 executed live against a real merchant agent demo with raw evidence captured.
  • Independent clean-machine reproduction from committed sources: verdicts identical.
  • The two-seal discipline (run + reproduction) applied ever since.
seals#0001#0002
amw-paypal-agent-demo

Live round · reproduced

engage

Run your code before your users do

Two working tracks, both ending in the same artifact: a sealed verdict anyone can verify. Engagements begin with written scope; nothing about an engagement changes a verdict.

A

Coordinated production round

Point us at a live agent that moves real money; we run the battery under written scope authorization: funded test wallet, agreed time window on 48 hours' notice, a no-debit/refund arrangement for probe transactions, full request capture shared with you. Deliverable: a sealed production-round report — either documenting that the production endpoint held against the full battery, or documenting the exact boundary where defects become live money movement.

from $990 / round

B

Fix verification

Our batteries become your acceptance rig. You ship a patched artifact; we re-run the sealed suite against the published release and seal PASS/FAIL per finding. A clean seal is an independently checkable artifact you can cite to integrators and in your security narrative.

Per-finding sealed verdict

terms of engagement

  • Fixed per-round fee or a short retainer, agreed before any run.
  • Reports and seals remain the lab's to publish; vendor responses get equal space.
  • Production rounds require written scope, a funded test wallet, and a no-debit arrangement.
  • Private resolution is welcome — the seal is not a weapon, it is a receipt.
Start via GitHubOne-pager (PDF)argus.testlab@gmail.com

github.com/igor-vii/Argus-Agent-Test-Lab · issues open

  • §1Praise in public where it is earned — as loudly as failures get stated.
  • §2Break in public where it is broken — with line numbers, not adjectives.
  • §3Vendor responses get equal space alongside every published finding.
  • §4No vendor operates our stands; no engagement buys a verdict.