Benchmarks

Measured, not marketed. Every quantitative claim traces to a harness you can run offline in fixture mode — here's each one and how to reproduce it.

Every quantitative claim about Sirius traces to a harness in bench/. Each one measures a single number, prints it, and exits. All of them run offline in fixture mode — the sirius binary, the parent CLIs, and a live ledger are not required — so the numbers below are met in simulation today and become live measurements once the binary lands.

bun run bench/soak.ts
bun run bench/gate-escape.ts
bun run bench/receipts.ts
bun run bench/wasted-work.ts
bun run bench/claim-mode.ts

What each harness measures

The methods

soak.ts — claim integrity. Drives 4 workers through the full loop against a deliberately contentious shared pool, interleaved tick by tick so workers really hold resources while others try to claim them. An independent shadow auditor — separate from the lock under test — counts any moment two workers hold the same resource. It reports double_claims (target 0) alongside the count of correctly rejected contended claims, proving the contention was real.

bun run bench/soak.ts --duration=30m           # full 30-minute-equivalent workload
bun run bench/soak.ts --duration=30m --realtime  # pace to wall-clock

gate-escape.ts — gate escape rate. Replays a corpus of known regressions (default 95, across 4 repos) through a simulated affected-tests gate. An escape is a regression the SAFE tier fails to select a test for. The simulated miss rate is tuned to Hayvenhurst’s observed ~1.8% floor, so the number sits realistically just under the 2% target rather than a suspicious 0%.

receipts.ts — provenance coverage. Seeds a ledger of completed iterations, every one filing a two-way receipt, plus non-done outcomes that must not count against coverage. Coverage = done iterations with a two-way receipt ÷ done iterations. Pass --broken to drop one reverse stamp and prove the metric detects sub-100%.

wasted-work.ts — wasted-work ceiling. Sums tokens on iterations whose outcome is not completed. The cost model reflects the claim-order payoff: a 409 release costs almost nothing (backed off before the agent ran) while a post-work failure costs a full agent pass.

claim-mode.ts — policy comparison. Replays one contentious workload under all three claim modes and compares tokens per completed issue. The honest result: at low contention never-claim wins and blanket claim-first is a needless tax; from moderate contention up, adaptive is cheaper than both static modes. Adaptive is deliberately not a universal winner — at very low contention it trails never slightly while it learns, which is the truthful outcome.

Why fixture mode is honest

Each harness models the ledger and the parents’ hard-lock and gate semantics faithfully (bench/lib/ledger.ts mirrors the ledger schema 1:1), and ends with a machine-parseable METRIC line that CI asserts on. That proves the harnesses are sound and the targets are met in simulation. What it does not yet claim is a live measurement on your repo — that arrives with the binary, and the same harnesses gain a --live mode to produce it.