EqualifyEverything / equalify-iris-bench

fix(report): count what the campaign spent, not what it delivered
The first real bench put $9.26 through the deployment and got 25 pages back. The report said $2.50, because `cost` summed delivered runs only — and the whole point of the number is to answer "what will 2000 URLs cost", which is a question about spend, not about the subset that worked. Iris reports a failed run's tokens in its own diagnostics, and they are not small: a run that extracted 25 pages and then died in review has already paid for all 25. Here two such runs were 73% of the bill. So `cost.tokens` and `by_agent` now cover every attempt, with `tokens_delivered` kept separately for comparing documents. `usd_on_failures` names the part that bought nothing, and `usd_per_delivered_page` — everything spent over the pages anyone received — replaces the delivered-only median in the per-1000-pages projection. That is 3.7x apart here ($0.0999 vs $0.3704), and the projection is the line people quote. `usd_per_page_p50` stays, relabelled as what a page costs when nothing goes wrong, because that is the right number for ranking documents against each other and the wrong one for a budget. The smoke test had encoded the old behaviour as 16400 tokens — two of three runs at 8200 each — so it asserted the bug. It now asserts 24600 and that the two figures differ in the direction they must.
Blake Bertuccelli-Booth Blake Bertuccelli-Booth committed on Aug 24, 2026, 04:34 PM
Showing 2 changed files +54 additions -17 deletions
M src/report.mjs +42 -13
Browse files at this commit →