EqualifyEverything / equalify-iris-bench

A harness for asking a live Iris deployment the three questions a 2000-URL list of real-world PDFs can answer: how often does it succeed, how long does it take, and what does it cost per page. It lives outside equalify-iris because it is a client of the public API with dependencies and gigabytes of cached PDFs the service should not carry, and because it has to be pointable at any deployment — including the live one — without a release. Three stages, each resumable, because a campaign of this size will be interrupted: prepare downloads every URL (Iris does not fetch them — /v1/sessions is multipart only), screens it, and splits what is too long. run submits, polls, and captures status, log, diagnostics and output per document. report turns that into results.jsonl and a summary. The decisions worth arguing with: Nothing hardcodes a limit. Page caps, image ceilings and rate limits all come from GET /v1/limits at run time, which is the reason that endpoint exists and deliberately does not name the model behind its numbers. A harness that baked them in would prefilter the corpus against last month's deployment and report the difference as a change in Iris. The one exception is the rasterization DPI, which Iris does not publish; it only ever produces a risk flag, never an exclusion, so a stale guess costs a failed run rather than a missing document. PDFs are detected by magic bytes, not content-type. Servers hand out PDFs as application/octet-stream and sign-in pages as application/pdf, and only one of those errors is visible from a header. Oversize documents are split, not dropped. An unfiltered real-world corpus skews long, so excluding everything over the page cap would bias the whole accuracy baseline toward short documents. Chunks are labelled, because a chunk's review score is not comparable to a whole document's — the reviewer saw a document with no beginning — and the report keeps the two populations apart. At most four chunks come from one PDF, and the dropped tail is printed: a bound that does not announce itself reads as "we covered everything". Failed runs are captured in full. A corpus's most useful artifacts are from the documents that went wrong, and the log is the only account of why, while the diagnostics still carry what the run spent getting there. Success is reported twice. Once over items submitted, and once over every URL the CSV contained — where a URL counts only if every chunk of it came back. The second number is lower and is the one an outside claim has to survive, because 404s, login walls and encrypted files are part of the world. Accuracy is not agreement with ground truth; nobody has hand-authored the correct HTML for 2000 PDFs. It is the review loop's own signals (iterations, unresolved issues, dropped links, id collisions, failed page verifies) plus one independent check: a local axe re-lint of the delivered HTML. That re-lint exists because of a real gap — Iris's FINAL lint, the one describing the document actually handed back, goes to the deployment-wide quality tally and has no per-session endpoint. So src/lint.mjs is a rule-for-rule port of src/pipeline/lint.ts with axe-core and jsdom pinned exactly, and a test runs both linters over the same fixtures and requires identical verdicts. Structure counts sit beside it, because a clean lint on a nearly empty document is a failure no rule catches. Cost is computed here and never in Iris, at a stated date under stated assumptions. Every rate carries a source and a checked date, an unrecognized model costs null rather than zero and is counted loudly, and Bedrock ids are flagged estimate_only — the live deployment is billed by AWS at AWS's rates, so these dollars compare documents against each other and are not an invoice. Pacing matches the deployment rather than fighting it: concurrency defaults to the server's max_concurrent_runs, submissions are spaced from the published upload_per_minute, 429s honour Retry-After, and a 400 is not retried because it is a verdict about the file. Closing sessions is opt-in: on 2000 documents that is 2000 regression-fixture captures against a deployment that has processed seventeen, which should be a decision and not a side effect of measuring. npm test exercises all three stages against a stub of the API over a hand-written multi-page PDF — no token, no network, no queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>