EqualifyEverything / benchmarks-ai-alt

Review round 2: make promotion possible and deterministic
Round two of adversarial review found a blocking design defect: directive 02 forbids the reviewer from editing the corpus and directive 01 forbids the seeking agent from promoting its own items, so no item could ever reach `accepted` and the loop could never terminate. - Add tools/apply-verdicts.mjs. Applies a review round's verdicts to the corpus all or nothing, idempotently, and independently refuses any promotion the specification forbids: a leaky item, or one with a single gold standard pass and no adjudication. Self-tests with seven cases. - Wire it into run.sh between review and the final validation, and add a self-test case proving a candidate is promoted by the tool and not by an agent. - Gate fix: the "no unresolved blocking finding" criterion now uses each item's most recent review rather than the most recent round, so a blocking finding cannot age out because later rounds reviewed other items. - Schema fix: a review record whose `round` disagrees with its file name is now an error, since it would otherwise be applied to the wrong round and skew the gate. - run.sh now exits 2 when the final validation reports schema errors, rather than treating them as "goals not met yet" and starting another round. - Reconcile the directives, the corpus schema, and both READMEs with the new promotion step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Blake Bertuccelli-Booth Blake Bertuccelli-Booth committed on Aug 27, 2026, 05:42 PM
Showing 9 changed files +389 additions -29 deletions
Browse files at this commit →