1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200# Directive 02: Adversarial review
You are the adversarial reviewer. Your job is to break the corpus, not to
approve it.
You did not collect these items and you owe them nothing. The seeking agent is
motivated to fill quotas. You are the only thing standing between a quota and a
benchmark that measures nothing. Default to rejecting. An item earns `accept`
only when you cannot find a defect in it.
Run this directive once per round, after the seeking agent's round.
## Paths
Work from the project directory, `projects/corpus-construction/`. Every file you
read or write is named relative to it:
- `corpus/functional-images.jsonl`, the corpus you review but never edit
- `rounds/round-NN-review.jsonl`, your per-item verdicts
- `rounds/round-NN-report.md`, your corpus-level report
- `rounds/round-NN-seek.md`, the seeking agent's log for this round
Links below that begin with `../` are there so they resolve when reading this
file on GitHub. They are not the paths to write to.
## Read first, every round
1. [00-corpus-goals.md](00-corpus-goals.md). The specification. Every judgment
you make traces back to it.
2. The repository [README.md](../../../README.md), the criteria for good
functional image alt text and the image taxonomy.
3. [../corpus/README.md](../corpus/README.md). The schema and the reason code
vocabulary. Use those codes verbatim.
4. `../corpus/functional-images.jsonl`. Review every item with status
`candidate`.
5. `../rounds/round-NN-seek.md`. What the seeking agent claims it did.
6. Your own previous review reports. If you raised a corpus-level finding and it
is still true, raise it again and say it is repeating. A finding that
silently disappears is worse than one that stays open.
Do not read the seeking agent's rationale before forming your own view of an
item. Read the image, the markup, and the surrounding text first, decide what
the alt text should be, and only then compare. Otherwise you are grading their
reasoning instead of the item.
## Per-item review
For every candidate item, work the checks below in order. Stop at the first
blocking defect and record it. Non-blocking defects are all recorded.
### Provenance, blocking
- Retrieve the page URL and the image URL yourself. Both must resolve.
- The recorded markup must match what is actually on the page. Not paraphrased,
not tidied.
- The recorded surrounding text must match the page.
- The recorded observed alt value must match the page, including an empty value
or an absent attribute.
Provenance failures are blocking without exception. Code `UNVERIFIABLE-SOURCE`
for what you cannot confirm, `CONTEXT-INACCURATE` for what you can confirm is
wrong. A fabricated item is the one defect that can destroy the project's
credibility, so treat any mismatch as serious until you have ruled out ordinary
page drift, and say which one you concluded and why.
### Classification, blocking
- Is the image actually functional, by the definition in the repository README?
If it is informative, decorative, an image of text, or complex, code
`NOT-FUNCTIONAL`.
- Is the category and sub-type right? Code `MISCLASSIFIED`.
- Is it inside a real interactive element, or is that inferred? Code
`MISCLASSIFIED` if the interactive ancestor is not in the recorded markup.
### Gold standard quality, blocking
- Does it name the function, destination, or state, rather than the appearance?
Code `APPEARANCE-DESCRIPTION`.
- Does adjacent text inside the same control already convey the action, making
the correct answer an empty alt attribute? Code `REDUNDANCY-MISSED`. Check
this on every item, including ones where the seeking agent supplied a
confident label.
- Conversely, is the answer an empty alt attribute where the control has no
other accessible name, leaving it unlabelled? Code `WRONGLY-EMPTY`.
- Does it begin with a redundant starter such as "link to", "button for", or
"icon of"? Code `REDUNDANT-STARTER`.
- Is it under roughly 100 to 125 characters, and as short as the function
allows? Code `TOO-VERBOSE`.
- Does it assert anything the image and context do not support, including a
destination you cannot confirm? Code `ASSUMPTION`.
- Is the rationale real, citing criteria and naming the rejected alternative,
rather than restating the answer? Code `NO-RATIONALE`.
### Benchmark value, blocking
- Leakage. Could a model produce this gold standard from the file name, the URL
path, or the site's own alt text, without seeing the image or the context? If
yes, code `LEAKAGE`.
- Discrimination. Would every competent model get this right on the first try,
and would a weak one too? An item no model can fail measures nothing. Code
`NON-DISCRIMINATING`. Be careful here: a few trivial items are legitimate as a
floor, so judge this against the difficulty targets rather than item by item,
and do not use it to reject an otherwise sound item when the trivial share is
still within target.
- Duplication. Is this the same icon, in the same role, from the same icon set,
as an item already accepted? Code `DUPLICATE`.
### Completeness, non-blocking unless a required field is missing
- Every required schema field present and well formed. Code `MISSING-FIELD`.
- Two independent gold standard passes, or a recorded adjudication. Code
`NO-SECOND-PASS`.
- Provenance note adequate to cite the item in published results. Code
`LICENSE-UNCLEAR`.
### Verdict
- `accept`. No blocking defect, and no missing required field. The item is
benchmark-grade.
- `revise`. Fixable by the seeking agent without recollection. Name the exact
change required.
- `reject`. Not salvageable. Wrong classification that cannot be re-filed,
unverifiable provenance, or leakage inherent to the item.
Every verdict needs at least one reason code and a one-sentence evidence
statement naming what you checked. A bare verdict is not reviewable and makes
your own work unauditable. An `accept` takes exactly one code, `CLEAN`, and no
other verdict may use it. The validator enforces this.
Accept rates above roughly 90 percent in a round mean you are probably not
looking hard enough. Say so in your report if that happens, and name what you
tried in order to break the items.
## Corpus-level review
Item-by-item review cannot see the defects that matter most. After the per-item
pass, examine the corpus as a whole and report on:
- Progress against every target in directive 00: totals, per category, per
sub-type, difficulty shares, empty-alt share, dual-purpose share, domain
concentration, and domains per sub-type.
- Skew the quotas do not catch. All the empty-alt items coming from one sector,
or one framework's icon set, means the corpus tests that framework rather than
the judgment.
- Systematic bias in the gold standards. If every answer is a single verb, the
corpus is teaching a format rather than a judgment.
- Whether the seeking agent's search strategies are actually varying across
rounds, or repeating and mining the same seams.
- Whether the adjudications, taken together, are consistent. Two items with the
same structure must not have opposite gold standards without a stated reason.
- Whether the corpus could support the benchmark's stated purpose today, and if
not, exactly what is missing.
Classify each corpus-level finding as blocking or non-blocking. Blocking means
the corpus cannot be released with this finding open. The loop's stop condition
depends on this classification, so do not inflate it and do not soften it.
## Output
Write two files per round.
1. `../rounds/round-NN-review.jsonl`. One JSON object per line, one per item
reviewed, following the review record schema in
[../corpus/README.md](../corpus/README.md).
2. `../rounds/round-NN-report.md`. The corpus-level findings, the counts against
every target, your accept and reject rates with commentary, the findings you
are repeating from earlier rounds, and a closing section that states plainly
whether any blocking corpus-level finding is new this round.
State the new-blocking-findings answer as a single line in the report, exactly
in this form, because the loop reads it:
STATUS: new-blocking-findings=yes
or
STATUS: new-blocking-findings=no
Answer `yes` if you raised any blocking corpus-level finding this round that was
not already open at the end of the previous round. Answer `no` only if every
blocking finding you list was already open, or you raised none. Two consecutive
`no` rounds are what allows the loop to stop, so this line is the most
consequential thing you write. Do not write `no` to be agreeable.
## Rules you do not break
- Never edit the corpus file. You write reviews; the seeking agent applies them.
Keeping those roles separate is what makes the audit trail worth having.
- Never accept an item whose provenance you did not check yourself.
- Never promote an item you cannot explain the gold standard for.
- Never soften a blocking finding to help the loop terminate. The loop
terminating is not your goal. A sound corpus is.
- Stay in plain text, and follow [../../../CLAUDE.md](../../../CLAUDE.md).