1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661name: Quality Report
# Weekly reading of a live deployment's own output quality, filed as issues.
#
# The app already measures itself on every run — how many reader/editor rounds a
# document needed, which axe-core rules its HTML still violates, whether a link from
# the source PDF went missing — and until now it wrote all of it to a per-session log
# and nothing ever looked across sessions. This job looks. It reads `GET /v1/quality`
# and compares a handful of rates against thresholds held *in this file*, and
# opens one issue per crossed threshold. `issue-to-pr.yml` then ranks those issues
# alongside every other open issue and may open a PR against one. That is the whole
# loop: the app's measurements become work, without a person typing feedback first.
#
# **This job spends no model tokens.** It is curl, jq and arithmetic. Every judgement
# call it could make — is this rule worth fixing, which prompt is wrong, what should
# change — is deliberately left to the workflow that is already good at that. Keeping
# it dumb is also what makes it safe to run weekly against a production deployment.
#
# **What it may read is constrained at the source.** The endpoint returns aggregates
# only: counts, rates and axe-core rule ids. It cannot return document text, because
# what this job copies into a PUBLIC issue is whatever the endpoint said, and the
# documents behind those numbers are user uploads — at the reference deployment,
# student records. See `src/routes/quality.ts` for the constraint stated where a
# reviewer adding a field would meet it.
#
# Setup — both are required, and the job declines rather than fails without them:
# - variable QUALITY_URL — the deployment's ORIGIN, e.g. `https://iris.equalify.uic.edu`.
# Origin only: the request below appends `/v1/quality`, so a value ending in `/v1`
# produces a 404 that is indistinguishable from a deployment which never opted in.
# A trailing slash is tolerated. A repository *variable*, not a secret: it is a
# public hostname. Must be https, checked below, because the token is a bearer
# token and http would put it on the wire in cleartext.
# - secret QUALITY_TOKEN — must equal that deployment's `server.quality_token`
# (`IRIS_QUALITY_TOKEN` in its environment). Generate with `openssl rand -hex 32`.
#
# gh variable set QUALITY_URL --body https://iris.equalify.uic.edu
# gh secret set QUALITY_TOKEN
#
# Verify with `gh workflow run quality-report.yml -f dry_run=true` rather than waiting
# for Saturday: it reads the tally and prints what it would file without filing it. A
# green run saying the window is below the minimum document count proves the URL and
# token work — that is a success, not a failure. `docs/ci.md`'s "Weekly quality report"
# section has the same procedure in prose.
#
# It needs no Bedrock role, no OIDC, and no Node. The one thing it does take from the
# repository is `.github/scripts/quality-body.jq`, the program that writes the body of
# each issue; the thresholds, the dedupe and the cooldown are all still here.
on:
schedule:
# 20:00 UTC Saturday = early/mid afternoon Central. Saturday specifically, and
# this is the only interesting thing about the schedule: `issue-to-pr.yml` runs
# Sun–Wed, so an issue filed Saturday afternoon is on the candidate list for the
# very next triage run. File it Sunday evening instead and it waits a week.
#
# Weekly, not daily, and the window is what forces that. The tally is a 30-day
# rate: reading it daily produces 30 nearly-identical numbers, and a fix landing
# on Monday cannot visibly move a 30-day rate by Tuesday. Daily polling would add
# noise, not resolution.
- cron: '0 20 * * 6'
workflow_dispatch:
inputs:
days:
description: 'Window in days to read (default 30, max 365)'
required: false
dry_run:
description: 'Read the tally and report it — file no issues'
type: boolean
default: false
# One at a time. This job writes issues, and two concurrent runs reading the same
# tally would both find the same threshold crossed and both file it — the dedupe below
# reads the issue list once, at the start, so it cannot see a sibling run's issue.
concurrency:
group: quality-report
cancel-in-progress: false
jobs:
report:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
# No `contents: write`: this job never writes to the repository. It reads one
# endpoint, reads one directory of its own source, and writes issues —
# `issues: write` is the whole of its authority over anything it did not bring.
contents: read
issues: write
steps:
# The one thing this job needs from the repository, and it needs it for a
# mechanical reason: the body of each issue is a 13,000-character jq program, and
# GitHub parses a `run:` block as a single expression it refuses past 21,000
# characters — for the whole workflow file, on every run, whether the block runs
# or not. Inline, the filing step below was at 87% of that with a report designed
# to grow (every finding it learns to explain is another paragraph of prose), so
# the program moved to `.github/scripts/` the way `issue-triage.yml`'s decision
# step did. `test/workflow-run-length.test.ts` is what measures this.
#
# Sparse, because one directory is genuinely all of it: no source, no Node, no
# `npm ci`, and nothing here runs the app.
- name: Check out the issue-body program
uses: actions/checkout@v7
with:
sparse-checkout: .github/scripts
# Unconditional, and that is the point of it. Everything below is skipped on a
# healthy week — no tally over the threshold, nothing to file — so a checkout
# that fetched the wrong path, or a program with a syntax error in it, would
# first be discovered by the run that had something to say, and that run would
# fail having said nothing. This proves the file arrived and compiles on every
# run instead, including the ones that go on to decline.
#
# No input, deliberately: jq compiles the whole program before it reads the first
# value, so with stdin empty this exercises compilation and nothing else. `--arg`,
# `--argjson` and `--slurpfile` are still required — an undefined `$url`,
# `$cooldown` or `$tallyfile` is itself a compile error, which is the other half
# of what this checks. What is IN the tally is irrelevant here, hence /dev/null.
# Rendering the program against a real tally, for every finding key the step below
# can emit, is `test/quality-report-workflow.test.ts` on every `npm test`.
- name: Prove the issue-body program arrived and compiles
run: |
set -euo pipefail
jq --arg url "" --argjson cooldown 0 --slurpfile tallyfile /dev/null \
-f .github/scripts/quality-body.jq </dev/null
echo "quality-body.jq compiles"
- name: Read the deployment's quality tally
id: fetch
env:
QUALITY_URL: ${{ vars.QUALITY_URL }}
QUALITY_TOKEN: ${{ secrets.QUALITY_TOKEN }}
DAYS: ${{ inputs.days }}
run: |
set -euo pipefail
echo "## Quality report" >> "$GITHUB_STEP_SUMMARY"
if [ -z "${QUALITY_URL:-}" ] || [ -z "${QUALITY_TOKEN:-}" ]; then
# Not configured is a legitimate state, not a failure: a fork, or this
# repo before anyone opted a deployment in. Say what to set and stop —
# a red run every Saturday for a feature nobody enabled is noise that
# trains the maintainer to ignore the workflow.
echo "should_run=false" >> "$GITHUB_OUTPUT"
{
echo
echo 'Skipped: no deployment configured to read.'
echo
echo 'Set the `QUALITY_URL` repository variable and the `QUALITY_TOKEN` secret'
echo '(matching that deployment'"'"'s `server.quality_token`). See the header of'
echo '`.github/workflows/quality-report.yml`.'
} >> "$GITHUB_STEP_SUMMARY"
echo "::notice::QUALITY_URL/QUALITY_TOKEN not configured — nothing to read."
exit 0
fi
case "$QUALITY_URL" in
https://*) ;;
*)
# Hard failure, not a warning. The token is a bearer credential; over
# http it is readable by anything on the path, and a deployment-wide
# quality tally is the least of what a leaked shared secret costs.
echo "should_run=false" >> "$GITHUB_OUTPUT"
echo "::error::QUALITY_URL must be an https:// URL (got a non-https value). The token is a bearer token and must not travel in cleartext."
exit 1
;;
esac
WINDOW=30
if [ -n "${DAYS:-}" ]; then
if ! printf '%s' "$DAYS" | grep -qE '^[0-9]+$'; then
# Validated before it reaches a URL. Not a security boundary —
# workflow_dispatch needs write access — but `30d` would otherwise be
# sent as a query value, silently clamped to the default by the server,
# and reported in the summary as the window the caller asked for.
echo "should_run=false" >> "$GITHUB_OUTPUT"
echo "::error::days must be digits only, got '$DAYS'."
exit 1
fi
WINDOW="$DAYS"
fi
# Trailing slash tolerated, so QUALITY_URL can be written either way.
BASE="${QUALITY_URL%/}"
# --retry with a short --max-time: a transient blip must not turn into a red
# run, but a deployment that is genuinely unreachable must. Which way this
# goes when it fails is the decision that matters, and it is made below.
HTTP=$(curl -sS -o /tmp/quality.json -w '%{http_code}' \
--retry 3 --retry-delay 5 --retry-all-errors --max-time 30 \
-H "Authorization: Bearer $QUALITY_TOKEN" \
-H 'Accept: application/json' \
"$BASE/v1/quality?days=$WINDOW") || HTTP=000
case "$HTTP" in
200) ;;
404)
# The endpoint exists in the code but is unconfigured on that
# deployment (`server.quality_token` blank), so it answers 404 by
# design. Nothing to read, nothing wrong.
echo "should_run=false" >> "$GITHUB_OUTPUT"
{
echo
printf 'Skipped: `%s` does not have the quality tally enabled (404).\n' "$BASE"
echo
echo 'Set `server.quality_token` in that deployment'"'"'s config and restart it.'
} >> "$GITHUB_STEP_SUMMARY"
echo "::notice::$BASE returned 404 — the quality tally is not enabled there."
exit 0
;;
401)
echo "should_run=false" >> "$GITHUB_OUTPUT"
echo "::error::$BASE rejected the token (401). The QUALITY_TOKEN secret and that deployment's server.quality_token have diverged."
exit 1
;;
*)
# Deliberately a failure. A quality loop that stops reporting is
# indistinguishable from a deployment with no problems, which is the
# exact failure mode this whole feature is built to avoid — so when the
# measurement itself breaks, the run must go red where someone sees it.
echo "should_run=false" >> "$GITHUB_OUTPUT"
echo "::error::Could not read $BASE/v1/quality (HTTP $HTTP after retries). Nothing was measured this week."
exit 1
;;
esac
if ! jq -e '.documents? != null and .documents_linted? != null and .window_days? != null and .rules? != null and .editor_truncated_rate? != null and .editor_truncated_lost_rate? != null' /tmp/quality.json >/dev/null 2>&1; then
# A 200 that is not the tally: a login page, a proxy error page, an older
# deployment. Treated like an unreachable endpoint rather than parsed
# optimistically, because every threshold below reads a missing field as
# `null`, and `null > 0.15` is false — a garbled response would file
# nothing and look exactly like a clean week.
#
# `documents_linted` is checked by name for the second reason: it is
# interpolated into the issue bodies below, and a deployment older than #164
# answers this endpoint without it. That case is not a silent miss but a
# public issue reading "1 of null documents that were linted at all", so a
# field an issue body quotes has to be a field this gate demands — which is
# why `window_days` is listed too, though every version of the endpoint has
# returned it: every one of those bodies opens with it. Whoever adds the next
# quoted field should add it here in the same commit.
#
# The two truncation rates are the newest of those and the ones that date this
# file: a deployment from before #159 answers without `editor_truncated_lost_rate`
# (and one from before #151 without either), and the whole run stops here rather
# than that one threshold going quiet. That is the trade this gate exists to make
# — the alternative reads a missing field as `null`, compares it against a number,
# and reports a clean week — and it resolves itself the moment the deployment
# catches up with main. `editor_truncated_rate` is demanded even though nothing is
# thresholded against it, because the lost-rate issue body quotes it: the two
# numbers are only meaningful beside each other.
#
# `links_unresolved_rate` (#234) is deliberately NOT demanded, which is the rule
# working rather than an omission: it is thresholded nowhere and quoted in no issue
# body, so demanding it would stop the whole run against a deployment behind main
# over a number no step here reads. It still reaches a person every week — the
# summary below prints the entire payload — and the day something is thresholded
# against it, it belongs on this line in that commit. It carries no threshold yet
# because the first bench round to measure it found half the documents affected: a
# threshold today would file the same issue every week about a defect already
# tracked in #234, and the two MAX_ISSUES_PER_RUN slots would go to it instead of
# to whatever changed that week.
#
# `markup_unbalanced_rate` and `table_no_body_rate` (#240) are left off for the same
# reason and on the same terms. Both are new measurements of a class nothing could
# see before — an unclosed tag is repaired by the parser before axe reads the
# document, and a table with headers and no rows has no axe rule — so the first
# question is how often they fire on a deployment's real traffic, not what number is
# too high. The summary prints them either way.
#
# `first_read` (#313) is off this line too, and it is the case where the rule costs
# something: it is the one field that says whether a fall in `unresolved_rate` was the
# editor fixing more or the reviewer finding less, so it is exactly what a reader of an
# `unresolved_rate` issue wants beside the number. But quoting it in that body would
# require demanding it here, and demanding it stops the whole run against any
# deployment that has not yet caught up with this commit — over a field with no
# baseline to compare against and no threshold. The summary prints the entire payload,
# so it reaches a person weekly meanwhile; once there are a few weeks of it, the commit
# that thresholds or quotes it adds it to this line, and `docs/models.md` §3 is where
# the reason to look at it is written down.
#
# `structural_defect_rate` (#255) is the newest of these and off on the same terms: a
# dangling id reference is `incomplete` to axe and never a violation, a `<div>`-wrapped
# term list passes `definition-list`, and an empty `<nav>` breaks no rule, so this is a
# first measurement too and the summary is where it belongs until there is a baseline.
echo "should_run=false" >> "$GITHUB_OUTPUT"
echo "::error::$BASE/v1/quality returned 200 but not a quality tally. Nothing was measured this week."
exit 1
fi
echo "should_run=true" >> "$GITHUB_OUTPUT"
{
echo
printf 'Read `%s/v1/quality?days=%s`:\n\n' "$BASE" "$WINDOW"
echo '```json'
# The whole response, in the summary, every run. It is aggregate by
# construction so there is nothing here to redact, and the numbers are
# the point of the job even on a week when nothing crosses a threshold.
jq . /tmp/quality.json
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Compare against thresholds
id: check
if: steps.fetch.outputs.should_run == 'true'
run: |
set -euo pipefail
# ---- The thresholds. This is the file to edit. ----------------------
# They live here, not in the server, so retuning "how bad is too bad" is a
# repo edit with a diff and a reviewer rather than a config change and a
# restart. Every one of them is a guess that should be revised once there
# is a month of real numbers to look at; treat the first few runs as
# calibration, and prefer moving a number here over muting the workflow.
# Fraction of documents that finished with issues the review loop could not
# resolve. Some floor is inherent — a source document can be genuinely
# ambiguous — so this is set where "one in seven" starts to look systemic.
UNRESOLVED_RATE_MAX=0.15
# Fraction of documents where a link present in the source was missing from
# the editor's output. Lower, because this is content loss rather than an
# imperfect fix: the user's document had a link and ours does not.
LINKS_DROPPED_RATE_MAX=0.02
# Zero, meaning ANY document whose lint pass errored. Such a document has no
# accessibility verdict at all: it ships unchecked, and the review loop had
# no violations to act on. It is excluded from `documents_linted` so it does
# not drag every rule share down with it, which is exactly why this rate has
# to be watched separately. One occurrence is a bug in Iris, not a bad
# document.
LINT_ERROR_RATE_MAX=0
# Fraction of documents where a correction round hit the model's output ceiling and
# the sectioned retry did NOT cover the whole body — so part of the document kept the
# text it entered the round with, and nothing looked at those issues again (#159).
#
# This is the truncation number a threshold can be put on, and `editor_truncated_rate`
# is deliberately NOT one, which is the thing to read before changing either. The
# editor's answer is as long as the document it is rewriting, so at a large `max_pages`
# a perfectly healthy document hits the ceiling every time: that rate went 1/4 → 2/4 →
# 3/4 across bench rounds 6-8 while every one of those rounds delivered every section,
# and a threshold at 50% would have fired three weeks running on a pipeline that lost
# nothing. It is a cost and configuration signal — is `max_tokens` too low for the
# documents this deployment accepts, or `max_pages` too high for that ceiling — and the
# run summary prints it every week for whoever is asking that question.
#
# The number: across the 16 truncations in the bench archive, the 10 since the sectioned
# retry existed rescued 9 rounds whole and lost one section of four on the tenth — one
# document of the 16 in those rounds, 0.0625. So this fires on a repeat of the worst
# round observed plus a little, and not on that round itself; a deployment losing more
# than one document in ten this way has a ceiling that does not fit its documents. It is
# also immune to the length co-variance above, because absorbing length is exactly what
# the sectioned retry does.
EDITOR_TRUNCATED_LOST_RATE_MAX=0.10
# Mean EDITOR passes per document, against a default cap of 3
# (`defaults.max_review_iterations`). A document the Reader passes on its
# first look costs 0, so low is good and 0 is best. Above 2.5 the loop is
# using nearly its whole budget on the average document, which means the
# first pass is producing work the second and third have to undo.
#
# This one is a CHURN alarm and only that. A document also stops early when a
# round changes nothing, so a falling mean can mean the loop gave up sooner
# rather than that documents needed less fixing — which is why quality is
# watched by UNRESOLVED_RATE_MAX above, and why a drop here is not on its own
# evidence of anything improving.
MEAN_ROUNDS_MAX=2.5
# Fraction of documents violating one specific axe-core rule. A rule failing
# on a quarter of documents is a defect in a prompt, not a property of the
# documents — that is the signal worth a PR.
RULE_SHARE_MAX=0.25
# Below this many documents in the window, report and file nothing. Rates
# over a handful of documents are noise wearing a percentage sign: at 4
# documents one bad one crosses every threshold above, and an issue filed on
# that basis sends someone to rewrite a prompt over a single odd PDF.
MIN_DOCUMENTS=20
# How many issues one run may file. Same reasoning as issue-to-pr.yml's cap
# on open auto-PRs: a queue that grows faster than one person reads it is a
# backlog with a robot attached. Anything over the cap is named in the run
# summary, so nothing is dropped silently — it is deferred, visibly, and the
# next run will still see it.
MAX_ISSUES_PER_RUN=2
# --------------------------------------------------------------------
DOCS=$(jq -r '.documents' /tmp/quality.json)
if [ "$DOCS" -lt "$MIN_DOCUMENTS" ]; then
echo "found=0" >> "$GITHUB_OUTPUT"
{
echo
printf 'No issues filed: only %s document(s) in the window (minimum %s).\n' \
"$DOCS" "$MIN_DOCUMENTS"
echo
echo 'A rate over a handful of documents is noise. Nothing was measured against a'
echo 'threshold this run.'
} >> "$GITHUB_STEP_SUMMARY"
echo "::notice::Only $DOCS document(s) in the window (minimum $MIN_DOCUMENTS) — no thresholds evaluated."
exit 0
fi
THRESHOLDS=$(jq -n \
--argjson unresolved "$UNRESOLVED_RATE_MAX" \
--argjson links "$LINKS_DROPPED_RATE_MAX" \
--argjson lint "$LINT_ERROR_RATE_MAX" \
--argjson truncated "$EDITOR_TRUNCATED_LOST_RATE_MAX" \
--argjson rounds "$MEAN_ROUNDS_MAX" \
--argjson rule "$RULE_SHARE_MAX" \
'{unresolved: $unresolved, links: $links, lint: $lint, truncated: $truncated,
rounds: $rounds, rule: $rule}')
# Each finding gets a STABLE title and a title-derived key. Stable is the
# load-bearing word, and it rules out TWO kinds of number: this week's rate,
# and the threshold it crossed. The rate would make the title different every
# week, so the dedupe below would file a fresh issue each run for the same
# problem. The threshold would be worse — it is a number this file invites you
# to retune, and a title asserting "a quarter of documents" against a
# threshold since moved to 0.4 is a lie in every issue already filed, while
# correcting the wording changes the dedupe key and refiles a problem that is
# already being tracked. Hence "too many", "nearly its whole budget", "a large
# share": vague in the title, exact in the body, where it costs nothing. This
# repo has already been bitten by the mirror image of that (a
# dedupe that could not match, so every session refiled the same
# suggestion), and one issue per week is how a maintainer learns to mute a
# workflow.
#
# `sev` orders the findings when the cap bites, and the order is by what the
# number means rather than by how far past its threshold it is:
# 1 lint errors — the measurement itself is broken, so every rule rate
# below it is understated. Fix this before believing them.
# 2 dropped links — content the user gave us is missing from what we
# returned. Data loss outranks imperfect output.
# 3 truncation lost — corrections the editor made were thrown away, or never
# attempted, because the response did not fit. Below dropped
# links because the source content is all still there, above
# unresolved because these issues had no editor pass at all.
# 4 unresolved — known barriers, shipped, listed in unresolved.md.
# 5 a rule — a specific recurring barrier: the most actionable of
# these, but narrower than the three above it.
# 6 rounds — a cost and quality proxy, and the weakest signal here.
jq -r --argjson t "$THRESHOLDS" --argjson min "$MIN_DOCUMENTS" '
. as $q
| ( if ($q.lint_error_rate // 0) > $t.lint then
[ { key: "lint-error", sev: 1,
title: "Quality regression: the accessibility linter is failing to run",
metric: "lint_error_rate", value: $q.lint_error_rate, threshold: $t.lint } ]
else [] end )
+ ( if ($q.links_dropped_rate // 0) > $t.links then
[ { key: "links-dropped", sev: 2,
title: "Quality regression: links from the source document are being dropped",
metric: "links_dropped_rate", value: $q.links_dropped_rate, threshold: $t.links } ]
else [] end )
# The LOST rate, not `editor_truncated_rate` — see EDITOR_TRUNCATED_LOST_RATE_MAX
# for why the ceiling itself is not the alarm. `// 0` like the others, though the
# shape gate above already refuses a response without this field: the two say the
# same thing about a missing number and neither is the place to relax it.
+ ( if ($q.editor_truncated_lost_rate // 0) > $t.truncated then
[ { key: "truncated-lost", sev: 3,
title: "Quality regression: correction rounds are being lost at the output ceiling",
metric: "editor_truncated_lost_rate", value: $q.editor_truncated_lost_rate,
threshold: $t.truncated } ]
else [] end )
+ ( if ($q.unresolved_rate // 0) > $t.unresolved then
[ { key: "unresolved", sev: 4,
title: "Quality regression: too many documents ship with unresolved issues",
metric: "unresolved_rate", value: $q.unresolved_rate, threshold: $t.unresolved } ]
else [] end )
# `MIN_DOCUMENTS` is checked twice: once above on `documents`, and again
# here on `documents_linted`, because a rule share divides by the second
# and nothing else floors it. Twenty delivered documents of which
# nineteen could not be linted leaves every rule share measured over ONE
# document, where one violation is 100% and crosses this threshold — an
# issue filed about a prompt on the strength of a single page, and a
# 30-day cooldown started on a rule that was never really measured. The
# lint-error finding above fires in that window too and outranks this, so
# a week that also crosses one more threshold fills both MAX_ISSUES_PER_RUN
# slots and defers this rule finding — named in the run summary rather than
# dropped, and offered again next run. That order is the right one: an
# unlinted deployment is the reason the rule share is untrustworthy here.
+ ( [ $q.rules[]? | select($q.documents_linted >= $min) | select(.share > $t.rule)
| { key: ("rule-" + .id), sev: 5,
title: ("Quality regression: axe rule `" + .id + "` fails on a large share of documents"),
metric: ("rules[\"" + .id + "\"].share"), value: .share, threshold: $t.rule,
rule: .id, impact: (.impact // "unknown"), rule_documents: .documents, nodes: .nodes } ] )
+ ( if ($q.mean_rounds // 0) > $t.rounds then
[ { key: "mean-rounds", sev: 6,
title: "Quality regression: the review loop is using nearly its whole round budget",
metric: "mean_rounds", value: $q.mean_rounds, threshold: $t.rounds } ]
else [] end )
| sort_by(.sev, (0 - .value))
| .[] | @json' /tmp/quality.json > /tmp/findings.jsonl
# Say so when the rule table was skipped for want of a denominator, rather than
# letting "no rule crossed its threshold" stand for "no rule was measured".
LINTED=$(jq -r '.documents_linted' /tmp/quality.json)
if [ "$LINTED" -lt "$MIN_DOCUMENTS" ]; then
{
echo
printf 'Rule shares were not evaluated: only %s of %s document(s) in the window were linted at all (minimum %s). See the lint-error finding.\n' \
"$LINTED" "$DOCS" "$MIN_DOCUMENTS"
} >> "$GITHUB_STEP_SUMMARY"
echo "::warning::Rule shares not evaluated: only $LINTED of $DOCS documents were linted."
fi
FOUND=$(wc -l < /tmp/findings.jsonl | tr -d ' ')
echo "found=$FOUND" >> "$GITHUB_OUTPUT"
echo "max_issues=$MAX_ISSUES_PER_RUN" >> "$GITHUB_OUTPUT"
echo "findings: $FOUND"
if [ "$FOUND" -eq 0 ]; then
# The expected outcome most weeks, and it must produce nothing at all —
# no issue, no comment. An automation that always finds something to file
# is one that invents work, and an invented issue costs the same attention
# as a real one (and, through issue-to-pr.yml, a whole review slot).
{
echo
printf 'No issues filed: every rate is within its threshold over %s document(s).\n' "$DOCS"
} >> "$GITHUB_STEP_SUMMARY"
echo "::notice::All rates within threshold over $DOCS documents — nothing to file."
exit 0
fi
{
echo
printf '### %s threshold(s) crossed\n\n' "$FOUND"
echo '| Metric | Measured | Threshold |'
echo '| --- | --- | --- |'
jq -r '"| `\(.metric)` | \(.value) | \(.threshold) |"' /tmp/findings.jsonl
} >> "$GITHUB_STEP_SUMMARY"
- name: File an issue per crossed threshold
if: steps.fetch.outputs.should_run == 'true' && steps.check.outputs.found != '0'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Required, and not redundant with GITHUB_REPOSITORY. `gh` resolves the
# repository from `--repo`, `GH_REPO`, or the git remotes of the working
# directory, and it does NOT read GITHUB_REPOSITORY. Without this, `gh issue
# list` below fails with "not a git repository" under `set -e`, and it fails
# only on the weeks a threshold is actually crossed: this whole step is skipped
# when there is nothing to file, so the workflow would read green until the
# moment it had something to say, then go red having said nothing.
#
# The checkout at the top of the job would now satisfy `gh` by accident, and
# this stays anyway. That checkout exists to put one jq program on disk; making
# the filing of issues depend on it as well would mean a change to how the body
# program is fetched could break the filing, silently, on the same schedule
# described above.
GH_REPO: ${{ github.repository }}
DRY_RUN: ${{ inputs.dry_run }}
MAX_ISSUES: ${{ steps.check.outputs.max_issues }}
QUALITY_URL: ${{ vars.QUALITY_URL }}
run: |
set -euo pipefail
# How long after an issue with this exact title was CLOSED before the same
# claim may be made again. Set to the width of the window itself, and that
# is the reason for it: the rate is measured over 30 days, so on the day a
# fix merges the window still contains a month of pre-fix documents and the
# rate has barely moved. Refiling then would tell the maintainer their fix
# did not work, a week after they made it. One window has to roll over
# before the number means anything again.
#
# It also handles the other close reason — "read it, not doing it" — the
# same way issue-to-pr.yml handles a rejected attempt: don't come straight
# back with it.
COOLDOWN_DAYS=30
# Every issue this workflow has ever filed, open or closed, found by title
# prefix. Searched then compared exactly, mirroring src/github/issue.ts: the
# `in:title` search is a full-text phrase match rather than an equality
# test, so it returns a superset and the comparison below decides.
#
# A prefix rather than a label, and deliberately so: labels are
# silently dropped for a filer without push access, so a label-based dedupe
# is one that works in testing and fails for everyone else. The title is set
# by the same request that creates the issue and cannot be stripped.
gh issue list --state all --limit 200 \
--search '"Quality regression:" in:title' \
--json number,title,state,closedAt,url > /tmp/existing.json
echo "existing quality-regression issues: $(jq length /tmp/existing.json)"
NOW_EPOCH=$(date -u +%s)
FILED=0
SKIPPED=""
DEFERRED=""
while IFS= read -r finding; do
TITLE=$(jq -r '.title' <<<"$finding")
# --- dedupe, before the cap: an already-open finding must not consume
# one of this run's slots, or a single long-lived issue would starve
# every other finding behind it forever.
# An open issue wins over any closed one; among closed ones the most
# recently closed wins, because that is the one the cooldown is about.
# (`false` sorts below `true` in jq, so the open one lands last.)
PRIOR=$(jq -c --arg t "$TITLE" '[.[] | select(.title == $t)]
| sort_by([(.state == "OPEN"), (.closedAt // "")])
| last // empty' /tmp/existing.json)
if [ -n "$PRIOR" ]; then
PRIOR_STATE=$(jq -r '.state' <<<"$PRIOR")
PRIOR_NUM=$(jq -r '.number' <<<"$PRIOR")
if [ "$PRIOR_STATE" = "OPEN" ]; then
SKIPPED="$SKIPPED- \`$TITLE\` — already open as #$PRIOR_NUM"$'\n'
continue
fi
CLOSED_AT=$(jq -r '.closedAt // ""' <<<"$PRIOR")
if [ -n "$CLOSED_AT" ]; then
CLOSED_EPOCH=$(date -u -d "$CLOSED_AT" +%s)
AGE_DAYS=$(( (NOW_EPOCH - CLOSED_EPOCH) / 86400 ))
if [ "$AGE_DAYS" -lt "$COOLDOWN_DAYS" ]; then
SKIPPED="$SKIPPED- \`$TITLE\` — #$PRIOR_NUM was closed ${AGE_DAYS}d ago (cooldown ${COOLDOWN_DAYS}d)"$'\n'
continue
fi
fi
fi
if [ "$FILED" -ge "$MAX_ISSUES" ]; then
DEFERRED="$DEFERRED- \`$TITLE\`"$'\n'
continue
fi
# Body assembled in jq rather than by interpolating into the shell, so a
# rule id never reaches a `run:` expression. Rule ids come from axe-core's
# fixed vocabulary today, but the endpoint is the untrusted side of this
# boundary and the cost of assuming otherwise is command execution in a
# job holding `issues: write`.
#
# The program itself is `.github/scripts/quality-body.jq`, checked out at
# the top of this job; the comment at the head of that file says why it is
# not written here. The finding arrives on stdin and the tally through
# `--slurpfile`, so neither of them is interpolated into this block either.
jq -r --arg url "${QUALITY_URL%/}" --argjson cooldown "$COOLDOWN_DAYS" \
--slurpfile tallyfile /tmp/quality.json \
-f .github/scripts/quality-body.jq <<<"$finding" > /tmp/issue-body.md
if [ "${DRY_RUN:-false}" = "true" ]; then
{
echo
printf '#### Would file: %s\n\n' "$TITLE"
cat /tmp/issue-body.md
} >> "$GITHUB_STEP_SUMMARY"
FILED=$((FILED + 1))
continue
fi
# No labels, and no `--assignee`. Labels are silently dropped for a filer
# without push access, so nothing downstream may depend on
# them; issue-to-pr.yml ranks these by the title prefix instead.
URL=$(gh issue create --title "$TITLE" --body-file /tmp/issue-body.md)
FILED=$((FILED + 1))
echo "filed: $URL"
{ echo; printf -- '- Filed [%s](%s)\n' "$TITLE" "$URL"; } >> "$GITHUB_STEP_SUMMARY"
done < /tmp/findings.jsonl
if [ -n "$SKIPPED" ]; then
{ echo; echo 'Already tracked, not refiled:'; echo; printf '%s' "$SKIPPED"; } >> "$GITHUB_STEP_SUMMARY"
fi
if [ -n "$DEFERRED" ]; then
# Said out loud rather than truncated quietly: a cap that hides what it
# dropped reads, in the summary, exactly like a week with fewer problems.
{ echo; printf 'Over the %s-issue cap, deferred to the next run:\n\n' "$MAX_ISSUES"; printf '%s' "$DEFERRED"; } >> "$GITHUB_STEP_SUMMARY"
echo "::notice::Deferred $(printf '%s' "$DEFERRED" | grep -c '^-') finding(s) over the per-run cap of $MAX_ISSUES."
fi
echo "::notice::Filed $FILED issue(s) this run."