EqualifyEverything / equalify-iris-bench

fix(prepare): stop keeping 5.4 GB to show 265 MB of corpus
Four documents left 5.4 GB in cache/chunks. 3.3 GB of it was scratch: poppler's splitter writes one PDF per page before uniting them, into a .tmp- directory per chunk that nothing ever removed. Another 1.9 GB was four 477 MB chunks that had already been measured, recorded as `chunk_too_large`, and can never be sent anywhere. At four URLs that is untidy; at two thousand it is the reason the run stops. So the scratch directory is created only by the splitter that needs one and removed in a finally, and an unsendable chunk is deleted once its size has been recorded — the measurement is the part worth keeping, and the record no longer carries a `path` to a file that is gone. Both of these are poppler behaviours, and this machine has qpdf, so neither was covered: the default probe picks qpdf and the poppler branch never ran. --splitter qpdf|poppler forces one, which makes the test deterministic about which splitter it is describing — and is worth having anyway, since the two are not interchangeable and the difference is exactly what this commit is about.
Blake Bertuccelli-Booth Blake Bertuccelli-Booth committed on Aug 24, 2026, 04:31 PM
Showing 3 changed files +68 additions -21 deletions
M README.md +9
M src/prepare.mjs +35 -10
Browse files at this commit →