23 of the 25 most-starred AI evaluation repositories have an Evaluator Trust Boundary defect.

We took the 25 most-starred AI evaluation repositories on GitHub and asked one question of each: does anything the instrument reads sit inside the evaluated model’s writability cone? 23 fail the question. One more is under coordinated disclosure until November, which will make it 24 of 25. The single clean case is the one that does what the taxonomy prescribes: it validates the judge’s structured output instead of parsing free text.

The defects are not exotic. The first JSON object in the judge’s free text wins. The word “Yes” anywhere in the judge’s reply means correct. The last two bare numbers in the reply are the score. Ground truth read from inside the machine being graded. Every finding is public with a receipt: a filed issue, a runnable proof of concept, and the exact file and line.

The census

Star counts and ranking frozen 2026-08-13 via the GitHub API. “Filed” means a public issue or pull request exists upstream; receipts for the 18 earlier filings are in the evidence table. The five newest filings ship standalone PoCs that download the genuine upstream file at the pinned commit, verify its sha256, and run the real parser.

The control case is the point

Phoenix was audited on its core evaluation path and came back clean: structured verdicts, labels validated against the allowed value set, and a loud failure on anything malformed. That is not a research breakthrough. It is a one-file architectural decision, and 24 of 25 repositories made the other one. The class is not inevitable. It is just unexamined.

Honesty notes

• “Top 25 by stars” is a proxy for “most used.” The frozen pool and ranking script are the defense against cherry-picking, and they are public.
• Phoenix was audited on its core evals path; its server and agent surfaces were out of scope.
• The withheld repository was reported through coordinated disclosure before this census was published. Its details join this table when the embargo lifts.
• A defect in a harness is not a claim that any specific published score is wrong. It is a claim that the score cannot arbitrate.