The evaluator trusts a mean over a silently reduced sample. Unscorable, refused, or errored samples are silently excluded from the mean, inflating the score.
Fix
Count them, or surface them as a separate reported quantity.
In the study
6 instances
Of
76
Samples the scorer could not score are left out of the average without being reported. If 20 of 100 samples error and the other 80 average 0.9, the headline reads 0.9 over what the reader assumes is 100 samples. When failures correlate with the behavior being measured, which they often do, removing them moves the number in the direction the evaluated system benefits from.
The scorer is reading the shape of the evaluated system's failures. Whatever makes a sample unscorable, a refusal, an output the parser rejects, a timeout, decides whether that sample counts at all.
UK AISI's Inspect AI fixed one: scoring metrics silently dropped inconclusive and errored samples from the denominator. Other public cases include attack-success rates that drop None detector scores and metric objects that do not track how many values were missing.
The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy
Illustrative Python, reduced to the line that matters. The public reports below link the real code.
accuracy = mean(s.value for s in scores if s.value is not None)
scored = [s for s in scores if s.value is not None]
report = {
"accuracy": sum(s.value for s in scored) / len(scores), # unscorable counts against
"unscorable": len(scores) - len(scored),
"n": len(scores),
}Because failed samples are being removed from the denominator rather than counted. Each failure that would have scored low is no longer in the average, so the average rises.
Count them as wrong, or report them as a separate number next to the score with the total n. Either is defensible. Dropping them without saying so is not.
UK AISI Inspect AI. Scoring metrics silently dropped inconclusive and errored samples from the denominator.
Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.
Presidio PII guardrail fails open on analyzer error and skips /v1/responses input and streaming bytes
UKGovernmentBEIS/inspect_ai #4286landed
Scoring metrics silently drop inconclusive/errored samples from the denominator and don't bound their range
Fix FileDetector output/result misalignment (yield None for skipped outputs)
Expose NaN-denominator inflation on EvaluationResult (unscoreable-as-pass)
Surface ASR denominator-drop from None detector scores (unscoreable-as-pass)
Track none_count on Stat to surface denominator-drop (unscoreable-as-pass)
Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.