The evaluator trusts an error as a pass. A backend, parse, or vendor error is coerced to “safe” or “pass.”
Fix
Fail closed. Flag the sample. Never default to safe.
In the study
6 instances
Of
76
An error inside the evaluator is turned into a passing result. A missing score defaults to passing, an exception handler returns allow, a third-party safety service that times out lets the request through. The code path that should mean we do not know produces the same output as we checked and it was fine.
That makes the error itself something the evaluated system can reach. Output the parser cannot read, a reply with no verdict in it, or content that makes a vendor call fail all land on the passing branch. The public cases include an LLM-as-judge guardrail whose missing overall score defaulted to a passing 100, safety rails that allowed everything when the safety backend was down, and a verdict parser that failed open.
Promptfoo fixed one in October 2026: an llm-rubric judge response with no verdict defaulted to a pass. The judge now fails closed.
The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy
Illustrative Python, reduced to the line that matters. The public reports below link the real code.
try:
score = parse_verdict(judge_reply)
except Exception:
score = 1.0 # judge failed, assume the answer was finetry:
score = parse_verdict(judge_reply)
except VerdictParseError:
return Score(value=None, status="unscorable",
reason="no verdict in judge reply")Record the sample as unscorable, flag it, and report how many there were. Do not default it to pass, and do not default it to safe. A grader that cannot tell you its answer has not given you one.
Because the evaluated system can cause the error. When a malformed or verdict-less reply scores as a pass, producing one is a way to pass, and nothing in the reported number shows it happened.
Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.
llm_as_a_judge guardrail fails open: missing overall_score defaults to 100 (pass)
Lakera v1 guardrail: setting category_thresholds silently disables blocking for flagged categories it doesn't name
Third-party safety rails FAIL OPEN on backend error (safety vendor down = guardrail allows everything)
is_content_safe verdict parser fails OPEN, bypassing self-check / content-safety / fact-checking rails
OpenHands/software-agent-sdk #4157
LLMSecurityAnalyzer trusts model self-assessed risk level: any action classified LOW auto-executes without human confirmation
Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.