The evaluator trusts a self-reported execution result. A runner trusts a self-reported PASSED, an always-equal return, or spoofed stdout.
Fix
Verify against frozen ground truth the policy cannot influence.
In the study
7 instances
Of
76
The runner believes what the code under test reports about its own execution. A PASSED line in the test output, a return value compared with ==, an anti-cheat canary inside the environment: each is produced by, or reachable from, the code being evaluated.
So the code under test can print PASSED, or return an object whose __eq__ always returns True, and score as if it had solved the task. Public cases include code-execution benchmarks that a dunder override passes without solving anything, and anti-cheat checks the policy can subvert.
SWE-bench fixed one in August 2026: its grader counted PASSED lines that the evaluated patch could print itself. It now checks the test command's exit status.
The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy
Illustrative Python, reduced to the line that matters. The public reports below link the real code.
out = subprocess.run(["pytest", test_file], capture_output=True, text=True).stdout passed = "PASSED" in out # the code under test can print this
proc = subprocess.run(["pytest", test_file], capture_output=True, text=True) passed = proc.returncode == 0 # the runner's own evidence, not lines the code wrote
Before the fix, a patch could print PASSED lines that the grader counted as passing tests. The accepted fix checks the test command's exit status instead of trusting lines in its output.
Do not compare objects the candidate constructed. Serialize the result to plain data, compare it with the expected value in the grader's own process, and check the type as well as equality.
Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.
Tool permission/policy guardrails not applied on /v1/responses & /v1/messages routes; blocklist bypassed by name variants
TAP/PAIR LLM-judge prompt interpolates target response unescaped: model output can forge Rating rail and flip jailbreak to safe
SWE-bench/SWE-bench #620landed
Detect stdout spoofing via test exit code (model can forge PASSED lines)
Detect return-value deception via dunder override (same exploit class as HumanEval)
open-compass/opencompass #2535
Return-value deception via __eq__ override: model can pass code-exec benchmarks without solving problems
Detect return-value deception via dunder override (model can score 100% without solving problems)
Close two anti-cheat bypasses in canary and dockerfile checks
Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.