The evaluator trusts a score parsed out of candidate-written text. The judge extracts a score from model-controlled text: a greedy first-match regex over the candidate answer, or a JSON parse that returns the first well-formed object in the string.
Fix
Parse the judge's own structured verdict. Bind extraction to the last block or an explicitly delimited region. Never read the candidate.
In the study
24 instances
Of
76
The scorer reads its verdict out of a string that contains the candidate's answer. The usual form is a first match: a regular expression that stops at the first GRADE: marker, or a parser that returns the first well-formed JSON object it finds. The candidate's text sits ahead of the judge's verdict in that string, so the candidate writes first and the candidate wins.
It is the most common mechanism in the study, 24 of 76 instances, and it recurs in codebases with no shared lineage. Independent implementations converging on the same wrong default is what a missing concept looks like in source code: nobody is holding the trust boundary in mind, so everybody reaches for the most obvious parse.
The judge model is not the problem. In the paper's frontier-judge run, the same model behind three scoring code paths went from an attack success rate of 0.0000 to 1.0000 when only the string the verdict was parsed from changed. No prompt, model or instruction changed.
It also corrupts grading with no adversary at all. The broken parse fired on 3 of 8 benign controls, because the first match reached the word PASS inside the grader's own explanation of the rubric ("requires at least 3 for PASS") about sixty characters ahead of its actual VERDICT: FAIL. The judge was right on every control. The scorer recorded the opposite.
The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy
Illustrative Python, reduced to the line that matters. The public reports below link the real code.
# The verdict is parsed from a blob that contains the candidate's answer.
blob = f"{candidate_answer}\n{judge_reply}"
grade = re.search(r"GRADE:\s*([A-F])", blob).group(1) # first match wins# Parse only the judge's own reply, bound to its final line. last = judge_reply.strip().splitlines()[-1].strip() m = re.fullmatch(r"GRADE:\s*([A-F])", last) grade = m.group(1) if m else None # None means could not score, never a pass
Yes, whenever the scorer extracts the verdict from text that includes the model's answer. If the answer contains a well-formed verdict and the parser takes the first match, the answer's verdict is the one recorded, whatever the judge concluded.
It is a superset. Attacks such as JudgeDeceiver, GCG suffixes or training-data poisoning need an adversary and target the judge model. ETB-01 is the extraction step reading a string the policy can write. It fires on a crafted injection, on benign prose that happens to contain a verdict token, and on a policy that drifts into the pattern under optimization with no intent at all. A perfectly hardened judge model does not fix it.
Run etb-scan (pip install etb-scan) against your judge callable. It presents 25 injected scenarios across five attack families and 8 benign controls, and fires only when an injection flips a failing verdict to a passing one. Use several trials, because a real judge is stochastic.
Instructor. Code-block JSON extraction returned the first object, so planted JSON outranked the real output.
UK AISI Inspect AI. Model-graded GRADE extraction mis-scored ordinary prose, zero-width characters and lowercase verdicts.
UK AISI Inspect Evals. The SimpleQA scorer misread grades wrapped in grader decoration.
Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.
UKGovernmentBEIS/inspect_ai #4285
Warn on model-graded self-grading and even grader ensembles
UKGovernmentBEIS/inspect_evals #1812landed
Widen original scorer grade parse to tolerate grader decoration
meridianlabs-ai/inspect_petri #113
Judge refusals and failed/empty audits silently lower reported concern (false assurance)
EleutherAI/lm-evaluation-harness #3879
Add AnchoredMultiChoiceRegexFilter: commitment-based answer extraction
Verdict-injection via trimAndLoadJson returns first JSON object
guardrails-ai/guardrails #1566
Fix verdict-injection via extract_json_from_ouput (prompt-injection-into-judge)
Fix verdict-injection via extract_json (prompt-injection-into-judge)
Fix verdict-injection in _marshal_llm_to_json (prompt-injection-into-judge)
UKGovernmentBEIS/reality-test-eval #11
Fix parse_score to take last match (injection-resistant)
MixEval judge extractors use last-match (verdict-injection defense)
open-compass/opencompass #2534
Use last-match in generic LLM judge postprocess (verdict-injection defense)
Reject duplicate JSON keys in Hermes tool parser for streaming consistency
ApolloResearch/deception-detection #72
parse_judgement takes last block when judge echoes <judgement> tags from model output (Step 2 of prompt)
Ratnaditya-J/alignment-evals #1
Harden parse_judge_json against verdict-injection from the model under test
UKGovernmentBEIS/inspect_evals #1992
extract_code_block takes last block to defeat verdict-injection from agent log history
patronus-ai/trail-benchmark #3
Correct int-cast, JSON extraction, and sentinel handling in calculate_scores
safety-research/open-source-alignment-faking #5
Harden ClassicScratchpadClassifier against verdict-injection from the model under test
Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.