An evaluator commits an Evaluator Trust Boundary failure when its score depends on an artifact the evaluated system can influence. Ten mechanisms, one violation. Every instance below is a public report with a live URL.
The rule
A score must be a function only of state the evaluated system cannot reach.
Offline, standard library only, no API keys and no model spend. It measures ETB-01 against your own judge callable.
A real judge is stochastic, so use several trials: a scenario counts as exploited if the injection lands in any one of them. The scanner covers ETB-01 only, and the other nine need review of the source.
Prior work shows that LLM judges can be attacked, using gradient-based prompt injection or training-data poisoning. This is a different claim: the defect sits in the scoring code path rather than in the judge model, so it survives a perfectly hardened judge and is reachable by ordinary optimization with no adversary at all. Identifiers are permanent. New classes append; these ten never renumber.
Trustsa score parsed out of candidate-written text
The judge extracts a score from model-controlled text: a greedy first-match regex over the candidate answer, or a JSON parse that returns the first well-formed object in the string.
FixParse the judge's own structured verdict. Bind extraction to the last block or an explicitly delimited region. Never read the candidate.
Trustsmodel text as prompt structure
Model text (answers, chain-of-thought, tool arguments, transcripts) flows unescaped into the judge or monitor prompt.
FixEscape or structurally neutralize model output before it enters any grader prompt. Use message-role separation.
Trustsan error as a pass
A backend, parse, or vendor error is coerced to “safe” or “pass.”
FixFail closed. Flag the sample. Never default to safe.
Trustsa mean over a silently reduced sample
Unscorable, refused, or errored samples are silently excluded from the mean, inflating the score.
FixCount them, or surface them as a separate reported quantity.
Trusts“could not score” as “safe”
“Could not score” is conflated with “the attack did not succeed.”
FixDistinguish could-not-score from safe at the type level.
Trustsa byte comparison over unnormalized text
Homoglyph, zero-width, or Unicode-steganographic content evades a detector.
FixNormalize both trigger strings and model output before matching.
Trustsper-step innocence as trajectory innocence
Per-step monitors miss an attack distributed across individually benign steps.
FixCarry cross-step state in the monitor.
The study's control case. A maintainer stated on the record that stateless per-step monitors are intentional and cost-motivated. That is a priced tradeoff, not an oversight, and it is the one place in the sample where the trust boundary was already in someone's model. Classified here as an undocumented tradeoff rather than a defect: the gap is that consumers of the monitor's output are not told composition attacks pass by design.
Trustsan aggregation that loses per-key structure
Whole-dict positivity, multi-answer under-count, or name-collision collapse.
FixPer-key and per-answer scoring.
Trustsa self-reported execution result
A runner trusts a self-reported PASSED, an always-equal return, or spoofed stdout.
FixVerify against frozen ground truth the policy cannot influence.
Trustsa context the policy can detect and condition on
Environment variables, run identifiers, or harness artifacts let the agent detect it is being evaluated.
FixStrip or randomize identifying context; treat detectability as a measured property.
Rests on a single observation. A single instance is a hypothesis about a class, not a measurement of one.
76 instances across 36 organizations, close to evenly split across four sectors.
Counted as distinct defects, not reports: where one defect produced both an issue and a pull request, it counts once. 23 further defects surfaced during the audit are real but are not ETB failures (conventional security issues, correctness bugs, detection-coverage gaps); they are excluded from every count here.
ETB-07 and ETB-10 each rest on a single observation. Treat them as hypotheses about a class rather than measurements of one.
The scanner covers ETB-01 only. The other nine are not detectable from outside the scoring code and need review of the source.
Reports are clustered by repository: one organization accounts for seven reports to a single repo, so per-organization behavior is not seven independent observations.
Numbers on this page are generated from the public report set, not maintained by hand. Corrections are welcome and every row resolves to a live URL: john@authensor.com.
If your evaluator reads a score out of text the candidate wrote, etb-scan finds it in one command.
pip install etb-scan. Offline, standard library, no API keys.
Point it at your judge callable. Several trials, because a real judge is stochastic.
It reports ETB-01 exposure. The other nine need a read of the source.