The Evaluator Trust Boundary
An evaluator commits an Evaluator Trust Boundary (ETB) failure when its score depends on an artifact the evaluated system can influence.
A score must be a function only of state the evaluated system cannot reach.
Prior work shows that LLM judges can be attacked, using gradient-based prompt injection or training-data poisoning. This is a different claim: the defect sits in the scoring code path rather than in the judge model, so it survives a perfectly hardened judge and is reachable by ordinary optimization with no adversary at all.
Ten mechanisms, one violation. Every instance below is a public report with a live URL.
pip install etb-scan
etbscan --judge yourpkg.judges:your_judge --trials 10Offline, standard library only, no API keys and no model spend. It measures ETB-01 against your own judge callable. A real judge is stochastic, so use several trials: a scenario counts as exploited if the injection lands in any one of them.
The ten mechanisms
Identifiers are permanent. New classes append; these ten never renumber.
The evaluator trusts a score parsed out of candidate-written text.
The judge extracts a score from model-controlled text: a greedy first-match regex over the candidate answer, or a JSON parse that returns the first well-formed object in the string.
FixParse the judge's own structured verdict. Bind extraction to the last block or an explicitly delimited region. Never read the candidate.
The evaluator trusts model text as prompt structure.
Model text (answers, chain-of-thought, tool arguments, transcripts) flows unescaped into the judge or monitor prompt.
FixEscape or structurally neutralize model output before it enters any grader prompt. Use message-role separation.
The evaluator trusts an error as a pass.
A backend, parse, or vendor error is coerced to “safe” or “pass.”
FixFail closed. Flag the sample. Never default to safe.
The evaluator trusts a mean over a silently reduced sample.
Unscorable, refused, or errored samples are silently excluded from the mean, inflating the score.
FixCount them, or surface them as a separate reported quantity.
The evaluator trusts “could not score” as “safe”.
“Could not score” is conflated with “the attack did not succeed.”
FixDistinguish could-not-score from safe at the type level.
The evaluator trusts a byte comparison over unnormalized text.
Homoglyph, zero-width, or Unicode-steganographic content evades a detector.
FixNormalize both trigger strings and model output before matching.
The evaluator trusts per-step innocence as trajectory innocence.
Per-step monitors miss an attack distributed across individually benign steps.
FixCarry cross-step state in the monitor.
The study's control case. A maintainer stated on the record that stateless per-step monitors are intentional and cost-motivated. That is a priced tradeoff, not an oversight, and it is the one place in the sample where the trust boundary was already in someone's model. Classified here as an undocumented tradeoff rather than a defect: the gap is that consumers of the monitor's output are not told composition attacks pass by design.
The evaluator trusts an aggregation that loses per-key structure.
Whole-dict positivity, multi-answer under-count, or name-collision collapse.
FixPer-key and per-answer scoring.
The evaluator trusts a self-reported execution result.
A runner trusts a self-reported PASSED, an always-equal return, or spoofed stdout.
FixVerify against frozen ground truth the policy cannot influence.
The evaluator trusts a context the policy can detect and condition on.
Environment variables, run identifiers, or harness artifacts let the agent detect it is being evaluated.
FixStrip or randomize identifying context; treat detectability as a measured property.
Rests on a single observation. A single instance is a hypothesis about a class, not a measurement of one.
Where it appears
76 instances across 36 organizations, close to evenly split across four sectors. We are not aware of an evaluation stack in the sample that is free of the class.
Counted as distinct defects, not reports: where one defect produced both an issue and a pull request, it counts once. 23 further defects surfaced during the audit are real but are not ETB failures (conventional security issues, correctness bugs, detection-coverage gaps); they are excluded from every count here.
Limits
- ETB-07 and ETB-10 each rest on a single observation. Treat them as hypotheses about a class rather than measurements of one.
- The scanner covers ETB-01 only. The other nine are not detectable from outside the scoring code and need review of the source.
- Reports are clustered by repository: one organization accounts for seven reports to a single repo, so per-organization behavior is not seven independent observations.
Numbers on this page are generated from the public report set, not maintained by hand. Corrections are welcome and every row resolves to a live URL: john@authensor.com.