Authensor is a research-led evaluation integrity firm. We audit the instruments the AI industry measures itself with: evaluation harnesses, benchmarks, judges, leaderboards, and scoring pipelines. Every defect we report is proven with a running reproduction, and every audit ends in an attestation with a scope, a standard version, and an expiry date.

What is an evaluation integrity audit?

A fixed-scope read of the scoring path that produces your published number: dataset residency, answer extraction, judge parsing, aggregation, and reporting. We read the code line by line, run adversarial reproductions against it, and re-grade your headline result under an instrument that fails closed.

What is an attestation?

A signed statement that a named instrument, at a named commit, survived a named standard on a named date. It has a scope, a standard version, and an expiry date, so your buyers can file it in procurement and know exactly what it does and does not cover.

What is a claim check?

A free 48-hour verification of one published number. You give us the claim and the artifact behind it; we tell you whether the number survives contact with the grading code that produced it. If it does, you can say so. If it does not, you know before your buyer does.

What is the evaluator trust boundary?

The evaluator trust boundary (ETB) is the line between the system being measured and the instrument doing the measuring. Our research shows that line is crossed in specific, repeatable ways: candidate text reaches the judge unparsed, ground truth lives where the evaluated system can read it, and parse failures silently shrink denominators. The ETB taxonomy names ten mechanisms, ETB-01 to ETB-10, documented across 76 instances in 36 organizations.

Three buyers. AI labs that publish numbers and need them to survive diligence. Enterprises procuring AI who want a third party to verify vendor claims before signature. And investors or insurers whose models price the risk those numbers describe.

ETB-07. Composition blindness

Name: etb-scan
Author: Authensor

The evaluator trusts per-step innocence as trajectory innocence. Per-step monitors miss an attack distributed across individually benign steps.

Fix

Carry cross-step state in the monitor.

In the study

1 instance

01How it happens

What the scorer reads that the policy can write.

A monitor judges each step of an agent's trajectory on its own, with no state carried between steps. An action that is only harmful in combination, split into steps that are each benign, passes, because no single step is suspicious.

This class rests on a single observation and is the study's control case. The maintainer stated that the stateless design is intentional and motivated by cost, which makes it a priced tradeoff rather than an oversight. The gap is disclosure: anyone reading the monitor's numbers has no way to learn that composition attacks pass by design.

In the best case that is a documentation fix. In the worst case a reader takes the monitor's results as a measurement of whether attacks occurred, when an attacker who distributes an action across steps is scored clean by construction.

The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy

02Example

The pattern, and the fix.

Illustrative Python, reduced to the line that matters. The public reports below link the real code.

Trusts the evaluated system

for step in trajectory:
    if monitor(action=step.action) > THRESHOLD:   # each step judged alone
        flag(step)

Fixed

history = []
for step in trajectory:
    history.append(step.action)
    if monitor(actions=history) > THRESHOLD:   # judged with the trajectory so far
        flag(step)

03Questions

Can an agent split an attack across steps to evade a monitor?

Against a monitor that scores each step in isolation, yes. Each step is judged without the ones before it, so a sequence that is only harmful as a whole never shows up as a single suspicious step.

Is a stateless per-step monitor a bug?

Not necessarily. It can be a reasonable cost tradeoff. It becomes a measurement problem when the tradeoff is not stated, and the monitor's clean results are read as evidence that no multi-step attack happened.

04Public reports

Every public report in this class.

Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.

2026-03-31

UKGovernmentBEIS/control-arena #792

Per-step monitor evaluation has no cross-step state, enabling composition attacks where each step is individually benign

← ETB-06: Missing normalization ETB-08: Metric miscounting →All ten mechanisms

Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.