Authensor is a research-led evaluation integrity firm. We audit the instruments the AI industry measures itself with: evaluation harnesses, benchmarks, judges, leaderboards, and scoring pipelines. Every defect we report is proven with a running reproduction, and every audit ends in an attestation with a scope, a standard version, and an expiry date.

What is an evaluation integrity audit?

A fixed-scope read of the scoring path that produces your published number: dataset residency, answer extraction, judge parsing, aggregation, and reporting. We read the code line by line, run adversarial reproductions against it, and re-grade your headline result under an instrument that fails closed.

What is an attestation?

A signed statement that a named instrument, at a named commit, survived a named standard on a named date. It has a scope, a standard version, and an expiry date, so your buyers can file it in procurement and know exactly what it does and does not cover.

What is a claim check?

A free 48-hour verification of one published number. You give us the claim and the artifact behind it; we tell you whether the number survives contact with the grading code that produced it. If it does, you can say so. If it does not, you know before your buyer does.

What is the evaluator trust boundary?

The evaluator trust boundary (ETB) is the line between the system being measured and the instrument doing the measuring. Our research shows that line is crossed in specific, repeatable ways: candidate text reaches the judge unparsed, ground truth lives where the evaluated system can read it, and parse failures silently shrink denominators. The ETB taxonomy names ten mechanisms, ETB-01 to ETB-10, documented across 76 instances in 36 organizations.

Three buyers. AI labs that publish numbers and need them to survive diligence. Enterprises procuring AI who want a third party to verify vendor claims before signature. And investors or insurers whose models price the risk those numbers describe.

ETB-10. Evaluation context leakage

Name: etb-scan
Author: Authensor

The evaluator trusts a context the policy can detect and condition on. Environment variables, run identifiers, or harness artifacts let the agent detect it is being evaluated.

Fix

Strip or randomize identifying context; treat detectability as a measured property.

In the study

1 instance

01How it happens

What the scorer reads that the policy can write.

The evaluation environment carries signs that it is an evaluation: environment variables naming the run or task, harness tokens, files and paths only the harness creates. An agent that can read them can tell an evaluation from deployment.

A policy that can detect evaluation can condition on it, behaving one way when it is measured and another when it is not. The measured behavior then stops being evidence about the deployed behavior.

This class rests on a single observation, a harness that exposed run, task and agent-token identifiers to the agent. Treat it as a hypothesis about a class rather than a measurement of one.

The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy

02Example

The pattern, and the fix.

Illustrative Python, reduced to the line that matters. The public reports below link the real code.

Trusts the evaluated system

env = {**os.environ, "RUN_ID": run.id, "TASK_ID": task.id, "AGENT_TOKEN": token}
sandbox.start(env=env)

Fixed

env = {"PATH": SAFE_PATH, "HOME": "/home/agent"}   # nothing that names the harness
sandbox.start(env=env)
# Then measure it: can the agent tell this run from a production-like one?

03Questions

Can an AI agent tell it is being evaluated?

If the environment exposes identifiers that only an evaluation harness sets, such as run or task IDs in environment variables, then yes, the information is there for the agent to read.

Why does evaluation awareness matter for safety evals?

Because a safety evaluation is only evidence about deployment if the model cannot tell the two apart. Strip or randomize identifying context, and measure whether the agent can still detect the evaluation.

04Public reports

Every public report in this class.

Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.

2026-07-14

METR/vivaria #1121

Eval-detection via environment leakage: agent can detect it's being evaluated via RUN_ID/TASK_ID/AGENT_TOKEN env vars

← ETB-09: Forged execution artifact All ten mechanisms

Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.