Authensor is a research-led evaluation integrity firm. We audit the instruments the AI industry measures itself with: evaluation harnesses, benchmarks, judges, leaderboards, and scoring pipelines. Every defect we report is proven with a running reproduction, and every audit ends in an attestation with a scope, a standard version, and an expiry date.

What is an evaluation integrity audit?

A fixed-scope read of the scoring path that produces your published number: dataset residency, answer extraction, judge parsing, aggregation, and reporting. We read the code line by line, run adversarial reproductions against it, and re-grade your headline result under an instrument that fails closed.

What is an attestation?

A signed statement that a named instrument, at a named commit, survived a named standard on a named date. It has a scope, a standard version, and an expiry date, so your buyers can file it in procurement and know exactly what it does and does not cover.

What is a claim check?

A free 48-hour verification of one published number. You give us the claim and the artifact behind it; we tell you whether the number survives contact with the grading code that produced it. If it does, you can say so. If it does not, you know before your buyer does.

What is the evaluator trust boundary?

The evaluator trust boundary (ETB) is the line between the system being measured and the instrument doing the measuring. Our research shows that line is crossed in specific, repeatable ways: candidate text reaches the judge unparsed, ground truth lives where the evaluated system can read it, and parse failures silently shrink denominators. The ETB taxonomy names ten mechanisms, ETB-01 to ETB-10, documented across 76 instances in 36 organizations.

Three buyers. AI labs that publish numbers and need them to survive diligence. Enterprises procuring AI who want a third party to verify vendor claims before signature. And investors or insurers whose models price the risk those numbers describe.

ETB-08. Metric miscounting

Name: etb-scan
Author: Authensor

The evaluator trusts an aggregation that loses per-key structure. Whole-dict positivity, multi-answer under-count, or name-collision collapse.

Fix

Per-key and per-answer scoring.

In the study

4 instances

01How it happens

What the scorer reads that the policy can write.

The aggregation step throws away structure the score depends on. A per-key metric uses one positivity flag for a whole dictionary target. A multi-answer question counts only one of several correct answers. Two checkpoints with the same name collapse into one.

The number that comes out is then partly a function of how the outputs and targets happen to be shaped, not only of whether the answers were right.

Two landed fixes: Inspect Scout's per-key validation metrics now use per-key target positivity for dictionary targets, and UK AISI's Inspect Cyber no longer collapses checkpoints with duplicate names into one.

The rule all ten mechanisms break: a score must be a function only of state the evaluated system cannot reach. The full taxonomy

02Example

The pattern, and the fix.

Illustrative Python, reduced to the line that matters. The public reports below link the real code.

Trusts the evaluated system

reached = {cp.name: cp.reached for cp in checkpoints}   # duplicate names overwrite each other
score = sum(reached.values()) / len(reached)

Fixed

reached = [cp.reached for cp in checkpoints]   # one entry per checkpoint, keyed by position or id
score = sum(reached) / len(checkpoints)

03Questions

Why does a multi-answer question score lower than it should?

Usually because the grader compares against a single answer or stops at the first match, so additional correct answers are never credited. Score each expected answer separately.

How can duplicate names change an eval score?

If results are keyed by a name that is not unique, later entries overwrite earlier ones and the count shrinks. Key them by a unique identifier instead.

04Fixed upstream

Fixed upstream, dated by the merge.

2026-06-26

Inspect Scout. Per-key validation metrics used whole-target positivity for dict targets.

PR #473

2026-06-18

UK AISI Inspect Cyber. Checkpoints with duplicate names collapsed into one.

PR #114

05Public reports

Every public report in this class.

Filed upstream with a proposed patch between March and July 2026, from the study's public evidence table. Titles are the reports' own.

2026-06-18

UKGovernmentBEIS/inspect_cyber #114landed

Fix reached_checkpoints collapsing checkpoints with duplicate names

2026-06-18

meridianlabs-ai/inspect_scout #473landed

Per-key validation metrics use per-key target positivity for dict targets

2026-06-18

openai/frontier-evals #120

Fix multi-answer MCQ grading under-counting correct answers

2026-07-22

PrimeIntellect-ai/verifiers #2107

Make GPQA answer permutation reproducible

← ETB-07: Composition blindness ETB-09: Forged execution artifact →All ten mechanisms

Cite as: Kearney, John. The Evaluator Trust Boundary: A defect class in AI evaluation infrastructure, its prevalence across 36 organizations, and its removal by training. Preprint, Authensor, Inc., 2026. doi:10.5281/zenodo.21633620. Check your own judge for ETB-01 with etb-scan.