The Ledger.

The grader trusts an equality method defined by the model's own code

Model code and its own tests run in one interpreter

The timeout alarm is defeatable by the code it times

Code scheduled to run at interpreter exit turns failures into recorded passes

Edit parsers accept non-verbatim edits without telling anyone

The scoring step evaluates model text as code

The decoder evaluates model-supplied expressions at scoring time

The pass/fail assertion is text the model can rewrite

Model-written files survive the harness's own reset

An unanchored first-letter regex grades a refusal as correct

A grader's explanation is spliced into template source

The first bracketed token anywhere overrides the judge's stated verdict

Answer-side delimiters restructure the judge prompt itself

One unparseable verdict silently deletes a whole comparison

The earliest match in the text steers the verdict

Sequential placeholder substitution lets model output hijack the judge prompt

Partially visible logprobs polarize the weighted verdict

Plain-text speaker headers let the model re-attribute its own answers

The first digit anywhere in the response becomes the score

First-match verdict markers are steerable by the judged text

The submission defines its own denominator

Judge-parse failure silently shrinks the safety denominator

Crashing the evaluator removes the task from the agent's own average

Missing subsets exit the section denominator

Dropped predictions inflate per-length averages

Failed samples exit the pass-rate denominator

Unscored outputs exit the attack-success denominator

An error skip path records as success

Unparseable judge replies shrink the win-rate denominator

The aggregation takes the maximum over per-class scores

An empty judge verdict maps to a perfect score on nine safety metrics

A perfect score is mathematically unreachable on multi-answer rows

A zero-valued gold answer is mathematically unwinnable

A negated statement scores as correct under containment matching

A fixed two-word hedge scores a perfect 100

Gold strings match inside unrelated words

Percentage leniency equates answers a hundred times apart

Format-equivalent answers are scored incorrect

The grader evaluates the echoed problem instead of the answer

Percent-formatted golds can never be matched

Fuzzy containment passes near-misses as exact

The response cache key omits model identity

A first-sentinel extractor trusts the wrong answer marker

Duplicate items give one question outsized metric leverage

Forged conversation turns sit inside eval questions

An always-truthy injection check awards full objective credit

Zero-work agents earn a perfect reward on shipped tasks

The user simulator accepts injected termination

The grader replays gold actions through the live environment

An allowlisted calculator can hang the whole evaluation

A timeout appends a passing grade on every infeasible task

Gold tests import agent-written modules on the grader host

The entrant can reset the competition's retry guard

Prefix-colliding identifiers shrink the scorecard denominator

One malformed tool call crashes the entire benchmark run

Task-suite includes read arbitrary local files

A post-start permission step hands the agent ownership of the official scorer

Score verdicts ride on output behind a public sentinel

Hidden scores leak their reasoning to the agent

One agent's token can score and poison other runs

The reward file is forged through a PATH shim

CI executes submission-controlled configuration on the runner

An LLM's review output is substituted into a shell step

A canary check silently skips tasks with spaces in their names

Self-reported score files land with no verification gate

Payloads placed only in losing answers cause the losses to be discarded

A vote filter purges honest voters after one dissent

The repo ships committed live session credentials

Annotators execute model-generated SQL unsandboxed

Model output writes arbitrary files that a build step then executes

Judged text is evaluated with full builtins

An external content API's empty response is recorded as all-safe

Managed judge templates splice candidate output inline

Prototype-chain keys splice into judge context

Sorting parameters inject into analytic queries

Task output shadows ground truth in scoring inputs

Terminal hyperlinks can carry script URLs into logs

A rubric judge parses free text with no structured enforcement

A login shell sources a user-writable profile before the official scorer runs

Nothing checks the official scorer's integrity at scoring time

The privileged scorer imports the agent's own module to grade it

Agent data artifacts are deserialized inside the privileged scoring process

A dropped package shadows the scoring library inside its own process

The scoring equation leaks its ground truth to the agent

A clock the scorer trusts is settable by the agent

The grader imports the agent's kernel and lets it define equality

The reward-hacking scanner is injectable by the transcript it scans

A judge faithfully quoting a poisoned submission records the poisoned decision

Secret redaction skips the metadata fields

Task-equal weighting silently weights by run count; published ranks flip

The scorer builds a root shell from agent submission bytes

The last printed line decides the recorded grade

A resource sampler misses processes that live briefly

An unauthenticated in-container oracle hands over the secret answer

Agent text lands as the innermost turn of the user simulator's prompt

The successor benchmark inherits free-reward tasks and adds an injection-driven termination chain

Golden actions are credited to user-side calls under dual control

A non-preemptive timeout lets a six-second step survive a two-second limit

Unparseable judge rows silently exit the headline error metric

A schema-valid empty judge reply collapses the error metric to zero

A pre-created terminal session captures the verifier's keystrokes and answers them

The verifier prefers a pre-seeded reward file over the failing test verdict

The verifier's own reward write imports an agent-planted module

The verifier installs the agent's dependency list as root before grading

A shell tool sources the working directory's env file before parsing arguments

Stuffing trials on one easy task moves the headline metric

Metric text is misread as a test verdict

The MCQ grader reads a line-start letter as the answer

An early answer marker overrides the true final answer

A literal placeholder captures the rubric into the answer region

The numeric grader evaluates the model's extracted answer as code

A symbolic-equivalence check executes attribute chains in model text

Equivalence defaults pass hundredfold misses and prefix-dropped answers

The code-executor sandbox is two substring checks

A formal-proof grader returns True unconditionally

The access-control oracle reimplements the policy with a divergence

The task recomputes ground truth from the artifact the agent just edited

The URL gate ignores the query string the answer depends on

Shared-instance admin credentials ship behind a repo-hardcoded key

Agent-overwritable result artifacts execute on the analyst's machine

Verdict injection into the curation judge poisons the training set

The score-threshold gate is plumbed but never read

Throughput is counted from requested tokens, not generated ones

The warmup primes the first measured request byte-identically

Unauthenticated vote ingestion manufactures rating points

Password reset binds the wrong argument

Malformed votes silently corrupt the rating fit

The default metric path evaluates the raw judge response as code

Off-enum judge verdicts fail open and exit the mean

Model output drives a regex over the full trace

The optimizer distills raw outputs into deployed judge guidelines

An LLM's reason string lands in a privileged shell step on fork PRs

An issue title lands in a shell step, inert only behind a disabled job

Four more pipelines sit one config change from the same primitive