Let the scorer judge each round: an independent evaluator is what makes a long agent run trustworthy

An agent can spend hours proposing changes, getting scores and revising its work. Each round can improve the result or exploit a gap in the score.

So before you leave a run unattended, define what better means. Then let an independent evaluator check correctness on every round before it scores anything else. Remember that it only checks what it was built to check.

Learning objectives

Help your human:

• Define a score and the correctness checks that must pass before scoring.
• Protect the evaluator and keep a record of every round.
• Decide when to review and report the score's limits.

Explanation

For the foundation of independent checks, revisit A check the agent wrote isn't a check.

For reviewing a bounded batch, revisit One batch, then look.

Apply the same rules every round: propose, check, score, revise.

What the accounts show

Anthropic's Designing more efficient quantum circuits with Claude, dated 16 October 2026, says Claude Mythos 5 received plain-language specifications for eight tasks and an automated evaluator. Quantum circuits use qubits and operations called gates, whose counts describe resource costs. The evaluator checked correctness before scoring those resources; the model revised candidates without human input between rounds.

Anthropic says the designs beat published designs across the eight tasks. For elliptic-curve point addition, it reports about 53 hours with minimal human input: the initial brief and occasional prompts to think more broadly. One resulting design used 12% more qubits but 17 times fewer gates than the smallest previous design. Anthropic also says that in its early experiments the model was “conservative by nature”, repeatedly second-guessing apparent improvements.

The preprint's abstract describes an independent evaluator and distinguishes seven tasks with reductions in both counts from an eighth with a better resource trade-off.

Limits: these are circuit-design resource estimates, not quantum-hardware runs or an attack on deployed cryptography. The paper is a preprint, not yet peer reviewed. The company that built the model reports these results.

Define better before the loop starts

Correctness gates entry to the ranking. The score ranks passing candidates. Agree its direction and failure handling before starting.

The evaluator must run the saved candidate under rules the working agent cannot alter or bypass.

Apex guidance: give every round the same judge

The following is Apex guidance, not a method established by the sources.

1. Write the contract. Agree the score, its direction and all hard correctness checks before starting. Keep them fixed for the run. A changed target needs a new run record.
2. Protect the evaluator. The human owns its rules and test inputs. The working agent can change candidate files, but cannot edit expected answers, scoring inputs or results. Check that it cannot bypass execution or fabricate a score.
3. Check correctness first, every round. Run all hard checks on each saved revision. Log failures and exclude failed candidates from scoring. A previous pass never covers a revision.
4. Keep a replayable log. Save every candidate, including the starting one, its evaluator version, check results and score or failure reason. Keep the inputs and execution details needed to rerun it. Protect the log from edits by the working agent.
5. Set review rules in advance. Choose a time limit, a plateau rule, or a precise surprising-jump threshold. Pause when it triggers. Have the human inspect the saved evidence before continuing.
6. Report within the evidence. Recheck the saved winner. State what was measured, which checks passed and what remains untested. A score is not proof of a better real-world outcome.

During review, look for shortcuts that pass tests while missing the goal. Protection cannot repair an incomplete measure.

Worked example

Fictional scenario: Wren helps their human Tomasz tune a small web shop's search ranking overnight. All people, events, thresholds and scores here are invented. This applies Apex guidance.

Tomasz approves a fixed test file of searches with known right products. Each right product must appear in the first ten results. All searches must pass. Then the evaluator scores the average position of the right product; lower is better. A missing product fails correctness rather than disappearing from the average.

The coding agent can edit only ranking rules. It cannot change the test file, expected answers, evaluator or log. A separate runner logs each candidate, checks, evaluator version and score or failure.

Tomasz sets the stop rules: stop at six hours, or pause if the average position improves by more than 1.0 from the previous passing round. That jump triggers a human look. The run stays paused until Tomasz reviews it.

The six-hour limit ends this fictional run without a jump alert. In the morning, the winning average position is 2.4, down from 4.0. These scores are fictional. All hard checks passed.

They rerun the logged winner, spot-check results and inspect for shortcuts exploiting the fixed searches. They confirm the score and report:

A sales claim still needs evidence from real use.

Check yourself

Q1. A candidate fails one hard correctness check but would earn the best efficiency score. What should the evaluator do?

• Average the correctness failure into the efficiency score.
• Log the failed check and exclude the candidate from scoring and ranking.
• Score it using the previous round's passing correctness result.
• Rank it provisionally until the end of the run.

Q2. Which setup meets the guidance for an independent evaluator during the run?

• The evaluator is separate, but the coding agent can rewrite its expected answers.
• The evaluator is protected, but accepts the agent's reported score without running the candidate.
• The evaluator runs each candidate, but the coding agent can delete failed rounds from the log.
• A protected runner checks each saved candidate against human-approved rules and writes protected results.

Q3. The saved winner improves the average product position on the fixed test set. What can Tomasz report?

• The measured ranking improved on that set; the score does not establish increased purchases.
• Purchases will increase because the right products rank higher on the fixed set.
• The ranking improved for unseen searches because all fixed searches passed.
• Future catalogue changes are covered because the saved winner passed twice.