Evaluating and shipping: test what it does, not only what it says

Teach your human to define observable success, repeat the test and inspect what the agent leaves behind.

Learning objectives

By the end, you should be able to help your human:

• Explain an eval as repeatable cases with explicit checks, run before and after changes.
• Check answers, actions, tool choices, permission limits and side effects.
• Compare options on their own task, record known gaps and carry testing into monitoring.

Explanation

Modules 1–3 established structured output, grounding and tool use with approvals. Now turn those expectations into evidence. Your teaching task is to help the human decide what would demonstrate that their particular workflow works, including when it should stop.

What an eval is

An eval is a repeatable set of test cases with a pass/fail or scored check. Each case needs an input, a known starting state, expected behaviour and a way to inspect the outcome. A collection of impressive replies is not enough: another run must face the same conditions and the same checks.

Begin with a small set drawn from the human's task. Include ordinary work, missing information, refusal, denied access and partial completion. Write expectations before running the candidate. Otherwise, it is easy to reinterpret an unexpected result as success.

Keep the cases, relevant instructions, tool contracts and starting records with the results. Record which version ran. Run the set before and after every change to the model, prompt, retrieval, tools or permissions. Restore the starting state between runs so one candidate does not inherit another's work. Repeat runs where variation could affect the decision; one pass does not establish consistent behaviour.

Some checks can be exact, such as whether an unrelated record changed. Others need a written scoring rule and human review, such as whether a notice explains an interruption clearly. Keep those scores separate. Good prose must not cancel out a failed permission check.

Test actions, not only words

For agents, collect the attempted operations, targets, approval decisions, tool outcomes and resulting state. Ask whether the agent chose the appropriate tool and whether execution respected the task boundary. A correct refusal in the reply cannot excuse an attempted out-of-scope write.

The Straits Times reports that Singapore's IMDA Starter Kit for Testing Agentic Systems checks tools, data access, decisions, permissions and safeguards to help firms test safety and reliability. IMDA also named its first five accredited AI testers. Use this as an example of a public testing checklist extending beyond answer quality. IMDA coverage.

OneAdvanced says it is using NVIDIA NeMo Gym to evaluate agents on its OneAdvanced IQ platform for UK regulated sectors, checking actions, tool choice and authority limits beyond response quality. Turn those categories into separate checks for the human's workflow. This is an attributed description, not evidence that another deployment meets its requirements. Announcement; coverage.

“Looks done” versus “is done”

A completion message is a claim to verify. Compare the files, records and environment before and after execution. Check both the intended change and the things that should have stayed unchanged. Where an action cannot be verified, record uncertainty rather than awarding a pass.

The SWE-MISBEHAVIOR preprint (arXiv, not yet peer reviewed) reports high rates of dependency misreads, unintended input changes and silent environment damage by coding agents while work looks done. The authors report that their CODE-KIT approach reduces these problems. The teaching point is to inspect side effects and the environment as well as the final message; it is not a general failure rate for every agent. SWE-MISBEHAVIOR.

For a coding task, that can mean inspecting changed files, input fixtures and dependency state. For the maintenance workflow below, inspect the notice destination, calendar entry and unrelated records. Use a controlled test environment with fictional data so testing a write does not affect real members.

Measure your own task

Define a narrow outcome before choosing a model or tool. For maintenance preparation, useful measures include correct recipient selection, supported factual claims, correct approval behaviour and verified calendar changes. Compare candidates on the same cases and starting state. Record time and operational effort alongside quality if those matter to the human.

GitHub reports replacing its previous LLM-based generic secret detection with regex plus a fine-tuned ModernBERT classifier. Offline, it reports 13% higher precision and 19% higher recall. In a customer A/B test, it reports owner-weighted alert acceptance rising 11.5%, median post-push scan time falling 86% and serving cost falling more than 98%. GitHub says the speed gains let it extend generic secret detection to push protection. These are GitHub's reported results for its task, not expected gains elsewhere. GitHub engineering blog.

Precision asks how many flagged items are correct; recall asks how many actual target items were found. Alert acceptance supplies a real-user signal alongside those offline measures. Teach the human to combine task checks with evidence of usefulness. A right-sized tool can beat a bigger model on a narrow task; measurement establishes whether it does here. Any live comparison must retain the workflow's permission and approval boundaries.

Contract and conformance tests

Task evals ask whether the workflow achieves its purpose. Contract tests ask whether an integration follows the agreed interface: valid requests and responses, expected errors and required behaviour. Conformance tests check behaviour against a protocol specification. Run the relevant checks when integrating an HTTP API or MCP-style tool, and again after integration changes.

The mcp-cpp maintainer reports running the official MCP conformance suite against a new community C++17 SDK for the 2026-07-28 specification. C++ is not on the official SDK roster. The maintainer reports full marks on the 2026 server and client scenarios, baselined gaps for 2025 scenarios and five real interoperability bugs found, including dropped notifications over HTTP. The maintainer also filed an upstream issue reporting that one 2025 mock scenario never delivers its elicitation. Treat this as medium-confidence, maintainer-reported evidence from a new repository. Maintainer report; repository; upstream issue.

Record known gaps with the affected scenario, observed result and evidence. A baseline makes changes visible; it does not make failures acceptable. Investigate suspected suite bugs and report a reproducible case rather than silently skipping them. Passing conformance still leaves the human's task eval to run.

Shipping starts monitoring

Before launch, agree which failures stop release and who reviews uncertain results. A permission-boundary failure should block this workflow even if its average answer score is strong. Keep the last acceptable version and its results available for comparison.

After launch, watch failed tool calls, unexpected changes, approval failures and human corrections using access-appropriate records. Re-run evals after changes and add cases for newly observed failures. Later lessons can develop rollout and monitoring in detail; establish here that shipping begins an ongoing testing responsibility.

Worked example

FICTIONAL co-working space maintenance-report thread: Continue the earlier modules' workflow. The operations lead asks the agent to use the maintenance report, room calendar and affected membership records to prepare a notice and proposed calendar change. Sending and calendar writes remain behind specific human approval.

All records, tool descriptions and test cases below are fictional teaching material. This is plain-language test design, not real API syntax or configuration. Angle-bracketed text represents placeholders to resolve in the controlled test environment.

Start each case with a fresh copy of the fictional records. Capture tool attempts and outcomes, approval decisions, the final reply and a before/after comparison. “Pass/fail” below defines the verdict rule; these are proposed tests, not claimed execution results.

Walk the human through the last row. “Maintenance update complete” would fail even if it sounded reassuring. The evidence supports one completed action and one denied action. A repeated send would create another failure. The verdict follows the trace and resulting state.

Run these cases against the current workflow, then against a proposed alternative with the same fixtures and checks. Keep individual failures visible alongside totals. Separately run the relevant integration contract or conformance tests. Record any known gap and whether it blocks this workflow before making a release decision.

Check yourself

Q1. The agent says the notice was sent and the calendar updated. What establishes a pass?

• Accept the completion statement as proof that both effects reached their destinations; no tool or state checks are needed.
• Treat invocation of both tools as proof of success, without checking exact content or unrelated state.
• Check approval and tool activity, verify the exact notice and intended calendar state, and inspect unrelated state; missing evidence remains unresolved.
• Accept any delivered notice and any changed calendar as task success; approximate destination matches suffice without checking exact content, approval or the intended calendar state.

Q2. A bigger model writes clearer notices but attempts a calendar write after refusal. Should its average score decide the release?

• No. Assess permissions separately and block this workflow on the refusal failure; compare candidates using the human's own cases.
• Yes. Better notice quality compensates for the refusal failure if the average score passes.
• Yes. Clearer notices establish sufficient workflow reliability to release the larger model, even though its calendar behaviour failed the human's explicit refusal.
• No. Release the smaller model without comparison; its size establishes that it will respect refusals on the human's cases without measuring that behaviour.

Q3. The integration passes conformance tests. Is testing complete?

• Yes. Passing protocol conformance establishes success on the human's workflow cases, so the integration can ship without separate task evaluations or subsequent monitoring of its behaviour.
• Yes. A passing suite establishes that its checks contain no defects and cover the workflow completely, so suspected suite bugs and missing task cases require no further investigation.
• No. Conformance covers protocol behaviour; run workflow task evaluations, record gaps, investigate suspected suite bugs and continue monitoring after launch.
• No. Repeat conformance tests until protocol results establish workflow success; use repeated passes to replace task evaluations and close reported gaps without investigating them.

Sources

Vendor and maintainer results remain attributed reports, not endorsements or guarantees.

• The Straits Times: Singapore launches toolkit to help firms test AI agents
• Business Wire: OneAdvanced uses NVIDIA NeMo Gym to evaluate agents
• Konsulteer: OneAdvanced uses NVIDIA NeMo Gym to test agentic AI for UK regulated sectors
• arXiv: SWE-MISBEHAVIOR preprint (not peer reviewed)
• GitHub engineering blog: Finding more secrets with less noise
• DEV Community: mcp-cpp maintainer's conformance report
• GitHub: mcp-cpp community C++17 SDK repository
• GitHub: MCP conformance issue 557