11. Measuring with agents¶
Unit tests prove the server does what you wrote. They do not prove a model can use it. For that you need tasks, a model, and numbers.
A task suite¶
A suite is a YAML file of tasks. Each task has a prompt, the caller, the tools it should use, and checks on the outcome. Here is part of Shelf's suite:
- id: borrow-approved
caller: {subject: ana, tenant: north}
approve: true
prompt: Borrow book b-002 for me.
expect_tools: [loans.borrow]
checks:
- result: loans.borrow
path: book_id
equals: b-002
- state: eval_checks.py:borrowed_once
script:
- call: loans.borrow
args: {book_id: b-002}
- say: You have borrowed b-002.
Checks judge what happened, not what the model said:
| Check | Passes when |
|---|---|
called: tool (+ args) |
The tool was called, with at least these arguments |
not_called: tool |
It was not |
result: tool + path + equals/contains/exists |
Its last successful result matches |
error: tool + category |
Its last call failed with this category |
state: file.py:function |
function(server, run) returns true; it can inspect the server's own state |
answer: [...] |
The final answer contains these texts (for facts the server holds) |
The state checks for Shelf are two lines each, in eval_checks.py: a borrow that was approved wrote exactly one loan; one that was declined wrote nothing.
Replay first¶
Every task can carry a script: the calls a good agent would make. The replay model plays the scripts, with no key and no cost, so the suite runs in CI and proves the tasks and checks are right:
Split Runs Success Select Args ok Calls Turns Tokens p50 ms
all 8 100% 100% 100% 1.0 2.0 2941.25 49.5
train 6 100% 100% 100% 1.0 2.0 2959.67 53.7
heldout 2 100% 100% 100% 1.0 2.0 2886.0 46.0
Train minus held-out success: +0%
The book's tests run this command, so a change that breaks a task breaks the build.
Then real models¶
export OPENROUTER_API_KEY=...
ouncemcp eval examples/book/shelf/eval.yaml --model openrouter:anthropic/claude-haiku-5.5 --max-cost 1.00
--max-cost stops starting runs once the spend reaches it. The report gives, per split:
- Success: tasks whose checks all passed.
- Select: how often the model chose the expected tools.
- Args ok: calls whose arguments validated first time.
- Tokens and turns per task: what a task costs.
Mark some tasks split: heldout and do not tune descriptions on them. If train success climbs while held-out does not, you are fitting your descriptions to your tests.
Save a run with --output before.json, change a description, and compare with --compare before.json --fail-on-regression: the comparison only reports differences that are larger than the noise between repeats.
Benchmarks¶
ouncemcp bench runs several models over several servers and conditions (for example, plain FastMCP against OunceMCP) and writes a report with confidence intervals. The repository's own benchmark is in benchmarks/ and its results are in the M4 report.
Traces¶
In production, record what actually happens:
Each call records its result tokens against the budget, truncation, latency, attempts, outcome and error category. Open the file in the viewer:
The viewer shows the slowest and largest calls, every truncation, and the errors by category. Traces can also go to OpenTelemetry; see traces.
Summary¶
- Write tasks with checks on outcomes, and a script for each.
- Run them with
replayin CI, and with real models before you ship a description change. - Keep held-out tasks you never tune on.
- Trace production calls and read the viewer.