Evaluating with task suites¶
ouncemcp eval runs tasks against a server the way an agent would and judges the outcome on the server (proposal 0019).
uv pip install -e ".[eval]"
ouncemcp eval examples/inventory/eval.yaml
ouncemcp eval examples/inventory/eval.yaml --output baseline.json
ouncemcp eval examples/inventory/eval.yaml --compare baseline.json --fail-on-regression
A task:
- id: close-approved
caller: {subject: ana, tenant: acme} # in-process servers: run as this verified caller
approve: true # answer to approval requests
prompt: Close ticket T-1.
expect_tools: [tickets.close]
checks:
- result: tickets.close
path: status
equals: closed
- state: eval_checks.py:t1_closed_once # function(server, run) -> bool
script: # what the replay model does
- call: tickets.close
args: {ticket_id: T-1}
- say: Ticket T-1 is closed.
- Put some tasks in
split: heldoutand keep them out of description tuning; the report shows the train minus held-out gap. - Use
{{last.<tool>.<path>}}in script arguments to reuse a value from an earlier result, andpage: next_cursorto follow cursors. --serverruns the same suite against another server, for example a plain FastMCP version, to compare tokens per task.-
The replay model checks the server, not a model's judgment. To measure a real model, run the same suite through OpenRouter (proposal 0026):
export OPENROUTER_API_KEY=... ouncemcp eval examples/inventory/eval.yaml --model openrouter:anthropic/claude-haiku-5.5 \ --repeats 3 --max-cost 2.00--max-cost(default $1.00) stops the suite before it spends more. The report shows the provider's token counts, the cost per task and which model answered. Real models vary between runs, so use several--repeatsand compare rates. - Write checks about outcomes, not about a script: a real model may reach the right result by another path, or rightly refuse.approve: falsemakes the simulated user decline confirmations, for tasks where the user would not want the write.
--html report.html writes the report as one page that opens offline: splits, tasks with their failed checks, regressions and the environment (proposal 0020).
Read traces from a run with ouncemcp trace calls.jsonl (traces).