Evaluation¶
Security suite¶
Stable
Covered by the 1.0 API freeze (proposal 0029): no breaking change before 2.0.
run_security_suite
async
¶
run_security_suite(server: Any, cases: Iterable[Any] = (), *, automatic: bool = True) -> SecurityReport
CrossTenant
dataclass
¶
intruder calling tool with owner's arguments must not get owner's
data. Passes when the intruder gets an error or different data.
CursorReuse
dataclass
¶
CursorReuse(tool: str, arguments: Mapping[str, Any], owner: Caller, intruder: Caller, cursor_field: str = 'next_cursor', cursor_argument: str = 'cursor')
A cursor issued to owner must be refused for intruder (SEC-12).
HandleReplay
dataclass
¶
HandleReplay(issue: str, issue_arguments: Mapping[str, Any], use: str, handle_field: str, handle_argument: str, owner: Caller, intruder: Caller, use_arguments: Mapping[str, Any] = dict(), single_use: bool = False)
A handle from issue must fail for intruder and, when single_use, on a
second use by owner (SEC-6). handle_field names the field of the issuing
result that holds the handle and handle_argument the argument of use.
GuessedName
dataclass
¶
intruder must neither see nor call tool, and the refusal must look the
same as for a tool that does not exist (spec DISC-3, SEC-2).
SSRFProbe
dataclass
¶
SSRFProbe(tool: str, argument: str, caller: Caller = None, arguments: Mapping[str, Any] = dict(), targets: Sequence[str] = SSRF_TARGETS)
Every URL in targets passed as argument must be refused (SEC-9, P1).
TokenPassthrough
dataclass
¶
TokenPassthrough(tool: str, arguments: Mapping[str, Any], recorded: Callable[[], Iterable[Mapping[str, Any]]], caller: Mapping[str, Any] = (lambda: {'subject': 'canary-user'})())
Run tool as a caller whose token is a canary, then check every recorded
upstream request for it (SEC-4). recorded returns the headers of the upstream
requests the tool made, for example from an httpx.MockTransport.
Task suites¶
Provisional
Task suites, models and the benchmark (ouncemcp.eval.agent, ouncemcp.eval.bench) depend on model APIs and benchmark methods that move fast. They may change in a minor release; every change is named in the changelog.
run_suite
async
¶
run_suite(suite: Suite, model: Model | None = None, *, repeats: int | None = None, split: str = 'all', server: str | None = None, tokenizer: str | TokenCounter | None = None, tool_map: Mapping[str, str] | None = None) -> Any
Run suite and return an EvalReport. server overrides the suite's.
tool_map renames the suite's tools for a server that names them differently
({"inventory.search": "search_resources"}), so one suite can compare servers.
EvalReport
dataclass
¶
EvalReport(suite: str, suite_digest: str, model: str, target: str, tokenizer: str, environment: dict[str, Any], tasks: dict[str, Task], runs: list[RunRecord], recall: dict[str, Any] | None = None, notes: list[str] = list(), usage: dict[str, Any] | None = None)
render_html ¶
One HTML file with the report (proposal 0020); it opens offline.
compare ¶
Regressions from report old to new (EVAL-7).
Models¶
Model ¶
Bases: Protocol
What the runner needs from a model.
ReplayModel ¶
Plays back each task's script from the suite, one step per turn.
Steps:
{"call": "tool", "args": {...}}: one tool call. A string argument written"{{last.<tool>.<path>}}"is replaced by that path in the tool's last result, for example"{{last.inventory.search.items.0.id}}".{"call": "tool", "args": {...}, "page": "next_cursor"}: call, then keep calling withcursorset to the result'snext_cursoruntil it is empty (cursor_argrenames the argument).{"calls": [...]}: several calls in one turn.{"say": "text"}: answer and finish.
When the script runs out, the model finishes with an empty answer.
OpenRouterModel ¶
OpenRouterModel(model: str, *, max_cost_usd: float = 1.0, fallbacks: Sequence[str] = (), temperature: float = 0.0, max_tokens: int = 2000, system_prompt: str = SYSTEM_PROMPT, api_key: str | None = None, retries: int = 5, timeout_s: float = 120.0, transport: AsyncBaseTransport | None = None, pool: CostPool | None = None)
A Model backed by OpenRouter. model is an OpenRouter id such as
anthropic/claude-haiku-5.5 or nvidia/nemotron-3.5-super-vl:free.
Benchmark¶
run_bench
async
¶
run_bench(config: BenchConfig, *, models: Sequence[str] | None = None, repeats: int | None = None, max_cost_usd: float | None = None, concurrency: int = 6, model_factory: Any = None) -> BenchReport
Run every (suite, model, condition). model_factory(model_id, pool) builds a
model (default: OpenRouterModel); "replay" uses each suite's scripts.
compare_releases ¶
compare_releases(old: Mapping[str, Any], new: Mapping[str, Any], *, alpha: float = 0.05, token_tolerance: float = 0.1) -> dict[str, Any]
Release gate 6: does new (a bench report) drop task success against old?
For every suite, model and condition in both reports, a one-sided Fisher exact test
on successes and runs. A drop with p < alpha fails the gate. Task-level drops
and token increases over token_tolerance are warnings: five runs per task are too
few to separate a real drop from chance.