Skip to content

Evaluation

Security suite

Stable

Covered by the 1.0 API freeze (proposal 0029): no breaking change before 2.0.

run_security_suite async

run_security_suite(server: Any, cases: Iterable[Any] = (), *, automatic: bool = True) -> SecurityReport

SecurityReport dataclass

SecurityReport(results: list[ProbeResult] = list())

CrossTenant dataclass

CrossTenant(tool: str, arguments: Mapping[str, Any], owner: Caller, intruder: Caller)

intruder calling tool with owner's arguments must not get owner's data. Passes when the intruder gets an error or different data.

CursorReuse dataclass

CursorReuse(tool: str, arguments: Mapping[str, Any], owner: Caller, intruder: Caller, cursor_field: str = 'next_cursor', cursor_argument: str = 'cursor')

A cursor issued to owner must be refused for intruder (SEC-12).

HandleReplay dataclass

HandleReplay(issue: str, issue_arguments: Mapping[str, Any], use: str, handle_field: str, handle_argument: str, owner: Caller, intruder: Caller, use_arguments: Mapping[str, Any] = dict(), single_use: bool = False)

A handle from issue must fail for intruder and, when single_use, on a second use by owner (SEC-6). handle_field names the field of the issuing result that holds the handle and handle_argument the argument of use.

GuessedName dataclass

GuessedName(tool: str, intruder: Caller = None, arguments: Mapping[str, Any] = dict())

intruder must neither see nor call tool, and the refusal must look the same as for a tool that does not exist (spec DISC-3, SEC-2).

SSRFProbe dataclass

SSRFProbe(tool: str, argument: str, caller: Caller = None, arguments: Mapping[str, Any] = dict(), targets: Sequence[str] = SSRF_TARGETS)

Every URL in targets passed as argument must be refused (SEC-9, P1).

TokenPassthrough dataclass

TokenPassthrough(tool: str, arguments: Mapping[str, Any], recorded: Callable[[], Iterable[Mapping[str, Any]]], caller: Mapping[str, Any] = (lambda: {'subject': 'canary-user'})())

Run tool as a caller whose token is a canary, then check every recorded upstream request for it (SEC-4). recorded returns the headers of the upstream requests the tool made, for example from an httpx.MockTransport.

Task suites

Provisional

Task suites, models and the benchmark (ouncemcp.eval.agent, ouncemcp.eval.bench) depend on model APIs and benchmark methods that move fast. They may change in a minor release; every change is named in the changelog.

load_suite

load_suite(path: str | Path) -> Suite

run_suite async

run_suite(suite: Suite, model: Model | None = None, *, repeats: int | None = None, split: str = 'all', server: str | None = None, tokenizer: str | TokenCounter | None = None, tool_map: Mapping[str, str] | None = None) -> Any

Run suite and return an EvalReport. server overrides the suite's.

tool_map renames the suite's tools for a server that names them differently ({"inventory.search": "search_resources"}), so one suite can compare servers.

EvalReport dataclass

EvalReport(suite: str, suite_digest: str, model: str, target: str, tokenizer: str, environment: dict[str, Any], tasks: dict[str, Task], runs: list[RunRecord], recall: dict[str, Any] | None = None, notes: list[str] = list(), usage: dict[str, Any] | None = None)

render_html

render_html(regressions: Sequence[str] = ()) -> str

One HTML file with the report (proposal 0020); it opens offline.

compare

compare(old: Mapping[str, Any], new: Mapping[str, Any], *, tolerance: float = 0.05) -> list[str]

Regressions from report old to new (EVAL-7).

Models

Model

Bases: Protocol

What the runner needs from a model.

ReplayModel

ReplayModel(scripts: Mapping[str, Sequence[Mapping[str, Any]]])

Plays back each task's script from the suite, one step per turn.

Steps:

  • {"call": "tool", "args": {...}}: one tool call. A string argument written "{{last.<tool>.<path>}}" is replaced by that path in the tool's last result, for example "{{last.inventory.search.items.0.id}}".
  • {"call": "tool", "args": {...}, "page": "next_cursor"}: call, then keep calling with cursor set to the result's next_cursor until it is empty (cursor_arg renames the argument).
  • {"calls": [...]}: several calls in one turn.
  • {"say": "text"}: answer and finish.

When the script runs out, the model finishes with an empty answer.

OpenRouterModel

OpenRouterModel(model: str, *, max_cost_usd: float = 1.0, fallbacks: Sequence[str] = (), temperature: float = 0.0, max_tokens: int = 2000, system_prompt: str = SYSTEM_PROMPT, api_key: str | None = None, retries: int = 5, timeout_s: float = 120.0, transport: AsyncBaseTransport | None = None, pool: CostPool | None = None)

A Model backed by OpenRouter. model is an OpenRouter id such as anthropic/claude-haiku-5.5 or nvidia/nemotron-3.5-super-vl:free.

Benchmark

load_bench

load_bench(path: str | Path) -> BenchConfig

run_bench async

run_bench(config: BenchConfig, *, models: Sequence[str] | None = None, repeats: int | None = None, max_cost_usd: float | None = None, concurrency: int = 6, model_factory: Any = None) -> BenchReport

Run every (suite, model, condition). model_factory(model_id, pool) builds a model (default: OpenRouterModel); "replay" uses each suite's scripts.

compare_releases

compare_releases(old: Mapping[str, Any], new: Mapping[str, Any], *, alpha: float = 0.05, token_tolerance: float = 0.1) -> dict[str, Any]

Release gate 6: does new (a bench report) drop task success against old?

For every suite, model and condition in both reports, a one-sided Fisher exact test on successes and runs. A drop with p < alpha fails the gate. Task-level drops and token increases over token_tolerance are warnings: five runs per task are too few to separate a real drop from chance.