0004: Pluggable tokenizer¶
Status: implemented for M1 (BUD-6, interface, tiktoken encodings and custom callables). Provider token-counting APIs are P1 and not part of this change. See spec section 8.1.2.
Problem¶
Token counts were tied to one tokenizer. TokenCounter could only wrap a tiktoken encoding, OunceServer(tokenizer=...) took only an encoding name, and ouncemcp check and context ignored the server's tokenizer and always counted with o200k_base. The numbers are a consistent estimate and not a host's bill, since model families tokenize differently (M0 report, "Known limitations"). A budget set against one tokenizer can be too loose or too tight for another, and a report that does not say which tokenizer produced a number cannot be compared with another report.
Proposal¶
Choosing a tokenizer¶
from ouncemcp import OunceServer, get_counter
OunceServer("inventory") # o200k_base, as before
OunceServer("inventory", tokenizer="cl100k_base") # any tiktoken encoding
OunceServer("inventory", tokenizer=my_count, tokenizer_name="acme-tokenizer-2.1") # a function
OunceServer("inventory", tokenizer=MyCounter()) # a TokenCounter subclass
counter = get_counter(my_count, name="acme-tokenizer-2.1") # outside a server
get_counter(tokenizer, *, name=None)accepts an encoding name, aTokenCounter, or a callablestr -> int. Encoding counters are cached.nameis required for a bare callable and an error for anything else, because every report states the tokenizer used and a function has no name worth reporting.TokenCounteris now an abstract base class with anameand acount(text)method.count_json, and a defaulttruncate_textthat searches by prefix length, are provided. A subclass only implementscount.TiktokenCounter(encoding)is the previous behavior, with the same exact truncation at token boundaries. An unknown encoding raisesValueErrorthat lists the valid names and points to the callable option.CallableCounter(fn, *, name)wraps a function. Its result must be a non-negative integer, orcountraisesValueErrornaming the tokenizer. Integer-like values (__index__) are accepted, floats andNoneare not.TokenCounter.encoding_namestays as an alias ofname.
Contract for custom counters¶
Shaping and truncation rely on one property: counts grow with the text. A prefix of a text must not have noticeably more tokens than the whole text (a few tokens of slack are fine, as at a token boundary). Counting words, characters or bytes, a vendor tokenizer, and any BPE tokenizer all satisfy this. A counter that is not monotonic can make shaping stop early or keep too much. Results are always checked with the counter's own count, so a result never goes over budget as that counter measures it. tests/test_bud6_tokenizer.py has a Hypothesis test of this for counters of different sizes.
Reports name the tokenizer¶
| Report | Where the name appears |
|---|---|
ouncemcp check |
Tokenizer line in text output, tokenizer in --json (unchanged key, now the counter's name) |
ouncemcp context |
the token summary line ends with counted with <name> |
ServerCost (cost_report, measure_tools) |
tokenizer field, to_dict() and render_text() |
| Tool results | _meta.ouncemcp.tokenizer, next to result_tokens |
| Too-large errors | _meta.ouncemcp.tokenizer, next to min_tokens and budget |
| Startup summary on stderr | already named it |
check and context now count with the server's own tokenizer, and --tokenizer ENCODING overrides it with a tiktoken encoding. For a plain FastMCP server the default is o200k_base. An unknown encoding exits with status 2 and the list of valid names. Counting with a custom function from the command line is not supported, because it would mean loading arbitrary code from a flag. Set it on the server.
Alternatives considered¶
- Accept a callable without a name and use its
__name__. Rejected. Lambdas and partials give<lambda>, and two different functions often share a name, so reports could not be compared. A required name costs one argument. - Keep
TokenCounterconcrete and add acount=option. Rejected. It would make one class both a tiktoken wrapper and a function wrapper, and subclassing for a vendor tokenizer would still need an escape hatch. An abstract base with two small implementations is simpler. - Accept tiktoken model names such as
gpt-4o. Deferred. The report would then name an encoding the user did not type. Encoding names are the stable identifier. A model-to-encoding helper can be added without changing this API. - A
--tokenizeroption that takes a Python path. Rejected for the reason above. - Provider token-counting APIs now. The spec makes them P1. They need network access, credentials and recorded responses for tests. The callable interface lets a user wrap one today, and a built-in adapter can follow.
Impact¶
- Default behavior is unchanged. The default is still
o200k_baseand the inventory example still measures 488 definition tokens and 1,512 tokens for the four-task session. - The same server counts differently under each tokenizer. The inventory example's definitions are 488 tokens with
o200k_base, 454 withcl100k_base, and 500 withp50k_baseandgpt2. Budgets are set against one tokenizer, so changing it can move a tool across the warn or fail thresholds. - No added work per call.
TiktokenCounter.countis the previouscount.
Tests (tests/test_bud6_tokenizer.py): encodings match tiktoken, unknown encodings and misuse of name, caching and pass-through, callable validation and error messages, subclassing, truncation by prefix search (Hypothesis, against a chars/4 counter), the same server reported under three tokenizers with per-tool breakdowns that sum to the total, names in check, context, results and too-large errors, budgets enforced in the chosen tokenizer (a character counter keeps fewer items than o200k_base), shaping within budget for counters of different sizes (Hypothesis), and the CLI flag and the server's own tokenizer.
Migration¶
- Code that built
TokenCounter("enc")directly should callget_counter("enc")orTiktokenCounter("enc").TokenCountercan no longer be instantiated, since it is abstract. Nothing in the repository did this outsideget_counter. ouncemcp checkon a server created withtokenizer=now reports that tokenizer. Before, it reportedo200k_basewhatever the server used.- Clients reading
_meta.ouncemcpget one more key,tokenizer.