Tokenize text, exactly.

The ids Hugging Face tokenizers would give, from toks's C core running as WebAssembly in this tab, and how fast it got there. Nothing leaves your browser.

Text in. The model's ids out, byte for byte what hf tokenizers 0.23.2 returns, for every tokenizer type below.

$ toks encode loading…

measured in this tab, on your machine · wasm64, toks's scalar tier · toks

tokenizer

text

loading toks…

Exact. Every supported tokenizer is checked against Hugging Face tokenizers id for id, 3.96 million cases a night, across both tiers and four platforms.

Fast. A small C library with hand-written NEON and AVX2 kernels, 13-151x faster than hf tokenizers on the speed table. This page runs the portable C twin of those kernels.

Private. The tokenizer and your text stay in this tab: a static page, no server, no telemetry. Apache-2.0, so it can live inside your product too.

A tokenizer that isn't exact isn't a tokenizer.

GitHub Releases

toks is Actual Computer's tokenizer: give it the tokenizer.json a model ships with and it returns exactly the ids Hugging Face tokenizers returns, from a C library with no runtime and no dependencies, plus a Python package with an hf-style API. Built to live inside inference engines; open source under Apache-2.0.