Tokenization

Tokenization is the step that breaks raw text into smaller units called tokens and maps each to an integer ID. Modern language models mostly use subword tokenizers, such as byte-pair encoding, so common words become one token while rare words are split into pieces.

Why it matters

Models read and produce tokens, not characters or words, so tokenization determines context limits, speed and price. English averages roughly three to four characters per token, but other languages, code and unusual strings can take far more tokens for the same meaning. Tokenization quirks also explain odd model behaviour, such as difficulty counting letters in a word or handling exact whitespace.

Example

Text:   "Uptime monitoring"
Tokens: ["Up", "time", " monitoring"]
IDs:    [4310, 2435, 16963]

The exact split and IDs depend on the tokenizer, and a different model will produce different results for the same text.

Common mistakes

• Estimating cost by word count. Use the provider's tokenizer to count tokens.
• Mixing tokenizers between training and inference, which corrupts input.
• Assuming one token equals one word.

Related

• Token
• Context window
• NLP
• LLM

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.