A token is the basic unit of text a language model processes. A tokenizer splits input into tokens, which may be whole words, parts of words, punctuation, or spaces, and maps each to a number the model can use.
Why it matters
Tokens determine how much text fits in the context window and how much an API call costs, since providers commonly bill input and output tokens separately. They also drive speed, because output is generated one token at a time.
Example
A tokenizer might split text like this (exact splits vary by model):
"Uptime monitoring" -> ["Up", "time", " monitoring"]As a rough rule of thumb for English, a token is a few characters, so a short paragraph is on the order of 100 tokens. Use the provider's own tokenizer for exact counts.
Common mistakes
- Assuming one token equals one word or that all models tokenize the same way.
- Forgetting that the response also consumes tokens against limits.
- Estimating costs for non-English text or code with English averages, which can differ substantially.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.