Tokenization

Tokenization is the step that breaks raw text into smaller units called tokens and maps each to an integer ID. Modern language models mostly use subword tokenizers, such as byte-pair encoding, so common words become one token while rare words are split into pieces.

Why it matters

Models read and produce tokens, not characters or words, so tokenization determines context limits, speed and price. English averages roughly three to four characters per token, but other languages, code and unusual strings can take far more tokens for the same meaning. Tokenization quirks also explain odd model behaviour, such as difficulty counting letters in a word or handling exact whitespace.

Example

Text:   "Uptime monitoring"
Tokens: ["Up", "time", " monitoring"]
IDs:    [4310, 2435, 16963]

The exact split and IDs depend on the tokenizer, and a different model will produce different results for the same text.

Common mistakes

• Estimating cost by word count. Use the provider's tokenizer to count tokens.
• Mixing tokenizers between training and inference, which corrupts input.
• Assuming one token equals one word.

Related

• Token
• Context window
• NLP
• LLM

यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.