Token (AI)

A token is the basic unit of text a language model processes. A tokenizer splits input into tokens, which may be whole words, parts of words, punctuation, or spaces, and maps each to a number the model can use.

Why it matters

Tokens determine how much text fits in the context window and how much an API call costs, since providers commonly bill input and output tokens separately. They also drive speed, because output is generated one token at a time.

Example

A tokenizer might split text like this (exact splits vary by model):

"Uptime monitoring" -> ["Up", "time", " monitoring"]

As a rough rule of thumb for English, a token is a few characters, so a short paragraph is on the order of 100 tokens. Use the provider's own tokenizer for exact counts.

Common mistakes

• Assuming one token equals one word or that all models tokenize the same way.
• Forgetting that the response also consumes tokens against limits.
• Estimating costs for non-English text or code with English averages, which can differ substantially.

Related

• Context window
• Large language model
• Rate limiting
• Prompt

यह पेज सामान्य जानकारी है, जिसे हमारी संपादकीय टीम ने लिखा है और दस्तावेज़ीकरण बदलने पर अपडेट किया जाता है। सर्वर और प्रदाता अलग-अलग होते हैं, इसलिए बदलाव पहले सुरक्षित तरीके से जाँचें। देखें: संपादकीय नीति और अस्वीकरण.