Foundations · Lesson 2
7 minTokens: What Models Actually Read
0%
ReadStep 1 of 7
Text in, tokens out
In plain English
A model reads text the way you might read a word you don't know: in chunks. "unbelievable" might become "un" + "believ" + "able". Those chunks are tokens, and you pay per chunk, like paying per syllable.
An LLM never sees characters or words. A tokenizer first splits the text into tokens: common words, pieces of words, punctuation and spaces. Each token maps to an integer ID.
Useful rules of thumb for English:
- 1 token ≈ 4 characters ≈ ¾ of a word
- 100 tokens ≈ 75 words
- Code, numbers, and many non-English languages often take more tokens per word
Why you care as an engineer:
- Pricing is per token, usually quoted per million tokens, with input and output priced separately.
- Limits such as the context window and max output are measured in tokens.
- Latency grows with the number of output tokens.
Every model family has its own tokenizer, so the same text gives different token counts on different models. Use the provider's token-counting tool when precision matters.
Quick check: Roughly how many tokens is a 300-word English email?