AI glossary

What is Token?

The unit of text a language model reads and writes: a common word, part of a word, a punctuation mark or a space.

In English, one token is roughly 4 characters or ¾ of a word. API pricing, context windows and output limits are all measured in tokens, and each model family has its own tokenizer.

▶Practice it hands-onTokens: What Models Actually Read

Related terms

Tokenizer Context Window

← All 43 AI terms