AI glossary

What is Tokenizer?

The component that splits text into tokens and maps each one to an integer ID before the model processes it.

Most modern tokenizers use variants of byte-pair encoding (BPE) learned from training data. The same sentence can produce different token counts on different models.

▶Practice it hands-onTokens: What Models Actually Read

Related terms

Token

← All 43 AI terms