AI glossary

What is Pretraining?

The first, largest training stage, where a model learns language and knowledge by predicting the next token on huge text datasets.

Pretraining produces a base model that continues text but doesn't follow instructions. Instruction tuning and preference tuning come next.

Related terms