How LLMs Are Built (and Why They Hallucinate)

ReadStep 1 of 8

Three stages from raw text to assistant

In plain English

First the model reads a huge library (pretraining). Then it goes to etiquette school to learn to answer questions (instruction tuning). Then people give it feedback on which answers are better (preference tuning).

Every chat model you use went through roughly these stages:

Pretraining. Next-token prediction over a huge corpus of text and code. This is where the model learns language, facts and patterns. The result is a base model: a powerful text continuer.
Instruction tuning (supervised fine-tuning, SFT). Training on example conversations written to show good behavior. The model learns to follow instructions and answer in a helpful format.
Preference tuning. People (or AI raters guided by written principles) compare answers, and the model is trained toward the preferred ones. Methods include RLHF (reinforcement learning from human feedback) and DPO (direct preference optimization).

Newer reasoning models add reinforcement learning on problems with checkable answers, such as math and code, which teaches them to work through long chains of reasoning before answering.

Pretraining data stops at a knowledge cutoff. The model knows nothing after that date unless you put it in the prompt.

Web-scale text

Pretraining

Base model

Instruction tuning

Preference tuning

Assistant

Quick check: Which stage produces a model that simply continues text, without following instructions?