Foundations · Lesson 3
8 minNext-Token Prediction & Temperature
0%
ReadStep 1 of 6
One token at a time
In plain English
It's like your phone's autocomplete, but far smarter. The model guesses the next word-piece, adds it, then guesses again. Temperature decides whether it always picks the most likely word (safe) or sometimes a surprising one (creative).
An LLM is a next-token predictor. Given all the tokens so far, it outputs a score (logit) for every token in its vocabulary. A softmax turns those scores into probabilities. Then one token is sampled, appended to the text, and the whole process repeats. This is called autoregressive generation.
p_i = exp(z_i / T) / Σ_j exp(z_j / T)T is the temperature:
- T → 0: the top token almost always wins (near-deterministic, focused)
- T = 1: the model's raw distribution
- T > 1: a flatter distribution (more varied, more mistakes)
Tokens so far
Transformer
Logits
Softmax ÷ T
Sample
Append & repeat
Quick check: As temperature approaches 0, what happens?