Foundations · Lesson 5
6 minContext Windows, Latency & Cost
0%
ReadStep 1 of 5
Bigger isn't always better
In plain English
A context window is like a desk. A bigger desk fits more papers, but if you pile 500 pages on it, it gets slower, costs more, and the important page in the middle gets missed.
Modern models accept huge context windows, from 100k to over 1M tokens. That doesn't mean you should fill them:
- Cost: you pay for every input token on every call.
- Latency: long prompts increase time to first token (TTFT).
- Quality: research such as "Lost in the Middle" (Liu et al., 2023) found that models use information at the start and end of long contexts better than information buried in the middle.
Latency has two parts:
total time ≈ TTFT + output_tokens / tokens_per_secondOutput generation is usually the slow part. A 1,000-token answer at 50 tokens/s takes 20 seconds.
Streaming the response doesn't make it finish sooner, but users see text right away, so it feels much faster.
Quick check: For a long answer, which part usually takes the most time?