AI glossary

What is Latency?

How long an AI request takes. Time to first token (TTFT) measures when output starts, and total time includes generating every token.

Output generation usually dominates total time. Track p95 latency, not just averages, and stream responses to improve perceived speed.

Related terms