AI latency is how long an AI system takes to respond. It is often measured as the time until the first word appears and the time to finish the whole answer. Short delays feel natural, while long ones make customers give up or talk over a voice assistant.
Latency depends on the model size, the length of the prompt and answer, any extra steps such as searching documents or calling other systems, network distance to the servers and how busy the provider is. Larger, more capable models and "reasoning" modes are usually slower. Streaming the reply as it is generated makes it feel faster.
It matters most for live situations: a website chat, a drive-through voice assistant, a phone answering agent or a staff tool at a busy counter. Background work such as summarising reviews overnight can tolerate slow responses, so a cheaper and slower setup may be fine.
Test at realistic times, not just quiet moments, since latency can spike at peak load. Weigh speed against quality and cost, and consider smaller models, shorter prompts and caching of common answers. Have a fallback, such as a simple message or human handover, when a response takes too long.
Latency shapes whether customers find an AI helpful or frustrating, so it deserves as much attention as accuracy.