The Latency Budget That Shapes Every Conversational AI Decision

Most AI conversations you've had with a machine were bad not because the model was wrong, but because the model was slow. A two-second silence after you stop speaking feels like the system froze. By the time it starts replying, the caller has either repeated themselves, hung up, or started a new thought — and the conversation is over.

Building the AI Calling Assistant, we treated latency not as an optimisation target but as the single constraint that shaped every architectural choice. Here's what that looked like in practice.

The budget

Counting from the moment a caller stops speaking to the moment the AI begins replying, sub-second response is the threshold. Below that, the interaction feels natural. Above it, you can feel the seam.

That single number — less than 1,000 milliseconds — has to absorb every stage of the pipeline:

• End-of-speech detection (the system has to decide the caller is actually done, not just pausing mid-sentence)
• Final speech-to-text transcription (close the streaming transcript, commit the text)
• Dialogue state update (what does this new input mean in the context of the conversation so far?)
• Large-language-model inference (generate the reply)
• Text-to-speech synthesis (turn text into audio)
• Audio encoding and carrier delivery (get the first audio frame on the wire)

If any one of those takes more than 150–200 ms, the whole budget is blown.

What that forces

Everything streams. You don't wait for the final transcript before starting model inference. You don't wait for the full model completion before starting TTS. You don't wait for complete TTS audio before starting playback. The pipeline looks more like a series of coupled conveyor belts than a request-response sequence.

State is in memory, not in a database. In the hot path of a live call, every round-trip to PostgreSQL is an eternity. Session state lives in Redis; persistence writes are dispatched to background workers after the audio has already left. The database never sees the data until the conversation is over (or at least past the current turn).

Provider choice becomes a runtime decision. Different speech-to-text engines, language models, and voice synthesis services have different latency profiles — and they change, quarterly, as vendors release new versions. Hard-coding a provider is locking yourself to whatever its latency is today. Every external service in the system sits behind a common internal interface with provider resolved at runtime from config.

The orchestrator is a state machine, not a chat loop. Pure LLM chat systems can drift, pause to "think," or generate long preambles that blow the latency budget in a single response. We use a central orchestrator that coordinates the language model with specialised capabilities — script adherence, keypad input, compliance checks, call summarisation, fallback handling. The LLM handles free-form understanding; everything latency-critical is deterministic.

What you give up

Speed has a cost. A streaming pipeline makes it harder to do "pre-flight" validation — you can't scan the full caller utterance for prohibited content before responding, because you're already responding while they're still speaking. You compensate with mid-stream interruption: if a compliance signal fires during generation, the TTS cuts out and a safe fallback replaces it. It works, but it's architecturally more complex than a batch pipeline would be.

You also give up the simplicity of "one provider per function." Running adapters across multiple STT, LLM, and TTS vendors means you're constantly A/B-testing in production. The dashboard has a page dedicated to just that: previewing voices, swapping defaults, testing connectivity — because the provider landscape moves faster than any code base.

The payoff

When the pipeline works, the experience is the payoff. Callers don't talk about the AI being fast — they talk about it being conversational. The difference between "frustrating robot" and "trained human agent" is, almost entirely, the 800 milliseconds between the end of their sentence and the start of the reply.

Latency is the architecture. Everything else is downstream of it.