Express course · No. 35
Voice changes everything about an AI product. It has to listen, understand, think, and reply — fast enough to feel like a conversation, not a walkie-talkie. Inside is a pipeline of three pieces and one genuinely hard problem: doing it all in real time.
18 concepts · a diagram for each · cheat sheet at the end
A voice AI isn't one thing — it's a chain of three jobs. See the pipeline first and everything else in this course falls into place.
Map: 18 concepts in seven groups
A subway map: first you see the whole lines, then you ride station by station.
The course's concepts fall into seven groups. First, the pipeline itself. Then its edges: the ear (speech to text) and the mouth (text to speech). Then the central difficulty — latency. Then the choreography of conversation — who speaks when. Then the newer path — realtime models. And finally, how to use voice wisely.
How to read each concept
A passport: the same fields for everyone, so they're easy to compare.
Every concept below has the same parts: its English name (remember it — it's how you'll find it in docs), an everyday picture, a diagram, and a card with fields: what it is, how it works, example, catch, and related concepts. One idea runs through the whole course: latency is everything.
Voice Pipeline — ear, brain, mouth
A person in a conversation: they hear what you say, think about it, and reply. Three distinct acts that feel like one.
A voice AI does three things in sequence. It listens — turns speech into text. It thinks — a model reads that text and writes a reply. It speaks — turns the reply back into audio. The stages have names: Speech-to-Text (STT), the model (LLM), and Text-to-Speech (TTS). This chain is also called a cascade. Most voice AI is built exactly this way.
- What it is
A chain of three stages: speech to text, the model, text to speech.
- How it works
- Voice in → transcribe → reason → synthesise → voice out.
- Example
The user says "a table for two", STT writes it down, the model replies, TTS speaks the reply.
- Catch
- Three stages in a row — three sources of delay.
- Related
Made of Speech-to-Text, the LLM Core, and Text-to-Speech; the alternative is Speech-to-Speech.
LLM Core — same model, new edges
You already know how to think and reply in words. Learning to converse by voice just adds listening and speaking around that same core.
The reasoning core is the same language model (LLM) from every other course. Context, prompting, tools, structured output — all of it still applies to the text in the middle. What's new is only the audio on both ends: turning sound into text and text into sound, reliably and fast. You don't relearn the brain. You add the ear and the mouth — and learn the one thing that makes voice hard: doing it all within the tight timing a conversation demands.
- What it is
- A regular text model in the middle of the voice pipeline.
- How it works
- Takes text from STT, writes a text reply for TTS.
- Example
The same prompt, the same tools, and the same output structure as in a text chatbot.
- Catch
- The model only sees text — it doesn't know how something was said.
- Related
The middle of the Voice Pipeline; Speech-to-Speech removes it.
A voice AI is a pipeline: the ear (STT), the brain (the model), and the mouth (TTS). You already know the brain; what's new is the audio at the edges and the timing between the stages.
The first stage turns sound into words. It's remarkably good now, but its mistakes shape everything downstream, so it's worth knowing what it can and can't do.
Speech-to-Text — sound becomes text
A court stenographer typing every word as it's spoken — turning the flow of speech into a record the rest of the system can read.
Speech-to-Text (STT) — also called transcription or ASR (automatic speech recognition) — converts spoken audio into written text. It's the ear of the system: the user talks, and STT produces the words the model will reason over. Modern STT is impressively accurate and handles natural, messy speech far better than the clunky voice recognition of years past. This is the step that lets a person simply talk instead of type.
- What it is
- The ear of the system: turns speech into text.
- How it works
- Listens to audio and outputs the words the model will read.
- Example
- The user speaks — out comes the string "a table for two".
- Catch
- Not perfect: accents, noise, names, and jargon cause errors.
- Related
The first stage of the Voice Pipeline; its mistakes are Error Propagation.
Error Propagation — the ear's mistakes flow downstream
If the stenographer mishears one word, every reader after them inherits the mistake. It's baked in at the start.
STT isn't perfect: accents, background noise, crosstalk, technical terms, and names can all cause mis-transcriptions. And because STT is the first stage, its errors propagate — the model reasons over whatever text it was given. One misheard word can quietly derail the whole response. A voice system is only as good as its transcription. So design for imperfect input: the model should handle slightly garbled text gracefully, and you watch the conditions where the ear struggles most.
- What it is
- A transcription error becomes the input for every later stage.
- How it works
The model doesn't hear the audio — it trusts the text it was given.
- Example
STT mishears a name — the model confidently looks up the wrong person.
- What to do
Design for imperfect input; watch the noise and accents of your real users.
- Related
- The weak spot of Speech-to-Text; the LLM Core can't see it.
Streaming STT — transcribe as the user speaks
A stenographer who types continuously as you talk, not one who waits for the whole speech to finish — the text appears in step with the words.
STT can be streaming — transcribing continuously as the user speaks, rather than waiting for them to finish and then processing the whole clip. This matters for speed: the system starts working sooner and reacts the moment the user stops, instead of adding a long pause. Transcribe-at-the-end versus transcribe-as-you-go is one of the first latency decisions in a voice system, and streaming is usually what makes it feel live.
- What it is
- Transcribing continuously while the user is still speaking.
- How it works
Text builds up during the speech — by the end of the sentence it's almost ready.
- Example
The user finishes — the system replies right away, not after a pause to transcribe.
- Catch
- Without it, every reply gets a noticeable pause in front.
- Related
- The twin of Streaming TTS; part of Stream & Overlap.
STT is the ear: it turns speech into text for the model. Its errors flow downstream, so design for imperfect input — and transcribe as the user speaks.
The last stage turns the model's text reply back into a voice. It's the part the user actually hears, so its quality shapes how the whole product feels.
Text-to-Speech — text becomes a voice
A narrator reading a script aloud — turning written words into natural, expressive speech, not a recital.
Text-to-Speech (TTS), or speech synthesis, converts the model's text reply into audio. It's the mouth of the system. Modern TTS produces remarkably natural voices, with realistic intonation and rhythm, far beyond the flat robotic speech of old. A great pipeline with a robotic voice still feels cheap. A natural voice makes the whole interaction feel human — much of why voice products have become viable at all.
- What it is
- The mouth of the system: turns the reply text into a voice.
- How it works
Takes the model's text and synthesises speech with intonation and rhythm.
- Example
- The string "Table booked" becomes a warm, natural phrase.
- Catch
- A robotic voice makes even a great pipeline feel cheap.
- Related
The last stage of the Voice Pipeline; Voice Persona sets its tone, Streaming TTS its speed.
Voice Persona — the voice carries the experience
The same words said warmly or coldly land completely differently. The listener responds to the delivery, not just the content.
In a voice product, how something is said matters as much as what is said. The chosen voice, its tone, its pacing — these are product decisions that shape trust and comfort, the way visual design does in a screen product. A voice that's too fast, too flat, or oddly inflected undermines an otherwise good answer. So TTS isn't an afterthought bolted on at the end. The voice (the voice persona) is chosen and tuned as deliberately as any other interface element.
- What it is
- The chosen voice, tone, and pace — the "face" of a voice product.
- How it works
- Delivery changes how the very same answer is received.
- Example
"Your order is delayed" in a calm, warm voice — versus the same words, fast and flat.
- Catch
- A default, untuned voice can undercut trust in good answers.
- Related
- A setting of Text-to-Speech; part of Conversation Flow.
Streaming TTS — speak while it writes
Someone who starts speaking their answer as they think of it, rather than silently composing the whole thing and delivering it in one block.
Just as STT can stream in, TTS can stream out — speaking the start of the reply while the rest is still being generated. No need to wait for the full text and full audio before making a sound. The user hears a reply begin almost immediately instead of sitting in silence. Combined with the model streaming its text, this is how a voice agent answers without an awkward gap: the speech starts as soon as there's something to say.
- What it is
- Voicing the start of the reply while the rest is still generated.
- How it works
The model emits text in chunks; TTS turns each chunk into audio right away.
- Example
The first words of the reply sound almost instantly, not after a silence.
- Catch
Without streaming, the user sits in silence while the whole reply is made.
- Related
- The twin of Streaming STT; part of Stream & Overlap.
TTS is the mouth: it voices the model's reply, and modern voices sound natural enough to carry the whole experience. Stream the speech so the reply begins at once, not after a silence.
Here is what makes voice genuinely hard. A conversation has a rhythm, and the whole three-stage pipeline has to fit inside it — or the magic breaks.
Latency Budget — a conversation has a tight window
In real talk, a reply that comes a beat late feels awkward; one that comes after several seconds feels broken. The window for "natural" is small.
People expect a conversational reply within a fraction of a second. A delay that would be invisible in a chat app is glaring in speech — a couple of seconds of silence after you finish talking feels like the system froze. This tight window — the latency budget — is the defining constraint of voice AI. The whole round trip — hear, transcribe, think, synthesise, speak — has to fit inside it. Miss it, and even a perfect answer feels broken.
- What it is
The response-time window a live conversation tolerates — a fraction of a second.
- How it works
- Everything from "hear" to "speak" has to fit inside that window.
- Example
A two-second pause nobody notices in chat sounds like "the system froze" in voice.
- Catch
- Miss it, and even a correct answer feels broken.
- Related
Eaten by Latency Stacking; saved by Stream & Overlap and Speech-to-Speech.
Latency Stacking — the delays add up
A relay race where the total time is every runner's leg combined — a slow handoff anywhere makes the whole team late.
The cruel part is that latency accumulates across the pipeline. Time to transcribe + time for the model to think + time to synthesise — all stack into the pause the user feels. Each stage being "fast enough" alone isn't sufficient if their sum blows the budget. This is why voice is harder than text: a text chat waits only on the model, while voice waits on three stages in series — and the conversational window is far tighter.
- What it is
- The delays of the three stages add up into one total pause.
- How it works
- Total delay = STT + model + TTS, when run strictly in order.
- Example
Every stage is "fast", yet together they leave an awkward silence after each sentence.
- Catch
- Optimising one stage isn't enough — the user feels the sum.
- Related
Breaks the Latency Budget; fixed by Stream & Overlap or Speech-to-Speech.
Stream & Overlap — run the stages in parallel
A simultaneous interpreter who translates as you speak and starts voicing the reply mid-thought — overlapping the stages instead of doing them strictly one after another.
The key technique is to stream and overlap the stages instead of running them strictly in sequence. Transcribe while the user speaks. Let the model start generating before the question is fully processed. Start speaking the reply before it's fully written. Overlapping the three stages, instead of waiting for each to finish, is how a voice system collapses the total delay into something conversational. The whole art of low-latency voice is keeping the pipeline flowing continuously, never stopping and starting at each stage.
- What it is
- All three stages stream and run at the same time, not in turn.
- How it works
STT transcribes as you go, the model starts early, TTS speaks while the model writes.
- Example
A voice agent replies with no gap, though inside it still has the same three stages.
- Catch
A streaming system is harder to build than "finish a stage, pass it on".
- Related
Combines Streaming STT and Streaming TTS; the main way to fit the Latency Budget.
A conversation has a tight latency budget, and the three stages' delays add up. Stream and overlap them — transcribe, think, and speak in parallel rather than in sequence.
Beyond speed, natural conversation has a choreography of who speaks when. That's what separates a real-feeling voice agent from a clumsy one.
Turn-taking — knowing when the user has finished
A good listener can tell a pause for breath from the end of a sentence — they don't jump in every time you take a moment.
A voice system has to detect when the user has actually finished speaking, versus just paused mid-thought. This is turn-taking: knowing when it's the system's turn. Jump in too early and you cut the user off; wait too long and the conversation drags. The judgement is surprisingly hard and central to feeling natural. Humans do it effortlessly from subtle cues; a voice agent has to approximate it well enough that the back-and-forth flows instead of stumbling.
- What it is
- The call: is this pause a breath, or the end of the turn?
- How it works
- The system listens to the pauses and decides when it's its turn.
- Example
"I'd like to book… [pause] …for Friday" — after the first pause it's too early to reply.
- Catch
- Too early cuts the user off; too late makes the conversation drag.
- Related
- Together with Barge-in, it makes up Conversation Flow.
Barge-in — yield when interrupted
When you start talking over someone, they stop and listen — they don't keep plowing through their sentence as if you said nothing.
Real conversation includes interruption. If the user starts speaking while the AI is talking — that's barge-in — a natural system stops, listens, and responds to the new input, instead of finishing its scripted reply over the top of them. Supporting interruption is a hallmark of a good voice agent; a poor one talks over you obliviously. Handling barge-in means the system is always listening, even while it speaks, ready to yield the floor the instant the user takes it.
- What it is
- The ability to stop when the user interrupts.
- How it works
The system listens even while speaking; when it hears a voice, it stops and listens.
- Example
The agent lists options, the user says "the second one!" — the agent switches at once.
- Catch
- Without barge-in the agent talks over people and seems deaf.
- Related
- Together with Turn-taking, it makes up Conversation Flow.
Conversation Flow — the flow is the product
A good conversation partner and an awkward one say similar words — the difference is timing, listening, and giving way. The flow is the experience.
Whether a voice agent feels natural or robotic comes down largely to this choreography: sensing turns, yielding to interruptions, not leaving dead air, and not talking over the user. These conversation flow details matter as much as the accuracy of the answer, because a correct response delivered with clumsy timing still feels broken. Designing a voice product means designing the conversation, not just the answers. The rhythm of who speaks when is a first-class part of the experience, not a polish step.
- What it is
- The rhythm of the talk: who speaks when, pauses, giving way.
- How it works
- Built from turn-taking, barge-in, and no dead air.
- Example
Two agents with identical answers: one feels like a person, the other like an answering machine.
- Catch
- A correct answer with clumsy timing still feels broken.
- Related
Built from Turn-taking and Barge-in; depends on the Latency Budget.
Natural voice needs turn-taking — knowing when the user finished — and barge-in — stopping when interrupted. The rhythm of who speaks when is as much the product as the answers themselves.
The classic cascade is being joined by a newer approach: models that handle voice directly, end to end, collapsing the stages to cut latency. It's worth knowing where the field is heading.
Speech-to-Speech — one model instead of three
Instead of a chain of "a translator writes it down, a thinker answers, a speaker reads it out" — one person who hears, thinks, and replies at once. A direct flight instead of two connections.
A newer class of realtime, or speech-to-speech, models handles audio directly. Voice in, voice out, without separate transcribe → reason → synthesise steps. One model instead of three in a chain — the end-to-end approach. The big win is latency: no boundaries between stages means no delays passing data between them. The second win is that nothing is lost in conversion: the model hears how something was said, not just the words, keeps tone and emotion, and can reply with matching expression.
- What it is
- One model that takes voice in and answers directly in voice.
- How it works
No separate STT and TTS — so no hand-offs between stages and no hand-off delays.
- Example
The model hears irritation in the user's voice and answers more calmly — a text cascade can't see that.
- Catch
- The middle is a black box: less visibility and control.
- Related
The alternative to the Voice Pipeline; the most direct way into the Latency Budget.
Pipeline vs End-to-End — a real choice
You can travel by a series of trains and control each leg, or take one direct service that's faster but gives you less say over the route. Different trade-offs, both valid.
The classic pipeline and the end-to-end model are both real options, each with a price. The pipeline gives you control and flexibility at every stage — swap the STT, inspect the transcript, use any model — at the cost of latency and complexity. End-to-end gives speed and naturalness but less visibility and control over the middle. Knowing both exist lets you choose deliberately: the transparent, flexible pipeline or the fast, fluid end-to-end approach, depending on what your product needs most.
- What it is
- The architecture choice for a voice product.
- Pipeline
Swap STT/TTS, read the transcript, use any model. Cost: latency and more moving parts.
- End-to-end
Lowest latency, tone and emotion preserved. Cost: less visibility and control.
- When to use which
Need control and inspectability — pipeline; need speed and fluidity — end-to-end.
- Related
- Compares the Voice Pipeline and Speech-to-Speech.
Realtime models take voice in and out directly: fewer stages, less latency, and tone survives. Pipeline versus end-to-end is a real trade-off: control and flexibility, or speed and naturalness.
Using voice well means choosing it for the right situations and respecting that latency, more than anything, decides whether it feels magical or broken.
Modality Fit — voice where it belongs
You'd rather speak directions while driving and type a long form at your desk — the right input depends entirely on the situation.
Voice is powerful where it fits: hands busy, accessibility, natural conversation, situations where talking is faster or safer than typing. It's a poor fit where text is better: precise input, privacy in public, careful review. The skill is matching the modality to the moment, not adding voice because it's impressive. Voice where text would serve better is a downgrade; voice where speaking is the natural act is transformative. Choose it for the situation, not the novelty.
- What it is
- Choosing voice or text for the specific situation.
- Voice fits
- Hands busy, driving, accessibility, natural conversation.
- Text fits
- Precise input, privacy in public, careful review.
- Catch
- Voice "because it's cool" where text is better is a downgrade.
- Related
The first question before anything else; without it, Latency-First is moot.
Latency-First Design — budget for latency from day one
A chef who plans the whole meal around the dish that takes longest — designing around the binding constraint, not discovering it at the end.
Because latency decides everything, design for it from the start (latency-first): streaming at every stage, an end-to-end model where speed is critical, and a plan for when a stage is slow. Voice features that ignore latency until late tend to feel sluggish and broken no matter how good the answers are. Treat response time as the central design constraint of a voice product — the one number that, missed, ruins everything else.
- What it is
- Response time is the main constraint everything is built around.
- How it works
Set the budget → stream everywhere → end-to-end where critical → plan for a slow stage.
- Example
Time the full round trip in the first prototype and reject decisions that break it.
- Catch
- Latency discovered at the end is hard to fix.
- Related
Protects the Latency Budget; relies on Stream & Overlap and Speech-to-Speech.
How it all connects
The whole subway map at once: now you can see where the lines cross.
The cascade causes latency stacking, which hits the budget. Streaming and overlap, or speech-to-speech, rescue it. Fit the budget and build a conversation flow from turn-taking and barge-in, and the conversation feels alive. And modality fit decides whether you need voice at all.
- Does voice genuinely fit the situation — or would text serve the user better?
- Is each stage streaming — transcribe, think, and speak overlapping, not in strict sequence?
- Does the total latency feel conversational, or is there an awkward gap?
- Does it handle turn-taking — knowing when the user finished — and barge-in?
- Is the voice natural enough to carry the experience?
- Pipeline or end-to-end — have I chosen the architecture for my latency and control needs?
- You choose voice for situations where speaking genuinely fits, not for novelty.
- Every stage streams and overlaps, so the reply starts almost instantly.
- The total latency feels conversational, with no awkward silence.
- The agent handles turn-taking and barge-in, so the flow feels natural.
- You chose pipeline or end-to-end deliberately, designing around the latency budget.
Cheat sheet: all 18 concepts
Pipeline
- Voice Pipelinecascade
STT → model → TTS: ear, brain, mouth in sequence.
- LLM Corethe reasoning core
The same text model in the middle; only the edges are new.
Listen
- Speech-to-TextSTT, ASR
The ear: turns speech into text.
- Error Propagationerrors flow downstream
A transcription error becomes the model's input.
- Streaming STTtranscribe as you go
Text is ready by the end of the sentence — no pause before reacting.
Speak
- Text-to-SpeechTTS, synthesis
The mouth: turns the reply text into a voice.
- Voice Personavoice and delivery
Tone and pace are part of the product, not a detail.
- Streaming TTSstreaming synthesis
Starts speaking while the reply is still being written.
Latency
- Latency Budgetresponse-time window
Reply within a fraction of a second — or it feels frozen.
- Latency Stackingaccumulated latency
The three stages' delays add up into one.
- Stream & Overlapstages in parallel
All stages streaming at once — the main way to make it in time.
Turns
- Turn-takingturn detection
Tell a pause for breath from the end of a sentence.
- Barge-ininterruption handling
Interrupted — stop, listen, respond.
- Conversation Flowrhythm of the talk
Who speaks when is as much the product as the answers.
Realtime
- Speech-to-Speechrealtime, end-to-end
One model: voice in, voice out. Less latency, tone survives.
- Pipeline vs End-to-Endarchitecture choice
Control and flexibility versus speed and naturalness.
Using it well
- Modality Fitwhen voice belongs
Voice where speaking is natural; text where precision matters.
- Latency-First Designlatency from day one
Build everything around the response-time budget.
Voice means listen, think, and speak fast enough to feel like a conversation. Stream and overlap the pipeline, handle turn-taking and interruptions, choose pipeline or end-to-end deliberately — and treat latency as the constraint everything else bends to.