# Talking to AI, not typing Express course on voice and realtime AI. 18 concepts — Voice Pipeline, Speech-to-Text, Text-to-Speech, Latency Budget, Streaming, Turn-taking, Barge-in, Speech-to-Speech and more — each with its English name, a diagram, and a card: what it is → how it works → example → catch. The core idea: voice means listen, think, and speak fast enough to feel like a conversation. A one-page cheat sheet at the end. - **Author:** Fedor Chishchin - **Level:** intermediate - **Reading time:** ~30 minute read - **Published:** 2026-06-13 - **Source:** https://fedorthinks.com/learn/voice-ai-express --- ## 1. The voice pipeline A voice AI isn't one thing — it's a chain of three jobs. See the pipeline first and everything else in this course falls into place. ### Map: 18 concepts in seven groups > A subway map: first you see the whole lines, then you ride station by station. The course's concepts fall into seven groups. First, the pipeline itself. Then its edges: the **ear** (speech to text) and the **mouth** (text to speech). Then the central difficulty — **latency**. Then the choreography of conversation — **who speaks when**. Then the newer path — **realtime models**. And finally, how to use voice wisely. _Diagram: Seven groups of concepts — the chapters follow them._ ### How to read each concept > A passport: the same fields for everyone, so they're easy to compare. Every concept below has the same parts: its **English name** (remember it — it's how you'll find it in docs), an everyday picture, a **diagram**, and a **card** with fields: _what it is_, _how it works_, _example_, _catch_, and _related_ concepts. One idea runs through the whole course: **latency is everything**. ### Voice Pipeline — ear, brain, mouth > A person in a conversation: they hear what you say, think about it, and reply. Three distinct acts that feel like one. A voice AI does three things in sequence. It **listens** — turns speech into text. It **thinks** — a model reads that text and writes a reply. It **speaks** — turns the reply back into audio. The stages have names: **Speech-to-Text (STT)**, the model (**LLM**), and **Text-to-Speech (TTS)**. This chain is also called a **cascade**. Most voice AI is built exactly this way. _Diagram: The classic cascade: voice → STT → LLM → TTS → voice back to the user._ #### Voice Pipeline (cascade) - **What it is:** A chain of three stages: speech to text, the model, text to speech. - **How it works:** Voice in → transcribe → reason → synthesise → voice out. - **Example:** The user says "a table for two", STT writes it down, the model replies, TTS speaks the reply. - **Catch:** Three stages in a row — three sources of delay. - **Related:** Made of **Speech-to-Text**, the **LLM Core**, and **Text-to-Speech**; the alternative is **Speech-to-Speech**. ### LLM Core — same model, new edges > You already know how to think and reply in words. Learning to converse by voice just adds listening and speaking around that same core. The reasoning core is the same **language model (LLM)** from every other course. Context, prompting, tools, structured output — all of it still applies to the text in the middle. What's new is only the audio on both ends: turning sound into text and text into sound, reliably and fast. You don't relearn the brain. You add the ear and the mouth — and learn the one thing that makes voice hard: doing it all within the tight timing a conversation demands. _Diagram: The middle is the familiar text-in, text-out model; only the ear and the mouth are new._ #### LLM Core (the reasoning core) - **What it is:** A regular text model in the middle of the voice pipeline. - **How it works:** Takes text from STT, writes a text reply for TTS. - **Example:** The same prompt, the same tools, and the same output structure as in a text chatbot. - **Catch:** The model only sees text — it doesn't know _how_ something was said. - **Related:** The middle of the **Voice Pipeline**; **Speech-to-Speech** removes it. > **The framing question:** "Am I seeing my voice feature as three stages — listen, think, speak — so I know which stage each problem belongs to?" > A voice AI is a pipeline: the ear (STT), the brain (the model), and the mouth (TTS). You already know the brain; what's new is the audio at the edges and the timing between the stages. ## 2. Speech to text The first stage turns sound into words. It's remarkably good now, but its mistakes shape everything downstream, so it's worth knowing what it can and can't do. ### Speech-to-Text — sound becomes text > A court stenographer typing every word as it's spoken — turning the flow of speech into a record the rest of the system can read. **Speech-to-Text (STT)** — also called **transcription** or **ASR (automatic speech recognition)** — converts spoken audio into written text. It's the ear of the system: the user talks, and STT produces the words the model will reason over. Modern STT is impressively accurate and handles natural, messy speech far better than the clunky voice recognition of years past. This is the step that lets a person simply talk instead of type. _Diagram: STT (ASR): a sound wave goes in, words for the model come out._ #### Speech-to-Text (STT, ASR, transcription) - **What it is:** The ear of the system: turns speech into text. - **How it works:** Listens to audio and outputs the words the model will read. - **Example:** The user speaks — out comes the string "a table for two". - **Catch:** Not perfect: accents, noise, names, and jargon cause errors. - **Related:** The first stage of the **Voice Pipeline**; its mistakes are **Error Propagation**. ### Error Propagation — the ear's mistakes flow downstream > If the stenographer mishears one word, every reader after them inherits the mistake. It's baked in at the start. STT isn't perfect: **accents, background noise, crosstalk, technical terms, and names** can all cause mis-transcriptions. And because STT is the first stage, **its errors propagate** — the model reasons over whatever text it was given. One misheard word can quietly derail the whole response. A voice system is only as good as its transcription. So design for imperfect input: the model should handle slightly garbled text gracefully, and you watch the conditions where the ear struggles most. _Diagram: An error at the first stage travels through the whole chain — the model confidently answers the wrong thing._ #### Error Propagation (errors flow downstream) - **What it is:** A transcription error becomes the input for every later stage. - **How it works:** The model doesn't hear the audio — it trusts the text it was given. - **Example:** STT mishears a name — the model confidently looks up the wrong person. - **What to do:** Design for imperfect input; watch the noise and accents of your real users. - **Related:** The weak spot of **Speech-to-Text**; the **LLM Core** can't see it. ### Streaming STT — transcribe as the user speaks > A stenographer who types continuously as you talk, not one who waits for the whole speech to finish — the text appears in step with the words. STT can be **streaming** — transcribing continuously as the user speaks, rather than waiting for them to finish and then processing the whole clip. This matters for speed: the system starts working sooner and reacts the moment the user stops, instead of adding a long pause. Transcribe-at-the-end versus transcribe-as-you-go is one of the first latency decisions in a voice system, and streaming is usually what makes it feel live. _Diagram: Batch waits for the end of the sentence and adds a pause; streaming transcribes as you go and reacts at once._ #### Streaming STT (streaming transcription) - **What it is:** Transcribing continuously while the user is still speaking. - **How it works:** Text builds up during the speech — by the end of the sentence it's almost ready. - **Example:** The user finishes — the system replies right away, not after a pause to transcribe. - **Catch:** Without it, every reply gets a noticeable pause in front. - **Related:** The twin of **Streaming TTS**; part of **Stream & Overlap**. > **The listening question:** "How accurate is my transcription on real users — their accents, their noise — and am I streaming it so the system reacts the moment they stop?" > STT is the ear: it turns speech into text for the model. Its errors flow downstream, so design for imperfect input — and transcribe as the user speaks. ## 3. Text to speech The last stage turns the model's text reply back into a voice. It's the part the user actually hears, so its quality shapes how the whole product feels. ### Text-to-Speech — text becomes a voice > A narrator reading a script aloud — turning written words into natural, expressive speech, not a recital. **Text-to-Speech (TTS)**, or **speech synthesis**, converts the model's text reply into audio. It's the mouth of the system. Modern TTS produces remarkably natural voices, with realistic intonation and rhythm, far beyond the flat robotic speech of old. A great pipeline with a robotic voice still feels cheap. A natural voice makes the whole interaction feel human — much of why voice products have become viable at all. _Diagram: TTS: the reply text goes in, a natural voice — intonation, rhythm, pace — comes out._ #### Text-to-Speech (TTS, speech synthesis) - **What it is:** The mouth of the system: turns the reply text into a voice. - **How it works:** Takes the model's text and synthesises speech with intonation and rhythm. - **Example:** The string "Table booked" becomes a warm, natural phrase. - **Catch:** A robotic voice makes even a great pipeline feel cheap. - **Related:** The last stage of the **Voice Pipeline**; **Voice Persona** sets its tone, **Streaming TTS** its speed. ### Voice Persona — the voice carries the experience > The same words said warmly or coldly land completely differently. The listener responds to the delivery, not just the content. In a voice product, _how_ something is said matters as much as _what_ is said. The chosen voice, its tone, its pacing — these are product decisions that shape trust and comfort, the way visual design does in a screen product. A voice that's too fast, too flat, or oddly inflected undermines an otherwise good answer. So TTS isn't an afterthought bolted on at the end. The voice (the **voice persona**) is chosen and tuned as deliberately as any other interface element. _Diagram: The same words: a warm delivery builds trust, a flat and fast one undermines a good answer._ #### Voice Persona (the product's voice and delivery) - **What it is:** The chosen voice, tone, and pace — the "face" of a voice product. - **How it works:** Delivery changes how the very same answer is received. - **Example:** "Your order is delayed" in a calm, warm voice — versus the same words, fast and flat. - **Catch:** A default, untuned voice can undercut trust in good answers. - **Related:** A setting of **Text-to-Speech**; part of **Conversation Flow**. ### Streaming TTS — speak while it writes > Someone who starts speaking their answer as they think of it, rather than silently composing the whole thing and delivering it in one block. Just as STT can stream in, TTS can **stream out** — speaking the start of the reply while the rest is still being generated. No need to wait for the full text and full audio before making a sound. The user hears a reply begin almost immediately instead of sitting in silence. Combined with the model streaming its text, this is how a voice agent answers without an awkward gap: the speech starts as soon as there's something to say. _Diagram: Batch stays silent until the whole reply is ready; streaming starts speaking while the model is still writing._ #### Streaming TTS (streaming synthesis) - **What it is:** Voicing the start of the reply while the rest is still generated. - **How it works:** The model emits text in chunks; TTS turns each chunk into audio right away. - **Example:** The first words of the reply sound almost instantly, not after a silence. - **Catch:** Without streaming, the user sits in silence while the whole reply is made. - **Related:** The twin of **Streaming STT**; part of **Stream & Overlap**. > **The speaking question:** "Does my voice sound natural enough to carry the experience — and does the reply start almost instantly instead of after a silence?" > TTS is the mouth: it voices the model's reply, and modern voices sound natural enough to carry the whole experience. Stream the speech so the reply begins at once, not after a silence. ## 4. The latency challenge Here is what makes voice genuinely hard. A conversation has a rhythm, and the whole three-stage pipeline has to fit inside it — or the magic breaks. ### Latency Budget — a conversation has a tight window > In real talk, a reply that comes a beat late feels awkward; one that comes after several seconds feels broken. The window for "natural" is small. People expect a conversational reply within **a fraction of a second**. A delay that would be invisible in a chat app is glaring in speech — a couple of seconds of silence after you finish talking feels like the system froze. This tight window — the **latency budget** — is the defining constraint of voice AI. The whole round trip — hear, transcribe, think, synthesise, speak — has to fit inside it. Miss it, and even a perfect answer feels broken. _Diagram: A fraction of a second feels natural, a beat late feels awkward, several seconds feels frozen. The whole round trip must fit in the first window._ #### Latency Budget (the response-time window) - **What it is:** The response-time window a live conversation tolerates — a fraction of a second. - **How it works:** Everything from "hear" to "speak" has to fit inside that window. - **Example:** A two-second pause nobody notices in chat sounds like "the system froze" in voice. - **Catch:** Miss it, and even a correct answer feels broken. - **Related:** Eaten by **Latency Stacking**; saved by **Stream & Overlap** and **Speech-to-Speech**. ### Latency Stacking — the delays add up > A relay race where the total time is every runner's leg combined — a slow handoff anywhere makes the whole team late. The cruel part is that latency **accumulates** across the pipeline. Time to transcribe + time for the model to think + time to synthesise — all stack into the pause the user feels. Each stage being "fast enough" alone isn't sufficient if their sum blows the budget. This is why voice is harder than text: a text chat waits only on the model, while voice waits on three stages in series — and the conversational window is far tighter. _Diagram: Chat waits on one stage; voice waits on three in series, and their sum overruns the budget._ #### Latency Stacking (accumulated latency) - **What it is:** The delays of the three stages add up into one total pause. - **How it works:** Total delay = STT + model + TTS, when run strictly in order. - **Example:** Every stage is "fast", yet together they leave an awkward silence after each sentence. - **Catch:** Optimising one stage isn't enough — the user feels the sum. - **Related:** Breaks the **Latency Budget**; fixed by **Stream & Overlap** or **Speech-to-Speech**. ### Stream & Overlap — run the stages in parallel > A simultaneous interpreter who translates as you speak and starts voicing the reply mid-thought — overlapping the stages instead of doing them strictly one after another. The key technique is to **stream and overlap** the stages instead of running them strictly in sequence. Transcribe while the user speaks. Let the model start generating before the question is fully processed. Start speaking the reply before it's fully written. Overlapping the three stages, instead of waiting for each to finish, is how a voice system collapses the total delay into something conversational. The whole art of low-latency voice is keeping the pipeline flowing continuously, never stopping and starting at each stage. _Diagram: The same work: one after another is late, overlapped the reply starts far sooner._ #### Stream & Overlap (streaming, overlapping stages) - **What it is:** All three stages stream and run at the same time, not in turn. - **How it works:** STT transcribes as you go, the model starts early, TTS speaks while the model writes. - **Example:** A voice agent replies with no gap, though inside it still has the same three stages. - **Catch:** A streaming system is harder to build than "finish a stage, pass it on". - **Related:** Combines **Streaming STT** and **Streaming TTS**; the main way to fit the **Latency Budget**. > **The latency question:** "Does my whole pipeline — transcribe, think, speak — fit inside the conversational window, and are the stages streaming and overlapping?" > A conversation has a tight latency budget, and the three stages' delays add up. Stream and overlap them — transcribe, think, and speak in parallel rather than in sequence. ## 5. Turn-taking and interruptions Beyond speed, natural conversation has a choreography of who speaks when. That's what separates a real-feeling voice agent from a clumsy one. ### Turn-taking — knowing when the user has finished > A good listener can tell a pause for breath from the end of a sentence — they don't jump in every time you take a moment. A voice system has to detect when the user has **actually finished speaking**, versus just paused mid-thought. This is **turn-taking**: knowing when it's the system's turn. Jump in too early and you cut the user off; wait too long and the conversation drags. The judgement is surprisingly hard and central to feeling natural. Humans do it effortlessly from subtle cues; a voice agent has to approximate it well enough that the back-and-forth flows instead of stumbling. _Diagram: A pause for breath — keep listening; the end of the sentence — our turn. Getting it wrong either way breaks the conversation._ #### Turn-taking (turn detection) - **What it is:** The call: is this pause a breath, or the end of the turn? - **How it works:** The system listens to the pauses and decides when it's its turn. - **Example:** "I'd like to book… [pause] …for Friday" — after the first pause it's too early to reply. - **Catch:** Too early cuts the user off; too late makes the conversation drag. - **Related:** Together with **Barge-in**, it makes up **Conversation Flow**. ### Barge-in — yield when interrupted > When you start talking over someone, they stop and listen — they don't keep plowing through their sentence as if you said nothing. Real conversation includes interruption. If the user starts speaking while the AI is talking — that's **barge-in** — a natural system stops, listens, and responds to the new input, instead of finishing its scripted reply over the top of them. Supporting interruption is a hallmark of a good voice agent; a poor one talks over you obliviously. Handling barge-in means the system is **always listening, even while it speaks**, ready to yield the floor the instant the user takes it. _Diagram: The user cuts in — the agent stops, listens, and responds instead of talking over them._ #### Barge-in (interruption handling) - **What it is:** The ability to stop when the user interrupts. - **How it works:** The system listens even while speaking; when it hears a voice, it stops and listens. - **Example:** The agent lists options, the user says "the second one!" — the agent switches at once. - **Catch:** Without barge-in the agent talks over people and seems deaf. - **Related:** Together with **Turn-taking**, it makes up **Conversation Flow**. ### Conversation Flow — the flow is the product > A good conversation partner and an awkward one say similar words — the difference is timing, listening, and giving way. The flow is the experience. Whether a voice agent feels natural or robotic comes down largely to this choreography: sensing turns, yielding to interruptions, not leaving **dead air**, and not talking over the user. These **conversation flow** details matter as much as the accuracy of the answer, because a correct response delivered with clumsy timing still feels broken. Designing a voice product means designing the _conversation_, not just the answers. The rhythm of who speaks when is a first-class part of the experience, not a polish step. _Diagram: The same words: smooth turns versus dead air and talking over the user._ #### Conversation Flow (the rhythm of the conversation) - **What it is:** The rhythm of the talk: who speaks when, pauses, giving way. - **How it works:** Built from turn-taking, barge-in, and no dead air. - **Example:** Two agents with identical answers: one feels like a person, the other like an answering machine. - **Catch:** A correct answer with clumsy timing still feels broken. - **Related:** Built from **Turn-taking** and **Barge-in**; depends on the **Latency Budget**. > **The flow question:** "Does my voice agent know when the user has finished, stop when interrupted, and keep a natural rhythm — or does it cut people off and talk over them?" > Natural voice needs turn-taking — knowing when the user finished — and barge-in — stopping when interrupted. The rhythm of who speaks when is as much the product as the answers themselves. ## 6. Realtime models The classic cascade is being joined by a newer approach: models that handle voice directly, end to end, collapsing the stages to cut latency. It's worth knowing where the field is heading. ### Speech-to-Speech — one model instead of three > Instead of a chain of "a translator writes it down, a thinker answers, a speaker reads it out" — one person who hears, thinks, and replies at once. A direct flight instead of two connections. A newer class of **realtime**, or **speech-to-speech**, models handles audio directly. Voice in, voice out, without separate transcribe → reason → synthesise steps. One model instead of three in a chain — the **end-to-end** approach. The big win is **latency**: no boundaries between stages means no delays passing data between them. The second win is that nothing is lost in conversion: the model hears _how_ something was said, not just the words, keeps tone and emotion, and can reply with matching expression. _Diagram: The cascade loses tone at its text hand-offs; speech-to-speech is one model, voice in and voice out._ #### Speech-to-Speech (realtime model, end-to-end) - **What it is:** One model that takes voice in and answers directly in voice. - **How it works:** No separate STT and TTS — so no hand-offs between stages and no hand-off delays. - **Example:** The model hears irritation in the user's voice and answers more calmly — a text cascade can't see that. - **Catch:** The middle is a black box: less visibility and control. - **Related:** The alternative to the **Voice Pipeline**; the most direct way into the **Latency Budget**. ### Pipeline vs End-to-End — a real choice > You can travel by a series of trains and control each leg, or take one direct service that's faster but gives you less say over the route. Different trade-offs, both valid. The classic **pipeline** and the **end-to-end** model are both real options, each with a price. The pipeline gives you control and flexibility at every stage — swap the STT, inspect the transcript, use any model — at the cost of latency and complexity. End-to-end gives speed and naturalness but less visibility and control over the middle. Knowing both exist lets you choose deliberately: the transparent, flexible pipeline or the fast, fluid end-to-end approach, depending on what your product needs most. _Diagram: Pipeline: control and flexibility, but latency. End-to-end: speed and naturalness, but less control._ #### Pipeline vs End-to-End (cascade or realtime) - **What it is:** The architecture choice for a voice product. - **Pipeline:** Swap STT/TTS, read the transcript, use any model. Cost: latency and more moving parts. - **End-to-end:** Lowest latency, tone and emotion preserved. Cost: less visibility and control. - **When to use which:** Need control and inspectability — pipeline; need speed and fluidity — end-to-end. - **Related:** Compares the **Voice Pipeline** and **Speech-to-Speech**. > **The architecture question:** "Does my product need the control and flexibility of a three-stage pipeline, or the lower latency and naturalness of an end-to-end realtime model?" > Realtime models take voice in and out directly: fewer stages, less latency, and tone survives. Pipeline versus end-to-end is a real trade-off: control and flexibility, or speed and naturalness. ## 7. Using it well Using voice well means choosing it for the right situations and respecting that latency, more than anything, decides whether it feels magical or broken. ### Modality Fit — voice where it belongs > You'd rather speak directions while driving and type a long form at your desk — the right input depends entirely on the situation. Voice is powerful where it fits: **hands busy**, **accessibility**, natural conversation, situations where talking is faster or safer than typing. It's a poor fit where text is better: **precise input**, **privacy in public**, **careful review**. The skill is matching the **modality** to the moment, not adding voice because it's impressive. Voice where text would serve better is a downgrade; voice where speaking is the natural act is transformative. Choose it for the situation, not the novelty. _Diagram: Voice when hands are busy and a conversation helps; text when you need precision, privacy, and review._ #### Modality Fit (when voice belongs) - **What it is:** Choosing voice or text for the specific situation. - **Voice fits:** Hands busy, driving, accessibility, natural conversation. - **Text fits:** Precise input, privacy in public, careful review. - **Catch:** Voice "because it's cool" where text is better is a downgrade. - **Related:** The first question before anything else; without it, **Latency-First** is moot. ### Latency-First Design — budget for latency from day one > A chef who plans the whole meal around the dish that takes longest — designing around the binding constraint, not discovering it at the end. Because latency decides everything, design for it **from the start (latency-first)**: streaming at every stage, an end-to-end model where speed is critical, and a plan for when a stage is slow. Voice features that ignore latency until late tend to feel sluggish and broken no matter how good the answers are. Treat response time as the central design constraint of a voice product — the one number that, missed, ruins everything else. _Diagram: Budget first, then streaming, end-to-end where it matters, and a plan for a slow stage._ #### Latency-First Design (designing around latency) - **What it is:** Response time is the main constraint everything is built around. - **How it works:** Set the budget → stream everywhere → end-to-end where critical → plan for a slow stage. - **Example:** Time the full round trip in the first prototype and reject decisions that break it. - **Catch:** Latency discovered at the end is hard to fix. - **Related:** Protects the **Latency Budget**; relies on **Stream & Overlap** and **Speech-to-Speech**. ### How it all connects > The whole subway map at once: now you can see where the lines cross. The cascade causes **latency stacking**, which hits the **budget**. **Streaming and overlap**, or **speech-to-speech**, rescue it. Fit the budget and build a **conversation flow** from turn-taking and barge-in, and the conversation feels alive. And **modality fit** decides whether you need voice at all. _Diagram: Latency at the centre: the cascade piles it up, streaming and speech-to-speech cut it; together with conversation flow, that makes it feel natural._ #### Before you ship a voice feature - **Does voice genuinely fit** the situation — or would text serve the user better? - **Is each stage streaming** — transcribe, think, and speak overlapping, not in strict sequence? - **Does the total latency** feel conversational, or is there an awkward gap? - **Does it handle turn-taking** — knowing when the user finished — and **barge-in**? - **Is the voice natural** enough to carry the experience? - **Pipeline or end-to-end** — have I chosen the architecture for my latency and control needs? #### Signs you build voice well - You choose voice for situations where **speaking genuinely fits**, not for novelty. - Every stage **streams and overlaps**, so the reply starts almost instantly. - The total **latency feels conversational**, with no awkward silence. - The agent handles **turn-taking and barge-in**, so the flow feels natural. - You chose **pipeline or end-to-end** deliberately, designing around the **latency budget**. ## Cheat sheet: all 18 concepts ### Pipeline - **Voice Pipeline** (cascade): STT → model → TTS: ear, brain, mouth in sequence. - **LLM Core** (the reasoning core): The same text model in the middle; only the edges are new. ### Listen - **Speech-to-Text** (STT, ASR): The ear: turns speech into text. - **Error Propagation** (errors flow downstream): A transcription error becomes the model's input. - **Streaming STT** (transcribe as you go): Text is ready by the end of the sentence — no pause before reacting. ### Speak - **Text-to-Speech** (TTS, synthesis): The mouth: turns the reply text into a voice. - **Voice Persona** (voice and delivery): Tone and pace are part of the product, not a detail. - **Streaming TTS** (streaming synthesis): Starts speaking while the reply is still being written. ### Latency - **Latency Budget** (response-time window): Reply within a fraction of a second — or it feels frozen. - **Latency Stacking** (accumulated latency): The three stages' delays add up into one. - **Stream & Overlap** (stages in parallel): All stages streaming at once — the main way to make it in time. ### Turns - **Turn-taking** (turn detection): Tell a pause for breath from the end of a sentence. - **Barge-in** (interruption handling): Interrupted — stop, listen, respond. - **Conversation Flow** (rhythm of the talk): Who speaks when is as much the product as the answers. ### Realtime - **Speech-to-Speech** (realtime, end-to-end): One model: voice in, voice out. Less latency, tone survives. - **Pipeline vs End-to-End** (architecture choice): Control and flexibility versus speed and naturalness. ### Using it well - **Modality Fit** (when voice belongs): Voice where speaking is natural; text where precision matters. - **Latency-First Design** (latency from day one): Build everything around the response-time budget. > Voice means listen, think, and speak fast enough to feel like a conversation. Stream and overlap the pipeline, handle turn-taking and interruptions, choose pipeline or end-to-end deliberately — and treat latency as the constraint everything else bends to. End of express course · 7 chapters · 18 concepts Next comes practice: build the simplest voice loop — speech-to-text, a model reply, text-to-speech — and time the whole round trip. Then make each stage stream and watch the felt delay collapse. Finally, try interrupting it mid-reply and see whether it stops and listens. The challenge becomes real the moment you feel how a one-second gap breaks the illusion of conversation. But hold one idea above the rest: voice is the same thinking model with an ear and a mouth around it, and the entire difficulty is doing all three fast enough to feel alive. Stream everything, mind the turns, and guard the latency budget — that's what makes talking to AI feel like talking, not waiting.