# Models that see and hear, not just read Express course on multimodal AI — models that take images, audio, and video, not only text. 20 concepts — modality, mixed input, vision encoder, OCR, scene understanding, structured output, text-to-image, visual prompt injection, image token cost, and more — each with its English name, a diagram, an example, and a card. The point: the same engineering you learned for text, plus new powers and new traps. A cheat sheet of every concept at the end. - **Author:** Fedor Chishchin - **Level:** intermediate - **Reading time:** ~30 minute read - **Published:** 2026-06-13 - **Source:** https://fedorthinks.com/learn/multimodal-ai-express --- ## 1. Beyond text The first thing to grasp is simply what changed: models stopped being limited to reading and writing words. Once a model can take in images and sound, a whole new range of problems opens up. ### The map: 20 concepts in seven groups > A subway map: first you see the whole network, then you ride it station by station. First, three **basics**: what a modality is, mixed input, and why the engineering stays the same. Then six groups, one per chapter: how a model **sees**, what it **unlocks**, building it **into a product**, **generating** other modalities, the new **pitfalls**, and **using it well**. Every concept that follows has the same shape: its **English name** (remember it — it's the term), a picture from life, a **diagram**, and a **card** with fields like _what it is_, _idea_, _example_, _catch_, and _related_. _Diagram: Basics on top; below, the six groups that chapters 2–7 follow._ ### Modality / Multimodal — a kind of data, and a model that handles several > A person who can read, but also see, listen, and speak — each sense is a different channel for taking in the world, not a different brain. A **modality** is a type of data — text, images, audio, video. A model that handles more than one is **multimodal**: it can take an image _and_ text together, or produce speech from a description. For years, models were text-only — read words, write words. The shift is that a single model can now perceive across modalities, the way a person uses several senses, rather than being deaf and blind to everything but text. _Diagram: Text, image, audio, video — different modalities flow into one multimodal model._ #### Multimodal Model (modality) - **What it is:** A model that works with several data types: text, images, audio, video. - **Modality:** One type of data — one "channel," like one human sense. - **Example:** Describe a photo, read a chart, transcribe a meeting, answer a question about a screenshot. - **Related:** Enables **mixed input**; everything else in this course follows from this one capability. ### Mixed Input — a picture and words in one question > Handing someone a photo and asking "what's wrong with this?" — they look and answer in words. One question, two kinds of input, one response. The power of multimodal isn't handling images _instead_ of text — it's handling them _together_. This is **mixed input**: show the model a screenshot and ask a question about it, give it a chart and a written request, send a photo with instructions. The model reasons over all of it at once. This combination is what makes multimodal genuinely new: input is no longer one channel, so your prompts can mix words and pictures the way a real conversation does. _Diagram: A screenshot and a question go in together — the model reasons over both and answers in words._ #### Mixed Input (interleaved input) - **What it is:** An image (or audio) and text in the same request. - **Idea:** The model reasons over every part of the input at once, not in turns. - **Example:** An error screenshot + "why?"; a chart + "what's the trend?". - **Related:** In a product this is a **multimodal request** — ordinary context assembly with a picture inside. ### Same Engineering — the text skills still apply > Learning to see doesn't make you forget how to read — the new sense adds to what you could already do, it doesn't replace it. Everything you learned for text — **prompting**, **context**, **structured output**, **grounding**, **evals** — still applies to multimodal. The image or audio is just another part of the input you assemble into the context; the model still produces an output you constrain, validate, and measure. So multimodal isn't a separate discipline to relearn from scratch; it's the same engineering with an extra kind of input and output. Carry your text skills over and add the new modality's specifics on top. _Diagram: Text and image go through the same disciplines: prompting, context, schema, grounding, evals._ #### Same Engineering (skills carry over) - **Idea:** An image is one more input; all the text engineering applies to it. - **Example:** The same JSON schema, the same validation, the same evals — only now a photo goes in. - **Catch:** Thinking "this is something else entirely" and dropping proven practice. - **Related:** On top come the **modality edges**: injection, cost, perception limits. > **The framing question:** "Does this task involve more than words — an image, a chart, audio — that the model could perceive directly, instead of me forcing everything into text first?" > A modality is a kind of data; a multimodal model handles several — images and audio, not just text — and can mix them in one input. All your text skills still apply, with a new sense added. ## 2. How a model sees It helps to know, roughly, how a model takes in an image — because it demystifies what multimodal can and can't do, and why it sometimes misreads what's right in front of it. ### Shared Representation — an image becomes the same kind of tokens > A translator who turns both spoken words and sign language into the same written notes — different inputs, converted to one common form the brain works with. Under the hood, a multimodal model converts an image into the same kind of internal representation it uses for text. A **vision encoder** turns pixels into a sequence of **image tokens**, while text is split into text tokens as usual. Both sequences land in a **shared representation** — which is exactly why the model can answer a text question about a picture: both inputs have been translated into one common language inside the model. You don't need the details; knowing that image and text become comparable is enough. _Diagram: Pixels via a vision encoder, words via a tokenizer — and it's all one sequence for the model._ #### Shared Representation (vision encoder, image tokens) - **What it is:** Image and text are translated into one internal form — a sequence. - **How it works:** Vision encoder: pixels → image tokens; the model then sees them next to the words. - **Why it matters:** It's how a question in words gets connected to what's in the picture. - **Related:** Image tokens cost money — see **image token cost**. ### OCR vs Scene Understanding — reading the text ≠ understanding the scene > You can read the label on a jar and separately understand what the jar is for — two different acts, even though both involve looking at the same object. Two distinct capabilities get blurred. Reading the **text within an image** — a sign, a document, a screenshot — is roughly what used to need a separate **OCR (optical character recognition)** tool; a multimodal model can do it directly. **Scene understanding** — what's happening, what objects are present, what a chart means — is different. A good multimodal model does both, but it's worth distinguishing "extract the words from this image" from "interpret what this image shows," because they're different asks with different reliability. _Diagram: One image, two asks: OCR pulls out the exact text, scene understanding the meaning._ #### OCR vs Scene Understanding (reading text vs interpreting) - **OCR:** Extract the words and numbers written in the image. - **Scene:** Interpret it: what's shown, what's happening, what the chart means. - **Example:** "What does the label say?" — OCR. "What is this object and what's it for?" — scene. - **Catch:** Different asks, different reliability — be clear which one you're making. - **Related:** Both run into **perception limits**. ### Perception Limits — seeing has limits > Even sharp eyes miss fine print, misjudge a blurry photo, or misread a cluttered diagram — seeing is powerful but not flawless. A model's vision is impressive but imperfect — these are its **perception limits**. It can misread small or low-quality text, miss details in a busy image, miscount objects, or confidently misdescribe something subtle. The perception is genuinely useful but not a precise instrument — treat what the model "sees" as a strong interpretation, not a guaranteed reading. It's tempting to assume that because the model _can_ see, it sees correctly; like its text answers, its visual ones can be confidently wrong. _Diagram: Small print, blur, clutter, many objects — and sometimes a confident mistake._ #### Perception Limits (limits of model vision) - **What it is:** The model's vision is a strong interpretation, not a guaranteed reading. - **Where it breaks:** Small or poor text, cluttered images, counting objects, subtle details. - **Catch:** "Can see" doesn't mean "sees correctly." - **Related:** The root of **confident misreading**; handled by **validation & grounding**. > **The perception question:** "Am I asking the model to read text in an image or to interpret a scene — and how reliable does the perception need to be for this task to be safe?" > A model converts an image into the same internal form as text, so it can reason over both together. Reading text in an image differs from understanding the scene — and the perception, though powerful, can be confidently wrong. ## 3. What it unlocks The reason multimodal matters is the range of real problems it makes solvable. Seeing the concrete uses shows why it's more than a novelty. ### Document Understanding — documents, forms, screenshots > An assistant who can glance at a paper form or a screen and pull out exactly what you need, instead of you typing it all in by hand. A huge practical use is **document understanding**: reading documents, forms, and screenshots — extracting data from an invoice, understanding a PDF's layout, answering questions about what's on a screen. Before multimodal, this needed brittle, specialised tools; now a model can look at the document and work with it directly, structure and all. Anywhere information lives in a visual format rather than clean text, multimodal turns "a human has to read and retype this" into something a model can handle. _Diagram: Invoice, form, screenshot — the model reads them, layout included, and hands back data._ #### Document Understanding (reading documents) - **Problem:** Information is locked in invoices, forms, and screenshots — a human reads and retypes it. - **Idea:** The model looks at the document and pulls out what's needed, layout and all. - **Example:** Pull the total, date, and vendor out of a PDF invoice. - **Before:** Brittle, specialised tools for every format. - **Related:** Ask for the result as **structured output** and check it with **validation**. ### Visual Analysis — images and charts > Showing an expert a graph and asking what it means — they read the visual and explain the trend, no spreadsheet required. Multimodal models can do **visual analysis**: describe a photo, read and interpret a chart or diagram, spot what's in an image, compare two pictures. You can hand the model a sales chart and ask for the trend, a product photo and ask what's wrong, a diagram and ask it to explain. This turns images from things only humans could interpret into inputs your software can reason about — opening up any workflow where the meaningful information is visual. _Diagram: Chart → trend, product photo → defect, diagram → a plain explanation._ #### Visual Analysis (analysing images and charts) - **What it is:** Describe, explain, and compare what's shown. - **Example:** Sales chart → "what's the trend?"; product photo → "what's wrong?"; diagram → "explain." - **Catch:** Interpretation is **scene understanding** — less precise than OCR. - **Related:** Opens up workflows where the meaning lives in a picture. ### Accessibility & Reach — who and what you can serve > A guide who describes the scene aloud for someone who can't see it — turning a visual world into words anyone can use. Multimodal also widens who and what your product can serve — **accessibility** and **reach**: describing images for blind and visually-impaired users, letting people point a camera instead of typing, working with content that was never in text form. The same capability that reads a chart can narrate a photo or caption a video. Beyond any single feature, multimodal expands the _surface_ of what an AI product can take as input — from "type your question" to "show me, tell me, or play me what you mean." _Diagram: Beyond “type” come “show,” “tell,” and “play” — and images described for people who can't see them._ #### Accessibility & Reach (a wider input surface) - **What it is:** A wider input surface: show, tell, play — not only type. - **Example:** Photo descriptions for low-vision users; pointing a camera instead of typing; video captions. - **Why it matters:** The product serves people and content it couldn't reach before. - **Related:** The same capability as **visual analysis**, pointed at people. > **The unlock question:** "Is there information in my problem that's trapped in a visual or audio form — a document, a chart, a photo — that multimodal could let the model use directly?" > Multimodal unlocks reading documents and screenshots, analysing images and charts, and widening reach through accessibility — turning visual and audio information into something software can reason about. ## 4. Multimodal in your product Building with multimodal is mostly the engineering you already know, with images or audio added to the input. The familiar disciplines carry straight over. ### Multimodal Request — send the image alongside the text > Including a photo with your written question in the same message — the recipient sees both and answers the whole thing at once. In practice, you build a **multimodal request** much like a text one: you assemble the context, but now it can include an image (or audio) along with your text instructions. "Here's a screenshot, and here's what I want to know about it" goes to the model as one combined input. This is just **context engineering** with a richer input — the image is one more thing you put in the window, deliberately, alongside the words. The assembly mindset you already have applies directly. _Diagram: Instructions, an image, and a question in the context window — the model gets them as one input._ #### Multimodal Request (context engineering with an image) - **Idea:** Assemble the context as usual, but put an image or audio in it. - **Example:** Instructions + screenshot + "what's wrong here?" — in one request. - **Catch:** Throwing in an image "just in case" is both noise in the context and extra cost. - **Related:** The implementation of **mixed input**; priced by **image token cost**. ### Structured Output — chaos in, clean data out > A form-filler who looks at a messy receipt and writes the total, date, and vendor into neat labelled boxes — chaos in, clean data out. When you use multimodal to extract information — pull fields from a document, classify an image, read a chart — you still want **structured output**: ask the model to return clean, schema-shaped data your code can use, not prose. A multimodal model reading a receipt should hand you `{total, date, vendor}`, not a paragraph. The same structured-output discipline that turns a text model into a reliable component does the same for a vision model. The modality is new; the bridge to your code isn't. _Diagram: Receipt photo → schema-shaped data your code takes. A paragraph of prose your code can't use._ #### Structured Output (schema-shaped data) - **Problem:** The model describes the picture in prose — your code can't do anything with it. - **Idea:** Ask for data in a schema: fields, types, shape. - **Example:** Receipt → `{total, date, vendor}`, not a paragraph. - **When to use:** Any time you extract data from an image. - **Related:** Next step: **validation & grounding** of those fields. ### Validation & Grounding — check it, as always > You double-check what someone reports back from a quick glance, especially the important details — a second look on what matters. Because the model's perception can be wrong, you apply the same reliability discipline: **validate** the extracted data, **ground** answers in what's actually verifiable, and keep a human on high-stakes reads. A model misreading a number off an invoice is the visual version of a **hallucination**, so you treat its visual output as untrusted-until-checked, exactly as you would its text. The lesson from every other course holds: the model is a fallible component, and a new modality doesn't change that. _Diagram: Model's read → schema and rule checks → checked against the source → a human on what matters → use._ #### Validation & Grounding (verify before you trust) - **Problem:** The model can misread a number — and sound sure about it. - **Idea:** Output is untrusted until checked: schema, rules, source, human. - **Example:** The invoice total doesn't match the sum of the lines → retry or route to a human. - **Related:** The answer to **perception limits** and **confident misreading**. > **The product question:** "Am I treating the image as just another input to engineer — assembled into context, returned as structured output, validated — the same way I'd treat text?" > Building multimodal is context engineering with a richer input: send the image alongside the text, still ask for structured output, and validate and ground the result — the model is a fallible component, modality aside. ## 5. Generating across modalities So far we've talked about models taking in images and audio. The other direction — models producing them — is its own large area, worth naming so you know where it fits. ### Generation — text-to-image, text-to-speech, and more > An artist who paints what you describe, or a voice actor who speaks your script — generation is the mirror image of perception. Just as models can take in images and sound, other models **generate** them: **text-to-image** models paint a picture from a description, **text-to-speech (TTS)** turns words into a spoken voice, and there are models for music, video, and more. Generation is the flip side of understanding — another modality as output rather than input. It's a vast field of its own, but the key point is that "multimodal" spans both directions: a model can perceive other modalities, and a model can produce them. _Diagram: Top: perception, an image or sound goes in. Bottom: generation, an image, speech, music, or video comes out._ #### Generation (text-to-image, text-to-speech (TTS)) - **What it is:** A model produces another modality: an image, speech, music, video. - **Example:** An illustration from a description; speech for a voice product. - **Idea:** The mirror of perception: the modality is the output, not the input. - **Related:** Runs on the **brief → draft → review** loop; reach for it **deliberately**. ### Brief → Draft → Review — the same engineering posture > You direct an artist with a clear brief and then review the result before using it — the same loop, whatever they're producing. Building with generative modalities follows the same posture as everything else, the **brief → draft → review** loop: a clear instruction (the prompt), an output you treat as a draft to review, and a human in control of using it. A generated image or voice is the model's confident attempt, to be checked and edited, not blindly shipped — the product-design discipline applies whether the output is text or a picture. So you don't need an entirely new playbook; the same "fallible component, you stay in control" stance carries over. _Diagram: Brief → generate → draft → human review → use. Not right yet — edit the brief and go again._ #### Brief → Draft → Review (the generation loop) - **Idea:** What's generated is a draft; a human reviews it before it's used. - **Example:** Generate an illustration, look at it, refine the prompt, generate again. - **Catch:** Shipping an image or a voice "as is," unchecked. - **Related:** The same posture as **same engineering** for text. ### Deliberate Generation — know it exists, reach for it on purpose > You don't commission a custom painting for a job that needs a sentence — you use the medium the task actually calls for. Generative modalities are powerful for the right job — illustrations, synthesised speech for a voice product, video — but they're a **deliberate choice**, not a default to sprinkle everywhere. Reach for image or audio generation when the output genuinely needs to be in that modality, and stick to text when text does the job. Knowing this whole capability exists, and where it fits, is enough for now — the point is that multimodal is a two-way street, perceiving and producing across senses. _Diagram: Does the output really need to be an image or a voice? Yes — generate. No — stay with text._ #### Deliberate Generation (generate only when needed) - **Question:** Does the output really need another modality? - **Yes:** Illustrations, speech for a voice product, video. - **No:** Text is cleaner, cheaper, and faster — stay with it. - **Related:** A special case of **channel fit** — for output instead of input. > **The generation question:** "Does this task genuinely need output in another modality — an image, a voice — or am I reaching for generation when text would do the job?" > Multimodal runs both ways: models also generate images, speech, and more. The same engineering posture applies — a clear brief, a reviewed draft, a human in control — and you reach for it only when the output truly needs that modality. ## 6. The new pitfalls Multimodal adds new failure modes on top of the familiar ones. A few specific traps catch teams who treat images as if they were as safe and cheap as text. ### Visual Prompt Injection — an attack hidden in a picture > A photograph with instructions written in it that the eye barely notices but the machine reads perfectly — a message smuggled past you in plain sight. A multimodal model reads _everything_ in an image, including text a human might miss. That makes images a channel for **prompt injection**: an attacker can hide instructions in a picture — faint text, embedded in the pixels — that the model dutifully follows. This is **visual prompt injection**, the visual version of the injection problem from the security course, and it's nastier because you can't easily see the attack. Treat any image the model ingests as untrusted input that might be carrying instructions, not just innocent pixels. _Diagram: Faint text in a photo slips past a human, but the model reads and follows it. The defense: treat the image as untrusted._ #### Visual Prompt Injection (injection via images) - **What it is:** Instructions hidden in an image that the model follows. - **Example:** Faint "ignore previous instructions" text on a product photo. - **Why nastier:** The attack is hard to see — a human doesn't notice it. - **Defense:** Every image is untrusted input, not "just pixels." - **Related:** The visual form of **prompt injection** from the security course. ### Image Token Cost — a picture costs more than a thousand words > Sending a photograph instead of a sentence — far more to transmit and process, and the bill reflects it. An image is worth far more than a thousand words to the meter — this is **image token cost**. Processing one consumes many more tokens than a short text prompt, so multimodal calls can be substantially more expensive. A feature that sends high-resolution images on every request can run up a surprising bill. The model-economics discipline applies with extra force here — be deliberate about image size and how often you send one, because the cost of "just include the picture" is much higher than including a line of text. _Diagram: A short prompt, one image, a high-res image, the same image on every call — the bill grows._ #### Image Token Cost (images are expensive input) - **What it is:** One image consumes many more tokens than a short text prompt. - **Example:** A high-res screenshot on every request — a surprising bill. - **How to cut it:** Lower the resolution; send the image only when it's needed. - **Related:** Follows from **shared representation**: an image = many tokens. ### Confident Misreading — the quiet failure > Someone who glances at a blurry sign and confidently tells you the wrong number — fast, sure, and wrong. The familiar hallucination problem takes a visual form — **confident misreading**: a model can confidently misread a figure, miscount items, or describe something that isn't in the image — and sound just as sure as when it's right. Because the input is visual, these errors can be harder to catch than a textual slip. So for anything where a misread matters — a number off a document, a medical image, a safety check — you verify, keep a human in the loop, and never assume "it can see it" means "it read it right." The model's confidence is no more reliable about images than about text. _Diagram: The sign says “8,” the model confidently says “3.” The defense: verify, a human in the loop, no assumptions._ #### Confident Misreading (visual hallucination) - **What it is:** The model misreads or misdescribes an image — and sounds sure. - **Example:** The wrong figure off an invoice, the wrong count, a detail that isn't there. - **Why quiet:** A visual error is harder to catch than a textual slip. - **Defense:** Verify, a human on high-stakes reads, no "but it can see it." - **Related:** A consequence of **perception limits**; handled by **validation & grounding**. > **The pitfall question:** "Could this image carry a hidden injection, am I accounting for its higher cost, and have I assumed the model reads it correctly when it might be confidently wrong?" > Multimodal's new traps: images can smuggle prompt injections, they cost far more tokens than text, and the model can misread them as confidently as it hallucinates in text. Treat images as untrusted, pricey, and fallibly read. ## 7. Using it well Using multimodal well is mostly applying everything you already know to a richer input, while respecting the new modality's specific costs and risks. ### Channel Fit — match the channel to the task > You bring in the camera when the problem is something to look at, and stick to typing when it's something to say — matching the channel to the task. The decision is simple — **channel fit**: use multimodal when the meaningful information genuinely lives in an image or audio — a document to read, a scene to interpret, speech to understand — rather than forcing it into text first or avoiding it when it would help. But don't reach for it reflexively where text is cleaner, faster, and cheaper. The skill is recognising when a problem is actually visual or auditory, and letting the model perceive it directly, instead of either ignoring that capability or overusing it. _Diagram: Where does the meaning live? In an image or audio — multimodal. In words — text._ #### Channel Fit (the right channel for the task) - **Question:** Where does the meaning live — in an image, in sound, or in words? - **Example:** A scanned invoice — multimodal. A question the user types anyway — text. - **Catch:** Two mistakes: ignoring the capability and overusing it. - **Related:** The same idea for output is **deliberate generation**. ### Modality Edges — every discipline, plus the modality's own > A pilot rated for a new aircraft brings all their existing skills and adds the specifics of the new plane — not starting over, but extending. Multimodal isn't a reason to forget what you learned. Assemble the image into context deliberately, ask for structured output, validate and ground the result, watch the cost, and treat images as untrusted. Then add the **modality edges** — what's specific to this modality: visual injection, higher token cost, perception limits. Everything from the text courses still governs; multimodal just extends it with a new kind of input and its own handful of traps. Build on what you know, and learn only the new edges. _Diagram: The foundation is the text disciplines. On top, three new edges: injection, cost, perception limits._ #### Modality Edges (new edges on old disciplines) - **Idea:** The old disciplines + the modality's specifics, not everything from scratch. - **Old:** Context, structured output, validation, grounding, cost. - **New:** Visual injection, image token cost, perception limits. - **Related:** Ties **same engineering** together with every **pitfall** of chapter 6. ### How the concepts connect > An experienced engineer gaining a new sense: everything they could do stays, and on top come new powers — and new ways to be wrong. A **multimodal model** **perceives** images and sound (through a **shared representation**) and **produces** them (**generation**). Perception unlocks **uses** — documents, charts, accessibility — but has **limits** and brings **pitfalls**: injection, cost, confident misreads. Both are kept in check by the **same engineering**: multimodal request, structured output, validation & grounding. _Diagram: The model perceives and produces; the uses are built with the same engineering, and the limits and pitfalls are guarded by it._ #### Before you ship a multimodal feature - **Is the information genuinely visual or audio** — or am I using multimodal where text is cleaner? - **Am I assembling the image into context** deliberately, like any other input? - **Am I asking for structured output** when extracting data from an image? - **Could the image carry a hidden injection** — am I treating it as untrusted? - **Have I accounted for the much higher token cost** of images? - **Am I validating the perception** and keeping a human on high-stakes reads? #### Signs you use multimodal well - You reach for it when the information is truly **visual or audio**, not by reflex. - You **assemble the image into context** and ask for **structured output** when extracting data. - You treat images as **untrusted**, alert to **visual injection**. - You account for the **higher cost** and the **perception limits**. - You **validate and keep a human** on reads where a misread would matter. ## Cheat sheet: every concept in the course ### Basics - **Modality / Multimodal Model** (kind of data, model for several): A modality is a type of data; a multimodal model handles several. - **Mixed Input** (interleaved input): Image and text in one request — the model reasons over both. - **Same Engineering** (skills carry over): All the text engineering still applies to images. ### How a model sees - **Shared Representation** (vision encoder, image tokens): Pixels become tokens in one space with the words. - **OCR vs Scene Understanding** (text vs scene): Reading the words in a picture isn't the same as understanding it. - **Perception Limits** (limits of model vision): Vision is a strong interpretation, not a guarantee. ### What it unlocks - **Document Understanding** (reading documents): Invoices, forms, screenshots — no retyping. - **Visual Analysis** (images and charts): A trend from a chart, a defect in a photo, a diagram explained. - **Accessibility & Reach** (wider input surface): Show, tell, play — not only type. ### In your product - **Multimodal Request** (context engineering with an image): The image is one more part of the assembled context. - **Structured Output** (schema-shaped data): Data in a schema, not prose. - **Validation & Grounding** (verify before you trust): Visual output is untrusted until checked. ### Generation - **Generation** (text-to-image, TTS): A model produces an image, speech, music, video. - **Brief → Draft → Review** (the generation loop): What's generated is a draft; a human decides. - **Deliberate Generation** (generate only when needed): Generate only when the output truly needs that modality. ### Pitfalls - **Visual Prompt Injection** (injection via images): Instructions hidden in a picture. - **Image Token Cost** (images are expensive): An image costs many more tokens than text. - **Confident Misreading** (visual hallucination): A wrong read, delivered with confidence. ### Using it well - **Channel Fit** (the right channel): Multimodal when the meaning lives in an image or sound. - **Modality Edges** (the new edges): The old disciplines plus injection, cost, and perception limits. > Multimodal extends your text engineering to images and audio: reach for it when the information is visual, carry over every discipline — context, structured output, validation, cost — and respect the new traps of injection, cost, and confident misreading. End of express course · 7 chapters · 20 concepts Next comes practice: take a task where information is stuck in a visual form — a screenshot, a receipt, a chart — and hand the image to a multimodal model, asking for structured output you can use. Then test its limits: blur the image, add small print, and watch where it misreads. Finally, treat one image as hostile and see whether hidden text could steer it. The capability clicks the moment the model reads a document you'd have had to retype. But hold one idea above the rest: multimodal lets a model perceive more than words, but it's the same fallible component underneath — carry over everything you know, and respect that seeing, like reading, can be confidently wrong.