Express course · No. 05
Two programs can only cooperate if they agree on the rules — what to send, in what shape, and who speaks when. The protocol is that agreement, and picking the right one is choosing how the conversation goes.
Essence only · One picture per protocol · Examples over RFCs
A protocol is the agreed set of rules two programs use to talk — what a message looks like, and who may say what, when. Get the rules right and wildly different systems understand each other.
A protocol is a language plus etiquette
Two diplomats who share no native tongue still negotiate — because they agree in advance on one language and on the order of speaking. Break either and it's just noise.
For two programs, the protocol fixes both: the format of the message (so each side can read it) and the choreography (who speaks first, when a reply comes, how it ends). HTTP, gRPC, WebSocket — each is just a different agreement about those two things. Same job, different manners.
Protocols come in layers
A letter: the postal system moves the envelope and doesn't care what's inside; the language on the page is between you and the reader. Two separate agreements, stacked.
Underneath sits a transport protocol (TCP or UDP) that just moves bytes between machines. On top sits an application protocol (HTTP, gRPC, MQTT) that gives those bytes meaning. You usually work at the top — but when things get slow or flaky, the bottom layer leaks through, so it pays to know it's there.
Every protocol answers three questions
A conversation needs three things settled: who starts, what language, and whether you take turns or both talk at once.
Who initiates — does the client ask, or can the server push? What shape the data takes — JSON text, compact binary, a continuous stream? The timing — one shot, or a long-lived line where either side can speak? Almost every difference between protocols is a different answer to those three.
There is no best protocol, only a fit
You wouldn't send a wedding invitation by walkie-talkie, or coordinate a rescue by post.
A protocol is tuned to a shape of conversation: rare and reliable, or constant and fast; one-to-one, or one-to-many; human-readable, or as small as possible. The skill isn't knowing the newest protocol — it's matching the protocol to how the two sides actually need to talk.
A protocol is an agreement. Pick it by the shape of the conversation, not by what's fashionable.
Under almost everything you build sit a few protocols you rarely touch directly but always depend on. Know them — they decide what's even possible above.
TCP: a reliable line that guarantees delivery
A phone call where the other person says "got it" after every sentence — slower, but nothing is lost or scrambled.
TCP sets up a connection and guarantees every byte arrives, in order, retrying what gets lost. It's the workhorse under the web, email, and most APIs. You pay a little in setup and latency for the promise that the data is complete and correct — usually a great trade.
UDP: fire and forget, fast and lossy
Shouting across a noisy room — quick, but if a word is missed you don't stop to repeat it, you just keep going.
UDP sends packets with no guarantee of arrival or order. That sounds bad until you want speed over perfection: video calls, live games, DNS lookups — where a dropped frame beats a stutter while you wait for a re-send. (HTTP/3, below, is built on it.)
HTTP: the request-response language of the web
A shop counter: you ask for one thing, you get one answer, and the clerk forgets you the moment you walk away.
HTTP rides on TCP and defines the familiar request/response with verbs (GET, POST, PUT, DELETE) and status codes (200, 404, 500). It's stateless — each request stands alone — which is exactly what lets the web scale to billions of calls. Nearly every API you'll touch speaks it.
HTTPS and TLS: the same, but sealed
The same letter, now inside a tamper-proof envelope with the sender's verified signature on the outside.
TLS wraps HTTP (and other protocols) in encryption and identity, so nobody in between can read or forge the traffic. It's the "S" in HTTPS — and no longer optional: browsers, app stores, and APIs assume it. If data leaves your machine, it should be inside TLS.
HTTP/1.1 → 2 → 3: the same pipe, made wider
A single-lane road that became a multi-lane highway — then got rebuilt so one stalled car no longer blocks every lane.
HTTP/1.1 handled one request at a time per connection; HTTP/2 multiplexes many over one; HTTP/3 moves onto QUIC (over UDP) to kill "head-of-line blocking," open connections in roughly one round trip, and survive a phone switching from Wi-Fi to cellular without dropping. The same requests you already write — a faster road underneath.
You build at the top of the stack, but the bottom decides what's possible. And encryption isn't a layer you bolt on later.
Most app-to-server and service-to-service talk is "ask a question, get an answer." Three protocols dominate, trading simplicity for power in different ways.
REST: resources over plain HTTP verbs
A well-organised library where every book has a fixed address, and you use the same four actions everywhere: look at it, add one, change it, remove it.
REST models everything as resources at URLs (/users/42) acted on with HTTP verbs, usually returning JSON. It's simple, cacheable, debuggable in a browser, and understood everywhere — the default for public APIs. The weak spots: you often get more data than you need, or must make several calls to fill one screen.
GraphQL: ask for exactly the fields you want
A buffet where, instead of fixed set menus, you hand the kitchen a precise list and get back exactly that — no more, no less.
GraphQL gives one endpoint where the client writes a query for precisely the fields it needs, in a single round trip — ending the over- and under-fetching of REST. Brilliant when many different clients (web, mobile, partners) need different slices of a complex data graph. The price: caching is harder, and the flexibility pushes real complexity onto your server.
gRPC: call a remote function like a local one
An intercom between two rooms of the same building — terse, fast, and private, because both sides already share the same shorthand.
gRPC uses Protocol Buffers (compact binary) over HTTP/2 to let one service call another's methods directly, with a typed contract and streaming built in. It's fast and strict — ideal for internal service-to-service traffic. The trade-offs: it isn't natively browser-friendly, and the binary payload isn't human-readable, so you debug with tools, not your eyes.
And the elder: SOAP
A formal contract in triplicate, stamped and notarised — heavy, but unambiguous.
Before REST, SOAP wrapped calls in strict XML envelopes with formal contracts. It's verbose and old-fashioned, but its rigour still runs in banking, payments, and enterprise integrations. You probably won't build a new one — but you may well have to talk to one.
REST to be understood by everyone. GraphQL to let clients choose. gRPC to be fast between your own services.
Plain request/response has a gap: the server can't speak first. When the client needs to hear about things as they happen, you reach for one of these.
Polling: keep asking 'anything new?'
A child on a road trip asking "are we there yet?" every two minutes. Simple — and mostly wasted breath.
The crudest option: the client re-requests on a timer. Long-polling improves it by holding each request open until there's actually something to say, then the client immediately asks again. It works everywhere and needs no special protocol — but it's wasteful and laggy next to a real stream. A fine fallback, a poor default.
Server-Sent Events: a one-way stream from the server
A radio broadcast — the station streams to you continuously; you just listen, you don't talk back on the same channel.
SSE keeps one HTTP connection open and lets the server push a stream of updates to the client, with automatic reconnection, for almost no overhead. Perfect for one-way feeds: live scores, notifications, a ticking dashboard — and it's how an LLM streams tokens into a chat UI. If the client doesn't need to reply on the same line, SSE is the simplest real-time tool.
WebSocket: a two-way line, open both ways
An open phone line left off the hook — either side can speak the instant they have something, with no redialing.
A WebSocket upgrades an HTTP connection into a persistent, full-duplex channel where client and server send messages freely, at the lowest latency and tiny per-message overhead. The right tool when both sides talk constantly: chat, multiplayer games, collaborative editing, live trading. It's more to operate than SSE, so use it when you genuinely need two-way.
Webhooks: don't call us, we'll call you
Leaving your number with a shop so they ring you when your order arrives — instead of walking back every hour to check.
A webhook flips the direction: you register a URL, and the other service makes an HTTP request to it when an event happens. Example: Stripe POSTs to your endpoint the moment a payment succeeds; GitHub pings your server on every push. It's the standard way services notify each other without anyone sitting in a polling loop.
If the server must speak first, stop polling. Stream one way with SSE, both ways with WebSocket.
Sometimes A shouldn't call B directly at all. Put a broker between them, and the sender can fire and forget while the receiver works at its own pace.
The idea: a middleman that holds messages
A post office between sender and recipient — you drop the letter and leave; they collect it when ready. Neither has to be present at the same moment.
Instead of a direct call, the sender hands a message to a broker, which delivers it to one or more receivers. That decouples them in time (the receiver can be slow, or briefly down) and in knowledge (the sender needn't know who's listening). It's the backbone of resilient, spiky, asynchronous systems.
Message queues (AMQP / RabbitMQ): one job, one worker
An order spike at a diner — tickets pile on the rail, and whichever cook is free grabs the next one.
A queue holds tasks until a worker takes one, processes it, and acknowledges it. Protocols like AMQP (RabbitMQ) add smart routing, retries, and delivery guarantees. Example: dropping "send welcome email" or "generate invoice" onto a queue so the web request returns instantly while a worker handles the slow part.
Kafka: a durable log many can replay
A newspaper archive — every issue kept in order, and any number of readers can start anywhere and read forward at their own speed.
Kafka is less a queue than a durable, ordered log of events that many consumers read independently, and that stays around after it's read. It's built for enormous throughput. Example: one "order placed" stream feeding analytics, search indexing, and email at once — each consumer tracking its own position. A queue: the message is gone once handled. Kafka: the history stays.
MQTT: tiny pub/sub for unreliable networks
A walkie-talkie channel for a fleet of delivery vans — short bursts, low power, fine if one briefly drives through a tunnel.
MQTT is an ultra-light publish/subscribe protocol built for IoT: tiny messages, minimal battery and bandwidth, and tolerance for flaky links. Example: thousands of sensors, smart-home devices, or vehicles reporting telemetry over cellular. When the "client" is a cheap chip on a weak network, MQTT fits where HTTP won't.
Don't make the caller wait for slow work. Hand it to a broker and move on.
A protocol is the envelope; the payload inside still needs a shape both sides agree on. That choice is a quiet trade between readability and speed.
JSON: readable, universal, the default
A handwritten note anyone can read at a glance — a little long, but no decoder needed.
JSON is text, human-readable, and supported everywhere — the default body for web and REST APIs. You can eyeball it in a browser and debug by reading. The cost is size and parse speed: it's verbose and not the fastest — usually fine, occasionally the bottleneck at scale.
Protocol Buffers and friends: small, fast, binary
A barcode — unreadable to you, but scanned instantly and packed tight. You need the schema to decode it.
Binary formats like Protocol Buffers (used by gRPC), plus MessagePack and Avro, serialise data against a shared schema into a compact, fast payload. The win is size and speed; the cost is that you can't read it by eye, and both sides must share the schema. Worth it for high-volume, internal, performance-critical traffic.
XML: verbose, formal, still around
A legal document — heavy with tags and structure, precise, and nobody's first choice for a quick note.
XML predates JSON and carries rich schema and validation, at the cost of being bulky. You'll meet it in SOAP services, document formats, and configuration. Rarely the choice for something new, but still load-bearing in plenty of older systems you'll have to integrate with.
Text to be understood, binary to be fast. Most APIs are right to default to readable.
You don't pick one protocol for a system — you pick the right one for each conversation. Real apps run several at once.
Match the protocol to the conversation
A toolbox with a hammer, a screwdriver, a wrench. You don't argue which is best — you look at the job in front of you.
Decide by the shape of the talk: a public API many clients use is REST; two internal services needing speed is gRPC; a client picking exact fields is GraphQL; the server streaming updates is SSE; both sides live is WebSocket; slow background work is a queue; one service telling another that an event happened is a webhook. The question is never "which protocol" but "which one, for this line."
Start simple; reach for the exotic only when it hurts
You don't lay a railway to carry one sack of flour across the yard.
Most products go a very long way on REST + JSON over HTTPS, plus a queue for background jobs. gRPC, GraphQL, Kafka, and WebSockets are answers to specific pains — service-to-service latency, hungry clients, huge event volumes, live interaction. Adopt them when you feel that exact pain, not because they sound modern. The simplest protocol that fits the conversation usually wins.
- Who starts the conversation — the client, or the server? - How often, and how big are the messages? - One-to-one, or one-to-many over time? - Does it need to be readable, or as small and fast as possible? - Who's the client — a browser, a mobile app, another of my services, a tiny device? - Is plain REST + JSON enough before I reach for something fancier?
- You picked gRPC for a public API that browsers have to call. - WebSockets where a refresh button would do. - Kafka to move a few hundred messages a day. - GraphQL for one client that fetches the same three fields every time. - A binary format for data you constantly need to read and debug by hand.
- Client and server agree with no glue code translating between them. - The protocol matches the traffic — streaming for streams, request/response for questions. - You can debug it with the tools the protocol expects — a browser, logs, or proto tooling. - Encryption is on by default, everywhere. - Each conversation uses the simplest protocol that fits it, and no more.
Modern apps aren't built on one protocol. They're built on the right protocol for each conversation — with TLS around all of them.