Software Architect · Module 10
The moment the system crosses the boundary of a single process, you get latency, retries, partial failures, and an ambiguous truth.
Partial failure · timeout · retry · consistency · CAP
In a distributed system, failure can be partial: one component is sure the operation went through, another one doesn't know it yet.
The network isn't a function
Sending a letter doesn't mean the addressee got it, read it, and understood it. The network is the same.
A call to a neighboring service can end in success, timeout, network error, or a response that comes back after the client has already given up. The server may have executed the operation, but the client didn't get the response. The client may have retried, and the server got a duplicate.
So distributed design needs timeouts, retries with backoff, idempotency, circuit breakers, correlation ids, and clear lines of responsibility. Without those, the system sometimes works only because load is low.
Consistency costs money
You can synchronize the clocks in every room of a building. The bigger the building, the more expensive perfect accuracy gets.
Strong consistency is convenient for the user and the developer, but it constrains availability and latency. Eventual consistency scales better and survives failures, but it requires compensations, pending states, and a UX that's honest about an intermediate state.
The architect shouldn't throw the CAP theorem around as a slogan. The concrete question matters more: where does the business require strict consistency, and where is propagation delay acceptable?
The complexity of a distributed system shows up at the boundaries: money, inventory, notifications, reporting.
Example: an order and an email as different guarantees
A receipt matters more than a marketing email. Losing the receipt is an incident; losing the email is an annoyance.
Creating the order and charging the money have to be tightly controlled. The email confirmation can go through an outbox and a retry worker. If the email is delayed, the order is still valid.
That separation of guarantees keeps you from dragging a distributed transaction into a place where eventual consistency is enough.
Anti-example: a synchronous chain of seven services
If getting to work depends on seven elevators in a row, one stuck elevator ruins the whole day.
The frontend calls the API, the API calls checkout, checkout calls pricing, pricing calls catalog, then identity, payment, notification. Each has a 30-second timeout. When something goes wrong, the user waits and the engineers hunt for where exactly it broke.
Synchronous chains multiply latency and the probability of failure. They need to be shortened, cached, moved to async, or designed with explicit degradation paths.
- What happens on a timeout after a successful execution?
- Which operations have to be idempotent?
- Where is strong consistency required?
- Which partial failure will the user see, and how will the UI explain it?