Software Architect · Module 17
Observability answers one question: can we understand what the system is doing — without deploying new code and without guessing from second-hand symptoms.
Logs · metrics · traces · correlation id · alerting
You design observability alongside the system — not bolt it on after the first big incident.
The three signals solve different problems
The thermometer, the map, and the dashcam each show a different side of the trip. One instrument isn't enough.
Metrics show aggregate state: latency, error rate, queue depth, memory, throughput. Logs give you events and context. Traces show the path of a request across services.
The architect has to decide which signals matter for the critical user journeys. For checkout: payment latency, failed authorisations, retry count, outbox lag, provider errors. For a content site: cache hit ratio, build failures, p95 page load, search freshness.
An alert has to demand action
An alarm that goes off every hour for no reason quickly becomes part of the background noise.
Bad alerts create fatigue. A good alert is tied to user impact or to a meaningful approach to a dangerous limit. It has an owner, a severity, a runbook, and a clear action.
Not everything needs to wake someone up at night. Error logs with no SLO impact can live on a dashboard or in a ticket. A drop in successful payments should page on-call immediately.
During an incident, observability should cut time-to-understanding — not add noise.
Example: a correlation id across the whole request path
A case number in court lets you pull every document, witness, and ruling. Without it, you're searching from memory.
Every inbound request gets a correlation id. It travels through the API, the worker, broker messages, and downstream calls. Logs and traces carry it. Support pulls the id from the user's error, and an engineer can see the full request path at a glance.
That collapses MTTR: the team isn't searching for "something around 14:03" — they're looking at a specific chain of events.
Anti-example: logs as a stream of text
A book with no chapters, no page numbers, and no table of contents holds information — but you can't actually use it.
The service writes something failed, bad request, payment error — with no user id, tenant id, order id, provider code, retry count, or severity. In the incident review, the team realises the data was there but you can't investigate from it.
Structured logs with context beat pretty strings.
- Which SLOs are visible in metrics?
- Can you trace a single request across all the services?
- Which alerts demand immediate action?
- Which fields do logs need to support an investigation?