Chapter 16 — Observability (Logs, Metrics, Traces)
Chapter 16 — Observability (Logs, Metrics, Traces)
Hey everyone! Welcome back to Namaste System Design! 🙏
You've built a beautiful distributed system: load balancers, caches, queues, a dozen microservices. Then at 3am, a user tweets "your checkout is broken!" — and you have no idea where. Was it the payment service? A slow database? A full queue? Without observability, you're flying blind. This chapter gives you eyes.
What we will cover:
- Observability vs monitoring (the doctor analogy)
- The three pillars: logs, metrics, traces — with real examples of each
- Distributed tracing (following one request across services)
- Alerting, and the RED & USE methods
- SLIs, SLOs, and SLAs — with worked numbers
- What big companies actually use
- Interview Questions
1. The Doctor Analogy
┌─────────────────────────────────────────────────────────────┐ │ OBSERVABILITY = being able to understand what's happening │ │ INSIDE your system from the outside, without redeploying. │ │ │ │ Like a doctor reading your vitals 🩺: │ │ • Temperature, pulse, BP → METRICS (numbers over time) │ │ • Medical history notes → LOGS (what happened, when) │ │ • Tracing the path of a → TRACES (follow one thing │ │ pain through the body through the whole system) │ └─────────────────────────────────────────────────────────────┘
Monitoring vs observability: monitoring watches things you knew to check ("is CPU high?"). Observability lets you investigate problems you didn't predict ("why is this specific user's checkout slow?"). Monitoring answers known questions; observability lets you ask new ones.
2. The Three Pillars
Every observability tool you'll ever touch is really just one of three things. Here's the map before we zoom in:
┌──────────────┬────────────────────────┬──────────────────────┐ │ PILLAR │ WHAT IT IS │ ANSWERS │ ├──────────────┼────────────────────────┼──────────────────────┤ │ 1. LOGS │ Timestamped text │ "WHAT happened at │ │ │ records of events │ 10:42:07?" │ │ │ │ │ │ 2. METRICS │ Numbers over time │ "How MANY / how │ │ │ (counts, rates, gauges)│ MUCH / how fast?" │ │ │ │ │ │ 3. TRACES │ The full path of ONE │ "WHERE in the │ │ │ request across services│ journey did it │ │ │ │ slow down / break?" │ └──────────────┴────────────────────────┴──────────────────────┘
2.1 Logs — the diary
A log is one line of text, written the moment something happens. Timestamp + what happened. That's it.
2026-07-08 10:42:07 ERROR [payment-svc] card declined, order=55
2026-07-08 10:42:08 INFO [order-svc] order 55 cancelled
Structured version (JSON — searchable by field):
{"ts":"2026-07-08T10:42:07Z","level":"ERROR",
"service":"payment-svc","order_id":55,"msg":"card declined"}
✅ GREAT FOR "What exactly happened, to this exact order?"
❌ WEAK AT Seeing trends across thousands of requests at once
(too much text to eyeball)
👉 USE IT WHEN You already know roughly where to look and need
the exact detail — an error message, a stack trace,
a specific user's order ID.
At scale, you don't SSH into 50 servers to grep their log files. You ship every log line to one central place — ELK (Elasticsearch/Logstash/Kibana) or Loki — so you can search across the whole fleet in one query.
2.2 Metrics — the vital signs
A metric is a single number, tracked over time. No text, no detail — just a value and a timestamp, sampled every few seconds.
requests_per_second{service="payment-svc"} = 4200
error_rate{service="payment-svc"} = 2.3%
p99_latency_ms{service="payment-svc"} = 340
cpu_usage{host="app-server-7"} = 78%
✅ GREAT FOR Dashboards, trend lines, and alerts — cheap to
store because it's just numbers, not full text
❌ WEAK AT Telling you WHY a number changed — a metric shows
THAT error_rate jumped, not which order failed
👉 USE IT WHEN You want the big picture — "is the system
healthy right now?" — or you need something an
alert can watch 24/7.
Tools: Prometheus collects and stores the numbers; Grafana draws the dashboards on top of them.
2.3 Traces — the journey map
A trace follows one single request across every service it touches, timing each hop. Each hop is called a span.
ONE checkout request, traced across services:
[gateway] 5ms → [order-svc] 20ms → [payment-svc] 800ms ⚠️
→ [inventory] 15ms
Total: 840ms.
Span tree (parent → children, each with its own duration):
trace ABC123 (840ms total)
└─ gateway 5ms
└─ order-svc 20ms
├─ payment-svc 800ms ⚠️ the slow one
└─ inventory 15ms
✅ GREAT FOR "WHICH of my 10 microservices is the slow one?"
— instantly, instead of guessing for hours
❌ WEAK AT Being cheap — capturing a full trace on every
single request is expensive, so most systems
only trace a sample (e.g. 1 in 100 requests)
👉 USE IT WHEN You have more than a couple of services talking
to each other. This is THE microservices
superpower from Chapter 15.
Visual Comparison
| Pillar | Granularity | Storage cost | Best question it answers |
|---|---|---|---|
| Logs | One event | ❌ Expensive (full text) | "What exactly happened?" |
| Metrics | One number | ✅ Cheap | "Is the system healthy right now?" |
| Traces | One request, many services | ❌ Expensive (usually sampled) | "Which service is slow?" |
What Do Big Companies Actually Use?
In practice, nobody picks just one pillar — you need all three, wired together. A common real-world stack:
METRICS Prometheus (collect) + Grafana (dashboards)
LOGS ELK stack (Elasticsearch/Logstash/Kibana) or Loki
TRACES OpenTelemetry (the standard for instrumenting code)
+ Jaeger or Zipkin (storing/viewing the traces)
ALL-IN-ONE Datadog, New Relic, Honeycomb — commercial
platforms that bundle logs+metrics+traces together
The real workflow: a metric alert fires ("error rate is up") → you check the trace to find WHICH service is failing → you read that service's logs to find WHY. The three pillars aren't alternatives, they're a funnel — metrics tell you something's wrong, traces tell you where, logs tell you why.
3. Distributed Tracing — How It Works
When a request enters the system, it's tagged with a unique trace ID. That ID is passed along to every service the request touches, usually in an HTTP header. Later, you collect all the pieces with the same ID and reassemble the full journey.
Request enters → assign trace-id: ABC123
│ (ID travels in headers to every hop)
▼
gateway(ABC123) → order(ABC123) → payment(ABC123) → db(ABC123)
Each hop = a "SPAN" (with its own start time, end time, and
parent span). Collect all spans sharing trace-id ABC123 →
see the whole request as a waterfall, like the span tree
in 2.3 above.
Tools: OpenTelemetry (the standard way to emit spans),
Jaeger, Zipkin (store and visualize them).
This is the answer to "which of my 10 services broke?" from Chapter 15 — distributed tracing is what makes microservices debuggable.
4. Alerting & The RED / USE Methods
You can't stare at Grafana 24/7. Alerts watch metrics and page a human the moment something's wrong — so you don't have to be watching.
4.1 Alert rules — watch symptoms, not just causes
IF error_rate > 5% for 2 minutes → PAGE the on-call engineer 📟 IF p99_latency > 1s → send a warning IF disk > 90% full → page before it fills
✅ GOLDEN RULE Alert on SYMPTOMS users feel (errors, latency),
not just causes (high CPU). High CPU with no
impact on users doesn't need to wake anyone up.
❌ AVOID "Alert fatigue" — too many noisy, low-value
alerts, and people start ignoring ALL of them,
including the real ones.
But which metrics should you even be watching? Two well-known checklists answer that:
4.2 The RED Method — for services
For anything that handles requests (an API, a microservice), track three numbers:
R — Rate requests_per_second = 4200 E — Errors error_rate = 2.3% D — Duration p99_latency_ms = 340 Example alert built from RED: IF Errors > 5% AND Rate > 100 req/s for 2 min → page (ignores noisy blips when traffic is near-zero)
👉 Use it when: you're instrumenting a service and don't know where to start. RED is the "default dashboard" for any request-driven system.
4.3 The USE Method — for resources
For anything that's a finite resource (CPU, disk, a connection pool), track three different numbers:
U — Utilization cpu_usage = 78% (how busy is it?)
S — Saturation queue_depth = 240 (how much is queued
waiting for it?)
E — Errors disk_errors = 3 (is it failing?)
Example: a connection pool at 95% utilization AND rising
saturation (queued requests waiting for a connection) is
about to become your bottleneck — USE catches this early.
👉 Use it when: you're watching infrastructure — CPU, memory, disk, connection pools, thread pools — rather than a request-handling service. RED and USE together cover almost everything you'd want to alert on.
5. SLIs, SLOs, and SLAs — The Promise Chain
These three letters get mixed up constantly. Think of them as three steps: measure it → aim for it → promise it.
SLI (Indicator) → the actual measured number, right now
e.g. "this month, 99.95% of requests succeeded"
SLO (Objective) → your internal target for that indicator
e.g. "we aim for 99.9% success" — alert if we
dip below this
SLA (Agreement) → the PROMISE made to customers, with penalties
e.g. "99.9% uptime or we refund you" — usually
set looser than the SLO, so you breach your
own internal target before you breach the
customer's contract
Worked example: turning 99.9% into an error budget
An SLO of "99.9% success" sounds abstract until you convert it into a number of allowed failures — this is called an error budget.
SLO: 99.9% of requests succeed → 0.1% are ALLOWED to fail
At 4,000 requests/sec, over 30 days:
total requests ≈ 4,000 × 86,400 × 30 ≈ 10.4 billion
allowed failures = 0.1% of 10.4B ≈ 10.4 million requests
That "10.4 million failures" is your error budget for the
month. Burn through it faster than expected (say, half of
it gone in the first 3 days) → that's exactly the kind of
thing an alert should catch, before the SLA is at risk.
Relationship: you measure SLIs continuously, you aim for SLOs (a stricter internal bar), and you promise SLAs (a looser external bar, with money on the line). A health check — the /health endpoint the load balancer pings (Chapter 04) — is one of the simplest SLIs you can build: it's a live yes/no reading of "is this instance OK?"
Interview Questions — Quick Fire!
Q: What are the three pillars of observability?
"Logs, metrics, and traces. Logs are timestamped records of discrete events — they tell you what happened, like a specific error for a specific order. Metrics are numeric measurements over time like request rate, error rate, and latency — they tell you how much and how fast, and drive dashboards and alerts. Traces follow a single request across all the services it touches — they tell you where in the journey something slowed down or failed. In practice they work together: a metric alert tells you something's wrong, a trace tells you where, and a log tells you why."
Q: What's the difference between monitoring and observability?
"Monitoring is watching predefined signals you already know to check, like CPU or error rate, and it answers known questions. Observability is the broader ability to understand the internal state of a system from its outputs, so you can investigate problems you didn't anticipate — like why one specific user's request was slow. Monitoring answers known questions; observability lets you ask new ones."
Q: What is distributed tracing and why is it important?
"Distributed tracing tags each incoming request with a unique trace ID that's propagated to every service the request touches. Each service records a span with timing, and all spans sharing the trace ID are reassembled into a tree showing the request's full path. It's essential in microservices because a single request spans many services, and tracing instantly shows which service caused a slowdown or error — for example a 840ms checkout request where payment-svc alone took 800ms — which would otherwise take hours to find by guessing."
Q: What's the RED method, and what's USE? When would you use each?
"RED is Rate, Errors, Duration — the three numbers to track for anything that serves requests, like a microservice. USE is Utilization, Saturation, Errors — the three numbers to track for a finite resource, like CPU, disk, or a connection pool. I'd put RED on my API services and USE on my infrastructure. Together they cover almost every alert I need without having to guess what to watch."
Q: What's the difference between an SLA, SLO, and SLI?
"An SLI is the indicator — the actual measured value, like the percentage of successful requests. An SLO is the objective — your internal target for that indicator, like 99.9% success, which you alert on. An SLA is the agreement — the formal promise to customers, often with financial penalties if breached. You measure SLIs, aim for stricter SLOs internally, and commit to looser SLAs externally. A 99.9% SLO at 4,000 requests a second gives you an error budget of about 10 million allowed failures a month — burning that budget too fast is itself something worth alerting on."
Q: What makes a good alert?
"It should fire on symptoms that users actually feel — like elevated error rates or high latency — rather than raw causes like high CPU that may not affect users. It should be actionable and not too noisy, because too many false alarms cause alert fatigue and people start ignoring them. The goal is that every page means a real problem worth waking someone for."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Observability | Understand a running system from outside; ask new questions, not just known ones. |
| Logs | What happened, when. Structured (JSON) + centralized (ELK/Loki) for search. |
| Metrics | Numbers over time. Cheap, powers dashboards + alerts (Prometheus/Grafana). |
| Traces | Follow one request across services via a trace ID + spans → find the slow hop. |
| RED / USE | RED (Rate/Errors/Duration) for services; USE (Utilization/Saturation/Errors) for resources. |
| SLI/SLO/SLA | Measure SLIs, aim for stricter SLOs, promise looser SLAs. SLO → error budget. |
| Alerts | Alert on user-felt symptoms, not raw causes. Avoid alert fatigue. |
What's Next?
Season 4 complete — you now have the full toolbox! In Chapter 17 we begin the grand finale, Season 5: The Interview. First up: a repeatable framework to attack ANY system design question, so you never freeze at the whiteboard again.
Keep designing, keep scaling! See you in the next one!
Post a Comment