Chapter 16 — Observability (Logs, Metrics, Traces)
Chapter 16 — Observability (Logs, Metrics, Traces)
Hey everyone! Welcome back to Namaste System Design! 🙏
You've built a beautiful distributed system: load balancers, caches, queues, a dozen microservices. Then at 3am, a user tweets "your checkout is broken!" — and you have no idea where. Was it the payment service? A slow database? A full queue? Without observability, you're flying blind. This chapter gives you eyes.
What we will cover:
- Observability vs monitoring (the doctor analogy)
- The three pillars: logs, metrics, traces
- What each pillar answers
- Distributed tracing (following one request across services)
- Alerting & the RED/USE methods
- Health checks & SLAs/SLOs
- Interview Questions
1. The Doctor Analogy
┌─────────────────────────────────────────────────────────────┐ │ OBSERVABILITY = being able to understand what's happening │ │ INSIDE your system from the outside, without redeploying. │ │ │ │ Like a doctor reading your vitals 🩺: │ │ • Temperature, pulse, BP → METRICS (numbers over time) │ │ • Medical history notes → LOGS (what happened, when) │ │ • Tracing the path of a → TRACES (follow one thing │ │ pain through the body through the whole system) │ └─────────────────────────────────────────────────────────────┘
Monitoring vs observability: monitoring watches things you knew to check ("is CPU high?"). Observability lets you investigate problems you didn't predict ("why is this specific user's checkout slow?"). Monitoring answers known questions; observability lets you ask new ones.
2. The Three Pillars
┌──────────────┬────────────────────────┬──────────────────────┐ │ PILLAR │ WHAT IT IS │ ANSWERS │ ├──────────────┼────────────────────────┼──────────────────────┤ │ 1. LOGS │ Timestamped text │ "WHAT happened at │ │ │ records of events │ 10:42:07?" │ │ │ │ │ │ 2. METRICS │ Numbers over time │ "How MANY / how │ │ │ (counts, rates, gauges)│ MUCH / how fast?" │ │ │ │ │ │ 3. TRACES │ The full path of ONE │ "WHERE in the │ │ │ request across services│ journey did it │ │ │ │ slow down / break?" │ └──────────────┴────────────────────────┴──────────────────────┘
Pillar 1 — Logs
2026-07-08 10:42:07 ERROR [payment-svc] card declined, order=55 2026-07-08 10:42:08 INFO [order-svc] order 55 cancelled Detailed, specific, great for "what exactly happened." At scale → ship to a central place (ELK / Loki) so you can search across ALL servers, not SSH into each one. TIP: use STRUCTURED logs (JSON) so they're searchable.
Pillar 2 — Metrics
requests_per_second: 4200 error_rate: 2.3% p99_latency_ms: 340 cpu_usage: 78% Cheap to store, perfect for DASHBOARDS & ALERTS. Tools: Prometheus (collect) + Grafana (visualize).
Pillar 3 — Traces (the microservices superpower)
ONE checkout request, traced across services:
[gateway] 5ms → [order-svc] 20ms → [payment-svc] 800ms ⚠️
→ [inventory] 15ms
Total 840ms. The trace INSTANTLY shows payment-svc is the
culprit. Without tracing, you'd guess for hours.
3. Distributed Tracing — How It Works
When a request enters the system, it's tagged with a unique trace ID. That ID is passed along to every service the request touches. Later, you collect all the pieces with the same ID and reassemble the full journey.
Request enters → assign trace-id: ABC123
│ (ID travels in headers to every hop)
▼
gateway(ABC123) → order(ABC123) → payment(ABC123) → db(ABC123)
Each hop = a "SPAN" (with start/end time). Collect all spans
with trace-id ABC123 → see the whole request as a waterfall.
Tools: OpenTelemetry (standard), Jaeger, Zipkin.
This is the answer to "which of my 10 services broke?" from Chapter 15 — distributed tracing is what makes microservices debuggable.
4. Alerting — Don't Watch Dashboards All Day
You can't stare at Grafana 24/7. Alerts watch metrics and page a human when something's wrong.
IF error_rate > 5% for 2 minutes → PAGE the on-call engineer 📟 IF p99_latency > 1s → send a warning IF disk > 90% full → page before it fills GOLDEN RULE: alert on SYMPTOMS users feel (errors, latency), not just causes (high CPU). Avoid "alert fatigue" — too many noisy alerts and people start ignoring them.
Two handy metric recipes
RED (for request-driven services):
Rate · Errors · Duration
USE (for resources like CPU/disk):
Utilization · Saturation · Errors
5. Health Checks, SLAs, SLOs, SLIs
HEALTH CHECK → a /health endpoint the load balancer pings
to know if a server is alive (Chapter 04).
SLI (Indicator) → the actual measured number
e.g. "99.95% of requests succeeded"
SLO (Objective) → your internal target
e.g. "99.9% success" — alert if we dip below
SLA (Agreement) → the PROMISE to customers (with penalties)
e.g. "99.9% uptime or we refund you"
Relationship: you measure SLIs, aim for SLOs (stricter),
and promise SLAs (looser, with money on the line).
Interview Questions — Quick Fire!
Q: What are the three pillars of observability?
"Logs, metrics, and traces. Logs are timestamped records of discrete events — they tell you what happened. Metrics are numeric measurements over time like request rate, error rate, and latency — they tell you how much and how fast, and drive dashboards and alerts. Traces follow a single request across all the services it touches — they tell you where in the journey something slowed down or failed."
Q: What's the difference between monitoring and observability?
"Monitoring is watching predefined signals you already know to check, like CPU or error rate, and it answers known questions. Observability is the broader ability to understand the internal state of a system from its outputs, so you can investigate problems you didn't anticipate — like why one specific user's request was slow. Monitoring answers known questions; observability lets you ask new ones."
Q: What is distributed tracing and why is it important?
"Distributed tracing tags each incoming request with a unique trace ID that's propagated to every service the request touches. Each service records a span with timing, and all spans sharing the trace ID are reassembled into the request's full path. It's essential in microservices because a single request spans many services, and tracing instantly shows which service caused a slowdown or error, which would otherwise take hours to find."
Q: What's the difference between an SLA, SLO, and SLI?
"An SLI is the indicator — the actual measured value, like the percentage of successful requests. An SLO is the objective — your internal target for that indicator, like 99.9% success, which you alert on. An SLA is the agreement — the formal promise to customers, often with financial penalties if breached. You measure SLIs, aim for stricter SLOs internally, and commit to looser SLAs externally."
Q: What makes a good alert?
"It should fire on symptoms that users actually feel — like elevated error rates or high latency — rather than raw causes like high CPU that may not affect users. It should be actionable and not too noisy, because too many false alarms cause alert fatigue and people start ignoring them. The goal is that every page means a real problem worth waking someone for."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Observability | Understand a running system from outside; ask new questions, not just known ones. |
| Logs | What happened, when. Structured + centralized for search. |
| Metrics | Numbers over time. Power dashboards + alerts (Prometheus/Grafana). |
| Traces | Follow one request across services via a trace ID → find the slow hop. |
| Alerts & SLOs | Alert on user-felt symptoms; measure SLIs, target SLOs, promise SLAs. |
What's Next?
Season 4 complete — you now have the full toolbox! In Chapter 17 we begin the grand finale, Season 5: The Interview. First up: a repeatable framework to attack ANY system design question, so you never freeze at the whiteboard again.
Keep designing, keep scaling! See you in the next one!
Post a Comment