Observability

Must-know concept2 hIntermediate

Logging, metrics, tracing, alerting.

You cannot fix what you cannot see. Observability is the discipline of leaving enough breadcrumbs in production that you can answer questions you didn't anticipate. Three signals — logs, metrics, traces — each answer a different question.

The big idea

The three pillars are not redundant; they're complementary.

Attribute	Pillar	Best at
Logs	Narrative of a single event	"What happened in this request?"
Metrics	Counts and aggregates over time	"Is the system OK right now?"
Traces	A request as it crosses N services	"Why was this request slow?"

Logs — but structured

Plain-text logs are useless at scale. Use structured logging: every log line is a JSON object with named fields.

{ "ts": "2025-11-12T08:00:00Z", "level": "info",
  "msg": "order.placed", "order_id": 9001, "user_id": 42,
  "request_id": "req_abc", "latency_ms": 87 }

Searchable, aggregatable, machine-parsable.

Don't

08:00:00 INFO Order 9001 placed by user 42 in 87ms (req_abc)

Each parser regex is its own bug.

Threads to weave through every log:

request_id / trace_id: Stamp every request at the gateway; propagate via headers to every downstream call.
user_id / tenant_id: Filter all logs for one customer's tickets.
service / version: Find the bad deploy.
level: debug / info / warn / error — never log info 10× per request.

Metrics

A metric is a number you can graph. Three flavours:

Counter: Monotonic — http_requests_total. Take the rate over time.
Gauge: Up and down — db_connections_active, queue_depth.
Histogram / summary: Distribution of values — request_latency_seconds. The basis of percentiles.

The RED method for any service:

R — Raterequests/sec

E — Errorserror rate

D — Durationp50/p95/p99

Plus the USE method for any resource:

U — Utilisation% busy

S — Saturationqueue depth

E — Errorserrors/sec

Tracing

A trace stitches together every span (DB call, RPC, internal function) for one request across services.

trace_id=abc123
├─ HTTP GET /orders                   [api-gateway]   142ms
│  ├─ auth.verify                      [auth-svc]      9ms
│  ├─ orders.list                      [orders-svc]   118ms
│  │  ├─ db.query (SELECT … FROM orders)              92ms  ← the slow span
│  │  └─ cache.lookup                                   2ms
│  └─ encode_response                                   3ms

Two minutes of staring at a trace replaces an hour of guessing. Use OpenTelemetry as the API; most backends (Jaeger, Tempo, Honeycomb, Datadog) speak it.

Alerting

You'll have hundreds of metrics; alert on a handful.

Alert on symptoms, not causes
"User-facing latency p99 > 2s" beats "CPU > 80% on db-3."
Page only on what wakes you usefully
A page that you ignore is worse than no page.
Pair every page with a runbook
The first thing the on-call sees should be "if this fired, do X."
Track alert noise
A flapping alert that fires nightly trains people to ignore real alerts.

In practice

Start small: structured logs with request IDs, RED metrics on every service, traces on critical paths. Promote one engineer per quarter to observability owner — somebody who watches dashboards, removes noisy alerts, and adds the field everyone needs but nobody added. The pillar is staff time as much as tooling.

Key takeaways

Logs = narrative; Metrics = aggregate; Traces = cross-service path.
Structured logs (JSON) with consistent fields beat plain-text by orders of magnitude.
RED for services, USE for resources — start with those eight numbers.
OpenTelemetry is the API; pick any backend that speaks it.
Alert on symptoms, sample traces, pair pages with runbooks.

Checkpoint questions

Use these to test whether the lesson is clear enough to explain without rereading.

1What question can logs answer that metrics or traces cannot?
2Which RED metrics would you capture for a user-facing service?
3How does a trace help debug latency across multiple services?
4What makes an alert actionable instead of noisy?

References

External resources for going deeper after the lesson above.