SE

Debugging workflow

Must-know concept2 hBeginner

Reproduce, isolate, inspect logs, form hypotheses, fix, and verify.

Debugging is disciplined evidence gathering. The goal is not to guess faster; it is to make the bug small, observable, and impossible to misunderstand.

The big idea

A reliable debugging loop is boring on purpose:

Reproduceone command
Localizewhich layer
Observelogs/data
Hypothesizefalsifiable
Fixsmall patch
Verifysame repro

When the loop is clear, even hard bugs feel tractable.

Start with reproduction

Write the smallest command, request, or test that fails every time.

curl -i http://localhost:8000/api/orders/42
pytest tests/test_orders.py::test_missing_order_returns_404 -q
npm test -- --runInBand order-detail

If the bug only happens sometimes, your first task is to collect the conditions: input, user, time, version, environment, concurrency, and external dependencies.

Trace the request flow

Do not debug "the app" as one blob. Walk the path:

L6

browser, curl, mobile app

L5

nginx, CDN, gateway

L4

method, path, auth, validation

L3

business rule and branching

L2

query, transaction, job state

L1

timeout, retry, contract drift

At each layer, ask: what did it receive, what did it decide, and what did it send next?

Make hypotheses falsifiable

Weak hypothesis: "Auth is broken."

Strong hypothesis: "The route rejects this request because the Authorization header is missing after the nginx proxy forwards it."

The strong version tells you what to inspect:

curl -v http://localhost:8000/api/me -H "Authorization: Bearer test"
docker compose logs api | rg "request_id|Authorization|401"

Read evidence, not vibes

Logs
Best for exact events: request IDs, user IDs, branches taken, errors.
Metrics
Best for shape: latency, error rate, queue depth, saturation.
Traces
Best for multi-service latency and where time disappeared.
Database state
Best for whether the system stored what you think it stored.

Add temporary instrumentation only when needed, and remove noisy debug prints before the fix lands.

Verify the fix

A fix is not done when the error disappears once. It is done when:

  • The original failing reproduction now passes.
  • At least one adjacent failure case is checked.
  • The right layer owns the fix.
  • The behavior is covered by a test or a saved smoke command.

In practice

Take a past production bug and rewrite it as a debugging trace: request, layer, evidence, root cause, patch, verification. If the trace has gaps, that is exactly where your future logs or tests should improve.

Key takeaways

  • First make the bug reproducible; then make it small.
  • Trace the full flow instead of guessing which layer is broken.
  • A useful hypothesis predicts what evidence would prove it wrong.
  • Use logs, metrics, traces, and database state for different questions.
  • Verification means rerunning the original repro and checking nearby failure cases.

Checkpoint questions

Use these to test whether the lesson is clear enough to explain without rereading.

  1. 1Can you reproduce the bug with the smallest reliable command or request?
  2. 2What layer owns the failure: client, route, service, database, queue, or infrastructure?
  3. 3Which log line, metric, trace, or curl output would falsify your current hypothesis?
  4. 4How do you prove the fix worked and did not only hide the symptom?

References

External resources for going deeper after the lesson above.