Message queues

Must-know concept2 hIntermediate

Kafka, RabbitMQ, SQS — async processing.

A queue decouples the thing that makes work from the thing that does the work. It smooths spikes, enables retries, and lets you scale producers and consumers independently. The trade-off is a whole new class of failure modes.

The big idea

Replace the synchronous call with a queue:

API → queue → worker pool
              → DB
              → email
              → analytics

Request returns in 50ms. Slow work happens later.

Don't

API → DB → email → analytics → response

Request blocks for 8s; the email server timing out fails the whole order.

Two shapes: log vs broker

The two dominant designs differ in what "a message" is.

Attribute	Log (Kafka, Pulsar, Kinesis)	Broker (RabbitMQ, SQS)
Storage	Append-only log, retained for days	Queue; message gone after consumed
Replay	Yes — read from any offset	No (or limited)
Multiple consumers	Each consumer group reads independently	Each message goes to one consumer
Ordering	Per-partition strict	Per-queue best-effort
Best for	Event sourcing, analytics, replay	Task queues, work distribution

Delivery semantics

The defining question: what does "the message was delivered" actually mean?

At-most-once: May be lost; never duplicated. "Fire and forget."
At-least-once: Never lost; may be duplicated. The pragmatic default — your handlers must be idempotent.
Exactly-once: Never lost, never duplicated. Available in some systems (Kafka transactions), with caveats. Hard to do truly E2E.

What goes in a message

A message is a promise of work to do. Keep it small and self-contained.

{
  "id": "msg_8e4f2c…",                  // unique → idempotency key
  "type": "order.placed",
  "version": 1,
  "occurred_at": "2025-11-12T08:00:00Z",
  "data": {
    "order_id": 9001,
    "user_id": 42
  }
}

Don't put giant payloads in messages (PDFs, video). Store them in object storage and pass a reference. Don't put state that may have changed — send IDs, let the handler re-read fresh state.

Failure handling

Every queue system has at minimum these three concepts:

Visibility timeout / lease
When a worker picks up a message, the queue hides it for N seconds. If the worker crashes before acking, the message reappears for another worker.
Retries with backoff
Failed messages go back to the queue, ideally with exponential backoff and jitter. Configure a max retry count.
Dead-letter queue (DLQ)
After max retries, the message moves to a DLQ for human inspection. Set up alerts — a growing DLQ is a P1 incident.

Ordering

In Kafka, ordering is per partition. You partition by a key (order_id, user_id), and all messages for the same key arrive in order to the same consumer. Across keys, there is no global order. Design around this — if your handler depends on "things happened in this order," your partition key must contain everything that defines that order.

In practice

The first time you reach for a queue, you'll be tempted to model business logic as 12 small queues + 12 small workers. Resist. Start with one queue, one worker pool, simple message shapes. Split when you have a real reason: independent scaling needs, different SLAs, or genuinely different domains.

Key takeaways

Queues decouple producers from consumers — smooth spikes, enable retries, scale independently.
Logs (Kafka) for replay/analytics; brokers (Rabbit, SQS) for work distribution.
Pick at-least-once + idempotent handlers over chasing exactly-once.
Keep messages small; pass references to large payloads.
Visibility timeout, retries with backoff, and a DLQ are non-negotiable.
Kafka ordering is per partition — your partition key defines the unit of order.

Checkpoint questions

Use these to test whether the lesson is clear enough to explain without rereading.

1What work should move from a synchronous request into a queue?
2How do producers, brokers, consumers, acknowledgements, and retries fit together?
3Why must consumers usually be idempotent?
4When would you choose Kafka, RabbitMQ, or SQS-style queues?

References

External resources for going deeper after the lesson above.