Load balancing

A load balancer is the front door to your service: it spreads traffic across instances, removes broken ones, and (when configured well) handles the boring 90% of failures so your application doesn't have to.

The big idea

You have N identical app instances and one stream of incoming requests. The load balancer distributes the requests, watches health, and pretends to be one service to clients.

L4 vs L7

The first decision: at which layer does the LB look at traffic?

For most services: L7. The smart routing pays for the CPU.

Algorithms

How does the LB pick which backend gets the next request?

Round robin

Cycle through instances in order. Simple; assumes uniform request cost — wrong often.

Least connections

Send to the instance with the fewest active connections. Good when requests vary in duration.

Random

Pick uniformly at random. Surprisingly competitive; trivially scalable.

Power of two choices

Pick two random instances, send to the less-loaded one. Mathematically wins over both round-robin and pure random in practice.

Consistent hashing

Same key (user ID, session) always goes to the same backend. Used for sticky sessions and sharded caches.

Health checks

Without them, a load balancer cheerfully forwards traffic to a dead instance.

health_check:
  path: /healthz
  interval: 5s
  timeout: 2s
  healthy_threshold: 2
  unhealthy_threshold: 3

Two flavours of health check coexist:

1. 1

   Liveness

   Is the process responsive? 200 on a trivial endpoint. Used by orchestrators (K8s) to restart wedged pods.

2. 2

   Readiness

   Is this instance ready to take traffic? Returns 503 during startup, drain, or dependency outages. Used by the load balancer to remove instances from rotation.

Sticky sessions (and why to avoid them)

A sticky session pins a user to one backend by cookie or IP hash. Useful when each backend caches per-user state in memory. The cost: failover is painful (the user's data is on a now-dead instance), and rolling deploys break stickiness.

Better: make the backend stateless and store session data in Redis or a signed cookie. The load balancer goes back to round-robin and life is simpler.

TLS termination

Most production setups terminate TLS at the load balancer. The LB has the cert, decrypts traffic, and forwards plain HTTP over the private network to the backends. Saves CPU on each instance and centralises certificate rotation.

In practice

The load balancer is also your operational handle for:

• Rolling deploys — drain connections from old instances before stopping them.
• Blue-green / canary — weight traffic between two pools.
• Rate limiting + WAF — block bad traffic before it reaches your app.
• Geo-routing — send users to their nearest region.

A surprising number of "incidents" are LB misconfigurations: a stale target group, a health check pointing at the wrong path, a TLS cert expiring. Treat the LB config like production code.