Scaling strategies

"Vertical or horizontal" is shorthand for a deeper choice: do I throw a bigger machine at the problem, or change the design to share it across many small ones? The deeper choice shapes every other piece of your architecture.

The big idea

Two axes:

For nearly every modern app: vertical for the database, horizontal for everything else.

Make services stateless

A stateless service stores no per-user state in memory between requests. Every request can be routed to any instance. That's the magic that makes horizontal scaling work.

const session = await redis.get(`session:${sid}`);
if (!session) return res.status(401).end();

session = await redis.get(f"session:{sid}")
if session is None:
    return Response(status_code=401)

const sessions: Map<string, Session> = new Map();   // dies with the process
sessions.set(sid, session);

sessions: dict[str, Session] = {}   # dies with the process
sessions[sid] = session

Connection pooling

Each app instance opens DB connections; if 50 instances each open 100 connections, that's 5,000 connections — far more than Postgres can handle. Pool inside each instance, and use a connection pooler (PgBouncer, RDS Proxy) in front of the database.

Scaling the database

Stateful tiers are harder. The escalation:

1. 1

   Vertical first

   Bigger instance. Postgres on a 64-core box goes a long way.

2. 2

   Read replicas

   Reads go to replicas; writes still hit the primary. Watch for replication lag bugs.

3. 3

   Sharding / partitioning

   Split data across multiple primaries by some key (user_id, tenant_id). Now you have a distributed system; the rules change.

4. 4

   Specialised stores

   Move hot workloads off: ElasticSearch for search, Redis for sessions, Kafka for streams, ClickHouse for analytics.

Async over sync where you can

Synchronous work limits throughput to the latency of the slowest dependency. Pushing slow work to a queue/worker frees the request path:

Order placed → return immediately, send the receipt email asynchronously. The user sees "order confirmed" in milliseconds; the email is queued for a worker.

Watch for hidden bottlenecks

Even a stateless horizontal service still funnels through:

The database

Eventually you hit max connections or write throughput.

Caches

Hot keys serialise on a single Redis instance.

Locks

Anywhere SELECT ... FOR UPDATE happens, contention kills throughput.

Third-party APIs

Their rate limit is your scaling ceiling.

In practice

Most teams scale prematurely. The right order:

1. Profile the slow path.
2. Add an index, cache the obvious answer, batch the N+1 queries.
3. Add app instances (cheap, easy).
4. Then consider read replicas, queues, sharding.

By the time you're sharding the database, you should know the system intimately. If you don't, sharding will make things worse.