Why We Need Distributed Systems

Sat, 01 Aug 2026 00:00:00 GMT

The one server setup nobody wants to admit they miss

You have a web app on a single server. Nginx, your application code, and Postgres, all on the same box. A few dozen people use it during the day. Response times are good. When something breaks you SSH in, tail the logs, and usually know what went wrong inside a minute.

This is a nice place to be. Most teams leave it earlier than they should.

The reason it’s nice isn’t nostalgia. A single machine gives you things that are genuinely expensive to buy back later. Your app talks to your database through a function call measured in microseconds, not a network round trip measured in milliseconds. There’s exactly one copy of your data, so “which version is correct” isn’t a question anyone has to ask. There’s one set of logs. Deploys are push, restart, done.

None of that is nothing. Every one of those properties have a price tag once you go distributed, and you pay it forever.

So the honest framing isn’t “when do I get to build a distributed system.” It’s “what forces me off this machine.”

What actually forces you off

Four things, and they arrive in a fairly predictable order.

CPU goes first for most applications. Every server has a fixed number of cores, and once your business logic, JSON serialization, and template rendering saturate them, requests start queuing. The symptom isn’t a crash. It’s response times that creep from 80ms to 400ms over a few weeks while everyone argues about whether it’s the database.

Memory goes next, and it goes badly. Your app and your database are competing for the same RAM. When you run out, the OS starts swapping, and memory access that took nanoseconds now takes milliseconds. Performance doesn’t degrade gracefully here. It falls off a cliff, and it usually does so at 2am on a Saturday.

Disk I/O is the quiet one. SSDs pushed this ceiling much further out than it used to be, but a single drive still has a throughput limit, and a Postgres instance doing a few thousand transactions per second will find it.

Network bandwidth matters if you’re streaming, serving large payloads, or holding thousands of open WebSocket connections. One machine, one NIC.

And then there’s the fifth thing, which isn’t a resource limit at all. That box is a single point of failure. Hardware fault, kernel panic, a bad deploy, someone filling the disk with a runaway log file. Doesn’t matter which. Everything is down. Every user, every request.

You can throw money at the first four. You can’t throw money at the fifth.

Buying a bigger box works until it doesn’t

The instinctive fix is vertical scaling. More cores, more RAM, faster disks. And this works, genuinely works, for longer than distributed systems evangelists like to admit.

What kills it is the cost curve. Eight cores to sixteen might cost you double. Sixteen to thirty two, maybe four times. Past that you’re shopping for specialised hardware at ten to twenty times commodity pricing, and you’re doing it to buy a machine that is still, at the end of all that spending, one machine that can die.

Vertical scaling is a strategy with a known expiry date. The only question is whether you hit it before your growth stalls.

Going out instead of up

The alternative is to split the work across several smaller machines.

The canonical first step is unglamorous. A load balancer in front of two or three app servers, with the database moved to its own dedicated box and a read replica behind it. Same traffic, but each server now sits at 35% CPU instead of 95%. There’s headroom. When you need more, you add another commodity server and the load balancer picks it up.

That’s the whole idea. It’s not complicated. What’s complicated is everything that follows from it.

What you get

Capacity that scales with your budget, not with hardware limits. The ceiling becomes how many machines you’re willing to run, not how fast the fastest available CPU is.

Failures that don’t take everything down. One server dies, the others keep serving. Users might see a blip during failover. They don’t see a maintenance page.

The ability to scale parts independently. Your database and your app servers almost never hit their limits at the same time. Distributed architecture lets you add read replicas without touching the app tier, or add app servers without touching the database.

Proximity to users. A request from Tokyo served out of Tokyo is 20ms. Served out of Virginia it’s 200ms. For an interactive product, that difference is the entire feel of the thing.

What you give up, which is more than people expect

This is the part that gets undersold in every architecture diagram.

Every function call becomes a network request. That request can fail. It can time out. It can arrive out of order. Worst of all, it can succeed on the receiving end while the response gets lost on the way back, so your caller believes the operation failed when it actually happened. Every interaction between machines now has failure modes that simply did not exist on one box. Idempotency stops being something nice to have and becomes a requirement in places you won’t anticipate.

Failure stops being binary. On one server, you’re up or you’re down. In a distributed system, Server A is healthy, Server B is thrashing, the database is mid backup and slow, and roughly 8% of requests are failing for reasons nobody can reproduce. “Is the site up?” becomes a question without a clean answer, which means your monitoring has to get much better before your architecture gets much bigger.

The network is not reliable and will remind you. Packets drop. Latency spikes. Partitions happen, and they don’t happen cleanly, you rarely get a total split. You get the app server that can reach the database but not the cache, or the two app servers that can’t see each other but can both see the database and are now both convinced they’re the leader.

State gets genuinely hard. Two copies of the data on two servers. A user updates their profile against Server A. The next request lands on Server B. What does it see? This sounds like a trivial question and it is not. It’s one of the hardest problems in the field, it has formal machinery built around it called consistency models, and it deserves it’s own post rather than a paragraph here.

If your workload fits on one machine, none of the above is a price worth paying. Say that out loud in the design review.

Three properties, permanently in tension

Nearly every architecture argument you’ll have reduces to one of three things.

Scalability is whether you can absorb more load by adding resources, without rewriting anything. That last clause is the whole point. Plenty of systems can handle 10x traffic if you’re willing to spend three months restructuring them first. That isn’t scalability, that’s a rebuild.

Resiliency is whether the system keeps working when things break, and things break constantly. Servers crash, disks fill, deploys ship bugs. Resiliency isn’t about preventing failure, it’s about containing how far a failure spreads and recovering without a human in the loop. Degraded but functioning beats down.

Maintainability is whether people can understand, operate, and change the system. This one gets treated as a soft concern and it absolutely isn’t. A system nobody can debug in production, nobody can deploy without dread, and nobody can modify without breaking something adjacent is a liability regardless of how well it scales.

The tension is the interesting part. Redundancy and failover make a system more resilient and less maintainable, less moving parts is not something you get for free. Caching and partitioning make it more scalable and introduce entirely new failure modes. There’s no configuration where all three are maximised. You’re picking a point on a surface based on what your product actually needs, and being honest about that beats copying whatever architecture was in the last conference talk you watched.

The pieces you’ll end up with

Most modern distributed systems converge on roughly the same components.

Clients. Browsers, mobile apps, other services. They’re outside your control, they’re on worse networks than you assume, and they will retry aggressively at the worst moment.

CDN. A globally distributed cache for static assets. Your JS, CSS, and images get served from a node near the user. This absorbs the easy traffic so your servers can spend their cycles on requests that actually need computing.

Load balancer. Distributes incoming requests across your app servers. Works at different layers depending on how much you need it to know, whether that’s routing based on DNS, Layer 4 routing on IP and port, or Layer 7 reading HTTP headers and making decisions based on path or host.

Application services. Your business logic, often split into several independently deployable pieces. Auth, orders, payments. They talk to each other synchronously over HTTP or gRPC, or asynchronously through a queue. Splitting these too early is one of the most common and most expensive mistakes in this space.

Database. Usually Postgres or MySQL for anything transactional. A primary handling writes, one or more replicas handling reads. You get fault tolerance, promote a replica, and read scaling, spread the queries, from the same arrangement.

Cache. Redis or Memcached sitting between the app and the database. Microsecond responses instead of millisecond ones. The tax is that you now maintain two copies of the truth, and reconciling them is your problem forever.

Message queue. Kafka, SQS, RabbitMQ. Lets Service A hand off work and move on instead of blocking on Service B. This decouples services in time, which means they no longer have to be healthy simultaneously. That property is worth more than the throughput gains people usually cite.

None of these are mandatory. Add each one when something is actually hurting, not because the diagram looks incomplete without it.

codetheorem.dev