Distributed Systems

A distributed system is a group of computers that work together over a network and look like one system to their users. Almost every tool a data engineer touches is one: Kafka, Spark, Flink, Trino, Cassandra, Snowflake, BigQuery, and every lakehouse on object storage. You spread work over many machines to store more data, handle more traffic and survive failures, and in exchange you inherit problems a single machine never has: networks drop messages, machines fail independently, and clocks disagree.

This section explains the handful of ideas behind those trade-offs. Once they click, the documentation of most data tools reads very differently.

Partial failure

Some parts break, others don't

A node can be dead, slow or just unreachable, and from the outside these look the same.

No shared memory

Nodes only exchange messages

Every message can be delayed, duplicated or lost. Agreement has to be built on top.

No global clock

"Now" depends on who you ask

Ordering events across machines needs care, not just timestamps.

Why go distributed: scaling up vs scaling out

When one machine can’t keep up, you have two options. Scale up (vertical scaling): move to a bigger machine with more CPU, memory and disk. Scale out (horizontal scaling): add more ordinary machines and split the data and work between them, so the group behaves as one cluster.

$ add-capacity --strategy up|out
A bigger machine vs more machines
SCALE UP · vertical SCALE OUT · horizontal largest machine you can buy db-1 4 CPU · 16 GB node-1 A–Z all db-1 16 CPU · 64 GB db-1 192 CPU · 2 TB price grows faster largest machine you can buy one machine, one failure takes all coordinator routes · merges node-1node-2node-3 A–HI–PQ–Z 1/31/31/3 node-1node-2node-3node-4 A–FG–LM–RS–Z 1/41/41/41/4 SELECT count(*) FROM orders 120 + 98 + 131 + 105 = 454
  1. Both sides start the same: one machine holds all the data and does all the work.
  2. Scale up: move to a bigger machine. No code changes, no new failure modes. This is the right first move surprisingly often.
  3. But there is a ceiling: the biggest instance on the market. Price rises faster than capacity near the top, and it is still one machine that can fail.
  4. Scale out: split the data across three nodes by key range. A coordinator (or a smart client) knows which node holds which keys.
  5. Need more? Add a node and rebalance: some keys move to node-4 and each node now holds a quarter. Capacity grows in small steps with no hard ceiling.
  6. A query fans out: every node counts its part in parallel and the coordinator merges the partial results. This is how Spark, Trino and most warehouses work.
Scale up (vertical) Scale out (horizontal)
How Bigger CPU, RAM, disk on one machine More machines, data and work split between them
Limit The largest machine available Coordination and network costs, not hardware
Failure One machine down means everything down A node can fail; replicas on other nodes take over
Complexity Low: same software, same queries High: sharding, rebalancing, network partitions, consistency trade-offs
Examples A larger Postgres instance, a bigger single-node DuckDB box Kafka brokers, Spark executors, Cassandra rings, Snowflake/BigQuery compute

A practical rule: scale up while you can, because a single machine is simpler and today’s large instances are very large. Scale out when the data or traffic outgrows the biggest sensible machine, or when you need the system to survive a machine failure.

What’s in this section

Spreading the data

  • Replication: copies on several machines; leaders, followers, sync vs async and replication lag.
  • Sharding: splitting data across machines; range vs hash partitioning and choosing a shard key.
  • Indexing: finding rows without scanning everything; B-trees, index types and local vs global indexes.

Keeping copies correct

  • CAP: what a replicated database must give up when the network splits.
  • PACELC: the everyday trade-off between latency and consistency.
  • Consistency vs Eventual Consistency: consistency models, quorums and conflict resolution.
  • Clock: why timestamps lie, and Lamport, vector and hybrid clocks.

Changing data safely

Processing data

A good reading order for newcomers: replication, sharding, CAP, consistency, then the rest in any order.

Learning Resources

Books

  • Designing Data-Intensive Applications (2nd ed.), Martin Kleppmann and Chris Riccomini: the standard reference for everything in this section
  • Distributed Systems, Maarten van Steen and Andrew Tanenbaum: free digital edition
  • Designing Distributed Systems, Brendan Burns: patterns for containerized systems
  • Database Internals, Alex Petrov

Courses

Articles