Distributed Systems
A distributed system is a group of computers that work together over a network and look like one system to their users. Almost every tool a data engineer touches is one: Kafka, Spark, Flink, Trino, Cassandra, Snowflake, BigQuery, and every lakehouse on object storage. You spread work over many machines to store more data, handle more traffic and survive failures, and in exchange you inherit problems a single machine never has: networks drop messages, machines fail independently, and clocks disagree.
This section explains the handful of ideas behind those trade-offs. Once they click, the documentation of most data tools reads very differently.
Some parts break, others don't
A node can be dead, slow or just unreachable, and from the outside these look the same.
Nodes only exchange messages
Every message can be delayed, duplicated or lost. Agreement has to be built on top.
"Now" depends on who you ask
Ordering events across machines needs care, not just timestamps.
Why go distributed: scaling up vs scaling out
When one machine can’t keep up, you have two options. Scale up (vertical scaling): move to a bigger machine with more CPU, memory and disk. Scale out (horizontal scaling): add more ordinary machines and split the data and work between them, so the group behaves as one cluster.
- Both sides start the same: one machine holds all the data and does all the work.
- Scale up: move to a bigger machine. No code changes, no new failure modes. This is the right first move surprisingly often.
- But there is a ceiling: the biggest instance on the market. Price rises faster than capacity near the top, and it is still one machine that can fail.
- Scale out: split the data across three nodes by key range. A coordinator (or a smart client) knows which node holds which keys.
- Need more? Add a node and rebalance: some keys move to node-4 and each node now holds a quarter. Capacity grows in small steps with no hard ceiling.
- A query fans out: every node counts its part in parallel and the coordinator merges the partial results. This is how Spark, Trino and most warehouses work.
| Scale up (vertical) | Scale out (horizontal) | |
|---|---|---|
| How | Bigger CPU, RAM, disk on one machine | More machines, data and work split between them |
| Limit | The largest machine available | Coordination and network costs, not hardware |
| Failure | One machine down means everything down | A node can fail; replicas on other nodes take over |
| Complexity | Low: same software, same queries | High: sharding, rebalancing, network partitions, consistency trade-offs |
| Examples | A larger Postgres instance, a bigger single-node DuckDB box | Kafka brokers, Spark executors, Cassandra rings, Snowflake/BigQuery compute |
A practical rule: scale up while you can, because a single machine is simpler and today’s large instances are very large. Scale out when the data or traffic outgrows the biggest sensible machine, or when you need the system to survive a machine failure.
What’s in this section
Spreading the data
- Replication: copies on several machines; leaders, followers, sync vs async and replication lag.
- Sharding: splitting data across machines; range vs hash partitioning and choosing a shard key.
- Indexing: finding rows without scanning everything; B-trees, index types and local vs global indexes.
Keeping copies correct
- CAP: what a replicated database must give up when the network splits.
- PACELC: the everyday trade-off between latency and consistency.
- Consistency vs Eventual Consistency: consistency models, quorums and conflict resolution.
- Clock: why timestamps lie, and Lamport, vector and hybrid clocks.
Changing data safely
- ACID: transaction guarantees and isolation levels inside one database.
- Distributed Transactions: two-phase commit, sagas and the outbox pattern across machines.
- Normalization or Denormalization: store each fact once, or copy it where it is read.
Processing data
- Batch and Stream Processing: process data in scheduled chunks or event by event.
A good reading order for newcomers: replication, sharding, CAP, consistency, then the rest in any order.
Learning Resources
Books
- Designing Data-Intensive Applications (2nd ed.), Martin Kleppmann and Chris Riccomini: the standard reference for everything in this section
- Distributed Systems, Maarten van Steen and Andrew Tanenbaum: free digital edition
- Designing Distributed Systems, Brendan Burns: patterns for containerized systems
- Database Internals, Alex Petrov
Courses
- Distributed Systems lecture series, Martin Kleppmann (University of Cambridge): short, clear videos
- MIT 6.5840 (formerly 6.824) Distributed Systems: labs build Raft and a sharded key-value store; 2020 lecture videos
Articles
- Fallacies of distributed computing explained, Arnon Rotem-Gal-Oz
- Jepsen analyses: how real databases behave when things go wrong