DataBurst Data Engineering Wiki - Roadmap Step-by-Step: https://wiki.databurst.tech/docs/roadmap/ - Introduction: https://wiki.databurst.tech/docs/roadmap/introduction/ - Programming Language: https://wiki.databurst.tech/docs/roadmap/programming-language/ - Python: https://wiki.databurst.tech/docs/roadmap/programming-language/python/ - Java: https://wiki.databurst.tech/docs/roadmap/programming-language/java/ - Scala: https://wiki.databurst.tech/docs/roadmap/programming-language/scala/ - Go: https://wiki.databurst.tech/docs/roadmap/programming-language/go/ - Rust: https://wiki.databurst.tech/docs/roadmap/programming-language/rust/ - Operating System: https://wiki.databurst.tech/docs/roadmap/operating-system/ - Windows: https://wiki.databurst.tech/docs/roadmap/operating-system/windows/ - Linux: https://wiki.databurst.tech/docs/roadmap/operating-system/linux/ - Networking: https://wiki.databurst.tech/docs/roadmap/networking/ - Protocols: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ - TCP/IP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/tcp/ - DNS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/dns/ - HTTP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/http/ - HTTPS: https://wiki.databurst.tech/docs/roadmap/networking/protocols/https/ - SSH: https://wiki.databurst.tech/docs/roadmap/networking/protocols/ssh/ - UDP: https://wiki.databurst.tech/docs/roadmap/networking/protocols/udp/ - Concepts: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ - IP addressing: https://wiki.databurst.tech/docs/roadmap/networking/concepts/ip-addressing/ - DHCP: https://wiki.databurst.tech/docs/roadmap/networking/concepts/dhcp/ - Firewalls: https://wiki.databurst.tech/docs/roadmap/networking/concepts/firewalls/ - VPN: https://wiki.databurst.tech/docs/roadmap/networking/concepts/vpn/ - General Software Engineering Skills: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/ - OAuth: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/oauth/ - JWT: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/jwt/ - GraphQL: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/graphql/ - REST: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/rest/ - gRPC: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/grpc/ - Caching: https://wiki.databurst.tech/docs/roadmap/general-software-engineering-skills/caching/ - Web Frameworks and API Development: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/ - Python: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/python/ - Go: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/go/ - Java: https://wiki.databurst.tech/docs/roadmap/web-frameworks-and-api-development/java/ - Version Control System: https://wiki.databurst.tech/docs/roadmap/version-control-system/ - git: https://wiki.databurst.tech/docs/roadmap/version-control-system/git/ - Version Control System Hosting: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/ - Github: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/github/ - Gitlab: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/gitlab/ - BitBucket: https://wiki.databurst.tech/docs/roadmap/version-control-system-hosting/bitbucket/ - Distributed Systems Concepts: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/ - CAP: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/cap/ - PACELC: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/pacelc/ - Sharding: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/sharding/ - Clock: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/clock/ - ACID: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/acid/ - Replication: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/replication/ - Distributed Transactions: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/transaction/ - Indexing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/indexing/ - Batch and Stream Processing: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/batch-and-stream-processing/ - Consistency vs Eventual Consistency: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/consistency-vs-eventual-consistency/ - Normalization or Denormalization: https://wiki.databurst.tech/docs/roadmap/distributed-systems-concepts/normalization-or-denormalization/ - SQL Fundamentals: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/ - Basic SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/basic-sql/ - Advanced SQL: https://wiki.databurst.tech/docs/roadmap/sql-fundamentals/advanced-sql/ - Databases: https://wiki.databurst.tech/docs/roadmap/databases/ - Relational: https://wiki.databurst.tech/docs/roadmap/databases/relational/ - NoSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/ - Columnar: https://wiki.databurst.tech/docs/roadmap/databases/nosql/columnar/ - Document Based: https://wiki.databurst.tech/docs/roadmap/databases/nosql/document-based/ - Key-Value: https://wiki.databurst.tech/docs/roadmap/databases/nosql/key-value/ - Time Series: https://wiki.databurst.tech/docs/roadmap/databases/nosql/time-series/ - Vector: https://wiki.databurst.tech/docs/roadmap/databases/nosql/vector/ - Graph: https://wiki.databurst.tech/docs/roadmap/databases/nosql/graph/ - NewSQL: https://wiki.databurst.tech/docs/roadmap/databases/nosql/newsql/ - Data Modeling: https://wiki.databurst.tech/docs/roadmap/data-modeling/ - Star Schema (Ralph Kimball): https://wiki.databurst.tech/docs/roadmap/data-modeling/star-schema/ - Snowflake: https://wiki.databurst.tech/docs/roadmap/data-modeling/snowflake/ - Data Vault: https://wiki.databurst.tech/docs/roadmap/data-modeling/data-vault/ - Data Management Architecture: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/ - Relational Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/relational-data-warehouse/ - Data Lake: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lake/ - Modern Data Warehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/modern-data-warehouse/ - Data Fabric: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-fabric/ - Data Mart: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mart/ - Data Lakehouse: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-lakehouse/ - Data Mesh: https://wiki.databurst.tech/docs/roadmap/data-management-architecture/data-mesh/ - Data Integration: https://wiki.databurst.tech/docs/roadmap/data-integration/ - ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/etl/ - ELT: https://wiki.databurst.tech/docs/roadmap/data-integration/elt/ - CDC: https://wiki.databurst.tech/docs/roadmap/data-integration/cdc/ - Reverse ETL: https://wiki.databurst.tech/docs/roadmap/data-integration/reverse-etl/ - Storage: https://wiki.databurst.tech/docs/roadmap/storage/ - Block Storage: https://wiki.databurst.tech/docs/roadmap/storage/block-storage/ - Object Storage: https://wiki.databurst.tech/docs/roadmap/storage/object-storage/ - File Formats and Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/ - File Formats: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/ - CSV: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/csv/ - JSON: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/json/ - Parquet: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/parquet/ - XML: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/file-formats/xml/ - Serialization: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/ - Avro: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/avro/ - Protocol Buffers: https://wiki.databurst.tech/docs/roadmap/file-formats-and-serialization/serialization/protobuf/ - Query Engine: https://wiki.databurst.tech/docs/roadmap/query-engine/ - Apache Hive: https://wiki.databurst.tech/docs/roadmap/query-engine/hive/ - SparkSQL: https://wiki.databurst.tech/docs/roadmap/query-engine/sparksql/ - Trino: https://wiki.databurst.tech/docs/roadmap/query-engine/trino/ - Presto: https://wiki.databurst.tech/docs/roadmap/query-engine/presto/ - Data Architecture: https://wiki.databurst.tech/docs/roadmap/data-architecture/ - Kappa: https://wiki.databurst.tech/docs/roadmap/data-architecture/kappa/ - Lambda: https://wiki.databurst.tech/docs/roadmap/data-architecture/lambda/ - Data Processing: https://wiki.databurst.tech/docs/roadmap/data-processing/ - Pipeline Orchestration: https://wiki.databurst.tech/docs/roadmap/pipeline-orchestration/ - Messaging and Streaming Platform: https://wiki.databurst.tech/docs/roadmap/messaging-and-streaming-platform/ - Data Governance: https://wiki.databurst.tech/docs/roadmap/data-governance/ - Data Quality: https://wiki.databurst.tech/docs/roadmap/data-governance/data-quality/ - Data Visualization: https://wiki.databurst.tech/docs/roadmap/data-visualization/ - Containerization: https://wiki.databurst.tech/docs/roadmap/containerization/ - Docker: https://wiki.databurst.tech/docs/roadmap/containerization/docker/ - Podman: https://wiki.databurst.tech/docs/roadmap/containerization/podman/ - containerd: https://wiki.databurst.tech/docs/roadmap/containerization/containerd/ - Container Orchestration: https://wiki.databurst.tech/docs/roadmap/container-orchestration/ - Kubernetes: https://wiki.databurst.tech/docs/roadmap/container-orchestration/kubernetes/ - Docker Swarm: https://wiki.databurst.tech/docs/roadmap/container-orchestration/docker-swarm/ - Apache Mesos: https://wiki.databurst.tech/docs/roadmap/container-orchestration/apache-mesos/ - CI/CD: https://wiki.databurst.tech/docs/roadmap/cicd/ - DataOps: https://wiki.databurst.tech/docs/roadmap/cicd/dataops/ - Infrastructure as Code: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/ - Configuration Management: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/configuration-management/ - Infrastructure Provisioning: https://wiki.databurst.tech/docs/roadmap/infrastructure-as-code/infrastructure-provisioning/ - Observability: https://wiki.databurst.tech/docs/roadmap/observability/ - Logging: https://wiki.databurst.tech/docs/roadmap/observability/logging/ - Monitoring: https://wiki.databurst.tech/docs/roadmap/observability/monitoring/ - Tracing: https://wiki.databurst.tech/docs/roadmap/observability/tracing/ - Glossary: https://wiki.databurst.tech/docs/roadmap/glossary/
$ man sharding # a.k.a. partitioning, horizontal scaling
# Sharding **Sharding** splits one large dataset into smaller pieces (shards, also called partitions) and puts each piece on a different machine. Every row lives on exactly one shard, so together the machines can store more data and absorb more writes than any single server. The same idea is called *partitioning* in Kafka, Cassandra, Spark and most data tools. Sharding is **horizontal scaling** (add machines). Its alternative is **vertical scaling** (buy a bigger machine), which is simpler and should usually come first. ## The whole game is the shard key A **shard key** (or partition key) is the column that decides where a row goes. Two classic ways to map a key to a shard behave very differently. {{< fig kicker="$ ./route --key user_id --shards 3" title="Range vs hash partitioning on the same keys" >}} user_id: 7 18 29 44 63 88 RANGE by key order HASH by hash(key) SHARD 1 · 1–33 SHARD 2 · 34–66 SHARD 3 · 67–100 SHARD 1 · h % 3 = 0 SHARD 2 · h % 3 = 1 SHARD 3 · h % 3 = 2 7 18 29 44 63 88 WHERE user_id BETWEEN 20 AND 50 → 2 of 3 shards 96 97 98 99 HOT: every new signup 7 63 18 44 29 88 98 96 99 97 WHERE user_id BETWEEN 20 AND 50 → ask all 3 shards
  1. Six users, three machines. The same keys will be placed two ways: by range and by hash.
  2. Range partitioning: each shard owns a contiguous key range, like volumes of an encyclopedia.
  3. Range queries are cheap: users 20 to 50 live on 2 of 3 shards, and the rest are never touched.
  4. But keys that grow over time (auto-increment IDs, timestamps) all land on the last shard. One machine takes every write: a hot spot.
  5. Hash partitioning: hash(user_id) % 3 scatters keys pseudo-randomly, so new signups spread evenly across all shards.
  6. The price: neighbors are no longer together. The same range query must ask every shard and merge the answers (scatter-gather).
{{< /fig >}} ## Partitioning strategies | Strategy | How a key is placed | Good at | Bad at | Used by | |---|---|---|---|---| | **Range** | Sorted key ranges per shard | Range scans, ordered reads | Hot spots on increasing keys | HBase, Bigtable, CockroachDB, TiDB, MongoDB ranged shards | | **Hash** | `hash(key)` decides the shard | Even spread of reads and writes | Range queries become scatter-gather | Cassandra, DynamoDB, Kafka (default partitioner), Redis Cluster | | **Consistent hashing** | Keys and nodes on a hash ring | Adding a node moves only ~1/N of keys | Uneven without virtual nodes | Dynamo-style stores, Cassandra vnodes, CDNs | | **Directory / lookup** | A mapping table says where each key lives | Moving single tenants, uneven tenants | The directory is one more service to keep available | Vitess lookup vindexes, many multi-tenant SaaS setups | Plain `hash(key) % N` has a nasty property: change N from 3 to 4 and almost every key moves. Real systems avoid that with consistent hashing or with **many more partitions than nodes** (for example 256 partitions on 4 nodes) so rebalancing just moves whole partitions. ## Choosing a shard key A good shard key has **high cardinality**, spreads **writes evenly**, and matches **how you query**: - `user_id` or `tenant_id` for a SaaS app: most queries are "everything for this user", so they hit one shard. - `country` is a bad key: a handful of values, and one country may hold half your users. - `created_at` alone is a bad key for range sharding: today's writes all go to one shard. Combine it with a hashed prefix, or bucket by `hash(user_id)` first. The same rule applies to data engineering tools: Kafka keys decide which partition holds an entity's events (and therefore their order), and Spark or Iceberg partition columns decide which files a query can skip. ## Sharding and replication together Sharding and [replication]({{< relref "/docs/roadmap/distributed-systems-concepts/replication" >}}) solve different problems and are almost always combined. | | Replication | Sharding | |---|---|---| | What each node holds | A full copy | A slice | | Scales | Reads, availability | Storage, writes, and reads | | A node dies | Another copy serves the data | That slice is gone unless the shard is also replicated | | Typical layout | 1 leader + 2 followers | N shards, each with its own leader + followers | ## What sharding costs you - **Cross-shard queries**: joins and aggregations across shards need scatter-gather or data shuffling. - **Cross-shard transactions**: atomically updating rows on two shards needs a protocol like two-phase commit. See [Distributed Transactions]({{< relref "/docs/roadmap/distributed-systems-concepts/transaction" >}}). - **Secondary indexes**: an index on `email` when you shard by `user_id` is either local (every shard is searched) or global (writes touch another shard). See [Indexing]({{< relref "/docs/roadmap/distributed-systems-concepts/indexing" >}}). - **Operations**: rebalancing, resharding and backups across many machines. ## Common mistakes - **Sharding too early.** A single PostgreSQL server with good indexes, table partitioning and read replicas handles terabytes. Shard when one machine truly cannot keep up. - **A monotonic shard key with range partitioning.** Creates a permanent hot shard. - **Ignoring hot keys.** Even with hashing, one celebrity user or one huge tenant can overload a shard. Split or salt those keys. - **`% N` routing in application code.** Makes adding a shard a full data migration. - **Picking a key that does not match queries.** If most queries do not include the shard key, every query is scatter-gather. ## Learning Resources ### Books - *Designing Data-Intensive Applications* (2nd ed.), Martin Kleppmann and Chris Riccomini: the partitioning (sharding) chapter ### Docs and papers - [Dynamo: Amazon's Highly Available Key-value Store](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf): consistent hashing and virtual nodes in practice - [Vitess documentation: Sharding](https://vitess.io/docs/reference/features/sharding/): how YouTube-scale MySQL is sharded - [Citus: Choosing the distribution column](https://docs.citusdata.com/en/stable/sharding/data_modeling.html): shard key design for PostgreSQL - [Understanding Database Sharding](https://www.digitalocean.com/community/tutorials/understanding-database-sharding), DigitalOcean ### Videos - [What is Database Sharding?](https://www.youtube.com/watch?v=5faMjKuB9bc) - [Sharding | Horizontal Scaling | System Design Interview Basics](https://www.youtube.com/watch?v=0DzRdQ-sOTg)