DataBurst Data Engineering Wiki

Columnar

Mon, 01 Jan 0001 00:00:00 +0000

$ clickhouse-client -q "SELECT city, sum(amount) FROM orders GROUP BY city"

Columnar and Wide-Column Databases

A columnar (column-oriented) database stores each column of a table separately on disk, so a query that needs 3 of 200 columns reads only those 3, and similar values packed together compress 5 to 10 times. This makes columnar engines such as ClickHouse, Apache Druid, Apache Pinot, Snowflake and BigQuery the standard for analytics: scans and aggregations over billions of rows.

TCP/IP

Mon, 01 Jan 0001 00:00:00 +0000

$ man 7 tcp # RFC 791 (IP) · RFC 9293 (TCP)

TCP/IP

TCP/IP is the protocol suite the internet runs on. IP (Internet Protocol) moves individual packets from one machine to another by address, with no promises: packets can be lost, duplicated or reordered. TCP (Transmission Control Protocol) runs on top and turns that unreliable packet service into a reliable, ordered byte stream between two programs. Your database driver, HTTP client and Kafka producer all sit on a TCP connection.

Document Based

Mon, 01 Jan 0001 00:00:00 +0000

Document Databases

A document database stores each record as a self-contained document, usually JSON or a binary form of it (MongoDB’s BSON), grouped into collections. A document can nest objects and arrays, and two documents in the same collection may have different fields. The idea: data that the application reads together is stored together, so one lookup returns a whole order with its items and shipping address, without joins.

Key-Value

Mon, 01 Jan 0001 00:00:00 +0000

Key-Value Databases

A key-value store is the simplest database model: you put(key, value) and get(key), and the database treats the value as an opaque blob or a small data structure. Because every operation names exactly one key, the store can hash the key to find the right machine and answer in well under a millisecond. Caches, user sessions, feature flags, rate limiters and shopping carts are classic uses.

How it works

Lookup by key only. There is no “find all users in Tehran” unless you build that index yourself. Design keys around how you read: session:8f3a, cart:u-42, rate:ip:10.0.0.7:2026-11-16T09:30.
Partitioning by hash. In a cluster, hash(key) decides which node owns the key. Redis Cluster uses 16,384 hash slots; DynamoDB hashes the partition key. Adding nodes moves only some keys. See Sharding.
In memory or on disk. Redis and Valkey keep everything in RAM (with optional snapshots and an append-only file for durability). DynamoDB, RocksDB and FoundationDB store data on SSD and are fully durable.
Expiry. A TTL deletes a key automatically after a time, which is what makes key-value stores natural caches.

Popular key-value stores

Store	Storage	Notes
Redis	In memory	Rich value types: strings, hashes, lists, sets, sorted sets, streams. Licensing changed in 2024 (RSAL/SSPL); Redis 8 (2025) added AGPLv3 as an open-source option.
Valkey	In memory	BSD-licensed fork of Redis 7.2 under the Linux Foundation, created in 2024. Drop-in compatible; used by AWS ElastiCache and Google Memorystore.
Amazon DynamoDB	Durable, managed	Key-value and document. Partition key plus optional sort key, single-digit-millisecond latency at any scale, pay per request or provisioned capacity.
RocksDB	Embedded, on disk	An LSM-tree library, not a server. The storage engine inside Kafka Streams, Flink state, CockroachDB and many others.
etcd	Durable, Raft-replicated	Small, strongly consistent store for configuration and coordination (Kubernetes state). Not for bulk data.

When to use one

Good fit: caching expensive query results, sessions, counters, leaderboards (Redis sorted sets), idempotency keys, deduplication in pipelines, lookups at very high request rates.
Poor fit: anything that needs ad hoc queries, joins, or range scans over values. If you keep adding secondary indexes by hand, you want a different database.

Common mistakes

Treating a cache as the source of truth. Redis with default settings can lose recent writes on a crash. Keep the real data in a durable database.
Hot keys. One key (a global counter, a celebrity’s profile) receives most traffic and overloads a single node or DynamoDB partition. Spread it (counter:{shard}) or cache locally.
Huge values. Multi-megabyte values block single-threaded Redis and waste network. Store large blobs in object storage and keep a pointer.
No TTL on cache keys, so memory fills up and eviction starts dropping keys you needed.

Learning Resources

Redis documentation and Redis University
Valkey documentation
Amazon DynamoDB Developer Guide
The DynamoDB Book, Alex DeBrie: single-table design and access-pattern modeling
Dynamo: Amazon’s Highly Available Key-value Store (2007)
Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service (USENIX ATC 2022)

Time Series

Mon, 01 Jan 0001 00:00:00 +0000

Time-Series Databases

A time-series database (TSDB) stores measurements indexed by time: CPU usage every 15 seconds, the temperature of a sensor every minute, a stock price on every trade. Each point is a timestamp, a value, and a set of labels (tags) that identify the series, such as host=web-1, region=eu. Writes are almost always appends of recent data, and queries ask for ranges and aggregates (“average latency per 5 minutes over the last day”), so TSDBs optimize for exactly that.

Vector

Mon, 01 Jan 0001 00:00:00 +0000

$ SELECT name FROM products ORDER BY embedding <=> :query LIMIT 3;

Vector Databases

A vector database stores embeddings, lists of numbers (typically 384 to 3,072 of them) that a machine-learning model produces for a piece of text, an image or a product, and answers one question fast: which stored vectors are closest to this one? Items with similar meaning end up close together, so “closest vectors” means “most similar items”. This powers semantic search, recommendations, deduplication and retrieval-augmented generation (RAG), where an LLM is given the most relevant document chunks before it answers.

Graph

Mon, 01 Jan 0001 00:00:00 +0000

Graph Databases

A graph database stores data as nodes (people, accounts, products) and relationships between them (FOLLOWS, PAID, BOUGHT), and both can carry properties. Its strength is queries that walk many hops: “friends of friends who bought this”, “accounts within 4 transfers of a known fraudster”, “which services break if this one goes down”. In a relational database each hop is another join; in a native graph database each node points directly at its neighbours, so a hop costs roughly the same no matter how big the whole graph is.

NewSQL

Mon, 01 Jan 0001 00:00:00 +0000

NewSQL (Distributed SQL)

NewSQL, now usually called distributed SQL, describes relational databases that keep SQL and ACID transactions but spread data across many machines like NoSQL systems do. You talk to it like PostgreSQL or MySQL, while underneath the tables are split into ranges, every range is replicated with a consensus protocol (Paxos or Raft), and a node or even a whole datacenter can fail without losing committed data. The trade-off is latency: a write must reach a majority of replicas before it commits.

Avro

Mon, 01 Jan 0001 00:00:00 +0000

$ avro-tools getschema orders.avro

Avro

Apache Avro is a binary serialization format whose schema is written in JSON. A record is encoded as just its values, in schema order, with no field names or tags, so Avro data is very compact. The catch: you can only decode the bytes if you know the exact schema they were written with. Avro’s key feature is what it does with that: it resolves the writer’s schema against the reader’s schema, so old and new versions of a producer and consumer can keep talking.

CSV

Mon, 01 Jan 0001 00:00:00 +0000

CSV

CSV (comma-separated values) is a plain-text format for tables: one record per line, fields separated by a delimiter (usually a comma), and an optional header row with column names. Every spreadsheet, database and programming language can read and write it, which is why it is still the most common way to hand data from one system to another.

order_id,city,amount,created_at
1,Tehran,250,2026-11-01T10:15:00Z
2,Shiraz,120,2026-11-01T10:17:42Z
3,"Tehran, Iran",450,2026-11-01T10:20:05Z

How it works

There is no single CSV standard, but RFC 4180 describes what most tools agree on:

DNS

Mon, 01 Jan 0001 00:00:00 +0000

$ dig www.example.com # RFC 1034 / 1035, 1987

DNS

The Domain Name System is the internet’s phone book: it turns a name like www.example.com into an IP address like 203.0.113.10. It is not one database but a distributed hierarchy: the root servers know who runs .com, the .com servers know who runs example.com, and that domain’s own servers hold its records. Answers are cached everywhere along the way for a time set by the record’s TTL.

Firewalls

Mon, 01 Jan 0001 00:00:00 +0000

$ sudo nft list ruleset

Firewalls

A firewall checks each network packet or connection against a list of rules and decides to allow it or drop it. Rules match on things like source and destination IP, port, protocol and, in a stateful firewall, whether the packet belongs to a connection that was already allowed. Firewalls run on a single host (Linux nftables, Windows Defender Firewall), at the network edge (appliances from Palo Alto, Fortinet, Cisco), and in the cloud as security groups and network ACLs.

HTTP

Mon, 01 Jan 0001 00:00:00 +0000

$ curl -i https://api.shop.com/orders/42 # RFC 9110

HTTP

HTTP (Hypertext Transfer Protocol) is a request/response protocol: a client sends a request naming a method and a resource, and the server sends back one response with a status code, headers and an optional body. It started as the protocol for web pages and is now the default for APIs, object storage (every S3 GET and PUT is an HTTP call), webhooks and most SaaS data sources your pipelines pull from.

HTTPS

Mon, 01 Jan 0001 00:00:00 +0000

$ openssl s_client -connect shop.com:443 # RFC 8446 (TLS 1.3)

HTTPS

HTTPS is plain HTTP sent inside a TLS (Transport Layer Security) connection, usually on port 443. TLS gives three guarantees: confidentiality (nobody on the path can read the data), integrity (nobody can change it unnoticed) and authentication (you are really talking to shop.com, proven by a certificate). “SSL” is the name of TLS’s long-dead predecessor; every SSL version and TLS 1.0 and 1.1 are deprecated. Use TLS 1.3, with 1.2 as a fallback.

JSON

Mon, 01 Jan 0001 00:00:00 +0000

JSON

JSON (JavaScript Object Notation) is a text format for structured data built from two shapes: objects ({"key": value}) and arrays ([v1, v2]), with strings, numbers, true, false and null as values. It is defined by RFC 8259, it’s what nearly every REST API returns, and it’s the usual shape of application logs and webhook events.

{
  "order_id": 7,
  "user": {"id": 42, "name": "Ali"},
  "city": "Tehran",
  "items": [
    {"sku": "TEA-01", "qty": 2, "price": 3.5},
    {"sku": "CUP-07", "qty": 1, "price": 12.0}
  ],
  "paid": true,
  "created_at": "2026-11-16T09:30:00Z"
}

How it differs from CSV

JSON is nested and self-describing: every record carries its own field names, and a field can hold another object or a list. That makes it a natural fit for events whose shape varies, and an awkward fit for tables, because an order with three items does not map to one row.

Kappa

Mon, 01 Jan 0001 00:00:00 +0000

$ ./kappa --replay-from earliest

Kappa Architecture

The Kappa architecture handles both real-time and historical processing with a single stream processing pipeline. All events are kept in a durable, replayable log (usually Apache Kafka). The streaming job reads the log and maintains the output tables continuously. When logic changes, you do not run a separate batch job: you start a new version of the streaming job that replays the log from the beginning, let it catch up, and switch readers over. Jay Kreps proposed it in 2014 as a simpler alternative to Lambda.

Lambda

Mon, 01 Jan 0001 00:00:00 +0000

$ ./lambda --batch spark --speed flink

Lambda Architecture

The Lambda architecture processes the same events along two paths: a batch layer that periodically recomputes accurate results from all historical data, and a speed layer that processes new events in real time to cover the gap since the last batch run. A serving layer merges both so queries get answers that are complete and fresh. Nathan Marz described it around 2011, when stream processors were not yet reliable enough to be the only source of truth.

Parquet

Mon, 01 Jan 0001 00:00:00 +0000

$ parquet-tools inspect orders.parquet

Parquet

Apache Parquet is an open, columnar file format. It’s the default storage format of the modern data stack: Spark, Trino, DuckDB, ClickHouse, Snowflake and BigQuery all read it, and lakehouse table formats (Iceberg, Delta Lake, Hudi) are, underneath, folders of Parquet files plus metadata.

Two ideas make it fast: store columns, not rows, and write statistics so readers can skip data.

Idea 1: store columns, not rows

$ SELECT SUM(amount) FROM orders

Same table, two layouts on disk

id	user	city	amount
1	ali	TEH	250
2	sara	TEH	120
3	reza	TEH	450
4	mina	SHZ	980
5	omid	SHZ	600
6	nika	ISF	1200

SELECT SUM(amount) FROM orders;

amount (needed)other columnsread from disk

Row layout · CSV, PostgreSQL heap

Columnar layout · Parquet, ORC

6 orders, 4 columns. The query needs only one column: amount.
Row layout: each row's values sit next to each other on disk. Great for "fetch order #4", the OLTP case.
Columnar layout: all ids, then all users, then all cities, then all amounts.
The row store must read everything: amounts are interleaved with columns we don't need.
The column store reads only the amount block: 4× less I/O here. A real table with 200 columns? Often 50–100×.
Similar values now sit together, so they compress: run-length encoding turns TEH,TEH,TEH into TEH×3. Add dictionary encoding + zstd and files shrink 5–10× vs CSV.

Idea 2: statistics let you skip data

A Parquet file isn’t one big blob. Rows are cut into row groups (typically ~128 MB). Inside each row group, every column is a column chunk, split into pages. The footer stores the schema plus min / max / null-count for every column chunk.

Protocol Buffers

Mon, 01 Jan 0001 00:00:00 +0000

$ protoc --python_out=. order.proto

Protocol Buffers

Protocol Buffers (Protobuf) is Google’s language-neutral binary serialization format. You describe messages in a .proto file, a compiler (protoc or buf) generates classes for Go, Java, Python, Rust and a dozen other languages, and those classes encode objects into compact bytes. Each field is identified on the wire by a field number, not by its name, which is what lets services upgrade one at a time. Protobuf is the payload format of gRPC and is common on Kafka topics alongside Avro.

SSH

Mon, 01 Jan 0001 00:00:00 +0000

$ ssh deploy@db1.internal # RFC 4251–4254

SSH

SSH (Secure Shell) gives you an encrypted, authenticated connection to a remote machine over an untrusted network, usually on TCP port 22. You use it to get a shell on a server, run a remote command, copy files (scp, sftp, rsync), and tunnel other traffic, such as a PostgreSQL connection to a database that only accepts traffic from inside a private network. OpenSSH is the implementation on almost every Linux and macOS machine, and it ships with Windows.

UDP

Mon, 01 Jan 0001 00:00:00 +0000

$ man 7 udp # RFC 768, 1980

UDP

UDP (User Datagram Protocol) sends independent messages, called datagrams, from one port to another with no connection, no acknowledgments, no retransmission and no ordering. Each datagram either arrives whole or not at all. That makes UDP fast and simple, and it leaves reliability to the application, if the application wants it.

TCP and UDP under the same packet loss

$ send 3 messages, 1 gets lost

TCP and UDP, same network, same loss

Left: TCP. Right: UDP. Each side sends three messages over the same lossy network.
TCP spends a round trip on the handshake before any data. UDP just sends: three datagrams, and #2 is lost.
UDP is already finished. The receiver got #1 and #3. Nothing tells either side that #2 is missing.
TCP sends its three segments. Segment 2 is lost too.
The receiver ACKs only what arrived in order, so the sender retransmits segment 2. Segment 3 waits in a buffer meanwhile.
TCP delivers all three, in order, but later. UDP delivered two, immediately. Which is better depends on the app.

The whole header is 8 bytes

Field	Size	Meaning
Source port	16 bits	Where replies go
Destination port	16 bits	Which program receives it (53 = DNS)
Length	16 bits	Header + data, in bytes
Checksum	16 bits	Detects corruption (mandatory in IPv6)

Compare that with TCP’s 20 to 60 bytes of sequence numbers, ACKs, windows and flags. Because there is no connection, the server keeps no per-client state, which is why one DNS server can answer millions of clients.

VPN

Mon, 01 Jan 0001 00:00:00 +0000

$ wg-quick up wg0

VPN

A VPN (Virtual Private Network) builds an encrypted tunnel across an untrusted network so that a remote machine behaves as if it were plugged into a private one. Your laptop’s packets to an internal address are encrypted, wrapped in an ordinary-looking outer packet addressed to a VPN gateway, and unwrapped there. For a data engineer, that is how you reach a warehouse, a database or an internal Airflow UI that has no public address.

XML

Mon, 01 Jan 0001 00:00:00 +0000

XML

XML (eXtensible Markup Language) is a text format that stores data as a tree of elements wrapped in opening and closing tags, with optional attributes on each tag. It is a W3C standard from 1998, and although JSON replaced it for most new APIs, data engineers still meet it constantly: bank and payment messages (ISO 20022), government and tax feeds, SOAP services, RSS, Office files (.xlsx and .docx are zipped XML) and many legacy exports.