Parquet
Apache Parquet is an open, columnar file format. It’s the default storage format of the modern data stack: Spark, Trino, DuckDB, ClickHouse, Snowflake and BigQuery all read it, and lakehouse table formats (Iceberg, Delta Lake, Hudi) are, underneath, folders of Parquet files plus metadata.
Two ideas make it fast: store columns, not rows, and write statistics so readers can skip data.
Idea 1: store columns, not rows
| id | user | city | amount |
|---|---|---|---|
| 1 | amy | NYC | 250 |
| 2 | sam | NYC | 120 |
| 3 | ray | NYC | 450 |
| 4 | max | TOR | 980 |
| 5 | eva | TOR | 600 |
| 6 | leo | CHI | 1200 |
- 6 orders, 4 columns. The query needs only one column: amount.
- Row layout: each row's values sit next to each other on disk. Great for "fetch order #4", the OLTP case.
- Columnar layout: all ids, then all users, then all cities, then all amounts.
- The row store must read everything: amounts are interleaved with columns we don't need.
- The column store reads only the amount block: 4× less I/O here. A real table with 200 columns? Often 50–100×.
- Similar values now sit together, so they compress: run-length encoding turns NYC,NYC,NYC into NYC×3. Add dictionary encoding + zstd and files are often several times smaller than the same CSV.
Idea 2: statistics let you skip data
A Parquet file isn’t one big blob. Rows are cut into row groups (commonly 64–128 MB; Spark and parquet-java default to 128 MB, DuckDB to ~122k rows). Inside each row group, every column is a column chunk, split into pages. The footer stores the schema plus statistics (min / max / null count) for every column chunk, so a reader can decide what to skip before touching the data.
- The file: two row groups, each holding four column chunks, plus a footer.
- The reader opens the footer first: a few KB, read with one request.
- Row group 0 has max amount 450, so no row can match
amount > 500. It is never read. - In row group 1, only the id and amount chunks are fetched (column pruning). 2 of 8 chunks read.
This is why sorting or clustering data before writing matters: if
amountis sorted, min/max ranges don’t overlap and the engine skips almost everything. Iceberg and Delta lift these same stats up to the table level to skip whole files.
Encodings and compression
| Technique | What it does | Works best on |
|---|---|---|
| Dictionary | Replace values with small integer codes | Low-cardinality strings (city, status) |
| RLE / bit-packing | Store value × count |
Repeated or sorted values, dictionary codes |
| Delta | Store differences between neighbors | Timestamps, increasing IDs |
| zstd / snappy | General compression per page | Everything, on top of the above |
When not to use Parquet
- Row-at-a-time writes or updates: Parquet files are immutable. Use a database, or a table format (Iceberg / Delta) that rewrites files for you.
- Tiny files: thousands of 10 KB files kill performance. Aim for 100 MB – 1 GB per file.
- Message passing between services: use Avro or Protobuf.
Try it
import duckdb
duckdb.sql("COPY (SELECT * FROM 'orders.csv') TO 'orders.parquet' (FORMAT parquet, COMPRESSION zstd)")
duckdb.sql("SELECT * FROM parquet_metadata('orders.parquet')").show() # row groups + min/max stats
duckdb.sql("SELECT SUM(amount) FROM 'orders.parquet' WHERE amount > 500").show()Learning Resources
- Apache Parquet documentation
- Dremel: Interactive Analysis of Web-Scale Datasets: the paper behind nested columnar storage
- DuckDB: Querying Parquet
- See also: CSV, JSON