$ parquet-tools inspect orders.parquet

Parquet

Apache Parquet is an open, columnar file format. It’s the default storage format of the modern data stack: Spark, Trino, DuckDB, ClickHouse, Snowflake and BigQuery all read it, and lakehouse table formats (Iceberg, Delta Lake, Hudi) are, underneath, folders of Parquet files plus metadata.

Two ideas make it fast: store columns, not rows, and write statistics so readers can skip data.

Idea 1: store columns, not rows

$ SELECT SUM(amount) FROM orders
Same table, two layouts on disk
idusercityamount
1amyNYC250
2samNYC120
3rayNYC450
4maxTOR980
5evaTOR600
6leoCHI1200
SELECT SUM(amount) FROM orders;
amount (needed)other columnsread from disk
Row layout · CSV, PostgreSQL heap
Columnar layout · Parquet, ORC
  1. 6 orders, 4 columns. The query needs only one column: amount.
  2. Row layout: each row's values sit next to each other on disk. Great for "fetch order #4", the OLTP case.
  3. Columnar layout: all ids, then all users, then all cities, then all amounts.
  4. The row store must read everything: amounts are interleaved with columns we don't need.
  5. The column store reads only the amount block: 4× less I/O here. A real table with 200 columns? Often 50–100×.
  6. Similar values now sit together, so they compress: run-length encoding turns NYC,NYC,NYC into NYC×3. Add dictionary encoding + zstd and files are often several times smaller than the same CSV.

Idea 2: statistics let you skip data

A Parquet file isn’t one big blob. Rows are cut into row groups (commonly 64–128 MB; Spark and parquet-java default to 128 MB, DuckDB to ~122k rows). Inside each row group, every column is a column chunk, split into pages. The footer stores the schema plus statistics (min / max / null count) for every column chunk, so a reader can decide what to skip before touching the data.

$ SELECT id, amount FROM orders WHERE amount > 500
Predicate pushdown with min/max stats
orders.parquet Row group 0 · rows 1–3 id user city amount Row group 1 · rows 4–6 id user city amount Footer schema · RG0 amount: min 120, max 450 · RG1 amount: min 600, max 1200 skipped · max 450 ≤ 500 2 / 8 chunks read
  1. The file: two row groups, each holding four column chunks, plus a footer.
  2. The reader opens the footer first: a few KB, read with one request.
  3. Row group 0 has max amount 450, so no row can match amount > 500. It is never read.
  4. In row group 1, only the id and amount chunks are fetched (column pruning). 2 of 8 chunks read.

This is why sorting or clustering data before writing matters: if amount is sorted, min/max ranges don’t overlap and the engine skips almost everything. Iceberg and Delta lift these same stats up to the table level to skip whole files.

Encodings and compression

Technique What it does Works best on
Dictionary Replace values with small integer codes Low-cardinality strings (city, status)
RLE / bit-packing Store value × count Repeated or sorted values, dictionary codes
Delta Store differences between neighbors Timestamps, increasing IDs
zstd / snappy General compression per page Everything, on top of the above

When not to use Parquet

  • Row-at-a-time writes or updates: Parquet files are immutable. Use a database, or a table format (Iceberg / Delta) that rewrites files for you.
  • Tiny files: thousands of 10 KB files kill performance. Aim for 100 MB – 1 GB per file.
  • Message passing between services: use Avro or Protobuf.

Try it

import duckdb
duckdb.sql("COPY (SELECT * FROM 'orders.csv') TO 'orders.parquet' (FORMAT parquet, COMPRESSION zstd)")
duckdb.sql("SELECT * FROM parquet_metadata('orders.parquet')").show()   # row groups + min/max stats
duckdb.sql("SELECT SUM(amount) FROM 'orders.parquet' WHERE amount > 500").show()

Learning Resources