$ spark-submit count_orders_by_city.py

Data Processing

Data processing is the step where raw data becomes something useful: filtered, cleaned, joined, aggregated and written somewhere else. When the data fits on one machine, a SQL query or a Python script is enough. When it doesn’t, or when it never stops arriving, you need an engine that splits the work across many cores or machines and survives one of them dying halfway through.

Almost every distributed engine, from Google’s MapReduce (2004) and Hadoop to Apache Spark and Flink today, follows the same shape: split the input, process the pieces in parallel, shuffle by key, combine the results.

Split, map, shuffle, reduce

$ SELECT city, COUNT(*) FROM orders GROUP BY city
Counting orders per city on three machines
INPUTMAPSHUFFLEREDUCEOUTPUT SPLIT 1NYC NYC TOR SPLIT 2NYC CHI TOR SPLIT 3NYC NYC TOR MAP 1NYC:2 TOR:1 MAP 2NYC:1 CHI:1 TOR:1 MAP 3NYC:2 TOR:1 REDUCE 1key NYC REDUCE 2keys TOR, CHI 2+1+2 = 5 TOR 3 · CHI 1 NYC 5TOR 3CHI 1 MAP 2 · NODE DIED re-run split 2 only Other tasks keep their results. Only the lost task is re-run on another node.
  1. The input (a folder of Parquet files, a Kafka topic) is cut into splits, and each machine gets some of them.
  2. Map: each task reads its split and emits (city, count) pairs. It pre-aggregates locally, so it sends NYC:2, not NYC:1 twice.
  3. Shuffle: pairs are routed by hash of the key, so every NYC goes to reducer 1. This is the expensive step: data crosses the network and often hits disk.
  4. Reduce: each reducer combines the values for its keys and writes the result.
  5. A machine dies. The engine knows which split that task read, so it re-runs only that task elsewhere. This is how a 1,000-node job survives failures.

Spark calls the parts between shuffles stages and plans them from your SQL or DataFrame code. Flink does the same continuously: a keyed stream is a never-ending shuffle, and reducers keep running totals in state.

Batch, streaming and in between

  • Batch: process a bounded dataset (yesterday’s orders) and finish. Simple, cheap, easy to re-run.
  • Streaming: process an unbounded stream event by event and keep results up to date. Fresh, but needs state, checkpoints and handling of late data.
  • Micro-batch: small batches every few seconds (Spark Structured Streaming).

See Batch and Stream Processing for a side-by-side simulation.

Engines in 2026

Engine Type Best at
Apache Spark Batch + micro-batch streaming Large ETL, lakehouse jobs, ML prep. SQL, Python, Scala. Spark 4 (2025) added a VARIANT type for semi-structured data and a native XML reader.
Apache Flink Streaming first (batch too) Low-latency stateful streaming, event time, exactly-once state
Apache Beam Unified API Write once, run on Dataflow, Flink or Spark
Kafka Streams Streaming library Stream processing inside a Java service, no separate cluster
Warehouse SQL + dbt Batch (ELT) Transformations where the data already lives: Snowflake, BigQuery, Databricks SQL, ClickHouse
DuckDB, Polars Single-node, vectorized Anything up to hundreds of GB on one machine, fast and cheap
Ray, Dask Python-native distributed ML pipelines and Python workloads that outgrow one machine
RisingWave, Materialize Streaming SQL databases Continuously updated SQL views over streams

What this table leaves out on purpose: Kafka itself moves data but doesn’t process it (see Messaging and Streaming Platforms); Apache Arrow is an in-memory columnar format that many of these engines use internally; Hadoop MapReduce, Storm and Samza are legacy and rarely chosen for new work.

How to choose

  1. Does it fit on one machine? Data under a few hundred GB: try DuckDB or Polars first. It’s often faster than a cluster and costs nothing to run.
  2. Is the data already in a warehouse? Transform it there with SQL and dbt.
  3. Does it need results in seconds? Use Flink, Kafka Streams or a streaming database.
  4. Otherwise, large-scale batch on files: Spark.

Common mistakes

  • Data skew. One key (a huge customer, city = NULL) sends most rows to one reducer, and the whole job waits for it. Salt the key or handle the hot key separately.
  • Too many shuffles. Every JOIN, GROUP BY and DISTINCT on a new key is a shuffle. Filter and project early; broadcast small tables in joins.
  • The small-files problem. Thousands of tiny output files make every later read slow. Compact or repartition before writing.
  • Row-by-row Python UDFs in Spark. Use built-in functions or vectorized (Arrow-based) UDFs.
  • Reaching for a cluster too early. Distributed systems add cost and failure modes. Scale out when one machine really isn’t enough.

Learning Resources

Papers

Books

Docs and courses