Data Processing
Data processing is the step where raw data becomes something useful: filtered, cleaned, joined, aggregated and written somewhere else. When the data fits on one machine, a SQL query or a Python script is enough. When it doesn’t, or when it never stops arriving, you need an engine that splits the work across many cores or machines and survives one of them dying halfway through.
Almost every distributed engine, from Google’s MapReduce (2004) and Hadoop to Apache Spark and Flink today, follows the same shape: split the input, process the pieces in parallel, shuffle by key, combine the results.
Split, map, shuffle, reduce
- The input (a folder of Parquet files, a Kafka topic) is cut into splits, and each machine gets some of them.
- Map: each task reads its split and emits (city, count) pairs. It pre-aggregates locally, so it sends NYC:2, not NYC:1 twice.
- Shuffle: pairs are routed by hash of the key, so every NYC goes to reducer 1. This is the expensive step: data crosses the network and often hits disk.
- Reduce: each reducer combines the values for its keys and writes the result.
- A machine dies. The engine knows which split that task read, so it re-runs only that task elsewhere. This is how a 1,000-node job survives failures.
Spark calls the parts between shuffles stages and plans them from your SQL or DataFrame code. Flink does the same continuously: a keyed stream is a never-ending shuffle, and reducers keep running totals in state.
Batch, streaming and in between
- Batch: process a bounded dataset (yesterday’s orders) and finish. Simple, cheap, easy to re-run.
- Streaming: process an unbounded stream event by event and keep results up to date. Fresh, but needs state, checkpoints and handling of late data.
- Micro-batch: small batches every few seconds (Spark Structured Streaming).
See Batch and Stream Processing for a side-by-side simulation.
Engines in 2026
| Engine | Type | Best at |
|---|---|---|
| Apache Spark | Batch + micro-batch streaming | Large ETL, lakehouse jobs, ML prep. SQL, Python, Scala. Spark 4 (2025) added a VARIANT type for semi-structured data and a native XML reader. |
| Apache Flink | Streaming first (batch too) | Low-latency stateful streaming, event time, exactly-once state |
| Apache Beam | Unified API | Write once, run on Dataflow, Flink or Spark |
| Kafka Streams | Streaming library | Stream processing inside a Java service, no separate cluster |
| Warehouse SQL + dbt | Batch (ELT) | Transformations where the data already lives: Snowflake, BigQuery, Databricks SQL, ClickHouse |
| DuckDB, Polars | Single-node, vectorized | Anything up to hundreds of GB on one machine, fast and cheap |
| Ray, Dask | Python-native distributed | ML pipelines and Python workloads that outgrow one machine |
| RisingWave, Materialize | Streaming SQL databases | Continuously updated SQL views over streams |
What this table leaves out on purpose: Kafka itself moves data but doesn’t process it (see Messaging and Streaming Platforms); Apache Arrow is an in-memory columnar format that many of these engines use internally; Hadoop MapReduce, Storm and Samza are legacy and rarely chosen for new work.
How to choose
- Does it fit on one machine? Data under a few hundred GB: try DuckDB or Polars first. It’s often faster than a cluster and costs nothing to run.
- Is the data already in a warehouse? Transform it there with SQL and dbt.
- Does it need results in seconds? Use Flink, Kafka Streams or a streaming database.
- Otherwise, large-scale batch on files: Spark.
Common mistakes
- Data skew. One key (a huge customer,
city = NULL) sends most rows to one reducer, and the whole job waits for it. Salt the key or handle the hot key separately. - Too many shuffles. Every
JOIN,GROUP BYandDISTINCTon a new key is a shuffle. Filter and project early; broadcast small tables in joins. - The small-files problem. Thousands of tiny output files make every later read slow. Compact or repartition before writing.
- Row-by-row Python UDFs in Spark. Use built-in functions or vectorized (Arrow-based) UDFs.
- Reaching for a cluster too early. Distributed systems add cost and failure modes. Scale out when one machine really isn’t enough.
Learning Resources
Papers
- MapReduce: Simplified Data Processing on Large Clusters: Dean and Ghemawat, 2004
- Resilient Distributed Datasets: the paper behind Spark
- The Dataflow Model: the ideas behind Beam and Flink
Books
- Designing Data-Intensive Applications, Chapters 10 and 11: Martin Kleppmann
- Learning Spark, 2nd Edition
- Streaming Systems: Akidau, Chernyak and Lax