Data Architecture

A data processing architecture decides how data flows from sources to the answers people query, and in particular how you combine historical, complete results with fresh, real-time ones. The two classic patterns are named after Greek letters:

  • Lambda: run a batch layer for accurate history and a speed layer for recent events, then merge both at query time.
  • Kappa: use one stream processing path for everything, and reprocess history by replaying a durable log.
Lambda Kappa
Processing paths Two (batch + stream) One (stream)
Codebases Two, must agree One
Reprocessing Rerun the batch job Replay the log into a new job version
Main requirement Batch and stream engines A replayable log with long retention

Before these, read Batch and Stream Processing. For where the data ends up, see Data Management Architecture.

In 2026 many teams use a third, pragmatic option: stream events into a lakehouse table with Flink or Spark Structured Streaming, and run batch jobs on that same table. One storage layer, two kinds of compute.

Learning Resources

Presto Kappa