$ ./kappa --replay-from earliest

Kappa Architecture

The Kappa architecture handles both real-time and historical processing with a single stream processing pipeline. All events are kept in a durable, replayable log (usually Apache Kafka). The streaming job reads the log and maintains the output tables continuously. When logic changes, you do not run a separate batch job: you start a new version of the streaming job that replays the log from the beginning, let it catch up, and switch readers over. Jay Kreps proposed it in 2014 as a simpler alternative to Lambda.

Reprocessing without a batch layer

$ flink run orders_by_city_v2.jar # source starts at earliest offset
One log, one code path, replay to fix history
KAFKA TOPIC · orders · offsets 0 → 13 · long retention 012345678910111213 new → v1 STREAM JOB v1 live, at head of log orders_by_city_v1 table SERVING dashboards, API v2 STREAM JOB v2 replaying from offset 0 orders_by_city_v2 filling up caught up, live complete v1 stopped, table dropped
  1. The log keeps every order event with its offset. It is the source of truth, so retention must cover all the history you may need to reprocess.
  2. Job v1 reads the log continuously and keeps orders_by_city_v1 up to date. Dashboards read that table.
  3. The logic changes (a bug fix, a new column). Start job v2 with the same code path, reading from offset 0 into a new table. v1 keeps serving meanwhile.
  4. v2 replays history much faster than real time and reaches the head of the log. Both tables are now live.
  5. Point readers at v2, then stop v1 and drop its table. One codebase, and history was recomputed without a batch system.

Requirements

  • A replayable log with long retention. Kafka with tiered storage (KIP-405, production-ready since Kafka 3.9), Redpanda or WarpStream, or a compacted topic for entity state. If old events are deleted, they cannot be reprocessed.
  • A stateful stream processor with exactly-once state and event-time handling: Apache Flink, Kafka Streams or Spark Structured Streaming. See Batch and Stream Processing.
  • Versioned outputs, so a new job version writes to a new table or topic and readers can switch atomically.
  • Deterministic logic: replaying the same events must produce the same results (no calls to “now()” or to external APIs whose answers change).

Lambda vs Kappa

Lambda Kappa
Code paths Batch + stream Stream only
Reprocessing Rerun batch job Replay log into new job version
Operational cost Two systems One system, but a big log
Best for Heavy historical recomputation, mixed tools Event-driven data where all inputs flow through a log
Weak spot Logic duplicated and drifting Very large replays are slow and need capacity planning

When Kappa fits

  • All inputs are already events (clickstream, IoT, CDC from databases).
  • Freshness matters and you want one definition of each metric.
  • History fits in the log, or older history can be bootstrapped from a lakehouse snapshot before switching to the stream.

It fits less well when you must join with large, slowly changing reference data from many batch sources, or when reprocessing years of data through a stream engine is too slow or too costly. A common 2026 hybrid is Kappa-style streaming into Iceberg or Delta tables, with batch jobs reading the same tables for heavy backfills.

Common mistakes

  • Kafka retention of 7 days (the default) and then discovering you cannot replay last quarter.
  • Overwriting the live output table during a replay, so dashboards show partial numbers.
  • Under-provisioning the replay: catching up on months of events needs far more parallelism than live processing.

Learning Resources

Articles

Books