Once Kafka existed as LinkedIn's central log, a new problem showed up almost immediately: dozens of teams wanted to compute something continuously from those streams — enriching an event with profile data, computing rolling counts for relevance models, joining two topics together — and they were all solving the same infrastructure problems from scratch. Checkpointing progress, recovering from failure without reprocessing everything or dropping events, and managing local state that could grow larger than memory were being reinvented team by team, usually badly. LinkedIn built Samza to make stream processing a first-class piece of infrastructure with the same seriousness that Kafka gave to the log itself, rather than treating it as an afterthought bolted onto whatever compute framework happened to be handy.

Borrowing Kafka's own abstractions for state

The defining decision in Samza's design was to lean on Kafka's partitioning model instead of inventing a separate one. A Samza job is a set of tasks, each assigned one or more Kafka partitions, and each task processes its partitions independently and in order. Because Kafka already guarantees ordering within a partition and durability across the cluster, Samza didn't need to build its own replicated log for input data — it just consumed one that already existed.

The more distinctive piece was how Samza handled local state. Instead of keeping all state in memory or reaching out to a remote store for every lookup, each task got its own embedded key-value store (backed by RocksDB) co-located with the task. That store was also continuously backed up to a dedicated Kafka "changelog" topic. If a task died and was rescheduled elsewhere, Samza could rebuild its local state by replaying the changelog rather than recomputing from the beginning of time or losing state entirely. This meant jobs could hold state far larger than fits comfortably in memory — counters, windows, join tables — while still recovering cleanly from failure.

text
input partition --> task (local RocksDB state) --> changelog topic (Kafka)
                                    |
                                    v
                              output topic

Fault tolerance without a special-purpose coordinator

Samza deliberately reused existing infrastructure rather than building new coordination machinery. Early versions ran on YARN for resource allocation and process management, treating a stream processing job much like a long-running YARN application instead of inventing a bespoke cluster manager. Task-to-partition assignment and checkpointing followed the same "the log is the source of truth" philosophy as the rest of LinkedIn's streaming stack: a task's position was just an offset into its input partitions, checkpointed periodically, so recovery meant resuming from the last checkpoint and replaying forward.

This gave Samza at-least-once processing semantics by default, which was sufficient for the great majority of internal use cases — counting, joining, enriching — where idempotent downstream writes made occasional reprocessing harmless.

Powering relevance and monitoring pipelines

Samza became the engine behind a wide range of LinkedIn's real-time pipelines: computing derived metrics and relevance signals feeding features like the news feed and notifications, powering call-graph and operational monitoring by processing service log streams in real time, and handling the kind of continuous joins and aggregations that would otherwise have required either brittle in-house code or a slow batch round-trip through Hadoop. LinkedIn open sourced Samza and it became an Apache project, used alongside Kafka at other companies that had adopted the same log-centric architecture.

Operational gotchas of stateful stream jobs

Samza's bet — local state next to a partitioned log, restored from changelog — fails in production when state size outruns disk, when a rebalance restores checkpoints that are minutes behind a bursty topic, or when a job is not actually idempotent and a restore replays side effects. Mid-size teams copy "stateful streaming" for sessionization or feature generation and then treat RocksDB on the task as free. It is not: compaction, changelog traffic, and host replacement become the job.

A concrete failure mode is a join between a high-rate stream and a slowly changing table where the table changelog was compacted too aggressively; after a bounce, the job cannot rebuild the lookup and starts dropping enrichments. Another is processing-time logic that looks correct until a partition stalls and event-time watermarks freeze. Steal Kafka-plus-local-store only if you can explain restore time after losing a host. Otherwise, push state into a serving store Venice-style and keep the processor thin. Mid-size steal: start with stateless filters and aggregations that can replay from the log, add keyed state only for windows you can bound, and never write to an external database from a task unless that write is idempotent and observable. Rebalances that look like "just shuffling tasks" are user-visible latency cliffs if restore is slow.

What you can borrow

  • If you already have a durable, partitioned log, build your stream processor's fault tolerance on top of it rather than inventing a parallel durability mechanism.
  • Co-locating local state with the task that owns it, backed by a changelog for recovery, scales further than routing every state lookup over the network.
  • At-least-once semantics plus idempotent writes is usually good enough — exactly-once is expensive to guarantee and often unnecessary for the actual use case.
  • Reuse your existing resource manager and log infrastructure for coordination instead of building bespoke cluster-management machinery for a new subsystem.