Caching sits directly in the critical path of almost everything Twitter serves — timelines, social graph lookups, tweet objects — which meant that even small inefficiencies in the caching layer translated into real, aggregate cost and latency at Twitter's request volume. Stock Memcached, the industry-standard caching daemon that Twitter started with like most large web companies, was built for general-purpose use rather than for the specific operational demands of running at Twitter's scale: predictable memory behavior under high churn, low per-connection overhead with enormous connection counts, and visibility into exactly what the cache was doing at any moment. Rather than continuing to run stock Memcached and work around its limitations at the application layer, Twitter invested directly in owning and reshaping its caching infrastructure.

Twemcache and the memory fragmentation problem

Twitter's first major step was Twemcache, a fork of Memcached tuned for Twitter's specific workload characteristics and operational needs. One recurring pain point with stock Memcached at high scale was memory fragmentation from its slab allocator under Twitter's particular mix of item sizes and churn rates, which could leave a cache instance with plenty of free memory in aggregate but unable to satisfy new allocations efficiently. Twemcache added instrumentation, stats, and operational improvements suited to running thousands of cache instances reliably, treating the kind of visibility an SRE needs — per-instance stats, connection behavior, eviction patterns — as a first-class feature rather than something to infer indirectly.

Pelikan: rebuilding the cache as a framework

Twitter's caching investment matured further with Pelikan, a framework for building cache servers rather than a single fixed cache implementation. The insight behind Pelikan was that "cache" isn't really one workload — a cache backing a hot, small, read-dominated dataset behaves very differently from one backing a large, write-heavy dataset with short-lived items — and a single monolithic implementation optimized for one pattern inevitably makes compromises for the others. Pelikan provided modular building blocks (storage backends, protocols, eviction policies) that could be assembled into a cache server tuned for a specific workload, including a segment-based storage engine designed to reduce the fragmentation and metadata overhead that had been a persistent issue with slab-based allocators.

text
workload profile --> Pelikan building blocks --> tuned cache server
   (hot/small, write-heavy, TTL-short, etc.)      (storage engine + protocol + eviction policy)

Treating cache infrastructure as a product

The throughline across Twitter's caching evolution is treating caching as owned, first-class infrastructure with its own dedicated engineering investment, rather than a commodity dependency you install and leave alone. That meant building deep operational tooling — dashboards, alerting, capacity models specific to cache behavior — and being willing to fork or rebuild lower layers of the stack when off-the-shelf software's design assumptions stopped matching Twitter's actual traffic patterns. At Twitter's request volume, cache hit rate and eviction behavior directly moved user-facing latency and back-end database load, which made this kind of investment pay for itself in ways that wouldn't be justified at smaller scale.

Operational gotchas of forking the cache

Twitter's path from memcached to owned cache infrastructure came from needing features and operations the upstream did not give them. The failure mode for a mid-size team is forking too early: you now own a C codebase and a cluster, and you still have hot keys. Steal the operational lessons — consistent hashing, connection pooling, leases — from Twemproxy or a managed Redis, and fork only when you have a specific, measured gap.

The concrete failure mode is a custom cache that no longer takes security patches because the fork drifted. Another is client libraries that speak only the fork's protocol, so a rollback to memcached is fiction. Operational gotcha: cache as a source of truth for tweets that should have been in the timeline store; a flush becomes a public incident. Keep TTL and a rebuild. Multi-DC cache replication of unauthenticated tweets is a product leak if private status bits ride along. Separate keys. Twitter-scale justified Pelikan and similar. Your scale justifies not running a unique daemon. Measure hit rate, tail latency, and error amplification on node death. If losing one cache node takes the site to the database, you needed replicas or shedding, not a fork. The steal is treating cache as a product with an on-call, not a leftover apt package.

What you can borrow

  • Don't assume general-purpose infrastructure was tuned for your workload — profile it under your actual traffic before assuming the defaults are fine.
  • Memory fragmentation and allocator behavior are easy to overlook until you're running at a scale where they show up as real capacity waste; they're worth understanding before that point, not after.
  • "Cache" is not one workload — a system optimized for hot small data and one optimized for high-churn write-heavy data have genuinely different ideal designs.
  • Deep operational visibility (per-instance stats, eviction patterns, connection behavior) is what actually lets you run infrastructure well at scale, not just the core serving logic.