Project Inversion: Stopping Feature Work to Rebuild the Platform
Why LinkedIn paused most new feature development for months to overhaul its deployment, testing, and infrastructure practices before the site could scale further.
Engineering at scale
12 articles · 48 min of reading
Why LinkedIn paused most new feature development for months to overhaul its deployment, testing, and infrastructure practices before the site could scale further.
How LinkedIn built Brooklin as a single, generic streaming pipeline service to replace a sprawl of one-off connectors moving data between Kafka, Espresso, and beyond.
How LinkedIn built LIquid, an in-memory distributed graph engine, to answer connection-degree and shortest-path queries over its member network in real time.
Why LinkedIn built Pinot to answer analytical, slice-and-dice queries directly in member-facing products with sub-second latency at huge scale.
How LinkedIn's tangle of point-to-point data pipelines led to Kafka's commit-log abstraction, and how one internal tool became industry infrastructure.
How LinkedIn built Databus to stream low-latency, ordered change events out of Oracle and MySQL years before change data capture became a common pattern.
Why LinkedIn built Venice to bridge offline-computed and streaming-computed data into a low-latency read-only serving store for machine learning features.
How LinkedIn built T-REX, its internal experimentation platform, to make running and trusting thousands of concurrent A/B tests routine rather than risky.
How LinkedIn's move to a service-oriented architecture forced a standardized REST framework, and why schema-driven API contracts paid off at scale.
Why LinkedIn built its own distributed document database to replace Oracle for online member data, and how it married timeline consistency with Kafka.
How LinkedIn built Apache Samza to pair with Kafka, using partitioned logs and local state to make stateful stream processing durable and rebalanceable.
How LinkedIn adapted Amazon's Dynamo paper into Voldemort, a highly available key-value store built for read-heavy, low-latency online serving.