Chaos Engineering at Netflix: Breaking Things on Purpose
How a painful database outage pushed Netflix to build Chaos Monkey, the Simian Army, and a culture that tests failure before failure finds you.
Engineering at scale
14 articles · 56 min of reading
How a painful database outage pushed Netflix to build Chaos Monkey, the Simian Army, and a culture that tests failure before failure finds you.
Why Netflix's personalization system is architected as many specialized algorithms for ranking, rows, and artwork rather than a single model.
Why Netflix helped create Apache Iceberg to fix Hive table limitations, then rethought its data platform around a data-mesh ownership model.
Why Netflix moved its API layer from the homegrown Falcor model to federated GraphQL, and built the open-source DGS framework to get there.
How Netflix decides which of several images to show for the same title, and why artwork selection became its own machine-learning problem.
Why Netflix built Conductor, a workflow orchestration engine, to replace scattered state machines buried inside individual microservices.
How Netflix built Keystone, its Kafka- and Flink-based real-time data platform, and why it later migrated the processing layer from Samza to Flink.
How Netflix rebuilt its video encoding pipeline around per-title complexity analysis and the VMAF quality metric to cut bandwidth without hurting quality.
How Netflix built Titus, its own container management platform on EC2, to bring containers into an infrastructure stack built around AWS instances.
Why Netflix built Spinnaker to standardize continuous delivery across hundreds of teams and multiple cloud providers instead of scripting deploys by hand.
The story of Netflix's edge gateway and circuit-breaker library, and why isolating failure mattered more than avoiding it in a microservices fleet.
How Netflix's Atlas telemetry platform handles dimensional time-series data at a scale where most off-the-shelf monitoring systems fall over.
How Netflix wrapped Memcached in EVCache for wide-area replication and built Dynomite to give Redis a Dynamo-style distributed backbone.
Why Netflix stopped renting third-party CDN capacity and started shipping its own caching appliances directly into internet service providers.