Cheaper Than Triple Replication: Erasure Coding in Windows Azure Storage
How Microsoft replaced three full copies of blob data with a local reconstruction code that keeps durability high without tripling disk.
Tag
26 articles · 98 min of reading
How Microsoft replaced three full copies of blob data with a local reconstruction code that keeps durability high without tripling disk.
How Redis async replication and Sentinel (or Cluster) elect a new primary, and why a missed replica can still accept writes you will never see again.
Zonal VMs, zone-redundant services, and regional pairs: what a datacenter fire takes, and what a region-level event still takes.
Why databases write the log before the page, how REDO and UNDO recover, and the fsync choices that turn a power loss into silent corruption or lost commits.
Horizontal versus vertical autoscaling in Kubernetes, custom metrics, and the cooldown and request-sizing mistakes that cause flapping.
Dean and Barroso, CACM 2013: latency variability, hedged and tied requests, and the queueing that makes averages look fine.
How Stripe models money movement with immutable ledger entries and explicit state machines to guarantee correctness even when networks and services fail.
How 2PC actually runs, why a crashed coordinator freezes the world, and how Raft/Paxos-backed transaction coordinators changed the picture.
What the Raft paper actually guarantees, how leader election and log replication work, and the operational failure modes etcd and Consul still hit.
How Shopify built Toxiproxy to deliberately simulate latency, timeouts, and bandwidth limits so engineers could test resilience before production did.
How Google's Site Reliability Engineering practice used error budgets and a hard cap on toil to align reliability work with product velocity.
Why Stripe deliberately triggers failures and runs large-scale load tests against production-like systems before its highest-traffic days of the year.
How DoorDash designs for graceful degradation and load shedding so a spike or partial outage on one side of its marketplace doesn't take down the rest.
WhatsApp famously ran hundreds of millions of users on a strikingly small engineering team by designing operational simplicity into the system itself.
How Stripe layers per-request rate limiting with system-wide load shedding to keep its payments API available when traffic spikes or dependencies slow down.
Inside the public postmortem for Slack's January 2021 outage, where a networking degradation turned into a self-reinforcing overload across the stack.
How AWS uses cell-based architecture and shuffle sharding to keep one customer's or one partition's failure from taking down everyone else.
How the transactional outbox pattern makes updating a database and publishing an event atomic, without a distributed transaction.
Why most postmortems produce a document nobody acts on, and what separates a blameless postmortem process that genuinely prevents repeat incidents.
How Slack structures incident severity, on-call response, and blameless postmortems to turn outages into durable operational improvements.
Shopify runs months of scripted load tests and failure drills against production infrastructure to prepare for Black Friday Cyber Monday.
Idempotency keys, atomic check-and-write, and why PUT being idempotent by spec doesn't make it the right tool for a retried write.
Why deadlocks happen even in well-designed schemas, how to read a deadlock log, and the lock-ordering habits that prevent them.
What Amazon S3's famous '99.999999999% durability' figure actually means statistically, and the engineering that backs the promise.