Vitess at YouTube: Horizontal Sharding Without Leaving MySQL
YouTube built Vitess to shard MySQL with a proxy, a lock server, and resharding workflows so a massive video site could keep SQL instead of betting the company on a new store.
Category
81 articles · 290 min of reading
YouTube built Vitess to shard MySQL with a proxy, a lock server, and resharding workflows so a massive video site could keep SQL instead of betting the company on a new store.
Delta Lake put an ordered transaction log in front of Parquet files so Spark jobs could get ACID tables on cheap object stores that do not speak databases.
How Azure Cosmos DB made consistency a per-request choice instead of a cluster-wide religion, and what that means for globally distributed apps.
How Alibaba and Ant Group built OceanBase as a Paxos-replicated, shared-nothing SQL system that had to survive Singles' Day traffic without a single primary as the bottleneck.
How ByteDance's ByteHouse built an analytics warehouse experience on ClickHouse-shaped execution for internal and cloud customers who needed interactive SQL on huge event streams.
How ClickHouse's MergeTree family stores data in immutable parts and merges them in the background, which is why inserts are fast and mutations are not.
How CockroachDB splits tables into ranges, replicates them with Raft, and serves from multiple regions without one global write master.
How CockroachDB implements SQL — including serializable transactions — on a distributed key-value map, and why contention shows up as retries instead of silent anomalies.
How MongoDB replica sets elect a primary, copy operations through the oplog, and expose write concern as a durability choice rather than a default you ignore.
How MongoDB's WiredTiger engine brought document-level concurrency, compression, and checkpointing — replacing MMAPv1's collection-level locks.
How Redis kept a single-threaded command loop to make operations atomic and latency predictable — and where that design still bites under big keys and slow commands.
Aggregate-oriented persistence without lazy-loading surprises, how to map one-to-many without Hibernate, and when you should stay on JPA anyway.
State-based versus operation-based CRDTs, the data types that actually ship (counters, maps, text), and when last-write-wins is honest enough.
Why databases write the log before the page, how REDO and UNDO recover, and the fsync choices that turn a power loss into silent corruption or lost commits.
Thomson, Diamond, Weng, Ren, Shao, and Abadi, SIGMOD 2012: agree on input order, then execute lock-free replicas in lockstep.
How B-trees update pages in place and LSM trees turn writes into sorted flushes, with the read/write/space trade-offs that show up in MySQL, Postgres, RocksDB, and Cassandra.
VTGate, VSchema, and scatter vs targeted queries: the routing layer that made YouTube's MySQL split look like one database.
Chunks, chunk exclusion, compression, and retention policies: when a hypertable beats rolling your own monthly partitions.
Labels, relationship types, and property indexes: how Cypher plans stay cheap, and why a graph is not an excuse to skip data modeling.
MergeTree local execution versus elastic warehouses: when you want ClickHouse's latency, and when you want Snowflake/BigQuery's isolation.
How micro-partitions make undrop, time travel, and clone cheap at first — and why Fail-safe plus retained history still shows up on the bill.
Distributed tables, reference tables, and the query router: what Citus can push down, and which joins become network products.
The layer model: a strictly serializable ordered map, transactions via the sequencer, and why Record Layer and Document Layer exist.
Spanner waits out clock uncertainty; CockroachDB combines hybrid logical clocks with uncertainty intervals. Same serializability goal, different physics.