Megatron-LM: Slicing Transformers Across GPUs on Purpose
NVIDIA's Megatron-LM paper showed tensor and pipeline parallelism so Transformer layers larger than one GPU's memory could still train efficiently on DGX-class boxes.
Tag
81 articles · 287 min of reading
NVIDIA's Megatron-LM paper showed tensor and pipeline parallelism so Transformer layers larger than one GPU's memory could still train efficiently on DGX-class boxes.
NCCL implements all-reduce and friends across GPUs and nodes so data-parallel training can use NVLink, NVSwitch, and InfiniBand without each framework rewriting rings.
Zaharia et al.'s NSDI 2012 RDD paper showed that coarse-grained lineage can rebuild lost partitions without replicating every intermediate byte.
Bing's IndexServe stack turns a continuously updated web index into low-latency ranked results, with partitioning, replication, and a strict serving budget.
How Microsoft's DeepSpeed library partitioned optimizer state, gradients, and parameters so teams could train models that would not fit on a single GPU's memory.
How Microsoft Service Fabric's reliable collections and replica lifecycle let services own partitioned state instead of pushing every byte to an external database.
Microsoft's Orleans runtime treats grains as always-addressable actors so game and cloud services can skip manual placement, lifecycle, and locking.
How Azure Cosmos DB made consistency a per-request choice instead of a cluster-wide religion, and what that means for globally distributed apps.
Castro and Liskov's PBFT: 3f+1 replicas, three-phase agreement, and when crash-fault Raft is the wrong threat model.
Fischer, Lynch, and Paterson 1985: why deterministic consensus cannot be both safe and live in a fully asynchronous network with one crash.
How Alibaba and Ant Group built OceanBase as a Paxos-replicated, shared-nothing SQL system that had to survive Singles' Day traffic without a single primary as the bottleneck.
How Booking.com's availability search combines cached hotel inventory, partner feeds, and last-minute checks so a search result still has a room at checkout.
How ClickHouse Distributed tables send query pieces to shards, merge results on a coordinator, and fail in ways a single-node MergeTree never will.
How CockroachDB splits tables into ranges, replicates them with Raft, and serves from multiple regions without one global write master.
How CockroachDB implements SQL — including serializable transactions — on a distributed key-value map, and why contention shows up as retries instead of silent anomalies.
How Elasticsearch wraps Lucene indexes as shards with replicas so search can scale out — and why shard count is a decision you will live with.
How Etsy built activity and following feeds when millions of shops produce events and naive fanout would melt the write path.
How GitLab extracted Git disk operations into Gitaly so the Rails app no longer spoke filesystem paths to a shared NFS server.
How HashiCorp Consul combined Serf gossip for membership with a Raft-backed catalog so services could find healthy peers without a shared hosts file.
How HashiCorp Nomad scheduled jobs across a fleet with evaluations, allocations, and bin packing — without requiring a control plane that looks like a second operating system.
How MongoDB replica sets elect a primary, copy operations through the oplog, and expose write concern as a durability choice rather than a default you ignore.
How PayPal's scale forced an accounting-shaped ledger for balances and transfers so money could move across instruments without a single mutable number as the truth.
How PayPal's architecture outgrew two-phase commit across databases, and why eventual consistency with compensating actions became the payments pattern.
How Redis async replication and Sentinel (or Cluster) elect a new primary, and why a missed replica can still accept writes you will never see again.