JStorm: Alibaba's Storm Fork When the In-House Stream Had to Outrun the Upstream
How Alibaba forked Apache Storm into JStorm to get a faster, more operable stream processor for commerce and logging pipelines at internal scale.
Tag
17 articles · 62 min of reading
How Alibaba forked Apache Storm into JStorm to get a faster, more operable stream processor for commerce and logging pipelines at internal scale.
How Coinbase-style onchain indexers ingest blocks, handle reorgs, and serve balances without treating a node RPC as a database.
How Confluent Schema Registry stores Avro/Protobuf/JSON schemas with compatibility modes so producers and consumers can evolve without a coordinated outage.
How Elasticsearch ingest pipelines transform documents on the way in, and how Index Lifecycle Management rolls data from hot to delete without custom jobs.
How Etsy built activity and following feeds when millions of shops produce events and naive fanout would melt the write path.
The columnar storage and execution-tree design behind Google's Dremel engine, and how it became the foundation for BigQuery.
How to decide when features need real-time streaming computation versus a scheduled batch job, and how to keep the two consistent with each other.
How Spotify's event pipeline evolved from batch log collection off servers to a real-time, cloud-native Pub/Sub system serving hundreds of consuming teams.
Why Netflix helped create Apache Iceberg to fix Hive table limitations, then rethought its data platform around a data-mesh ownership model.
Why Airbnb kept getting conflicting numbers for the same metric across teams, and how Minerva's centralized definitions and certification process fixed it.
How Netflix built Keystone, its Kafka- and Flink-based real-time data platform, and why it later migrated the processing layer from Samza to Flink.
How Google's Percolator system replaced batch MapReduce reindexing with incremental, transactional updates and cut search index latency dramatically.
How DoorDash built a unified, Kafka-based event pipeline named Iguazu to feed analytics and machine learning from a sprawling microservices fleet.
How Spotify combines collaborative filtering, text analysis, and audio features in a weekly batch pipeline to power Discover Weekly for every user.
How Uber built Hudi to bring record-level updates, deletes, and incremental processing to a Hadoop data lake designed for immutable batch files.
Why cron and ad hoc scripts stopped working for Airbnb's data pipelines, and how the internal tool built to fix it became the industry-standard workflow scheduler.
How Google's 2004 MapReduce paper turned a painful distributed-systems problem into a simple programming model — and accidentally created Hadoop.