Google
Dremel: How Google Made Trillion-Row Queries Feel Interactive
The columnar storage and execution-tree design behind Google's Dremel engine, and how it became the foundation for BigQuery.
3 min read
Tag
Data Engineering
9 articles
Machine Learning
Batch vs. Streaming Feature Computation
How to decide when features need real-time streaming computation versus a scheduled batch job, and how to keep the two consistent with each other.
3 min read
Spotify
From Log Files to Cloud Pub/Sub: Spotify's Event Delivery Evolution
How Spotify's event pipeline evolved from batch log collection off servers to a real-time, cloud-native Pub/Sub system serving hundreds of consuming teams.
2 min read
Airbnb
Minerva: How Airbnb Built One Source of Truth for Metrics
Why Airbnb kept getting conflicting numbers for the same metric across teams, and how Minerva's centralized definitions and certification process fixed it.
2 min read
Netflix
Keystone: The Pipeline That Moves Trillions of Events a Day at Netflix
How Netflix built Keystone, its Kafka- and Flink-based real-time data platform, and why it later migrated the processing layer from Samza to Flink.
3 min read
DoorDash
Iguazu: DoorDash's Kafka-Based Backbone for Analytics and ML
How DoorDash built a unified, Kafka-based event pipeline named Iguazu to feed analytics and machine learning from a sprawling microservices fleet.
3 min read
Spotify
The Data Pipeline Behind Spotify's Discover Weekly
How Spotify combines collaborative filtering, text analysis, and audio features in a weekly batch pipeline to power Discover Weekly for every user.
3 min read
Airbnb
How a Scheduling Problem at Airbnb Became Apache Airflow
Why cron and ad hoc scripts stopped working for Airbnb's data pipelines, and how the internal tool built to fix it became the industry-standard workflow scheduler.
2 min read
Google
The MapReduce Paper: How Two Functions Started the Big Data Era
How Google's 2004 MapReduce paper turned a painful distributed-systems problem into a simple programming model — and accidentally created Hadoop.
3 min read