MLflow: An Experiment Log That Outlived the Notebook
Databricks open-sourced MLflow so teams could record parameters, metrics, and artifacts from training runs without each group inventing a spreadsheet protocol.
Engineering at scale
5 articles · 15 min of reading
Databricks open-sourced MLflow so teams could record parameters, metrics, and artifacts from training runs without each group inventing a spreadsheet protocol.
Unity Catalog centralized identities, tables, and permissions so lakehouse objects could be governed like warehouse objects instead of as raw S3 prefixes.
Databricks built Photon as a native, vectorized engine so SQL and Spark workloads could leave interpreted row-at-a-time execution on the table.
Delta Lake put an ordered transaction log in front of Parquet files so Spark jobs could get ACID tables on cheap object stores that do not speak databases.
Zaharia et al.'s NSDI 2012 RDD paper showed that coarse-grained lineage can rebuild lost partitions without replicating every intermediate byte.