Data lakes fail governance first. Tables are paths. Permissions are IAM policies on buckets that do not know what a column is. Every engine — Spark, BI tools, notebooks — has a different notion of "user." Databricks Unity Catalog is a cross-workspace, three-level namespace (catalog.schema.table) with a metastore that issues table access, audits, and — over time — row and column filters, volumes for files, and model objects. The company bet that the lakehouse would not be trusted in regulated shops until the catalog, not the object store, was the control plane.
Identity and the table as the unit of grant
Unity Catalog binds to enterprise identity (SSO, SCIM) so a grant to a group survives a cluster that dies every night. Table ACLs are evaluated at query planning, not only at S3 GET time. That distinction matters: an IAM role that can read a bucket can still exfiltrate Parquet if the engine is bypassed. Databricks therefore combines metastore checks with credential vending — short-lived storage credentials scoped to what the catalog allowed — so the engine does not hold a god-mode instance profile. This is the same pattern warehouses used for decades, applied to objects.
Lineage and audit logs are the other half. If you cannot answer "who read pii_customers last week" you do not have governance, you have a wiki. Unity Catalog's lineage graph (jobs to tables to dashboards) is how platform teams close that loop. It is only as good as the engines that report lineage. Jobs that write with raw S3 APIs around the catalog recreate the shadow lake.
Open formats, closed control plane, multi-cloud reality
Databricks open-sourced Unity Catalog APIs and pushed Iceberg/Delta interoperability so the catalog is not only a lock-in story. The operational reality is still a metastore that must be highly available and correctly replicated across regions. A catalog outage is a company-wide read-only event for every governed table. Multi-cloud (AWS, Azure, GCP) means storage credentials, KMS keys, and network paths differ; the catalog's job is to hide that without hiding the blast radius of a mis-granted location.
Fine-grained security (row filters, column masks) is implemented as query rewriting or engine-enforced predicates. The failure mode is a client that does not honor them — an old JDBC driver, an export job. Defense in depth still wants sensitive columns in separate tables or encrypted, not only masked in SQL. Another failure mode is metastore sprawl: every sandbox creates a catalog that nobody owns, which is Hive metastore chaos with a better name. Platform teams need a catalog taxonomy (prod, shared, sandbox) and a deletion policy.
The steal for a mid-size company is not "deploy Unity." It is: one namespace, identity-based grants, credential vending instead of long-lived bucket keys, and an audit log that engines cannot opt out of. Until those exist, your lake is a shared disk.
What you can borrow
- Make the catalog the only supported way to find and read tables; punish raw path access in prod.
- Grant to groups from the identity provider; do not maintain IAM users per cluster.
- Vend short-lived storage credentials scoped to authorized objects, not a workspace-wide key.
- Require engines to emit lineage and audit events, or treat them as ungoverned.
- Implement row/column security in the engine and assume hostile clients exist; split truly sensitive data.