#Unity Catalog and Governance

Governance is the spine of every enterprise Databricks conversation, and the thing customers get most wrong. If you can only be excellent at two topics, make them this and performance.

#1. The object model

METASTORE  (one per region)
│
├── CATALOG                     ← the isolation unit: environment, domain, or both
│     └── SCHEMA
│           ├── TABLE           managed | external
│           ├── VIEW            incl. dynamic views
│           ├── MATERIALIZED VIEW
│           ├── VOLUME          governed non-tabular files (managed | external)
│           ├── FUNCTION        incl. row-filter and column-mask functions
│           └── MODEL           registered ML models
│
├── STORAGE CREDENTIAL          → cloud IAM identity Databricks assumes
├── EXTERNAL LOCATION           → (storage credential + path) the grantable unit of raw storage
├── CONNECTION                  → to a federated source (foreign catalogs)
├── SHARE / RECIPIENT           → Delta Sharing
└── CLEAN ROOM

Three-level namespace: catalog.schema.object. Say "three-level namespace" in interview — it is the shibboleth that distinguishes UC-native thinking from legacy Hive-metastore thinking.

#Managed vs external tables

Recommend managed by default. The automatic maintenance alone usually justifies it, and "we made everything external because we might leave one day" is a costly insurance policy against an event that rarely happens.

#Storage credentials and external locations

The chain is: storage credential (an IAM role / managed identity Databricks can assume) + a path = an external location, which is the object you grant on. Users never touch cloud credentials; UC vends short-lived, scoped access at query time. This "credential vending" point is worth making explicitly, because it is the answer to "so who can read the bucket directly?"

#2. The privilege model

ANSI-style GRANT, with inheritance down the hierarchy: a grant on a catalog applies to all its schemas and their objects.

The privileges that matter:

Ownership is separate from privileges: the owner can always manage and grant on the object. Own objects with groups, never individuals. Individually-owned production tables become orphaned the day someone leaves — this is a genuine enterprise failure mode and a good thing to raise unasked.

Design principle to state: "Grant to groups, at the catalog or schema level, never to users on individual tables. If your permission model requires per-table grants, the schema design is wrong. Permissions should fall out of the structure."

#3. Fine-grained access control

Three mechanisms, increasingly capable.

#Dynamic views

A view whose logic calls current_user() or is_account_group_member() to filter rows or mask columns. Works everywhere, but the logic is scattered across view definitions, and it only protects consumers who use the view.

#Row filters and column masks

UC functions attached to the table itself:

• Row filter — a function returning a boolean, evaluated per row; the table returns only rows for which it is true.
• Column mask — a function transforming a column's value based on who is asking.

Superior to dynamic views because the policy lives with the table, so it applies no matter how the table is reached. This is the standard answer for "how do you do row-level security".

#ABAC — attribute/tag-based policies

Tag columns and objects (e.g. pii, classification=restricted), then write policies against the tags rather than against objects. Governance then scales with the data: a new table tagged pii inherits masking automatically, with no new grants written.

The scaling argument to make: "Object-based grants are O(objects × groups) and grow forever. Tag-based policies are O(policies), and new data inherits governance on arrival. At 14 teams and tens of thousands of tables, that's the difference between a governance model and a governance backlog."

#4. Lineage, discovery and audit

• Lineage is captured automatically at table and column level for operations run through UC, across notebooks, jobs, pipelines and dashboards — and it extends to models and, increasingly, to AI assets. Use it for impact analysis ("what breaks if I change this column?"), regulatory evidence, and finding the actual owner of a mystery table.
• System tables are the governed, queryable operational record: billing/usage, access audit, query history, lineage, compute, and more. Everything you would otherwise build an observability project for. Answer cost, audit and adoption questions with system-table queries — it is concrete and it is what a real practitioner reaches for.
• Discovery: catalog search, tags, comments, and AI-generated documentation; Business Glossary brings authoritative business definitions alongside the technical assets.

The semantic layer (2026). Unity Catalog Metrics lets you define KPIs — revenue, churn, active customer — as governed, reusable objects queryable from SQL, BI tools, APIs and agents, with Domains grouping assets along business lines. (Summit 2026) This matters more than it sounds: it is the fix for "every dashboard computes revenue differently", and it is the substrate agents need in order to answer business questions consistently. Expect to be asked about the semantic layer; it is a current topic.

#5. Delta Sharing and clean rooms

Delta Sharing is an open protocol for sharing live data without copying it:

• Databricks-to-Databricks — richest experience, shares tables, views, volumes, models, notebooks.
• Open sharing — any client implementing the protocol (pandas, Spark, Power BI, Tableau), via a credential file. The recipient does not need Databricks.
• Sharing is live — the recipient reads current data, not a nightly extract. This kills a whole category of FTP-and-reconcile integration.
• OpenSharing (2026) extends the protocol under Linux Foundation governance to cover AI assets and Iceberg REST-compatible clients, and on-premises/private-cloud sources. (Summit 2026)

Clean rooms allow multiple parties to run approved computations across their combined data without either party seeing the other's raw rows — the standard answer for advertising measurement, retail/CPG collaboration, and cross-bank fraud consortia.

The commercial framing: "Sharing turns your data estate from a cost centre into a distribution channel — the same governance model that protects it internally now lets you monetise or collaborate externally without building an export pipeline per partner."

#6. Lakehouse Federation

Query external systems in place — Snowflake, SQL Server, Postgres, Redshift, BigQuery, MySQL and others — via a connection plus a foreign catalog, so external tables appear in the three-level namespace and are governed by the same UC permissions and lineage.

Where it genuinely fits:

• Migration runway — govern and query the legacy estate from day one, migrate workload by workload instead of big-bang. This is its best use and a strong migration story.
• Low-volume reference data that is not worth replicating.
• Exploration before committing to an ingestion pipeline.

Where it does not: high-volume, high-frequency analytical workloads. You inherit the source system's performance and you put analytical load on an operational database. Say this plainly — over-selling federation is a credibility loss with any experienced customer.

#7. The migration from Hive metastore to UC

A common real engagement, so have the shape ready.

1. Assess — inventory tables, jobs, and access patterns; find external locations and who writes where. System tables and lineage do much of this.
2. Set up the foundation — metastore, storage credentials, external locations, account-level groups, catalog design and naming standard.
3. Upgrade metadata — SYNC external tables into UC; decide table-by-table whether to remain external or convert to managed.
4. Cut over compute — UC requires a supported access mode; older jobs assuming no-isolation clusters or direct DBFS paths need remediation. This is where the real work is — budget for it explicitly rather than discovering it mid-programme.
5. Migrate permissions — do not lift and shift legacy table ACLs. Redesign around groups and inheritance; this is the one chance to fix the permission sprawl.
6. Decommission — disable the legacy metastore and remove direct storage access so it cannot be bypassed. Governance that can be bypassed is not governance — say this.

Known friction to name proactively: direct DBFS/path access habits, RDD-level code, unsupported libraries on shared access modes, machine-to-machine identity, and the political fight about who owns which catalog. The last one is usually the longest pole.

#8. Questions to answer cold

1. Why does SELECT alone not grant access? What else is needed?
2. Managed vs external tables — recommend one and defend it.
3. Row filters versus dynamic views — why prefer one?
4. How does ABAC change the scaling properties of a governance model?
5. What is a storage credential, and who ever handles cloud keys?
6. When is Lakehouse Federation the right answer, and when is it a trap?
7. How do you prove to an auditor who read a PII table last quarter?
8. Walk through a Hive metastore → UC migration and name where it usually goes wrong.