#Lakehouse Architecture and Enterprise Topology

"Design secure, scalable and high-performing Lakehouse platforms" — at enterprise scale this is mostly a topology and operating-model question, not a pipeline question.

#1. Control plane vs compute plane

The distinction every Databricks security and networking conversation starts from. Be able to draw it in 90 seconds.

┌──────────────────────── DATABRICKS CONTROL PLANE (Databricks cloud account) ───────────────────┐
│  Web application · REST APIs · Notebooks & source · Job scheduler & orchestration               │
│  Cluster manager · Unity Catalog metastore service · Query history & system tables              │
└───────────────────────────────────────┬─────────────────────────────────────────────────────────┘
                                        │  (control messages; no customer data at rest)
        ┌───────────────────────────────┴────────────────────────────────┐
        ▼                                                                ▼
┌─────────────────────────────────┐                      ┌────────────────────────────────────┐
│ CLASSIC COMPUTE PLANE           │                      │ SERVERLESS COMPUTE PLANE           │
│ (your cloud account / VPC)      │                      │ (Databricks-managed account)       │
│ • VMs you can see and tag       │                      │ • Instant start, scale to zero     │
│ • Your network controls         │                      │ • Network controls via NCC /       │
│ • Full instance-type control    │                      │   egress policy / Private Link     │
└─────────────────────────────────┘                      └────────────────────────────────────┘
                     │                                                    │
                     └─────────────────► CLOUD STORAGE ◄──────────────────┘
                              (S3 / ADLS / GCS — your account, your data)

The sentence that answers most security questions: "Your data lives in your cloud storage. The control plane orchestrates; it isn't where the data sits. The decision that actually matters is whether compute runs in your VPC (classic) or Databricks' (serverless), and that drives your network design — customer-managed VPC and Private Link for classic, network connectivity configurations and egress control for serverless."

#2. The object hierarchy

ACCOUNT  (one per cloud provider; identity, billing, admin, metastore management)
  ├── METASTORE  — one per region; the Unity Catalog root
  │     └── CATALOG  ── SCHEMA ── { TABLE · VIEW · VOLUME · FUNCTION · MODEL }
  └── WORKSPACE  (many; a working environment, attached to one metastore)
        └── compute · jobs · dashboards · notebooks · SQL warehouses

Two facts people get wrong, so state them cleanly:

• A metastore is regional, and a workspace attaches to exactly one. Multiple workspaces in a region share one metastore — that shared metastore is what makes data discoverable and governable across teams.
• Identity lives at the account level (users, groups, service principals via SCIM) and is federated down to workspaces. Groups defined once, granted everywhere.

#3. How many workspaces? How many metastores?

The question you will be asked, phrased as "we have 14 teams, how should we structure this?" The wrong answer is a number. The right answer is a set of criteria and then a recommendation.

#What a workspace boundary actually gives you

• A blast-radius and access boundary for compute, jobs and notebooks
• A unit for cost attribution and cloud-account separation
• Separate cluster policies, quotas and admin delegation
• Regional placement and, where needed, data-residency alignment

#What it costs you

• Duplicated administration, policies and CI/CD targets
• Fragmented discovery unless UC is used well
• More network plumbing per workspace

#The pattern that works at enterprise scale

Environment separation is non-negotiable; team separation is a judgement call.

Region: eu-west-1                      ONE METASTORE
│
├── Workspace: platform-prod ──────┐
├── Workspace: domain-a-prod       ├── catalogs: prod_sales, prod_finance, prod_risk …
├── Workspace: domain-b-prod ──────┘        (bound to prod workspaces only)
│
├── Workspace: shared-dev ─────────┬── catalogs: dev_sales, dev_finance …
└── Workspace: shared-test ────────┘

• One metastore per region, catalogs as the isolation unit per environment and domain.
• Workspace-catalog binding so a dev workspace physically cannot attach production catalogs — this is what makes a single metastore safe across environments.
• A workspace per domain in production when domains have genuinely different admin, compliance or cost-ownership needs; a shared workspace when they do not. Do not create 14 workspaces reflexively.
• Separate metastores per environment only when you need hard isolation and are willing to give up cross-environment lineage and sharing — it makes promoting between environments harder, and the usual answer is catalogs plus binding instead.

The interview line: "I'd default to one metastore per region with catalogs as the isolation unit, and use workspace bindings so dev can't see prod. I'd add workspaces where there's a real administrative or regulatory boundary — not per team by default, because each workspace is recurring operational cost and the governance model already gives me the isolation."

#4. The medallion architecture — and its honest critique

SOURCES ──▶ BRONZE ──────────▶ SILVER ──────────▶ GOLD ──────▶ CONSUMPTION
            raw, append-only    cleansed,          business      BI · ML ·
            schema-on-read      conformed,         aggregates    apps · agents
            immutable history   deduplicated,      dimensional
            full fidelity       quality-enforced   models

• Bronze: exact source fidelity, append-only, with ingestion metadata (source file, ingest time, batch id). Never transform destructively here — bronze is your ability to reprocess.
• Silver: deduplicated, typed, conformed, quality-checked; the layer most engineering teams actually build against. CDC is usually applied here.
• Gold: business-level aggregates and dimensional models, serving BI, apps and agents.

The critique to have ready (it shows you have done this in anger, not read a blog):

"Medallion is a useful default and a bad religion. Three failure modes I watch for: teams create all three layers for data that needed one, so you pay three times to move a lookup table; 'silver' becomes a dumping ground with no owner; and the layer boundaries get mistaken for team boundaries, which creates hand-offs and queues. I use it as a naming convention for quality guarantees, and I'm happy to collapse layers where the guarantee is already met."

How many layers for a small source? Two is often right. Say so — it signals judgement over dogma.

#5. Data mesh on Unity Catalog

Mesh maps cleanly onto UC, which makes it a common enterprise ask.

The honest caveat to volunteer: mesh fails organisationally far more often than technically. If the domains do not have their own engineers and their own budget, you get "mesh" as a naming convention over a central team that is now also blamed for everything. Ask about funding and staffing before endorsing mesh — and say that in the interview, because it demonstrates you have seen it fail.

#6. Environments, promotion and CI/CD

The unit of promotion is code plus configuration, never data.

• Databricks Asset Bundles (DABs) — declare jobs, pipelines, clusters and permissions as YAML in the repo, deploy the same bundle to dev/test/prod with per-target overrides. This is the standard answer for "how do you do CI/CD" and you should name it immediately.
• Git folders / repo integration for source control of notebooks and code.
• Environment parity via catalogs: the same code refers to ${catalog}.schema.table, with the catalog supplied per target.
• Test data strategy: SHALLOW CLONE from production into a dev catalog gives realistic data instantly with no storage copy — subject to the data-sensitivity question, which is where masking and synthetic data come in.
• Pipeline of record: Git → CI validates and runs tests → bundle deploys to test → integration tests → promote to prod. Service principals, not personal tokens, own production deployment.

#7. Disaster recovery

A question that separates architects from engineers, because the interesting part is what isn't automatically replicated.

What you must handle explicitly:

1. Data — cloud storage replication (cross-region bucket replication) or DEEP CLONE on a schedule. Storage replication is cheaper and continuous; deep clone gives you transactionally consistent, table-level control.
2. Metastore — a UC metastore is regional. A second region needs its own metastore, and the catalog/permission structure has to be reproduced. This is the part people forget.
3. Workspace configuration — clusters, policies, jobs, permissions. Reproduce from code (asset bundles + Terraform), never by hand. If your DR plan is a runbook of UI clicks, it is not a DR plan.
4. Identity and secrets — groups, service principals, secret scopes, external credentials.
5. Orchestration state — where a failed-over pipeline resumes; streaming checkpoints are region-specific and generally need to be re-established.

Postures:

The line: "Most analytics platforms claim they need active/active and actually need pilot light with a tested restore. I'd rather have a four-hour RTO we've genuinely rehearsed than a one-hour RTO on paper that nobody has ever executed."

#8. Reference architecture — enterprise, regulated, multi-domain

The picture to be able to draw cold. Practise until it takes four minutes.

  SOURCES                 INGEST                  LAKEHOUSE (UC-governed)            CONSUME
┌───────────┐     ┌────────────────────┐    ┌──────────────────────────────┐   ┌──────────────┐
│ SaaS apps ├────▶│ Lakeflow Connect   ├───▶│  BRONZE   (raw, append)      │   │ BI / AI-BI   │
│ (SFDC,    │     │ (managed connectors)│    │     │                        │   │ dashboards   │
│  Workday) │     └────────────────────┘    │     ▼                        │   ├──────────────┤
├───────────┤     ┌────────────────────┐    │  SILVER   (conformed, DQ)    │──▶│ Genie / NL   │
│ Databases ├────▶│ CDC ingest         ├───▶│     │                        │   │ analytics    │
│ (OLTP)    │     │ (Lakeflow / CDC)   │    │     ▼                        │   ├──────────────┤
├───────────┤     └────────────────────┘    │   GOLD    (products, metrics)│──▶│ ML / agents  │
│ Files /   ├────▶┌────────────────────┐    └──────────────────────────────┘   ├──────────────┤
│ object    │     │ Auto Loader        │              │                        │ Apps / APIs  │
├───────────┤     └────────────────────┘              │                        ├──────────────┤
│ Events /  ├────▶┌────────────────────┐              │                        │ Delta Sharing│
│ Kafka     │     │ Structured Stream  ├──────────────┘                        │ to partners  │
└───────────┘     └────────────────────┘                                       └──────────────┘

     CROSS-CUTTING:  Unity Catalog (access · lineage · audit · discovery · tags/ABAC)
                     Network isolation (Private Link · egress control) · CMK · compliance profile
                     Asset Bundles + Git CI/CD · System tables → cost, usage, audit observability

Narrate it in this order: governance first (it is the spine), then ingest, then the layers, then consumption, then operations. Candidates who start at "and then we put Kafka here" lose the plot; starting with the governance spine signals enterprise thinking immediately.

#9. Questions to answer cold

1. Draw control plane vs compute plane and say what changes with serverless.
2. How many metastores for a company operating in the EU and the US, with dev/test/prod? Why?
3. What does workspace-catalog binding buy you, and what does it not protect against?
4. When would you not use three medallion layers?
5. What in a Databricks estate is not covered by cloud storage replication in a DR event?
6. How do you promote a pipeline from dev to prod without promoting data?
7. A customer wants a workspace per team, 40 teams. Talk them through it.