Lakehouse Prepchief architect

#Security, Networking and Compliance

"Design secure, scalable and high-performing Lakehouse platforms." In enterprise loops there is usually a dedicated security interviewer, and they are checking whether you have actually deployed into a regulated environment or only read about it.

#1. The layered model

Present security in layers. It is how security architects think, and it keeps you from producing an unstructured list of features.

1. IDENTITY        SSO/SAML · SCIM provisioning · account-level groups · service principals (OAuth M2M)
2. NETWORK         no public IPs · customer-managed VPC · Private Link · egress control · IP access lists
3. COMPUTE         access modes · cluster policies · runtime versions · no-code-execution boundaries
4. DATA            Unity Catalog grants · row filters & column masks · ABAC tags · credential vending
5. ENCRYPTION      in transit · at rest · customer-managed keys (managed services + storage)
6. AUDIT           system.access.audit · lineage · query history · alerting
7. COMPLIANCE      compliance security profile (HIPAA · PCI-DSS · FedRAMP) · enhanced monitoring
8. AI RUNTIME      Unity AI Gateway policies · guardrails · MCP registry · agent tracing

Layer 8 is new and worth naming unprompted — it shows you are current.

#2. Identity

  • SSO/SAML for humans; SCIM to provision users and groups from the IdP (Entra ID, Okta) at the account level, federating into workspaces.
  • Groups are the unit of everything. Grant to groups, own objects with groups. Sourced from the IdP so joiners/movers/leavers are handled by an existing process rather than by a Databricks admin remembering.
  • Service principals with OAuth (M2M) for automation. Personal access tokens in production are a finding — a person leaves, production breaks, and the audit trail attributes machine activity to a human. Say this; it is a credible, specific opinion.
  • Least privilege on cloud identities too — the storage credential's IAM role should reach only the paths it must.

#3. Network

Classic compute (your VPC):

  • Secure cluster connectivity — cluster nodes have no public IP addresses and open no inbound ports; they initiate outbound connections to the control plane.
  • Customer-managed VPC — deploy into your own network with your subnets, routing and security groups, satisfying enterprise network standards.
  • Private Link — private connectivity for the front-end (users → workspace) and back-end (compute → control plane), keeping traffic off the public internet.
  • Egress controls — firewall/NAT rules restricting where clusters may call out, preventing exfiltration and unapproved package downloads.
  • IP access lists — restrict workspace access to corporate ranges/VPN.

Serverless compute: the controls move but do not disappear — network connectivity configurations (NCCs) to reach your resources privately, serverless egress control to constrain outbound destinations, and private connectivity to your storage. When a customer says "serverless means we lose network control", this is the correction, and it is a very common objection worth rehearsing (see the objections drill).

#4. Compute and code-execution boundaries

Access modes matter for security, not just compatibility. Unity Catalog enforces fine-grained permissions only where user code is properly isolated; a shared/standard access-mode cluster enforces row filters and column masks correctly, whereas legacy no-isolation modes cannot — which is exactly why the UC migration forces access-mode changes.

Cluster policies are the enterprise control surface: constrain instance types, enforce auto-termination, mandate tags, fix runtime versions, cap autoscaling, require Photon or not. Policies are how you make good behaviour the default rather than a training problem.

"Policies are where cost control and security control are the same lever. A policy that mandates tags and auto-termination is a FinOps control; the same policy pinning runtime versions and instance families is a security control."

#5. Encryption and keys

  • In transit: TLS throughout, including intra-cluster encryption options.
  • At rest: cloud-native encryption by default.
  • Customer-managed keys (CMK): for managed services (notebooks, secrets in the control plane) and for workspace/DBFS and managed storage. Gives you revocation as a control — the real reason regulated customers want it.

The honest framing when a customer demands CMK: "Ask what threat it addresses. If the answer is 'we need to be able to revoke access independently of the provider', CMK is exactly right. If the answer is 'the policy says so', we should still do it, but let's not pretend it changes the risk model much." Candid, and it lands well with security people.

#6. Audit and monitoring

  • system.access.audit — who did what, when, from where. The answer to every auditor question.
  • Lineage — regulatory evidence for data flow and impact analysis.
  • Query history / system.query.history — what was run, by whom, at what cost.
  • Enhanced security monitoring — additional hardening and monitoring for regulated workloads.
  • Alerting — the point of audit data is detection, not archaeology. Build alerts on privilege escalation, mass export, unusual access to tagged-sensitive data, and failed access spikes.

Lakewatch (2026) extends this direction: an agentic SIEM keeping security data in open lakehouse format, with agents handling detection authoring, alert triage and threat hunting. (Summit 2026) It is also a good talking point for the "why keep security data in the lakehouse" conversation: SIEM ingestion pricing causes teams to drop data, which creates detection blind spots.

#7. Compliance

The compliance security profile enables additional hardening and controls for regimes such as HIPAA, PCI-DSS and FedRAMP, including enforced monitoring and restricted configurations. What matters in interview is not reciting the list but knowing the shape of a compliance conversation:

  1. Which regime, and which controls actually apply to this workload? (Most of the regime will not.)
  2. Data residency — where must data physically stay? Drives region and metastore design.
  3. Right to erasure (GDPR) — how do you delete a person from a lakehouse? Deletion vectors make the delete cheap; the hard part is finding every copy. Governed lineage plus limiting uncontrolled copies is the real answer.
  4. Retention — both minimum (keep for seven years) and maximum (delete after N). Note these conflict, and the conflict is resolved by policy, not technology.
  5. Segregation of duties — who can grant, who can deploy, who can read production data. Service principals and group-based ownership.
  6. Evidence — can you produce it on demand without a project? System tables, lineage, audit.

The GDPR answer worth rehearsing: "Technically, erasure is a DELETE, and with deletion vectors it's cheap. Architecturally, the hard question is enumeration — how confident are we that we know every place that person's data landed? That's a governance question, and it's why I'm strict about uncontrolled extracts. Every CSV someone dropped on a share drive is an erasure obligation you can't discharge."

#8. The regulated reference deployment

Be able to draw this.

     Corporate network                    YOUR CLOUD ACCOUNT
   ┌──────────────────┐        ┌──────────────────────────────────────┐
   │  Users (SSO/IdP) │        │  Customer-managed VPC                │
   │        │         │        │  ┌────────────────────────────────┐  │
   └────────┼─────────┘        │  │ Clusters: no public IPs        │  │
            │ front-end        │  │ Egress via NAT + firewall      │  │
            │ Private Link     │  └───────────────┬────────────────┘  │
            ▼                  │                  │ back-end          │
   ┌──────────────────┐        │                  │ Private Link      │
   │ Databricks       │◀───────┼──────────────────┘                   │
   │ CONTROL PLANE    │        │  ┌────────────────────────────────┐  │
   │ + UC metastore   │        │  │ Storage (CMK, private endpoint)│  │
   └──────────────────┘        │  └────────────────────────────────┘  │
                               └──────────────────────────────────────┘
   Controls: IP access lists · SCIM groups · cluster policies · UC grants + ABAC
             audit → SIEM/Lakewatch · compliance security profile · CMK

#9. Questions to answer cold

  1. Where does customer data live, and what is in the control plane?
  2. A CISO says serverless is unacceptable because compute leaves their VPC. Respond.
  3. Front-end vs back-end Private Link — what does each protect?
  4. Why does Unity Catalog constrain which cluster access modes you can use?
  5. How do you satisfy a GDPR erasure request across a lakehouse?
  6. How do you prove to an auditor who accessed a PII column last quarter?
  7. What are cluster policies and why are they both a cost and a security control?
  8. What governs what an agent is permitted to do at runtime?