#Architecture Design Scenarios

Twelve scenarios of the kind used in senior Databricks loops. Work each one before reading the model answer — reading first converts a drill into a lecture and teaches you nothing.

Method: 25 minutes each. Talk out loud. Draw on paper. Use CLAIM: Context (ask) → Levers → Architecture → Implications → Measure.

Universal opening — the discovery set. Before any design, ask some subset of: volumes and growth · latency/freshness SLO and the decision it drives · number and skill of users · regulatory posture and data residency · incumbent estate and contracts · budget and timeline · who operates it after go-live · what "success" is measured by in 12 months.

Never more than five before you start designing — discovery that never converges is its own failure.

#Scenario 1 — Global retailer, Hadoop to lakehouse

"5,000-node on-prem Hadoop cluster. 8 PB. 2,000 Hive/Spark jobs, some written a decade ago. Oracle and Teradata warehouses alongside. 40 data engineers. The contract renewal is in 18 months and the board has approved a cloud migration. Design it and tell me how you'd sequence it."

Discovery: Which of the 2,000 jobs actually matter (usually 20% carry the value)? What is the actual data-transfer constraint — bandwidth and time to move 8 PB? Is Teradata in scope or a follow-on? What is the regulatory data-residency position? Which teams own which pipelines? Is the 18 months hard (contract) or soft?

Levers:

1. Migration pattern — lift-and-shift vs re-architect vs strangler. Answer: strangler, almost always. Big-bang migrations of 2,000 jobs fail publicly.
2. Data movement strategy — 8 PB does not move over the internet on a whim. Physical transfer appliances for the historical bulk; dual-write or replication for the active tail.
3. Governance-first or workload-first — governance first, or you will migrate the mess.

Architecture:

• Phase 0 (weeks 1–8): foundation. Landing zone, networking, UC metastore, catalog and naming standards, account groups from the IdP, cluster policies, tagging taxonomy, CI/CD with asset bundles. Nothing migrates until the paved road exists.
• Phase 1: assess and triage. Inventory with lineage; classify jobs into retire (frequently 30–40% are dead or duplicated — this is the cheapest win in the programme), rehost (mechanical translation), re-architect (the valuable ones), retain (out of scope).
• Phase 2: data. Historical bulk via physical transfer into bronze; active data dual-written or CDC-replicated so both sides are live during transition. Lakehouse Federation over Teradata/Oracle so consumers can be pointed at Databricks early even before the data moves — this is the trick that decouples consumer migration from data migration.
• Phase 3: workload waves. Migrate by business domain, not by technology. Each wave: migrate, run in parallel, reconcile outputs, cut over, decommission. Parallel-run with automated reconciliation is the control that makes cutover politically possible.
• Phase 4: decommission — and be ruthless about it. Un-decommissioned legacy is the single biggest cause of migrations that never finish and never bank the savings.

Implications: the risk is not technical, it is the tail — the last 15% of jobs with no owner and no tests. Budget disproportionate time for it. Also: 40 engineers who know Hadoop need to become 40 engineers who know Databricks; enablement is a workstream, not an afterthought.

Measure: jobs migrated vs retired; cost per workload before/after; parallel-run reconciliation pass rate; legacy footprint actually switched off; engineer certification counts.

The senior line: "The 18-month contract date is the real architecture constraint. I'd work backwards from it and be explicit that anything not migrated by month 14 gets an extension or a retirement decision — because the failure mode here isn't a bad design, it's running two platforms for three years and banking none of the savings."

#Scenario 2 — Bank, regulated, multi-region

"Tier-1 bank. Data must not leave its region. Three regions. Regulator requires full audit and provable access control. Risk teams need PII; analysts must not see it. Design the platform."

Discovery: Which regulations specifically (GDPR, local banking rules, Schrems II)? Does "not leave the region" include metadata? Are there global reporting requirements that need aggregates to cross regions? Who is the accountable data owner per domain? Existing IdP and privileged-access tooling?

Levers: metastore-per-region is forced by residency · whether aggregates may cross regions (usually yes, with approval — this unblocks global reporting) · fine-grained access approach (ABAC vs per-object grants) · classic vs serverless compute given network policy.

Architecture:

• One metastore per region, no cross-region sharing of raw data. Regional workspaces.
• Global reporting via approved, aggregated, de-identified extracts shared using Delta Sharing from each region into a reporting catalog — the aggregate crosses, the raw does not.
• Access control: account groups from the IdP; ABAC with tags (pii, classification) driving column masks and row filters, so a new table tagged pii inherits protection automatically. Risk users are in a group exempted by policy; analysts see masked values.
• Network: customer-managed VPC, no public IPs, front- and back-end Private Link, egress restricted; CMK for managed services and storage; compliance security profile enabled.
• Audit: system.access.audit streamed to the SIEM; alerting on privilege change, bulk export, and access to pii-tagged assets; lineage retained as regulatory evidence.
• Segregation of duties: service principals deploy to production; no human has standing write access to production data; break-glass is time-boxed and alerted.

Implications: three of everything means three times the operational surface — invest heavily in IaC and asset bundles so the regions are identical by construction, not by discipline. Per-region cost attribution needs the tagging taxonomy from day one.

Measure: time to produce an access-evidence report (target: minutes, from system tables); number of standing privileged accounts (target: zero); config drift between regions (target: zero, enforced by CI).

#Scenario 3 — The slow dashboard

"Executive dashboard takes 45 seconds to load. They doubled the warehouse size and it's now 40 seconds. What's going on?"

The doubling detail is the tell: near-linear insensitivity to compute means compute is not the bottleneck.

Evidence: query profile — files pruned vs scanned, bytes read vs result size, queue time vs execution time, spill, Photon vs fallback. Then: is it one query or twelve? Is time in the warehouse at all, or in the BI tool rendering?

Likely causes, ranked: dashboard queries hitting raw/bronze tables with no aggregation · no effective data skipping (no clustering on filter columns) · a dozen separate queries where one modelled table would do · queuing under concurrency rather than slow individual queries · the BI tool re-querying instead of using cached results.

Fix: a modelled gold layer with materialised aggregates for the dashboard's actual shapes; liquid clustering on the real filter columns; separate the BI warehouse from ETL; serverless with appropriate sizing and auto-stop. Then revisit sizing — downward, probably.

Prevent: a standard that dashboards never query bronze; monthly review of the top 20 queries by total cost; a "dashboard SLO" owned by the platform team.

The line: "Doubling the warehouse and getting 11% back tells us we're I/O and layout bound, not compute bound. I'd rather halve the data read than double the machine — one of those makes the bill smaller."

#Scenario 4 — Real-time customer-facing analytics

"We want to show merchants a live dashboard of their sales. 50,000 merchants, up to 5,000 concurrent users, sub-second page loads, data no more than a minute old."

Discovery: is "sub-second" the query or the page? What query shapes — fixed set or ad-hoc? Read-only? What happens if it's 5 seconds stale — does a merchant make a different decision?

Levers: serving-layer choice (duplicate into a specialised store vs serve from the lakehouse) · pre-aggregation strategy · multi-tenancy and per-merchant isolation.

Architecture:

• Events → streaming ingest → bronze → incremental aggregation into a per-merchant pre-aggregated gold table (the query shapes are fixed, so pre-aggregate ruthlessly; this is the highest-leverage decision).
• Serving: the classical answer duplicates into Pinot/Druid/ClickHouse. The 2026 answer is to evaluate Lakehouse//RT, which targets ~10–100 ms on governed Delta/Iceberg with flat latency at high concurrency — no copy, no separate governance. (Databricks) Caveat it honestly: beta, read-only at launch — benchmark on their query shapes before committing.
• Row-level isolation by merchant via row filters, so multi-tenancy is enforced by the platform rather than by every application query remembering a WHERE clause.
• Freshness via streaming with a short trigger, or real-time mode if genuinely needed.

Implications: if Lakehouse//RT doesn't meet the benchmark, the fallback is a serving store and you accept the duplication — design the pipeline so that swap is a sink change, not a re-architecture.

Measure: p50/p95/p99 query latency at target concurrency; freshness lag; cost per thousand dashboard loads.

#Scenario 5 — The GenAI mandate

"Our CEO wants AI agents across the business in six months. We have 400 TB across 30 systems, mostly ungoverned. Where do you start?"

The answer is not "start with agents." But do not simply refuse the mandate — reframe it into something deliverable, and deliver something visible fast.

Discovery: which business process, specifically, and what does success look like in money or hours? Who owns the data for that process? What is the appetite for a wrong answer reaching a customer?

Architecture and sequence:

1. Pick one high-value, low-blast-radius use case and instrument it end to end. Internal knowledge assistants and support-agent copilots are the usual best first bets — high value, low regulatory exposure, tolerant of imperfection because a human is in the loop.
2. Governed context for that domain only — the subset of the 400 TB that use case needs, into UC with permissions, lineage, tags. Do not attempt to govern 30 systems first; that programme takes two years and delivers nothing in six months.
3. Retrieval + evaluation harness before prompts. Ground-truth question set, recall@k measured, LLM judges, MLflow tracing. The eval harness is the deliverable that makes iteration possible.
4. Runtime governance from day one — Unity AI Gateway for spend caps, routing, guardrails and tracing. Retrofitting cost control after ten teams are live is painful.
5. Then scale: Domains and Metrics to scope context; Genie for analytics; agents where a process genuinely benefits.

Implications: be explicit with the CEO that the constraint is governed context, not models — and that the six-month outcome is one process transformed and a platform others can build on, not agents everywhere. Frame it as ambitious sequencing, not as saying no.

Measure: for the use case — deflection rate, handling-time reduction, answer faithfulness score, cost per interaction. For the platform — time for a second team to ship their use case (should be weeks, not months). That second number is the real proof of a platform.

#Scenario 6 — Cost crisis

"Databricks spend went from £80k to £400k a month in a year. The CFO wants it halved by next quarter. Go."

Do not start cutting. Start measuring — and say so.

Evidence (week 1): system.billing.usage joined to tags, workspace, SKU and job. Produce: spend by team, by workload, by SKU; top 20 jobs by cost; all-purpose vs jobs compute split; idle time; warehouse utilisation; serverless vs classic mix. You cannot manage what isn't attributed.

The usual findings, in order: scheduled work on all-purpose clusters · interactive clusters with no auto-termination · always-on warehouses that should be serverless with auto-stop · full refreshes that should be incremental · layout problems making every query read too much · oversized clusters chosen by superstition · ungoverned AI spend.

Sequence: quick wins first (policies enforcing auto-termination and jobs compute; auto-stop on warehouses) — typically 15–30% with no engineering. Then the structural work (incremental processing, clustering, predictive optimisation). Then the governance that stops regression: tagging enforced by policy, budget policies, per-team showback, and a monthly cost review.

The reframe for the CFO: "I can get a fast 20–30% from configuration. Beyond that, I'd want to change the question from 'is Databricks expensive' to 'which use cases are worth what they cost' — because once spend is attributed to product teams, they optimise without me policing anything. If we halve it by cutting capacity we'll be back here in six months with angry users."

Measure: cost per workload, not total cost. Total cost rising while cost per workload falls is a successful platform — make sure the CFO understands that before the next board meeting.

#Scenario 7 — Streaming CDC at scale

"200 source tables in Oracle. Need them in the lakehouse, no more than five minutes stale, with full history for audit. Design the ingestion."

Discovery: true five-minute need, or "as fresh as possible"? Schema change frequency? Deletes in source — hard or soft? Is history needed as SCD2 or just as an audit log? Source DB load tolerance?

Architecture:

• CDC capture — managed Lakeflow Connect connector where available, otherwise Debezium → Kafka. Log-based capture, never query-based polling (polling misses deletes and hammers the source).
• Bronze: append-only raw change records with operation type and source LSN/SCN, plus ingestion metadata. This is the audit history — never mutate it.
• Silver: AUTO CDC / APPLY CHANGES with a sequencing column to handle out-of-order events, producing current-state (SCD1) and/or history (SCD2) declaratively. Do not hand-write 200 merges.
• Scale pattern: metadata-driven. One parameterised pipeline definition over a control table of 200 entries — not 200 hand-built pipelines. This is the answer that distinguishes an architect.
• Schema evolution: additive changes handled automatically; breaking changes routed to a quarantine path and alerted rather than failing the pipeline.

Implications: five minutes across 200 tables means the merge cost matters — deletion vectors, clustering on merge keys, and grouping small tables into shared pipelines to avoid 200 sets of compute overhead.

Measure: end-to-end lag per table (p50/p95); reconciliation counts vs source; schema-change incidents caught vs missed.

#Scenario 8 — Data mesh for 14 domains

"14 business domains, each with their own analysts, all complaining the central team is a bottleneck. Leadership read about data mesh. Design the operating model."

The trap: this is an org question wearing a technology costume. Answer it as one.

Discovery — ask these before endorsing anything: do domains have their own engineers, or just analysts? Who funds them? Who is accountable when a domain's data product breaks at 2am? What is the central team's actual bottleneck — building pipelines, or approving access?

Architecture:

• Catalog per domain, owned by the domain, with domain groups as owners.
• Central platform team owns the paved road: cluster policies, asset bundle templates, CI/CD, base pipelines, the tagging taxonomy, and the governance policies — not the pipelines themselves.
• Federated governance: global policies (ABAC on classification tags, retention, naming) enforced centrally; local decisions (schema, modelling, SLAs) made by domains.
• Data products with owners, documentation, quality expectations and published SLAs; discovery via UC search, Domains, and Business Glossary.
• Interoperability: shared conformed dimensions and governed Metrics, or every domain invents its own definition of "customer" and the mesh becomes 14 silos with better branding.

Implications — say this plainly: "Mesh fails organisationally far more often than technically. If the domains don't have engineers and budget, you get the same central team with more meetings and a new vocabulary. I'd want to see at least two domains staffed and funded before we call it a mesh — and I'd start with those two rather than all 14."

Measure: time from a domain's idea to a published data product; central team's share of tickets that are approvals vs builds (should fall); number of duplicate definitions of core entities.

#Scenario 9 — Multi-cloud mandate

"Board wants no single-cloud lock-in. We're on AWS; they want Azure capability. What do you do?"

Challenge the premise politely, then serve it.

Discovery: what risk is the board actually pricing — vendor failure, commercial leverage, a regulatory requirement, or an acquisition? Each implies a different answer, and usually it is commercial leverage.

The honest analysis: true active/active multi-cloud roughly doubles operational cost and complexity, and the egress economics of moving data between clouds are punishing. What people usually want is portability (the ability to move without a rewrite), not simultaneity.

Architecture for portability: open table formats (Delta/Iceberg) so the data is not trapped · Unity Catalog as the governance layer, consistent across clouds · infrastructure and jobs as code (Terraform + asset bundles) so an environment can be rebuilt elsewhere · avoid deep cloud-native service coupling in pipelines · Databricks itself runs on all three, which is a genuine part of the answer.

Architecture if they truly need multi-cloud: run distinct workloads per cloud with a clear allocation rule, not a mirror; share via Delta Sharing rather than replicate; accept and budget the duplicated operational cost explicitly so it is a decision, not a surprise.

The line: "I can give the board portability for maybe 10% extra effort, and simultaneity for roughly 100%. Let me show them what each buys, and let them choose with the number in front of them."

#Scenario 10 — Rescuing a failing programme

"Eighteen months in. £4m spent. The business says nothing has been delivered. You're brought in. First 30 days?"

Days 1–10 — listen and measure. Interview stakeholders on all sides, separately. Get the facts from system tables: what actually runs, what is used, what it costs, who logs in. Find out what the business asked for originally and compare it to what is being built — the gap is usually the whole story.

Days 10–20 — diagnose. Sort the failure into its real category, because the fix differs entirely:

• Scope — building a platform when the business wanted three reports.
• Sequencing — foundations for two years with nothing visible.
• Skills — a team learning on a critical path.
• Ownership — no one empowered to decide, so nothing is decided.
• Technology — genuinely wrong choice. This is the rarest, and the one everyone blames.

Days 20–30 — re-plan, visibly. Pick one business outcome deliverable in 6–8 weeks and ship it. Kill or park everything not on that path. Publish a plan with dates and an honest statement of what was wrong. Re-establish trust with delivery, not with a roadmap.

The line: "Failing programmes are almost never rescued with better architecture. They're rescued by shipping something the business recognises as valuable, quickly, and rebuilding credibility from there. My first deliverable is a win, not a design."

#Scenario 11 — Unity Catalog migration under pressure

"400 tables, legacy Hive metastore, 60 jobs, three teams, and an audit finding that says we can't prove who accesses PII. Six weeks. Go."

The audit finding is the lever — it gives you the mandate. Use it.

Sequence: foundation (metastore, groups from IdP, catalogs, external locations) → inventory and classify, tagging PII first (the audit is about PII, so lead with it) → SYNC external tables and choose managed vs external per table → remediate compute access modes (the real work; find this early) → redesign permissions around groups rather than lifting legacy ACLs → cut over jobs in waves → remove direct storage access so the old path cannot be used.

The six-week reality check to state: "Six weeks is realistic for the PII-tagged subset and the audit evidence, which is what the finding actually requires. All 400 tables and 60 jobs is a three-to-four month programme, mostly because of access-mode remediation on the older jobs. I'd propose we close the finding in six weeks and land the rest on a published plan — that's a better conversation with the auditor than missing a date."

Pushing back with a credible alternative scores far better than accepting an impossible date.

#Scenario 12 — The build-versus-buy challenge

"Our engineering team says they can build this on open-source Spark and Iceberg on Kubernetes for a fraction of the licence cost. Why are we paying Databricks?"

Take the argument seriously; it is often made by good engineers, and dismissing it loses them.

Concede what is true: the components are genuinely open source. A strong team can absolutely build it. Spark, Iceberg and the surrounding tooling are mature. For some organisations — ones with deep platform engineering and unusual requirements — building is the right answer, and saying so is what makes the rest of your argument credible.

Then name what the comparison usually omits:

• The licence is a fraction of TCO. The dominant cost is engineering time — who is on call for the platform, who upgrades Spark, who builds the governance layer, who debugs the state store at 3am.
• Governance is the hard part, not compute. Unity Catalog-equivalent — lineage, ABAC, audit, sharing, AI asset governance — is years of work and is never finished.
• Opportunity cost: engineers building a platform are not building products. The question is not "can we?" but "is this our differentiation?"
• Key-person risk: the three people who understand the bespoke platform will not all still be there in three years.

The honest close: "I'd put real numbers on both, including two or three fully-loaded platform engineers and an on-call rota, over a three-year horizon. If building still wins, build — I've seen organisations where it does. In my experience it wins less often than the licence line suggests, because the licence is the visible cost and the engineering is the invisible one."

#How to self-assess

After each scenario, score yourself:

Anything below 3 on Discovery or Trade-offs is the highest-yield thing you can fix.