#Deep-Dive Q&A Bank

150 rapid-fire questions with the answer you should give — crisp, then stop. Over-answering is the most common technical-round failure; say the sharp version, then let them ask for depth.

How to drill: cover the answers. Say yours out loud. Compare. Mark anything you fumbled and revisit tomorrow. Do not read this passively — it will feel productive and change nothing.

#Spark execution (1–30)

1. Job vs stage vs task? A job is one action; stages are bounded by shuffle boundaries; a task is one partition's work in one slot.
2. What creates a stage boundary? A wide dependency — a shuffle.
3. Name five shuffling operations. groupBy, join (non-broadcast), distinct, repartition, window functions, orderBy.
4. repartition vs coalesce? repartition shuffles and can increase or decrease partitions evenly; coalesce only reduces, without a full shuffle, and can leave uneven partitions.
5. Four phases of Catalyst? Parse → analyse (catalog resolution) → optimise (rule-based, then cost-based) → physical planning and code generation.
6. What does the analyser do that the parser doesn't? Resolves names against the catalog: tables, columns, types, functions.
7. Name five logical optimisations. Predicate pushdown, column pruning, constant folding, boolean simplification, limit pushdown.
8. What is whole-stage codegen? Fusing the operators of a stage into one generated JVM method to eliminate virtual calls and keep values in registers.
9. How do you see codegen in a plan? Operators marked with * and a codegen stage id; an operator without it fell out of codegen.
10. What is Tungsten? The memory/CPU efficiency layer: binary UnsafeRow format, managed memory, cache-aware algorithms, codegen.
11. What does AQE do? Coalesces shuffle partitions, switches join strategies at runtime, and splits skewed partitions.
12. Why does AQE need a shuffle? The shuffle write is the first point with ground-truth statistics rather than estimates.
13. What can AQE not fix? Anything before the first shuffle, a fundamentally wrong plan, or bad data layout.
14. What is DPP? Dynamic partition pruning — pushing a dimension's filtered key set down to prune fact-table partitions at runtime.
15. Three join strategies and when each? Broadcast hash (one small side), shuffle hash (smaller side fits per partition), sort-merge (two large sides; the robust default).
16. How does a broadcast join fail? The "small" side isn't — driver OOM collecting it, or executor memory blown holding copies.
17. When do you get a broadcast nested loop join? No equi-join condition. Usually a query bug.
18. What is CBO and what does it need? Cost-based optimisation for join order and strategy; it needs fresh table and column statistics.
19. Execution vs storage memory? They share one region and borrow from each other; execution can evict borrowed storage, but not vice versa.
20. What is spill? Execution memory exhausted, so intermediate data is written to local disk. A slowdown, not an error.
21. Where do you see spill? Spill (memory) and spill (disk) columns in the stage's task metrics.
22. How do you detect skew? Max task duration or shuffle read far above the median in the slow stage — roughly 5–10×+.
23. Four skew remedies in order? AQE skew handling → filter junk keys → broadcast the small side → salt the hot keys.
24. What is salting? Appending a random suffix to the hot key and fanning out the other side into one copy per salt value, so the hot key spreads across tasks.
25. When would you not salt? When AQE already handles it, when you can broadcast, or when filtering a junk key fixes it — salting adds real complexity.
26. Why is collect() dangerous? It materialises the whole result in driver memory.
27. What is Photon? A vectorised C++ execution engine using columnar batches and SIMD, replacing JVM execution for supported operators.
28. When does Photon not help? UDF-heavy Python/Scala workloads, and anywhere execution isn't the bottleneck — and it bills at a higher rate.
29. What happens to unsupported operators under Photon? They fall back to the Spark engine; a query can be partly Photon.
30. What is Spark Connect? A client/server split where the client sends unresolved logical plans over gRPC — what makes thin clients and serverless practical.

#Delta Lake (31–60)

31. What is a Delta table physically? Parquet data files plus a _delta_log that defines which files are in the table.
32. Why checkpoints? So readers don't replay every commit from zero; they read the latest checkpoint plus subsequent JSON commits.
33. Name six log actions. metaData, add, remove, protocol, commitInfo, txn, cdc, domainMetadata.
34. What's in an add action beyond the path? Per-file statistics — row count, per-column min/max and null counts.
35. Why does Delta avoid LIST on object storage? The file set is metadata in the log, so no directory scan is needed.
36. How does Delta achieve atomicity? A commit succeeds only if it can create version N+1 that doesn't already exist — put-if-absent.
37. Explain OCC on conflict. The loser re-reads the new snapshot, checks whether the conflict is semantic, and retries or fails.
38. WriteSerializable vs Serializable? WriteSerializable (default) allows some concurrent appends to be reordered relative to each other; Serializable is strict. Readers always get snapshot isolation.
39. How do you reduce write conflicts? Disjoin writers across files/partitions, make merge conditions prunable, use idempotent streaming writes.
40. What is data skipping? Using per-file min/max statistics to eliminate files without reading them.
41. Why might skipping not work? Filter columns outside the indexed statistics set, or a layout where every file's range spans everything.
42. Z-order vs liquid clustering? Z-order interleaves keys into a space-filling curve and requires a full rewrite with fixed keys; liquid clusters incrementally and lets keys evolve.
43. When would you still partition? Hard physical separation — residency, partition-level retention, or an external engine depending on the layout.
44. What's the over-partitioning failure mode? High-cardinality partition columns producing masses of tiny files; metadata and task overhead swamp any pruning.
45. What are deletion vectors? A bitmap marking logically deleted rows so a small delete doesn't rewrite the whole file — merge-on-read.
46. What new maintenance do deletion vectors create? Vectors accumulate and degrade reads; OPTIMIZE/REORG ... PURGE materialises them.
47. What is Change Data Feed? Row-level change records with _change_type, commit version and timestamp, readable by range.
48. What bounds time travel? Log retention (history) and deleted-file retention (data files), and VACUUM enforcing the latter.
49. What does VACUUM actually delete? Tombstoned data files past the retention threshold — the one Delta operation that destroys data.
50. Why is there a safety check on short VACUUM retention? A long-running reader may still be reading an old snapshot.
51. What is column mapping for? Decoupling logical from physical column names, enabling rename/drop and special characters without a rewrite.
52. Shallow vs deep clone? Shallow copies metadata and references source files (instant, great for test environments); deep copies data too (the DR building block).
53. What are table features? Discrete capabilities a table declares (deletion vectors, liquid clustering, row tracking) with minimum reader/writer requirements.
54. Risk of enabling a table feature? Older clients and external engines may no longer be able to read the table.
55. What is UniForm? Exposing Iceberg (and Hudi) metadata over the same Parquet files so other engines read a Delta table without copying.
56. What does Iceberg v3 add? Deletion vectors, row lineage and the VARIANT type — closing the gap with Delta and enabling shared Parquet files.
57. Delta or Iceberg — how do you answer? Increasingly a catalog question, not a format one; pick what your engines read natively and invest in governance.
58. What is VARIANT for? Semi-structured data in an open binary format that's navigable without string parsing.
59. What is Predictive Optimization? Automatic OPTIMIZE/VACUUM/statistics maintenance on managed tables, driven by usage.
60. Why do idempotent streaming writes matter? txnAppId/txnVersion let a retried batch be recognised and not double-applied.

#Unity Catalog and governance (61–85)

61. Three-level namespace? catalog.schema.object.
62. How many metastores? One per region; a workspace attaches to exactly one.
63. Where does identity live? At the account level — users, groups, service principals via SCIM, federated into workspaces.
64. Why doesn't SELECT alone work? You also need USE CATALOG and USE SCHEMA to traverse to the object.
65. Managed vs external tables? Managed: UC controls storage, drop removes data, predictive optimisation applies. External: you control the path.
66. Default recommendation? Managed, for the automatic maintenance and simpler governance.
67. What is a storage credential? A cloud IAM identity Databricks assumes; combined with a path it forms an external location, which is the grantable unit.
68. Do users ever handle cloud keys? No — UC vends short-lived scoped credentials at query time.
69. Row filter vs dynamic view? A row filter attaches to the table, so it applies however the table is reached; a view only protects consumers who use the view.
70. What is a column mask? A UC function transforming a column value based on who is asking.
71. What is ABAC and why does it scale? Policies written against tags rather than objects — governance becomes O(policies) instead of O(objects × groups), and new data inherits it.
72. Who should own production tables? Groups, never individuals — individually-owned tables are orphaned when someone leaves.
73. What is workspace-catalog binding? Restricting which workspaces may attach a catalog — how one metastore stays safe across environments.
74. What lineage does UC capture? Table and column level automatically, across notebooks, jobs, pipelines, dashboards, and models.
75. Name four system tables. Billing/usage, access audit, query history, lineage, compute.
76. How do you prove who read a PII table last quarter? Query system.access.audit.
77. What is Delta Sharing? An open protocol for live data sharing without copying, to Databricks or any client implementing it.
78. What is OpenSharing? The 2026 Linux Foundation-hosted evolution extending Delta Sharing to AI assets, Iceberg REST clients and on-prem sources.
79. What is a clean room? Approved computation across multiple parties' data without either seeing the other's raw rows.
80. What is Lakehouse Federation? Querying external systems in place via connections and foreign catalogs, governed by UC.
81. When is federation a trap? High-volume analytical workloads — you inherit the source's performance and load its operational database.
82. What are Unity Catalog Metrics? Governed, reusable KPI definitions queryable from SQL, BI, APIs and agents — the fix for inconsistent metric definitions.
83. What are Domains? Business-aligned grouping of assets, which also scopes agent context retrieval.
84. Biggest risk in a Hive-to-UC migration? Compute access-mode remediation on older jobs — and lifting legacy ACLs instead of redesigning permissions.
85. When is a UC migration actually complete? When direct storage access is removed so the old path can't be used — governance that can be bypassed isn't governance.

#Architecture and topology (86–105)

86. Control plane vs compute plane? Control plane orchestrates (web app, APIs, scheduler, metastore service); compute plane runs the work and touches your data.
87. Where is your data? In your cloud storage account, always.
88. Classic vs serverless compute plane? Classic runs in your cloud account/VPC; serverless runs in Databricks' with NCC, egress control and private connectivity as the controls.
89. One workspace per team — good idea? Usually not by default; each is recurring operational cost. Split on genuine administrative, regulatory or cost boundaries.
90. What forces a second metastore? A second region, or a hard isolation requirement you're willing to pay for in lost lineage and sharing.
91. Medallion layers? Bronze raw/append, silver conformed and quality-enforced, gold business-level.
92. When are three layers wrong? Small or already-clean sources — you pay three times to move a lookup table.
93. What belongs in bronze metadata? Source file/offset, ingestion timestamp, batch id — so you can always reprocess and audit.
94. How do you promote to production? Databricks Asset Bundles from Git via CI, deployed by a service principal, with catalog supplied per target. Code and config promote; data does not.
95. How do you make realistic dev data? Shallow clone from production into a dev catalog — instant, no storage copy — subject to masking of sensitive data.
96. What's not covered by storage replication in DR? The metastore, workspace configuration, identity/secrets, and orchestration/checkpoint state.
97. Most common DR mistake? A runbook of UI clicks instead of infrastructure as code — and never rehearsing it.
98. Sensible default DR posture for analytics? Pilot light with a genuinely rehearsed restore, rather than a paper active/active.
99. How does data mesh map to UC? Catalog per domain, central paved road, federated governance via account groups and tag policies.
100. Why does mesh usually fail? Domains without their own engineers or budget — you get the central team with more meetings.
101. What's the unit of cost attribution? Tags on clusters, warehouses and jobs flowing into billing system tables.
102. When should tagging be designed? Before the first workspace. Retrofitting tags across an estate is miserable.
103. What is a cluster policy? A constraint template on compute — instance types, auto-termination, tags, runtime — making good behaviour the default.
104. Why are policies both cost and security controls? The same mechanism enforces tagging and auto-termination as enforces runtime versions and instance families.
105. What's the spine of an enterprise design? Governance. Narrate it first; ingest and layers follow.

#Pipelines and streaming (106–125)

106. What is Auto Loader? Incremental file ingestion with durable tracking of processed files, schema inference/evolution, and exactly-once semantics.
107. Why not just list the directory? Listing cost grows with directory size, not with new data.
108. Auto Loader's two discovery modes? Directory listing and file notification (cloud events) for very large directories.
109. Streaming table vs materialized view? Streaming table incrementally ingests append-only sources; a materialized view keeps a query's results fresh.
110. Three expectation behaviours? Warn (track only), drop row, fail the update.
111. How do you do SCD2 without a MERGE? AUTO CDC / APPLY CHANGES declaratively, with a sequencing column for out-of-order events.
112. 200 CDC tables — how do you build it? Metadata-driven: one parameterised pipeline over a control table, not 200 hand-built pipelines.
113. What's required for end-to-end exactly-once? A replayable source plus an idempotent or transactional sink, with offsets tracked in the checkpoint.
114. Is that exactly-once delivery? No — exactly-once effect on the sink.
115. What's in a streaming checkpoint? Source offsets, a write-ahead log of planned batches, and stateful operator state.
116. What does a watermark do? Bounds how late data may be, letting Spark evict old state and emit final results.
117. What happens without one? State grows without limit and the job degrades until it dies.
118. Why RocksDB for state? Keeps large state off the JVM heap, avoiding GC pressure at scale.
119. What is the AvailableNow trigger? Process everything available, then stop — streaming's bookkeeping with batch economics.
120. Why is it under-used? Most "we need streaming" requirements are actually incremental batch.
121. Question to ask before agreeing to "real-time"? What decision is made on this data, and does 5 seconds versus 5 minutes change it?
122. Small files in streaming — cause and fix? Short triggers writing frequently; fix with longer triggers, optimised writes/auto-compaction, or AvailableNow.
123. Danger after a backlog? An unbounded first batch — always bound maxOffsetsPerTrigger/maxFilesPerTrigger.
124. Keep Airflow or use Lakeflow Jobs? Keep Airflow if it's the enterprise conductor across many systems; use native jobs if the work lives in Databricks.
125. Four layers of data quality? Schema enforcement, declarative expectations, monitoring/drift detection, and producer contracts.

#Performance and cost (126–140)

126. First question when a job slows down? What changed — code, data volume, runtime, cluster config, upstream schema?
127. Second? Where is the time — one dominant stage or spread across many?
128. Most useful single number? Max ÷ median task duration in the slowest stage.
129. Second most useful? Bytes read versus bytes actually needed.
130. How do you separate a data problem from a code problem? Re-run the current code against a pinned prior version using time travel.
131. Dashboard slow, doubling the warehouse barely helped — meaning? It's I/O and layout bound, not compute bound.
132. Why is MERGE slow? Non-prunable merge condition, no deletion vectors, undeduplicated source, non-broadcastable source, or layout not aligned to the merge key.
133. Six steps of the optimisation hierarchy? Do less work → read less data → shuffle less → fix distribution → faster engine → more hardware.
134. Why is "add nodes" last? It always costs more and often fixes nothing — a skewed job gets more expensive and equally slow.
135. Biggest single enterprise cost finding? Scheduled work running on all-purpose (interactive) compute.
136. Second biggest? Idle interactive clusters with no or long auto-termination.
137. How do you answer a cost question credibly? With system-table evidence, broken down by team, workload and SKU — not opinion.
138. What should you optimise, total cost or cost per workload? Cost per workload. Rising total with falling unit cost is a successful platform.
139. How do you make cost self-regulating? Attribute it. Tagged, showback-visible spend makes teams optimise without policing.
140. Target file size after compaction? Roughly 128 MB–1 GB, auto-tuned by table size.

#AI, security and judgement (141–150)

141. Why the lakehouse for AI? The differentiator is governed context and runtime controls, not the model — everyone has the same models.
142. RAG answers are bad — first check? Retrieval, not the prompt: was the correct chunk even in the retrieved set?
143. Five things that determine RAG quality? Chunking, retrieval quality measured, permission-aware filtering, evaluation, grounding with citations.
144. Unity Catalog vs Unity AI Gateway? UC governs assets (who may use this model/tool/table); the Gateway governs interactions at runtime — spend, routing, policies, guardrails, tracing.
145. How do you stop agent spend running away? Gateway cost attribution with hard caps and smart routing, from day one rather than retrofitted.
146. When is fine-tuning justified? When the gap is behaviour or format, not knowledge — and only after a measured RAG baseline.
147. CISO says serverless leaves their VPC — response? Data stays in their storage; controls move to network connectivity configurations, serverless egress control and private connectivity.
148. How do you satisfy GDPR erasure? The delete is cheap with deletion vectors; the hard part is enumerating every copy, which is a governance problem.
149. When would you not recommend Databricks? Small data with simple BI and no AI roadmap, pure OLTP, or an organisation with no capacity to own a platform.
150. What do you say when you don't know? "I don't know that specifically — here's how I'd reason about it and how I'd verify." Then reason well.