#Flashcards
Sorted by yield — how likely the card is to matter × how likely you are to fumble it. Tier 1 is the highest-return material in this repo per minute spent.
How to use: cover the right column. Say your answer aloud. Mark ✗ on anything you fumble and re-run only the ✗ cards tomorrow. Three passes of Tier 1 beats one pass of everything.
Schedule for a ten-day sprint: Tier 1 daily. Tier 2 on days 3, 5, 7, 9. Tier 3 twice total.
#Tier 1 — daily (the 30 that carry the most weight)
| Prompt | Answer |
|---|---|
| First question when a job slows | What changed — code, data volume, runtime, cluster config, upstream schema? |
| The single most useful performance number | Max ÷ median task duration in the slowest stage |
| Skew threshold worth investigating | Roughly 5–10× max/median |
| Four skew remedies, in order | AQE skew handling → filter junk keys → broadcast small side → salt |
| Optimisation hierarchy, six steps | Do less work → read less → shuffle less → fix distribution → faster engine → more hardware |
| Why "add nodes" is last | Always costs more, often fixes nothing — a skewed job gets pricier and equally slow |
| Separate a data problem from a code problem | Re-run current code against a pinned prior version via time travel |
| What a Delta table physically is | Parquet files + _delta_log that defines which files are in the table |
| How Delta gets atomicity | Commit succeeds only if version N+1 doesn't already exist (put-if-absent) |
| Two writers, same instant | One wins N+1; the other re-reads, checks for semantic conflict, retries or fails. No corruption, no locks |
What add actions carry |
Per-file stats: row count, per-column min/max, null counts — the basis of data skipping |
| Why Delta avoids LIST on object storage | The file set is metadata in the log, not a directory scan |
| Z-order vs liquid clustering | Z-order: full rewrite, fixed keys, space-filling curve. Liquid: incremental, evolvable keys |
| Over-partitioning failure mode | High-cardinality partition column → masses of tiny files; overhead swamps pruning |
| Deletion vectors | Bitmap marking deleted rows so small deletes don't rewrite whole files (merge-on-read) |
| Four phases of Catalyst | Parse → analyse → optimise → physical plan + codegen |
| Three things AQE does | Coalesce shuffle partitions, switch join strategy, split skewed partitions |
| What AQE cannot fix | Anything before the first shuffle, a fundamentally wrong plan, bad layout |
| Why AQE needs a shuffle | Shuffle write is the first point with ground truth instead of estimates |
| Three join strategies | Broadcast hash (one small side), shuffle hash, sort-merge (robust default) |
| Execution vs storage memory | Share one region, borrow from each other; execution can evict borrowed storage, not vice versa |
| What spill is | Execution memory exhausted → intermediate data to local disk. Slowdown, not an error |
| When Photon doesn't help | UDF-heavy work, or where execution isn't the bottleneck — and it bills at a higher rate |
| Three-level namespace | catalog.schema.object |
| Why SELECT alone fails | Also need USE CATALOG and USE SCHEMA to traverse |
| Why ABAC scales | Policies on tags → O(policies), not O(objects × groups); new data inherits governance |
| Control plane vs compute plane | Control plane orchestrates; compute runs work and touches data. Data always in your storage |
| UC vs Unity AI Gateway | UC governs assets; the Gateway governs interactions at runtime — spend, routing, policies, guardrails, tracing |
| RAG is bad — first check | Retrieval, not the prompt: was the right chunk even retrieved? |
| When NOT to use Databricks | Small data + simple BI + no AI roadmap; pure OLTP; no capacity to own a platform |
#Tier 2 — every other day
| Prompt | Answer |
|---|---|
| What bounds time travel | Log retention (history) + deleted-file retention (data), enforced by VACUUM |
| What VACUUM destroys | Tombstoned files past retention — the one Delta op that actually deletes data |
| Managed vs external tables | Managed: UC owns storage, drop removes data, predictive optimisation applies. Default to managed |
| What a storage credential is | A cloud IAM identity Databricks assumes; + path = external location, the grantable unit |
| Do users handle cloud keys? | No — UC vends short-lived scoped credentials at query time |
| Row filter vs dynamic view | Row filter attaches to the table so it applies however the table is reached |
| Who should own prod tables | Groups, never individuals — individual ownership orphans on departure |
| Workspace-catalog binding | Restricts which workspaces can attach a catalog — how one metastore stays safe across environments |
| How many metastores | One per region; second region or hard isolation forces another |
| What's NOT in storage replication for DR | Metastore, workspace config, identity/secrets, orchestration and checkpoint state |
| Unit of promotion | Code + config via asset bundles. Never data |
| Realistic dev data | Shallow clone from prod — instant, no storage copy — subject to masking |
| What Auto Loader solves | Listing cost grows with directory size, not new data; it tracks processed files durably |
| Streaming table vs materialized view | Streaming table ingests append-only sources; MV keeps a query's results fresh |
| Three expectation behaviours | Warn (track), drop row, fail update |
| SCD2 without a MERGE | AUTO CDC / APPLY CHANGES with a sequencing column |
| 200 CDC tables — how | Metadata-driven: one parameterised pipeline over a control table |
| Exactly-once requires | Replayable source + idempotent/transactional sink, offsets in the checkpoint |
| Is it exactly-once delivery? | No — exactly-once effect on the sink |
| What a watermark does | Bounds lateness so state can be evicted and results finalised |
| No watermark ⇒ | Unbounded state growth; the job slowly dies |
AvailableNow trigger |
Process all available then stop — streaming bookkeeping, batch economics |
| Question before agreeing to "real-time" | What decision does this drive, and does 5s vs 5min change it? |
| Why MERGE is slow | Non-prunable condition, no deletion vectors, undeduplicated source, layout not on merge key |
| Biggest enterprise cost finding | Scheduled work on all-purpose (interactive) compute |
| Second biggest | Idle interactive clusters with no auto-termination |
| Optimise total cost or unit cost? | Cost per workload. Rising total + falling unit cost = a successful platform |
| How cost becomes self-regulating | Attribute it — tagged showback makes teams optimise without policing |
| Cluster policies are | Constraints on compute: instance types, auto-termination, tags, runtime. Both a cost and a security control |
| Serverless network controls | Network connectivity configurations, serverless egress control, private connectivity to storage |
| Where the UC migration actually hurts | Compute access-mode remediation on older jobs |
| When a UC migration is done | When direct storage access is removed so the old path can't be used |
| Five things that determine RAG quality | Chunking, measured retrieval, permission-aware filtering, evaluation, grounding/citations |
| When fine-tuning is justified | Gap is behaviour/format, not knowledge — and only after a measured RAG baseline |
| GDPR erasure in a lakehouse | Delete is cheap with deletion vectors; the hard part is enumerating every copy |
#Tier 3 — twice before the interview
| Prompt | Answer |
|---|---|
| Lakehouse//RT and Reyden | Real-time SQL warehouse type; ~10ms small / sub-100ms large, ~12k QPS claimed; on governed Delta/Iceberg, no copy. Beta, read-only |
| LTAP | Operational data written directly to Delta/Iceberg in UC — analytics with no CDC pipeline, no second copy. Announced, coming |
| Lakebase | Serverless Postgres on lake storage, UC-governed; sub-500ms creation and branching; cross-cloud DR |
| Iceberg v3 (GA) | Deletion vectors, row lineage, VARIANT; Delta and Iceberg can share Parquet files |
| Unity Catalog Metrics | Governed reusable KPI definitions across SQL, BI, APIs and agents — fixes inconsistent metrics |
| Domains | Business-aligned asset grouping; also scopes agent context retrieval |
| Omnigent | Open-source meta-harness above agent harnesses: composition, control, collaboration |
| OpenSharing | Linux Foundation evolution of Delta Sharing — AI assets, Iceberg REST clients, on-prem |
| Lakewatch | Agentic SIEM on open lakehouse format |
| Old name → new | DLT → Lakeflow Declarative Pipelines; Workflows → Lakeflow Jobs; online tables → Lakebase |
| UniForm | Iceberg/Hudi metadata over the same Parquet files — read a Delta table as Iceberg, no copy |
| Delta or Iceberg? | A catalog question now, not a format one. Pick what your engines read; invest in governance |
| Table feature risk | Enabling one can make the table unreadable to older clients and external engines |
| Column mapping | Decouples logical from physical names — enables rename/drop without rewrite |
| Shallow vs deep clone | Shallow: metadata only, instant. Deep: copies data, the DR building block |
| DPP | Dynamic partition pruning — dimension's filtered keys prune fact partitions at runtime |
| Spark Connect | Client sends unresolved logical plans over gRPC — enables thin clients and serverless |
| Federation's trap | High-volume analytics — you inherit source performance and load an operational DB |
| Why mesh fails | Domains without their own engineers and budget |
| Four value levers | Cost reduction, productivity, revenue enablement, risk reduction |
| Which levers to lead with | Cost and risk (verifiable); be honest that revenue attribution is hard |
| Biggest risk to a migration business case | Not switching the legacy platform off — tie funding to decommissioning milestones |
#Write your own
The best cards are the ones you got wrong. After every drill session, add three:
| Prompt | Answer |
|---|---|