#Delta Lake Internals

The second named JD bullet. Depth here separates people who use Delta from people who can reason about it when it misbehaves.

#1. The transaction log is the table

A Delta table is: Parquet data files + a _delta_log/ directory that is the single source of truth about which files belong to the table.

my_table/
├── part-00000-....snappy.parquet          ← data (immutable once written)
├── part-00001-....snappy.parquet
└── _delta_log/
    ├── 00000000000000000000.json          ← commit 0  (atomic unit)
    ├── 00000000000000000001.json          ← commit 1
    ├── ...
    ├── 00000000000000000010.checkpoint.parquet   ← periodic state rollup
    └── _last_checkpoint                   ← pointer so readers skip replaying from zero

Reading a table = find the latest checkpoint, replay the JSON commits after it, and compute the current set of active files. That is why Delta avoids the expensive LIST operations that plague Hive-style tables on object storage: the file list is metadata, not a directory scan.

#Actions inside a commit

The add action's statistics are the performance story. Each records row count and per-column min, max and null counts — by default for a bounded number of leading columns (commonly the first 32, configurable). These stats drive data skipping: a query with WHERE event_date = '2026-09-01' consults the log, eliminates every file whose min/max range excludes that value, and never opens them.

Consequence worth saying aloud: "If your filter columns aren't in the statistics set, or your data is laid out so that every file's min/max spans the whole range, you get no skipping — you read the whole table no matter how clever the query is. Layout is what makes skipping work."

#2. ACID via optimistic concurrency control

Delta does not lock. Each writer:

1. Reads the current table version N and records what it read.
2. Does its work, writing new data files (no visible effect yet).
3. Attempts to commit its actions atomically as version N+1 — succeeding only if N+1 does not already exist ("put-if-absent").
4. If someone else got there first, it re-reads, checks whether the conflict is real, and retries.

Conflict resolution is semantic, not blind:

• Two blind appends → generally compatible, retry succeeds.
• A concurrent write that deleted files this transaction depended on → genuine conflict, may fail.
• MERGE against overlapping data → the classic real conflict.

Reducing conflicts in practice: partition or otherwise disjoin writers so they touch different files; make MERGE conditions selective enough to prune; serialise pipelines that genuinely contend on the same rows; use idempotent writes (txnAppId/txnVersion) for streaming.

Isolation levels:

• WriteSerializable (default) — serialisable for writes that conflict, but permits some concurrent appends to be reordered relative to each other. Higher throughput; the right default.
• Serializable — the strictest; all operations behave as if serially executed.
• Readers always get snapshot isolation: a query reads one consistent version and is not affected by concurrent writes.

Asked: "Two writers commit to the same table at the same instant. What happens?" Exactly one wins the N+1 slot. The other detects the collision, reloads the new snapshot, checks whether its work actually conflicts with what changed, and either retries transparently or fails with a concurrent-modification error. No data is corrupted and no lock was taken.

#3. Data layout — the lever that matters most

Four mechanisms, and knowing when each applies is a reliable senior signal.

#Partitioning (physical directories)

Classic Hive-style. Good for a low-cardinality, frequently-filtered column (typically a date). The failure mode is over-partitioning: partitioning on a high-cardinality column produces thousands of directories holding tiny files, and the metadata and task overhead swamps any pruning benefit. Guidance in practice: don't partition tables below roughly a terabyte, and aim for at least ~1 GB per partition.

#Z-ordering

OPTIMIZE table ZORDER BY (a, b) interleaves the bits of multiple columns along a space-filling curve, co-locating rows with similar values across several dimensions so min/max stats become selective on all of them. Costs: it is a full rewrite of the data in scope each time, and the key set is effectively fixed at design time. Effective on up to ~3–4 columns; beyond that the benefit per column dilutes.

#Liquid clustering — the modern default

CLUSTER BY (a, b) on the table.

• Incremental: new data is clustered as written; reorganisation is bounded rather than whole-table.
• Evolvable: you can change clustering keys later without rewriting history.
• Replaces both partitioning and Z-ordering for most tables, and removes the over-partitioning failure mode entirely.
• Automatic mode (CLUSTER BY AUTO) lets the platform select keys from observed query patterns.

When you would still partition: a hard requirement for physical separation — e.g. regulatory data residency, partition-level retention/deletion, or an external engine that depends on the directory layout.

#File sizing and compaction

Small files are the tax on every streaming and CDC workload: more tasks, more metadata, more listing, worse skipping. Mechanisms: OPTIMIZE (bin-packing compaction), optimised writes and auto-compaction at write time, and Predictive Optimization, which runs OPTIMIZE/VACUUM/statistics maintenance automatically on managed tables based on usage. On enterprise platforms, turning on predictive optimisation is usually a better answer than writing maintenance jobs per table — say so.

#4. Deletion vectors — merge-on-read for Delta

Historically, deleting one row from a file meant rewriting the whole file (copy-on-write). Deletion vectors change that: the file stays, and a compact bitmap records which row positions are logically deleted. Readers apply the bitmap during the scan.

• Wins: dramatically cheaper DELETE, UPDATE and MERGE, especially for small, scattered changes — the GDPR-erasure and CDC-update patterns.
• Cost: read-side work to apply vectors, and accumulating vectors degrade scans over time.
• Cleanup: OPTIMIZE (and REORG TABLE ... APPLY (PURGE)) materialises the deletions by rewriting the affected files and clearing the vectors.
• Compatibility: it is a table feature — readers must support it. Relevant when external engines read your tables.

Asked: "What changed about MERGE performance in recent years?" Deletion vectors (avoid rewriting whole files for small changes), better file skipping on the matched-file set, and the general move to liquid clustering so the matched files are fewer.

#5. Time travel, retention and VACUUM

SELECT * FROM t VERSION AS OF 42;
SELECT * FROM t TIMESTAMP AS OF '2026-09-01T00:00:00';
RESTORE TABLE t TO VERSION AS OF 42;

Two independent retention settings govern how far back you can actually go:

VACUUM permanently deletes files that are tombstoned and older than the retention threshold. After vacuuming, time travel to versions depending on those files fails. The safety interlock exists because a long-running reader may still be reading an old snapshot — which is exactly why lowering the retention below the default requires explicitly overriding a safety check. In interview, name the risk rather than the syntax: "VACUUM is the one Delta operation that actually destroys data, so it's governed by retention policy, not convenience."

#6. Change Data Feed

Enable with delta.enableChangeDataFeed = true. The table then records row-level changes with _change_type (insert, update_preimage, update_postimage, delete), _commit_version and _commit_timestamp, readable by version or timestamp range.

Use it for: incrementally propagating silver → gold, feeding downstream consumers without re-reading whole tables, and auditing. Know the boundaries: it only captures changes from the point it was enabled, and reading it is subject to the same retention limits as time travel.

#7. Schema evolution and constraints

• Schema enforcement is on by default — a write with a mismatched schema fails rather than silently corrupting the table. This is a feature; describe it as such.
• Evolution via mergeSchema on write or ALTER TABLE; MERGE supports schema evolution too.
• Column mapping (name or id mode) decouples the logical column name from the Parquet physical name, enabling RENAME/DROP COLUMN and special characters without a full rewrite.
• Constraints: NOT NULL, CHECK; plus generated columns (derived, and usable for partition pruning) and identity columns.
• Clone: SHALLOW CLONE copies metadata only, referencing the source's files — instant, ideal for test environments and experimentation; DEEP CLONE copies data too, which is the building block for cross-region replication and DR.

#8. Protocol versions and table features

A Delta table declares minimum reader and writer versions, and modern tables enumerate discrete table features (deletion vectors, liquid clustering, row tracking, column mapping, timestamp-without-timezone, and so on).

Why an architect must care: enabling a feature can make a table unreadable by older clients and third-party engines. Before enabling anything on a shared table, enumerate every reader — including the BI tool, the external Spark cluster, and the partner consuming via Delta Sharing. This is a classic avoidable production incident and a good thing to volunteer unprompted.

#9. Delta and Iceberg — the convergence

The format war has effectively resolved into interoperability, and you should be able to narrate this confidently because customers ask constantly.

• UniForm lets a Delta table expose Iceberg (and Hudi) metadata over the same Parquet files, so Iceberg-native engines can read it without a copy.
• Unity Catalog exposes an Iceberg REST catalog endpoint, so external Iceberg clients can discover and read governed tables.
• Iceberg v3 reached GA on the platform in 2026, with managed Iceberg tables in the runtime. Iceberg v3 brings deletion vectors, row lineage and the VARIANT type — features that align closely with Delta's, and allow Delta and Iceberg tables to share physical Parquet files without rewriting data. (Summit 2026 launches)

How to answer "Delta or Iceberg?" in the room:

"Increasingly the wrong question. The formats have converged on the same capabilities — v3 brings Iceberg deletion vectors and row lineage, and the same Parquet files can be exposed both ways. What I'd actually decide on is the catalog: where governance, lineage and access control live, because that's the part that's genuinely hard to change later. Pick the format that your engines read natively, and put the effort into the catalog."

That answer scores well because it reframes a religious question as an architectural one.

#10. VARIANT and semi-structured data

VARIANT stores semi-structured data (JSON-like) in an open, binary, efficiently-navigable encoding, rather than as a string parsed at query time. It gives you schema flexibility with far better performance than get_json_object over a string column, and is supported across Delta, Iceberg v3 and Parquet. The architectural use: bronze-layer landing of evolving payloads without either freezing a schema too early or paying the string-parsing tax forever.

#11. Ten questions to answer cold

1. What is physically in _delta_log, and what is a checkpoint for?
2. How does Delta avoid listing files on object storage, and why does that matter?
3. Which action carries statistics, and which queries do they help?
4. Explain OCC end to end, including what happens on conflict.
5. WriteSerializable vs Serializable — what is actually different?
6. Partitioning vs Z-order vs liquid clustering — pick one for a 50 TB event table and justify it.
7. What do deletion vectors change, and what new maintenance do they create?
8. Why is time travel bounded, and which two settings bound it?
9. What breaks when you enable a new table feature on a shared table?
10. A customer asks "should we standardise on Iceberg instead?" — answer in 60 seconds.