#Apache Spark Internals

The depth expected of someone who "remains hands-on with Spark performance, debugging and optimisation". Organised so that each section ends with what an interviewer actually asks.

#1. The execution model, precisely

Get the vocabulary exactly right — imprecision here is the fastest way to sound rusty.

Query / DataFrame action
      │
      ▼
   JOB            one per action (collect, write, count, save)
      │
      ▼
   STAGE          bounded by shuffle boundaries; stages run sequentially where dependent
      │
      ▼
   TASK           one per partition; the unit of scheduling; runs in one slot on one executor

• Driver — holds the SparkSession, builds and optimises the plan, schedules tasks, tracks metadata. Single point of failure; also the thing that OOMs when you collect() a large result or broadcast something enormous.
• Executor — JVM (plus Photon's native layer) running tasks and holding cached blocks. Slots per executor ≈ cores per executor.
• Narrow dependency — each output partition depends on one input partition (filter, map, union, select). Pipelined inside a stage, no data movement.
• Wide dependency / shuffle — output partitions depend on many input partitions (groupBy, join without broadcast, repartition, distinct, window functions). Creates a stage boundary and writes shuffle files to local disk.

Say it this way: "Stages are separated by shuffles. So when I'm tuning, the first question is always: how many shuffles does this plan have, and does each one earn its place?"

#2. Catalyst — the four phases

SQL / DataFrame
   │
   ├─▶ 1. PARSE      → Unresolved Logical Plan   (syntax only; table names not yet resolved)
   │
   ├─▶ 2. ANALYSE    → Resolved Logical Plan     (catalog lookup: tables, columns, types, functions)
   │
   ├─▶ 3. OPTIMISE   → Optimised Logical Plan    (rule-based; then cost-based for joins)
   │
   └─▶ 4. PLAN       → Physical Plan → Code Gen  (strategies → selected physical plan → JVM bytecode)

Phase 3 rules worth naming: predicate pushdown, projection/column pruning, constant folding, boolean simplification, null propagation, limit pushdown, subquery decorrelation, partition pruning. Cost-based optimisation (CBO) uses table and column statistics — collected via ANALYZE TABLE — to reorder joins and pick join strategies. CBO only helps if statistics exist and are fresh; a common real-world finding is a bad join order caused by missing stats.

Phase 4 — whole-stage code generation. Rather than the classic Volcano model where every operator is a virtual next() call, Spark fuses the operators in a stage into a single generated Java method, compiled at runtime (via Janino). This collapses virtual calls and keeps intermediate values in CPU registers. In the physical plan, operators marked with * and a bracketed number are fused into that codegen stage. When you see an operator without the *, it fell out of codegen — worth knowing why.

Asked: "Walk me through what happens when I run a SQL query." Answer with the four phases, then add the runtime layer: "…and then AQE can change that physical plan mid-flight, which is the part people forget."

#3. Tungsten

The memory and CPU efficiency layer underneath Catalyst:

• UnsafeRow — a compact binary row format operated on directly, avoiding JVM object overhead and garbage collection pressure per row.
• Off-heap / explicitly managed memory — Spark manages its own memory regions rather than leaving it to the JVM heap, which makes memory behaviour predictable.
• Cache-aware algorithms — sorting and hashing structured to respect CPU cache lines.
• Whole-stage codegen — as above.

The one-liner: "Catalyst decides what to do; Tungsten decides how efficiently the JVM does it."

#4. Adaptive Query Execution (AQE)

On by default in modern runtimes. The key insight: at a shuffle boundary, Spark has real statistics about the data it just wrote — so it can re-plan the rest of the query with facts rather than estimates.

Three things AQE does:

The critical limitation to state in interview: AQE re-optimises at shuffle boundaries, using statistics from completed stages. It cannot fix what happens before the first shuffle, it cannot rescue a fundamentally wrong plan, and it does not remove the need for good file layout and statistics. It is a runtime safety net, not a substitute for design.

Asked: "Why does AQE need a shuffle to work?" "Because the shuffle write is the first point where Spark has ground truth about partition sizes rather than estimates from statistics. The map output statistics are the input to the re-planning."

#5. Joins — the decision that dominates performance

What to check when a join is slow:

1. Is it a sort-merge join that should be a broadcast? (Stats missing, or threshold too low.)
2. Is it a broadcast that is killing the driver or executors? (Side is bigger than assumed.)
3. Is the key skewed? (One task at 40 minutes, the rest at 30 seconds.)
4. Is it a nested loop join because the join condition is not an equality? (Check the plan.)

Dynamic Partition Pruning (DPP): when joining a large partitioned fact table to a filtered dimension, Spark can push the filter's resulting key set down to prune fact-table partitions at runtime — a large win in star schemas that estimates alone cannot achieve. Requires the fact table to be partitioned on the join key and, typically, a broadcastable dimension side.

#6. Skew — the single most common production pathology

What it is: the partitioning key's value distribution is uneven, so one or a few tasks receive far more rows than the rest. The stage's duration becomes the slowest task's duration.

How you recognise it in the Spark UI: in the stage's task summary, compare max duration and max shuffle read to the median. A max/median ratio above ~5–10× on either is skew until proven otherwise.

Causes, in order of frequency: a dominant default value (NULL, -1, 'UNKNOWN', 0) · a genuine power-law business key (one hypermarket, one mega-customer) · a low-cardinality join key · a date key where one day holds a backfill.

Remedies, in order of preference:

1. Let AQE handle it — skew join handling splits the oversized partitions automatically. Check it is enabled and that the stage is actually eligible (it applies to sort-merge joins at shuffle boundaries).
2. Filter the junk key — if NULL/-1 rows cannot join meaningfully, exclude them before the join and union them back if needed. Frequently the whole fix.
3. Broadcast the small side — if you can broadcast, skew on the large side stops mattering for the join.
4. Salting — append a random suffix 0..N to the hot key on the large side, and explode the small side into N copies, one per salt value. Join on (key, salt). Costs an N× fan-out of the small side. Use when the hot keys are few and known.
5. Isolate and union — process the hot keys as a separate job with their own parallelism, union the results. Ugly, effective, and sometimes the only option.

Asked: "Describe salting and when you would not use it." "…I wouldn't salt when AQE already splits the skewed partitions, when the small side is broadcastable, or when the skew is a junk key I can just filter — salting adds real complexity and a fan-out cost, so it's the fourth thing I try, not the first."

#7. Memory, spill and OOM

Unified memory model. The executor's usable memory is split between:

• Execution memory — shuffles, joins, sorts, aggregations.
• Storage memory — cached/persisted blocks and broadcast variables.

These share one region and borrow from each other: storage can lend to execution and vice versa, but execution can evict borrowed storage, while storage cannot evict execution. Beyond that region sits user memory (your data structures) and a reserved portion.

Spill means execution memory was exhausted and Spark wrote intermediate data to local disk to continue. It is a correctness-preserving slowdown, not an error. In the Spark UI look for "Spill (memory)" and "Spill (disk)" columns in the stage's task metrics.

Rule of thumb worth stating: aim for shuffle partitions in the low-hundreds-of-MB range. Too large → spill and skew sensitivity. Too small → scheduling overhead and tiny files. AQE coalescing handles the "too small" side automatically; the "too large" side is your design problem.

#8. Photon

A vectorised query engine written in C++, replacing parts of JVM execution on supported operators.

• Why it is faster: columnar in-memory batches, SIMD vectorisation, no JVM object/GC overhead, and memory-efficient hash tables. Best on scan-heavy, filter-heavy, join-and-aggregate SQL and ETL.
• What it accelerates well: SQL and DataFrame operations, scans with data skipping, joins, aggregations, MERGE/UPDATE/DELETE on Delta, writes.
• Where it helps least: workloads dominated by Python/Scala UDFs, heavy ML/UDF-bound code, and small queries where the bottleneck is not execution.
• Fallback: unsupported operators run on the Spark engine. A query can be partly Photon. In the Spark UI / query profile, Photon operators are labelled — if you expected Photon and see Spark operators, find out which operator caused the fallback.
• Commercials: Photon consumes a higher DBU rate. The architect's point is that it usually still reduces total cost because the job finishes disproportionately faster — but you must measure cost per workload, not DBU rate. A UDF-heavy job on Photon can genuinely cost more.

Asked: "Is Photon always worth it?" "No — and that's the interesting part. It bills at a higher rate, so it wins when it can actually accelerate the plan. Scan and join heavy SQL, almost always. A pipeline that's 80% Python UDF, often not. I'd benchmark the actual workload and compare total cost, not rate."

#9. Structured Streaming

Model: an unbounded input treated as a table that grows; each trigger processes the new data as a micro-batch and updates results. Fault tolerance comes from a checkpoint holding source offsets, a write-ahead log of planned batches, and stateful operator state.

Delivery semantics: exactly-once end-to-end requires a replayable source (Kafka, files, Delta) and an idempotent or transactional sink (Delta is transactional). Spark tracks which offsets belong to which batch, so a re-run of a failed batch produces the same result rather than duplicates. Be precise in interview: "exactly-once effect on the sink, via replayable source plus atomic commit — not exactly-once message delivery."

Triggers:

Watermarks and state. Stateful operations (windowed aggregations, stream-stream joins, deduplication, arbitrary stateful processing) accumulate state. A watermark declares how late data may be, which lets Spark evict state older than that bound and emit final results. Without a watermark, state grows without limit — this is the classic "my streaming job slowly dies" incident. State is held in a state store, with RocksDB the standard choice at scale because it keeps state off the JVM heap and avoids GC pressure.

Failure modes to name:

• State growth from a missing or too-generous watermark.
• Small-file explosion from a short trigger interval writing to Delta (fix: longer trigger, auto compaction / optimised writes, or AvailableNow).
• Checkpoint incompatibility after changing a stateful query's shape — some changes require a new checkpoint and a state rebuild. Plan this into deployment.
• Source throttling: unbounded maxOffsetsPerTrigger / maxFilesPerTrigger causing an enormous first batch after a backlog. Always bound the first batch.

Asked: "A streaming job's latency is growing steadily over days. Diagnose." Batch duration trending up → check state store size and watermark configuration → check input rate vs processing rate → check for small-file accumulation on the sink → check for a skewed key concentrating state. Most often: unbounded state.

#10. Spark Connect and serverless

Spark Connect decouples the client from the driver: the client builds an unresolved logical plan and sends it over gRPC to a remote Spark server, which analyses, optimises and executes it. This is what makes thin clients, multi-language support, and shared/serverless compute practical, and it removes the old model where the client was welded into the driver JVM. Practical consequence: some APIs that relied on JVM internals (deep RDD manipulation, certain SparkContext internals) behave differently or are unavailable in Connect-based environments — a real migration consideration for older codebases.

Serverless compute removes cluster configuration and start-up latency: capacity is managed by Databricks, starts in seconds, and scales without you sizing instance types. The architectural trade-off to articulate:

The architect's take: serverless converts an engineering problem (right-sizing clusters, idle waste, start-up latency) into a governance problem (budget policies, tagging, egress rules). At enterprise scale that is usually a good trade, because governance scales across teams and tuning does not.

#11. Twelve questions to be able to answer cold

1. Stage vs task vs job — define each in one sentence.
2. Which operations cause a shuffle, and why does repartition differ from coalesce?
3. Four phases of Catalyst; what happens in each.
4. What does whole-stage codegen actually produce, and how do you see it in a plan?
5. Three things AQE does and the one thing it fundamentally cannot do.
6. Broadcast vs shuffle hash vs sort-merge — selection criteria and failure mode of each.
7. How do you detect skew in the Spark UI, and what are four remedies in preference order?
8. Execution vs storage memory — who can evict whom?
9. What is spill, and is it an error?
10. When is Photon not worth enabling?
11. What must be true for end-to-end exactly-once in Structured Streaming?
12. Why does a missing watermark eventually kill a streaming job?

If any of these is shaky, that is your next hour — not the next file.