#Ingestion, Pipelines and Orchestration

Naming here changed materially — using the old names is the cheapest way to sound out of date. Lakeflow is the umbrella for data engineering: Connect (ingestion), Declarative Pipelines (formerly Delta Live Tables), Jobs (formerly Workflows), and Designer (visual authoring).

#1. Choosing an ingestion mechanism

Auto Loader, precisely. It tracks which files it has already processed in a scalable way (rather than listing a directory every run), supports directory listing and file notification modes (the latter using cloud event services for very large directories), handles schema inference and evolution with a rescue column for unexpected fields, and gives exactly-once ingestion semantics.

Asked: "Why not just read the directory each time?" "Listing cost grows with the directory, not with new data — at ten million files you're spending minutes listing to find a hundred new files. Auto Loader keeps its own durable record of what's been seen, and in notification mode the cloud tells us about new files instead of us asking."

#2. Declarative pipelines

You declare what the tables are and their dependencies; the platform infers the DAG, manages orchestration, retries, checkpoints, and incremental processing.

The object types:

• Streaming table — incrementally ingests from an append-only source. Each record processed once.
• Materialized view — a query whose results are kept fresh, incrementally recomputed where possible. The gold-layer workhorse.
• View — intermediate, not materialised.

Expectations are declarative data-quality constraints:

CONSTRAINT valid_id EXPECT (id IS NOT NULL) ON VIOLATION DROP ROW
CONSTRAINT valid_amt EXPECT (amount > 0)    ON VIOLATION FAIL UPDATE
-- omit ON VIOLATION to record the violation as a metric but keep the row

Three behaviours: warn (track only), drop row, fail the update. Violation metrics are retained per run, which is what makes data quality observable rather than aspirational.

CDC is handled declaratively (AUTO CDC / APPLY CHANGES), including out-of-order handling via a sequencing column and SCD Type 1 and Type 2 — you declare the semantics instead of writing and debugging a MERGE. This is the answer to most "how do we build CDC" questions, and it is far more robust than hand-rolled merges.

Real-time mode extends declarative pipelines to continuous processing with very low end-to-end latency, which is what lets teams consolidate workloads that previously needed Flink. (Summit 2026)

When not to use declarative pipelines: highly imperative multi-system workflows, heavy non-table side effects, or an existing well-run dbt/Airflow estate where the migration cost exceeds the benefit. Say this — blanket advocacy reads as vendor parroting.

#3. Orchestration

Lakeflow Jobs is the native orchestrator: multi-task DAGs, conditional execution, retries, parameter passing, file/table triggers, scheduled or continuous, with 50+ external integrations.

The pragmatic architect answer: "If Airflow is already the enterprise scheduler and orchestrates twenty other systems, I'd keep it as the conductor and call Databricks jobs from it, rather than fight a religious war. If Databricks is where the work lives, native jobs remove a whole component from the estate, and that's one fewer thing to run at 3am."

#4. Streaming design patterns

Pattern: bronze streaming, silver CDC, gold materialised. Auto Loader (or Kafka) → bronze streaming table (append, raw) → AUTO CDC into silver (deduplicated, SCD) → materialised views for gold aggregates. Covers the majority of enterprise requirements and is a clean thing to draw.

Pattern: incremental batch. Use the AvailableNow trigger — you get streaming's checkpointing and exactly-once bookkeeping with batch economics, running every 15 minutes instead of continuously. Most "we need streaming" requirements are actually this. Ask what the decision latency is: if a human looks at the dashboard each morning, five-second freshness is expensive theatre.

Pattern: late and out-of-order data. Watermark bounded by real-world lateness; reconciliation job for data arriving beyond the watermark; bronze retained so you can always reprocess.

The freshness question to ask every customer:

"What decision gets made with this data, and how quickly does it change the decision?"

That question, asked early, is one of the strongest discovery signals you can show in a panel.

#5. Data quality at enterprise scale

Four layers, and being able to enumerate them is a differentiator:

1. Schema enforcement — Delta rejects mismatched writes by default.
2. Declarative expectations — in-pipeline constraints with warn/drop/fail semantics.
3. Monitoring — profile tables and detect drift over time; alert on distribution shifts, not just failures.
4. Contracts — agreements with producers about schema, semantics and change notice. The organisational layer; without it the other three are archaeology.

The line: "Most data quality programmes are detection projects. The leverage is in the contract with the producer — everything downstream of that is you finding out late."

#6. Questions to answer cold

1. Auto Loader vs a directory listing — why does it matter at scale?
2. Streaming table vs materialized view — when each?
3. How do you implement SCD Type 2 without writing a MERGE?
4. Three expectation behaviours and when you'd use each.
5. Customer says "we need real-time". What do you ask before agreeing?
6. Keep Airflow or move to Lakeflow Jobs? Defend either.
7. Where does AvailableNow beat continuous streaming?