#AI, GenAI and Agents on Databricks

You do not need to be an ML engineer, but a chief architect in 2026 who cannot architect a governed GenAI system is not credible. This is also where the platform is moving fastest — and where a customer's hardest current questions live.

#1. The argument that frames everything

"The differentiator in enterprise AI is no longer the model — it's the governed context you can feed it and the controls you can put around it."

Everyone has access to the same frontier models. What they do not have is your data, correctly permissioned, with lineage, with consistent business definitions, and with spend and safety controls at runtime. That is the entire lakehouse-for-AI argument, and it is the thesis Databricks organised its 2026 platform around: the competition moved from "who builds the best agent" to "who governs the data, context, cost and security underneath it". (Summit 2026 recap)

Learn that framing. It answers "why Databricks for AI" better than any feature list.

#2. The stack

#3. RAG — the architecture you must be able to draw

  INGEST                     INDEX                      RETRIEVE                GENERATE
┌──────────┐   parse/    ┌────────────┐   sync     ┌──────────────┐         ┌──────────────┐
│ Documents├────chunk───▶│ Delta table├───────────▶│ Vector Search│────┐    │ LLM endpoint │
│ (volumes)│   enrich    │ + metadata │   (managed)│ index        │    │    │ (FM API or   │
└──────────┘             └────────────┘            └──────────────┘    │    │  external)   │
                               │                          ▲            ├───▶└──────┬───────┘
                               │  permissions,            │            │           │
                               │  lineage, tags      hybrid + filters   │           ▼
                               ▼                                        │    ┌──────────────┐
                        UNITY CATALOG ◀── access control applied ───────┘    │   Response   │
                                                                             │  + citations │
   GOVERNED BY: Unity AI Gateway (budgets · routing · guardrails · tracing)   └──────────────┘
   OBSERVED BY: MLflow tracing + evaluation  ·  EVALUATED BY: LLM judges + ground-truth set

The five things that actually determine RAG quality — say these, not "we use vector search":

1. Chunking strategy. Semantic/structural chunking beats fixed-size almost always. Chunk size and overlap are the highest-leverage tuning knobs and the most neglected.
2. Retrieval quality, measured. Hybrid (vector + keyword) plus reranking. Measure recall@k against a labelled set before touching the prompt — most "the model is bad" complaints are retrieval failures.
3. Metadata filtering and permissions. The index must respect who is asking. The failure mode that ends projects: a chatbot that cheerfully surfaces the HR folder. Filter at retrieval by the user's entitlements; never rely on the model to decline.
4. Evaluation. A ground-truth question set, LLM judges for faithfulness/relevance/toxicity, tracked over time in MLflow. Without this you are shipping vibes.
5. Grounding and citations. Every answer traceable to a source chunk. Non-negotiable in regulated industries and the thing that builds user trust fastest.

Asked: "Our RAG system gives bad answers. Where do you start?" "Retrieval before generation, every time. I'd take fifty real failing questions and check whether the correct chunk was in the retrieved set at all. If it wasn't, it's a chunking, embedding or filtering problem and the prompt is irrelevant. If it was there and the answer was still wrong, then it's a prompt, context-window or model problem. Teams almost always tune the prompt first and then wonder why nothing improves."

That answer is a reliable standout.

#4. Agents, and governing them

Agent Bricks provides a managed path to building, evaluating and deploying agents, with support for mainstream agent SDKs, managed memory, and sandboxed execution. Omnigent sits above agent harnesses (Claude Code, Codex, custom frameworks) as an open-source meta-harness — composition across harnesses, stateful policy control, and real-time human-agent collaboration — with a managed variant that inherits Unity Catalog and Unity AI Gateway governance. (Databricks docs, Unity AI Gateway)

The governance problem an architect must solve (and can now name concretely):

The distinction to state crisply: "Unity Catalog governs the assets — who may use this model, this tool, this table. Unity AI Gateway governs the interactions — what actually happens at runtime, what it costs, and what the agent is allowed to do. You need both; most organisations have neither and are running agents on an honour system."

#5. Evaluation — the discipline that separates pilots from production

• Offline: a curated question/answer set with expected behaviour; automated scoring on every change. Treat it as a test suite, because it is one.
• LLM judges: scoring faithfulness, relevance, harmfulness, and answer completeness at scale; calibrated against human labels on a sample.
• Online: user feedback, deflection/escalation rates, and traces — MLflow tracing gives per-request visibility into retrieval and generation steps.
• Regression gating: no prompt or model change ships without the eval suite passing. This is the single control that keeps a GenAI product stable, and almost nobody has it.

The line: "The reason most GenAI pilots don't reach production isn't model quality — it's that the team has no way to prove a change made things better. Evaluation is the thing I'd build second, right after retrieval, and before anyone touches the prompt."

#6. Natural-language analytics

Genie provides conversational analytics over governed data; Genie One (2026) extends this to a cross-application agentic interface producing documents and artefacts, with Genie Ontology as a continuously-built context layer grounding responses in enterprise knowledge, and Genie Agents turning conversations into reusable agents deployable into Slack and Teams. (Summit 2026)

What determines whether NL analytics succeeds — the honest architect's answer:

1. A clean, well-modelled, well-documented gold layer. Natural language over a messy warehouse produces confident wrong answers, which is worse than no answers.
2. Consistent metric definitions — this is exactly what Unity Catalog Metrics provides.
3. Scoped context — point it at curated domains, not the entire catalog.
4. Permissions that hold — the answer must respect the asker's entitlements.

"Genie is a forcing function for data modelling. If your semantics are ambiguous, the LLM will pick one interpretation and state it with total confidence. I'd scope it to a curated domain with governed metrics before I'd open it to the estate."

#7. Build vs buy vs fine-tune

A question you will be asked. Answer as a decision tree, in cost order:

1. Prompt engineering with a strong base model — days, cheapest, try first.
2. RAG — when the gap is knowledge, not behaviour. Most enterprise use cases stop here.
3. Fine-tuning — when the gap is behaviour, format, tone or a domain idiom that prompting cannot reach; needs real labelled data and an eval harness, and it must be re-done as base models improve.
4. Pre-training — essentially never for an enterprise, and saying so builds credibility.

"I'd need to see a RAG baseline with measured failure modes before agreeing to fine-tune. 'Fine-tune it' is usually a proposal to spend six weeks solving a retrieval problem."

#8. Questions to answer cold

1. Draw a governed RAG architecture, including where permissions are enforced.
2. RAG answers are poor — what do you check first, and why not the prompt?
3. What does Unity AI Gateway govern that Unity Catalog does not?
4. How do you stop agent spend running away across ten teams?
5. When is fine-tuning justified?
6. A customer wants NL analytics over their warehouse next month. What do you tell them?
7. How do you evaluate a GenAI application, concretely?