[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-ai-functions":3,"mdc--l325sq-key":30,"related-org-databricks-databricks-ai-functions":2834,"related-repo-databricks-databricks-ai-functions":3017},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":26,"sourceUrl":28,"mdContent":29},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-31T05:53:33.562077",null,60,[],{"repoUrl":21,"stars":20,"forks":24,"topics":27,"description":23},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-ai-functions","---\nname: databricks-ai-functions\ndescription: \"Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).\"\ncompatibility: Requires databricks CLI (>= v1.0.0)\nmetadata:\n  version: \"0.2.0\"\nparent: databricks-core\n---\n\n# Databricks AI Functions\n\n> **Official Docs:** https:\u002F\u002Fdocs.databricks.com\u002Flarge-language-models\u002Fai-functions\n> Individual function reference: https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002F\n\n## Overview\n\nDatabricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as `UPPER()` or `LENGTH()`, and are optimized for batch inference at scale.\n\n**Always prefer a task-specific function over `ai_query`.** Reach for `ai_query` only when no task function fits (custom\u002Fexternal endpoints, multimodal, or JSON beyond `ai_extract`'s limits). Every function below shares a baseline: **DBR 15.1+** (notebooks) \u002F **15.4 ML LTS** (batch), **not on SQL Warehouse Classic**, and region must support AI Functions — the Prereqs column lists only what's *additional*.\n\n**Cost & speed — each call is an LLM inference (slow and billed per token).** Run a function **once** per row and persist the result to a Delta table; never re-invoke it on every downstream query. In demos, **avoid generating tables with millions of rows** — sample the input when needed so the demo runs quickly. Materialize once, then query the cheap Delta output.\n\nThe **Function** column links to the in-repo deep reference (full options, schemas, examples); **Docs** links to the official page.\n\n| Function | Task | Input | Output | Extra prereqs | Docs |\n|---|---|---|---|---|---|\n| [`ai_analyze_sentiment`](references\u002F1-task-functions.md#ai_analyze_sentiment) | Sentiment scoring | `content STRING` | `STRING` — `positive`\u002F`negative`\u002F`neutral`\u002F`mixed`, or `NULL` | — | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_analyze_sentiment) |\n| [`ai_classify`](references\u002F1-task-functions.md#ai_classify) | Fixed-label routing | `content STRING\\|VARIANT`, `labels` (2–500), `[options MAP]` | `VARIANT` — `{response:[label], error_message}` | — | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_classify) |\n| [`ai_extract`](references\u002F1-task-functions.md#ai_extract) | Entity \u002F field extraction | `content STRING\\|VARIANT`, `schema STRING` (JSON), `[options MAP]` | `VARIANT` — `{response:{…}, error_message, metadata}` | ≤256 fields, ≤12 nesting levels | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_extract) |\n| [`ai_fix_grammar`](references\u002F1-task-functions.md#ai_fix_grammar) | Grammar correction | `content STRING` | `STRING` (corrected) | — | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_fix_grammar) |\n| [`ai_gen`](references\u002F1-task-functions.md#ai_gen) | Free-form generation | `prompt STRING` | `STRING` | — | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_gen) |\n| [`ai_mask`](references\u002F1-task-functions.md#ai_mask) | PII redaction | `content STRING`, `labels ARRAY\u003CSTRING>` | `STRING` (entities → `[MASKED]`) | — | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_mask) |\n| [`ai_similarity`](references\u002F1-task-functions.md#ai_similarity) | Semantic similarity | `expr1 STRING`, `expr2 STRING` | `FLOAT` (0.0–1.0) | — | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_similarity) |\n| [`ai_summarize`](references\u002F1-task-functions.md#ai_summarize) | Summarization | `content STRING`, `[max_words INT]` (0 = uncapped) | `STRING` | Public Preview; English-tuned | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_summarize) |\n| [`ai_translate`](references\u002F1-task-functions.md#ai_translate) | Translation | `content STRING`, `to_lang STRING` | `STRING` | Langs: en, fr, de, hi, it, pt, es, th | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_translate) |\n| [`ai_parse_document`](references\u002F1-task-functions.md#ai_parse_document) | Parse PDF \u002F Office \u002F images | `content BINARY`, `[Map('version','2.0', …)]` | `VARIANT` — pages, elements, error_status | **DBR 17.3+**; ≤500 pages \u002F 100 MB | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_parse_document) |\n| [`ai_prep_search`](references\u002F1-task-functions.md#ai_prep_search) | RAG chunking from parsed docs | `parsed VARIANT`, `[options MAP]` | `VARIANT` — `{document:{contents, pages, source_uri}, error_status}` | **DBR 18.2+** (serverless env v3+) | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_prep_search) |\n| [`ai_query`](references\u002F2-ai-query.md) | Any serving endpoint (built-in foundation or custom), multimodal, complex JSON (**last resort**) | `endpoint STRING`, `request STRING\\|STRUCT`, `[returnType]`, `[failOnError BOOL]`, `[modelParameters STRUCT]`, `[responseFormat STRING]`, `[files]` | Parsed response; with `failOnError => false` a `STRUCT{response, errorMessage}` | **Pro\u002FServerless** warehouse; `CAN QUERY` on endpoint | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_query) |\n| [`ai_forecast`](references\u002F3-ai-forecast.md) | Time series forecasting (table-valued) | `observed TABLE`, `horizon`, `time_col`, `value_col`, `[group_col]`, `[prediction_interval_width]`, `[frequency]`, `[seed]`, `[parameters]` | Rows: time\u002Fgroup cols + per value `{v}_forecast`, `{v}_upper`, `{v}_lower` (DOUBLE) | **Pro\u002FServerless** warehouse; Public Preview | [↗](https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_forecast) |\n\nModels run under Apache 2.0 or LLAMA 3.3 Community License — you are responsible for compliance.\n\n## Patterns\n\n**Chain task functions to enrich a column in one pass.** `ai_classify`\u002F`ai_extract` return a VARIANT — read it with the colon operator (`:response`):\n\n```sql\nSELECT id,\n  ai_analyze_sentiment(content)                                                   AS sentiment,\n  ai_summarize(content, 30)                                                       AS summary,\n  ai_classify(content, '[\"technical\",\"billing\",\"other\"]', map('version','2.0')):response[0]::STRING AS category,\n  ai_extract(content, '[\"product\",\"error_code\",\"date\"]', map('version','2.0')):response:product::STRING AS product,\n  ai_fix_grammar(content)                                                         AS content_clean\nFROM raw_feedback;\n```\n\nIn **PySpark**, call any of these inside `expr(...)`: `df.withColumn(\"category\", expr(\"ai_classify(content, '[\\\"a\\\",\\\"b\\\"]', map('version','2.0')):response[0]::STRING\"))` — and read VARIANT fields via `selectExpr(\"col:response:field::STRING AS field\")`.\n\n**PII redaction before storage** — `ai_mask(content, ARRAY(entity_types))` returns text with entities → `[MASKED]`.\n\n```sql\nSELECT ai_mask(message, array('person','email','phone','address')) AS message_safe FROM raw_messages;\n```\n\n**Semantic matching \u002F dedup** — `ai_similarity` returns 0–1; self-join and threshold:\n\n```sql\nSELECT a.id, b.id, ai_similarity(a.name, b.name) AS score\nFROM companies a JOIN companies b ON a.id \u003C b.id\nWHERE ai_similarity(a.name, b.name) > 0.85;\n```\n\n**Forecasting** — table-valued; one row per future period (+ per group). Full param\u002Fgroup\u002Finterval forms → [3-ai-forecast.md](references\u002F3-ai-forecast.md):\n\n```sql\nSELECT * FROM ai_forecast(\n    observed => TABLE(SELECT date, sales FROM daily_sales),\n    horizon => '2026-12-31', time_col => 'date', value_col => 'sales');\n-- Returns: date, sales_forecast, sales_upper, sales_lower\n```\n\n**Nested JSON via `ai_query`** (last resort — only past `ai_extract`'s limits) — parse the response with `from_json`. Model names, multimodal `files =>`, `modelParameters`, SQL UDF → [2-ai-query.md](references\u002F2-ai-query.md):\n\n```sql\nSELECT from_json(\n    ai_query('databricks-claude-sonnet-4',\n        concat('Extract invoice as JSON with nested line_items array: ', text_blocks),\n        responseFormat => '{\"type\":\"json_object\"}', failOnError => false).response,\n    'STRUCT\u003Cnumero:STRING, total:DOUBLE, line_items:ARRAY\u003CSTRUCT\u003Ccode:STRING, qty:DOUBLE>>>'\n) AS invoice\nFROM parsed_documents;\n```\n\nDocument parsing (`ai_parse_document`) and RAG chunking (`ai_prep_search`) get their own staged pipeline below.\n\n## Document Processing Pipeline\n\nChain AI Functions stage-by-stage into Delta tables for batch document processing. The example is written as a **Spark Declarative Pipeline (SDP \u002F Lakeflow \u002F DLT)** — `CREATE OR REFRESH STREAMING TABLE` with `STREAM(...)` sources. To run the same logic standalone in a **notebook \u002F SQL warehouse**, swap each `CREATE OR REFRESH STREAMING TABLE x AS` for `CREATE OR REPLACE TABLE x AS` and drop the `STREAM(...)` wrappers. In SDP Python it's `@dp.table` with `from pyspark import pipelines as dp`.\n\n```sql\n-- Stage 1 — parse binary docs (any type), filter parse errors\nCREATE OR REFRESH STREAMING TABLE raw_parsed AS\nSELECT path,\n  concat_ws('\\n', transform(parsed:document:elements, e -> e:content::STRING)) AS text_blocks,\n  parsed:error_status AS parse_error\nFROM (\n  SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed\n  FROM STREAM read_files('\u002FVolumes\u002Fmy_catalog\u002Fdoc_processing\u002Flanding\u002F', format => 'binaryFile')\n)\nWHERE parsed:error_status IS NULL;\n\n-- Stage 2 — classify document type (cheap, no endpoint selection)\nCREATE OR REFRESH STREAMING TABLE classified_docs AS\nSELECT *,\n  ai_classify(text_blocks, '[\"invoice\",\"purchase_order\",\"receipt\",\"contract\",\"other\"]', map('version','2.0')):response[0]::STRING AS doc_type\nFROM STREAM raw_parsed;\n\n-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`\nCREATE OR REFRESH STREAMING TABLE extracted AS\nSELECT path, doc_type,\n  result:response:invoice_number::STRING AS invoice_number,\n  result:response:vendor_name::STRING    AS vendor_name,\n  result:response:total_amount::DOUBLE   AS total_amount,\n  result:error_message::STRING           AS extract_error\nFROM (\n  SELECT *, ai_extract(text_blocks,\n    '{\"invoice_number\":{\"type\":\"string\"},\"vendor_name\":{\"type\":\"string\"},\"total_amount\":{\"type\":\"number\"}}',\n    map('version','2.0')) AS result\n  FROM STREAM classified_docs WHERE doc_type = 'invoice' AND text_blocks IS NOT NULL\n);\n```\n\nIn a batch job, route the per-row error to a sidecar table instead of letting it crash the run: keep `ai_extract`'s `result:error_message` (VARIANT, colon-accessed, as above), and for `ai_query` pass `failOnError => false` and check `ai_response.errorMessage` (a STRUCT field, dot-accessed). See [2-ai-query.md](references\u002F2-ai-query.md).\n\n### Custom RAG Pipeline — Parse → Prep → Index\n\nFor retrieval rather than field extraction: `ai_parse_document` → `ai_prep_search` (semantic chunking + context enrichment, DBR 18.2+) → Vector Search Delta Sync index. `ai_prep_search` returns `chunk_id`, `chunk_to_retrieve`, and `chunk_to_embed` (enriched with title\u002Fheaders\u002Fpage) — **embed `chunk_to_embed`, return `chunk_to_retrieve` to the LLM.** Shown standalone; in an SDP swap `CREATE OR REPLACE TABLE` for `CREATE OR REFRESH STREAMING TABLE` + `STREAM read_files(...)`.\n\n```sql\nCREATE OR REPLACE TABLE parsed_chunks AS\nWITH prepped AS (\n  SELECT path AS source_path, ai_prep_search(ai_parse_document(content)) AS prep\n  FROM read_files('\u002FVolumes\u002Fmy_catalog\u002Fdoc_processing\u002Fdocs\u002F', format => 'binaryFile')\n)\nSELECT\n  variant_get(chunk, '$.chunk_id',          'STRING') AS chunk_id,\n  variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,\n  variant_get(chunk, '$.chunk_to_embed',    'STRING') AS chunk_to_embed,\n  source_path\nFROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY\u003CVARIANT>')) c AS chunk;\n```\n\nThen enable CDF (`ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)`) and use the **[databricks-vector-search](..\u002Fdatabricks-vector-search\u002FSKILL.md)** skill to build a Delta Sync index: PK `chunk_id`, embedding source `chunk_to_embed`, return `chunk_to_retrieve`.\n\n**Beyond batch:**\n- **Ask questions over the output** — point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the **[databricks-genie-agents](..\u002Fdatabricks-genie-agents\u002FSKILL.md)** skill.\n- **Low-latency \u002F serving** — to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the **[databricks-model-serving](..\u002Fdatabricks-model-serving\u002FSKILL.md)** skill.\n- **Production incremental ingestion** — for a runnable end-to-end streaming `ai_parse_document` job (checkpoints, `trigger(availableNow=True)`), see [databricks\u002Fbundle-examples · job_with_ai_parse_document](https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fbundle-examples\u002Ftree\u002Fmain\u002Fcontrib\u002Fjob_with_ai_parse_document).\n\n## Reference Files\n\n- [1-task-functions.md](references\u002F1-task-functions.md) — Deep reference for every task-specific function: full options\u002Fschemas (e.g. `ai_extract` v2.1 citations + confidence scores, `ai_classify` multilabel, `ai_parse_document` options + output schema, `ai_prep_search` chunk schema) and non-trivial examples. The Overview table above links to each function's section directly.\n- [2-ai-query.md](references\u002F2-ai-query.md) — `ai_query` complete reference: all parameters, structured output with `responseFormat`, multimodal `files =>`, UDF patterns, and error handling\n- [3-ai-forecast.md](references\u002F3-ai-forecast.md) — `ai_forecast` parameters, single-metric, multi-group, multi-metric, and confidence interval patterns\n\n## Common Issues\n\n| Issue | Solution |\n|---|---|\n| `ai_parse_document` not found | Requires DBR **17.3+**. Check cluster runtime. |\n| `ai_prep_search` not found | Requires DBR **18.2+** (serverless env v3+). |\n| `explode()` fails on a VARIANT | `explode` needs ARRAY — cast first: `explode(variant_get(prep, '$.document.contents', 'ARRAY\u003CVARIANT>'))`. |\n| Embedding the wrong RAG column | Embed `chunk_to_embed` (context-enriched); return `chunk_to_retrieve` to the LLM. |\n| `ai_forecast` fails | Requires **Pro or Serverless** SQL warehouse — not available on Classic or Starter. |\n| All functions return NULL | Input column is NULL. Filter with `WHERE col IS NOT NULL` before calling. |\n| `ai_translate` fails for a language | Supported (8): English (`en`), French (`fr`), German (`de`), Hindi (`hi`), Italian (`it`), Portuguese (`pt`), Spanish (`es`), Thai (`th`) — `to_lang` takes the code or the full name (see [references\u002F1-task-functions.md](references\u002F1-task-functions.md)). Use `ai_query` with a multilingual model for others. |\n| `ai_classify` returns unexpected labels | Use clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results. |\n| `ai_query` raises on some rows in a batch job | Add `failOnError => false` — returns a STRUCT with `.response` and `.errorMessage` (dot-accessed) instead of raising. |\n| Batch job runs slowly | Use DBR **15.4 ML LTS** cluster (not serverless or interactive) for optimized batch inference throughput. |\n",{"data":31,"body":36},{"name":4,"description":6,"compatibility":32,"metadata":33,"parent":35},"Requires databricks CLI (>= v1.0.0)",{"version":34},"0.2.0","databricks-core",{"type":37,"children":38},"root",[39,47,79,86,108,169,193,212,1235,1240,1246,1276,1352,1387,1410,1424,1440,1471,1488,1527,1578,1641,1660,1666,1737,2006,2053,2060,2151,2245,2288,2296,2375,2381,2468,2474,2828],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":46},"text","Databricks AI Functions",{"type":40,"tag":48,"props":49,"children":50},"blockquote",{},[51],{"type":40,"tag":52,"props":53,"children":54},"p",{},[55,61,63,71,73],{"type":40,"tag":56,"props":57,"children":58},"strong",{},[59],{"type":45,"value":60},"Official Docs:",{"type":45,"value":62}," ",{"type":40,"tag":64,"props":65,"children":69},"a",{"href":66,"rel":67},"https:\u002F\u002Fdocs.databricks.com\u002Flarge-language-models\u002Fai-functions",[68],"nofollow",[70],{"type":45,"value":66},{"type":45,"value":72},"\nIndividual function reference: ",{"type":40,"tag":64,"props":74,"children":77},{"href":75,"rel":76},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002F",[68],[78],{"type":45,"value":75},{"type":40,"tag":80,"props":81,"children":83},"h2",{"id":82},"overview",[84],{"type":45,"value":85},"Overview",{"type":40,"tag":52,"props":87,"children":88},{},[89,91,98,100,106],{"type":45,"value":90},"Databricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as ",{"type":40,"tag":92,"props":93,"children":95},"code",{"className":94},[],[96],{"type":45,"value":97},"UPPER()",{"type":45,"value":99}," or ",{"type":40,"tag":92,"props":101,"children":103},{"className":102},[],[104],{"type":45,"value":105},"LENGTH()",{"type":45,"value":107},", and are optimized for batch inference at scale.",{"type":40,"tag":52,"props":109,"children":110},{},[111,124,126,131,133,139,141,146,148,153,155,160,162,168],{"type":40,"tag":56,"props":112,"children":113},{},[114,116,122],{"type":45,"value":115},"Always prefer a task-specific function over ",{"type":40,"tag":92,"props":117,"children":119},{"className":118},[],[120],{"type":45,"value":121},"ai_query",{"type":45,"value":123},".",{"type":45,"value":125}," Reach for ",{"type":40,"tag":92,"props":127,"children":129},{"className":128},[],[130],{"type":45,"value":121},{"type":45,"value":132}," only when no task function fits (custom\u002Fexternal endpoints, multimodal, or JSON beyond ",{"type":40,"tag":92,"props":134,"children":136},{"className":135},[],[137],{"type":45,"value":138},"ai_extract",{"type":45,"value":140},"'s limits). Every function below shares a baseline: ",{"type":40,"tag":56,"props":142,"children":143},{},[144],{"type":45,"value":145},"DBR 15.1+",{"type":45,"value":147}," (notebooks) \u002F ",{"type":40,"tag":56,"props":149,"children":150},{},[151],{"type":45,"value":152},"15.4 ML LTS",{"type":45,"value":154}," (batch), ",{"type":40,"tag":56,"props":156,"children":157},{},[158],{"type":45,"value":159},"not on SQL Warehouse Classic",{"type":45,"value":161},", and region must support AI Functions — the Prereqs column lists only what's ",{"type":40,"tag":163,"props":164,"children":165},"em",{},[166],{"type":45,"value":167},"additional",{"type":45,"value":123},{"type":40,"tag":52,"props":170,"children":171},{},[172,177,179,184,186,191],{"type":40,"tag":56,"props":173,"children":174},{},[175],{"type":45,"value":176},"Cost & speed — each call is an LLM inference (slow and billed per token).",{"type":45,"value":178}," Run a function ",{"type":40,"tag":56,"props":180,"children":181},{},[182],{"type":45,"value":183},"once",{"type":45,"value":185}," per row and persist the result to a Delta table; never re-invoke it on every downstream query. In demos, ",{"type":40,"tag":56,"props":187,"children":188},{},[189],{"type":45,"value":190},"avoid generating tables with millions of rows",{"type":45,"value":192}," — sample the input when needed so the demo runs quickly. Materialize once, then query the cheap Delta output.",{"type":40,"tag":52,"props":194,"children":195},{},[196,198,203,205,210],{"type":45,"value":197},"The ",{"type":40,"tag":56,"props":199,"children":200},{},[201],{"type":45,"value":202},"Function",{"type":45,"value":204}," column links to the in-repo deep reference (full options, schemas, examples); ",{"type":40,"tag":56,"props":206,"children":207},{},[208],{"type":45,"value":209},"Docs",{"type":45,"value":211}," links to the official page.",{"type":40,"tag":213,"props":214,"children":215},"table",{},[216,253],{"type":40,"tag":217,"props":218,"children":219},"thead",{},[220],{"type":40,"tag":221,"props":222,"children":223},"tr",{},[224,229,234,239,244,249],{"type":40,"tag":225,"props":226,"children":227},"th",{},[228],{"type":45,"value":202},{"type":40,"tag":225,"props":230,"children":231},{},[232],{"type":45,"value":233},"Task",{"type":40,"tag":225,"props":235,"children":236},{},[237],{"type":45,"value":238},"Input",{"type":40,"tag":225,"props":240,"children":241},{},[242],{"type":45,"value":243},"Output",{"type":40,"tag":225,"props":245,"children":246},{},[247],{"type":45,"value":248},"Extra prereqs",{"type":40,"tag":225,"props":250,"children":251},{},[252],{"type":45,"value":209},{"type":40,"tag":254,"props":255,"children":256},"tbody",{},[257,350,425,496,548,599,666,727,787,845,911,981,1104],{"type":40,"tag":221,"props":258,"children":259},{},[260,274,279,288,335,340],{"type":40,"tag":261,"props":262,"children":263},"td",{},[264],{"type":40,"tag":64,"props":265,"children":267},{"href":266},"references\u002F1-task-functions.md#ai_analyze_sentiment",[268],{"type":40,"tag":92,"props":269,"children":271},{"className":270},[],[272],{"type":45,"value":273},"ai_analyze_sentiment",{"type":40,"tag":261,"props":275,"children":276},{},[277],{"type":45,"value":278},"Sentiment scoring",{"type":40,"tag":261,"props":280,"children":281},{},[282],{"type":40,"tag":92,"props":283,"children":285},{"className":284},[],[286],{"type":45,"value":287},"content STRING",{"type":40,"tag":261,"props":289,"children":290},{},[291,297,299,305,307,313,314,320,321,327,329],{"type":40,"tag":92,"props":292,"children":294},{"className":293},[],[295],{"type":45,"value":296},"STRING",{"type":45,"value":298}," — ",{"type":40,"tag":92,"props":300,"children":302},{"className":301},[],[303],{"type":45,"value":304},"positive",{"type":45,"value":306},"\u002F",{"type":40,"tag":92,"props":308,"children":310},{"className":309},[],[311],{"type":45,"value":312},"negative",{"type":45,"value":306},{"type":40,"tag":92,"props":315,"children":317},{"className":316},[],[318],{"type":45,"value":319},"neutral",{"type":45,"value":306},{"type":40,"tag":92,"props":322,"children":324},{"className":323},[],[325],{"type":45,"value":326},"mixed",{"type":45,"value":328},", or ",{"type":40,"tag":92,"props":330,"children":332},{"className":331},[],[333],{"type":45,"value":334},"NULL",{"type":40,"tag":261,"props":336,"children":337},{},[338],{"type":45,"value":339},"—",{"type":40,"tag":261,"props":341,"children":342},{},[343],{"type":40,"tag":64,"props":344,"children":347},{"href":345,"rel":346},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_analyze_sentiment",[68],[348],{"type":45,"value":349},"↗",{"type":40,"tag":221,"props":351,"children":352},{},[353,366,371,396,412,416],{"type":40,"tag":261,"props":354,"children":355},{},[356],{"type":40,"tag":64,"props":357,"children":359},{"href":358},"references\u002F1-task-functions.md#ai_classify",[360],{"type":40,"tag":92,"props":361,"children":363},{"className":362},[],[364],{"type":45,"value":365},"ai_classify",{"type":40,"tag":261,"props":367,"children":368},{},[369],{"type":45,"value":370},"Fixed-label routing",{"type":40,"tag":261,"props":372,"children":373},{},[374,380,382,388,390],{"type":40,"tag":92,"props":375,"children":377},{"className":376},[],[378],{"type":45,"value":379},"content STRING|VARIANT",{"type":45,"value":381},", ",{"type":40,"tag":92,"props":383,"children":385},{"className":384},[],[386],{"type":45,"value":387},"labels",{"type":45,"value":389}," (2–500), ",{"type":40,"tag":92,"props":391,"children":393},{"className":392},[],[394],{"type":45,"value":395},"[options MAP]",{"type":40,"tag":261,"props":397,"children":398},{},[399,405,406],{"type":40,"tag":92,"props":400,"children":402},{"className":401},[],[403],{"type":45,"value":404},"VARIANT",{"type":45,"value":298},{"type":40,"tag":92,"props":407,"children":409},{"className":408},[],[410],{"type":45,"value":411},"{response:[label], error_message}",{"type":40,"tag":261,"props":413,"children":414},{},[415],{"type":45,"value":339},{"type":40,"tag":261,"props":417,"children":418},{},[419],{"type":40,"tag":64,"props":420,"children":423},{"href":421,"rel":422},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_classify",[68],[424],{"type":45,"value":349},{"type":40,"tag":221,"props":426,"children":427},{},[428,440,445,467,482,487],{"type":40,"tag":261,"props":429,"children":430},{},[431],{"type":40,"tag":64,"props":432,"children":434},{"href":433},"references\u002F1-task-functions.md#ai_extract",[435],{"type":40,"tag":92,"props":436,"children":438},{"className":437},[],[439],{"type":45,"value":138},{"type":40,"tag":261,"props":441,"children":442},{},[443],{"type":45,"value":444},"Entity \u002F field extraction",{"type":40,"tag":261,"props":446,"children":447},{},[448,453,454,460,462],{"type":40,"tag":92,"props":449,"children":451},{"className":450},[],[452],{"type":45,"value":379},{"type":45,"value":381},{"type":40,"tag":92,"props":455,"children":457},{"className":456},[],[458],{"type":45,"value":459},"schema STRING",{"type":45,"value":461}," (JSON), ",{"type":40,"tag":92,"props":463,"children":465},{"className":464},[],[466],{"type":45,"value":395},{"type":40,"tag":261,"props":468,"children":469},{},[470,475,476],{"type":40,"tag":92,"props":471,"children":473},{"className":472},[],[474],{"type":45,"value":404},{"type":45,"value":298},{"type":40,"tag":92,"props":477,"children":479},{"className":478},[],[480],{"type":45,"value":481},"{response:{…}, error_message, metadata}",{"type":40,"tag":261,"props":483,"children":484},{},[485],{"type":45,"value":486},"≤256 fields, ≤12 nesting levels",{"type":40,"tag":261,"props":488,"children":489},{},[490],{"type":40,"tag":64,"props":491,"children":494},{"href":492,"rel":493},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_extract",[68],[495],{"type":45,"value":349},{"type":40,"tag":221,"props":497,"children":498},{},[499,512,517,525,535,539],{"type":40,"tag":261,"props":500,"children":501},{},[502],{"type":40,"tag":64,"props":503,"children":505},{"href":504},"references\u002F1-task-functions.md#ai_fix_grammar",[506],{"type":40,"tag":92,"props":507,"children":509},{"className":508},[],[510],{"type":45,"value":511},"ai_fix_grammar",{"type":40,"tag":261,"props":513,"children":514},{},[515],{"type":45,"value":516},"Grammar correction",{"type":40,"tag":261,"props":518,"children":519},{},[520],{"type":40,"tag":92,"props":521,"children":523},{"className":522},[],[524],{"type":45,"value":287},{"type":40,"tag":261,"props":526,"children":527},{},[528,533],{"type":40,"tag":92,"props":529,"children":531},{"className":530},[],[532],{"type":45,"value":296},{"type":45,"value":534}," (corrected)",{"type":40,"tag":261,"props":536,"children":537},{},[538],{"type":45,"value":339},{"type":40,"tag":261,"props":540,"children":541},{},[542],{"type":40,"tag":64,"props":543,"children":546},{"href":544,"rel":545},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_fix_grammar",[68],[547],{"type":45,"value":349},{"type":40,"tag":221,"props":549,"children":550},{},[551,564,569,578,586,590],{"type":40,"tag":261,"props":552,"children":553},{},[554],{"type":40,"tag":64,"props":555,"children":557},{"href":556},"references\u002F1-task-functions.md#ai_gen",[558],{"type":40,"tag":92,"props":559,"children":561},{"className":560},[],[562],{"type":45,"value":563},"ai_gen",{"type":40,"tag":261,"props":565,"children":566},{},[567],{"type":45,"value":568},"Free-form generation",{"type":40,"tag":261,"props":570,"children":571},{},[572],{"type":40,"tag":92,"props":573,"children":575},{"className":574},[],[576],{"type":45,"value":577},"prompt STRING",{"type":40,"tag":261,"props":579,"children":580},{},[581],{"type":40,"tag":92,"props":582,"children":584},{"className":583},[],[585],{"type":45,"value":296},{"type":40,"tag":261,"props":587,"children":588},{},[589],{"type":45,"value":339},{"type":40,"tag":261,"props":591,"children":592},{},[593],{"type":40,"tag":64,"props":594,"children":597},{"href":595,"rel":596},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_gen",[68],[598],{"type":45,"value":349},{"type":40,"tag":221,"props":600,"children":601},{},[602,615,620,635,653,657],{"type":40,"tag":261,"props":603,"children":604},{},[605],{"type":40,"tag":64,"props":606,"children":608},{"href":607},"references\u002F1-task-functions.md#ai_mask",[609],{"type":40,"tag":92,"props":610,"children":612},{"className":611},[],[613],{"type":45,"value":614},"ai_mask",{"type":40,"tag":261,"props":616,"children":617},{},[618],{"type":45,"value":619},"PII redaction",{"type":40,"tag":261,"props":621,"children":622},{},[623,628,629],{"type":40,"tag":92,"props":624,"children":626},{"className":625},[],[627],{"type":45,"value":287},{"type":45,"value":381},{"type":40,"tag":92,"props":630,"children":632},{"className":631},[],[633],{"type":45,"value":634},"labels ARRAY\u003CSTRING>",{"type":40,"tag":261,"props":636,"children":637},{},[638,643,645,651],{"type":40,"tag":92,"props":639,"children":641},{"className":640},[],[642],{"type":45,"value":296},{"type":45,"value":644}," (entities → ",{"type":40,"tag":92,"props":646,"children":648},{"className":647},[],[649],{"type":45,"value":650},"[MASKED]",{"type":45,"value":652},")",{"type":40,"tag":261,"props":654,"children":655},{},[656],{"type":45,"value":339},{"type":40,"tag":261,"props":658,"children":659},{},[660],{"type":40,"tag":64,"props":661,"children":664},{"href":662,"rel":663},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_mask",[68],[665],{"type":45,"value":349},{"type":40,"tag":221,"props":667,"children":668},{},[669,682,687,703,714,718],{"type":40,"tag":261,"props":670,"children":671},{},[672],{"type":40,"tag":64,"props":673,"children":675},{"href":674},"references\u002F1-task-functions.md#ai_similarity",[676],{"type":40,"tag":92,"props":677,"children":679},{"className":678},[],[680],{"type":45,"value":681},"ai_similarity",{"type":40,"tag":261,"props":683,"children":684},{},[685],{"type":45,"value":686},"Semantic similarity",{"type":40,"tag":261,"props":688,"children":689},{},[690,696,697],{"type":40,"tag":92,"props":691,"children":693},{"className":692},[],[694],{"type":45,"value":695},"expr1 STRING",{"type":45,"value":381},{"type":40,"tag":92,"props":698,"children":700},{"className":699},[],[701],{"type":45,"value":702},"expr2 STRING",{"type":40,"tag":261,"props":704,"children":705},{},[706,712],{"type":40,"tag":92,"props":707,"children":709},{"className":708},[],[710],{"type":45,"value":711},"FLOAT",{"type":45,"value":713}," (0.0–1.0)",{"type":40,"tag":261,"props":715,"children":716},{},[717],{"type":45,"value":339},{"type":40,"tag":261,"props":719,"children":720},{},[721],{"type":40,"tag":64,"props":722,"children":725},{"href":723,"rel":724},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_similarity",[68],[726],{"type":45,"value":349},{"type":40,"tag":221,"props":728,"children":729},{},[730,743,748,765,773,778],{"type":40,"tag":261,"props":731,"children":732},{},[733],{"type":40,"tag":64,"props":734,"children":736},{"href":735},"references\u002F1-task-functions.md#ai_summarize",[737],{"type":40,"tag":92,"props":738,"children":740},{"className":739},[],[741],{"type":45,"value":742},"ai_summarize",{"type":40,"tag":261,"props":744,"children":745},{},[746],{"type":45,"value":747},"Summarization",{"type":40,"tag":261,"props":749,"children":750},{},[751,756,757,763],{"type":40,"tag":92,"props":752,"children":754},{"className":753},[],[755],{"type":45,"value":287},{"type":45,"value":381},{"type":40,"tag":92,"props":758,"children":760},{"className":759},[],[761],{"type":45,"value":762},"[max_words INT]",{"type":45,"value":764}," (0 = uncapped)",{"type":40,"tag":261,"props":766,"children":767},{},[768],{"type":40,"tag":92,"props":769,"children":771},{"className":770},[],[772],{"type":45,"value":296},{"type":40,"tag":261,"props":774,"children":775},{},[776],{"type":45,"value":777},"Public Preview; English-tuned",{"type":40,"tag":261,"props":779,"children":780},{},[781],{"type":40,"tag":64,"props":782,"children":785},{"href":783,"rel":784},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_summarize",[68],[786],{"type":45,"value":349},{"type":40,"tag":221,"props":788,"children":789},{},[790,803,808,823,831,836],{"type":40,"tag":261,"props":791,"children":792},{},[793],{"type":40,"tag":64,"props":794,"children":796},{"href":795},"references\u002F1-task-functions.md#ai_translate",[797],{"type":40,"tag":92,"props":798,"children":800},{"className":799},[],[801],{"type":45,"value":802},"ai_translate",{"type":40,"tag":261,"props":804,"children":805},{},[806],{"type":45,"value":807},"Translation",{"type":40,"tag":261,"props":809,"children":810},{},[811,816,817],{"type":40,"tag":92,"props":812,"children":814},{"className":813},[],[815],{"type":45,"value":287},{"type":45,"value":381},{"type":40,"tag":92,"props":818,"children":820},{"className":819},[],[821],{"type":45,"value":822},"to_lang STRING",{"type":40,"tag":261,"props":824,"children":825},{},[826],{"type":40,"tag":92,"props":827,"children":829},{"className":828},[],[830],{"type":45,"value":296},{"type":40,"tag":261,"props":832,"children":833},{},[834],{"type":45,"value":835},"Langs: en, fr, de, hi, it, pt, es, th",{"type":40,"tag":261,"props":837,"children":838},{},[839],{"type":40,"tag":64,"props":840,"children":843},{"href":841,"rel":842},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_translate",[68],[844],{"type":45,"value":349},{"type":40,"tag":221,"props":846,"children":847},{},[848,861,866,882,892,902],{"type":40,"tag":261,"props":849,"children":850},{},[851],{"type":40,"tag":64,"props":852,"children":854},{"href":853},"references\u002F1-task-functions.md#ai_parse_document",[855],{"type":40,"tag":92,"props":856,"children":858},{"className":857},[],[859],{"type":45,"value":860},"ai_parse_document",{"type":40,"tag":261,"props":862,"children":863},{},[864],{"type":45,"value":865},"Parse PDF \u002F Office \u002F images",{"type":40,"tag":261,"props":867,"children":868},{},[869,875,876],{"type":40,"tag":92,"props":870,"children":872},{"className":871},[],[873],{"type":45,"value":874},"content BINARY",{"type":45,"value":381},{"type":40,"tag":92,"props":877,"children":879},{"className":878},[],[880],{"type":45,"value":881},"[Map('version','2.0', …)]",{"type":40,"tag":261,"props":883,"children":884},{},[885,890],{"type":40,"tag":92,"props":886,"children":888},{"className":887},[],[889],{"type":45,"value":404},{"type":45,"value":891}," — pages, elements, error_status",{"type":40,"tag":261,"props":893,"children":894},{},[895,900],{"type":40,"tag":56,"props":896,"children":897},{},[898],{"type":45,"value":899},"DBR 17.3+",{"type":45,"value":901},"; ≤500 pages \u002F 100 MB",{"type":40,"tag":261,"props":903,"children":904},{},[905],{"type":40,"tag":64,"props":906,"children":909},{"href":907,"rel":908},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_parse_document",[68],[910],{"type":45,"value":349},{"type":40,"tag":221,"props":912,"children":913},{},[914,927,932,947,962,972],{"type":40,"tag":261,"props":915,"children":916},{},[917],{"type":40,"tag":64,"props":918,"children":920},{"href":919},"references\u002F1-task-functions.md#ai_prep_search",[921],{"type":40,"tag":92,"props":922,"children":924},{"className":923},[],[925],{"type":45,"value":926},"ai_prep_search",{"type":40,"tag":261,"props":928,"children":929},{},[930],{"type":45,"value":931},"RAG chunking from parsed docs",{"type":40,"tag":261,"props":933,"children":934},{},[935,941,942],{"type":40,"tag":92,"props":936,"children":938},{"className":937},[],[939],{"type":45,"value":940},"parsed VARIANT",{"type":45,"value":381},{"type":40,"tag":92,"props":943,"children":945},{"className":944},[],[946],{"type":45,"value":395},{"type":40,"tag":261,"props":948,"children":949},{},[950,955,956],{"type":40,"tag":92,"props":951,"children":953},{"className":952},[],[954],{"type":45,"value":404},{"type":45,"value":298},{"type":40,"tag":92,"props":957,"children":959},{"className":958},[],[960],{"type":45,"value":961},"{document:{contents, pages, source_uri}, error_status}",{"type":40,"tag":261,"props":963,"children":964},{},[965,970],{"type":40,"tag":56,"props":966,"children":967},{},[968],{"type":45,"value":969},"DBR 18.2+",{"type":45,"value":971}," (serverless env v3+)",{"type":40,"tag":261,"props":973,"children":974},{},[975],{"type":40,"tag":64,"props":976,"children":979},{"href":977,"rel":978},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_prep_search",[68],[980],{"type":45,"value":349},{"type":40,"tag":221,"props":982,"children":983},{},[984,996,1007,1058,1077,1095],{"type":40,"tag":261,"props":985,"children":986},{},[987],{"type":40,"tag":64,"props":988,"children":990},{"href":989},"references\u002F2-ai-query.md",[991],{"type":40,"tag":92,"props":992,"children":994},{"className":993},[],[995],{"type":45,"value":121},{"type":40,"tag":261,"props":997,"children":998},{},[999,1001,1006],{"type":45,"value":1000},"Any serving endpoint (built-in foundation or custom), multimodal, complex JSON (",{"type":40,"tag":56,"props":1002,"children":1003},{},[1004],{"type":45,"value":1005},"last resort",{"type":45,"value":652},{"type":40,"tag":261,"props":1008,"children":1009},{},[1010,1016,1017,1023,1024,1030,1031,1037,1038,1044,1045,1051,1052],{"type":40,"tag":92,"props":1011,"children":1013},{"className":1012},[],[1014],{"type":45,"value":1015},"endpoint STRING",{"type":45,"value":381},{"type":40,"tag":92,"props":1018,"children":1020},{"className":1019},[],[1021],{"type":45,"value":1022},"request STRING|STRUCT",{"type":45,"value":381},{"type":40,"tag":92,"props":1025,"children":1027},{"className":1026},[],[1028],{"type":45,"value":1029},"[returnType]",{"type":45,"value":381},{"type":40,"tag":92,"props":1032,"children":1034},{"className":1033},[],[1035],{"type":45,"value":1036},"[failOnError BOOL]",{"type":45,"value":381},{"type":40,"tag":92,"props":1039,"children":1041},{"className":1040},[],[1042],{"type":45,"value":1043},"[modelParameters STRUCT]",{"type":45,"value":381},{"type":40,"tag":92,"props":1046,"children":1048},{"className":1047},[],[1049],{"type":45,"value":1050},"[responseFormat STRING]",{"type":45,"value":381},{"type":40,"tag":92,"props":1053,"children":1055},{"className":1054},[],[1056],{"type":45,"value":1057},"[files]",{"type":40,"tag":261,"props":1059,"children":1060},{},[1061,1063,1069,1071],{"type":45,"value":1062},"Parsed response; with ",{"type":40,"tag":92,"props":1064,"children":1066},{"className":1065},[],[1067],{"type":45,"value":1068},"failOnError => false",{"type":45,"value":1070}," a ",{"type":40,"tag":92,"props":1072,"children":1074},{"className":1073},[],[1075],{"type":45,"value":1076},"STRUCT{response, errorMessage}",{"type":40,"tag":261,"props":1078,"children":1079},{},[1080,1085,1087,1093],{"type":40,"tag":56,"props":1081,"children":1082},{},[1083],{"type":45,"value":1084},"Pro\u002FServerless",{"type":45,"value":1086}," warehouse; ",{"type":40,"tag":92,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":45,"value":1092},"CAN QUERY",{"type":45,"value":1094}," on endpoint",{"type":40,"tag":261,"props":1096,"children":1097},{},[1098],{"type":40,"tag":64,"props":1099,"children":1102},{"href":1100,"rel":1101},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_query",[68],[1103],{"type":45,"value":349},{"type":40,"tag":221,"props":1105,"children":1106},{},[1107,1120,1125,1190,1217,1226],{"type":40,"tag":261,"props":1108,"children":1109},{},[1110],{"type":40,"tag":64,"props":1111,"children":1113},{"href":1112},"references\u002F3-ai-forecast.md",[1114],{"type":40,"tag":92,"props":1115,"children":1117},{"className":1116},[],[1118],{"type":45,"value":1119},"ai_forecast",{"type":40,"tag":261,"props":1121,"children":1122},{},[1123],{"type":45,"value":1124},"Time series forecasting (table-valued)",{"type":40,"tag":261,"props":1126,"children":1127},{},[1128,1134,1135,1141,1142,1148,1149,1155,1156,1162,1163,1169,1170,1176,1177,1183,1184],{"type":40,"tag":92,"props":1129,"children":1131},{"className":1130},[],[1132],{"type":45,"value":1133},"observed TABLE",{"type":45,"value":381},{"type":40,"tag":92,"props":1136,"children":1138},{"className":1137},[],[1139],{"type":45,"value":1140},"horizon",{"type":45,"value":381},{"type":40,"tag":92,"props":1143,"children":1145},{"className":1144},[],[1146],{"type":45,"value":1147},"time_col",{"type":45,"value":381},{"type":40,"tag":92,"props":1150,"children":1152},{"className":1151},[],[1153],{"type":45,"value":1154},"value_col",{"type":45,"value":381},{"type":40,"tag":92,"props":1157,"children":1159},{"className":1158},[],[1160],{"type":45,"value":1161},"[group_col]",{"type":45,"value":381},{"type":40,"tag":92,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":45,"value":1168},"[prediction_interval_width]",{"type":45,"value":381},{"type":40,"tag":92,"props":1171,"children":1173},{"className":1172},[],[1174],{"type":45,"value":1175},"[frequency]",{"type":45,"value":381},{"type":40,"tag":92,"props":1178,"children":1180},{"className":1179},[],[1181],{"type":45,"value":1182},"[seed]",{"type":45,"value":381},{"type":40,"tag":92,"props":1185,"children":1187},{"className":1186},[],[1188],{"type":45,"value":1189},"[parameters]",{"type":40,"tag":261,"props":1191,"children":1192},{},[1193,1195,1201,1202,1208,1209,1215],{"type":45,"value":1194},"Rows: time\u002Fgroup cols + per value ",{"type":40,"tag":92,"props":1196,"children":1198},{"className":1197},[],[1199],{"type":45,"value":1200},"{v}_forecast",{"type":45,"value":381},{"type":40,"tag":92,"props":1203,"children":1205},{"className":1204},[],[1206],{"type":45,"value":1207},"{v}_upper",{"type":45,"value":381},{"type":40,"tag":92,"props":1210,"children":1212},{"className":1211},[],[1213],{"type":45,"value":1214},"{v}_lower",{"type":45,"value":1216}," (DOUBLE)",{"type":40,"tag":261,"props":1218,"children":1219},{},[1220,1224],{"type":40,"tag":56,"props":1221,"children":1222},{},[1223],{"type":45,"value":1084},{"type":45,"value":1225}," warehouse; Public Preview",{"type":40,"tag":261,"props":1227,"children":1228},{},[1229],{"type":40,"tag":64,"props":1230,"children":1233},{"href":1231,"rel":1232},"https:\u002F\u002Fdocs.databricks.com\u002Fsql\u002Flanguage-manual\u002Ffunctions\u002Fai_forecast",[68],[1234],{"type":45,"value":349},{"type":40,"tag":52,"props":1236,"children":1237},{},[1238],{"type":45,"value":1239},"Models run under Apache 2.0 or LLAMA 3.3 Community License — you are responsible for compliance.",{"type":40,"tag":80,"props":1241,"children":1243},{"id":1242},"patterns",[1244],{"type":45,"value":1245},"Patterns",{"type":40,"tag":52,"props":1247,"children":1248},{},[1249,1254,1255,1260,1261,1266,1268,1274],{"type":40,"tag":56,"props":1250,"children":1251},{},[1252],{"type":45,"value":1253},"Chain task functions to enrich a column in one pass.",{"type":45,"value":62},{"type":40,"tag":92,"props":1256,"children":1258},{"className":1257},[],[1259],{"type":45,"value":365},{"type":45,"value":306},{"type":40,"tag":92,"props":1262,"children":1264},{"className":1263},[],[1265],{"type":45,"value":138},{"type":45,"value":1267}," return a VARIANT — read it with the colon operator (",{"type":40,"tag":92,"props":1269,"children":1271},{"className":1270},[],[1272],{"type":45,"value":1273},":response",{"type":45,"value":1275},"):",{"type":40,"tag":1277,"props":1278,"children":1283},"pre",{"className":1279,"code":1280,"language":1281,"meta":1282,"style":1282},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","SELECT id,\n  ai_analyze_sentiment(content)                                                   AS sentiment,\n  ai_summarize(content, 30)                                                       AS summary,\n  ai_classify(content, '[\"technical\",\"billing\",\"other\"]', map('version','2.0')):response[0]::STRING AS category,\n  ai_extract(content, '[\"product\",\"error_code\",\"date\"]', map('version','2.0')):response:product::STRING AS product,\n  ai_fix_grammar(content)                                                         AS content_clean\nFROM raw_feedback;\n","sql","",[1284],{"type":40,"tag":92,"props":1285,"children":1286},{"__ignoreMap":1282},[1287,1298,1307,1316,1325,1334,1343],{"type":40,"tag":1288,"props":1289,"children":1292},"span",{"class":1290,"line":1291},"line",1,[1293],{"type":40,"tag":1288,"props":1294,"children":1295},{},[1296],{"type":45,"value":1297},"SELECT id,\n",{"type":40,"tag":1288,"props":1299,"children":1301},{"class":1290,"line":1300},2,[1302],{"type":40,"tag":1288,"props":1303,"children":1304},{},[1305],{"type":45,"value":1306},"  ai_analyze_sentiment(content)                                                   AS sentiment,\n",{"type":40,"tag":1288,"props":1308,"children":1310},{"class":1290,"line":1309},3,[1311],{"type":40,"tag":1288,"props":1312,"children":1313},{},[1314],{"type":45,"value":1315},"  ai_summarize(content, 30)                                                       AS summary,\n",{"type":40,"tag":1288,"props":1317,"children":1319},{"class":1290,"line":1318},4,[1320],{"type":40,"tag":1288,"props":1321,"children":1322},{},[1323],{"type":45,"value":1324},"  ai_classify(content, '[\"technical\",\"billing\",\"other\"]', map('version','2.0')):response[0]::STRING AS category,\n",{"type":40,"tag":1288,"props":1326,"children":1328},{"class":1290,"line":1327},5,[1329],{"type":40,"tag":1288,"props":1330,"children":1331},{},[1332],{"type":45,"value":1333},"  ai_extract(content, '[\"product\",\"error_code\",\"date\"]', map('version','2.0')):response:product::STRING AS product,\n",{"type":40,"tag":1288,"props":1335,"children":1337},{"class":1290,"line":1336},6,[1338],{"type":40,"tag":1288,"props":1339,"children":1340},{},[1341],{"type":45,"value":1342},"  ai_fix_grammar(content)                                                         AS content_clean\n",{"type":40,"tag":1288,"props":1344,"children":1346},{"class":1290,"line":1345},7,[1347],{"type":40,"tag":1288,"props":1348,"children":1349},{},[1350],{"type":45,"value":1351},"FROM raw_feedback;\n",{"type":40,"tag":52,"props":1353,"children":1354},{},[1355,1357,1362,1364,1370,1372,1378,1380,1386],{"type":45,"value":1356},"In ",{"type":40,"tag":56,"props":1358,"children":1359},{},[1360],{"type":45,"value":1361},"PySpark",{"type":45,"value":1363},", call any of these inside ",{"type":40,"tag":92,"props":1365,"children":1367},{"className":1366},[],[1368],{"type":45,"value":1369},"expr(...)",{"type":45,"value":1371},": ",{"type":40,"tag":92,"props":1373,"children":1375},{"className":1374},[],[1376],{"type":45,"value":1377},"df.withColumn(\"category\", expr(\"ai_classify(content, '[\\\"a\\\",\\\"b\\\"]', map('version','2.0')):response[0]::STRING\"))",{"type":45,"value":1379}," — and read VARIANT fields via ",{"type":40,"tag":92,"props":1381,"children":1383},{"className":1382},[],[1384],{"type":45,"value":1385},"selectExpr(\"col:response:field::STRING AS field\")",{"type":45,"value":123},{"type":40,"tag":52,"props":1388,"children":1389},{},[1390,1395,1396,1402,1404,1409],{"type":40,"tag":56,"props":1391,"children":1392},{},[1393],{"type":45,"value":1394},"PII redaction before storage",{"type":45,"value":298},{"type":40,"tag":92,"props":1397,"children":1399},{"className":1398},[],[1400],{"type":45,"value":1401},"ai_mask(content, ARRAY(entity_types))",{"type":45,"value":1403}," returns text with entities → ",{"type":40,"tag":92,"props":1405,"children":1407},{"className":1406},[],[1408],{"type":45,"value":650},{"type":45,"value":123},{"type":40,"tag":1277,"props":1411,"children":1413},{"className":1279,"code":1412,"language":1281,"meta":1282,"style":1282},"SELECT ai_mask(message, array('person','email','phone','address')) AS message_safe FROM raw_messages;\n",[1414],{"type":40,"tag":92,"props":1415,"children":1416},{"__ignoreMap":1282},[1417],{"type":40,"tag":1288,"props":1418,"children":1419},{"class":1290,"line":1291},[1420],{"type":40,"tag":1288,"props":1421,"children":1422},{},[1423],{"type":45,"value":1412},{"type":40,"tag":52,"props":1425,"children":1426},{},[1427,1432,1433,1438],{"type":40,"tag":56,"props":1428,"children":1429},{},[1430],{"type":45,"value":1431},"Semantic matching \u002F dedup",{"type":45,"value":298},{"type":40,"tag":92,"props":1434,"children":1436},{"className":1435},[],[1437],{"type":45,"value":681},{"type":45,"value":1439}," returns 0–1; self-join and threshold:",{"type":40,"tag":1277,"props":1441,"children":1443},{"className":1279,"code":1442,"language":1281,"meta":1282,"style":1282},"SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score\nFROM companies a JOIN companies b ON a.id \u003C b.id\nWHERE ai_similarity(a.name, b.name) > 0.85;\n",[1444],{"type":40,"tag":92,"props":1445,"children":1446},{"__ignoreMap":1282},[1447,1455,1463],{"type":40,"tag":1288,"props":1448,"children":1449},{"class":1290,"line":1291},[1450],{"type":40,"tag":1288,"props":1451,"children":1452},{},[1453],{"type":45,"value":1454},"SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score\n",{"type":40,"tag":1288,"props":1456,"children":1457},{"class":1290,"line":1300},[1458],{"type":40,"tag":1288,"props":1459,"children":1460},{},[1461],{"type":45,"value":1462},"FROM companies a JOIN companies b ON a.id \u003C b.id\n",{"type":40,"tag":1288,"props":1464,"children":1465},{"class":1290,"line":1309},[1466],{"type":40,"tag":1288,"props":1467,"children":1468},{},[1469],{"type":45,"value":1470},"WHERE ai_similarity(a.name, b.name) > 0.85;\n",{"type":40,"tag":52,"props":1472,"children":1473},{},[1474,1479,1481,1486],{"type":40,"tag":56,"props":1475,"children":1476},{},[1477],{"type":45,"value":1478},"Forecasting",{"type":45,"value":1480}," — table-valued; one row per future period (+ per group). Full param\u002Fgroup\u002Finterval forms → ",{"type":40,"tag":64,"props":1482,"children":1483},{"href":1112},[1484],{"type":45,"value":1485},"3-ai-forecast.md",{"type":45,"value":1487},":",{"type":40,"tag":1277,"props":1489,"children":1491},{"className":1279,"code":1490,"language":1281,"meta":1282,"style":1282},"SELECT * FROM ai_forecast(\n    observed => TABLE(SELECT date, sales FROM daily_sales),\n    horizon => '2026-12-31', time_col => 'date', value_col => 'sales');\n-- Returns: date, sales_forecast, sales_upper, sales_lower\n",[1492],{"type":40,"tag":92,"props":1493,"children":1494},{"__ignoreMap":1282},[1495,1503,1511,1519],{"type":40,"tag":1288,"props":1496,"children":1497},{"class":1290,"line":1291},[1498],{"type":40,"tag":1288,"props":1499,"children":1500},{},[1501],{"type":45,"value":1502},"SELECT * FROM ai_forecast(\n",{"type":40,"tag":1288,"props":1504,"children":1505},{"class":1290,"line":1300},[1506],{"type":40,"tag":1288,"props":1507,"children":1508},{},[1509],{"type":45,"value":1510},"    observed => TABLE(SELECT date, sales FROM daily_sales),\n",{"type":40,"tag":1288,"props":1512,"children":1513},{"class":1290,"line":1309},[1514],{"type":40,"tag":1288,"props":1515,"children":1516},{},[1517],{"type":45,"value":1518},"    horizon => '2026-12-31', time_col => 'date', value_col => 'sales');\n",{"type":40,"tag":1288,"props":1520,"children":1521},{"class":1290,"line":1318},[1522],{"type":40,"tag":1288,"props":1523,"children":1524},{},[1525],{"type":45,"value":1526},"-- Returns: date, sales_forecast, sales_upper, sales_lower\n",{"type":40,"tag":52,"props":1528,"children":1529},{},[1530,1540,1542,1547,1549,1555,1557,1563,1564,1570,1572,1577],{"type":40,"tag":56,"props":1531,"children":1532},{},[1533,1535],{"type":45,"value":1534},"Nested JSON via ",{"type":40,"tag":92,"props":1536,"children":1538},{"className":1537},[],[1539],{"type":45,"value":121},{"type":45,"value":1541}," (last resort — only past ",{"type":40,"tag":92,"props":1543,"children":1545},{"className":1544},[],[1546],{"type":45,"value":138},{"type":45,"value":1548},"'s limits) — parse the response with ",{"type":40,"tag":92,"props":1550,"children":1552},{"className":1551},[],[1553],{"type":45,"value":1554},"from_json",{"type":45,"value":1556},". Model names, multimodal ",{"type":40,"tag":92,"props":1558,"children":1560},{"className":1559},[],[1561],{"type":45,"value":1562},"files =>",{"type":45,"value":381},{"type":40,"tag":92,"props":1565,"children":1567},{"className":1566},[],[1568],{"type":45,"value":1569},"modelParameters",{"type":45,"value":1571},", SQL UDF → ",{"type":40,"tag":64,"props":1573,"children":1574},{"href":989},[1575],{"type":45,"value":1576},"2-ai-query.md",{"type":45,"value":1487},{"type":40,"tag":1277,"props":1579,"children":1581},{"className":1279,"code":1580,"language":1281,"meta":1282,"style":1282},"SELECT from_json(\n    ai_query('databricks-claude-sonnet-4',\n        concat('Extract invoice as JSON with nested line_items array: ', text_blocks),\n        responseFormat => '{\"type\":\"json_object\"}', failOnError => false).response,\n    'STRUCT\u003Cnumero:STRING, total:DOUBLE, line_items:ARRAY\u003CSTRUCT\u003Ccode:STRING, qty:DOUBLE>>>'\n) AS invoice\nFROM parsed_documents;\n",[1582],{"type":40,"tag":92,"props":1583,"children":1584},{"__ignoreMap":1282},[1585,1593,1601,1609,1617,1625,1633],{"type":40,"tag":1288,"props":1586,"children":1587},{"class":1290,"line":1291},[1588],{"type":40,"tag":1288,"props":1589,"children":1590},{},[1591],{"type":45,"value":1592},"SELECT from_json(\n",{"type":40,"tag":1288,"props":1594,"children":1595},{"class":1290,"line":1300},[1596],{"type":40,"tag":1288,"props":1597,"children":1598},{},[1599],{"type":45,"value":1600},"    ai_query('databricks-claude-sonnet-4',\n",{"type":40,"tag":1288,"props":1602,"children":1603},{"class":1290,"line":1309},[1604],{"type":40,"tag":1288,"props":1605,"children":1606},{},[1607],{"type":45,"value":1608},"        concat('Extract invoice as JSON with nested line_items array: ', text_blocks),\n",{"type":40,"tag":1288,"props":1610,"children":1611},{"class":1290,"line":1318},[1612],{"type":40,"tag":1288,"props":1613,"children":1614},{},[1615],{"type":45,"value":1616},"        responseFormat => '{\"type\":\"json_object\"}', failOnError => false).response,\n",{"type":40,"tag":1288,"props":1618,"children":1619},{"class":1290,"line":1327},[1620],{"type":40,"tag":1288,"props":1621,"children":1622},{},[1623],{"type":45,"value":1624},"    'STRUCT\u003Cnumero:STRING, total:DOUBLE, line_items:ARRAY\u003CSTRUCT\u003Ccode:STRING, qty:DOUBLE>>>'\n",{"type":40,"tag":1288,"props":1626,"children":1627},{"class":1290,"line":1336},[1628],{"type":40,"tag":1288,"props":1629,"children":1630},{},[1631],{"type":45,"value":1632},") AS invoice\n",{"type":40,"tag":1288,"props":1634,"children":1635},{"class":1290,"line":1345},[1636],{"type":40,"tag":1288,"props":1637,"children":1638},{},[1639],{"type":45,"value":1640},"FROM parsed_documents;\n",{"type":40,"tag":52,"props":1642,"children":1643},{},[1644,1646,1651,1653,1658],{"type":45,"value":1645},"Document parsing (",{"type":40,"tag":92,"props":1647,"children":1649},{"className":1648},[],[1650],{"type":45,"value":860},{"type":45,"value":1652},") and RAG chunking (",{"type":40,"tag":92,"props":1654,"children":1656},{"className":1655},[],[1657],{"type":45,"value":926},{"type":45,"value":1659},") get their own staged pipeline below.",{"type":40,"tag":80,"props":1661,"children":1663},{"id":1662},"document-processing-pipeline",[1664],{"type":45,"value":1665},"Document Processing Pipeline",{"type":40,"tag":52,"props":1667,"children":1668},{},[1669,1671,1676,1677,1683,1685,1691,1693,1698,1700,1706,1708,1714,1716,1721,1723,1729,1730,1736],{"type":45,"value":1670},"Chain AI Functions stage-by-stage into Delta tables for batch document processing. The example is written as a ",{"type":40,"tag":56,"props":1672,"children":1673},{},[1674],{"type":45,"value":1675},"Spark Declarative Pipeline (SDP \u002F Lakeflow \u002F DLT)",{"type":45,"value":298},{"type":40,"tag":92,"props":1678,"children":1680},{"className":1679},[],[1681],{"type":45,"value":1682},"CREATE OR REFRESH STREAMING TABLE",{"type":45,"value":1684}," with ",{"type":40,"tag":92,"props":1686,"children":1688},{"className":1687},[],[1689],{"type":45,"value":1690},"STREAM(...)",{"type":45,"value":1692}," sources. To run the same logic standalone in a ",{"type":40,"tag":56,"props":1694,"children":1695},{},[1696],{"type":45,"value":1697},"notebook \u002F SQL warehouse",{"type":45,"value":1699},", swap each ",{"type":40,"tag":92,"props":1701,"children":1703},{"className":1702},[],[1704],{"type":45,"value":1705},"CREATE OR REFRESH STREAMING TABLE x AS",{"type":45,"value":1707}," for ",{"type":40,"tag":92,"props":1709,"children":1711},{"className":1710},[],[1712],{"type":45,"value":1713},"CREATE OR REPLACE TABLE x AS",{"type":45,"value":1715}," and drop the ",{"type":40,"tag":92,"props":1717,"children":1719},{"className":1718},[],[1720],{"type":45,"value":1690},{"type":45,"value":1722}," wrappers. In SDP Python it's ",{"type":40,"tag":92,"props":1724,"children":1726},{"className":1725},[],[1727],{"type":45,"value":1728},"@dp.table",{"type":45,"value":1684},{"type":40,"tag":92,"props":1731,"children":1733},{"className":1732},[],[1734],{"type":45,"value":1735},"from pyspark import pipelines as dp",{"type":45,"value":123},{"type":40,"tag":1277,"props":1738,"children":1740},{"className":1279,"code":1739,"language":1281,"meta":1282,"style":1282},"-- Stage 1 — parse binary docs (any type), filter parse errors\nCREATE OR REFRESH STREAMING TABLE raw_parsed AS\nSELECT path,\n  concat_ws('\\n', transform(parsed:document:elements, e -> e:content::STRING)) AS text_blocks,\n  parsed:error_status AS parse_error\nFROM (\n  SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed\n  FROM STREAM read_files('\u002FVolumes\u002Fmy_catalog\u002Fdoc_processing\u002Flanding\u002F', format => 'binaryFile')\n)\nWHERE parsed:error_status IS NULL;\n\n-- Stage 2 — classify document type (cheap, no endpoint selection)\nCREATE OR REFRESH STREAMING TABLE classified_docs AS\nSELECT *,\n  ai_classify(text_blocks, '[\"invoice\",\"purchase_order\",\"receipt\",\"contract\",\"other\"]', map('version','2.0')):response[0]::STRING AS doc_type\nFROM STREAM raw_parsed;\n\n-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`\nCREATE OR REFRESH STREAMING TABLE extracted AS\nSELECT path, doc_type,\n  result:response:invoice_number::STRING AS invoice_number,\n  result:response:vendor_name::STRING    AS vendor_name,\n  result:response:total_amount::DOUBLE   AS total_amount,\n  result:error_message::STRING           AS extract_error\nFROM (\n  SELECT *, ai_extract(text_blocks,\n    '{\"invoice_number\":{\"type\":\"string\"},\"vendor_name\":{\"type\":\"string\"},\"total_amount\":{\"type\":\"number\"}}',\n    map('version','2.0')) AS result\n  FROM STREAM classified_docs WHERE doc_type = 'invoice' AND text_blocks IS NOT NULL\n);\n",[1741],{"type":40,"tag":92,"props":1742,"children":1743},{"__ignoreMap":1282},[1744,1752,1760,1768,1776,1784,1792,1800,1809,1818,1827,1837,1846,1855,1864,1873,1882,1890,1899,1908,1917,1926,1935,1944,1953,1961,1970,1979,1988,1997],{"type":40,"tag":1288,"props":1745,"children":1746},{"class":1290,"line":1291},[1747],{"type":40,"tag":1288,"props":1748,"children":1749},{},[1750],{"type":45,"value":1751},"-- Stage 1 — parse binary docs (any type), filter parse errors\n",{"type":40,"tag":1288,"props":1753,"children":1754},{"class":1290,"line":1300},[1755],{"type":40,"tag":1288,"props":1756,"children":1757},{},[1758],{"type":45,"value":1759},"CREATE OR REFRESH STREAMING TABLE raw_parsed AS\n",{"type":40,"tag":1288,"props":1761,"children":1762},{"class":1290,"line":1309},[1763],{"type":40,"tag":1288,"props":1764,"children":1765},{},[1766],{"type":45,"value":1767},"SELECT path,\n",{"type":40,"tag":1288,"props":1769,"children":1770},{"class":1290,"line":1318},[1771],{"type":40,"tag":1288,"props":1772,"children":1773},{},[1774],{"type":45,"value":1775},"  concat_ws('\\n', transform(parsed:document:elements, e -> e:content::STRING)) AS text_blocks,\n",{"type":40,"tag":1288,"props":1777,"children":1778},{"class":1290,"line":1327},[1779],{"type":40,"tag":1288,"props":1780,"children":1781},{},[1782],{"type":45,"value":1783},"  parsed:error_status AS parse_error\n",{"type":40,"tag":1288,"props":1785,"children":1786},{"class":1290,"line":1336},[1787],{"type":40,"tag":1288,"props":1788,"children":1789},{},[1790],{"type":45,"value":1791},"FROM (\n",{"type":40,"tag":1288,"props":1793,"children":1794},{"class":1290,"line":1345},[1795],{"type":40,"tag":1288,"props":1796,"children":1797},{},[1798],{"type":45,"value":1799},"  SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed\n",{"type":40,"tag":1288,"props":1801,"children":1803},{"class":1290,"line":1802},8,[1804],{"type":40,"tag":1288,"props":1805,"children":1806},{},[1807],{"type":45,"value":1808},"  FROM STREAM read_files('\u002FVolumes\u002Fmy_catalog\u002Fdoc_processing\u002Flanding\u002F', format => 'binaryFile')\n",{"type":40,"tag":1288,"props":1810,"children":1812},{"class":1290,"line":1811},9,[1813],{"type":40,"tag":1288,"props":1814,"children":1815},{},[1816],{"type":45,"value":1817},")\n",{"type":40,"tag":1288,"props":1819,"children":1821},{"class":1290,"line":1820},10,[1822],{"type":40,"tag":1288,"props":1823,"children":1824},{},[1825],{"type":45,"value":1826},"WHERE parsed:error_status IS NULL;\n",{"type":40,"tag":1288,"props":1828,"children":1830},{"class":1290,"line":1829},11,[1831],{"type":40,"tag":1288,"props":1832,"children":1834},{"emptyLinePlaceholder":1833},true,[1835],{"type":45,"value":1836},"\n",{"type":40,"tag":1288,"props":1838,"children":1840},{"class":1290,"line":1839},12,[1841],{"type":40,"tag":1288,"props":1842,"children":1843},{},[1844],{"type":45,"value":1845},"-- Stage 2 — classify document type (cheap, no endpoint selection)\n",{"type":40,"tag":1288,"props":1847,"children":1849},{"class":1290,"line":1848},13,[1850],{"type":40,"tag":1288,"props":1851,"children":1852},{},[1853],{"type":45,"value":1854},"CREATE OR REFRESH STREAMING TABLE classified_docs AS\n",{"type":40,"tag":1288,"props":1856,"children":1858},{"class":1290,"line":1857},14,[1859],{"type":40,"tag":1288,"props":1860,"children":1861},{},[1862],{"type":45,"value":1863},"SELECT *,\n",{"type":40,"tag":1288,"props":1865,"children":1867},{"class":1290,"line":1866},15,[1868],{"type":40,"tag":1288,"props":1869,"children":1870},{},[1871],{"type":45,"value":1872},"  ai_classify(text_blocks, '[\"invoice\",\"purchase_order\",\"receipt\",\"contract\",\"other\"]', map('version','2.0')):response[0]::STRING AS doc_type\n",{"type":40,"tag":1288,"props":1874,"children":1876},{"class":1290,"line":1875},16,[1877],{"type":40,"tag":1288,"props":1878,"children":1879},{},[1880],{"type":45,"value":1881},"FROM STREAM raw_parsed;\n",{"type":40,"tag":1288,"props":1883,"children":1885},{"class":1290,"line":1884},17,[1886],{"type":40,"tag":1288,"props":1887,"children":1888},{"emptyLinePlaceholder":1833},[1889],{"type":45,"value":1836},{"type":40,"tag":1288,"props":1891,"children":1893},{"class":1290,"line":1892},18,[1894],{"type":40,"tag":1288,"props":1895,"children":1896},{},[1897],{"type":45,"value":1898},"-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`\n",{"type":40,"tag":1288,"props":1900,"children":1902},{"class":1290,"line":1901},19,[1903],{"type":40,"tag":1288,"props":1904,"children":1905},{},[1906],{"type":45,"value":1907},"CREATE OR REFRESH STREAMING TABLE extracted AS\n",{"type":40,"tag":1288,"props":1909,"children":1911},{"class":1290,"line":1910},20,[1912],{"type":40,"tag":1288,"props":1913,"children":1914},{},[1915],{"type":45,"value":1916},"SELECT path, doc_type,\n",{"type":40,"tag":1288,"props":1918,"children":1920},{"class":1290,"line":1919},21,[1921],{"type":40,"tag":1288,"props":1922,"children":1923},{},[1924],{"type":45,"value":1925},"  result:response:invoice_number::STRING AS invoice_number,\n",{"type":40,"tag":1288,"props":1927,"children":1929},{"class":1290,"line":1928},22,[1930],{"type":40,"tag":1288,"props":1931,"children":1932},{},[1933],{"type":45,"value":1934},"  result:response:vendor_name::STRING    AS vendor_name,\n",{"type":40,"tag":1288,"props":1936,"children":1938},{"class":1290,"line":1937},23,[1939],{"type":40,"tag":1288,"props":1940,"children":1941},{},[1942],{"type":45,"value":1943},"  result:response:total_amount::DOUBLE   AS total_amount,\n",{"type":40,"tag":1288,"props":1945,"children":1947},{"class":1290,"line":1946},24,[1948],{"type":40,"tag":1288,"props":1949,"children":1950},{},[1951],{"type":45,"value":1952},"  result:error_message::STRING           AS extract_error\n",{"type":40,"tag":1288,"props":1954,"children":1956},{"class":1290,"line":1955},25,[1957],{"type":40,"tag":1288,"props":1958,"children":1959},{},[1960],{"type":45,"value":1791},{"type":40,"tag":1288,"props":1962,"children":1964},{"class":1290,"line":1963},26,[1965],{"type":40,"tag":1288,"props":1966,"children":1967},{},[1968],{"type":45,"value":1969},"  SELECT *, ai_extract(text_blocks,\n",{"type":40,"tag":1288,"props":1971,"children":1973},{"class":1290,"line":1972},27,[1974],{"type":40,"tag":1288,"props":1975,"children":1976},{},[1977],{"type":45,"value":1978},"    '{\"invoice_number\":{\"type\":\"string\"},\"vendor_name\":{\"type\":\"string\"},\"total_amount\":{\"type\":\"number\"}}',\n",{"type":40,"tag":1288,"props":1980,"children":1982},{"class":1290,"line":1981},28,[1983],{"type":40,"tag":1288,"props":1984,"children":1985},{},[1986],{"type":45,"value":1987},"    map('version','2.0')) AS result\n",{"type":40,"tag":1288,"props":1989,"children":1991},{"class":1290,"line":1990},29,[1992],{"type":40,"tag":1288,"props":1993,"children":1994},{},[1995],{"type":45,"value":1996},"  FROM STREAM classified_docs WHERE doc_type = 'invoice' AND text_blocks IS NOT NULL\n",{"type":40,"tag":1288,"props":1998,"children":2000},{"class":1290,"line":1999},30,[2001],{"type":40,"tag":1288,"props":2002,"children":2003},{},[2004],{"type":45,"value":2005},");\n",{"type":40,"tag":52,"props":2007,"children":2008},{},[2009,2011,2016,2018,2024,2026,2031,2033,2038,2040,2046,2048,2052],{"type":45,"value":2010},"In a batch job, route the per-row error to a sidecar table instead of letting it crash the run: keep ",{"type":40,"tag":92,"props":2012,"children":2014},{"className":2013},[],[2015],{"type":45,"value":138},{"type":45,"value":2017},"'s ",{"type":40,"tag":92,"props":2019,"children":2021},{"className":2020},[],[2022],{"type":45,"value":2023},"result:error_message",{"type":45,"value":2025}," (VARIANT, colon-accessed, as above), and for ",{"type":40,"tag":92,"props":2027,"children":2029},{"className":2028},[],[2030],{"type":45,"value":121},{"type":45,"value":2032}," pass ",{"type":40,"tag":92,"props":2034,"children":2036},{"className":2035},[],[2037],{"type":45,"value":1068},{"type":45,"value":2039}," and check ",{"type":40,"tag":92,"props":2041,"children":2043},{"className":2042},[],[2044],{"type":45,"value":2045},"ai_response.errorMessage",{"type":45,"value":2047}," (a STRUCT field, dot-accessed). See ",{"type":40,"tag":64,"props":2049,"children":2050},{"href":989},[2051],{"type":45,"value":1576},{"type":45,"value":123},{"type":40,"tag":2054,"props":2055,"children":2057},"h3",{"id":2056},"custom-rag-pipeline-parse-prep-index",[2058],{"type":45,"value":2059},"Custom RAG Pipeline — Parse → Prep → Index",{"type":40,"tag":52,"props":2061,"children":2062},{},[2063,2065,2070,2072,2077,2079,2084,2086,2092,2093,2099,2101,2107,2109,2128,2130,2136,2137,2142,2144,2150],{"type":45,"value":2064},"For retrieval rather than field extraction: ",{"type":40,"tag":92,"props":2066,"children":2068},{"className":2067},[],[2069],{"type":45,"value":860},{"type":45,"value":2071}," → ",{"type":40,"tag":92,"props":2073,"children":2075},{"className":2074},[],[2076],{"type":45,"value":926},{"type":45,"value":2078}," (semantic chunking + context enrichment, DBR 18.2+) → Vector Search Delta Sync index. ",{"type":40,"tag":92,"props":2080,"children":2082},{"className":2081},[],[2083],{"type":45,"value":926},{"type":45,"value":2085}," returns ",{"type":40,"tag":92,"props":2087,"children":2089},{"className":2088},[],[2090],{"type":45,"value":2091},"chunk_id",{"type":45,"value":381},{"type":40,"tag":92,"props":2094,"children":2096},{"className":2095},[],[2097],{"type":45,"value":2098},"chunk_to_retrieve",{"type":45,"value":2100},", and ",{"type":40,"tag":92,"props":2102,"children":2104},{"className":2103},[],[2105],{"type":45,"value":2106},"chunk_to_embed",{"type":45,"value":2108}," (enriched with title\u002Fheaders\u002Fpage) — ",{"type":40,"tag":56,"props":2110,"children":2111},{},[2112,2114,2119,2121,2126],{"type":45,"value":2113},"embed ",{"type":40,"tag":92,"props":2115,"children":2117},{"className":2116},[],[2118],{"type":45,"value":2106},{"type":45,"value":2120},", return ",{"type":40,"tag":92,"props":2122,"children":2124},{"className":2123},[],[2125],{"type":45,"value":2098},{"type":45,"value":2127}," to the LLM.",{"type":45,"value":2129}," Shown standalone; in an SDP swap ",{"type":40,"tag":92,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":45,"value":2135},"CREATE OR REPLACE TABLE",{"type":45,"value":1707},{"type":40,"tag":92,"props":2138,"children":2140},{"className":2139},[],[2141],{"type":45,"value":1682},{"type":45,"value":2143}," + ",{"type":40,"tag":92,"props":2145,"children":2147},{"className":2146},[],[2148],{"type":45,"value":2149},"STREAM read_files(...)",{"type":45,"value":123},{"type":40,"tag":1277,"props":2152,"children":2154},{"className":1279,"code":2153,"language":1281,"meta":1282,"style":1282},"CREATE OR REPLACE TABLE parsed_chunks AS\nWITH prepped AS (\n  SELECT path AS source_path, ai_prep_search(ai_parse_document(content)) AS prep\n  FROM read_files('\u002FVolumes\u002Fmy_catalog\u002Fdoc_processing\u002Fdocs\u002F', format => 'binaryFile')\n)\nSELECT\n  variant_get(chunk, '$.chunk_id',          'STRING') AS chunk_id,\n  variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,\n  variant_get(chunk, '$.chunk_to_embed',    'STRING') AS chunk_to_embed,\n  source_path\nFROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY\u003CVARIANT>')) c AS chunk;\n",[2155],{"type":40,"tag":92,"props":2156,"children":2157},{"__ignoreMap":1282},[2158,2166,2174,2182,2190,2197,2205,2213,2221,2229,2237],{"type":40,"tag":1288,"props":2159,"children":2160},{"class":1290,"line":1291},[2161],{"type":40,"tag":1288,"props":2162,"children":2163},{},[2164],{"type":45,"value":2165},"CREATE OR REPLACE TABLE parsed_chunks AS\n",{"type":40,"tag":1288,"props":2167,"children":2168},{"class":1290,"line":1300},[2169],{"type":40,"tag":1288,"props":2170,"children":2171},{},[2172],{"type":45,"value":2173},"WITH prepped AS (\n",{"type":40,"tag":1288,"props":2175,"children":2176},{"class":1290,"line":1309},[2177],{"type":40,"tag":1288,"props":2178,"children":2179},{},[2180],{"type":45,"value":2181},"  SELECT path AS source_path, ai_prep_search(ai_parse_document(content)) AS prep\n",{"type":40,"tag":1288,"props":2183,"children":2184},{"class":1290,"line":1318},[2185],{"type":40,"tag":1288,"props":2186,"children":2187},{},[2188],{"type":45,"value":2189},"  FROM read_files('\u002FVolumes\u002Fmy_catalog\u002Fdoc_processing\u002Fdocs\u002F', format => 'binaryFile')\n",{"type":40,"tag":1288,"props":2191,"children":2192},{"class":1290,"line":1327},[2193],{"type":40,"tag":1288,"props":2194,"children":2195},{},[2196],{"type":45,"value":1817},{"type":40,"tag":1288,"props":2198,"children":2199},{"class":1290,"line":1336},[2200],{"type":40,"tag":1288,"props":2201,"children":2202},{},[2203],{"type":45,"value":2204},"SELECT\n",{"type":40,"tag":1288,"props":2206,"children":2207},{"class":1290,"line":1345},[2208],{"type":40,"tag":1288,"props":2209,"children":2210},{},[2211],{"type":45,"value":2212},"  variant_get(chunk, '$.chunk_id',          'STRING') AS chunk_id,\n",{"type":40,"tag":1288,"props":2214,"children":2215},{"class":1290,"line":1802},[2216],{"type":40,"tag":1288,"props":2217,"children":2218},{},[2219],{"type":45,"value":2220},"  variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,\n",{"type":40,"tag":1288,"props":2222,"children":2223},{"class":1290,"line":1811},[2224],{"type":40,"tag":1288,"props":2225,"children":2226},{},[2227],{"type":45,"value":2228},"  variant_get(chunk, '$.chunk_to_embed',    'STRING') AS chunk_to_embed,\n",{"type":40,"tag":1288,"props":2230,"children":2231},{"class":1290,"line":1820},[2232],{"type":40,"tag":1288,"props":2233,"children":2234},{},[2235],{"type":45,"value":2236},"  source_path\n",{"type":40,"tag":1288,"props":2238,"children":2239},{"class":1290,"line":1829},[2240],{"type":40,"tag":1288,"props":2241,"children":2242},{},[2243],{"type":45,"value":2244},"FROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY\u003CVARIANT>')) c AS chunk;\n",{"type":40,"tag":52,"props":2246,"children":2247},{},[2248,2250,2256,2258,2267,2269,2274,2276,2281,2282,2287],{"type":45,"value":2249},"Then enable CDF (",{"type":40,"tag":92,"props":2251,"children":2253},{"className":2252},[],[2254],{"type":45,"value":2255},"ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)",{"type":45,"value":2257},") and use the ",{"type":40,"tag":56,"props":2259,"children":2260},{},[2261],{"type":40,"tag":64,"props":2262,"children":2264},{"href":2263},"..\u002Fdatabricks-vector-search\u002FSKILL.md",[2265],{"type":45,"value":2266},"databricks-vector-search",{"type":45,"value":2268}," skill to build a Delta Sync index: PK ",{"type":40,"tag":92,"props":2270,"children":2272},{"className":2271},[],[2273],{"type":45,"value":2091},{"type":45,"value":2275},", embedding source ",{"type":40,"tag":92,"props":2277,"children":2279},{"className":2278},[],[2280],{"type":45,"value":2106},{"type":45,"value":2120},{"type":40,"tag":92,"props":2283,"children":2285},{"className":2284},[],[2286],{"type":45,"value":2098},{"type":45,"value":123},{"type":40,"tag":52,"props":2289,"children":2290},{},[2291],{"type":40,"tag":56,"props":2292,"children":2293},{},[2294],{"type":45,"value":2295},"Beyond batch:",{"type":40,"tag":2297,"props":2298,"children":2299},"ul",{},[2300,2322,2342],{"type":40,"tag":2301,"props":2302,"children":2303},"li",{},[2304,2309,2311,2320],{"type":40,"tag":56,"props":2305,"children":2306},{},[2307],{"type":45,"value":2308},"Ask questions over the output",{"type":45,"value":2310}," — point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the ",{"type":40,"tag":56,"props":2312,"children":2313},{},[2314],{"type":40,"tag":64,"props":2315,"children":2317},{"href":2316},"..\u002Fdatabricks-genie-agents\u002FSKILL.md",[2318],{"type":45,"value":2319},"databricks-genie-agents",{"type":45,"value":2321}," skill.",{"type":40,"tag":2301,"props":2323,"children":2324},{},[2325,2330,2332,2341],{"type":40,"tag":56,"props":2326,"children":2327},{},[2328],{"type":45,"value":2329},"Low-latency \u002F serving",{"type":45,"value":2331}," — to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the ",{"type":40,"tag":56,"props":2333,"children":2334},{},[2335],{"type":40,"tag":64,"props":2336,"children":2338},{"href":2337},"..\u002Fdatabricks-model-serving\u002FSKILL.md",[2339],{"type":45,"value":2340},"databricks-model-serving",{"type":45,"value":2321},{"type":40,"tag":2301,"props":2343,"children":2344},{},[2345,2350,2352,2357,2359,2365,2367,2374],{"type":40,"tag":56,"props":2346,"children":2347},{},[2348],{"type":45,"value":2349},"Production incremental ingestion",{"type":45,"value":2351}," — for a runnable end-to-end streaming ",{"type":40,"tag":92,"props":2353,"children":2355},{"className":2354},[],[2356],{"type":45,"value":860},{"type":45,"value":2358}," job (checkpoints, ",{"type":40,"tag":92,"props":2360,"children":2362},{"className":2361},[],[2363],{"type":45,"value":2364},"trigger(availableNow=True)",{"type":45,"value":2366},"), see ",{"type":40,"tag":64,"props":2368,"children":2371},{"href":2369,"rel":2370},"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fbundle-examples\u002Ftree\u002Fmain\u002Fcontrib\u002Fjob_with_ai_parse_document",[68],[2372],{"type":45,"value":2373},"databricks\u002Fbundle-examples · job_with_ai_parse_document",{"type":45,"value":123},{"type":40,"tag":80,"props":2376,"children":2378},{"id":2377},"reference-files",[2379],{"type":45,"value":2380},"Reference Files",{"type":40,"tag":2297,"props":2382,"children":2383},{},[2384,2423,2453],{"type":40,"tag":2301,"props":2385,"children":2386},{},[2387,2393,2395,2400,2402,2407,2409,2414,2416,2421],{"type":40,"tag":64,"props":2388,"children":2390},{"href":2389},"references\u002F1-task-functions.md",[2391],{"type":45,"value":2392},"1-task-functions.md",{"type":45,"value":2394}," — Deep reference for every task-specific function: full options\u002Fschemas (e.g. ",{"type":40,"tag":92,"props":2396,"children":2398},{"className":2397},[],[2399],{"type":45,"value":138},{"type":45,"value":2401}," v2.1 citations + confidence scores, ",{"type":40,"tag":92,"props":2403,"children":2405},{"className":2404},[],[2406],{"type":45,"value":365},{"type":45,"value":2408}," multilabel, ",{"type":40,"tag":92,"props":2410,"children":2412},{"className":2411},[],[2413],{"type":45,"value":860},{"type":45,"value":2415}," options + output schema, ",{"type":40,"tag":92,"props":2417,"children":2419},{"className":2418},[],[2420],{"type":45,"value":926},{"type":45,"value":2422}," chunk schema) and non-trivial examples. The Overview table above links to each function's section directly.",{"type":40,"tag":2301,"props":2424,"children":2425},{},[2426,2430,2431,2436,2438,2444,2446,2451],{"type":40,"tag":64,"props":2427,"children":2428},{"href":989},[2429],{"type":45,"value":1576},{"type":45,"value":298},{"type":40,"tag":92,"props":2432,"children":2434},{"className":2433},[],[2435],{"type":45,"value":121},{"type":45,"value":2437}," complete reference: all parameters, structured output with ",{"type":40,"tag":92,"props":2439,"children":2441},{"className":2440},[],[2442],{"type":45,"value":2443},"responseFormat",{"type":45,"value":2445},", multimodal ",{"type":40,"tag":92,"props":2447,"children":2449},{"className":2448},[],[2450],{"type":45,"value":1562},{"type":45,"value":2452},", UDF patterns, and error handling",{"type":40,"tag":2301,"props":2454,"children":2455},{},[2456,2460,2461,2466],{"type":40,"tag":64,"props":2457,"children":2458},{"href":1112},[2459],{"type":45,"value":1485},{"type":45,"value":298},{"type":40,"tag":92,"props":2462,"children":2464},{"className":2463},[],[2465],{"type":45,"value":1119},{"type":45,"value":2467}," parameters, single-metric, multi-group, multi-metric, and confidence interval patterns",{"type":40,"tag":80,"props":2469,"children":2471},{"id":2470},"common-issues",[2472],{"type":45,"value":2473},"Common Issues",{"type":40,"tag":213,"props":2475,"children":2476},{},[2477,2493],{"type":40,"tag":217,"props":2478,"children":2479},{},[2480],{"type":40,"tag":221,"props":2481,"children":2482},{},[2483,2488],{"type":40,"tag":225,"props":2484,"children":2485},{},[2486],{"type":45,"value":2487},"Issue",{"type":40,"tag":225,"props":2489,"children":2490},{},[2491],{"type":45,"value":2492},"Solution",{"type":40,"tag":254,"props":2494,"children":2495},{},[2496,2521,2544,2576,2602,2627,2648,2750,2768,2809],{"type":40,"tag":221,"props":2497,"children":2498},{},[2499,2509],{"type":40,"tag":261,"props":2500,"children":2501},{},[2502,2507],{"type":40,"tag":92,"props":2503,"children":2505},{"className":2504},[],[2506],{"type":45,"value":860},{"type":45,"value":2508}," not found",{"type":40,"tag":261,"props":2510,"children":2511},{},[2512,2514,2519],{"type":45,"value":2513},"Requires DBR ",{"type":40,"tag":56,"props":2515,"children":2516},{},[2517],{"type":45,"value":2518},"17.3+",{"type":45,"value":2520},". Check cluster runtime.",{"type":40,"tag":221,"props":2522,"children":2523},{},[2524,2533],{"type":40,"tag":261,"props":2525,"children":2526},{},[2527,2532],{"type":40,"tag":92,"props":2528,"children":2530},{"className":2529},[],[2531],{"type":45,"value":926},{"type":45,"value":2508},{"type":40,"tag":261,"props":2534,"children":2535},{},[2536,2537,2542],{"type":45,"value":2513},{"type":40,"tag":56,"props":2538,"children":2539},{},[2540],{"type":45,"value":2541},"18.2+",{"type":45,"value":2543}," (serverless env v3+).",{"type":40,"tag":221,"props":2545,"children":2546},{},[2547,2558],{"type":40,"tag":261,"props":2548,"children":2549},{},[2550,2556],{"type":40,"tag":92,"props":2551,"children":2553},{"className":2552},[],[2554],{"type":45,"value":2555},"explode()",{"type":45,"value":2557}," fails on a VARIANT",{"type":40,"tag":261,"props":2559,"children":2560},{},[2561,2567,2569,2575],{"type":40,"tag":92,"props":2562,"children":2564},{"className":2563},[],[2565],{"type":45,"value":2566},"explode",{"type":45,"value":2568}," needs ARRAY — cast first: ",{"type":40,"tag":92,"props":2570,"children":2572},{"className":2571},[],[2573],{"type":45,"value":2574},"explode(variant_get(prep, '$.document.contents', 'ARRAY\u003CVARIANT>'))",{"type":45,"value":123},{"type":40,"tag":221,"props":2577,"children":2578},{},[2579,2584],{"type":40,"tag":261,"props":2580,"children":2581},{},[2582],{"type":45,"value":2583},"Embedding the wrong RAG column",{"type":40,"tag":261,"props":2585,"children":2586},{},[2587,2589,2594,2596,2601],{"type":45,"value":2588},"Embed ",{"type":40,"tag":92,"props":2590,"children":2592},{"className":2591},[],[2593],{"type":45,"value":2106},{"type":45,"value":2595}," (context-enriched); return ",{"type":40,"tag":92,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":45,"value":2098},{"type":45,"value":2127},{"type":40,"tag":221,"props":2603,"children":2604},{},[2605,2615],{"type":40,"tag":261,"props":2606,"children":2607},{},[2608,2613],{"type":40,"tag":92,"props":2609,"children":2611},{"className":2610},[],[2612],{"type":45,"value":1119},{"type":45,"value":2614}," fails",{"type":40,"tag":261,"props":2616,"children":2617},{},[2618,2620,2625],{"type":45,"value":2619},"Requires ",{"type":40,"tag":56,"props":2621,"children":2622},{},[2623],{"type":45,"value":2624},"Pro or Serverless",{"type":45,"value":2626}," SQL warehouse — not available on Classic or Starter.",{"type":40,"tag":221,"props":2628,"children":2629},{},[2630,2635],{"type":40,"tag":261,"props":2631,"children":2632},{},[2633],{"type":45,"value":2634},"All functions return NULL",{"type":40,"tag":261,"props":2636,"children":2637},{},[2638,2640,2646],{"type":45,"value":2639},"Input column is NULL. Filter with ",{"type":40,"tag":92,"props":2641,"children":2643},{"className":2642},[],[2644],{"type":45,"value":2645},"WHERE col IS NOT NULL",{"type":45,"value":2647}," before calling.",{"type":40,"tag":221,"props":2649,"children":2650},{},[2651,2661],{"type":40,"tag":261,"props":2652,"children":2653},{},[2654,2659],{"type":40,"tag":92,"props":2655,"children":2657},{"className":2656},[],[2658],{"type":45,"value":802},{"type":45,"value":2660}," fails for a language",{"type":40,"tag":261,"props":2662,"children":2663},{},[2664,2666,2672,2674,2680,2682,2688,2690,2696,2698,2704,2706,2712,2714,2720,2722,2727,2729,2735,2737,2741,2743,2748],{"type":45,"value":2665},"Supported (8): English (",{"type":40,"tag":92,"props":2667,"children":2669},{"className":2668},[],[2670],{"type":45,"value":2671},"en",{"type":45,"value":2673},"), French (",{"type":40,"tag":92,"props":2675,"children":2677},{"className":2676},[],[2678],{"type":45,"value":2679},"fr",{"type":45,"value":2681},"), German (",{"type":40,"tag":92,"props":2683,"children":2685},{"className":2684},[],[2686],{"type":45,"value":2687},"de",{"type":45,"value":2689},"), Hindi (",{"type":40,"tag":92,"props":2691,"children":2693},{"className":2692},[],[2694],{"type":45,"value":2695},"hi",{"type":45,"value":2697},"), Italian (",{"type":40,"tag":92,"props":2699,"children":2701},{"className":2700},[],[2702],{"type":45,"value":2703},"it",{"type":45,"value":2705},"), Portuguese (",{"type":40,"tag":92,"props":2707,"children":2709},{"className":2708},[],[2710],{"type":45,"value":2711},"pt",{"type":45,"value":2713},"), Spanish (",{"type":40,"tag":92,"props":2715,"children":2717},{"className":2716},[],[2718],{"type":45,"value":2719},"es",{"type":45,"value":2721},"), Thai (",{"type":40,"tag":92,"props":2723,"children":2725},{"className":2724},[],[2726],{"type":45,"value":225},{"type":45,"value":2728},") — ",{"type":40,"tag":92,"props":2730,"children":2732},{"className":2731},[],[2733],{"type":45,"value":2734},"to_lang",{"type":45,"value":2736}," takes the code or the full name (see ",{"type":40,"tag":64,"props":2738,"children":2739},{"href":2389},[2740],{"type":45,"value":2389},{"type":45,"value":2742},"). Use ",{"type":40,"tag":92,"props":2744,"children":2746},{"className":2745},[],[2747],{"type":45,"value":121},{"type":45,"value":2749}," with a multilingual model for others.",{"type":40,"tag":221,"props":2751,"children":2752},{},[2753,2763],{"type":40,"tag":261,"props":2754,"children":2755},{},[2756,2761],{"type":40,"tag":92,"props":2757,"children":2759},{"className":2758},[],[2760],{"type":45,"value":365},{"type":45,"value":2762}," returns unexpected labels",{"type":40,"tag":261,"props":2764,"children":2765},{},[2766],{"type":45,"value":2767},"Use clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results.",{"type":40,"tag":221,"props":2769,"children":2770},{},[2771,2781],{"type":40,"tag":261,"props":2772,"children":2773},{},[2774,2779],{"type":40,"tag":92,"props":2775,"children":2777},{"className":2776},[],[2778],{"type":45,"value":121},{"type":45,"value":2780}," raises on some rows in a batch job",{"type":40,"tag":261,"props":2782,"children":2783},{},[2784,2786,2791,2793,2799,2801,2807],{"type":45,"value":2785},"Add ",{"type":40,"tag":92,"props":2787,"children":2789},{"className":2788},[],[2790],{"type":45,"value":1068},{"type":45,"value":2792}," — returns a STRUCT with ",{"type":40,"tag":92,"props":2794,"children":2796},{"className":2795},[],[2797],{"type":45,"value":2798},".response",{"type":45,"value":2800}," and ",{"type":40,"tag":92,"props":2802,"children":2804},{"className":2803},[],[2805],{"type":45,"value":2806},".errorMessage",{"type":45,"value":2808}," (dot-accessed) instead of raising.",{"type":40,"tag":221,"props":2810,"children":2811},{},[2812,2817],{"type":40,"tag":261,"props":2813,"children":2814},{},[2815],{"type":45,"value":2816},"Batch job runs slowly",{"type":40,"tag":261,"props":2818,"children":2819},{},[2820,2822,2826],{"type":45,"value":2821},"Use DBR ",{"type":40,"tag":56,"props":2823,"children":2824},{},[2825],{"type":45,"value":152},{"type":45,"value":2827}," cluster (not serverless or interactive) for optimized batch inference throughput.",{"type":40,"tag":2829,"props":2830,"children":2831},"style",{},[2832],{"type":45,"value":2833},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2835,"total":3016},[2836,2853,2859,2876,2893,2913,2924,2947,2958,2975,2989,3002],{"slug":2837,"name":2837,"fn":2838,"description":2839,"org":2840,"tags":2841,"stars":20,"repoUrl":21,"updatedAt":2852},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2842,2845,2846,2849],{"name":2843,"slug":2844,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2847,"slug":2848,"type":15},"Knowledge Management","knowledge-management",{"name":2850,"slug":2851,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":4,"name":4,"fn":5,"description":6,"org":2854,"tags":2855,"stars":20,"repoUrl":21,"updatedAt":22},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2856,2857,2858],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":2860,"name":2860,"fn":2861,"description":2862,"org":2863,"tags":2864,"stars":20,"repoUrl":21,"updatedAt":2875},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2865,2868,2869,2872],{"name":2866,"slug":2867,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":2870,"slug":2871,"type":15},"Docker","docker",{"name":2873,"slug":2874,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":2877,"name":2877,"fn":2878,"description":2879,"org":2880,"tags":2881,"stars":20,"repoUrl":21,"updatedAt":2892},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2882,2885,2888,2891],{"name":2883,"slug":2884,"type":15},"Analytics","analytics",{"name":2886,"slug":2887,"type":15},"Dashboards","dashboards",{"name":2889,"slug":2890,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":2894,"name":2894,"fn":2895,"description":2896,"org":2897,"tags":2898,"stars":20,"repoUrl":21,"updatedAt":2912},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2899,2900,2903,2906,2909],{"name":9,"slug":8,"type":15},{"name":2901,"slug":2902,"type":15},"Design","design",{"name":2904,"slug":2905,"type":15},"Frontend","frontend",{"name":2907,"slug":2908,"type":15},"React","react",{"name":2910,"slug":2911,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":2914,"name":2914,"fn":2915,"description":2916,"org":2917,"tags":2918,"stars":20,"repoUrl":21,"updatedAt":2923},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2919,2920,2921,2922],{"name":2883,"slug":2884,"type":15},{"name":2886,"slug":2887,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":2925,"name":2925,"fn":2926,"description":2927,"org":2928,"tags":2929,"stars":20,"repoUrl":21,"updatedAt":2946},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2930,2931,2934,2937,2940,2943],{"name":9,"slug":8,"type":15},{"name":2932,"slug":2933,"type":15},"FastAPI","fastapi",{"name":2935,"slug":2936,"type":15},"Flask","flask",{"name":2938,"slug":2939,"type":15},"Gradio","gradio",{"name":2941,"slug":2942,"type":15},"Python","python",{"name":2944,"slug":2945,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",{"slug":35,"name":35,"fn":2948,"description":2949,"org":2950,"tags":2951,"stars":20,"repoUrl":21,"updatedAt":2957},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2952,2955,2956],{"name":2953,"slug":2954,"type":15},"Authentication","authentication",{"name":2866,"slug":2867,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":2959,"name":2959,"fn":2960,"description":2961,"org":2962,"tags":2963,"stars":20,"repoUrl":21,"updatedAt":2974},"databricks-dabs","manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2964,2967,2970,2971],{"name":2965,"slug":2966,"type":15},"Automation","automation",{"name":2968,"slug":2969,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":2972,"slug":2973,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":2976,"name":2976,"fn":2977,"description":2978,"org":2979,"tags":2980,"stars":20,"repoUrl":21,"updatedAt":2988},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2981,2982,2985,2986],{"name":17,"slug":18,"type":15},{"name":2983,"slug":2984,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":2987,"slug":1281,"type":15},"SQL","2026-07-31T05:53:32.561877",{"slug":2990,"name":2990,"fn":2991,"description":2992,"org":2993,"tags":2994,"stars":20,"repoUrl":21,"updatedAt":3001},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2995,2996,2999,3000],{"name":17,"slug":18,"type":15},{"name":2997,"slug":2998,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":2987,"slug":1281,"type":15},"2026-07-12T08:04:08.678282",{"slug":3003,"name":3003,"fn":3004,"description":3005,"org":3006,"tags":3007,"stars":20,"repoUrl":21,"updatedAt":3015},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3008,3009,3012],{"name":9,"slug":8,"type":15},{"name":3010,"slug":3011,"type":15},"Documentation","documentation",{"name":3013,"slug":3014,"type":15},"Reference","reference","2026-07-15T05:41:34.697746",31,{"items":3018,"total":3016},[3019,3026,3032,3039,3046,3054,3061],{"slug":2837,"name":2837,"fn":2838,"description":2839,"org":3020,"tags":3021,"stars":20,"repoUrl":21,"updatedAt":2852},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3022,3023,3024,3025],{"name":2843,"slug":2844,"type":15},{"name":9,"slug":8,"type":15},{"name":2847,"slug":2848,"type":15},{"name":2850,"slug":2851,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":3027,"tags":3028,"stars":20,"repoUrl":21,"updatedAt":22},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3029,3030,3031],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":2860,"name":2860,"fn":2861,"description":2862,"org":3033,"tags":3034,"stars":20,"repoUrl":21,"updatedAt":2875},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3035,3036,3037,3038],{"name":2866,"slug":2867,"type":15},{"name":9,"slug":8,"type":15},{"name":2870,"slug":2871,"type":15},{"name":2873,"slug":2874,"type":15},{"slug":2877,"name":2877,"fn":2878,"description":2879,"org":3040,"tags":3041,"stars":20,"repoUrl":21,"updatedAt":2892},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3042,3043,3044,3045],{"name":2883,"slug":2884,"type":15},{"name":2886,"slug":2887,"type":15},{"name":2889,"slug":2890,"type":15},{"name":9,"slug":8,"type":15},{"slug":2894,"name":2894,"fn":2895,"description":2896,"org":3047,"tags":3048,"stars":20,"repoUrl":21,"updatedAt":2912},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3049,3050,3051,3052,3053],{"name":9,"slug":8,"type":15},{"name":2901,"slug":2902,"type":15},{"name":2904,"slug":2905,"type":15},{"name":2907,"slug":2908,"type":15},{"name":2910,"slug":2911,"type":15},{"slug":2914,"name":2914,"fn":2915,"description":2916,"org":3055,"tags":3056,"stars":20,"repoUrl":21,"updatedAt":2923},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3057,3058,3059,3060],{"name":2883,"slug":2884,"type":15},{"name":2886,"slug":2887,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"slug":2925,"name":2925,"fn":2926,"description":2927,"org":3062,"tags":3063,"stars":20,"repoUrl":21,"updatedAt":2946},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3064,3065,3066,3067,3068,3069],{"name":9,"slug":8,"type":15},{"name":2932,"slug":2933,"type":15},{"name":2935,"slug":2936,"type":15},{"name":2938,"slug":2939,"type":15},{"name":2941,"slug":2942,"type":15},{"name":2944,"slug":2945,"type":15}]