[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-pipelines":3,"mdc--6ff8l3-key":36,"related-repo-databricks-databricks-pipelines":4459,"related-org-databricks-databricks-pipelines":4579},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":34,"mdContent":35},"databricks-pipelines","develop Databricks Lakeflow Spark pipelines","Develop Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks. Use when building batch or streaming data pipelines with Python or SQL. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,22,23],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"Data Pipeline","data-pipeline",{"name":20,"slug":21,"type":15},"Python","python",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},"SQL","sql",204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-31T05:53:34.774642",null,60,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":29},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-pipelines","---\nname: databricks-pipelines\ndescription: Develop Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks. Use when building batch or streaming data pipelines with Python or SQL. Invoke BEFORE starting implementation.\ncompatibility: Requires databricks CLI (>= v1.0.0)\nmetadata:\n  version: \"0.3.0\"\nparent: databricks-core\n---\n\n# Lakeflow Spark Declarative Pipelines Development\n\n**FIRST**: Use the parent `databricks-core` skill for CLI basics, authentication, profile selection, and data discovery commands.\n\n## Decision Tree\n\nUse this tree to determine which dataset type and features to use. Multiple features can apply to the same dataset — e.g., a Streaming Table can use Auto Loader for ingestion, Append Flows for fan-in, and Expectations for data quality. Choose the dataset type first, then layer on applicable features.\n\n```\nUser request → What kind of output?\n├── Intermediate\u002Freusable logic (not persisted) → Temporary View\n│   ├── Preprocessing\u002Ffiltering before Auto CDC → Temporary View feeding CDC flow\n│   ├── Shared intermediate streaming logic reused by multiple downstream tables\n│   ├── Pipeline-private helper logic (not published to catalog)\n│   └── Published to UC for external queries → Persistent View (SQL only)\n├── Persisted dataset\n│   ├── Source is streaming\u002Fincremental\u002Fcontinuously growing → Streaming Table\n│   │   ├── File ingestion (cloud storage, Volumes) → Auto Loader\n│   │   ├── Message bus (Kafka, Kinesis, Pub\u002FSub, Pulsar, Event Hubs) → streaming source read\n│   │   ├── Existing streaming\u002FDelta table → streaming read from table\n│   │   ├── CDC \u002F upserts \u002F track changes \u002F keep latest per key \u002F SCD Type 1 or 2 → Auto CDC\n│   │   ├── Multiple sources into one table → Append Flows (NOT union)\n│   │   ├── Historical backfill + live stream → one-time Append Flow + regular flow\n│   │   └── Windowed aggregation with watermark → stateful streaming\n│   └── Source is batch\u002Fhistorical\u002Ffull scan → Materialized View\n│       ├── Aggregation\u002Fjoin across full dataset (GROUP BY, SUM, COUNT, etc.)\n│       ├── Gold layer aggregation from streaming table → MV with batch read (spark.read \u002F no STREAM)\n│       ├── JDBC\u002FFederation\u002Fexternal batch sources\n│       └── Small static file load (reference data, no streaming read)\n├── Output to external system (Python only) → Sink\n│   ├── Existing external table not managed by this pipeline → Sink with format=\"delta\"\n│   │   (prefer fully-qualified dataset names if the pipeline should own the table — see Publishing Modes)\n│   ├── Kafka \u002F Event Hubs → Sink with format=\"kafka\" + @dp.append_flow(target=\"sink_name\")\n│   ├── Custom destination not natively supported → Sink with custom format\n│   ├── Custom merge\u002Fupsert logic per batch → ForEachBatch Sink (Public Preview)\n│   └── Multiple destinations per batch → ForEachBatch Sink (Public Preview)\n└── Data quality constraints → Expectations (on any dataset type)\n```\n\n## Common Traps\n\n- **Names** → SDP = LDP = Lakeflow Declarative Pipelines = (formerly) DLT. All interchangeable when the user mentions them.\n- **\"Create a table\" without specifying type** → ask whether the source is streaming or batch. Streaming source → Streaming Table; batch source → Materialized View. Mismatched pairs error at validation.\n- **Aggregation over a streaming source** → use a Materialized View with a batch read (`spark.read.table` \u002F `SELECT FROM` without `STREAM`). STs are append-only and don't recompute aggregates when source rows change; MVs do.\n- **Intermediate logic** → default to a Temporary View. Even for shared logic reused by multiple downstream tables. Use a Private MV\u002FST (`private=True` \u002F `CREATE PRIVATE ...`) only when materializing once saves significant reprocessing. For preprocessing before Auto CDC, the temp view is required — the CDC flow reads from `STREAM(view_name)` (SQL) or `spark.readStream.table(\"view_name\")` (Python).\n- **Union of streams** → use multiple Append Flows. UNION across streaming sources is an anti-pattern.\n- **Changing dataset type** → cannot change ST→MV or MV→ST in place. Full refresh does NOT help. Drop the existing table manually or rename the new dataset.\n- **`CREATE OR REFRESH` vs `CREATE`** → both parse for SQL datasets, but `CREATE OR REFRESH` is the idiomatic convention. For PRIVATE datasets: `CREATE OR REFRESH PRIVATE STREAMING TABLE` \u002F `... MATERIALIZED VIEW`.\n- **Kafka\u002FEvent Hubs sink serialization** → the `value` column is mandatory; serialize the row with `to_json(struct(*)) AS value`. See [sink-python.md](references\u002Fsink-python.md).\n- **Multi-column Auto CDC sequencing** → SQL: `SEQUENCE BY STRUCT(col1, col2)`. Python: `sequence_by=struct(\"col1\", \"col2\")`. See the auto-cdc references.\n- **Auto CDC TRUNCATE** (SCD Type 1 only) → SQL: `APPLY AS TRUNCATE WHEN condition`. Python: `apply_as_truncates=expr(\"condition\")`. Do NOT claim truncate is unsupported.\n- **Python-only features** → Sinks, ForEachBatch Sinks, CDC from snapshots, and custom data sources are Python-only. When the user is working in SQL, clarify this and suggest switching to Python.\n- **Recommend ONE clear approach** → present a single recommended path. Don't list anti-patterns or inferior alternatives — they confuse. Only mention alternatives when they genuinely offer different trade-offs.\n\n## Common Issues\n\nError → cause\u002Ffix mappings agents hit constantly. For DAB-bundle vs CLI-iteration deploy issues, see the workflow-specific reference files.\n\n| Error \u002F symptom | Cause \u002F fix |\n|-----------------|-------------|\n| Rejection of `CREATE OR REPLACE STREAMING TABLE` \u002F `MATERIALIZED VIEW` | `CREATE OR REPLACE` is standard SQL, NOT SDP. Use `CREATE OR REFRESH STREAMING TABLE` \u002F `CREATE OR REFRESH MATERIALIZED VIEW`. |\n| CLI errors on `databricks fs ls \u002FVolumes\u002F...` | The `dbfs:` prefix is required even for UC Volume paths: `databricks fs ls dbfs:\u002FVolumes\u002F\u003Ccatalog>\u002F\u003Cschema>\u002F\u003Cvolume>\u002F\u003Cpath>`. |\n| `DELTA_CLUSTERING_COLUMNS_DATATYPE_NOT_SUPPORTED` at first write | A `CLUSTER BY` column is BOOLEAN \u002F ARRAY \u002F MAP \u002F STRUCT \u002F BINARY. SDP doesn't pre-validate — verify with `DESCRIBE` before submitting. Cluster keys must be numeric \u002F string \u002F date \u002F timestamp. Full type rules in [references\u002Fperformance.md](references\u002Fperformance.md#cluster-key-data-types). |\n| `Cannot create streaming table from batch query` | In a streaming-table query you wrote `FROM read_files(...)` (batch). Use `FROM STREAM read_files(...)` so Auto Loader kicks in. |\n| `Column not found` at ingest time | `schemaHints` don't match the actual file schema. `DESCRIBE` a sample file and align the hints. |\n| Streaming reads fail with parser error | Use `FROM STREAM read_files(...)` for file ingestion and `FROM stream(table)` (or `FROM STREAM table_name` — legacy DLT, prefer function form) for table-to-table streams. Don't mix. |\n| Pipeline stuck `INITIALIZING` for serverless | Normal — first run takes a few minutes for cold start. Don't kill it. |\n| Materialized View doesn't incrementally refresh | Automatic incremental refresh for aggregations requires **serverless** + Delta row tracking on the source (`delta.enableRowTracking = true`). Without both, falls back to full recompute. Mention the serverless requirement when the user asks about incremental refresh. |\n| SCD2 query returns nothing \u002F \"column not found\" on `START_AT` | Lakeflow uses `__START_AT` \u002F `__END_AT` (double underscore). Current rows: `WHERE __END_AT IS NULL`. |\n| `error.exceptions[0].message` missing from your events output | Your `jq` is reading `.message` (which is just \"Update X is FAILED\"). Read `error.exceptions[0].message` for the real cause — see [2-rapid-iteration-with-cli.md](references\u002F2-rapid-iteration-with-cli.md#step-4-start-an-update-and-poll-that-update). |\n\n## Publishing Modes\n\nPipelines use a **default catalog and schema** configured in the pipeline settings. All datasets are published there unless overridden.\n\n- **Fully-qualified names**: Use `catalog.schema.table` in the dataset name to write to a different catalog\u002Fschema than the pipeline default. The pipeline creates the dataset there directly — no Sink needed.\n- **USE CATALOG \u002F USE SCHEMA**: SQL commands that change the current catalog\u002Fschema for all subsequent definitions in the same file.\n- **LIVE prefix**: Deprecated. Ignored in the default publishing mode.\n- When reading or defining datasets within the pipeline, use the dataset name only — do NOT use fully-qualified names unless the pipeline already does so or the user explicitly requests a different target catalog\u002Fschema.\n\n## API Reference\n\n**Before writing pipeline code for any feature, read the linked reference file.** Each table below maps the feature to the exact API and to the detail file for that (feature, language).\n\nSome features sit on top of others — read both:\n\n- **Auto Loader** \u002F **Auto CDC** \u002F **Sinks** target a streaming table → also read [streaming-table-python.md](references\u002Fstreaming-table-python.md) \u002F [streaming-table-sql.md](references\u002Fstreaming-table-sql.md).\n- **Expectations** attach to a dataset → also read the dataset definition file (streaming-table \u002F materialized-view \u002F temporary-view).\n\n### Dataset Definition APIs\n\n| Feature                    | Description                                                      | Python                            | SQL                                         | Skill (Py)                                              | Skill (SQL)                                       |\n| -------------------------- | ---------------------------------------------------------------- | --------------------------------- | ------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------- |\n| Streaming Table            | Continuous incremental processing, exactly-once, append-only.    | `@dp.table()` returning streaming DF | `CREATE OR REFRESH STREAMING TABLE`         | [streaming-table-python](references\u002Fstreaming-table-python.md) | [streaming-table-sql](references\u002Fstreaming-table-sql.md) |\n| Materialized View          | Physically stored query result, incrementally refreshed.         | `@dp.materialized_view()`         | `CREATE OR REFRESH MATERIALIZED VIEW`       | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Temporary View             | Pipeline-private, not persisted to Unity Catalog.                | `@dp.temporary_view()`            | `CREATE TEMPORARY VIEW`                     | [temporary-view-python](references\u002Ftemporary-view-python.md) | [temporary-view-sql](references\u002Ftemporary-view-sql.md) |\n| Persistent View (UC)       | Published to UC; query runs on access (no storage).              | N\u002FA — SQL only                    | `CREATE VIEW`                               | —                                                       | [view-sql](references\u002Fview-sql.md)                |\n| Streaming Table (explicit) | Empty target, populated by separate flows (Append Flow, AUTO CDC). | `dp.create_streaming_table()`     | `CREATE OR REFRESH STREAMING TABLE` (no AS) | [streaming-table-python](references\u002Fstreaming-table-python.md) | [streaming-table-sql](references\u002Fstreaming-table-sql.md) |\n\n### Flow and Sink APIs\n\n| Feature                      | Description                                                      | Python                       | SQL                                    | Skill (Py)                                                  | Skill (SQL)                                       |\n| ---------------------------- | ---------------------------------------------------------------- | ---------------------------- | -------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------- |\n| Append Flow                  | Fan-in: multiple sources → one streaming table. Use instead of UNION. | `@dp.append_flow()`     | `CREATE FLOW ... INSERT INTO`          | [streaming-table-python](references\u002Fstreaming-table-python.md) | [streaming-table-sql](references\u002Fstreaming-table-sql.md) |\n| Backfill Flow                | One-time historical load + ongoing live stream into same table. | `@dp.append_flow(once=True)` | `CREATE FLOW ... INSERT INTO ... ONCE` | [streaming-table-python](references\u002Fstreaming-table-python.md) | [streaming-table-sql](references\u002Fstreaming-table-sql.md) |\n| Sink (Delta\u002FKafka\u002FEH\u002Fcustom) | Write streaming output to external Delta \u002F Kafka \u002F Event Hubs.   | `dp.create_sink()`           | N\u002FA — Python only                      | [sink-python](references\u002Fsink-python.md)                    | —                                                 |\n| ForEachBatch Sink            | Custom per-batch Python logic (merge\u002Fupsert, multi-destination). Public Preview. | `@dp.foreach_batch_sink()` | N\u002FA — Python only         | [foreach-batch-sink-python](references\u002Fforeach-batch-sink-python.md) | —                                       |\n| RTM update flow              | Real-Time Mode: route a flow to a sink with sub-second latency. Public Preview. | `@dp.update_flow(target=...)` | N\u002FA — Python only          | [real-time-mode](references\u002Freal-time-mode.md)              | —                                                 |\n\n### CDC APIs\n\n| Feature                      | Description                                                          | Python                                      | SQL                             | Skill (Py)                                | Skill (SQL)                          |\n| ---------------------------- | -------------------------------------------------------------------- | ------------------------------------------- | ------------------------------- | ----------------------------------------- | ------------------------------------ |\n| Auto CDC (streaming source)  | SCD Type 1 (overwrite) or Type 2 (history) from a CDC feed.          | `dp.create_auto_cdc_flow()`                 | `AUTO CDC INTO ... FROM STREAM` | [auto-cdc-python](references\u002Fauto-cdc-python.md) | [auto-cdc-sql](references\u002Fauto-cdc-sql.md) |\n| Auto CDC (periodic snapshot) | Compare consecutive full snapshots to detect changes.                | `dp.create_auto_cdc_from_snapshot_flow()`   | N\u002FA — Python only               | [auto-cdc-python](references\u002Fauto-cdc-python.md) | —                                    |\n\nFor querying SCD Type 2 history tables (`__START_AT` \u002F `__END_AT`, point-in-time, joining facts with historical dimensions), see [scd-2-querying.md](references\u002Fscd-2-querying.md).\n\n### Data Quality APIs\n\n| Feature            | Description                                | Python                       | SQL                                                    | Skill (Py)                                            | Skill (SQL)                                     |\n| ------------------ | ------------------------------------------ | ---------------------------- | ------------------------------------------------------ | ----------------------------------------------------- | ----------------------------------------------- |\n| Expect (warn)      | Log violations, keep all rows.             | `@dp.expect()`               | `CONSTRAINT ... EXPECT (...)`                          | [expectations-python](references\u002Fexpectations-python.md) | [expectations-sql](references\u002Fexpectations-sql.md) |\n| Expect or drop     | Drop violating rows.                       | `@dp.expect_or_drop()`       | `CONSTRAINT ... EXPECT (...) ON VIOLATION DROP ROW`    | [expectations-python](references\u002Fexpectations-python.md) | [expectations-sql](references\u002Fexpectations-sql.md) |\n| Expect or fail     | Fail the pipeline on first violation.      | `@dp.expect_or_fail()`       | `CONSTRAINT ... EXPECT (...) ON VIOLATION FAIL UPDATE` | [expectations-python](references\u002Fexpectations-python.md) | [expectations-sql](references\u002Fexpectations-sql.md) |\n| Expect all (warn)  | Multiple constraints at once, warn only.   | `@dp.expect_all({})`         | Multiple `CONSTRAINT` clauses                          | [expectations-python](references\u002Fexpectations-python.md) | [expectations-sql](references\u002Fexpectations-sql.md) |\n| Expect all or drop | Multiple constraints, drop on violation.   | `@dp.expect_all_or_drop({})` | Multiple constraints with `DROP ROW`                   | [expectations-python](references\u002Fexpectations-python.md) | [expectations-sql](references\u002Fexpectations-sql.md) |\n| Expect all or fail | Multiple constraints, fail on violation.   | `@dp.expect_all_or_fail({})` | Multiple constraints with `FAIL UPDATE`                | [expectations-python](references\u002Fexpectations-python.md) | [expectations-sql](references\u002Fexpectations-sql.md) |\n\n### Reading Data APIs\n\n| Feature                           | Description                                            | Python                                         | SQL                                              | Skill (Py)                                              | Skill (SQL)                                       |\n| --------------------------------- | ------------------------------------------------------ | ---------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------- |\n| Batch read (pipeline dataset)     | Read a sibling table as a static DataFrame.            | `spark.read.table(\"name\")`                     | `SELECT ... FROM name`                           | —                                                       | —                                                 |\n| Streaming read (pipeline dataset) | Read a sibling table as a streaming DataFrame.         | `spark.readStream.table(\"name\")`               | `SELECT ... FROM STREAM(name)`                   | —                                                       | —                                                 |\n| Auto Loader (cloud files)         | Incrementally ingest new files from cloud storage.     | `spark.readStream.format(\"cloudFiles\")`        | `STREAM read_files(...)`                         | [auto-loader-python](references\u002Fauto-loader-python.md)  | [auto-loader-sql](references\u002Fauto-loader-sql.md)  |\n| Kafka source                      | Streaming read from Kafka topic.                       | `spark.readStream.format(\"kafka\")`             | `STREAM read_kafka(...)`                         | [kafka](references\u002Fkafka.md)                            | [kafka](references\u002Fkafka.md)                      |\n| Kinesis source                    | Streaming read from AWS Kinesis.                       | `spark.readStream.format(\"kinesis\")`           | `STREAM read_kinesis(...)`                       | —                                                       | —                                                 |\n| Pub\u002FSub source                    | Streaming read from GCP Pub\u002FSub.                       | `spark.readStream.format(\"pubsub\")`            | `STREAM read_pubsub(...)`                        | —                                                       | —                                                 |\n| Pulsar source                     | Streaming read from Apache Pulsar.                     | `spark.readStream.format(\"pulsar\")`            | `STREAM read_pulsar(...)`                        | —                                                       | —                                                 |\n| Event Hubs source                 | Streaming read from Azure Event Hubs (Kafka protocol). | `spark.readStream.format(\"kafka\")` + EH config | `STREAM read_kafka(...)` + EH config             | [kafka](references\u002Fkafka.md)                            | [kafka](references\u002Fkafka.md)                      |\n| JDBC \u002F Lakehouse Federation       | Batch read from external systems via federation.       | `spark.read.format(\"postgresql\")` etc.         | Direct table ref via federation catalog          | —                                                       | —                                                 |\n| Custom data source                | User-defined Python data source.                       | `spark.read[Stream].format(\"custom\")`          | N\u002FA — Python only                                | —                                                       | —                                                 |\n| Static file read (batch)          | One-shot load of files (no incremental tracking).      | `spark.read.format(\"json\"\\|\"csv\"\\|...).load()` | `read_files(...)` (no STREAM)                    | —                                                       | —                                                 |\n| Skip upstream change commits      | Ignore CDC commits on the upstream table.              | `.option(\"skipChangeCommits\", \"true\")`         | `read_stream(\"name\", skipChangeCommits => true)` | [streaming-table-python](references\u002Fstreaming-table-python.md) | [streaming-table-sql](references\u002Fstreaming-table-sql.md) |\n\n### Table\u002FSchema Feature APIs\n\n| Feature                      | Description                                                   | Python                                                | SQL                                     | Skill (Py)                                              | Skill (SQL)                                       |\n| ---------------------------- | ------------------------------------------------------------- | ----------------------------------------------------- | --------------------------------------- | ------------------------------------------------------- | ------------------------------------------------- |\n| Liquid clustering            | Adaptive multi-column data layout; replaces PARTITION + Z-ORDER. Prefer Auto clustering when possible | `cluster_by=[...]`                                 | `CLUSTER BY (col1, col2)`               | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Auto liquid clustering       | Databricks picks clustering keys from query patterns.         | `cluster_by_auto=True`                                | `CLUSTER BY AUTO`                       | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Partition columns            | Legacy fixed partitioning. Prefer Liquid Clustering.          | `partition_cols=[...]`                                | `PARTITIONED BY (col1, col2)`           | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Table properties             | Delta table properties (auto-optimize, CDF, retention).       | `table_properties={...}`                              | `TBLPROPERTIES (...)`                   | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Explicit schema              | Declare column types up front (vs inferred).                  | `schema=\"col1 TYPE, ...\"`                             | `(col1 TYPE, ...) AS`                   | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Generated columns            | Columns computed from other columns at write time.            | `schema=\"..., col TYPE GENERATED ALWAYS AS (expr)\"`   | `col TYPE GENERATED ALWAYS AS (expr)`   | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Row filter (Public Preview)  | UC fine-grained access: filter rows by a function.            | `row_filter=\"ROW FILTER fn ON (col)\"`                 | `WITH ROW FILTER fn ON (col)`           | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Column mask (Public Preview) | UC fine-grained access: mask a column with a function.        | `schema=\"..., col TYPE MASK fn USING COLUMNS (col2)\"` | `col TYPE MASK fn USING COLUMNS (col2)` | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n| Private dataset              | Materialized intermediate not published to UC.                | `private=True`                                        | `CREATE PRIVATE ...`                    | [materialized-view-python](references\u002Fmaterialized-view-python.md) | [materialized-view-sql](references\u002Fmaterialized-view-sql.md) |\n\n### Legacy DLT Syntax — always migrate\n\nThe tables above show **only the modern API**. If you see any of the following in user code, it is the legacy DLT syntax — **always migrate to the modern form**, do not extend it. Read [references\u002Fdlt-migration.md](references\u002Fdlt-migration.md) before suggesting changes so the conversion is correct (especially around `apply_changes` → `create_auto_cdc_flow` semantics and `partition_cols` → `cluster_by`).\n\n| If you see…                                                                 | …it's DLT. Migrate to                                                |\n| --------------------------------------------------------------------------- | -------------------------------------------------------------------- |\n| `import dlt`                                                                | `from pyspark import pipelines as dp`                                |\n| `@dlt.table(...)`, `@dlt.append_flow(...)`, `@dlt.expect*`                  | Same decorator name on `dp.*` (e.g. `@dp.table`, `@dp.expect_or_drop`). |\n| `@dlt.view(...)` (or `@dp.view(...)` if present in older code)              | `@dp.temporary_view(...)` — the modern API has no `view` decorator, only `temporary_view`. |\n| `dlt.read(\"name\")` \u002F `dlt.read_stream(\"name\")`                              | `spark.read.table(\"name\")` \u002F `spark.readStream.table(\"name\")`        |\n| `dp.read(...)` \u002F `dp.read_stream(...)`                                      | Also legacy — use `spark.read.table(...)` \u002F `spark.readStream.table(...)`. |\n| `dlt.apply_changes(...)` \u002F `dp.apply_changes(...)`                          | `dp.create_auto_cdc_flow(...)`. `sequence_by` accepts a column name (string) or `col(...)`; `stored_as_scd_type` is integer `2` for Type 2 or string `\"1\"` for Type 1. |\n| `dlt.apply_changes_from_snapshot(...)`                                      | `dp.create_auto_cdc_from_snapshot_flow(...)`                         |\n| `dlt.create_streaming_table(...)`                                           | `dp.create_streaming_table(...)`                                     |\n| `LIVE.\u003Cname>` prefix in SQL                                                 | Bare name (`SELECT FROM name` for batch, `SELECT FROM STREAM(name)` for streaming). `LIVE.` will error in modern pipelines. |\n| `CREATE LIVE TABLE` \u002F `CREATE STREAMING LIVE TABLE` | `CREATE OR REFRESH MATERIALIZED VIEW` \u002F `CREATE OR REFRESH STREAMING TABLE`. |\n| `CREATE TEMPORARY LIVE VIEW` (a.k.a. `CREATE LIVE VIEW`) | `CREATE TEMPORARY VIEW`. **Exception**: `CREATE TEMPORARY VIEW` does NOT support `CONSTRAINT` clauses for expectations — for the rare case where you need expectations on a temp view, `CREATE LIVE VIEW` is retained. See [temporary-view-sql.md](references\u002Ftemporary-view-sql.md#using-expectations-with-temporary-views) and [expectations-sql.md](references\u002Fexpectations-sql.md). |\n| `APPLY CHANGES INTO ... FROM STREAM ...` (SQL)                              | `AUTO CDC INTO ... FROM STREAM ...`                                  |\n| `partition_cols=[...]` \u002F `PARTITIONED BY (...)` + `ZORDER`                  | `cluster_by=[...]` \u002F `CLUSTER BY (...)` (Liquid Clustering).         |\n| `input_file_name()`                                                         | `_metadata.file_path` (SQL) \u002F `F.col(\"_metadata.file_path\")` (Python). |\n| `target=...` parameter on `create_streaming_table` \u002F pipeline config        | `schema=...`                                                         |\n\n## Language Selection (Python vs SQL)\n\nDecide before scaffolding — the choice picks template files (`.py` vs `.sql`) and which reference docs apply. Both can coexist, but pick a primary. When unsure, default to SQL for simplicity.\n\n| User signal | Pick |\n|-------------|------|\n| \"Python pipeline\", UDF, pandas, ML inference, pyspark | **Python** |\n| \"SQL pipeline\", \"SQL files\" | **SQL** |\n| \"Simple pipeline\", \"create a table\", \"an aggregation\" | **SQL** (simpler, use it as default) |\n| Complex parameterized logic, custom UDFs, ML | **Python** |\n\nIf ambiguous, ask. Stick with the chosen language unless the user explicitly switches.\n\n## Choose Your Workflow\n\nThree project shapes exist — pick before scaffolding. Default to A for production-bound work and C for exploration \u002F demo scaffolding.\n\n- **A: Standalone new pipeline project (DAB)** — pipeline IS the project, no existing `databricks.yml`. Scaffold with `databricks pipelines init --output-dir . --config-file init-config.json`. → [1-project-initialization-with-dab.md](references\u002F1-project-initialization-with-dab.md)\n- **B: Pipeline in an existing bundle (DAB)** — `databricks.yml` already exists. Add a `resources\u002F\u003Cname>.pipeline.yml` pointing at `src\u002F`. → [1-project-initialization-with-dab.md#workflow-b-pipeline-in-existing-bundle](references\u002F1-project-initialization-with-dab.md#workflow-b-pipeline-in-existing-bundle)\n- **C: Rapid CLI iteration (no bundle)** — prototyping. `databricks pipelines create \u002F start-update \u002F list-pipeline-events`; formalise into a bundle later if the work goes to production. → [2-rapid-iteration-with-cli.md](references\u002F2-rapid-iteration-with-cli.md)\n\n## Pipeline Structure\n\n- Follow the medallion pattern (Bronze → Silver → Gold) unless the user says otherwise. Keep it simple by default — just a few tables.\n- One dataset per file, named after the dataset. Transformation files live in `src\u002F` or `transformations\u002F`.\n- **Gold layer: preserve key business dimensions.** When aggregating into Gold, keep the dimensions analysts will filter \u002F slice by (location, department, product line, customer segment, time period). Over-aggregating loses information that can't be recovered downstream. If a dashboard is mentioned, every filter on it needs to be a column in the Gold table. Easier to aggregate further in queries than to recover lost dimensions.\n\n\n## Running a Pipeline\n\nPicking the right run command depends on the workflow chosen above.\n\n- **Workflow A \u002F B (DAB)** — Code changes only take effect after `databricks bundle deploy`. Always deploy before any run, dry run, or selective refresh.\n  ```bash\n  databricks bundle validate --profile \u003Cprofile>\n  databricks bundle deploy -t dev --profile \u003Cprofile>\n  databricks bundle run \u003Cpipeline_name> -t dev --profile \u003Cprofile>\n  databricks pipelines get \u003Cpipeline_id> --profile \u003Cprofile>      # status\n  ```\n  → Full DAB run + iteration details: [references\u002F1-project-initialization-with-dab.md#running-a-pipeline-workflow-a--b](references\u002F1-project-initialization-with-dab.md#running-a-pipeline-workflow-a--b)\n\n- **Workflow C (CLI, no bundle)** — Upload files to the workspace, then drive the pipeline directly. Re-upload after every code change.\n  ```bash\n  databricks workspace import-dir .\u002Fmy_pipeline \u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fmy_pipeline --overwrite\n  databricks pipelines start-update \u003Cpipeline_id>\n  ```\n  → Full CLI run + polling pattern: [references\u002F2-rapid-iteration-with-cli.md](references\u002F2-rapid-iteration-with-cli.md)\n\n**Refresh modes (both workflows):**\n\n- **Selective refresh** is preferred when you only need to run one table. Dependencies must already be materialized.\n- **Full refresh** is the most expensive and dangerous option and **can lead to data loss** (it reprocesses streaming sources from scratch, destroying streaming state). Use only when really necessary. Always suggest it as a follow-up the user must explicitly approve.\n\n**Always poll the update**, not top-level pipeline state — see the polling rationale in [2-rapid-iteration-with-cli.md#step-4-start-an-update-and-poll-that-update](references\u002F2-rapid-iteration-with-cli.md#step-4-start-an-update-and-poll-that-update). Same rule applies to bundle runs.\n\n## Reference Index\n\nProject & lifecycle:\n\n- [1-project-initialization-with-dab.md](references\u002F1-project-initialization-with-dab.md) — Workflows A and B.\n- [2-rapid-iteration-with-cli.md](references\u002F2-rapid-iteration-with-cli.md) — Workflow C; start-update + polling + error-extraction.\n- [pipeline-configuration.md](references\u002Fpipeline-configuration.md) — Full create\u002Fupdate JSON reference + variant snippets + multi-schema + platform constraints.\n- [performance.md](references\u002Fperformance.md) — Liquid Clustering, state management, joins, pre-aggregation, monitoring.\n- [dlt-migration.md](references\u002Fdlt-migration.md) — DLT → SDP conversions.\n\nCross-cutting patterns:\n\n- [streaming-patterns.md](references\u002Fstreaming-patterns.md) — Dedup, windowed aggregations, late data, rescue-data quarantine, anomaly detection, lag monitoring.\n- [scd-2-querying.md](references\u002Fscd-2-querying.md) — Current-state, point-in-time, joining facts with historical dims.\n- [kafka.md](references\u002Fkafka.md) — Kafka \u002F Event Hubs ingestion.\n- [real-time-mode.md](references\u002Freal-time-mode.md) — Real-Time Mode (RTM): sub-second continuous pipelines (`@dp.update_flow`, Kafka sinks, serverless or classic).\n\nAuto Loader format-specific options: [JSON](references\u002Foptions-json.md) · [CSV](references\u002Foptions-csv.md) · [XML](references\u002Foptions-xml.md) · [Parquet](references\u002Foptions-parquet.md) · [Avro](references\u002Foptions-avro.md) · [Text](references\u002Foptions-text.md) · [ORC](references\u002Foptions-orc.md).\n\nDataset, flow, CDC, expectation, Auto Loader, and sink references are listed per (feature, language) in the [API Reference tables above](#api-reference).\n",{"data":37,"body":42},{"name":4,"description":6,"compatibility":38,"metadata":39,"parent":41},"Requires databricks CLI (>= v1.0.0)",{"version":40},"0.3.0","databricks-core",{"type":43,"children":44},"root",[45,54,74,81,86,98,104,373,379,384,774,780,792,838,844,854,859,908,915,1187,1193,1436,1442,1564,1589,1595,1912,1918,2448,2454,2893,2899,2956,3560,3566,3586,3670,3675,3681,3686,3782,3788,3825,3831,3836,4212,4220,4250,4267,4273,4278,4330,4335,4386,4441,4453],{"type":46,"tag":47,"props":48,"children":50},"element","h1",{"id":49},"lakeflow-spark-declarative-pipelines-development",[51],{"type":52,"value":53},"text","Lakeflow Spark Declarative Pipelines Development",{"type":46,"tag":55,"props":56,"children":57},"p",{},[58,64,66,72],{"type":46,"tag":59,"props":60,"children":61},"strong",{},[62],{"type":52,"value":63},"FIRST",{"type":52,"value":65},": Use the parent ",{"type":46,"tag":67,"props":68,"children":70},"code",{"className":69},[],[71],{"type":52,"value":41},{"type":52,"value":73}," skill for CLI basics, authentication, profile selection, and data discovery commands.",{"type":46,"tag":75,"props":76,"children":78},"h2",{"id":77},"decision-tree",[79],{"type":52,"value":80},"Decision Tree",{"type":46,"tag":55,"props":82,"children":83},{},[84],{"type":52,"value":85},"Use this tree to determine which dataset type and features to use. Multiple features can apply to the same dataset — e.g., a Streaming Table can use Auto Loader for ingestion, Append Flows for fan-in, and Expectations for data quality. Choose the dataset type first, then layer on applicable features.",{"type":46,"tag":87,"props":88,"children":92},"pre",{"className":89,"code":91,"language":52},[90],"language-text","User request → What kind of output?\n├── Intermediate\u002Freusable logic (not persisted) → Temporary View\n│   ├── Preprocessing\u002Ffiltering before Auto CDC → Temporary View feeding CDC flow\n│   ├── Shared intermediate streaming logic reused by multiple downstream tables\n│   ├── Pipeline-private helper logic (not published to catalog)\n│   └── Published to UC for external queries → Persistent View (SQL only)\n├── Persisted dataset\n│   ├── Source is streaming\u002Fincremental\u002Fcontinuously growing → Streaming Table\n│   │   ├── File ingestion (cloud storage, Volumes) → Auto Loader\n│   │   ├── Message bus (Kafka, Kinesis, Pub\u002FSub, Pulsar, Event Hubs) → streaming source read\n│   │   ├── Existing streaming\u002FDelta table → streaming read from table\n│   │   ├── CDC \u002F upserts \u002F track changes \u002F keep latest per key \u002F SCD Type 1 or 2 → Auto CDC\n│   │   ├── Multiple sources into one table → Append Flows (NOT union)\n│   │   ├── Historical backfill + live stream → one-time Append Flow + regular flow\n│   │   └── Windowed aggregation with watermark → stateful streaming\n│   └── Source is batch\u002Fhistorical\u002Ffull scan → Materialized View\n│       ├── Aggregation\u002Fjoin across full dataset (GROUP BY, SUM, COUNT, etc.)\n│       ├── Gold layer aggregation from streaming table → MV with batch read (spark.read \u002F no STREAM)\n│       ├── JDBC\u002FFederation\u002Fexternal batch sources\n│       └── Small static file load (reference data, no streaming read)\n├── Output to external system (Python only) → Sink\n│   ├── Existing external table not managed by this pipeline → Sink with format=\"delta\"\n│   │   (prefer fully-qualified dataset names if the pipeline should own the table — see Publishing Modes)\n│   ├── Kafka \u002F Event Hubs → Sink with format=\"kafka\" + @dp.append_flow(target=\"sink_name\")\n│   ├── Custom destination not natively supported → Sink with custom format\n│   ├── Custom merge\u002Fupsert logic per batch → ForEachBatch Sink (Public Preview)\n│   └── Multiple destinations per batch → ForEachBatch Sink (Public Preview)\n└── Data quality constraints → Expectations (on any dataset type)\n",[93],{"type":46,"tag":67,"props":94,"children":96},{"__ignoreMap":95},"",[97],{"type":52,"value":91},{"type":46,"tag":75,"props":99,"children":101},{"id":100},"common-traps",[102],{"type":52,"value":103},"Common Traps",{"type":46,"tag":105,"props":106,"children":107},"ul",{},[108,119,129,163,204,214,224,268,302,328,353,363],{"type":46,"tag":109,"props":110,"children":111},"li",{},[112,117],{"type":46,"tag":59,"props":113,"children":114},{},[115],{"type":52,"value":116},"Names",{"type":52,"value":118}," → SDP = LDP = Lakeflow Declarative Pipelines = (formerly) DLT. All interchangeable when the user mentions them.",{"type":46,"tag":109,"props":120,"children":121},{},[122,127],{"type":46,"tag":59,"props":123,"children":124},{},[125],{"type":52,"value":126},"\"Create a table\" without specifying type",{"type":52,"value":128}," → ask whether the source is streaming or batch. Streaming source → Streaming Table; batch source → Materialized View. Mismatched pairs error at validation.",{"type":46,"tag":109,"props":130,"children":131},{},[132,137,139,145,147,153,155,161],{"type":46,"tag":59,"props":133,"children":134},{},[135],{"type":52,"value":136},"Aggregation over a streaming source",{"type":52,"value":138}," → use a Materialized View with a batch read (",{"type":46,"tag":67,"props":140,"children":142},{"className":141},[],[143],{"type":52,"value":144},"spark.read.table",{"type":52,"value":146}," \u002F ",{"type":46,"tag":67,"props":148,"children":150},{"className":149},[],[151],{"type":52,"value":152},"SELECT FROM",{"type":52,"value":154}," without ",{"type":46,"tag":67,"props":156,"children":158},{"className":157},[],[159],{"type":52,"value":160},"STREAM",{"type":52,"value":162},"). STs are append-only and don't recompute aggregates when source rows change; MVs do.",{"type":46,"tag":109,"props":164,"children":165},{},[166,171,173,179,180,186,188,194,196,202],{"type":46,"tag":59,"props":167,"children":168},{},[169],{"type":52,"value":170},"Intermediate logic",{"type":52,"value":172}," → default to a Temporary View. Even for shared logic reused by multiple downstream tables. Use a Private MV\u002FST (",{"type":46,"tag":67,"props":174,"children":176},{"className":175},[],[177],{"type":52,"value":178},"private=True",{"type":52,"value":146},{"type":46,"tag":67,"props":181,"children":183},{"className":182},[],[184],{"type":52,"value":185},"CREATE PRIVATE ...",{"type":52,"value":187},") only when materializing once saves significant reprocessing. For preprocessing before Auto CDC, the temp view is required — the CDC flow reads from ",{"type":46,"tag":67,"props":189,"children":191},{"className":190},[],[192],{"type":52,"value":193},"STREAM(view_name)",{"type":52,"value":195}," (SQL) or ",{"type":46,"tag":67,"props":197,"children":199},{"className":198},[],[200],{"type":52,"value":201},"spark.readStream.table(\"view_name\")",{"type":52,"value":203}," (Python).",{"type":46,"tag":109,"props":205,"children":206},{},[207,212],{"type":46,"tag":59,"props":208,"children":209},{},[210],{"type":52,"value":211},"Union of streams",{"type":52,"value":213}," → use multiple Append Flows. UNION across streaming sources is an anti-pattern.",{"type":46,"tag":109,"props":215,"children":216},{},[217,222],{"type":46,"tag":59,"props":218,"children":219},{},[220],{"type":52,"value":221},"Changing dataset type",{"type":52,"value":223}," → cannot change ST→MV or MV→ST in place. Full refresh does NOT help. Drop the existing table manually or rename the new dataset.",{"type":46,"tag":109,"props":225,"children":226},{},[227,244,246,251,253,259,260,266],{"type":46,"tag":59,"props":228,"children":229},{},[230,236,238],{"type":46,"tag":67,"props":231,"children":233},{"className":232},[],[234],{"type":52,"value":235},"CREATE OR REFRESH",{"type":52,"value":237}," vs ",{"type":46,"tag":67,"props":239,"children":241},{"className":240},[],[242],{"type":52,"value":243},"CREATE",{"type":52,"value":245}," → both parse for SQL datasets, but ",{"type":46,"tag":67,"props":247,"children":249},{"className":248},[],[250],{"type":52,"value":235},{"type":52,"value":252}," is the idiomatic convention. For PRIVATE datasets: ",{"type":46,"tag":67,"props":254,"children":256},{"className":255},[],[257],{"type":52,"value":258},"CREATE OR REFRESH PRIVATE STREAMING TABLE",{"type":52,"value":146},{"type":46,"tag":67,"props":261,"children":263},{"className":262},[],[264],{"type":52,"value":265},"... MATERIALIZED VIEW",{"type":52,"value":267},".",{"type":46,"tag":109,"props":269,"children":270},{},[271,276,278,284,286,292,294,301],{"type":46,"tag":59,"props":272,"children":273},{},[274],{"type":52,"value":275},"Kafka\u002FEvent Hubs sink serialization",{"type":52,"value":277}," → the ",{"type":46,"tag":67,"props":279,"children":281},{"className":280},[],[282],{"type":52,"value":283},"value",{"type":52,"value":285}," column is mandatory; serialize the row with ",{"type":46,"tag":67,"props":287,"children":289},{"className":288},[],[290],{"type":52,"value":291},"to_json(struct(*)) AS value",{"type":52,"value":293},". See ",{"type":46,"tag":295,"props":296,"children":298},"a",{"href":297},"references\u002Fsink-python.md",[299],{"type":52,"value":300},"sink-python.md",{"type":52,"value":267},{"type":46,"tag":109,"props":303,"children":304},{},[305,310,312,318,320,326],{"type":46,"tag":59,"props":306,"children":307},{},[308],{"type":52,"value":309},"Multi-column Auto CDC sequencing",{"type":52,"value":311}," → SQL: ",{"type":46,"tag":67,"props":313,"children":315},{"className":314},[],[316],{"type":52,"value":317},"SEQUENCE BY STRUCT(col1, col2)",{"type":52,"value":319},". Python: ",{"type":46,"tag":67,"props":321,"children":323},{"className":322},[],[324],{"type":52,"value":325},"sequence_by=struct(\"col1\", \"col2\")",{"type":52,"value":327},". See the auto-cdc references.",{"type":46,"tag":109,"props":329,"children":330},{},[331,336,338,344,345,351],{"type":46,"tag":59,"props":332,"children":333},{},[334],{"type":52,"value":335},"Auto CDC TRUNCATE",{"type":52,"value":337}," (SCD Type 1 only) → SQL: ",{"type":46,"tag":67,"props":339,"children":341},{"className":340},[],[342],{"type":52,"value":343},"APPLY AS TRUNCATE WHEN condition",{"type":52,"value":319},{"type":46,"tag":67,"props":346,"children":348},{"className":347},[],[349],{"type":52,"value":350},"apply_as_truncates=expr(\"condition\")",{"type":52,"value":352},". Do NOT claim truncate is unsupported.",{"type":46,"tag":109,"props":354,"children":355},{},[356,361],{"type":46,"tag":59,"props":357,"children":358},{},[359],{"type":52,"value":360},"Python-only features",{"type":52,"value":362}," → Sinks, ForEachBatch Sinks, CDC from snapshots, and custom data sources are Python-only. When the user is working in SQL, clarify this and suggest switching to Python.",{"type":46,"tag":109,"props":364,"children":365},{},[366,371],{"type":46,"tag":59,"props":367,"children":368},{},[369],{"type":52,"value":370},"Recommend ONE clear approach",{"type":52,"value":372}," → present a single recommended path. Don't list anti-patterns or inferior alternatives — they confuse. Only mention alternatives when they genuinely offer different trade-offs.",{"type":46,"tag":75,"props":374,"children":376},{"id":375},"common-issues",[377],{"type":52,"value":378},"Common Issues",{"type":46,"tag":55,"props":380,"children":381},{},[382],{"type":52,"value":383},"Error → cause\u002Ffix mappings agents hit constantly. For DAB-bundle vs CLI-iteration deploy issues, see the workflow-specific reference files.",{"type":46,"tag":385,"props":386,"children":387},"table",{},[388,407],{"type":46,"tag":389,"props":390,"children":391},"thead",{},[392],{"type":46,"tag":393,"props":394,"children":395},"tr",{},[396,402],{"type":46,"tag":397,"props":398,"children":399},"th",{},[400],{"type":52,"value":401},"Error \u002F symptom",{"type":46,"tag":397,"props":403,"children":404},{},[405],{"type":52,"value":406},"Cause \u002F fix",{"type":46,"tag":408,"props":409,"children":410},"tbody",{},[411,458,492,534,567,599,635,656,684,725],{"type":46,"tag":393,"props":412,"children":413},{},[414,433],{"type":46,"tag":415,"props":416,"children":417},"td",{},[418,420,426,427],{"type":52,"value":419},"Rejection of ",{"type":46,"tag":67,"props":421,"children":423},{"className":422},[],[424],{"type":52,"value":425},"CREATE OR REPLACE STREAMING TABLE",{"type":52,"value":146},{"type":46,"tag":67,"props":428,"children":430},{"className":429},[],[431],{"type":52,"value":432},"MATERIALIZED VIEW",{"type":46,"tag":415,"props":434,"children":435},{},[436,442,444,450,451,457],{"type":46,"tag":67,"props":437,"children":439},{"className":438},[],[440],{"type":52,"value":441},"CREATE OR REPLACE",{"type":52,"value":443}," is standard SQL, NOT SDP. Use ",{"type":46,"tag":67,"props":445,"children":447},{"className":446},[],[448],{"type":52,"value":449},"CREATE OR REFRESH STREAMING TABLE",{"type":52,"value":146},{"type":46,"tag":67,"props":452,"children":454},{"className":453},[],[455],{"type":52,"value":456},"CREATE OR REFRESH MATERIALIZED VIEW",{"type":52,"value":267},{"type":46,"tag":393,"props":459,"children":460},{},[461,472],{"type":46,"tag":415,"props":462,"children":463},{},[464,466],{"type":52,"value":465},"CLI errors on ",{"type":46,"tag":67,"props":467,"children":469},{"className":468},[],[470],{"type":52,"value":471},"databricks fs ls \u002FVolumes\u002F...",{"type":46,"tag":415,"props":473,"children":474},{},[475,477,483,485,491],{"type":52,"value":476},"The ",{"type":46,"tag":67,"props":478,"children":480},{"className":479},[],[481],{"type":52,"value":482},"dbfs:",{"type":52,"value":484}," prefix is required even for UC Volume paths: ",{"type":46,"tag":67,"props":486,"children":488},{"className":487},[],[489],{"type":52,"value":490},"databricks fs ls dbfs:\u002FVolumes\u002F\u003Ccatalog>\u002F\u003Cschema>\u002F\u003Cvolume>\u002F\u003Cpath>",{"type":52,"value":267},{"type":46,"tag":393,"props":493,"children":494},{},[495,506],{"type":46,"tag":415,"props":496,"children":497},{},[498,504],{"type":46,"tag":67,"props":499,"children":501},{"className":500},[],[502],{"type":52,"value":503},"DELTA_CLUSTERING_COLUMNS_DATATYPE_NOT_SUPPORTED",{"type":52,"value":505}," at first write",{"type":46,"tag":415,"props":507,"children":508},{},[509,511,517,519,525,527,533],{"type":52,"value":510},"A ",{"type":46,"tag":67,"props":512,"children":514},{"className":513},[],[515],{"type":52,"value":516},"CLUSTER BY",{"type":52,"value":518}," column is BOOLEAN \u002F ARRAY \u002F MAP \u002F STRUCT \u002F BINARY. SDP doesn't pre-validate — verify with ",{"type":46,"tag":67,"props":520,"children":522},{"className":521},[],[523],{"type":52,"value":524},"DESCRIBE",{"type":52,"value":526}," before submitting. Cluster keys must be numeric \u002F string \u002F date \u002F timestamp. Full type rules in ",{"type":46,"tag":295,"props":528,"children":530},{"href":529},"references\u002Fperformance.md#cluster-key-data-types",[531],{"type":52,"value":532},"references\u002Fperformance.md",{"type":52,"value":267},{"type":46,"tag":393,"props":535,"children":536},{},[537,546],{"type":46,"tag":415,"props":538,"children":539},{},[540],{"type":46,"tag":67,"props":541,"children":543},{"className":542},[],[544],{"type":52,"value":545},"Cannot create streaming table from batch query",{"type":46,"tag":415,"props":547,"children":548},{},[549,551,557,559,565],{"type":52,"value":550},"In a streaming-table query you wrote ",{"type":46,"tag":67,"props":552,"children":554},{"className":553},[],[555],{"type":52,"value":556},"FROM read_files(...)",{"type":52,"value":558}," (batch). Use ",{"type":46,"tag":67,"props":560,"children":562},{"className":561},[],[563],{"type":52,"value":564},"FROM STREAM read_files(...)",{"type":52,"value":566}," so Auto Loader kicks in.",{"type":46,"tag":393,"props":568,"children":569},{},[570,581],{"type":46,"tag":415,"props":571,"children":572},{},[573,579],{"type":46,"tag":67,"props":574,"children":576},{"className":575},[],[577],{"type":52,"value":578},"Column not found",{"type":52,"value":580}," at ingest time",{"type":46,"tag":415,"props":582,"children":583},{},[584,590,592,597],{"type":46,"tag":67,"props":585,"children":587},{"className":586},[],[588],{"type":52,"value":589},"schemaHints",{"type":52,"value":591}," don't match the actual file schema. ",{"type":46,"tag":67,"props":593,"children":595},{"className":594},[],[596],{"type":52,"value":524},{"type":52,"value":598}," a sample file and align the hints.",{"type":46,"tag":393,"props":600,"children":601},{},[602,607],{"type":46,"tag":415,"props":603,"children":604},{},[605],{"type":52,"value":606},"Streaming reads fail with parser error",{"type":46,"tag":415,"props":608,"children":609},{},[610,612,617,619,625,627,633],{"type":52,"value":611},"Use ",{"type":46,"tag":67,"props":613,"children":615},{"className":614},[],[616],{"type":52,"value":564},{"type":52,"value":618}," for file ingestion and ",{"type":46,"tag":67,"props":620,"children":622},{"className":621},[],[623],{"type":52,"value":624},"FROM stream(table)",{"type":52,"value":626}," (or ",{"type":46,"tag":67,"props":628,"children":630},{"className":629},[],[631],{"type":52,"value":632},"FROM STREAM table_name",{"type":52,"value":634}," — legacy DLT, prefer function form) for table-to-table streams. Don't mix.",{"type":46,"tag":393,"props":636,"children":637},{},[638,651],{"type":46,"tag":415,"props":639,"children":640},{},[641,643,649],{"type":52,"value":642},"Pipeline stuck ",{"type":46,"tag":67,"props":644,"children":646},{"className":645},[],[647],{"type":52,"value":648},"INITIALIZING",{"type":52,"value":650}," for serverless",{"type":46,"tag":415,"props":652,"children":653},{},[654],{"type":52,"value":655},"Normal — first run takes a few minutes for cold start. Don't kill it.",{"type":46,"tag":393,"props":657,"children":658},{},[659,664],{"type":46,"tag":415,"props":660,"children":661},{},[662],{"type":52,"value":663},"Materialized View doesn't incrementally refresh",{"type":46,"tag":415,"props":665,"children":666},{},[667,669,674,676,682],{"type":52,"value":668},"Automatic incremental refresh for aggregations requires ",{"type":46,"tag":59,"props":670,"children":671},{},[672],{"type":52,"value":673},"serverless",{"type":52,"value":675}," + Delta row tracking on the source (",{"type":46,"tag":67,"props":677,"children":679},{"className":678},[],[680],{"type":52,"value":681},"delta.enableRowTracking = true",{"type":52,"value":683},"). Without both, falls back to full recompute. Mention the serverless requirement when the user asks about incremental refresh.",{"type":46,"tag":393,"props":685,"children":686},{},[687,698],{"type":46,"tag":415,"props":688,"children":689},{},[690,692],{"type":52,"value":691},"SCD2 query returns nothing \u002F \"column not found\" on ",{"type":46,"tag":67,"props":693,"children":695},{"className":694},[],[696],{"type":52,"value":697},"START_AT",{"type":46,"tag":415,"props":699,"children":700},{},[701,703,709,710,716,718,724],{"type":52,"value":702},"Lakeflow uses ",{"type":46,"tag":67,"props":704,"children":706},{"className":705},[],[707],{"type":52,"value":708},"__START_AT",{"type":52,"value":146},{"type":46,"tag":67,"props":711,"children":713},{"className":712},[],[714],{"type":52,"value":715},"__END_AT",{"type":52,"value":717}," (double underscore). Current rows: ",{"type":46,"tag":67,"props":719,"children":721},{"className":720},[],[722],{"type":52,"value":723},"WHERE __END_AT IS NULL",{"type":52,"value":267},{"type":46,"tag":393,"props":726,"children":727},{},[728,739],{"type":46,"tag":415,"props":729,"children":730},{},[731,737],{"type":46,"tag":67,"props":732,"children":734},{"className":733},[],[735],{"type":52,"value":736},"error.exceptions[0].message",{"type":52,"value":738}," missing from your events output",{"type":46,"tag":415,"props":740,"children":741},{},[742,744,750,752,758,760,765,767,773],{"type":52,"value":743},"Your ",{"type":46,"tag":67,"props":745,"children":747},{"className":746},[],[748],{"type":52,"value":749},"jq",{"type":52,"value":751}," is reading ",{"type":46,"tag":67,"props":753,"children":755},{"className":754},[],[756],{"type":52,"value":757},".message",{"type":52,"value":759}," (which is just \"Update X is FAILED\"). Read ",{"type":46,"tag":67,"props":761,"children":763},{"className":762},[],[764],{"type":52,"value":736},{"type":52,"value":766}," for the real cause — see ",{"type":46,"tag":295,"props":768,"children":770},{"href":769},"references\u002F2-rapid-iteration-with-cli.md#step-4-start-an-update-and-poll-that-update",[771],{"type":52,"value":772},"2-rapid-iteration-with-cli.md",{"type":52,"value":267},{"type":46,"tag":75,"props":775,"children":777},{"id":776},"publishing-modes",[778],{"type":52,"value":779},"Publishing Modes",{"type":46,"tag":55,"props":781,"children":782},{},[783,785,790],{"type":52,"value":784},"Pipelines use a ",{"type":46,"tag":59,"props":786,"children":787},{},[788],{"type":52,"value":789},"default catalog and schema",{"type":52,"value":791}," configured in the pipeline settings. All datasets are published there unless overridden.",{"type":46,"tag":105,"props":793,"children":794},{},[795,813,823,833],{"type":46,"tag":109,"props":796,"children":797},{},[798,803,805,811],{"type":46,"tag":59,"props":799,"children":800},{},[801],{"type":52,"value":802},"Fully-qualified names",{"type":52,"value":804},": Use ",{"type":46,"tag":67,"props":806,"children":808},{"className":807},[],[809],{"type":52,"value":810},"catalog.schema.table",{"type":52,"value":812}," in the dataset name to write to a different catalog\u002Fschema than the pipeline default. The pipeline creates the dataset there directly — no Sink needed.",{"type":46,"tag":109,"props":814,"children":815},{},[816,821],{"type":46,"tag":59,"props":817,"children":818},{},[819],{"type":52,"value":820},"USE CATALOG \u002F USE SCHEMA",{"type":52,"value":822},": SQL commands that change the current catalog\u002Fschema for all subsequent definitions in the same file.",{"type":46,"tag":109,"props":824,"children":825},{},[826,831],{"type":46,"tag":59,"props":827,"children":828},{},[829],{"type":52,"value":830},"LIVE prefix",{"type":52,"value":832},": Deprecated. Ignored in the default publishing mode.",{"type":46,"tag":109,"props":834,"children":835},{},[836],{"type":52,"value":837},"When reading or defining datasets within the pipeline, use the dataset name only — do NOT use fully-qualified names unless the pipeline already does so or the user explicitly requests a different target catalog\u002Fschema.",{"type":46,"tag":75,"props":839,"children":841},{"id":840},"api-reference",[842],{"type":52,"value":843},"API Reference",{"type":46,"tag":55,"props":845,"children":846},{},[847,852],{"type":46,"tag":59,"props":848,"children":849},{},[850],{"type":52,"value":851},"Before writing pipeline code for any feature, read the linked reference file.",{"type":52,"value":853}," Each table below maps the feature to the exact API and to the detail file for that (feature, language).",{"type":46,"tag":55,"props":855,"children":856},{},[857],{"type":52,"value":858},"Some features sit on top of others — read both:",{"type":46,"tag":105,"props":860,"children":861},{},[862,898],{"type":46,"tag":109,"props":863,"children":864},{},[865,870,871,876,877,882,884,890,891,897],{"type":46,"tag":59,"props":866,"children":867},{},[868],{"type":52,"value":869},"Auto Loader",{"type":52,"value":146},{"type":46,"tag":59,"props":872,"children":873},{},[874],{"type":52,"value":875},"Auto CDC",{"type":52,"value":146},{"type":46,"tag":59,"props":878,"children":879},{},[880],{"type":52,"value":881},"Sinks",{"type":52,"value":883}," target a streaming table → also read ",{"type":46,"tag":295,"props":885,"children":887},{"href":886},"references\u002Fstreaming-table-python.md",[888],{"type":52,"value":889},"streaming-table-python.md",{"type":52,"value":146},{"type":46,"tag":295,"props":892,"children":894},{"href":893},"references\u002Fstreaming-table-sql.md",[895],{"type":52,"value":896},"streaming-table-sql.md",{"type":52,"value":267},{"type":46,"tag":109,"props":899,"children":900},{},[901,906],{"type":46,"tag":59,"props":902,"children":903},{},[904],{"type":52,"value":905},"Expectations",{"type":52,"value":907}," attach to a dataset → also read the dataset definition file (streaming-table \u002F materialized-view \u002F temporary-view).",{"type":46,"tag":909,"props":910,"children":912},"h3",{"id":911},"dataset-definition-apis",[913],{"type":52,"value":914},"Dataset Definition APIs",{"type":46,"tag":385,"props":916,"children":917},{},[918,952],{"type":46,"tag":389,"props":919,"children":920},{},[921],{"type":46,"tag":393,"props":922,"children":923},{},[924,929,934,938,942,947],{"type":46,"tag":397,"props":925,"children":926},{},[927],{"type":52,"value":928},"Feature",{"type":46,"tag":397,"props":930,"children":931},{},[932],{"type":52,"value":933},"Description",{"type":46,"tag":397,"props":935,"children":936},{},[937],{"type":52,"value":20},{"type":46,"tag":397,"props":939,"children":940},{},[941],{"type":52,"value":24},{"type":46,"tag":397,"props":943,"children":944},{},[945],{"type":52,"value":946},"Skill (Py)",{"type":46,"tag":397,"props":948,"children":949},{},[950],{"type":52,"value":951},"Skill (SQL)",{"type":46,"tag":408,"props":953,"children":954},{},[955,1003,1051,1100,1141],{"type":46,"tag":393,"props":956,"children":957},{},[958,963,968,979,987,995],{"type":46,"tag":415,"props":959,"children":960},{},[961],{"type":52,"value":962},"Streaming Table",{"type":46,"tag":415,"props":964,"children":965},{},[966],{"type":52,"value":967},"Continuous incremental processing, exactly-once, append-only.",{"type":46,"tag":415,"props":969,"children":970},{},[971,977],{"type":46,"tag":67,"props":972,"children":974},{"className":973},[],[975],{"type":52,"value":976},"@dp.table()",{"type":52,"value":978}," returning streaming DF",{"type":46,"tag":415,"props":980,"children":981},{},[982],{"type":46,"tag":67,"props":983,"children":985},{"className":984},[],[986],{"type":52,"value":449},{"type":46,"tag":415,"props":988,"children":989},{},[990],{"type":46,"tag":295,"props":991,"children":992},{"href":886},[993],{"type":52,"value":994},"streaming-table-python",{"type":46,"tag":415,"props":996,"children":997},{},[998],{"type":46,"tag":295,"props":999,"children":1000},{"href":893},[1001],{"type":52,"value":1002},"streaming-table-sql",{"type":46,"tag":393,"props":1004,"children":1005},{},[1006,1011,1016,1025,1033,1042],{"type":46,"tag":415,"props":1007,"children":1008},{},[1009],{"type":52,"value":1010},"Materialized View",{"type":46,"tag":415,"props":1012,"children":1013},{},[1014],{"type":52,"value":1015},"Physically stored query result, incrementally refreshed.",{"type":46,"tag":415,"props":1017,"children":1018},{},[1019],{"type":46,"tag":67,"props":1020,"children":1022},{"className":1021},[],[1023],{"type":52,"value":1024},"@dp.materialized_view()",{"type":46,"tag":415,"props":1026,"children":1027},{},[1028],{"type":46,"tag":67,"props":1029,"children":1031},{"className":1030},[],[1032],{"type":52,"value":456},{"type":46,"tag":415,"props":1034,"children":1035},{},[1036],{"type":46,"tag":295,"props":1037,"children":1039},{"href":1038},"references\u002Fmaterialized-view-python.md",[1040],{"type":52,"value":1041},"materialized-view-python",{"type":46,"tag":415,"props":1043,"children":1044},{},[1045],{"type":46,"tag":295,"props":1046,"children":1048},{"href":1047},"references\u002Fmaterialized-view-sql.md",[1049],{"type":52,"value":1050},"materialized-view-sql",{"type":46,"tag":393,"props":1052,"children":1053},{},[1054,1059,1064,1073,1082,1091],{"type":46,"tag":415,"props":1055,"children":1056},{},[1057],{"type":52,"value":1058},"Temporary View",{"type":46,"tag":415,"props":1060,"children":1061},{},[1062],{"type":52,"value":1063},"Pipeline-private, not persisted to Unity Catalog.",{"type":46,"tag":415,"props":1065,"children":1066},{},[1067],{"type":46,"tag":67,"props":1068,"children":1070},{"className":1069},[],[1071],{"type":52,"value":1072},"@dp.temporary_view()",{"type":46,"tag":415,"props":1074,"children":1075},{},[1076],{"type":46,"tag":67,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":52,"value":1081},"CREATE TEMPORARY VIEW",{"type":46,"tag":415,"props":1083,"children":1084},{},[1085],{"type":46,"tag":295,"props":1086,"children":1088},{"href":1087},"references\u002Ftemporary-view-python.md",[1089],{"type":52,"value":1090},"temporary-view-python",{"type":46,"tag":415,"props":1092,"children":1093},{},[1094],{"type":46,"tag":295,"props":1095,"children":1097},{"href":1096},"references\u002Ftemporary-view-sql.md",[1098],{"type":52,"value":1099},"temporary-view-sql",{"type":46,"tag":393,"props":1101,"children":1102},{},[1103,1108,1113,1118,1127,1132],{"type":46,"tag":415,"props":1104,"children":1105},{},[1106],{"type":52,"value":1107},"Persistent View (UC)",{"type":46,"tag":415,"props":1109,"children":1110},{},[1111],{"type":52,"value":1112},"Published to UC; query runs on access (no storage).",{"type":46,"tag":415,"props":1114,"children":1115},{},[1116],{"type":52,"value":1117},"N\u002FA — SQL only",{"type":46,"tag":415,"props":1119,"children":1120},{},[1121],{"type":46,"tag":67,"props":1122,"children":1124},{"className":1123},[],[1125],{"type":52,"value":1126},"CREATE VIEW",{"type":46,"tag":415,"props":1128,"children":1129},{},[1130],{"type":52,"value":1131},"—",{"type":46,"tag":415,"props":1133,"children":1134},{},[1135],{"type":46,"tag":295,"props":1136,"children":1138},{"href":1137},"references\u002Fview-sql.md",[1139],{"type":52,"value":1140},"view-sql",{"type":46,"tag":393,"props":1142,"children":1143},{},[1144,1149,1154,1163,1173,1180],{"type":46,"tag":415,"props":1145,"children":1146},{},[1147],{"type":52,"value":1148},"Streaming Table (explicit)",{"type":46,"tag":415,"props":1150,"children":1151},{},[1152],{"type":52,"value":1153},"Empty target, populated by separate flows (Append Flow, AUTO CDC).",{"type":46,"tag":415,"props":1155,"children":1156},{},[1157],{"type":46,"tag":67,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":52,"value":1162},"dp.create_streaming_table()",{"type":46,"tag":415,"props":1164,"children":1165},{},[1166,1171],{"type":46,"tag":67,"props":1167,"children":1169},{"className":1168},[],[1170],{"type":52,"value":449},{"type":52,"value":1172}," (no AS)",{"type":46,"tag":415,"props":1174,"children":1175},{},[1176],{"type":46,"tag":295,"props":1177,"children":1178},{"href":886},[1179],{"type":52,"value":994},{"type":46,"tag":415,"props":1181,"children":1182},{},[1183],{"type":46,"tag":295,"props":1184,"children":1185},{"href":893},[1186],{"type":52,"value":1002},{"type":46,"tag":909,"props":1188,"children":1190},{"id":1189},"flow-and-sink-apis",[1191],{"type":52,"value":1192},"Flow and Sink APIs",{"type":46,"tag":385,"props":1194,"children":1195},{},[1196,1226],{"type":46,"tag":389,"props":1197,"children":1198},{},[1199],{"type":46,"tag":393,"props":1200,"children":1201},{},[1202,1206,1210,1214,1218,1222],{"type":46,"tag":397,"props":1203,"children":1204},{},[1205],{"type":52,"value":928},{"type":46,"tag":397,"props":1207,"children":1208},{},[1209],{"type":52,"value":933},{"type":46,"tag":397,"props":1211,"children":1212},{},[1213],{"type":52,"value":20},{"type":46,"tag":397,"props":1215,"children":1216},{},[1217],{"type":52,"value":24},{"type":46,"tag":397,"props":1219,"children":1220},{},[1221],{"type":52,"value":946},{"type":46,"tag":397,"props":1223,"children":1224},{},[1225],{"type":52,"value":951},{"type":46,"tag":408,"props":1227,"children":1228},{},[1229,1274,1319,1358,1397],{"type":46,"tag":393,"props":1230,"children":1231},{},[1232,1237,1242,1251,1260,1267],{"type":46,"tag":415,"props":1233,"children":1234},{},[1235],{"type":52,"value":1236},"Append Flow",{"type":46,"tag":415,"props":1238,"children":1239},{},[1240],{"type":52,"value":1241},"Fan-in: multiple sources → one streaming table. Use instead of UNION.",{"type":46,"tag":415,"props":1243,"children":1244},{},[1245],{"type":46,"tag":67,"props":1246,"children":1248},{"className":1247},[],[1249],{"type":52,"value":1250},"@dp.append_flow()",{"type":46,"tag":415,"props":1252,"children":1253},{},[1254],{"type":46,"tag":67,"props":1255,"children":1257},{"className":1256},[],[1258],{"type":52,"value":1259},"CREATE FLOW ... INSERT INTO",{"type":46,"tag":415,"props":1261,"children":1262},{},[1263],{"type":46,"tag":295,"props":1264,"children":1265},{"href":886},[1266],{"type":52,"value":994},{"type":46,"tag":415,"props":1268,"children":1269},{},[1270],{"type":46,"tag":295,"props":1271,"children":1272},{"href":893},[1273],{"type":52,"value":1002},{"type":46,"tag":393,"props":1275,"children":1276},{},[1277,1282,1287,1296,1305,1312],{"type":46,"tag":415,"props":1278,"children":1279},{},[1280],{"type":52,"value":1281},"Backfill Flow",{"type":46,"tag":415,"props":1283,"children":1284},{},[1285],{"type":52,"value":1286},"One-time historical load + ongoing live stream into same table.",{"type":46,"tag":415,"props":1288,"children":1289},{},[1290],{"type":46,"tag":67,"props":1291,"children":1293},{"className":1292},[],[1294],{"type":52,"value":1295},"@dp.append_flow(once=True)",{"type":46,"tag":415,"props":1297,"children":1298},{},[1299],{"type":46,"tag":67,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":52,"value":1304},"CREATE FLOW ... INSERT INTO ... ONCE",{"type":46,"tag":415,"props":1306,"children":1307},{},[1308],{"type":46,"tag":295,"props":1309,"children":1310},{"href":886},[1311],{"type":52,"value":994},{"type":46,"tag":415,"props":1313,"children":1314},{},[1315],{"type":46,"tag":295,"props":1316,"children":1317},{"href":893},[1318],{"type":52,"value":1002},{"type":46,"tag":393,"props":1320,"children":1321},{},[1322,1327,1332,1341,1346,1354],{"type":46,"tag":415,"props":1323,"children":1324},{},[1325],{"type":52,"value":1326},"Sink (Delta\u002FKafka\u002FEH\u002Fcustom)",{"type":46,"tag":415,"props":1328,"children":1329},{},[1330],{"type":52,"value":1331},"Write streaming output to external Delta \u002F Kafka \u002F Event Hubs.",{"type":46,"tag":415,"props":1333,"children":1334},{},[1335],{"type":46,"tag":67,"props":1336,"children":1338},{"className":1337},[],[1339],{"type":52,"value":1340},"dp.create_sink()",{"type":46,"tag":415,"props":1342,"children":1343},{},[1344],{"type":52,"value":1345},"N\u002FA — Python only",{"type":46,"tag":415,"props":1347,"children":1348},{},[1349],{"type":46,"tag":295,"props":1350,"children":1351},{"href":297},[1352],{"type":52,"value":1353},"sink-python",{"type":46,"tag":415,"props":1355,"children":1356},{},[1357],{"type":52,"value":1131},{"type":46,"tag":393,"props":1359,"children":1360},{},[1361,1366,1371,1380,1384,1393],{"type":46,"tag":415,"props":1362,"children":1363},{},[1364],{"type":52,"value":1365},"ForEachBatch Sink",{"type":46,"tag":415,"props":1367,"children":1368},{},[1369],{"type":52,"value":1370},"Custom per-batch Python logic (merge\u002Fupsert, multi-destination). Public Preview.",{"type":46,"tag":415,"props":1372,"children":1373},{},[1374],{"type":46,"tag":67,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":52,"value":1379},"@dp.foreach_batch_sink()",{"type":46,"tag":415,"props":1381,"children":1382},{},[1383],{"type":52,"value":1345},{"type":46,"tag":415,"props":1385,"children":1386},{},[1387],{"type":46,"tag":295,"props":1388,"children":1390},{"href":1389},"references\u002Fforeach-batch-sink-python.md",[1391],{"type":52,"value":1392},"foreach-batch-sink-python",{"type":46,"tag":415,"props":1394,"children":1395},{},[1396],{"type":52,"value":1131},{"type":46,"tag":393,"props":1398,"children":1399},{},[1400,1405,1410,1419,1423,1432],{"type":46,"tag":415,"props":1401,"children":1402},{},[1403],{"type":52,"value":1404},"RTM update flow",{"type":46,"tag":415,"props":1406,"children":1407},{},[1408],{"type":52,"value":1409},"Real-Time Mode: route a flow to a sink with sub-second latency. Public Preview.",{"type":46,"tag":415,"props":1411,"children":1412},{},[1413],{"type":46,"tag":67,"props":1414,"children":1416},{"className":1415},[],[1417],{"type":52,"value":1418},"@dp.update_flow(target=...)",{"type":46,"tag":415,"props":1420,"children":1421},{},[1422],{"type":52,"value":1345},{"type":46,"tag":415,"props":1424,"children":1425},{},[1426],{"type":46,"tag":295,"props":1427,"children":1429},{"href":1428},"references\u002Freal-time-mode.md",[1430],{"type":52,"value":1431},"real-time-mode",{"type":46,"tag":415,"props":1433,"children":1434},{},[1435],{"type":52,"value":1131},{"type":46,"tag":909,"props":1437,"children":1439},{"id":1438},"cdc-apis",[1440],{"type":52,"value":1441},"CDC APIs",{"type":46,"tag":385,"props":1443,"children":1444},{},[1445,1475],{"type":46,"tag":389,"props":1446,"children":1447},{},[1448],{"type":46,"tag":393,"props":1449,"children":1450},{},[1451,1455,1459,1463,1467,1471],{"type":46,"tag":397,"props":1452,"children":1453},{},[1454],{"type":52,"value":928},{"type":46,"tag":397,"props":1456,"children":1457},{},[1458],{"type":52,"value":933},{"type":46,"tag":397,"props":1460,"children":1461},{},[1462],{"type":52,"value":20},{"type":46,"tag":397,"props":1464,"children":1465},{},[1466],{"type":52,"value":24},{"type":46,"tag":397,"props":1468,"children":1469},{},[1470],{"type":52,"value":946},{"type":46,"tag":397,"props":1472,"children":1473},{},[1474],{"type":52,"value":951},{"type":46,"tag":408,"props":1476,"children":1477},{},[1478,1527],{"type":46,"tag":393,"props":1479,"children":1480},{},[1481,1486,1491,1500,1509,1518],{"type":46,"tag":415,"props":1482,"children":1483},{},[1484],{"type":52,"value":1485},"Auto CDC (streaming source)",{"type":46,"tag":415,"props":1487,"children":1488},{},[1489],{"type":52,"value":1490},"SCD Type 1 (overwrite) or Type 2 (history) from a CDC feed.",{"type":46,"tag":415,"props":1492,"children":1493},{},[1494],{"type":46,"tag":67,"props":1495,"children":1497},{"className":1496},[],[1498],{"type":52,"value":1499},"dp.create_auto_cdc_flow()",{"type":46,"tag":415,"props":1501,"children":1502},{},[1503],{"type":46,"tag":67,"props":1504,"children":1506},{"className":1505},[],[1507],{"type":52,"value":1508},"AUTO CDC INTO ... FROM STREAM",{"type":46,"tag":415,"props":1510,"children":1511},{},[1512],{"type":46,"tag":295,"props":1513,"children":1515},{"href":1514},"references\u002Fauto-cdc-python.md",[1516],{"type":52,"value":1517},"auto-cdc-python",{"type":46,"tag":415,"props":1519,"children":1520},{},[1521],{"type":46,"tag":295,"props":1522,"children":1524},{"href":1523},"references\u002Fauto-cdc-sql.md",[1525],{"type":52,"value":1526},"auto-cdc-sql",{"type":46,"tag":393,"props":1528,"children":1529},{},[1530,1535,1540,1549,1553,1560],{"type":46,"tag":415,"props":1531,"children":1532},{},[1533],{"type":52,"value":1534},"Auto CDC (periodic snapshot)",{"type":46,"tag":415,"props":1536,"children":1537},{},[1538],{"type":52,"value":1539},"Compare consecutive full snapshots to detect changes.",{"type":46,"tag":415,"props":1541,"children":1542},{},[1543],{"type":46,"tag":67,"props":1544,"children":1546},{"className":1545},[],[1547],{"type":52,"value":1548},"dp.create_auto_cdc_from_snapshot_flow()",{"type":46,"tag":415,"props":1550,"children":1551},{},[1552],{"type":52,"value":1345},{"type":46,"tag":415,"props":1554,"children":1555},{},[1556],{"type":46,"tag":295,"props":1557,"children":1558},{"href":1514},[1559],{"type":52,"value":1517},{"type":46,"tag":415,"props":1561,"children":1562},{},[1563],{"type":52,"value":1131},{"type":46,"tag":55,"props":1565,"children":1566},{},[1567,1569,1574,1575,1580,1582,1588],{"type":52,"value":1568},"For querying SCD Type 2 history tables (",{"type":46,"tag":67,"props":1570,"children":1572},{"className":1571},[],[1573],{"type":52,"value":708},{"type":52,"value":146},{"type":46,"tag":67,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":52,"value":715},{"type":52,"value":1581},", point-in-time, joining facts with historical dimensions), see ",{"type":46,"tag":295,"props":1583,"children":1585},{"href":1584},"references\u002Fscd-2-querying.md",[1586],{"type":52,"value":1587},"scd-2-querying.md",{"type":52,"value":267},{"type":46,"tag":909,"props":1590,"children":1592},{"id":1591},"data-quality-apis",[1593],{"type":52,"value":1594},"Data Quality APIs",{"type":46,"tag":385,"props":1596,"children":1597},{},[1598,1628],{"type":46,"tag":389,"props":1599,"children":1600},{},[1601],{"type":46,"tag":393,"props":1602,"children":1603},{},[1604,1608,1612,1616,1620,1624],{"type":46,"tag":397,"props":1605,"children":1606},{},[1607],{"type":52,"value":928},{"type":46,"tag":397,"props":1609,"children":1610},{},[1611],{"type":52,"value":933},{"type":46,"tag":397,"props":1613,"children":1614},{},[1615],{"type":52,"value":20},{"type":46,"tag":397,"props":1617,"children":1618},{},[1619],{"type":52,"value":24},{"type":46,"tag":397,"props":1621,"children":1622},{},[1623],{"type":52,"value":946},{"type":46,"tag":397,"props":1625,"children":1626},{},[1627],{"type":52,"value":951},{"type":46,"tag":408,"props":1629,"children":1630},{},[1631,1680,1725,1770,1819,1866],{"type":46,"tag":393,"props":1632,"children":1633},{},[1634,1639,1644,1653,1662,1671],{"type":46,"tag":415,"props":1635,"children":1636},{},[1637],{"type":52,"value":1638},"Expect (warn)",{"type":46,"tag":415,"props":1640,"children":1641},{},[1642],{"type":52,"value":1643},"Log violations, keep all rows.",{"type":46,"tag":415,"props":1645,"children":1646},{},[1647],{"type":46,"tag":67,"props":1648,"children":1650},{"className":1649},[],[1651],{"type":52,"value":1652},"@dp.expect()",{"type":46,"tag":415,"props":1654,"children":1655},{},[1656],{"type":46,"tag":67,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":52,"value":1661},"CONSTRAINT ... EXPECT (...)",{"type":46,"tag":415,"props":1663,"children":1664},{},[1665],{"type":46,"tag":295,"props":1666,"children":1668},{"href":1667},"references\u002Fexpectations-python.md",[1669],{"type":52,"value":1670},"expectations-python",{"type":46,"tag":415,"props":1672,"children":1673},{},[1674],{"type":46,"tag":295,"props":1675,"children":1677},{"href":1676},"references\u002Fexpectations-sql.md",[1678],{"type":52,"value":1679},"expectations-sql",{"type":46,"tag":393,"props":1681,"children":1682},{},[1683,1688,1693,1702,1711,1718],{"type":46,"tag":415,"props":1684,"children":1685},{},[1686],{"type":52,"value":1687},"Expect or drop",{"type":46,"tag":415,"props":1689,"children":1690},{},[1691],{"type":52,"value":1692},"Drop violating rows.",{"type":46,"tag":415,"props":1694,"children":1695},{},[1696],{"type":46,"tag":67,"props":1697,"children":1699},{"className":1698},[],[1700],{"type":52,"value":1701},"@dp.expect_or_drop()",{"type":46,"tag":415,"props":1703,"children":1704},{},[1705],{"type":46,"tag":67,"props":1706,"children":1708},{"className":1707},[],[1709],{"type":52,"value":1710},"CONSTRAINT ... EXPECT (...) ON VIOLATION DROP ROW",{"type":46,"tag":415,"props":1712,"children":1713},{},[1714],{"type":46,"tag":295,"props":1715,"children":1716},{"href":1667},[1717],{"type":52,"value":1670},{"type":46,"tag":415,"props":1719,"children":1720},{},[1721],{"type":46,"tag":295,"props":1722,"children":1723},{"href":1676},[1724],{"type":52,"value":1679},{"type":46,"tag":393,"props":1726,"children":1727},{},[1728,1733,1738,1747,1756,1763],{"type":46,"tag":415,"props":1729,"children":1730},{},[1731],{"type":52,"value":1732},"Expect or fail",{"type":46,"tag":415,"props":1734,"children":1735},{},[1736],{"type":52,"value":1737},"Fail the pipeline on first violation.",{"type":46,"tag":415,"props":1739,"children":1740},{},[1741],{"type":46,"tag":67,"props":1742,"children":1744},{"className":1743},[],[1745],{"type":52,"value":1746},"@dp.expect_or_fail()",{"type":46,"tag":415,"props":1748,"children":1749},{},[1750],{"type":46,"tag":67,"props":1751,"children":1753},{"className":1752},[],[1754],{"type":52,"value":1755},"CONSTRAINT ... EXPECT (...) ON VIOLATION FAIL UPDATE",{"type":46,"tag":415,"props":1757,"children":1758},{},[1759],{"type":46,"tag":295,"props":1760,"children":1761},{"href":1667},[1762],{"type":52,"value":1670},{"type":46,"tag":415,"props":1764,"children":1765},{},[1766],{"type":46,"tag":295,"props":1767,"children":1768},{"href":1676},[1769],{"type":52,"value":1679},{"type":46,"tag":393,"props":1771,"children":1772},{},[1773,1778,1783,1792,1805,1812],{"type":46,"tag":415,"props":1774,"children":1775},{},[1776],{"type":52,"value":1777},"Expect all (warn)",{"type":46,"tag":415,"props":1779,"children":1780},{},[1781],{"type":52,"value":1782},"Multiple constraints at once, warn only.",{"type":46,"tag":415,"props":1784,"children":1785},{},[1786],{"type":46,"tag":67,"props":1787,"children":1789},{"className":1788},[],[1790],{"type":52,"value":1791},"@dp.expect_all({})",{"type":46,"tag":415,"props":1793,"children":1794},{},[1795,1797,1803],{"type":52,"value":1796},"Multiple ",{"type":46,"tag":67,"props":1798,"children":1800},{"className":1799},[],[1801],{"type":52,"value":1802},"CONSTRAINT",{"type":52,"value":1804}," clauses",{"type":46,"tag":415,"props":1806,"children":1807},{},[1808],{"type":46,"tag":295,"props":1809,"children":1810},{"href":1667},[1811],{"type":52,"value":1670},{"type":46,"tag":415,"props":1813,"children":1814},{},[1815],{"type":46,"tag":295,"props":1816,"children":1817},{"href":1676},[1818],{"type":52,"value":1679},{"type":46,"tag":393,"props":1820,"children":1821},{},[1822,1827,1832,1841,1852,1859],{"type":46,"tag":415,"props":1823,"children":1824},{},[1825],{"type":52,"value":1826},"Expect all or drop",{"type":46,"tag":415,"props":1828,"children":1829},{},[1830],{"type":52,"value":1831},"Multiple constraints, drop on violation.",{"type":46,"tag":415,"props":1833,"children":1834},{},[1835],{"type":46,"tag":67,"props":1836,"children":1838},{"className":1837},[],[1839],{"type":52,"value":1840},"@dp.expect_all_or_drop({})",{"type":46,"tag":415,"props":1842,"children":1843},{},[1844,1846],{"type":52,"value":1845},"Multiple constraints with ",{"type":46,"tag":67,"props":1847,"children":1849},{"className":1848},[],[1850],{"type":52,"value":1851},"DROP ROW",{"type":46,"tag":415,"props":1853,"children":1854},{},[1855],{"type":46,"tag":295,"props":1856,"children":1857},{"href":1667},[1858],{"type":52,"value":1670},{"type":46,"tag":415,"props":1860,"children":1861},{},[1862],{"type":46,"tag":295,"props":1863,"children":1864},{"href":1676},[1865],{"type":52,"value":1679},{"type":46,"tag":393,"props":1867,"children":1868},{},[1869,1874,1879,1888,1898,1905],{"type":46,"tag":415,"props":1870,"children":1871},{},[1872],{"type":52,"value":1873},"Expect all or fail",{"type":46,"tag":415,"props":1875,"children":1876},{},[1877],{"type":52,"value":1878},"Multiple constraints, fail on violation.",{"type":46,"tag":415,"props":1880,"children":1881},{},[1882],{"type":46,"tag":67,"props":1883,"children":1885},{"className":1884},[],[1886],{"type":52,"value":1887},"@dp.expect_all_or_fail({})",{"type":46,"tag":415,"props":1889,"children":1890},{},[1891,1892],{"type":52,"value":1845},{"type":46,"tag":67,"props":1893,"children":1895},{"className":1894},[],[1896],{"type":52,"value":1897},"FAIL UPDATE",{"type":46,"tag":415,"props":1899,"children":1900},{},[1901],{"type":46,"tag":295,"props":1902,"children":1903},{"href":1667},[1904],{"type":52,"value":1670},{"type":46,"tag":415,"props":1906,"children":1907},{},[1908],{"type":46,"tag":295,"props":1909,"children":1910},{"href":1676},[1911],{"type":52,"value":1679},{"type":46,"tag":909,"props":1913,"children":1915},{"id":1914},"reading-data-apis",[1916],{"type":52,"value":1917},"Reading Data APIs",{"type":46,"tag":385,"props":1919,"children":1920},{},[1921,1951],{"type":46,"tag":389,"props":1922,"children":1923},{},[1924],{"type":46,"tag":393,"props":1925,"children":1926},{},[1927,1931,1935,1939,1943,1947],{"type":46,"tag":397,"props":1928,"children":1929},{},[1930],{"type":52,"value":928},{"type":46,"tag":397,"props":1932,"children":1933},{},[1934],{"type":52,"value":933},{"type":46,"tag":397,"props":1936,"children":1937},{},[1938],{"type":52,"value":20},{"type":46,"tag":397,"props":1940,"children":1941},{},[1942],{"type":52,"value":24},{"type":46,"tag":397,"props":1944,"children":1945},{},[1946],{"type":52,"value":946},{"type":46,"tag":397,"props":1948,"children":1949},{},[1950],{"type":52,"value":951},{"type":46,"tag":408,"props":1952,"children":1953},{},[1954,1993,2032,2081,2128,2167,2206,2245,2291,2328,2362,2403],{"type":46,"tag":393,"props":1955,"children":1956},{},[1957,1962,1967,1976,1985,1989],{"type":46,"tag":415,"props":1958,"children":1959},{},[1960],{"type":52,"value":1961},"Batch read (pipeline dataset)",{"type":46,"tag":415,"props":1963,"children":1964},{},[1965],{"type":52,"value":1966},"Read a sibling table as a static DataFrame.",{"type":46,"tag":415,"props":1968,"children":1969},{},[1970],{"type":46,"tag":67,"props":1971,"children":1973},{"className":1972},[],[1974],{"type":52,"value":1975},"spark.read.table(\"name\")",{"type":46,"tag":415,"props":1977,"children":1978},{},[1979],{"type":46,"tag":67,"props":1980,"children":1982},{"className":1981},[],[1983],{"type":52,"value":1984},"SELECT ... FROM name",{"type":46,"tag":415,"props":1986,"children":1987},{},[1988],{"type":52,"value":1131},{"type":46,"tag":415,"props":1990,"children":1991},{},[1992],{"type":52,"value":1131},{"type":46,"tag":393,"props":1994,"children":1995},{},[1996,2001,2006,2015,2024,2028],{"type":46,"tag":415,"props":1997,"children":1998},{},[1999],{"type":52,"value":2000},"Streaming read (pipeline dataset)",{"type":46,"tag":415,"props":2002,"children":2003},{},[2004],{"type":52,"value":2005},"Read a sibling table as a streaming DataFrame.",{"type":46,"tag":415,"props":2007,"children":2008},{},[2009],{"type":46,"tag":67,"props":2010,"children":2012},{"className":2011},[],[2013],{"type":52,"value":2014},"spark.readStream.table(\"name\")",{"type":46,"tag":415,"props":2016,"children":2017},{},[2018],{"type":46,"tag":67,"props":2019,"children":2021},{"className":2020},[],[2022],{"type":52,"value":2023},"SELECT ... FROM STREAM(name)",{"type":46,"tag":415,"props":2025,"children":2026},{},[2027],{"type":52,"value":1131},{"type":46,"tag":415,"props":2029,"children":2030},{},[2031],{"type":52,"value":1131},{"type":46,"tag":393,"props":2033,"children":2034},{},[2035,2040,2045,2054,2063,2072],{"type":46,"tag":415,"props":2036,"children":2037},{},[2038],{"type":52,"value":2039},"Auto Loader (cloud files)",{"type":46,"tag":415,"props":2041,"children":2042},{},[2043],{"type":52,"value":2044},"Incrementally ingest new files from cloud storage.",{"type":46,"tag":415,"props":2046,"children":2047},{},[2048],{"type":46,"tag":67,"props":2049,"children":2051},{"className":2050},[],[2052],{"type":52,"value":2053},"spark.readStream.format(\"cloudFiles\")",{"type":46,"tag":415,"props":2055,"children":2056},{},[2057],{"type":46,"tag":67,"props":2058,"children":2060},{"className":2059},[],[2061],{"type":52,"value":2062},"STREAM read_files(...)",{"type":46,"tag":415,"props":2064,"children":2065},{},[2066],{"type":46,"tag":295,"props":2067,"children":2069},{"href":2068},"references\u002Fauto-loader-python.md",[2070],{"type":52,"value":2071},"auto-loader-python",{"type":46,"tag":415,"props":2073,"children":2074},{},[2075],{"type":46,"tag":295,"props":2076,"children":2078},{"href":2077},"references\u002Fauto-loader-sql.md",[2079],{"type":52,"value":2080},"auto-loader-sql",{"type":46,"tag":393,"props":2082,"children":2083},{},[2084,2089,2094,2103,2112,2121],{"type":46,"tag":415,"props":2085,"children":2086},{},[2087],{"type":52,"value":2088},"Kafka source",{"type":46,"tag":415,"props":2090,"children":2091},{},[2092],{"type":52,"value":2093},"Streaming read from Kafka topic.",{"type":46,"tag":415,"props":2095,"children":2096},{},[2097],{"type":46,"tag":67,"props":2098,"children":2100},{"className":2099},[],[2101],{"type":52,"value":2102},"spark.readStream.format(\"kafka\")",{"type":46,"tag":415,"props":2104,"children":2105},{},[2106],{"type":46,"tag":67,"props":2107,"children":2109},{"className":2108},[],[2110],{"type":52,"value":2111},"STREAM read_kafka(...)",{"type":46,"tag":415,"props":2113,"children":2114},{},[2115],{"type":46,"tag":295,"props":2116,"children":2118},{"href":2117},"references\u002Fkafka.md",[2119],{"type":52,"value":2120},"kafka",{"type":46,"tag":415,"props":2122,"children":2123},{},[2124],{"type":46,"tag":295,"props":2125,"children":2126},{"href":2117},[2127],{"type":52,"value":2120},{"type":46,"tag":393,"props":2129,"children":2130},{},[2131,2136,2141,2150,2159,2163],{"type":46,"tag":415,"props":2132,"children":2133},{},[2134],{"type":52,"value":2135},"Kinesis source",{"type":46,"tag":415,"props":2137,"children":2138},{},[2139],{"type":52,"value":2140},"Streaming read from AWS Kinesis.",{"type":46,"tag":415,"props":2142,"children":2143},{},[2144],{"type":46,"tag":67,"props":2145,"children":2147},{"className":2146},[],[2148],{"type":52,"value":2149},"spark.readStream.format(\"kinesis\")",{"type":46,"tag":415,"props":2151,"children":2152},{},[2153],{"type":46,"tag":67,"props":2154,"children":2156},{"className":2155},[],[2157],{"type":52,"value":2158},"STREAM read_kinesis(...)",{"type":46,"tag":415,"props":2160,"children":2161},{},[2162],{"type":52,"value":1131},{"type":46,"tag":415,"props":2164,"children":2165},{},[2166],{"type":52,"value":1131},{"type":46,"tag":393,"props":2168,"children":2169},{},[2170,2175,2180,2189,2198,2202],{"type":46,"tag":415,"props":2171,"children":2172},{},[2173],{"type":52,"value":2174},"Pub\u002FSub source",{"type":46,"tag":415,"props":2176,"children":2177},{},[2178],{"type":52,"value":2179},"Streaming read from GCP Pub\u002FSub.",{"type":46,"tag":415,"props":2181,"children":2182},{},[2183],{"type":46,"tag":67,"props":2184,"children":2186},{"className":2185},[],[2187],{"type":52,"value":2188},"spark.readStream.format(\"pubsub\")",{"type":46,"tag":415,"props":2190,"children":2191},{},[2192],{"type":46,"tag":67,"props":2193,"children":2195},{"className":2194},[],[2196],{"type":52,"value":2197},"STREAM read_pubsub(...)",{"type":46,"tag":415,"props":2199,"children":2200},{},[2201],{"type":52,"value":1131},{"type":46,"tag":415,"props":2203,"children":2204},{},[2205],{"type":52,"value":1131},{"type":46,"tag":393,"props":2207,"children":2208},{},[2209,2214,2219,2228,2237,2241],{"type":46,"tag":415,"props":2210,"children":2211},{},[2212],{"type":52,"value":2213},"Pulsar source",{"type":46,"tag":415,"props":2215,"children":2216},{},[2217],{"type":52,"value":2218},"Streaming read from Apache Pulsar.",{"type":46,"tag":415,"props":2220,"children":2221},{},[2222],{"type":46,"tag":67,"props":2223,"children":2225},{"className":2224},[],[2226],{"type":52,"value":2227},"spark.readStream.format(\"pulsar\")",{"type":46,"tag":415,"props":2229,"children":2230},{},[2231],{"type":46,"tag":67,"props":2232,"children":2234},{"className":2233},[],[2235],{"type":52,"value":2236},"STREAM read_pulsar(...)",{"type":46,"tag":415,"props":2238,"children":2239},{},[2240],{"type":52,"value":1131},{"type":46,"tag":415,"props":2242,"children":2243},{},[2244],{"type":52,"value":1131},{"type":46,"tag":393,"props":2246,"children":2247},{},[2248,2253,2258,2268,2277,2284],{"type":46,"tag":415,"props":2249,"children":2250},{},[2251],{"type":52,"value":2252},"Event Hubs source",{"type":46,"tag":415,"props":2254,"children":2255},{},[2256],{"type":52,"value":2257},"Streaming read from Azure Event Hubs (Kafka protocol).",{"type":46,"tag":415,"props":2259,"children":2260},{},[2261,2266],{"type":46,"tag":67,"props":2262,"children":2264},{"className":2263},[],[2265],{"type":52,"value":2102},{"type":52,"value":2267}," + EH config",{"type":46,"tag":415,"props":2269,"children":2270},{},[2271,2276],{"type":46,"tag":67,"props":2272,"children":2274},{"className":2273},[],[2275],{"type":52,"value":2111},{"type":52,"value":2267},{"type":46,"tag":415,"props":2278,"children":2279},{},[2280],{"type":46,"tag":295,"props":2281,"children":2282},{"href":2117},[2283],{"type":52,"value":2120},{"type":46,"tag":415,"props":2285,"children":2286},{},[2287],{"type":46,"tag":295,"props":2288,"children":2289},{"href":2117},[2290],{"type":52,"value":2120},{"type":46,"tag":393,"props":2292,"children":2293},{},[2294,2299,2304,2315,2320,2324],{"type":46,"tag":415,"props":2295,"children":2296},{},[2297],{"type":52,"value":2298},"JDBC \u002F Lakehouse Federation",{"type":46,"tag":415,"props":2300,"children":2301},{},[2302],{"type":52,"value":2303},"Batch read from external systems via federation.",{"type":46,"tag":415,"props":2305,"children":2306},{},[2307,2313],{"type":46,"tag":67,"props":2308,"children":2310},{"className":2309},[],[2311],{"type":52,"value":2312},"spark.read.format(\"postgresql\")",{"type":52,"value":2314}," etc.",{"type":46,"tag":415,"props":2316,"children":2317},{},[2318],{"type":52,"value":2319},"Direct table ref via federation catalog",{"type":46,"tag":415,"props":2321,"children":2322},{},[2323],{"type":52,"value":1131},{"type":46,"tag":415,"props":2325,"children":2326},{},[2327],{"type":52,"value":1131},{"type":46,"tag":393,"props":2329,"children":2330},{},[2331,2336,2341,2350,2354,2358],{"type":46,"tag":415,"props":2332,"children":2333},{},[2334],{"type":52,"value":2335},"Custom data source",{"type":46,"tag":415,"props":2337,"children":2338},{},[2339],{"type":52,"value":2340},"User-defined Python data source.",{"type":46,"tag":415,"props":2342,"children":2343},{},[2344],{"type":46,"tag":67,"props":2345,"children":2347},{"className":2346},[],[2348],{"type":52,"value":2349},"spark.read[Stream].format(\"custom\")",{"type":46,"tag":415,"props":2351,"children":2352},{},[2353],{"type":52,"value":1345},{"type":46,"tag":415,"props":2355,"children":2356},{},[2357],{"type":52,"value":1131},{"type":46,"tag":415,"props":2359,"children":2360},{},[2361],{"type":52,"value":1131},{"type":46,"tag":393,"props":2363,"children":2364},{},[2365,2370,2375,2384,2395,2399],{"type":46,"tag":415,"props":2366,"children":2367},{},[2368],{"type":52,"value":2369},"Static file read (batch)",{"type":46,"tag":415,"props":2371,"children":2372},{},[2373],{"type":52,"value":2374},"One-shot load of files (no incremental tracking).",{"type":46,"tag":415,"props":2376,"children":2377},{},[2378],{"type":46,"tag":67,"props":2379,"children":2381},{"className":2380},[],[2382],{"type":52,"value":2383},"spark.read.format(\"json\"|\"csv\"|...).load()",{"type":46,"tag":415,"props":2385,"children":2386},{},[2387,2393],{"type":46,"tag":67,"props":2388,"children":2390},{"className":2389},[],[2391],{"type":52,"value":2392},"read_files(...)",{"type":52,"value":2394}," (no STREAM)",{"type":46,"tag":415,"props":2396,"children":2397},{},[2398],{"type":52,"value":1131},{"type":46,"tag":415,"props":2400,"children":2401},{},[2402],{"type":52,"value":1131},{"type":46,"tag":393,"props":2404,"children":2405},{},[2406,2411,2416,2425,2434,2441],{"type":46,"tag":415,"props":2407,"children":2408},{},[2409],{"type":52,"value":2410},"Skip upstream change commits",{"type":46,"tag":415,"props":2412,"children":2413},{},[2414],{"type":52,"value":2415},"Ignore CDC commits on the upstream table.",{"type":46,"tag":415,"props":2417,"children":2418},{},[2419],{"type":46,"tag":67,"props":2420,"children":2422},{"className":2421},[],[2423],{"type":52,"value":2424},".option(\"skipChangeCommits\", \"true\")",{"type":46,"tag":415,"props":2426,"children":2427},{},[2428],{"type":46,"tag":67,"props":2429,"children":2431},{"className":2430},[],[2432],{"type":52,"value":2433},"read_stream(\"name\", skipChangeCommits => true)",{"type":46,"tag":415,"props":2435,"children":2436},{},[2437],{"type":46,"tag":295,"props":2438,"children":2439},{"href":886},[2440],{"type":52,"value":994},{"type":46,"tag":415,"props":2442,"children":2443},{},[2444],{"type":46,"tag":295,"props":2445,"children":2446},{"href":893},[2447],{"type":52,"value":1002},{"type":46,"tag":909,"props":2449,"children":2451},{"id":2450},"tableschema-feature-apis",[2452],{"type":52,"value":2453},"Table\u002FSchema Feature APIs",{"type":46,"tag":385,"props":2455,"children":2456},{},[2457,2487],{"type":46,"tag":389,"props":2458,"children":2459},{},[2460],{"type":46,"tag":393,"props":2461,"children":2462},{},[2463,2467,2471,2475,2479,2483],{"type":46,"tag":397,"props":2464,"children":2465},{},[2466],{"type":52,"value":928},{"type":46,"tag":397,"props":2468,"children":2469},{},[2470],{"type":52,"value":933},{"type":46,"tag":397,"props":2472,"children":2473},{},[2474],{"type":52,"value":20},{"type":46,"tag":397,"props":2476,"children":2477},{},[2478],{"type":52,"value":24},{"type":46,"tag":397,"props":2480,"children":2481},{},[2482],{"type":52,"value":946},{"type":46,"tag":397,"props":2484,"children":2485},{},[2486],{"type":52,"value":951},{"type":46,"tag":408,"props":2488,"children":2489},{},[2490,2535,2580,2625,2670,2715,2760,2805,2850],{"type":46,"tag":393,"props":2491,"children":2492},{},[2493,2498,2503,2512,2521,2528],{"type":46,"tag":415,"props":2494,"children":2495},{},[2496],{"type":52,"value":2497},"Liquid clustering",{"type":46,"tag":415,"props":2499,"children":2500},{},[2501],{"type":52,"value":2502},"Adaptive multi-column data layout; replaces PARTITION + Z-ORDER. Prefer Auto clustering when possible",{"type":46,"tag":415,"props":2504,"children":2505},{},[2506],{"type":46,"tag":67,"props":2507,"children":2509},{"className":2508},[],[2510],{"type":52,"value":2511},"cluster_by=[...]",{"type":46,"tag":415,"props":2513,"children":2514},{},[2515],{"type":46,"tag":67,"props":2516,"children":2518},{"className":2517},[],[2519],{"type":52,"value":2520},"CLUSTER BY (col1, col2)",{"type":46,"tag":415,"props":2522,"children":2523},{},[2524],{"type":46,"tag":295,"props":2525,"children":2526},{"href":1038},[2527],{"type":52,"value":1041},{"type":46,"tag":415,"props":2529,"children":2530},{},[2531],{"type":46,"tag":295,"props":2532,"children":2533},{"href":1047},[2534],{"type":52,"value":1050},{"type":46,"tag":393,"props":2536,"children":2537},{},[2538,2543,2548,2557,2566,2573],{"type":46,"tag":415,"props":2539,"children":2540},{},[2541],{"type":52,"value":2542},"Auto liquid clustering",{"type":46,"tag":415,"props":2544,"children":2545},{},[2546],{"type":52,"value":2547},"Databricks picks clustering keys from query patterns.",{"type":46,"tag":415,"props":2549,"children":2550},{},[2551],{"type":46,"tag":67,"props":2552,"children":2554},{"className":2553},[],[2555],{"type":52,"value":2556},"cluster_by_auto=True",{"type":46,"tag":415,"props":2558,"children":2559},{},[2560],{"type":46,"tag":67,"props":2561,"children":2563},{"className":2562},[],[2564],{"type":52,"value":2565},"CLUSTER BY AUTO",{"type":46,"tag":415,"props":2567,"children":2568},{},[2569],{"type":46,"tag":295,"props":2570,"children":2571},{"href":1038},[2572],{"type":52,"value":1041},{"type":46,"tag":415,"props":2574,"children":2575},{},[2576],{"type":46,"tag":295,"props":2577,"children":2578},{"href":1047},[2579],{"type":52,"value":1050},{"type":46,"tag":393,"props":2581,"children":2582},{},[2583,2588,2593,2602,2611,2618],{"type":46,"tag":415,"props":2584,"children":2585},{},[2586],{"type":52,"value":2587},"Partition columns",{"type":46,"tag":415,"props":2589,"children":2590},{},[2591],{"type":52,"value":2592},"Legacy fixed partitioning. Prefer Liquid Clustering.",{"type":46,"tag":415,"props":2594,"children":2595},{},[2596],{"type":46,"tag":67,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":52,"value":2601},"partition_cols=[...]",{"type":46,"tag":415,"props":2603,"children":2604},{},[2605],{"type":46,"tag":67,"props":2606,"children":2608},{"className":2607},[],[2609],{"type":52,"value":2610},"PARTITIONED BY (col1, col2)",{"type":46,"tag":415,"props":2612,"children":2613},{},[2614],{"type":46,"tag":295,"props":2615,"children":2616},{"href":1038},[2617],{"type":52,"value":1041},{"type":46,"tag":415,"props":2619,"children":2620},{},[2621],{"type":46,"tag":295,"props":2622,"children":2623},{"href":1047},[2624],{"type":52,"value":1050},{"type":46,"tag":393,"props":2626,"children":2627},{},[2628,2633,2638,2647,2656,2663],{"type":46,"tag":415,"props":2629,"children":2630},{},[2631],{"type":52,"value":2632},"Table properties",{"type":46,"tag":415,"props":2634,"children":2635},{},[2636],{"type":52,"value":2637},"Delta table properties (auto-optimize, CDF, retention).",{"type":46,"tag":415,"props":2639,"children":2640},{},[2641],{"type":46,"tag":67,"props":2642,"children":2644},{"className":2643},[],[2645],{"type":52,"value":2646},"table_properties={...}",{"type":46,"tag":415,"props":2648,"children":2649},{},[2650],{"type":46,"tag":67,"props":2651,"children":2653},{"className":2652},[],[2654],{"type":52,"value":2655},"TBLPROPERTIES (...)",{"type":46,"tag":415,"props":2657,"children":2658},{},[2659],{"type":46,"tag":295,"props":2660,"children":2661},{"href":1038},[2662],{"type":52,"value":1041},{"type":46,"tag":415,"props":2664,"children":2665},{},[2666],{"type":46,"tag":295,"props":2667,"children":2668},{"href":1047},[2669],{"type":52,"value":1050},{"type":46,"tag":393,"props":2671,"children":2672},{},[2673,2678,2683,2692,2701,2708],{"type":46,"tag":415,"props":2674,"children":2675},{},[2676],{"type":52,"value":2677},"Explicit schema",{"type":46,"tag":415,"props":2679,"children":2680},{},[2681],{"type":52,"value":2682},"Declare column types up front (vs inferred).",{"type":46,"tag":415,"props":2684,"children":2685},{},[2686],{"type":46,"tag":67,"props":2687,"children":2689},{"className":2688},[],[2690],{"type":52,"value":2691},"schema=\"col1 TYPE, ...\"",{"type":46,"tag":415,"props":2693,"children":2694},{},[2695],{"type":46,"tag":67,"props":2696,"children":2698},{"className":2697},[],[2699],{"type":52,"value":2700},"(col1 TYPE, ...) AS",{"type":46,"tag":415,"props":2702,"children":2703},{},[2704],{"type":46,"tag":295,"props":2705,"children":2706},{"href":1038},[2707],{"type":52,"value":1041},{"type":46,"tag":415,"props":2709,"children":2710},{},[2711],{"type":46,"tag":295,"props":2712,"children":2713},{"href":1047},[2714],{"type":52,"value":1050},{"type":46,"tag":393,"props":2716,"children":2717},{},[2718,2723,2728,2737,2746,2753],{"type":46,"tag":415,"props":2719,"children":2720},{},[2721],{"type":52,"value":2722},"Generated columns",{"type":46,"tag":415,"props":2724,"children":2725},{},[2726],{"type":52,"value":2727},"Columns computed from other columns at write time.",{"type":46,"tag":415,"props":2729,"children":2730},{},[2731],{"type":46,"tag":67,"props":2732,"children":2734},{"className":2733},[],[2735],{"type":52,"value":2736},"schema=\"..., col TYPE GENERATED ALWAYS AS (expr)\"",{"type":46,"tag":415,"props":2738,"children":2739},{},[2740],{"type":46,"tag":67,"props":2741,"children":2743},{"className":2742},[],[2744],{"type":52,"value":2745},"col TYPE GENERATED ALWAYS AS (expr)",{"type":46,"tag":415,"props":2747,"children":2748},{},[2749],{"type":46,"tag":295,"props":2750,"children":2751},{"href":1038},[2752],{"type":52,"value":1041},{"type":46,"tag":415,"props":2754,"children":2755},{},[2756],{"type":46,"tag":295,"props":2757,"children":2758},{"href":1047},[2759],{"type":52,"value":1050},{"type":46,"tag":393,"props":2761,"children":2762},{},[2763,2768,2773,2782,2791,2798],{"type":46,"tag":415,"props":2764,"children":2765},{},[2766],{"type":52,"value":2767},"Row filter (Public Preview)",{"type":46,"tag":415,"props":2769,"children":2770},{},[2771],{"type":52,"value":2772},"UC fine-grained access: filter rows by a function.",{"type":46,"tag":415,"props":2774,"children":2775},{},[2776],{"type":46,"tag":67,"props":2777,"children":2779},{"className":2778},[],[2780],{"type":52,"value":2781},"row_filter=\"ROW FILTER fn ON (col)\"",{"type":46,"tag":415,"props":2783,"children":2784},{},[2785],{"type":46,"tag":67,"props":2786,"children":2788},{"className":2787},[],[2789],{"type":52,"value":2790},"WITH ROW FILTER fn ON (col)",{"type":46,"tag":415,"props":2792,"children":2793},{},[2794],{"type":46,"tag":295,"props":2795,"children":2796},{"href":1038},[2797],{"type":52,"value":1041},{"type":46,"tag":415,"props":2799,"children":2800},{},[2801],{"type":46,"tag":295,"props":2802,"children":2803},{"href":1047},[2804],{"type":52,"value":1050},{"type":46,"tag":393,"props":2806,"children":2807},{},[2808,2813,2818,2827,2836,2843],{"type":46,"tag":415,"props":2809,"children":2810},{},[2811],{"type":52,"value":2812},"Column mask (Public Preview)",{"type":46,"tag":415,"props":2814,"children":2815},{},[2816],{"type":52,"value":2817},"UC fine-grained access: mask a column with a function.",{"type":46,"tag":415,"props":2819,"children":2820},{},[2821],{"type":46,"tag":67,"props":2822,"children":2824},{"className":2823},[],[2825],{"type":52,"value":2826},"schema=\"..., col TYPE MASK fn USING COLUMNS (col2)\"",{"type":46,"tag":415,"props":2828,"children":2829},{},[2830],{"type":46,"tag":67,"props":2831,"children":2833},{"className":2832},[],[2834],{"type":52,"value":2835},"col TYPE MASK fn USING COLUMNS (col2)",{"type":46,"tag":415,"props":2837,"children":2838},{},[2839],{"type":46,"tag":295,"props":2840,"children":2841},{"href":1038},[2842],{"type":52,"value":1041},{"type":46,"tag":415,"props":2844,"children":2845},{},[2846],{"type":46,"tag":295,"props":2847,"children":2848},{"href":1047},[2849],{"type":52,"value":1050},{"type":46,"tag":393,"props":2851,"children":2852},{},[2853,2858,2863,2871,2879,2886],{"type":46,"tag":415,"props":2854,"children":2855},{},[2856],{"type":52,"value":2857},"Private dataset",{"type":46,"tag":415,"props":2859,"children":2860},{},[2861],{"type":52,"value":2862},"Materialized intermediate not published to UC.",{"type":46,"tag":415,"props":2864,"children":2865},{},[2866],{"type":46,"tag":67,"props":2867,"children":2869},{"className":2868},[],[2870],{"type":52,"value":178},{"type":46,"tag":415,"props":2872,"children":2873},{},[2874],{"type":46,"tag":67,"props":2875,"children":2877},{"className":2876},[],[2878],{"type":52,"value":185},{"type":46,"tag":415,"props":2880,"children":2881},{},[2882],{"type":46,"tag":295,"props":2883,"children":2884},{"href":1038},[2885],{"type":52,"value":1041},{"type":46,"tag":415,"props":2887,"children":2888},{},[2889],{"type":46,"tag":295,"props":2890,"children":2891},{"href":1047},[2892],{"type":52,"value":1050},{"type":46,"tag":909,"props":2894,"children":2896},{"id":2895},"legacy-dlt-syntax-always-migrate",[2897],{"type":52,"value":2898},"Legacy DLT Syntax — always migrate",{"type":46,"tag":55,"props":2900,"children":2901},{},[2902,2904,2909,2911,2916,2918,2923,2925,2931,2933,2939,2941,2947,2948,2954],{"type":52,"value":2903},"The tables above show ",{"type":46,"tag":59,"props":2905,"children":2906},{},[2907],{"type":52,"value":2908},"only the modern API",{"type":52,"value":2910},". If you see any of the following in user code, it is the legacy DLT syntax — ",{"type":46,"tag":59,"props":2912,"children":2913},{},[2914],{"type":52,"value":2915},"always migrate to the modern form",{"type":52,"value":2917},", do not extend it. Read ",{"type":46,"tag":295,"props":2919,"children":2921},{"href":2920},"references\u002Fdlt-migration.md",[2922],{"type":52,"value":2920},{"type":52,"value":2924}," before suggesting changes so the conversion is correct (especially around ",{"type":46,"tag":67,"props":2926,"children":2928},{"className":2927},[],[2929],{"type":52,"value":2930},"apply_changes",{"type":52,"value":2932}," → ",{"type":46,"tag":67,"props":2934,"children":2936},{"className":2935},[],[2937],{"type":52,"value":2938},"create_auto_cdc_flow",{"type":52,"value":2940}," semantics and ",{"type":46,"tag":67,"props":2942,"children":2944},{"className":2943},[],[2945],{"type":52,"value":2946},"partition_cols",{"type":52,"value":2932},{"type":46,"tag":67,"props":2949,"children":2951},{"className":2950},[],[2952],{"type":52,"value":2953},"cluster_by",{"type":52,"value":2955},").",{"type":46,"tag":385,"props":2957,"children":2958},{},[2959,2975],{"type":46,"tag":389,"props":2960,"children":2961},{},[2962],{"type":46,"tag":393,"props":2963,"children":2964},{},[2965,2970],{"type":46,"tag":397,"props":2966,"children":2967},{},[2968],{"type":52,"value":2969},"If you see…",{"type":46,"tag":397,"props":2971,"children":2972},{},[2973],{"type":52,"value":2974},"…it's DLT. Migrate to",{"type":46,"tag":408,"props":2976,"children":2977},{},[2978,2999,3053,3100,3133,3171,3241,3262,3283,3326,3360,3433,3456,3499,3529],{"type":46,"tag":393,"props":2979,"children":2980},{},[2981,2990],{"type":46,"tag":415,"props":2982,"children":2983},{},[2984],{"type":46,"tag":67,"props":2985,"children":2987},{"className":2986},[],[2988],{"type":52,"value":2989},"import dlt",{"type":46,"tag":415,"props":2991,"children":2992},{},[2993],{"type":46,"tag":67,"props":2994,"children":2996},{"className":2995},[],[2997],{"type":52,"value":2998},"from pyspark import pipelines as dp",{"type":46,"tag":393,"props":3000,"children":3001},{},[3002,3026],{"type":46,"tag":415,"props":3003,"children":3004},{},[3005,3011,3013,3019,3020],{"type":46,"tag":67,"props":3006,"children":3008},{"className":3007},[],[3009],{"type":52,"value":3010},"@dlt.table(...)",{"type":52,"value":3012},", ",{"type":46,"tag":67,"props":3014,"children":3016},{"className":3015},[],[3017],{"type":52,"value":3018},"@dlt.append_flow(...)",{"type":52,"value":3012},{"type":46,"tag":67,"props":3021,"children":3023},{"className":3022},[],[3024],{"type":52,"value":3025},"@dlt.expect*",{"type":46,"tag":415,"props":3027,"children":3028},{},[3029,3031,3037,3039,3045,3046,3052],{"type":52,"value":3030},"Same decorator name on ",{"type":46,"tag":67,"props":3032,"children":3034},{"className":3033},[],[3035],{"type":52,"value":3036},"dp.*",{"type":52,"value":3038}," (e.g. ",{"type":46,"tag":67,"props":3040,"children":3042},{"className":3041},[],[3043],{"type":52,"value":3044},"@dp.table",{"type":52,"value":3012},{"type":46,"tag":67,"props":3047,"children":3049},{"className":3048},[],[3050],{"type":52,"value":3051},"@dp.expect_or_drop",{"type":52,"value":2955},{"type":46,"tag":393,"props":3054,"children":3055},{},[3056,3074],{"type":46,"tag":415,"props":3057,"children":3058},{},[3059,3065,3066,3072],{"type":46,"tag":67,"props":3060,"children":3062},{"className":3061},[],[3063],{"type":52,"value":3064},"@dlt.view(...)",{"type":52,"value":626},{"type":46,"tag":67,"props":3067,"children":3069},{"className":3068},[],[3070],{"type":52,"value":3071},"@dp.view(...)",{"type":52,"value":3073}," if present in older code)",{"type":46,"tag":415,"props":3075,"children":3076},{},[3077,3083,3085,3091,3093,3099],{"type":46,"tag":67,"props":3078,"children":3080},{"className":3079},[],[3081],{"type":52,"value":3082},"@dp.temporary_view(...)",{"type":52,"value":3084}," — the modern API has no ",{"type":46,"tag":67,"props":3086,"children":3088},{"className":3087},[],[3089],{"type":52,"value":3090},"view",{"type":52,"value":3092}," decorator, only ",{"type":46,"tag":67,"props":3094,"children":3096},{"className":3095},[],[3097],{"type":52,"value":3098},"temporary_view",{"type":52,"value":267},{"type":46,"tag":393,"props":3101,"children":3102},{},[3103,3119],{"type":46,"tag":415,"props":3104,"children":3105},{},[3106,3112,3113],{"type":46,"tag":67,"props":3107,"children":3109},{"className":3108},[],[3110],{"type":52,"value":3111},"dlt.read(\"name\")",{"type":52,"value":146},{"type":46,"tag":67,"props":3114,"children":3116},{"className":3115},[],[3117],{"type":52,"value":3118},"dlt.read_stream(\"name\")",{"type":46,"tag":415,"props":3120,"children":3121},{},[3122,3127,3128],{"type":46,"tag":67,"props":3123,"children":3125},{"className":3124},[],[3126],{"type":52,"value":1975},{"type":52,"value":146},{"type":46,"tag":67,"props":3129,"children":3131},{"className":3130},[],[3132],{"type":52,"value":2014},{"type":46,"tag":393,"props":3134,"children":3135},{},[3136,3152],{"type":46,"tag":415,"props":3137,"children":3138},{},[3139,3145,3146],{"type":46,"tag":67,"props":3140,"children":3142},{"className":3141},[],[3143],{"type":52,"value":3144},"dp.read(...)",{"type":52,"value":146},{"type":46,"tag":67,"props":3147,"children":3149},{"className":3148},[],[3150],{"type":52,"value":3151},"dp.read_stream(...)",{"type":46,"tag":415,"props":3153,"children":3154},{},[3155,3157,3163,3164,3170],{"type":52,"value":3156},"Also legacy — use ",{"type":46,"tag":67,"props":3158,"children":3160},{"className":3159},[],[3161],{"type":52,"value":3162},"spark.read.table(...)",{"type":52,"value":146},{"type":46,"tag":67,"props":3165,"children":3167},{"className":3166},[],[3168],{"type":52,"value":3169},"spark.readStream.table(...)",{"type":52,"value":267},{"type":46,"tag":393,"props":3172,"children":3173},{},[3174,3190],{"type":46,"tag":415,"props":3175,"children":3176},{},[3177,3183,3184],{"type":46,"tag":67,"props":3178,"children":3180},{"className":3179},[],[3181],{"type":52,"value":3182},"dlt.apply_changes(...)",{"type":52,"value":146},{"type":46,"tag":67,"props":3185,"children":3187},{"className":3186},[],[3188],{"type":52,"value":3189},"dp.apply_changes(...)",{"type":46,"tag":415,"props":3191,"children":3192},{},[3193,3199,3201,3207,3209,3215,3217,3223,3225,3231,3233,3239],{"type":46,"tag":67,"props":3194,"children":3196},{"className":3195},[],[3197],{"type":52,"value":3198},"dp.create_auto_cdc_flow(...)",{"type":52,"value":3200},". ",{"type":46,"tag":67,"props":3202,"children":3204},{"className":3203},[],[3205],{"type":52,"value":3206},"sequence_by",{"type":52,"value":3208}," accepts a column name (string) or ",{"type":46,"tag":67,"props":3210,"children":3212},{"className":3211},[],[3213],{"type":52,"value":3214},"col(...)",{"type":52,"value":3216},"; ",{"type":46,"tag":67,"props":3218,"children":3220},{"className":3219},[],[3221],{"type":52,"value":3222},"stored_as_scd_type",{"type":52,"value":3224}," is integer ",{"type":46,"tag":67,"props":3226,"children":3228},{"className":3227},[],[3229],{"type":52,"value":3230},"2",{"type":52,"value":3232}," for Type 2 or string ",{"type":46,"tag":67,"props":3234,"children":3236},{"className":3235},[],[3237],{"type":52,"value":3238},"\"1\"",{"type":52,"value":3240}," for Type 1.",{"type":46,"tag":393,"props":3242,"children":3243},{},[3244,3253],{"type":46,"tag":415,"props":3245,"children":3246},{},[3247],{"type":46,"tag":67,"props":3248,"children":3250},{"className":3249},[],[3251],{"type":52,"value":3252},"dlt.apply_changes_from_snapshot(...)",{"type":46,"tag":415,"props":3254,"children":3255},{},[3256],{"type":46,"tag":67,"props":3257,"children":3259},{"className":3258},[],[3260],{"type":52,"value":3261},"dp.create_auto_cdc_from_snapshot_flow(...)",{"type":46,"tag":393,"props":3263,"children":3264},{},[3265,3274],{"type":46,"tag":415,"props":3266,"children":3267},{},[3268],{"type":46,"tag":67,"props":3269,"children":3271},{"className":3270},[],[3272],{"type":52,"value":3273},"dlt.create_streaming_table(...)",{"type":46,"tag":415,"props":3275,"children":3276},{},[3277],{"type":46,"tag":67,"props":3278,"children":3280},{"className":3279},[],[3281],{"type":52,"value":3282},"dp.create_streaming_table(...)",{"type":46,"tag":393,"props":3284,"children":3285},{},[3286,3297],{"type":46,"tag":415,"props":3287,"children":3288},{},[3289,3295],{"type":46,"tag":67,"props":3290,"children":3292},{"className":3291},[],[3293],{"type":52,"value":3294},"LIVE.\u003Cname>",{"type":52,"value":3296}," prefix in SQL",{"type":46,"tag":415,"props":3298,"children":3299},{},[3300,3302,3308,3310,3316,3318,3324],{"type":52,"value":3301},"Bare name (",{"type":46,"tag":67,"props":3303,"children":3305},{"className":3304},[],[3306],{"type":52,"value":3307},"SELECT FROM name",{"type":52,"value":3309}," for batch, ",{"type":46,"tag":67,"props":3311,"children":3313},{"className":3312},[],[3314],{"type":52,"value":3315},"SELECT FROM STREAM(name)",{"type":52,"value":3317}," for streaming). ",{"type":46,"tag":67,"props":3319,"children":3321},{"className":3320},[],[3322],{"type":52,"value":3323},"LIVE.",{"type":52,"value":3325}," will error in modern pipelines.",{"type":46,"tag":393,"props":3327,"children":3328},{},[3329,3345],{"type":46,"tag":415,"props":3330,"children":3331},{},[3332,3338,3339],{"type":46,"tag":67,"props":3333,"children":3335},{"className":3334},[],[3336],{"type":52,"value":3337},"CREATE LIVE TABLE",{"type":52,"value":146},{"type":46,"tag":67,"props":3340,"children":3342},{"className":3341},[],[3343],{"type":52,"value":3344},"CREATE STREAMING LIVE TABLE",{"type":46,"tag":415,"props":3346,"children":3347},{},[3348,3353,3354,3359],{"type":46,"tag":67,"props":3349,"children":3351},{"className":3350},[],[3352],{"type":52,"value":456},{"type":52,"value":146},{"type":46,"tag":67,"props":3355,"children":3357},{"className":3356},[],[3358],{"type":52,"value":449},{"type":52,"value":267},{"type":46,"tag":393,"props":3361,"children":3362},{},[3363,3382],{"type":46,"tag":415,"props":3364,"children":3365},{},[3366,3372,3374,3380],{"type":46,"tag":67,"props":3367,"children":3369},{"className":3368},[],[3370],{"type":52,"value":3371},"CREATE TEMPORARY LIVE VIEW",{"type":52,"value":3373}," (a.k.a. ",{"type":46,"tag":67,"props":3375,"children":3377},{"className":3376},[],[3378],{"type":52,"value":3379},"CREATE LIVE VIEW",{"type":52,"value":3381},")",{"type":46,"tag":415,"props":3383,"children":3384},{},[3385,3390,3391,3396,3398,3403,3405,3410,3412,3417,3419,3425,3427,3432],{"type":46,"tag":67,"props":3386,"children":3388},{"className":3387},[],[3389],{"type":52,"value":1081},{"type":52,"value":3200},{"type":46,"tag":59,"props":3392,"children":3393},{},[3394],{"type":52,"value":3395},"Exception",{"type":52,"value":3397},": ",{"type":46,"tag":67,"props":3399,"children":3401},{"className":3400},[],[3402],{"type":52,"value":1081},{"type":52,"value":3404}," does NOT support ",{"type":46,"tag":67,"props":3406,"children":3408},{"className":3407},[],[3409],{"type":52,"value":1802},{"type":52,"value":3411}," clauses for expectations — for the rare case where you need expectations on a temp view, ",{"type":46,"tag":67,"props":3413,"children":3415},{"className":3414},[],[3416],{"type":52,"value":3379},{"type":52,"value":3418}," is retained. See ",{"type":46,"tag":295,"props":3420,"children":3422},{"href":3421},"references\u002Ftemporary-view-sql.md#using-expectations-with-temporary-views",[3423],{"type":52,"value":3424},"temporary-view-sql.md",{"type":52,"value":3426}," and ",{"type":46,"tag":295,"props":3428,"children":3429},{"href":1676},[3430],{"type":52,"value":3431},"expectations-sql.md",{"type":52,"value":267},{"type":46,"tag":393,"props":3434,"children":3435},{},[3436,3447],{"type":46,"tag":415,"props":3437,"children":3438},{},[3439,3445],{"type":46,"tag":67,"props":3440,"children":3442},{"className":3441},[],[3443],{"type":52,"value":3444},"APPLY CHANGES INTO ... FROM STREAM ...",{"type":52,"value":3446}," (SQL)",{"type":46,"tag":415,"props":3448,"children":3449},{},[3450],{"type":46,"tag":67,"props":3451,"children":3453},{"className":3452},[],[3454],{"type":52,"value":3455},"AUTO CDC INTO ... FROM STREAM ...",{"type":46,"tag":393,"props":3457,"children":3458},{},[3459,3482],{"type":46,"tag":415,"props":3460,"children":3461},{},[3462,3467,3468,3474,3476],{"type":46,"tag":67,"props":3463,"children":3465},{"className":3464},[],[3466],{"type":52,"value":2601},{"type":52,"value":146},{"type":46,"tag":67,"props":3469,"children":3471},{"className":3470},[],[3472],{"type":52,"value":3473},"PARTITIONED BY (...)",{"type":52,"value":3475}," + ",{"type":46,"tag":67,"props":3477,"children":3479},{"className":3478},[],[3480],{"type":52,"value":3481},"ZORDER",{"type":46,"tag":415,"props":3483,"children":3484},{},[3485,3490,3491,3497],{"type":46,"tag":67,"props":3486,"children":3488},{"className":3487},[],[3489],{"type":52,"value":2511},{"type":52,"value":146},{"type":46,"tag":67,"props":3492,"children":3494},{"className":3493},[],[3495],{"type":52,"value":3496},"CLUSTER BY (...)",{"type":52,"value":3498}," (Liquid Clustering).",{"type":46,"tag":393,"props":3500,"children":3501},{},[3502,3511],{"type":46,"tag":415,"props":3503,"children":3504},{},[3505],{"type":46,"tag":67,"props":3506,"children":3508},{"className":3507},[],[3509],{"type":52,"value":3510},"input_file_name()",{"type":46,"tag":415,"props":3512,"children":3513},{},[3514,3520,3522,3528],{"type":46,"tag":67,"props":3515,"children":3517},{"className":3516},[],[3518],{"type":52,"value":3519},"_metadata.file_path",{"type":52,"value":3521}," (SQL) \u002F ",{"type":46,"tag":67,"props":3523,"children":3525},{"className":3524},[],[3526],{"type":52,"value":3527},"F.col(\"_metadata.file_path\")",{"type":52,"value":203},{"type":46,"tag":393,"props":3530,"children":3531},{},[3532,3551],{"type":46,"tag":415,"props":3533,"children":3534},{},[3535,3541,3543,3549],{"type":46,"tag":67,"props":3536,"children":3538},{"className":3537},[],[3539],{"type":52,"value":3540},"target=...",{"type":52,"value":3542}," parameter on ",{"type":46,"tag":67,"props":3544,"children":3546},{"className":3545},[],[3547],{"type":52,"value":3548},"create_streaming_table",{"type":52,"value":3550}," \u002F pipeline config",{"type":46,"tag":415,"props":3552,"children":3553},{},[3554],{"type":46,"tag":67,"props":3555,"children":3557},{"className":3556},[],[3558],{"type":52,"value":3559},"schema=...",{"type":46,"tag":75,"props":3561,"children":3563},{"id":3562},"language-selection-python-vs-sql",[3564],{"type":52,"value":3565},"Language Selection (Python vs SQL)",{"type":46,"tag":55,"props":3567,"children":3568},{},[3569,3571,3577,3578,3584],{"type":52,"value":3570},"Decide before scaffolding — the choice picks template files (",{"type":46,"tag":67,"props":3572,"children":3574},{"className":3573},[],[3575],{"type":52,"value":3576},".py",{"type":52,"value":237},{"type":46,"tag":67,"props":3579,"children":3581},{"className":3580},[],[3582],{"type":52,"value":3583},".sql",{"type":52,"value":3585},") and which reference docs apply. Both can coexist, but pick a primary. When unsure, default to SQL for simplicity.",{"type":46,"tag":385,"props":3587,"children":3588},{},[3589,3605],{"type":46,"tag":389,"props":3590,"children":3591},{},[3592],{"type":46,"tag":393,"props":3593,"children":3594},{},[3595,3600],{"type":46,"tag":397,"props":3596,"children":3597},{},[3598],{"type":52,"value":3599},"User signal",{"type":46,"tag":397,"props":3601,"children":3602},{},[3603],{"type":52,"value":3604},"Pick",{"type":46,"tag":408,"props":3606,"children":3607},{},[3608,3623,3638,3655],{"type":46,"tag":393,"props":3609,"children":3610},{},[3611,3616],{"type":46,"tag":415,"props":3612,"children":3613},{},[3614],{"type":52,"value":3615},"\"Python pipeline\", UDF, pandas, ML inference, pyspark",{"type":46,"tag":415,"props":3617,"children":3618},{},[3619],{"type":46,"tag":59,"props":3620,"children":3621},{},[3622],{"type":52,"value":20},{"type":46,"tag":393,"props":3624,"children":3625},{},[3626,3631],{"type":46,"tag":415,"props":3627,"children":3628},{},[3629],{"type":52,"value":3630},"\"SQL pipeline\", \"SQL files\"",{"type":46,"tag":415,"props":3632,"children":3633},{},[3634],{"type":46,"tag":59,"props":3635,"children":3636},{},[3637],{"type":52,"value":24},{"type":46,"tag":393,"props":3639,"children":3640},{},[3641,3646],{"type":46,"tag":415,"props":3642,"children":3643},{},[3644],{"type":52,"value":3645},"\"Simple pipeline\", \"create a table\", \"an aggregation\"",{"type":46,"tag":415,"props":3647,"children":3648},{},[3649,3653],{"type":46,"tag":59,"props":3650,"children":3651},{},[3652],{"type":52,"value":24},{"type":52,"value":3654}," (simpler, use it as default)",{"type":46,"tag":393,"props":3656,"children":3657},{},[3658,3663],{"type":46,"tag":415,"props":3659,"children":3660},{},[3661],{"type":52,"value":3662},"Complex parameterized logic, custom UDFs, ML",{"type":46,"tag":415,"props":3664,"children":3665},{},[3666],{"type":46,"tag":59,"props":3667,"children":3668},{},[3669],{"type":52,"value":20},{"type":46,"tag":55,"props":3671,"children":3672},{},[3673],{"type":52,"value":3674},"If ambiguous, ask. Stick with the chosen language unless the user explicitly switches.",{"type":46,"tag":75,"props":3676,"children":3678},{"id":3677},"choose-your-workflow",[3679],{"type":52,"value":3680},"Choose Your Workflow",{"type":46,"tag":55,"props":3682,"children":3683},{},[3684],{"type":52,"value":3685},"Three project shapes exist — pick before scaffolding. Default to A for production-bound work and C for exploration \u002F demo scaffolding.",{"type":46,"tag":105,"props":3687,"children":3688},{},[3689,3721,3759],{"type":46,"tag":109,"props":3690,"children":3691},{},[3692,3697,3699,3705,3707,3713,3715],{"type":46,"tag":59,"props":3693,"children":3694},{},[3695],{"type":52,"value":3696},"A: Standalone new pipeline project (DAB)",{"type":52,"value":3698}," — pipeline IS the project, no existing ",{"type":46,"tag":67,"props":3700,"children":3702},{"className":3701},[],[3703],{"type":52,"value":3704},"databricks.yml",{"type":52,"value":3706},". Scaffold with ",{"type":46,"tag":67,"props":3708,"children":3710},{"className":3709},[],[3711],{"type":52,"value":3712},"databricks pipelines init --output-dir . --config-file init-config.json",{"type":52,"value":3714},". → ",{"type":46,"tag":295,"props":3716,"children":3718},{"href":3717},"references\u002F1-project-initialization-with-dab.md",[3719],{"type":52,"value":3720},"1-project-initialization-with-dab.md",{"type":46,"tag":109,"props":3722,"children":3723},{},[3724,3729,3731,3736,3738,3744,3746,3752,3753],{"type":46,"tag":59,"props":3725,"children":3726},{},[3727],{"type":52,"value":3728},"B: Pipeline in an existing bundle (DAB)",{"type":52,"value":3730}," — ",{"type":46,"tag":67,"props":3732,"children":3734},{"className":3733},[],[3735],{"type":52,"value":3704},{"type":52,"value":3737}," already exists. Add a ",{"type":46,"tag":67,"props":3739,"children":3741},{"className":3740},[],[3742],{"type":52,"value":3743},"resources\u002F\u003Cname>.pipeline.yml",{"type":52,"value":3745}," pointing at ",{"type":46,"tag":67,"props":3747,"children":3749},{"className":3748},[],[3750],{"type":52,"value":3751},"src\u002F",{"type":52,"value":3714},{"type":46,"tag":295,"props":3754,"children":3756},{"href":3755},"references\u002F1-project-initialization-with-dab.md#workflow-b-pipeline-in-existing-bundle",[3757],{"type":52,"value":3758},"1-project-initialization-with-dab.md#workflow-b-pipeline-in-existing-bundle",{"type":46,"tag":109,"props":3760,"children":3761},{},[3762,3767,3769,3775,3777],{"type":46,"tag":59,"props":3763,"children":3764},{},[3765],{"type":52,"value":3766},"C: Rapid CLI iteration (no bundle)",{"type":52,"value":3768}," — prototyping. ",{"type":46,"tag":67,"props":3770,"children":3772},{"className":3771},[],[3773],{"type":52,"value":3774},"databricks pipelines create \u002F start-update \u002F list-pipeline-events",{"type":52,"value":3776},"; formalise into a bundle later if the work goes to production. → ",{"type":46,"tag":295,"props":3778,"children":3780},{"href":3779},"references\u002F2-rapid-iteration-with-cli.md",[3781],{"type":52,"value":772},{"type":46,"tag":75,"props":3783,"children":3785},{"id":3784},"pipeline-structure",[3786],{"type":52,"value":3787},"Pipeline Structure",{"type":46,"tag":105,"props":3789,"children":3790},{},[3791,3796,3815],{"type":46,"tag":109,"props":3792,"children":3793},{},[3794],{"type":52,"value":3795},"Follow the medallion pattern (Bronze → Silver → Gold) unless the user says otherwise. Keep it simple by default — just a few tables.",{"type":46,"tag":109,"props":3797,"children":3798},{},[3799,3801,3806,3808,3814],{"type":52,"value":3800},"One dataset per file, named after the dataset. Transformation files live in ",{"type":46,"tag":67,"props":3802,"children":3804},{"className":3803},[],[3805],{"type":52,"value":3751},{"type":52,"value":3807}," or ",{"type":46,"tag":67,"props":3809,"children":3811},{"className":3810},[],[3812],{"type":52,"value":3813},"transformations\u002F",{"type":52,"value":267},{"type":46,"tag":109,"props":3816,"children":3817},{},[3818,3823],{"type":46,"tag":59,"props":3819,"children":3820},{},[3821],{"type":52,"value":3822},"Gold layer: preserve key business dimensions.",{"type":52,"value":3824}," When aggregating into Gold, keep the dimensions analysts will filter \u002F slice by (location, department, product line, customer segment, time period). Over-aggregating loses information that can't be recovered downstream. If a dashboard is mentioned, every filter on it needs to be a column in the Gold table. Easier to aggregate further in queries than to recover lost dimensions.",{"type":46,"tag":75,"props":3826,"children":3828},{"id":3827},"running-a-pipeline",[3829],{"type":52,"value":3830},"Running a Pipeline",{"type":46,"tag":55,"props":3832,"children":3833},{},[3834],{"type":52,"value":3835},"Picking the right run command depends on the workflow chosen above.",{"type":46,"tag":105,"props":3837,"children":3838},{},[3839,4098],{"type":46,"tag":109,"props":3840,"children":3841},{},[3842,3847,3849,3855,3857,4087,4091,4093],{"type":46,"tag":59,"props":3843,"children":3844},{},[3845],{"type":52,"value":3846},"Workflow A \u002F B (DAB)",{"type":52,"value":3848}," — Code changes only take effect after ",{"type":46,"tag":67,"props":3850,"children":3852},{"className":3851},[],[3853],{"type":52,"value":3854},"databricks bundle deploy",{"type":52,"value":3856},". Always deploy before any run, dry run, or selective refresh.",{"type":46,"tag":87,"props":3858,"children":3862},{"className":3859,"code":3860,"language":3861,"meta":95,"style":95},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","databricks bundle validate --profile \u003Cprofile>\ndatabricks bundle deploy -t dev --profile \u003Cprofile>\ndatabricks bundle run \u003Cpipeline_name> -t dev --profile \u003Cprofile>\ndatabricks pipelines get \u003Cpipeline_id> --profile \u003Cprofile>      # status\n","bash",[3863],{"type":46,"tag":67,"props":3864,"children":3865},{"__ignoreMap":95},[3866,3915,3962,4025],{"type":46,"tag":3867,"props":3868,"children":3871},"span",{"class":3869,"line":3870},"line",1,[3872,3877,3883,3888,3893,3899,3904,3910],{"type":46,"tag":3867,"props":3873,"children":3875},{"style":3874},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[3876],{"type":52,"value":8},{"type":46,"tag":3867,"props":3878,"children":3880},{"style":3879},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[3881],{"type":52,"value":3882}," bundle",{"type":46,"tag":3867,"props":3884,"children":3885},{"style":3879},[3886],{"type":52,"value":3887}," validate",{"type":46,"tag":3867,"props":3889,"children":3890},{"style":3879},[3891],{"type":52,"value":3892}," --profile",{"type":46,"tag":3867,"props":3894,"children":3896},{"style":3895},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[3897],{"type":52,"value":3898}," \u003C",{"type":46,"tag":3867,"props":3900,"children":3901},{"style":3879},[3902],{"type":52,"value":3903},"profil",{"type":46,"tag":3867,"props":3905,"children":3907},{"style":3906},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[3908],{"type":52,"value":3909},"e",{"type":46,"tag":3867,"props":3911,"children":3912},{"style":3895},[3913],{"type":52,"value":3914},">\n",{"type":46,"tag":3867,"props":3916,"children":3918},{"class":3869,"line":3917},2,[3919,3923,3927,3932,3937,3942,3946,3950,3954,3958],{"type":46,"tag":3867,"props":3920,"children":3921},{"style":3874},[3922],{"type":52,"value":8},{"type":46,"tag":3867,"props":3924,"children":3925},{"style":3879},[3926],{"type":52,"value":3882},{"type":46,"tag":3867,"props":3928,"children":3929},{"style":3879},[3930],{"type":52,"value":3931}," deploy",{"type":46,"tag":3867,"props":3933,"children":3934},{"style":3879},[3935],{"type":52,"value":3936}," -t",{"type":46,"tag":3867,"props":3938,"children":3939},{"style":3879},[3940],{"type":52,"value":3941}," dev",{"type":46,"tag":3867,"props":3943,"children":3944},{"style":3879},[3945],{"type":52,"value":3892},{"type":46,"tag":3867,"props":3947,"children":3948},{"style":3895},[3949],{"type":52,"value":3898},{"type":46,"tag":3867,"props":3951,"children":3952},{"style":3879},[3953],{"type":52,"value":3903},{"type":46,"tag":3867,"props":3955,"children":3956},{"style":3906},[3957],{"type":52,"value":3909},{"type":46,"tag":3867,"props":3959,"children":3960},{"style":3895},[3961],{"type":52,"value":3914},{"type":46,"tag":3867,"props":3963,"children":3965},{"class":3869,"line":3964},3,[3966,3970,3974,3979,3983,3988,3992,3997,4001,4005,4009,4013,4017,4021],{"type":46,"tag":3867,"props":3967,"children":3968},{"style":3874},[3969],{"type":52,"value":8},{"type":46,"tag":3867,"props":3971,"children":3972},{"style":3879},[3973],{"type":52,"value":3882},{"type":46,"tag":3867,"props":3975,"children":3976},{"style":3879},[3977],{"type":52,"value":3978}," run",{"type":46,"tag":3867,"props":3980,"children":3981},{"style":3895},[3982],{"type":52,"value":3898},{"type":46,"tag":3867,"props":3984,"children":3985},{"style":3879},[3986],{"type":52,"value":3987},"pipeline_nam",{"type":46,"tag":3867,"props":3989,"children":3990},{"style":3906},[3991],{"type":52,"value":3909},{"type":46,"tag":3867,"props":3993,"children":3994},{"style":3895},[3995],{"type":52,"value":3996},">",{"type":46,"tag":3867,"props":3998,"children":3999},{"style":3879},[4000],{"type":52,"value":3936},{"type":46,"tag":3867,"props":4002,"children":4003},{"style":3879},[4004],{"type":52,"value":3941},{"type":46,"tag":3867,"props":4006,"children":4007},{"style":3879},[4008],{"type":52,"value":3892},{"type":46,"tag":3867,"props":4010,"children":4011},{"style":3895},[4012],{"type":52,"value":3898},{"type":46,"tag":3867,"props":4014,"children":4015},{"style":3879},[4016],{"type":52,"value":3903},{"type":46,"tag":3867,"props":4018,"children":4019},{"style":3906},[4020],{"type":52,"value":3909},{"type":46,"tag":3867,"props":4022,"children":4023},{"style":3895},[4024],{"type":52,"value":3914},{"type":46,"tag":3867,"props":4026,"children":4028},{"class":3869,"line":4027},4,[4029,4033,4038,4043,4047,4052,4057,4061,4065,4069,4073,4077,4081],{"type":46,"tag":3867,"props":4030,"children":4031},{"style":3874},[4032],{"type":52,"value":8},{"type":46,"tag":3867,"props":4034,"children":4035},{"style":3879},[4036],{"type":52,"value":4037}," pipelines",{"type":46,"tag":3867,"props":4039,"children":4040},{"style":3879},[4041],{"type":52,"value":4042}," get",{"type":46,"tag":3867,"props":4044,"children":4045},{"style":3895},[4046],{"type":52,"value":3898},{"type":46,"tag":3867,"props":4048,"children":4049},{"style":3879},[4050],{"type":52,"value":4051},"pipeline_i",{"type":46,"tag":3867,"props":4053,"children":4054},{"style":3906},[4055],{"type":52,"value":4056},"d",{"type":46,"tag":3867,"props":4058,"children":4059},{"style":3895},[4060],{"type":52,"value":3996},{"type":46,"tag":3867,"props":4062,"children":4063},{"style":3879},[4064],{"type":52,"value":3892},{"type":46,"tag":3867,"props":4066,"children":4067},{"style":3895},[4068],{"type":52,"value":3898},{"type":46,"tag":3867,"props":4070,"children":4071},{"style":3879},[4072],{"type":52,"value":3903},{"type":46,"tag":3867,"props":4074,"children":4075},{"style":3906},[4076],{"type":52,"value":3909},{"type":46,"tag":3867,"props":4078,"children":4079},{"style":3895},[4080],{"type":52,"value":3996},{"type":46,"tag":3867,"props":4082,"children":4084},{"style":4083},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[4085],{"type":52,"value":4086},"      # status\n",{"type":46,"tag":4088,"props":4089,"children":4090},"br",{},[],{"type":52,"value":4092},"→ Full DAB run + iteration details: ",{"type":46,"tag":295,"props":4094,"children":4096},{"href":4095},"references\u002F1-project-initialization-with-dab.md#running-a-pipeline-workflow-a--b",[4097],{"type":52,"value":4095},{"type":46,"tag":109,"props":4099,"children":4100},{},[4101,4106,4108,4203,4206,4208],{"type":46,"tag":59,"props":4102,"children":4103},{},[4104],{"type":52,"value":4105},"Workflow C (CLI, no bundle)",{"type":52,"value":4107}," — Upload files to the workspace, then drive the pipeline directly. Re-upload after every code change.",{"type":46,"tag":87,"props":4109,"children":4111},{"className":3859,"code":4110,"language":3861,"meta":95,"style":95},"databricks workspace import-dir .\u002Fmy_pipeline \u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fmy_pipeline --overwrite\ndatabricks pipelines start-update \u003Cpipeline_id>\n",[4112],{"type":46,"tag":67,"props":4113,"children":4114},{"__ignoreMap":95},[4115,4171],{"type":46,"tag":3867,"props":4116,"children":4117},{"class":3869,"line":3870},[4118,4122,4127,4132,4137,4142,4147,4152,4157,4161,4166],{"type":46,"tag":3867,"props":4119,"children":4120},{"style":3874},[4121],{"type":52,"value":8},{"type":46,"tag":3867,"props":4123,"children":4124},{"style":3879},[4125],{"type":52,"value":4126}," workspace",{"type":46,"tag":3867,"props":4128,"children":4129},{"style":3879},[4130],{"type":52,"value":4131}," import-dir",{"type":46,"tag":3867,"props":4133,"children":4134},{"style":3879},[4135],{"type":52,"value":4136}," .\u002Fmy_pipeline",{"type":46,"tag":3867,"props":4138,"children":4139},{"style":3879},[4140],{"type":52,"value":4141}," \u002FWorkspace\u002FUsers\u002F",{"type":46,"tag":3867,"props":4143,"children":4144},{"style":3895},[4145],{"type":52,"value":4146},"\u003C",{"type":46,"tag":3867,"props":4148,"children":4149},{"style":3879},[4150],{"type":52,"value":4151},"use",{"type":46,"tag":3867,"props":4153,"children":4154},{"style":3906},[4155],{"type":52,"value":4156},"r",{"type":46,"tag":3867,"props":4158,"children":4159},{"style":3895},[4160],{"type":52,"value":3996},{"type":46,"tag":3867,"props":4162,"children":4163},{"style":3879},[4164],{"type":52,"value":4165},"\u002Fmy_pipeline",{"type":46,"tag":3867,"props":4167,"children":4168},{"style":3879},[4169],{"type":52,"value":4170}," --overwrite\n",{"type":46,"tag":3867,"props":4172,"children":4173},{"class":3869,"line":3917},[4174,4178,4182,4187,4191,4195,4199],{"type":46,"tag":3867,"props":4175,"children":4176},{"style":3874},[4177],{"type":52,"value":8},{"type":46,"tag":3867,"props":4179,"children":4180},{"style":3879},[4181],{"type":52,"value":4037},{"type":46,"tag":3867,"props":4183,"children":4184},{"style":3879},[4185],{"type":52,"value":4186}," start-update",{"type":46,"tag":3867,"props":4188,"children":4189},{"style":3895},[4190],{"type":52,"value":3898},{"type":46,"tag":3867,"props":4192,"children":4193},{"style":3879},[4194],{"type":52,"value":4051},{"type":46,"tag":3867,"props":4196,"children":4197},{"style":3906},[4198],{"type":52,"value":4056},{"type":46,"tag":3867,"props":4200,"children":4201},{"style":3895},[4202],{"type":52,"value":3914},{"type":46,"tag":4088,"props":4204,"children":4205},{},[],{"type":52,"value":4207},"→ Full CLI run + polling pattern: ",{"type":46,"tag":295,"props":4209,"children":4210},{"href":3779},[4211],{"type":52,"value":3779},{"type":46,"tag":55,"props":4213,"children":4214},{},[4215],{"type":46,"tag":59,"props":4216,"children":4217},{},[4218],{"type":52,"value":4219},"Refresh modes (both workflows):",{"type":46,"tag":105,"props":4221,"children":4222},{},[4223,4233],{"type":46,"tag":109,"props":4224,"children":4225},{},[4226,4231],{"type":46,"tag":59,"props":4227,"children":4228},{},[4229],{"type":52,"value":4230},"Selective refresh",{"type":52,"value":4232}," is preferred when you only need to run one table. Dependencies must already be materialized.",{"type":46,"tag":109,"props":4234,"children":4235},{},[4236,4241,4243,4248],{"type":46,"tag":59,"props":4237,"children":4238},{},[4239],{"type":52,"value":4240},"Full refresh",{"type":52,"value":4242}," is the most expensive and dangerous option and ",{"type":46,"tag":59,"props":4244,"children":4245},{},[4246],{"type":52,"value":4247},"can lead to data loss",{"type":52,"value":4249}," (it reprocesses streaming sources from scratch, destroying streaming state). Use only when really necessary. Always suggest it as a follow-up the user must explicitly approve.",{"type":46,"tag":55,"props":4251,"children":4252},{},[4253,4258,4260,4265],{"type":46,"tag":59,"props":4254,"children":4255},{},[4256],{"type":52,"value":4257},"Always poll the update",{"type":52,"value":4259},", not top-level pipeline state — see the polling rationale in ",{"type":46,"tag":295,"props":4261,"children":4262},{"href":769},[4263],{"type":52,"value":4264},"2-rapid-iteration-with-cli.md#step-4-start-an-update-and-poll-that-update",{"type":52,"value":4266},". Same rule applies to bundle runs.",{"type":46,"tag":75,"props":4268,"children":4270},{"id":4269},"reference-index",[4271],{"type":52,"value":4272},"Reference Index",{"type":46,"tag":55,"props":4274,"children":4275},{},[4276],{"type":52,"value":4277},"Project & lifecycle:",{"type":46,"tag":105,"props":4279,"children":4280},{},[4281,4290,4299,4310,4320],{"type":46,"tag":109,"props":4282,"children":4283},{},[4284,4288],{"type":46,"tag":295,"props":4285,"children":4286},{"href":3717},[4287],{"type":52,"value":3720},{"type":52,"value":4289}," — Workflows A and B.",{"type":46,"tag":109,"props":4291,"children":4292},{},[4293,4297],{"type":46,"tag":295,"props":4294,"children":4295},{"href":3779},[4296],{"type":52,"value":772},{"type":52,"value":4298}," — Workflow C; start-update + polling + error-extraction.",{"type":46,"tag":109,"props":4300,"children":4301},{},[4302,4308],{"type":46,"tag":295,"props":4303,"children":4305},{"href":4304},"references\u002Fpipeline-configuration.md",[4306],{"type":52,"value":4307},"pipeline-configuration.md",{"type":52,"value":4309}," — Full create\u002Fupdate JSON reference + variant snippets + multi-schema + platform constraints.",{"type":46,"tag":109,"props":4311,"children":4312},{},[4313,4318],{"type":46,"tag":295,"props":4314,"children":4315},{"href":532},[4316],{"type":52,"value":4317},"performance.md",{"type":52,"value":4319}," — Liquid Clustering, state management, joins, pre-aggregation, monitoring.",{"type":46,"tag":109,"props":4321,"children":4322},{},[4323,4328],{"type":46,"tag":295,"props":4324,"children":4325},{"href":2920},[4326],{"type":52,"value":4327},"dlt-migration.md",{"type":52,"value":4329}," — DLT → SDP conversions.",{"type":46,"tag":55,"props":4331,"children":4332},{},[4333],{"type":52,"value":4334},"Cross-cutting patterns:",{"type":46,"tag":105,"props":4336,"children":4337},{},[4338,4349,4358,4368],{"type":46,"tag":109,"props":4339,"children":4340},{},[4341,4347],{"type":46,"tag":295,"props":4342,"children":4344},{"href":4343},"references\u002Fstreaming-patterns.md",[4345],{"type":52,"value":4346},"streaming-patterns.md",{"type":52,"value":4348}," — Dedup, windowed aggregations, late data, rescue-data quarantine, anomaly detection, lag monitoring.",{"type":46,"tag":109,"props":4350,"children":4351},{},[4352,4356],{"type":46,"tag":295,"props":4353,"children":4354},{"href":1584},[4355],{"type":52,"value":1587},{"type":52,"value":4357}," — Current-state, point-in-time, joining facts with historical dims.",{"type":46,"tag":109,"props":4359,"children":4360},{},[4361,4366],{"type":46,"tag":295,"props":4362,"children":4363},{"href":2117},[4364],{"type":52,"value":4365},"kafka.md",{"type":52,"value":4367}," — Kafka \u002F Event Hubs ingestion.",{"type":46,"tag":109,"props":4369,"children":4370},{},[4371,4376,4378,4384],{"type":46,"tag":295,"props":4372,"children":4373},{"href":1428},[4374],{"type":52,"value":4375},"real-time-mode.md",{"type":52,"value":4377}," — Real-Time Mode (RTM): sub-second continuous pipelines (",{"type":46,"tag":67,"props":4379,"children":4381},{"className":4380},[],[4382],{"type":52,"value":4383},"@dp.update_flow",{"type":52,"value":4385},", Kafka sinks, serverless or classic).",{"type":46,"tag":55,"props":4387,"children":4388},{},[4389,4391,4397,4399,4405,4406,4412,4413,4419,4420,4426,4427,4433,4434,4440],{"type":52,"value":4390},"Auto Loader format-specific options: ",{"type":46,"tag":295,"props":4392,"children":4394},{"href":4393},"references\u002Foptions-json.md",[4395],{"type":52,"value":4396},"JSON",{"type":52,"value":4398}," · ",{"type":46,"tag":295,"props":4400,"children":4402},{"href":4401},"references\u002Foptions-csv.md",[4403],{"type":52,"value":4404},"CSV",{"type":52,"value":4398},{"type":46,"tag":295,"props":4407,"children":4409},{"href":4408},"references\u002Foptions-xml.md",[4410],{"type":52,"value":4411},"XML",{"type":52,"value":4398},{"type":46,"tag":295,"props":4414,"children":4416},{"href":4415},"references\u002Foptions-parquet.md",[4417],{"type":52,"value":4418},"Parquet",{"type":52,"value":4398},{"type":46,"tag":295,"props":4421,"children":4423},{"href":4422},"references\u002Foptions-avro.md",[4424],{"type":52,"value":4425},"Avro",{"type":52,"value":4398},{"type":46,"tag":295,"props":4428,"children":4430},{"href":4429},"references\u002Foptions-text.md",[4431],{"type":52,"value":4432},"Text",{"type":52,"value":4398},{"type":46,"tag":295,"props":4435,"children":4437},{"href":4436},"references\u002Foptions-orc.md",[4438],{"type":52,"value":4439},"ORC",{"type":52,"value":267},{"type":46,"tag":55,"props":4442,"children":4443},{},[4444,4446,4452],{"type":52,"value":4445},"Dataset, flow, CDC, expectation, Auto Loader, and sink references are listed per (feature, language) in the ",{"type":46,"tag":295,"props":4447,"children":4449},{"href":4448},"#api-reference",[4450],{"type":52,"value":4451},"API Reference tables above",{"type":52,"value":267},{"type":46,"tag":4454,"props":4455,"children":4456},"style",{},[4457],{"type":52,"value":4458},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4460,"total":4578},[4461,4478,4492,4509,4526,4546,4557],{"slug":4462,"name":4462,"fn":4463,"description":4464,"org":4465,"tags":4466,"stars":26,"repoUrl":27,"updatedAt":4477},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4467,4470,4471,4474],{"name":4468,"slug":4469,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":4472,"slug":4473,"type":15},"Knowledge Management","knowledge-management",{"name":4475,"slug":4476,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":4479,"name":4479,"fn":4480,"description":4481,"org":4482,"tags":4483,"stars":26,"repoUrl":27,"updatedAt":4491},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4484,4487,4488],{"name":4485,"slug":4486,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":4489,"slug":4490,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":4493,"name":4493,"fn":4494,"description":4495,"org":4496,"tags":4497,"stars":26,"repoUrl":27,"updatedAt":4508},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4498,4501,4502,4505],{"name":4499,"slug":4500,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":4503,"slug":4504,"type":15},"Docker","docker",{"name":4506,"slug":4507,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":4510,"name":4510,"fn":4511,"description":4512,"org":4513,"tags":4514,"stars":26,"repoUrl":27,"updatedAt":4525},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4515,4518,4521,4524],{"name":4516,"slug":4517,"type":15},"Analytics","analytics",{"name":4519,"slug":4520,"type":15},"Dashboards","dashboards",{"name":4522,"slug":4523,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":4527,"name":4527,"fn":4528,"description":4529,"org":4530,"tags":4531,"stars":26,"repoUrl":27,"updatedAt":4545},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4532,4533,4536,4539,4542],{"name":9,"slug":8,"type":15},{"name":4534,"slug":4535,"type":15},"Design","design",{"name":4537,"slug":4538,"type":15},"Frontend","frontend",{"name":4540,"slug":4541,"type":15},"React","react",{"name":4543,"slug":4544,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":4547,"name":4547,"fn":4548,"description":4549,"org":4550,"tags":4551,"stars":26,"repoUrl":27,"updatedAt":4556},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4552,4553,4554,4555],{"name":4516,"slug":4517,"type":15},{"name":4519,"slug":4520,"type":15},{"name":4485,"slug":4486,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":4558,"name":4558,"fn":4559,"description":4560,"org":4561,"tags":4562,"stars":26,"repoUrl":27,"updatedAt":4577},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4563,4564,4567,4570,4573,4574],{"name":9,"slug":8,"type":15},{"name":4565,"slug":4566,"type":15},"FastAPI","fastapi",{"name":4568,"slug":4569,"type":15},"Flask","flask",{"name":4571,"slug":4572,"type":15},"Gradio","gradio",{"name":20,"slug":21,"type":15},{"name":4575,"slug":4576,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",31,{"items":4580,"total":4578},[4581,4588,4594,4601,4608,4616,4623,4632,4643,4660,4671,4684],{"slug":4462,"name":4462,"fn":4463,"description":4464,"org":4582,"tags":4583,"stars":26,"repoUrl":27,"updatedAt":4477},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4584,4585,4586,4587],{"name":4468,"slug":4469,"type":15},{"name":9,"slug":8,"type":15},{"name":4472,"slug":4473,"type":15},{"name":4475,"slug":4476,"type":15},{"slug":4479,"name":4479,"fn":4480,"description":4481,"org":4589,"tags":4590,"stars":26,"repoUrl":27,"updatedAt":4491},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4591,4592,4593],{"name":4485,"slug":4486,"type":15},{"name":9,"slug":8,"type":15},{"name":4489,"slug":4490,"type":15},{"slug":4493,"name":4493,"fn":4494,"description":4495,"org":4595,"tags":4596,"stars":26,"repoUrl":27,"updatedAt":4508},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4597,4598,4599,4600],{"name":4499,"slug":4500,"type":15},{"name":9,"slug":8,"type":15},{"name":4503,"slug":4504,"type":15},{"name":4506,"slug":4507,"type":15},{"slug":4510,"name":4510,"fn":4511,"description":4512,"org":4602,"tags":4603,"stars":26,"repoUrl":27,"updatedAt":4525},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4604,4605,4606,4607],{"name":4516,"slug":4517,"type":15},{"name":4519,"slug":4520,"type":15},{"name":4522,"slug":4523,"type":15},{"name":9,"slug":8,"type":15},{"slug":4527,"name":4527,"fn":4528,"description":4529,"org":4609,"tags":4610,"stars":26,"repoUrl":27,"updatedAt":4545},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4611,4612,4613,4614,4615],{"name":9,"slug":8,"type":15},{"name":4534,"slug":4535,"type":15},{"name":4537,"slug":4538,"type":15},{"name":4540,"slug":4541,"type":15},{"name":4543,"slug":4544,"type":15},{"slug":4547,"name":4547,"fn":4548,"description":4549,"org":4617,"tags":4618,"stars":26,"repoUrl":27,"updatedAt":4556},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4619,4620,4621,4622],{"name":4516,"slug":4517,"type":15},{"name":4519,"slug":4520,"type":15},{"name":4485,"slug":4486,"type":15},{"name":9,"slug":8,"type":15},{"slug":4558,"name":4558,"fn":4559,"description":4560,"org":4624,"tags":4625,"stars":26,"repoUrl":27,"updatedAt":4577},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4626,4627,4628,4629,4630,4631],{"name":9,"slug":8,"type":15},{"name":4565,"slug":4566,"type":15},{"name":4568,"slug":4569,"type":15},{"name":4571,"slug":4572,"type":15},{"name":20,"slug":21,"type":15},{"name":4575,"slug":4576,"type":15},{"slug":41,"name":41,"fn":4633,"description":4634,"org":4635,"tags":4636,"stars":26,"repoUrl":27,"updatedAt":4642},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4637,4640,4641],{"name":4638,"slug":4639,"type":15},"Authentication","authentication",{"name":4499,"slug":4500,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":4644,"name":4644,"fn":4645,"description":4646,"org":4647,"tags":4648,"stars":26,"repoUrl":27,"updatedAt":4659},"databricks-dabs","manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4649,4652,4655,4656],{"name":4650,"slug":4651,"type":15},"Automation","automation",{"name":4653,"slug":4654,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":4657,"slug":4658,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":4661,"name":4661,"fn":4662,"description":4663,"org":4664,"tags":4665,"stars":26,"repoUrl":27,"updatedAt":4670},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4666,4667,4668,4669],{"name":4485,"slug":4486,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},"2026-07-31T05:53:32.561877",{"slug":4672,"name":4672,"fn":4673,"description":4674,"org":4675,"tags":4676,"stars":26,"repoUrl":27,"updatedAt":4683},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4677,4678,4681,4682],{"name":4485,"slug":4486,"type":15},{"name":4679,"slug":4680,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},"2026-07-12T08:04:08.678282",{"slug":4685,"name":4685,"fn":4686,"description":4687,"org":4688,"tags":4689,"stars":26,"repoUrl":27,"updatedAt":4697},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4690,4691,4694],{"name":9,"slug":8,"type":15},{"name":4692,"slug":4693,"type":15},"Documentation","documentation",{"name":4695,"slug":4696,"type":15},"Reference","reference","2026-07-15T05:41:34.697746"]