[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-lakeflow-connect":3,"mdc-duslrz-key":33,"related-org-databricks-databricks-lakeflow-connect":2623,"related-repo-databricks-databricks-lakeflow-connect":2810},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"databricks-lakeflow-connect","build managed ingestion pipelines into Databricks","Build managed ingestion pipelines into Databricks using Lakeflow Connect. Use when ingesting from SaaS apps (Salesforce, Workday Reports, ServiceNow, Google Analytics 4, HubSpot, Confluence) or databases (SQL Server cloud and on-prem; PostgreSQL\u002FMySQL CDC in PuPr) into Unity Catalog with serverless pipelines.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"ETL","etl",{"name":20,"slug":21,"type":15},"Data Pipeline","data-pipeline",{"name":9,"slug":8,"type":15},204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-12T08:04:21.243004",null,60,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-lakeflow-connect","---\nname: databricks-lakeflow-connect\ndescription: \"Build managed ingestion pipelines into Databricks using Lakeflow Connect. Use when ingesting from SaaS apps (Salesforce, Workday Reports, ServiceNow, Google Analytics 4, HubSpot, Confluence) or databases (SQL Server cloud and on-prem; PostgreSQL\u002FMySQL CDC in PuPr) into Unity Catalog with serverless pipelines.\"\ncompatibility: Requires databricks CLI (>= v0.294.0)\nmetadata:\n  version: \"0.1.0\"\nparent: databricks-core\n---\n\n# Lakeflow Connect\n\nBuild managed ingestion pipelines that pull from SaaS apps and databases into Unity Catalog Delta tables, governed end-to-end and powered by serverless Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables \u002F DLT).\n\n**Status:** mixed catalog — GA connectors for production use, plus Public Preview, Beta, and Private Preview connectors that expand over time. See the connector catalog below.\n\n---\n\n## What Is Lakeflow Connect?\n\nManaged connectors for ingesting data from SaaS applications and databases. The resulting ingestion pipeline is governed by Unity Catalog and powered by serverless compute and Lakeflow Spark Declarative Pipelines.\n\nThree frames to keep in mind:\n\n- **Simple and low-maintenance** — no client code to write, no message bus to operate; connector + UC Connection + a serverless pipeline.\n- **Unified with the lakehouse** — credentials stored in UC, output is governed Delta, runs on Jobs and SDP like any other workload.\n- **Efficient incremental processing** — change tracking \u002F CDC \u002F schema evolution \u002F retries are built in.\n\nThere are four architecture patterns:\n\n1. **SaaS pull** — connector reads from an external SaaS via OAuth or API key, lands in a streaming Delta table.\n2. **Database CDC via gateway** — an ingestion gateway runs in the customer's network, stages change events to a UC Volume, a serverless ingestion pipeline applies them as CDC into Delta.\n3. **Query-based** — for sources without native CDC (Oracle \u002F Teradata \u002F SQL Server \u002F PG \u002F MySQL query-based, Snowflake \u002F Redshift \u002F Synapse \u002F BigQuery via Foreign Catalog), the connector issues periodic queries instead of subscribing to a change feed.\n4. **Community connectors** — template-based, out of scope for this skill.\n\n---\n\n## Is Lakeflow Connect the right tool?\n\nDecide this before you build. Lakeflow Connect is the managed **pull** path for SaaS apps and databases — it is not the answer for every ingestion intent.\n\n| If the source is... | Use | Skill |\n|---|---|---|\n| A SaaS app or database with a managed connector (Salesforce, Workday, ServiceNow, GA4, HubSpot, Confluence, SQL Server, ...) | **Lakeflow Connect** | this skill |\n| Files on cloud object storage (S3 \u002F ADLS \u002F GCS) | Auto Loader | **databricks-pipelines** |\n| A source you want to query in place, no copy | Lakehouse Federation | — |\n| An app or device that **pushes** events at you | Zerobus | **databricks-zerobus-ingest** |\n| A partner offering a Delta share | Delta Sharing | — |\n\nFull reasoning, including the Federation-vs-Connect and Auto-Loader-vs-Connect trade-offs, is in [4-ingestion-decision-tree.md](references\u002F4-ingestion-decision-tree.md).\n\n---\n\n## Connector catalog\n\nLakeflow Connect ships connectors at multiple release stages. **GA** and **Public Preview** connectors are production-supported; **Beta** and **Private Preview** are early-access and not production-supported.\n\n### GA connectors\n\nFull coverage in this skill.\n\n| Source | Type | Auth | Reference |\n|--------|------|------|-----------|\n| Salesforce (Sales \u002F Service \u002F etc.) | SaaS pull | OAuth U2M | [1-saas-connectors.md](references\u002F1-saas-connectors.md) |\n| Workday Reports (RaaS) | SaaS pull | OAuth refresh token \u002F basic | [1-saas-connectors.md](references\u002F1-saas-connectors.md) |\n| ServiceNow | SaaS pull | OAuth U2M \u002F basic | [1-saas-connectors.md](references\u002F1-saas-connectors.md) |\n| Google Analytics 4 | SaaS pull (via BigQuery) | Service-account JSON | [1-saas-connectors.md](references\u002F1-saas-connectors.md) |\n| HubSpot | SaaS pull | OAuth | [1-saas-connectors.md](references\u002F1-saas-connectors.md) |\n| Confluence | SaaS pull | OAuth | [1-saas-connectors.md](references\u002F1-saas-connectors.md) |\n| SQL Server (cloud) | Database CDC | DB user + change tracking \u002F CDC | [2-database-connectors.md](references\u002F2-database-connectors.md) |\n| SQL Server (on-prem) | Database CDC | DB user + ExpressRoute \u002F Direct Connect | [2-database-connectors.md](references\u002F2-database-connectors.md) |\n\n### Public Preview connectors\n\nProduction-supported. Configuration may evolve before GA. Deep coverage is being added incrementally; until then, see the [public connector reference](https:\u002F\u002Fdocs.databricks.com\u002Faws\u002Fen\u002Fingestion\u002Flakeflow-connect\u002Fconnectors) for current setup steps.\n\n| Source | Type | Auth |\n|--------|------|------|\n| NetSuite | SaaS pull | OAuth |\n| Dynamics 365 | SaaS pull | OAuth |\n| PostgreSQL CDC | Database CDC | DB user + gateway |\n| MySQL CDC | Database CDC | DB user + gateway |\n| Oracle \u002F Teradata \u002F SQL Server \u002F PG \u002F MySQL (query-based) | Database query | DB user |\n| Snowflake \u002F Redshift \u002F Synapse \u002F BigQuery (Foreign Catalog) | Database query | Foreign Catalog |\n| SFTP | File pull | Key \u002F password |\n\n### Beta and Private Preview\n\nEarly-access connectors are not production-supported. The list changes month to month; check the [public connector reference](https:\u002F\u002Fdocs.databricks.com\u002Faws\u002Fen\u002Fingestion\u002Flakeflow-connect\u002Fconnectors) for current availability.\n\nFor the Lakeflow-Connect-vs-Auto-Loader-vs-Federation-vs-Delta-Sharing decision, see [4-ingestion-decision-tree.md](references\u002F4-ingestion-decision-tree.md).\n\n---\n\n## Required Tools\n\n- **Databricks CLI v0.294.0+** for `databricks pipelines create` and `databricks connections create`. Verify with `databricks --version`.\n- **Databricks SDK for Python** (`databricks-sdk>=0.85.0`) if you prefer SDK over CLI.\n- **Declarative Automation Bundles** if authoring as IaC (recommended for any pipeline that ships to a customer environment).\n\nNo extra connector-specific SDK is needed. Lakeflow Connect reuses the pipelines API surface — pipelines are created with an `ingestion_definition` block instead of a `libraries` block, but the API and CLI are otherwise the same.\n\n---\n\n## Prerequisites\n\nConfirm before creating any pipeline:\n\n1. **A Unity Catalog target** — catalog and schema must exist; the service principal or user creating the pipeline needs `USE CATALOG`, `USE SCHEMA`, `CREATE TABLE`, and `MODIFY` on the target schema.\n2. **A UC `CONNECTION` object** with credentials for the source. SaaS OAuth U2M connections must be created via the UI (Catalog Explorer); API-key and basic-auth connections can be created via CLI \u002F DAB.\n3. **For database connectors**: network reachability between the gateway (classic compute, customer VPC) and the source database. On-prem requires ExpressRoute (Azure) or Direct Connect (AWS).\n4. **For file connectors**: OAuth scope grants on the SaaS file repo (SharePoint \u002F Google Drive).\n\n---\n\n## Minimal Example — Salesforce ingestion pipeline\n\nThe canonical authoring path is JSON to `databricks pipelines create --json`. (There is no SQL `CREATE TABLE … FROM CONNECTION` syntax for Lakeflow Connect — that syntax exists only for Lakehouse Federation, which is a different product.)\n\n```bash\ndatabricks pipelines create --json '{\n  \"name\": \"salesforce_to_uc\",\n  \"ingestion_definition\": {\n    \"connection_name\": \"my_salesforce_oauth_connection\",\n    \"objects\": [\n      {\"table\": {\"source_schema\": \"salesforce\", \"source_table\": \"Account\",\n                 \"destination_catalog\": \"main\", \"destination_schema\": \"salesforce_raw\"}},\n      {\"table\": {\"source_schema\": \"salesforce\", \"source_table\": \"Opportunity\",\n                 \"destination_catalog\": \"main\", \"destination_schema\": \"salesforce_raw\"}}\n    ]\n  }\n}'\n```\n\nFor a DAB-authored version (the production path), see [1-saas-connectors.md](references\u002F1-saas-connectors.md).\n\n---\n\n## Running the pipeline\n\nOnce authored, deploy and trigger a run. The bundle path gives the cleanest run-by-key command:\n\n```bash\ndatabricks bundle deploy -t dev\ndatabricks bundle run salesforce_ingestion           # KEY = the pipeline resource key in the bundle; waits by default\ndatabricks bundle run salesforce_ingestion --no-wait\n```\n\nA pipeline created imperatively with `pipelines create --json` has no run-by-name CLI — start and poll an update by pipeline ID instead:\n\n```bash\ndatabricks pipelines start-update \u003Cpipeline-id>             # returns an update_id\ndatabricks pipelines get-update  \u003Cpipeline-id> \u003Cupdate-id>  # poll one update's status\ndatabricks pipelines list-updates \u003Cpipeline-id>             # recent updates and their states\n```\n\nThat asymmetry is one more reason to author with a Declarative Automation Bundle.\n\n---\n\n## Detailed guides\n\n| Topic | File | When to read |\n|-------|------|--------------|\n| SaaS connectors (Salesforce, Workday Reports, ServiceNow, GA4, HubSpot, Confluence) | [1-saas-connectors.md](references\u002F1-saas-connectors.md) | Unified SaaS pattern, per-connector deltas, OAuth flows, DAB stubs |\n| Database connectors (SQL Server cloud + on-prem) | [2-database-connectors.md](references\u002F2-database-connectors.md) | Gateway pattern, change tracking vs CDC, network setup |\n| Ingestion decision tree | [4-ingestion-decision-tree.md](references\u002F4-ingestion-decision-tree.md) | Lakeflow Connect vs Auto Loader vs Lakehouse Federation vs Delta Sharing |\n| Troubleshooting and monitoring | [5-troubleshooting-and-monitoring.md](references\u002F5-troubleshooting-and-monitoring.md) | Event log queries, common errors, escalation pointers |\n\n---\n\n## Workflow\n\nFor each new ingestion pipeline:\n\n1. **Pick the connector category** — SaaS \u002F database \u002F file \u002F push — and read the matching reference file.\n2. **Verify prerequisites** — UC target, source credentials, network path (for databases), region availability.\n3. **Create the UC `CONNECTION`** — UI for OAuth U2M, CLI \u002F DAB for everything else.\n4. **Author the pipeline** — `databricks pipelines create --json` for one-offs, DAB YAML for anything shipping to a customer.\n5. **Trigger the first run** and watch the event log; see [5-troubleshooting-and-monitoring.md](references\u002F5-troubleshooting-and-monitoring.md) for the SQL.\n6. **Schedule** the triggered pipeline with a Jobs `pipeline_task` (cron or interval). Lakeflow Connect supports triggered runs only — `continuous: false` selects triggered mode but is not itself a schedule, so the cadence comes from the Jobs trigger.\n\n---\n\n## Anti-patterns\n\nThree forms that look plausible but fail — wrong vs. right:\n\n**1. `CREATE TABLE ... FROM CONNECTION` is Lakehouse Federation, not Lakeflow Connect.**\n\n```sql\n-- WRONG: Federation syntax; no LFC equivalent exists\nCREATE TABLE main.salesforce_raw.account FROM CONNECTION my_salesforce_conn;\n```\n```json\n\u002F\u002F RIGHT: author an ingestion_definition (see the Minimal Example above)\n{\"ingestion_definition\": {\"connection_name\": \"my_salesforce_conn\", \"objects\": [\u002F* ... *\u002F]}}\n```\n\n**2. An ingestion pipeline carries `ingestion_definition`, never a `libraries` block.**\n\n```json\n\u002F\u002F WRONG: libraries is for a standard SDP pipeline running your notebooks\u002Ffiles\n{\"name\": \"salesforce_to_uc\", \"libraries\": [{\"notebook\": {\"path\": \"\u002FRepos\u002F...\u002Fingest\"}}]}\n\u002F\u002F RIGHT:\n{\"name\": \"salesforce_to_uc\", \"ingestion_definition\": {\"connection_name\": \"...\", \"objects\": []}}\n```\n\n**3. `continuous: true` is rejected — Lakeflow Connect is triggered-only.**\n\n```json\n\u002F\u002F WRONG: continuous mode fails at create\n{\"continuous\": true, \"ingestion_definition\": {\u002F* ... *\u002F}}\n\u002F\u002F RIGHT: continuous:false (or omit) + schedule with a Jobs pipeline_task\n{\"continuous\": false, \"ingestion_definition\": {\u002F* ... *\u002F}}\n```\n\n---\n\n## Important\n\n- **Triggered only, no continuous mode** — pipelines run on a schedule or on-demand, never continuously. Check the connector reference for the latest status.\n- **Compute-only billing** — Lakeflow Connect is billed in DBUs (no per-row fee). Database connectors also incur classic-compute gateway DBUs in addition to the serverless ingestion pipeline DBUs. See the [pricing page](https:\u002F\u002Fwww.databricks.com\u002Fproduct\u002Fpricing\u002Flakeflow-connect) for current rates.\n- **Salesforce auth is OAuth U2M only** — no machine-to-machine, no basic auth. Connection creation requires a UI walk-through.\n- **Database staging retention is 30 days** by default in the UC Volume between the gateway and the ingestion pipeline.\n- **Limits per pipeline** — most SaaS connectors cap at 250 tables per pipeline. Split across multiple pipelines if needed.\n- **This lands raw tables** — Lakeflow Connect writes source-faithful tables (the ingestion landing zone). Build the medallion Bronze\u002FSilver\u002FGold transforms on top of them with **databricks-pipelines**.\n\n---\n\n## Key Concepts\n\n- **UC `CONNECTION` is the credential anchor** — every Lakeflow Connect pipeline points at a UC connection. The connection owns the auth; the pipeline references it by name.\n- **Serverless ingestion pipeline + (optional) classic gateway** — SaaS connectors are pure serverless. Database connectors split into a customer-network gateway (classic) and a serverless ingestion pipeline (Delta-bound).\n- **CDC and schema evolution are built in** — for sources that support change tracking or CDC, the connector applies changes incrementally and evolves the target schema. Data-type changes typically require a full snapshot reload.\n- **Streaming Delta output** — destination tables are governed Delta tables; CDC sources are applied with change semantics (`APPLY CHANGES` \u002F AUTO CDC, or `apply_changes_from_snapshot` for snapshot sources). Compatible with downstream materialized views and Spark streaming.\n- **OAuth U2M is UI-only** — DAB \u002F CLI cannot bootstrap OAuth U2M connections. Hand the one-time browser step to a human (Catalog Explorer > External Data > Connections > Create connection > pick the source > sign in), then resume once `databricks connections get \u003Cconnection_name>` reports `READY`.\n\n---\n\n## Common Issues\n\nFor common errors and their fixes — duplicate-key violations, watermark \u002F cursor problems, schema evolution, gateway region availability, the `channel` runtime-channel setting, and pipelines that run but land no data — see [5-troubleshooting-and-monitoring.md](references\u002F5-troubleshooting-and-monitoring.md), which also has the event-log queries to diagnose them.\n\n---\n\n## Related Skills\n\n- **databricks-pipelines** — the SDP runtime that Lakeflow Connect pipelines run on. For Auto Loader and downstream pipeline patterns.\n- **databricks-zerobus-ingest** — push-based gRPC ingestion. Sibling to Lakeflow Connect's pull-based connectors.\n- **databricks-dabs** — author Lakeflow Connect pipelines as IaC.\n- **databricks-unity-catalog** — managing catalogs, schemas, and the UC `CONNECTION` objects that LFC credentials live in.\n- **databricks-jobs** — schedule ingestion pipelines with `pipeline_task`.\n\n---\n\n## Resources\n\n- [Lakeflow Connect public docs hub](https:\u002F\u002Fdocs.databricks.com\u002Faws\u002Fen\u002Fingestion\u002Flakeflow-connect)\n- [Connector reference (per-connector setup)](https:\u002F\u002Fdocs.databricks.com\u002Faws\u002Fen\u002Fingestion\u002Flakeflow-connect\u002Fconnectors)\n- [Pricing](https:\u002F\u002Fwww.databricks.com\u002Fproduct\u002Fpricing\u002Flakeflow-connect)\n",{"data":34,"body":39},{"name":4,"description":6,"compatibility":35,"metadata":36,"parent":38},"Requires databricks CLI (>= v0.294.0)",{"version":37},"0.1.0","databricks-core",{"type":40,"children":41},"root",[42,51,57,68,72,79,84,89,124,129,173,176,182,194,331,345,348,354,386,393,398,627,633,648,790,796,808,818,821,827,891,912,915,921,926,1008,1011,1017,1038,1191,1201,1204,1210,1215,1300,1313,1454,1459,1462,1468,1577,1580,1586,1591,1688,1691,1697,1702,1718,1743,1860,1882,2137,2153,2290,2293,2299,2376,2379,2385,2476,2479,2485,2504,2507,2513,2577,2580,2586,2617],{"type":43,"tag":44,"props":45,"children":47},"element","h1",{"id":46},"lakeflow-connect",[48],{"type":49,"value":50},"text","Lakeflow Connect",{"type":43,"tag":52,"props":53,"children":54},"p",{},[55],{"type":49,"value":56},"Build managed ingestion pipelines that pull from SaaS apps and databases into Unity Catalog Delta tables, governed end-to-end and powered by serverless Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables \u002F DLT).",{"type":43,"tag":52,"props":58,"children":59},{},[60,66],{"type":43,"tag":61,"props":62,"children":63},"strong",{},[64],{"type":49,"value":65},"Status:",{"type":49,"value":67}," mixed catalog — GA connectors for production use, plus Public Preview, Beta, and Private Preview connectors that expand over time. See the connector catalog below.",{"type":43,"tag":69,"props":70,"children":71},"hr",{},[],{"type":43,"tag":73,"props":74,"children":76},"h2",{"id":75},"what-is-lakeflow-connect",[77],{"type":49,"value":78},"What Is Lakeflow Connect?",{"type":43,"tag":52,"props":80,"children":81},{},[82],{"type":49,"value":83},"Managed connectors for ingesting data from SaaS applications and databases. The resulting ingestion pipeline is governed by Unity Catalog and powered by serverless compute and Lakeflow Spark Declarative Pipelines.",{"type":43,"tag":52,"props":85,"children":86},{},[87],{"type":49,"value":88},"Three frames to keep in mind:",{"type":43,"tag":90,"props":91,"children":92},"ul",{},[93,104,114],{"type":43,"tag":94,"props":95,"children":96},"li",{},[97,102],{"type":43,"tag":61,"props":98,"children":99},{},[100],{"type":49,"value":101},"Simple and low-maintenance",{"type":49,"value":103}," — no client code to write, no message bus to operate; connector + UC Connection + a serverless pipeline.",{"type":43,"tag":94,"props":105,"children":106},{},[107,112],{"type":43,"tag":61,"props":108,"children":109},{},[110],{"type":49,"value":111},"Unified with the lakehouse",{"type":49,"value":113}," — credentials stored in UC, output is governed Delta, runs on Jobs and SDP like any other workload.",{"type":43,"tag":94,"props":115,"children":116},{},[117,122],{"type":43,"tag":61,"props":118,"children":119},{},[120],{"type":49,"value":121},"Efficient incremental processing",{"type":49,"value":123}," — change tracking \u002F CDC \u002F schema evolution \u002F retries are built in.",{"type":43,"tag":52,"props":125,"children":126},{},[127],{"type":49,"value":128},"There are four architecture patterns:",{"type":43,"tag":130,"props":131,"children":132},"ol",{},[133,143,153,163],{"type":43,"tag":94,"props":134,"children":135},{},[136,141],{"type":43,"tag":61,"props":137,"children":138},{},[139],{"type":49,"value":140},"SaaS pull",{"type":49,"value":142}," — connector reads from an external SaaS via OAuth or API key, lands in a streaming Delta table.",{"type":43,"tag":94,"props":144,"children":145},{},[146,151],{"type":43,"tag":61,"props":147,"children":148},{},[149],{"type":49,"value":150},"Database CDC via gateway",{"type":49,"value":152}," — an ingestion gateway runs in the customer's network, stages change events to a UC Volume, a serverless ingestion pipeline applies them as CDC into Delta.",{"type":43,"tag":94,"props":154,"children":155},{},[156,161],{"type":43,"tag":61,"props":157,"children":158},{},[159],{"type":49,"value":160},"Query-based",{"type":49,"value":162}," — for sources without native CDC (Oracle \u002F Teradata \u002F SQL Server \u002F PG \u002F MySQL query-based, Snowflake \u002F Redshift \u002F Synapse \u002F BigQuery via Foreign Catalog), the connector issues periodic queries instead of subscribing to a change feed.",{"type":43,"tag":94,"props":164,"children":165},{},[166,171],{"type":43,"tag":61,"props":167,"children":168},{},[169],{"type":49,"value":170},"Community connectors",{"type":49,"value":172}," — template-based, out of scope for this skill.",{"type":43,"tag":69,"props":174,"children":175},{},[],{"type":43,"tag":73,"props":177,"children":179},{"id":178},"is-lakeflow-connect-the-right-tool",[180],{"type":49,"value":181},"Is Lakeflow Connect the right tool?",{"type":43,"tag":52,"props":183,"children":184},{},[185,187,192],{"type":49,"value":186},"Decide this before you build. Lakeflow Connect is the managed ",{"type":43,"tag":61,"props":188,"children":189},{},[190],{"type":49,"value":191},"pull",{"type":49,"value":193}," path for SaaS apps and databases — it is not the answer for every ingestion intent.",{"type":43,"tag":195,"props":196,"children":197},"table",{},[198,222],{"type":43,"tag":199,"props":200,"children":201},"thead",{},[202],{"type":43,"tag":203,"props":204,"children":205},"tr",{},[206,212,217],{"type":43,"tag":207,"props":208,"children":209},"th",{},[210],{"type":49,"value":211},"If the source is...",{"type":43,"tag":207,"props":213,"children":214},{},[215],{"type":49,"value":216},"Use",{"type":43,"tag":207,"props":218,"children":219},{},[220],{"type":49,"value":221},"Skill",{"type":43,"tag":223,"props":224,"children":225},"tbody",{},[226,247,268,286,314],{"type":43,"tag":203,"props":227,"children":228},{},[229,235,242],{"type":43,"tag":230,"props":231,"children":232},"td",{},[233],{"type":49,"value":234},"A SaaS app or database with a managed connector (Salesforce, Workday, ServiceNow, GA4, HubSpot, Confluence, SQL Server, ...)",{"type":43,"tag":230,"props":236,"children":237},{},[238],{"type":43,"tag":61,"props":239,"children":240},{},[241],{"type":49,"value":50},{"type":43,"tag":230,"props":243,"children":244},{},[245],{"type":49,"value":246},"this skill",{"type":43,"tag":203,"props":248,"children":249},{},[250,255,260],{"type":43,"tag":230,"props":251,"children":252},{},[253],{"type":49,"value":254},"Files on cloud object storage (S3 \u002F ADLS \u002F GCS)",{"type":43,"tag":230,"props":256,"children":257},{},[258],{"type":49,"value":259},"Auto Loader",{"type":43,"tag":230,"props":261,"children":262},{},[263],{"type":43,"tag":61,"props":264,"children":265},{},[266],{"type":49,"value":267},"databricks-pipelines",{"type":43,"tag":203,"props":269,"children":270},{},[271,276,281],{"type":43,"tag":230,"props":272,"children":273},{},[274],{"type":49,"value":275},"A source you want to query in place, no copy",{"type":43,"tag":230,"props":277,"children":278},{},[279],{"type":49,"value":280},"Lakehouse Federation",{"type":43,"tag":230,"props":282,"children":283},{},[284],{"type":49,"value":285},"—",{"type":43,"tag":203,"props":287,"children":288},{},[289,301,306],{"type":43,"tag":230,"props":290,"children":291},{},[292,294,299],{"type":49,"value":293},"An app or device that ",{"type":43,"tag":61,"props":295,"children":296},{},[297],{"type":49,"value":298},"pushes",{"type":49,"value":300}," events at you",{"type":43,"tag":230,"props":302,"children":303},{},[304],{"type":49,"value":305},"Zerobus",{"type":43,"tag":230,"props":307,"children":308},{},[309],{"type":43,"tag":61,"props":310,"children":311},{},[312],{"type":49,"value":313},"databricks-zerobus-ingest",{"type":43,"tag":203,"props":315,"children":316},{},[317,322,327],{"type":43,"tag":230,"props":318,"children":319},{},[320],{"type":49,"value":321},"A partner offering a Delta share",{"type":43,"tag":230,"props":323,"children":324},{},[325],{"type":49,"value":326},"Delta Sharing",{"type":43,"tag":230,"props":328,"children":329},{},[330],{"type":49,"value":285},{"type":43,"tag":52,"props":332,"children":333},{},[334,336,343],{"type":49,"value":335},"Full reasoning, including the Federation-vs-Connect and Auto-Loader-vs-Connect trade-offs, is in ",{"type":43,"tag":337,"props":338,"children":340},"a",{"href":339},"references\u002F4-ingestion-decision-tree.md",[341],{"type":49,"value":342},"4-ingestion-decision-tree.md",{"type":49,"value":344},".",{"type":43,"tag":69,"props":346,"children":347},{},[],{"type":43,"tag":73,"props":349,"children":351},{"id":350},"connector-catalog",[352],{"type":49,"value":353},"Connector catalog",{"type":43,"tag":52,"props":355,"children":356},{},[357,359,364,366,371,373,378,379,384],{"type":49,"value":358},"Lakeflow Connect ships connectors at multiple release stages. ",{"type":43,"tag":61,"props":360,"children":361},{},[362],{"type":49,"value":363},"GA",{"type":49,"value":365}," and ",{"type":43,"tag":61,"props":367,"children":368},{},[369],{"type":49,"value":370},"Public Preview",{"type":49,"value":372}," connectors are production-supported; ",{"type":43,"tag":61,"props":374,"children":375},{},[376],{"type":49,"value":377},"Beta",{"type":49,"value":365},{"type":43,"tag":61,"props":380,"children":381},{},[382],{"type":49,"value":383},"Private Preview",{"type":49,"value":385}," are early-access and not production-supported.",{"type":43,"tag":387,"props":388,"children":390},"h3",{"id":389},"ga-connectors",[391],{"type":49,"value":392},"GA connectors",{"type":43,"tag":52,"props":394,"children":395},{},[396],{"type":49,"value":397},"Full coverage in this skill.",{"type":43,"tag":195,"props":399,"children":400},{},[401,427],{"type":43,"tag":199,"props":402,"children":403},{},[404],{"type":43,"tag":203,"props":405,"children":406},{},[407,412,417,422],{"type":43,"tag":207,"props":408,"children":409},{},[410],{"type":49,"value":411},"Source",{"type":43,"tag":207,"props":413,"children":414},{},[415],{"type":49,"value":416},"Type",{"type":43,"tag":207,"props":418,"children":419},{},[420],{"type":49,"value":421},"Auth",{"type":43,"tag":207,"props":423,"children":424},{},[425],{"type":49,"value":426},"Reference",{"type":43,"tag":223,"props":428,"children":429},{},[430,456,480,504,529,553,576,603],{"type":43,"tag":203,"props":431,"children":432},{},[433,438,442,447],{"type":43,"tag":230,"props":434,"children":435},{},[436],{"type":49,"value":437},"Salesforce (Sales \u002F Service \u002F etc.)",{"type":43,"tag":230,"props":439,"children":440},{},[441],{"type":49,"value":140},{"type":43,"tag":230,"props":443,"children":444},{},[445],{"type":49,"value":446},"OAuth U2M",{"type":43,"tag":230,"props":448,"children":449},{},[450],{"type":43,"tag":337,"props":451,"children":453},{"href":452},"references\u002F1-saas-connectors.md",[454],{"type":49,"value":455},"1-saas-connectors.md",{"type":43,"tag":203,"props":457,"children":458},{},[459,464,468,473],{"type":43,"tag":230,"props":460,"children":461},{},[462],{"type":49,"value":463},"Workday Reports (RaaS)",{"type":43,"tag":230,"props":465,"children":466},{},[467],{"type":49,"value":140},{"type":43,"tag":230,"props":469,"children":470},{},[471],{"type":49,"value":472},"OAuth refresh token \u002F basic",{"type":43,"tag":230,"props":474,"children":475},{},[476],{"type":43,"tag":337,"props":477,"children":478},{"href":452},[479],{"type":49,"value":455},{"type":43,"tag":203,"props":481,"children":482},{},[483,488,492,497],{"type":43,"tag":230,"props":484,"children":485},{},[486],{"type":49,"value":487},"ServiceNow",{"type":43,"tag":230,"props":489,"children":490},{},[491],{"type":49,"value":140},{"type":43,"tag":230,"props":493,"children":494},{},[495],{"type":49,"value":496},"OAuth U2M \u002F basic",{"type":43,"tag":230,"props":498,"children":499},{},[500],{"type":43,"tag":337,"props":501,"children":502},{"href":452},[503],{"type":49,"value":455},{"type":43,"tag":203,"props":505,"children":506},{},[507,512,517,522],{"type":43,"tag":230,"props":508,"children":509},{},[510],{"type":49,"value":511},"Google Analytics 4",{"type":43,"tag":230,"props":513,"children":514},{},[515],{"type":49,"value":516},"SaaS pull (via BigQuery)",{"type":43,"tag":230,"props":518,"children":519},{},[520],{"type":49,"value":521},"Service-account JSON",{"type":43,"tag":230,"props":523,"children":524},{},[525],{"type":43,"tag":337,"props":526,"children":527},{"href":452},[528],{"type":49,"value":455},{"type":43,"tag":203,"props":530,"children":531},{},[532,537,541,546],{"type":43,"tag":230,"props":533,"children":534},{},[535],{"type":49,"value":536},"HubSpot",{"type":43,"tag":230,"props":538,"children":539},{},[540],{"type":49,"value":140},{"type":43,"tag":230,"props":542,"children":543},{},[544],{"type":49,"value":545},"OAuth",{"type":43,"tag":230,"props":547,"children":548},{},[549],{"type":43,"tag":337,"props":550,"children":551},{"href":452},[552],{"type":49,"value":455},{"type":43,"tag":203,"props":554,"children":555},{},[556,561,565,569],{"type":43,"tag":230,"props":557,"children":558},{},[559],{"type":49,"value":560},"Confluence",{"type":43,"tag":230,"props":562,"children":563},{},[564],{"type":49,"value":140},{"type":43,"tag":230,"props":566,"children":567},{},[568],{"type":49,"value":545},{"type":43,"tag":230,"props":570,"children":571},{},[572],{"type":43,"tag":337,"props":573,"children":574},{"href":452},[575],{"type":49,"value":455},{"type":43,"tag":203,"props":577,"children":578},{},[579,584,589,594],{"type":43,"tag":230,"props":580,"children":581},{},[582],{"type":49,"value":583},"SQL Server (cloud)",{"type":43,"tag":230,"props":585,"children":586},{},[587],{"type":49,"value":588},"Database CDC",{"type":43,"tag":230,"props":590,"children":591},{},[592],{"type":49,"value":593},"DB user + change tracking \u002F CDC",{"type":43,"tag":230,"props":595,"children":596},{},[597],{"type":43,"tag":337,"props":598,"children":600},{"href":599},"references\u002F2-database-connectors.md",[601],{"type":49,"value":602},"2-database-connectors.md",{"type":43,"tag":203,"props":604,"children":605},{},[606,611,615,620],{"type":43,"tag":230,"props":607,"children":608},{},[609],{"type":49,"value":610},"SQL Server (on-prem)",{"type":43,"tag":230,"props":612,"children":613},{},[614],{"type":49,"value":588},{"type":43,"tag":230,"props":616,"children":617},{},[618],{"type":49,"value":619},"DB user + ExpressRoute \u002F Direct Connect",{"type":43,"tag":230,"props":621,"children":622},{},[623],{"type":43,"tag":337,"props":624,"children":625},{"href":599},[626],{"type":49,"value":602},{"type":43,"tag":387,"props":628,"children":630},{"id":629},"public-preview-connectors",[631],{"type":49,"value":632},"Public Preview connectors",{"type":43,"tag":52,"props":634,"children":635},{},[636,638,646],{"type":49,"value":637},"Production-supported. Configuration may evolve before GA. Deep coverage is being added incrementally; until then, see the ",{"type":43,"tag":337,"props":639,"children":643},{"href":640,"rel":641},"https:\u002F\u002Fdocs.databricks.com\u002Faws\u002Fen\u002Fingestion\u002Flakeflow-connect\u002Fconnectors",[642],"nofollow",[644],{"type":49,"value":645},"public connector reference",{"type":49,"value":647}," for current setup steps.",{"type":43,"tag":195,"props":649,"children":650},{},[651,669],{"type":43,"tag":199,"props":652,"children":653},{},[654],{"type":43,"tag":203,"props":655,"children":656},{},[657,661,665],{"type":43,"tag":207,"props":658,"children":659},{},[660],{"type":49,"value":411},{"type":43,"tag":207,"props":662,"children":663},{},[664],{"type":49,"value":416},{"type":43,"tag":207,"props":666,"children":667},{},[668],{"type":49,"value":421},{"type":43,"tag":223,"props":670,"children":671},{},[672,688,704,721,737,755,772],{"type":43,"tag":203,"props":673,"children":674},{},[675,680,684],{"type":43,"tag":230,"props":676,"children":677},{},[678],{"type":49,"value":679},"NetSuite",{"type":43,"tag":230,"props":681,"children":682},{},[683],{"type":49,"value":140},{"type":43,"tag":230,"props":685,"children":686},{},[687],{"type":49,"value":545},{"type":43,"tag":203,"props":689,"children":690},{},[691,696,700],{"type":43,"tag":230,"props":692,"children":693},{},[694],{"type":49,"value":695},"Dynamics 365",{"type":43,"tag":230,"props":697,"children":698},{},[699],{"type":49,"value":140},{"type":43,"tag":230,"props":701,"children":702},{},[703],{"type":49,"value":545},{"type":43,"tag":203,"props":705,"children":706},{},[707,712,716],{"type":43,"tag":230,"props":708,"children":709},{},[710],{"type":49,"value":711},"PostgreSQL CDC",{"type":43,"tag":230,"props":713,"children":714},{},[715],{"type":49,"value":588},{"type":43,"tag":230,"props":717,"children":718},{},[719],{"type":49,"value":720},"DB user + gateway",{"type":43,"tag":203,"props":722,"children":723},{},[724,729,733],{"type":43,"tag":230,"props":725,"children":726},{},[727],{"type":49,"value":728},"MySQL CDC",{"type":43,"tag":230,"props":730,"children":731},{},[732],{"type":49,"value":588},{"type":43,"tag":230,"props":734,"children":735},{},[736],{"type":49,"value":720},{"type":43,"tag":203,"props":738,"children":739},{},[740,745,750],{"type":43,"tag":230,"props":741,"children":742},{},[743],{"type":49,"value":744},"Oracle \u002F Teradata \u002F SQL Server \u002F PG \u002F MySQL (query-based)",{"type":43,"tag":230,"props":746,"children":747},{},[748],{"type":49,"value":749},"Database query",{"type":43,"tag":230,"props":751,"children":752},{},[753],{"type":49,"value":754},"DB user",{"type":43,"tag":203,"props":756,"children":757},{},[758,763,767],{"type":43,"tag":230,"props":759,"children":760},{},[761],{"type":49,"value":762},"Snowflake \u002F Redshift \u002F Synapse \u002F BigQuery (Foreign Catalog)",{"type":43,"tag":230,"props":764,"children":765},{},[766],{"type":49,"value":749},{"type":43,"tag":230,"props":768,"children":769},{},[770],{"type":49,"value":771},"Foreign Catalog",{"type":43,"tag":203,"props":773,"children":774},{},[775,780,785],{"type":43,"tag":230,"props":776,"children":777},{},[778],{"type":49,"value":779},"SFTP",{"type":43,"tag":230,"props":781,"children":782},{},[783],{"type":49,"value":784},"File pull",{"type":43,"tag":230,"props":786,"children":787},{},[788],{"type":49,"value":789},"Key \u002F password",{"type":43,"tag":387,"props":791,"children":793},{"id":792},"beta-and-private-preview",[794],{"type":49,"value":795},"Beta and Private Preview",{"type":43,"tag":52,"props":797,"children":798},{},[799,801,806],{"type":49,"value":800},"Early-access connectors are not production-supported. The list changes month to month; check the ",{"type":43,"tag":337,"props":802,"children":804},{"href":640,"rel":803},[642],[805],{"type":49,"value":645},{"type":49,"value":807}," for current availability.",{"type":43,"tag":52,"props":809,"children":810},{},[811,813,817],{"type":49,"value":812},"For the Lakeflow-Connect-vs-Auto-Loader-vs-Federation-vs-Delta-Sharing decision, see ",{"type":43,"tag":337,"props":814,"children":815},{"href":339},[816],{"type":49,"value":342},{"type":49,"value":344},{"type":43,"tag":69,"props":819,"children":820},{},[],{"type":43,"tag":73,"props":822,"children":824},{"id":823},"required-tools",[825],{"type":49,"value":826},"Required Tools",{"type":43,"tag":90,"props":828,"children":829},{},[830,863,881],{"type":43,"tag":94,"props":831,"children":832},{},[833,838,840,847,848,854,856,862],{"type":43,"tag":61,"props":834,"children":835},{},[836],{"type":49,"value":837},"Databricks CLI v0.294.0+",{"type":49,"value":839}," for ",{"type":43,"tag":841,"props":842,"children":844},"code",{"className":843},[],[845],{"type":49,"value":846},"databricks pipelines create",{"type":49,"value":365},{"type":43,"tag":841,"props":849,"children":851},{"className":850},[],[852],{"type":49,"value":853},"databricks connections create",{"type":49,"value":855},". Verify with ",{"type":43,"tag":841,"props":857,"children":859},{"className":858},[],[860],{"type":49,"value":861},"databricks --version",{"type":49,"value":344},{"type":43,"tag":94,"props":864,"children":865},{},[866,871,873,879],{"type":43,"tag":61,"props":867,"children":868},{},[869],{"type":49,"value":870},"Databricks SDK for Python",{"type":49,"value":872}," (",{"type":43,"tag":841,"props":874,"children":876},{"className":875},[],[877],{"type":49,"value":878},"databricks-sdk>=0.85.0",{"type":49,"value":880},") if you prefer SDK over CLI.",{"type":43,"tag":94,"props":882,"children":883},{},[884,889],{"type":43,"tag":61,"props":885,"children":886},{},[887],{"type":49,"value":888},"Declarative Automation Bundles",{"type":49,"value":890}," if authoring as IaC (recommended for any pipeline that ships to a customer environment).",{"type":43,"tag":52,"props":892,"children":893},{},[894,896,902,904,910],{"type":49,"value":895},"No extra connector-specific SDK is needed. Lakeflow Connect reuses the pipelines API surface — pipelines are created with an ",{"type":43,"tag":841,"props":897,"children":899},{"className":898},[],[900],{"type":49,"value":901},"ingestion_definition",{"type":49,"value":903}," block instead of a ",{"type":43,"tag":841,"props":905,"children":907},{"className":906},[],[908],{"type":49,"value":909},"libraries",{"type":49,"value":911}," block, but the API and CLI are otherwise the same.",{"type":43,"tag":69,"props":913,"children":914},{},[],{"type":43,"tag":73,"props":916,"children":918},{"id":917},"prerequisites",[919],{"type":49,"value":920},"Prerequisites",{"type":43,"tag":52,"props":922,"children":923},{},[924],{"type":49,"value":925},"Confirm before creating any pipeline:",{"type":43,"tag":130,"props":927,"children":928},{},[929,970,988,998],{"type":43,"tag":94,"props":930,"children":931},{},[932,937,939,945,947,953,954,960,962,968],{"type":43,"tag":61,"props":933,"children":934},{},[935],{"type":49,"value":936},"A Unity Catalog target",{"type":49,"value":938}," — catalog and schema must exist; the service principal or user creating the pipeline needs ",{"type":43,"tag":841,"props":940,"children":942},{"className":941},[],[943],{"type":49,"value":944},"USE CATALOG",{"type":49,"value":946},", ",{"type":43,"tag":841,"props":948,"children":950},{"className":949},[],[951],{"type":49,"value":952},"USE SCHEMA",{"type":49,"value":946},{"type":43,"tag":841,"props":955,"children":957},{"className":956},[],[958],{"type":49,"value":959},"CREATE TABLE",{"type":49,"value":961},", and ",{"type":43,"tag":841,"props":963,"children":965},{"className":964},[],[966],{"type":49,"value":967},"MODIFY",{"type":49,"value":969}," on the target schema.",{"type":43,"tag":94,"props":971,"children":972},{},[973,986],{"type":43,"tag":61,"props":974,"children":975},{},[976,978,984],{"type":49,"value":977},"A UC ",{"type":43,"tag":841,"props":979,"children":981},{"className":980},[],[982],{"type":49,"value":983},"CONNECTION",{"type":49,"value":985}," object",{"type":49,"value":987}," with credentials for the source. SaaS OAuth U2M connections must be created via the UI (Catalog Explorer); API-key and basic-auth connections can be created via CLI \u002F DAB.",{"type":43,"tag":94,"props":989,"children":990},{},[991,996],{"type":43,"tag":61,"props":992,"children":993},{},[994],{"type":49,"value":995},"For database connectors",{"type":49,"value":997},": network reachability between the gateway (classic compute, customer VPC) and the source database. On-prem requires ExpressRoute (Azure) or Direct Connect (AWS).",{"type":43,"tag":94,"props":999,"children":1000},{},[1001,1006],{"type":43,"tag":61,"props":1002,"children":1003},{},[1004],{"type":49,"value":1005},"For file connectors",{"type":49,"value":1007},": OAuth scope grants on the SaaS file repo (SharePoint \u002F Google Drive).",{"type":43,"tag":69,"props":1009,"children":1010},{},[],{"type":43,"tag":73,"props":1012,"children":1014},{"id":1013},"minimal-example-salesforce-ingestion-pipeline",[1015],{"type":49,"value":1016},"Minimal Example — Salesforce ingestion pipeline",{"type":43,"tag":52,"props":1018,"children":1019},{},[1020,1022,1028,1030,1036],{"type":49,"value":1021},"The canonical authoring path is JSON to ",{"type":43,"tag":841,"props":1023,"children":1025},{"className":1024},[],[1026],{"type":49,"value":1027},"databricks pipelines create --json",{"type":49,"value":1029},". (There is no SQL ",{"type":43,"tag":841,"props":1031,"children":1033},{"className":1032},[],[1034],{"type":49,"value":1035},"CREATE TABLE … FROM CONNECTION",{"type":49,"value":1037}," syntax for Lakeflow Connect — that syntax exists only for Lakehouse Federation, which is a different product.)",{"type":43,"tag":1039,"props":1040,"children":1045},"pre",{"className":1041,"code":1042,"language":1043,"meta":1044,"style":1044},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","databricks pipelines create --json '{\n  \"name\": \"salesforce_to_uc\",\n  \"ingestion_definition\": {\n    \"connection_name\": \"my_salesforce_oauth_connection\",\n    \"objects\": [\n      {\"table\": {\"source_schema\": \"salesforce\", \"source_table\": \"Account\",\n                 \"destination_catalog\": \"main\", \"destination_schema\": \"salesforce_raw\"}},\n      {\"table\": {\"source_schema\": \"salesforce\", \"source_table\": \"Opportunity\",\n                 \"destination_catalog\": \"main\", \"destination_schema\": \"salesforce_raw\"}}\n    ]\n  }\n}'\n","bash","",[1046],{"type":43,"tag":841,"props":1047,"children":1048},{"__ignoreMap":1044},[1049,1087,1096,1105,1114,1123,1132,1141,1150,1159,1168,1177],{"type":43,"tag":1050,"props":1051,"children":1054},"span",{"class":1052,"line":1053},"line",1,[1055,1060,1066,1071,1076,1082],{"type":43,"tag":1050,"props":1056,"children":1058},{"style":1057},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1059],{"type":49,"value":8},{"type":43,"tag":1050,"props":1061,"children":1063},{"style":1062},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1064],{"type":49,"value":1065}," pipelines",{"type":43,"tag":1050,"props":1067,"children":1068},{"style":1062},[1069],{"type":49,"value":1070}," create",{"type":43,"tag":1050,"props":1072,"children":1073},{"style":1062},[1074],{"type":49,"value":1075}," --json",{"type":43,"tag":1050,"props":1077,"children":1079},{"style":1078},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1080],{"type":49,"value":1081}," '",{"type":43,"tag":1050,"props":1083,"children":1084},{"style":1062},[1085],{"type":49,"value":1086},"{\n",{"type":43,"tag":1050,"props":1088,"children":1090},{"class":1052,"line":1089},2,[1091],{"type":43,"tag":1050,"props":1092,"children":1093},{"style":1062},[1094],{"type":49,"value":1095},"  \"name\": \"salesforce_to_uc\",\n",{"type":43,"tag":1050,"props":1097,"children":1099},{"class":1052,"line":1098},3,[1100],{"type":43,"tag":1050,"props":1101,"children":1102},{"style":1062},[1103],{"type":49,"value":1104},"  \"ingestion_definition\": {\n",{"type":43,"tag":1050,"props":1106,"children":1108},{"class":1052,"line":1107},4,[1109],{"type":43,"tag":1050,"props":1110,"children":1111},{"style":1062},[1112],{"type":49,"value":1113},"    \"connection_name\": \"my_salesforce_oauth_connection\",\n",{"type":43,"tag":1050,"props":1115,"children":1117},{"class":1052,"line":1116},5,[1118],{"type":43,"tag":1050,"props":1119,"children":1120},{"style":1062},[1121],{"type":49,"value":1122},"    \"objects\": [\n",{"type":43,"tag":1050,"props":1124,"children":1126},{"class":1052,"line":1125},6,[1127],{"type":43,"tag":1050,"props":1128,"children":1129},{"style":1062},[1130],{"type":49,"value":1131},"      {\"table\": {\"source_schema\": \"salesforce\", \"source_table\": \"Account\",\n",{"type":43,"tag":1050,"props":1133,"children":1135},{"class":1052,"line":1134},7,[1136],{"type":43,"tag":1050,"props":1137,"children":1138},{"style":1062},[1139],{"type":49,"value":1140},"                 \"destination_catalog\": \"main\", \"destination_schema\": \"salesforce_raw\"}},\n",{"type":43,"tag":1050,"props":1142,"children":1144},{"class":1052,"line":1143},8,[1145],{"type":43,"tag":1050,"props":1146,"children":1147},{"style":1062},[1148],{"type":49,"value":1149},"      {\"table\": {\"source_schema\": \"salesforce\", \"source_table\": \"Opportunity\",\n",{"type":43,"tag":1050,"props":1151,"children":1153},{"class":1052,"line":1152},9,[1154],{"type":43,"tag":1050,"props":1155,"children":1156},{"style":1062},[1157],{"type":49,"value":1158},"                 \"destination_catalog\": \"main\", \"destination_schema\": \"salesforce_raw\"}}\n",{"type":43,"tag":1050,"props":1160,"children":1162},{"class":1052,"line":1161},10,[1163],{"type":43,"tag":1050,"props":1164,"children":1165},{"style":1062},[1166],{"type":49,"value":1167},"    ]\n",{"type":43,"tag":1050,"props":1169,"children":1171},{"class":1052,"line":1170},11,[1172],{"type":43,"tag":1050,"props":1173,"children":1174},{"style":1062},[1175],{"type":49,"value":1176},"  }\n",{"type":43,"tag":1050,"props":1178,"children":1180},{"class":1052,"line":1179},12,[1181,1186],{"type":43,"tag":1050,"props":1182,"children":1183},{"style":1062},[1184],{"type":49,"value":1185},"}",{"type":43,"tag":1050,"props":1187,"children":1188},{"style":1078},[1189],{"type":49,"value":1190},"'\n",{"type":43,"tag":52,"props":1192,"children":1193},{},[1194,1196,1200],{"type":49,"value":1195},"For a DAB-authored version (the production path), see ",{"type":43,"tag":337,"props":1197,"children":1198},{"href":452},[1199],{"type":49,"value":455},{"type":49,"value":344},{"type":43,"tag":69,"props":1202,"children":1203},{},[],{"type":43,"tag":73,"props":1205,"children":1207},{"id":1206},"running-the-pipeline",[1208],{"type":49,"value":1209},"Running the pipeline",{"type":43,"tag":52,"props":1211,"children":1212},{},[1213],{"type":49,"value":1214},"Once authored, deploy and trigger a run. The bundle path gives the cleanest run-by-key command:",{"type":43,"tag":1039,"props":1216,"children":1218},{"className":1041,"code":1217,"language":1043,"meta":1044,"style":1044},"databricks bundle deploy -t dev\ndatabricks bundle run salesforce_ingestion           # KEY = the pipeline resource key in the bundle; waits by default\ndatabricks bundle run salesforce_ingestion --no-wait\n",[1219],{"type":43,"tag":841,"props":1220,"children":1221},{"__ignoreMap":1044},[1222,1249,1276],{"type":43,"tag":1050,"props":1223,"children":1224},{"class":1052,"line":1053},[1225,1229,1234,1239,1244],{"type":43,"tag":1050,"props":1226,"children":1227},{"style":1057},[1228],{"type":49,"value":8},{"type":43,"tag":1050,"props":1230,"children":1231},{"style":1062},[1232],{"type":49,"value":1233}," bundle",{"type":43,"tag":1050,"props":1235,"children":1236},{"style":1062},[1237],{"type":49,"value":1238}," deploy",{"type":43,"tag":1050,"props":1240,"children":1241},{"style":1062},[1242],{"type":49,"value":1243}," -t",{"type":43,"tag":1050,"props":1245,"children":1246},{"style":1062},[1247],{"type":49,"value":1248}," dev\n",{"type":43,"tag":1050,"props":1250,"children":1251},{"class":1052,"line":1089},[1252,1256,1260,1265,1270],{"type":43,"tag":1050,"props":1253,"children":1254},{"style":1057},[1255],{"type":49,"value":8},{"type":43,"tag":1050,"props":1257,"children":1258},{"style":1062},[1259],{"type":49,"value":1233},{"type":43,"tag":1050,"props":1261,"children":1262},{"style":1062},[1263],{"type":49,"value":1264}," run",{"type":43,"tag":1050,"props":1266,"children":1267},{"style":1062},[1268],{"type":49,"value":1269}," salesforce_ingestion",{"type":43,"tag":1050,"props":1271,"children":1273},{"style":1272},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1274],{"type":49,"value":1275},"           # KEY = the pipeline resource key in the bundle; waits by default\n",{"type":43,"tag":1050,"props":1277,"children":1278},{"class":1052,"line":1098},[1279,1283,1287,1291,1295],{"type":43,"tag":1050,"props":1280,"children":1281},{"style":1057},[1282],{"type":49,"value":8},{"type":43,"tag":1050,"props":1284,"children":1285},{"style":1062},[1286],{"type":49,"value":1233},{"type":43,"tag":1050,"props":1288,"children":1289},{"style":1062},[1290],{"type":49,"value":1264},{"type":43,"tag":1050,"props":1292,"children":1293},{"style":1062},[1294],{"type":49,"value":1269},{"type":43,"tag":1050,"props":1296,"children":1297},{"style":1062},[1298],{"type":49,"value":1299}," --no-wait\n",{"type":43,"tag":52,"props":1301,"children":1302},{},[1303,1305,1311],{"type":49,"value":1304},"A pipeline created imperatively with ",{"type":43,"tag":841,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":49,"value":1310},"pipelines create --json",{"type":49,"value":1312}," has no run-by-name CLI — start and poll an update by pipeline ID instead:",{"type":43,"tag":1039,"props":1314,"children":1316},{"className":1041,"code":1315,"language":1043,"meta":1044,"style":1044},"databricks pipelines start-update \u003Cpipeline-id>             # returns an update_id\ndatabricks pipelines get-update  \u003Cpipeline-id> \u003Cupdate-id>  # poll one update's status\ndatabricks pipelines list-updates \u003Cpipeline-id>             # recent updates and their states\n",[1317],{"type":43,"tag":841,"props":1318,"children":1319},{"__ignoreMap":1044},[1320,1362,1417],{"type":43,"tag":1050,"props":1321,"children":1322},{"class":1052,"line":1053},[1323,1327,1331,1336,1341,1346,1352,1357],{"type":43,"tag":1050,"props":1324,"children":1325},{"style":1057},[1326],{"type":49,"value":8},{"type":43,"tag":1050,"props":1328,"children":1329},{"style":1062},[1330],{"type":49,"value":1065},{"type":43,"tag":1050,"props":1332,"children":1333},{"style":1062},[1334],{"type":49,"value":1335}," start-update",{"type":43,"tag":1050,"props":1337,"children":1338},{"style":1078},[1339],{"type":49,"value":1340}," \u003C",{"type":43,"tag":1050,"props":1342,"children":1343},{"style":1062},[1344],{"type":49,"value":1345},"pipeline-i",{"type":43,"tag":1050,"props":1347,"children":1349},{"style":1348},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1350],{"type":49,"value":1351},"d",{"type":43,"tag":1050,"props":1353,"children":1354},{"style":1078},[1355],{"type":49,"value":1356},">",{"type":43,"tag":1050,"props":1358,"children":1359},{"style":1272},[1360],{"type":49,"value":1361},"             # returns an update_id\n",{"type":43,"tag":1050,"props":1363,"children":1364},{"class":1052,"line":1089},[1365,1369,1373,1378,1383,1387,1391,1395,1399,1404,1408,1412],{"type":43,"tag":1050,"props":1366,"children":1367},{"style":1057},[1368],{"type":49,"value":8},{"type":43,"tag":1050,"props":1370,"children":1371},{"style":1062},[1372],{"type":49,"value":1065},{"type":43,"tag":1050,"props":1374,"children":1375},{"style":1062},[1376],{"type":49,"value":1377}," get-update",{"type":43,"tag":1050,"props":1379,"children":1380},{"style":1078},[1381],{"type":49,"value":1382},"  \u003C",{"type":43,"tag":1050,"props":1384,"children":1385},{"style":1062},[1386],{"type":49,"value":1345},{"type":43,"tag":1050,"props":1388,"children":1389},{"style":1348},[1390],{"type":49,"value":1351},{"type":43,"tag":1050,"props":1392,"children":1393},{"style":1078},[1394],{"type":49,"value":1356},{"type":43,"tag":1050,"props":1396,"children":1397},{"style":1078},[1398],{"type":49,"value":1340},{"type":43,"tag":1050,"props":1400,"children":1401},{"style":1062},[1402],{"type":49,"value":1403},"update-i",{"type":43,"tag":1050,"props":1405,"children":1406},{"style":1348},[1407],{"type":49,"value":1351},{"type":43,"tag":1050,"props":1409,"children":1410},{"style":1078},[1411],{"type":49,"value":1356},{"type":43,"tag":1050,"props":1413,"children":1414},{"style":1272},[1415],{"type":49,"value":1416},"  # poll one update's status\n",{"type":43,"tag":1050,"props":1418,"children":1419},{"class":1052,"line":1098},[1420,1424,1428,1433,1437,1441,1445,1449],{"type":43,"tag":1050,"props":1421,"children":1422},{"style":1057},[1423],{"type":49,"value":8},{"type":43,"tag":1050,"props":1425,"children":1426},{"style":1062},[1427],{"type":49,"value":1065},{"type":43,"tag":1050,"props":1429,"children":1430},{"style":1062},[1431],{"type":49,"value":1432}," list-updates",{"type":43,"tag":1050,"props":1434,"children":1435},{"style":1078},[1436],{"type":49,"value":1340},{"type":43,"tag":1050,"props":1438,"children":1439},{"style":1062},[1440],{"type":49,"value":1345},{"type":43,"tag":1050,"props":1442,"children":1443},{"style":1348},[1444],{"type":49,"value":1351},{"type":43,"tag":1050,"props":1446,"children":1447},{"style":1078},[1448],{"type":49,"value":1356},{"type":43,"tag":1050,"props":1450,"children":1451},{"style":1272},[1452],{"type":49,"value":1453},"             # recent updates and their states\n",{"type":43,"tag":52,"props":1455,"children":1456},{},[1457],{"type":49,"value":1458},"That asymmetry is one more reason to author with a Declarative Automation Bundle.",{"type":43,"tag":69,"props":1460,"children":1461},{},[],{"type":43,"tag":73,"props":1463,"children":1465},{"id":1464},"detailed-guides",[1466],{"type":49,"value":1467},"Detailed guides",{"type":43,"tag":195,"props":1469,"children":1470},{},[1471,1492],{"type":43,"tag":199,"props":1472,"children":1473},{},[1474],{"type":43,"tag":203,"props":1475,"children":1476},{},[1477,1482,1487],{"type":43,"tag":207,"props":1478,"children":1479},{},[1480],{"type":49,"value":1481},"Topic",{"type":43,"tag":207,"props":1483,"children":1484},{},[1485],{"type":49,"value":1486},"File",{"type":43,"tag":207,"props":1488,"children":1489},{},[1490],{"type":49,"value":1491},"When to read",{"type":43,"tag":223,"props":1493,"children":1494},{},[1495,1515,1535,1555],{"type":43,"tag":203,"props":1496,"children":1497},{},[1498,1503,1510],{"type":43,"tag":230,"props":1499,"children":1500},{},[1501],{"type":49,"value":1502},"SaaS connectors (Salesforce, Workday Reports, ServiceNow, GA4, HubSpot, Confluence)",{"type":43,"tag":230,"props":1504,"children":1505},{},[1506],{"type":43,"tag":337,"props":1507,"children":1508},{"href":452},[1509],{"type":49,"value":455},{"type":43,"tag":230,"props":1511,"children":1512},{},[1513],{"type":49,"value":1514},"Unified SaaS pattern, per-connector deltas, OAuth flows, DAB stubs",{"type":43,"tag":203,"props":1516,"children":1517},{},[1518,1523,1530],{"type":43,"tag":230,"props":1519,"children":1520},{},[1521],{"type":49,"value":1522},"Database connectors (SQL Server cloud + on-prem)",{"type":43,"tag":230,"props":1524,"children":1525},{},[1526],{"type":43,"tag":337,"props":1527,"children":1528},{"href":599},[1529],{"type":49,"value":602},{"type":43,"tag":230,"props":1531,"children":1532},{},[1533],{"type":49,"value":1534},"Gateway pattern, change tracking vs CDC, network setup",{"type":43,"tag":203,"props":1536,"children":1537},{},[1538,1543,1550],{"type":43,"tag":230,"props":1539,"children":1540},{},[1541],{"type":49,"value":1542},"Ingestion decision tree",{"type":43,"tag":230,"props":1544,"children":1545},{},[1546],{"type":43,"tag":337,"props":1547,"children":1548},{"href":339},[1549],{"type":49,"value":342},{"type":43,"tag":230,"props":1551,"children":1552},{},[1553],{"type":49,"value":1554},"Lakeflow Connect vs Auto Loader vs Lakehouse Federation vs Delta Sharing",{"type":43,"tag":203,"props":1556,"children":1557},{},[1558,1563,1572],{"type":43,"tag":230,"props":1559,"children":1560},{},[1561],{"type":49,"value":1562},"Troubleshooting and monitoring",{"type":43,"tag":230,"props":1564,"children":1565},{},[1566],{"type":43,"tag":337,"props":1567,"children":1569},{"href":1568},"references\u002F5-troubleshooting-and-monitoring.md",[1570],{"type":49,"value":1571},"5-troubleshooting-and-monitoring.md",{"type":43,"tag":230,"props":1573,"children":1574},{},[1575],{"type":49,"value":1576},"Event log queries, common errors, escalation pointers",{"type":43,"tag":69,"props":1578,"children":1579},{},[],{"type":43,"tag":73,"props":1581,"children":1583},{"id":1582},"workflow",[1584],{"type":49,"value":1585},"Workflow",{"type":43,"tag":52,"props":1587,"children":1588},{},[1589],{"type":49,"value":1590},"For each new ingestion pipeline:",{"type":43,"tag":130,"props":1592,"children":1593},{},[1594,1604,1614,1629,1646,1662],{"type":43,"tag":94,"props":1595,"children":1596},{},[1597,1602],{"type":43,"tag":61,"props":1598,"children":1599},{},[1600],{"type":49,"value":1601},"Pick the connector category",{"type":49,"value":1603}," — SaaS \u002F database \u002F file \u002F push — and read the matching reference file.",{"type":43,"tag":94,"props":1605,"children":1606},{},[1607,1612],{"type":43,"tag":61,"props":1608,"children":1609},{},[1610],{"type":49,"value":1611},"Verify prerequisites",{"type":49,"value":1613}," — UC target, source credentials, network path (for databases), region availability.",{"type":43,"tag":94,"props":1615,"children":1616},{},[1617,1627],{"type":43,"tag":61,"props":1618,"children":1619},{},[1620,1622],{"type":49,"value":1621},"Create the UC ",{"type":43,"tag":841,"props":1623,"children":1625},{"className":1624},[],[1626],{"type":49,"value":983},{"type":49,"value":1628}," — UI for OAuth U2M, CLI \u002F DAB for everything else.",{"type":43,"tag":94,"props":1630,"children":1631},{},[1632,1637,1639,1644],{"type":43,"tag":61,"props":1633,"children":1634},{},[1635],{"type":49,"value":1636},"Author the pipeline",{"type":49,"value":1638}," — ",{"type":43,"tag":841,"props":1640,"children":1642},{"className":1641},[],[1643],{"type":49,"value":1027},{"type":49,"value":1645}," for one-offs, DAB YAML for anything shipping to a customer.",{"type":43,"tag":94,"props":1647,"children":1648},{},[1649,1654,1656,1660],{"type":43,"tag":61,"props":1650,"children":1651},{},[1652],{"type":49,"value":1653},"Trigger the first run",{"type":49,"value":1655}," and watch the event log; see ",{"type":43,"tag":337,"props":1657,"children":1658},{"href":1568},[1659],{"type":49,"value":1571},{"type":49,"value":1661}," for the SQL.",{"type":43,"tag":94,"props":1663,"children":1664},{},[1665,1670,1672,1678,1680,1686],{"type":43,"tag":61,"props":1666,"children":1667},{},[1668],{"type":49,"value":1669},"Schedule",{"type":49,"value":1671}," the triggered pipeline with a Jobs ",{"type":43,"tag":841,"props":1673,"children":1675},{"className":1674},[],[1676],{"type":49,"value":1677},"pipeline_task",{"type":49,"value":1679}," (cron or interval). Lakeflow Connect supports triggered runs only — ",{"type":43,"tag":841,"props":1681,"children":1683},{"className":1682},[],[1684],{"type":49,"value":1685},"continuous: false",{"type":49,"value":1687}," selects triggered mode but is not itself a schedule, so the cadence comes from the Jobs trigger.",{"type":43,"tag":69,"props":1689,"children":1690},{},[],{"type":43,"tag":73,"props":1692,"children":1694},{"id":1693},"anti-patterns",[1695],{"type":49,"value":1696},"Anti-patterns",{"type":43,"tag":52,"props":1698,"children":1699},{},[1700],{"type":49,"value":1701},"Three forms that look plausible but fail — wrong vs. right:",{"type":43,"tag":52,"props":1703,"children":1704},{},[1705],{"type":43,"tag":61,"props":1706,"children":1707},{},[1708,1710,1716],{"type":49,"value":1709},"1. ",{"type":43,"tag":841,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":49,"value":1715},"CREATE TABLE ... FROM CONNECTION",{"type":49,"value":1717}," is Lakehouse Federation, not Lakeflow Connect.",{"type":43,"tag":1039,"props":1719,"children":1723},{"className":1720,"code":1721,"language":1722,"meta":1044,"style":1044},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","-- WRONG: Federation syntax; no LFC equivalent exists\nCREATE TABLE main.salesforce_raw.account FROM CONNECTION my_salesforce_conn;\n","sql",[1724],{"type":43,"tag":841,"props":1725,"children":1726},{"__ignoreMap":1044},[1727,1735],{"type":43,"tag":1050,"props":1728,"children":1729},{"class":1052,"line":1053},[1730],{"type":43,"tag":1050,"props":1731,"children":1732},{},[1733],{"type":49,"value":1734},"-- WRONG: Federation syntax; no LFC equivalent exists\n",{"type":43,"tag":1050,"props":1736,"children":1737},{"class":1052,"line":1089},[1738],{"type":43,"tag":1050,"props":1739,"children":1740},{},[1741],{"type":49,"value":1742},"CREATE TABLE main.salesforce_raw.account FROM CONNECTION my_salesforce_conn;\n",{"type":43,"tag":1039,"props":1744,"children":1748},{"className":1745,"code":1746,"language":1747,"meta":1044,"style":1044},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\u002F\u002F RIGHT: author an ingestion_definition (see the Minimal Example above)\n{\"ingestion_definition\": {\"connection_name\": \"my_salesforce_conn\", \"objects\": [\u002F* ... *\u002F]}}\n","json",[1749],{"type":43,"tag":841,"props":1750,"children":1751},{"__ignoreMap":1044},[1752,1760],{"type":43,"tag":1050,"props":1753,"children":1754},{"class":1052,"line":1053},[1755],{"type":43,"tag":1050,"props":1756,"children":1757},{"style":1272},[1758],{"type":49,"value":1759},"\u002F\u002F RIGHT: author an ingestion_definition (see the Minimal Example above)\n",{"type":43,"tag":1050,"props":1761,"children":1762},{"class":1052,"line":1089},[1763,1768,1773,1778,1782,1787,1792,1796,1801,1805,1809,1814,1819,1823,1828,1832,1837,1841,1845,1850,1855],{"type":43,"tag":1050,"props":1764,"children":1765},{"style":1078},[1766],{"type":49,"value":1767},"{",{"type":43,"tag":1050,"props":1769,"children":1770},{"style":1078},[1771],{"type":49,"value":1772},"\"",{"type":43,"tag":1050,"props":1774,"children":1776},{"style":1775},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1777],{"type":49,"value":901},{"type":43,"tag":1050,"props":1779,"children":1780},{"style":1078},[1781],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1783,"children":1784},{"style":1078},[1785],{"type":49,"value":1786},":",{"type":43,"tag":1050,"props":1788,"children":1789},{"style":1078},[1790],{"type":49,"value":1791}," {",{"type":43,"tag":1050,"props":1793,"children":1794},{"style":1078},[1795],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1797,"children":1798},{"style":1057},[1799],{"type":49,"value":1800},"connection_name",{"type":43,"tag":1050,"props":1802,"children":1803},{"style":1078},[1804],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1806,"children":1807},{"style":1078},[1808],{"type":49,"value":1786},{"type":43,"tag":1050,"props":1810,"children":1811},{"style":1078},[1812],{"type":49,"value":1813}," \"",{"type":43,"tag":1050,"props":1815,"children":1816},{"style":1062},[1817],{"type":49,"value":1818},"my_salesforce_conn",{"type":43,"tag":1050,"props":1820,"children":1821},{"style":1078},[1822],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1824,"children":1825},{"style":1078},[1826],{"type":49,"value":1827},",",{"type":43,"tag":1050,"props":1829,"children":1830},{"style":1078},[1831],{"type":49,"value":1813},{"type":43,"tag":1050,"props":1833,"children":1834},{"style":1057},[1835],{"type":49,"value":1836},"objects",{"type":43,"tag":1050,"props":1838,"children":1839},{"style":1078},[1840],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1842,"children":1843},{"style":1078},[1844],{"type":49,"value":1786},{"type":43,"tag":1050,"props":1846,"children":1847},{"style":1078},[1848],{"type":49,"value":1849}," [",{"type":43,"tag":1050,"props":1851,"children":1852},{"style":1272},[1853],{"type":49,"value":1854},"\u002F* ... *\u002F",{"type":43,"tag":1050,"props":1856,"children":1857},{"style":1078},[1858],{"type":49,"value":1859},"]}}\n",{"type":43,"tag":52,"props":1861,"children":1862},{},[1863],{"type":43,"tag":61,"props":1864,"children":1865},{},[1866,1868,1873,1875,1880],{"type":49,"value":1867},"2. An ingestion pipeline carries ",{"type":43,"tag":841,"props":1869,"children":1871},{"className":1870},[],[1872],{"type":49,"value":901},{"type":49,"value":1874},", never a ",{"type":43,"tag":841,"props":1876,"children":1878},{"className":1877},[],[1879],{"type":49,"value":909},{"type":49,"value":1881}," block.",{"type":43,"tag":1039,"props":1883,"children":1885},{"className":1745,"code":1884,"language":1747,"meta":1044,"style":1044},"\u002F\u002F WRONG: libraries is for a standard SDP pipeline running your notebooks\u002Ffiles\n{\"name\": \"salesforce_to_uc\", \"libraries\": [{\"notebook\": {\"path\": \"\u002FRepos\u002F...\u002Fingest\"}}]}\n\u002F\u002F RIGHT:\n{\"name\": \"salesforce_to_uc\", \"ingestion_definition\": {\"connection_name\": \"...\", \"objects\": []}}\n",[1886],{"type":43,"tag":841,"props":1887,"children":1888},{"__ignoreMap":1044},[1889,1897,2016,2024],{"type":43,"tag":1050,"props":1890,"children":1891},{"class":1052,"line":1053},[1892],{"type":43,"tag":1050,"props":1893,"children":1894},{"style":1272},[1895],{"type":49,"value":1896},"\u002F\u002F WRONG: libraries is for a standard SDP pipeline running your notebooks\u002Ffiles\n",{"type":43,"tag":1050,"props":1898,"children":1899},{"class":1052,"line":1089},[1900,1904,1908,1913,1917,1921,1925,1930,1934,1938,1942,1946,1950,1954,1959,1963,1968,1972,1976,1980,1984,1990,1994,1998,2002,2007,2011],{"type":43,"tag":1050,"props":1901,"children":1902},{"style":1078},[1903],{"type":49,"value":1767},{"type":43,"tag":1050,"props":1905,"children":1906},{"style":1078},[1907],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1909,"children":1910},{"style":1775},[1911],{"type":49,"value":1912},"name",{"type":43,"tag":1050,"props":1914,"children":1915},{"style":1078},[1916],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1918,"children":1919},{"style":1078},[1920],{"type":49,"value":1786},{"type":43,"tag":1050,"props":1922,"children":1923},{"style":1078},[1924],{"type":49,"value":1813},{"type":43,"tag":1050,"props":1926,"children":1927},{"style":1062},[1928],{"type":49,"value":1929},"salesforce_to_uc",{"type":43,"tag":1050,"props":1931,"children":1932},{"style":1078},[1933],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1935,"children":1936},{"style":1078},[1937],{"type":49,"value":1827},{"type":43,"tag":1050,"props":1939,"children":1940},{"style":1078},[1941],{"type":49,"value":1813},{"type":43,"tag":1050,"props":1943,"children":1944},{"style":1775},[1945],{"type":49,"value":909},{"type":43,"tag":1050,"props":1947,"children":1948},{"style":1078},[1949],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1951,"children":1952},{"style":1078},[1953],{"type":49,"value":1786},{"type":43,"tag":1050,"props":1955,"children":1956},{"style":1078},[1957],{"type":49,"value":1958}," [{",{"type":43,"tag":1050,"props":1960,"children":1961},{"style":1078},[1962],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1964,"children":1965},{"style":1057},[1966],{"type":49,"value":1967},"notebook",{"type":43,"tag":1050,"props":1969,"children":1970},{"style":1078},[1971],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1973,"children":1974},{"style":1078},[1975],{"type":49,"value":1786},{"type":43,"tag":1050,"props":1977,"children":1978},{"style":1078},[1979],{"type":49,"value":1791},{"type":43,"tag":1050,"props":1981,"children":1982},{"style":1078},[1983],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1985,"children":1987},{"style":1986},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1988],{"type":49,"value":1989},"path",{"type":43,"tag":1050,"props":1991,"children":1992},{"style":1078},[1993],{"type":49,"value":1772},{"type":43,"tag":1050,"props":1995,"children":1996},{"style":1078},[1997],{"type":49,"value":1786},{"type":43,"tag":1050,"props":1999,"children":2000},{"style":1078},[2001],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2003,"children":2004},{"style":1062},[2005],{"type":49,"value":2006},"\u002FRepos\u002F...\u002Fingest",{"type":43,"tag":1050,"props":2008,"children":2009},{"style":1078},[2010],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2012,"children":2013},{"style":1078},[2014],{"type":49,"value":2015},"}}]}\n",{"type":43,"tag":1050,"props":2017,"children":2018},{"class":1052,"line":1098},[2019],{"type":43,"tag":1050,"props":2020,"children":2021},{"style":1272},[2022],{"type":49,"value":2023},"\u002F\u002F RIGHT:\n",{"type":43,"tag":1050,"props":2025,"children":2026},{"class":1052,"line":1107},[2027,2031,2035,2039,2043,2047,2051,2055,2059,2063,2067,2071,2075,2079,2083,2087,2091,2095,2099,2103,2108,2112,2116,2120,2124,2128,2132],{"type":43,"tag":1050,"props":2028,"children":2029},{"style":1078},[2030],{"type":49,"value":1767},{"type":43,"tag":1050,"props":2032,"children":2033},{"style":1078},[2034],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2036,"children":2037},{"style":1775},[2038],{"type":49,"value":1912},{"type":43,"tag":1050,"props":2040,"children":2041},{"style":1078},[2042],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2044,"children":2045},{"style":1078},[2046],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2048,"children":2049},{"style":1078},[2050],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2052,"children":2053},{"style":1062},[2054],{"type":49,"value":1929},{"type":43,"tag":1050,"props":2056,"children":2057},{"style":1078},[2058],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2060,"children":2061},{"style":1078},[2062],{"type":49,"value":1827},{"type":43,"tag":1050,"props":2064,"children":2065},{"style":1078},[2066],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2068,"children":2069},{"style":1775},[2070],{"type":49,"value":901},{"type":43,"tag":1050,"props":2072,"children":2073},{"style":1078},[2074],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2076,"children":2077},{"style":1078},[2078],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2080,"children":2081},{"style":1078},[2082],{"type":49,"value":1791},{"type":43,"tag":1050,"props":2084,"children":2085},{"style":1078},[2086],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2088,"children":2089},{"style":1057},[2090],{"type":49,"value":1800},{"type":43,"tag":1050,"props":2092,"children":2093},{"style":1078},[2094],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2096,"children":2097},{"style":1078},[2098],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2100,"children":2101},{"style":1078},[2102],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2104,"children":2105},{"style":1062},[2106],{"type":49,"value":2107},"...",{"type":43,"tag":1050,"props":2109,"children":2110},{"style":1078},[2111],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2113,"children":2114},{"style":1078},[2115],{"type":49,"value":1827},{"type":43,"tag":1050,"props":2117,"children":2118},{"style":1078},[2119],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2121,"children":2122},{"style":1057},[2123],{"type":49,"value":1836},{"type":43,"tag":1050,"props":2125,"children":2126},{"style":1078},[2127],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2129,"children":2130},{"style":1078},[2131],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2133,"children":2134},{"style":1078},[2135],{"type":49,"value":2136}," []}}\n",{"type":43,"tag":52,"props":2138,"children":2139},{},[2140],{"type":43,"tag":61,"props":2141,"children":2142},{},[2143,2145,2151],{"type":49,"value":2144},"3. ",{"type":43,"tag":841,"props":2146,"children":2148},{"className":2147},[],[2149],{"type":49,"value":2150},"continuous: true",{"type":49,"value":2152}," is rejected — Lakeflow Connect is triggered-only.",{"type":43,"tag":1039,"props":2154,"children":2156},{"className":1745,"code":2155,"language":1747,"meta":1044,"style":1044},"\u002F\u002F WRONG: continuous mode fails at create\n{\"continuous\": true, \"ingestion_definition\": {\u002F* ... *\u002F}}\n\u002F\u002F RIGHT: continuous:false (or omit) + schedule with a Jobs pipeline_task\n{\"continuous\": false, \"ingestion_definition\": {\u002F* ... *\u002F}}\n",[2157],{"type":43,"tag":841,"props":2158,"children":2159},{"__ignoreMap":1044},[2160,2168,2226,2234],{"type":43,"tag":1050,"props":2161,"children":2162},{"class":1052,"line":1053},[2163],{"type":43,"tag":1050,"props":2164,"children":2165},{"style":1272},[2166],{"type":49,"value":2167},"\u002F\u002F WRONG: continuous mode fails at create\n",{"type":43,"tag":1050,"props":2169,"children":2170},{"class":1052,"line":1089},[2171,2175,2179,2184,2188,2192,2197,2201,2205,2209,2213,2217,2221],{"type":43,"tag":1050,"props":2172,"children":2173},{"style":1078},[2174],{"type":49,"value":1767},{"type":43,"tag":1050,"props":2176,"children":2177},{"style":1078},[2178],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2180,"children":2181},{"style":1775},[2182],{"type":49,"value":2183},"continuous",{"type":43,"tag":1050,"props":2185,"children":2186},{"style":1078},[2187],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2189,"children":2190},{"style":1078},[2191],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2193,"children":2194},{"style":1078},[2195],{"type":49,"value":2196}," true,",{"type":43,"tag":1050,"props":2198,"children":2199},{"style":1078},[2200],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2202,"children":2203},{"style":1775},[2204],{"type":49,"value":901},{"type":43,"tag":1050,"props":2206,"children":2207},{"style":1078},[2208],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2210,"children":2211},{"style":1078},[2212],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2214,"children":2215},{"style":1078},[2216],{"type":49,"value":1791},{"type":43,"tag":1050,"props":2218,"children":2219},{"style":1272},[2220],{"type":49,"value":1854},{"type":43,"tag":1050,"props":2222,"children":2223},{"style":1078},[2224],{"type":49,"value":2225},"}}\n",{"type":43,"tag":1050,"props":2227,"children":2228},{"class":1052,"line":1098},[2229],{"type":43,"tag":1050,"props":2230,"children":2231},{"style":1272},[2232],{"type":49,"value":2233},"\u002F\u002F RIGHT: continuous:false (or omit) + schedule with a Jobs pipeline_task\n",{"type":43,"tag":1050,"props":2235,"children":2236},{"class":1052,"line":1107},[2237,2241,2245,2249,2253,2257,2262,2266,2270,2274,2278,2282,2286],{"type":43,"tag":1050,"props":2238,"children":2239},{"style":1078},[2240],{"type":49,"value":1767},{"type":43,"tag":1050,"props":2242,"children":2243},{"style":1078},[2244],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2246,"children":2247},{"style":1775},[2248],{"type":49,"value":2183},{"type":43,"tag":1050,"props":2250,"children":2251},{"style":1078},[2252],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2254,"children":2255},{"style":1078},[2256],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2258,"children":2259},{"style":1078},[2260],{"type":49,"value":2261}," false,",{"type":43,"tag":1050,"props":2263,"children":2264},{"style":1078},[2265],{"type":49,"value":1813},{"type":43,"tag":1050,"props":2267,"children":2268},{"style":1775},[2269],{"type":49,"value":901},{"type":43,"tag":1050,"props":2271,"children":2272},{"style":1078},[2273],{"type":49,"value":1772},{"type":43,"tag":1050,"props":2275,"children":2276},{"style":1078},[2277],{"type":49,"value":1786},{"type":43,"tag":1050,"props":2279,"children":2280},{"style":1078},[2281],{"type":49,"value":1791},{"type":43,"tag":1050,"props":2283,"children":2284},{"style":1272},[2285],{"type":49,"value":1854},{"type":43,"tag":1050,"props":2287,"children":2288},{"style":1078},[2289],{"type":49,"value":2225},{"type":43,"tag":69,"props":2291,"children":2292},{},[],{"type":43,"tag":73,"props":2294,"children":2296},{"id":2295},"important",[2297],{"type":49,"value":2298},"Important",{"type":43,"tag":90,"props":2300,"children":2301},{},[2302,2312,2331,2341,2351,2361],{"type":43,"tag":94,"props":2303,"children":2304},{},[2305,2310],{"type":43,"tag":61,"props":2306,"children":2307},{},[2308],{"type":49,"value":2309},"Triggered only, no continuous mode",{"type":49,"value":2311}," — pipelines run on a schedule or on-demand, never continuously. Check the connector reference for the latest status.",{"type":43,"tag":94,"props":2313,"children":2314},{},[2315,2320,2322,2329],{"type":43,"tag":61,"props":2316,"children":2317},{},[2318],{"type":49,"value":2319},"Compute-only billing",{"type":49,"value":2321}," — Lakeflow Connect is billed in DBUs (no per-row fee). Database connectors also incur classic-compute gateway DBUs in addition to the serverless ingestion pipeline DBUs. See the ",{"type":43,"tag":337,"props":2323,"children":2326},{"href":2324,"rel":2325},"https:\u002F\u002Fwww.databricks.com\u002Fproduct\u002Fpricing\u002Flakeflow-connect",[642],[2327],{"type":49,"value":2328},"pricing page",{"type":49,"value":2330}," for current rates.",{"type":43,"tag":94,"props":2332,"children":2333},{},[2334,2339],{"type":43,"tag":61,"props":2335,"children":2336},{},[2337],{"type":49,"value":2338},"Salesforce auth is OAuth U2M only",{"type":49,"value":2340}," — no machine-to-machine, no basic auth. Connection creation requires a UI walk-through.",{"type":43,"tag":94,"props":2342,"children":2343},{},[2344,2349],{"type":43,"tag":61,"props":2345,"children":2346},{},[2347],{"type":49,"value":2348},"Database staging retention is 30 days",{"type":49,"value":2350}," by default in the UC Volume between the gateway and the ingestion pipeline.",{"type":43,"tag":94,"props":2352,"children":2353},{},[2354,2359],{"type":43,"tag":61,"props":2355,"children":2356},{},[2357],{"type":49,"value":2358},"Limits per pipeline",{"type":49,"value":2360}," — most SaaS connectors cap at 250 tables per pipeline. Split across multiple pipelines if needed.",{"type":43,"tag":94,"props":2362,"children":2363},{},[2364,2369,2371,2375],{"type":43,"tag":61,"props":2365,"children":2366},{},[2367],{"type":49,"value":2368},"This lands raw tables",{"type":49,"value":2370}," — Lakeflow Connect writes source-faithful tables (the ingestion landing zone). Build the medallion Bronze\u002FSilver\u002FGold transforms on top of them with ",{"type":43,"tag":61,"props":2372,"children":2373},{},[2374],{"type":49,"value":267},{"type":49,"value":344},{"type":43,"tag":69,"props":2377,"children":2378},{},[],{"type":43,"tag":73,"props":2380,"children":2382},{"id":2381},"key-concepts",[2383],{"type":49,"value":2384},"Key Concepts",{"type":43,"tag":90,"props":2386,"children":2387},{},[2388,2405,2415,2425,2451],{"type":43,"tag":94,"props":2389,"children":2390},{},[2391,2403],{"type":43,"tag":61,"props":2392,"children":2393},{},[2394,2396,2401],{"type":49,"value":2395},"UC ",{"type":43,"tag":841,"props":2397,"children":2399},{"className":2398},[],[2400],{"type":49,"value":983},{"type":49,"value":2402}," is the credential anchor",{"type":49,"value":2404}," — every Lakeflow Connect pipeline points at a UC connection. The connection owns the auth; the pipeline references it by name.",{"type":43,"tag":94,"props":2406,"children":2407},{},[2408,2413],{"type":43,"tag":61,"props":2409,"children":2410},{},[2411],{"type":49,"value":2412},"Serverless ingestion pipeline + (optional) classic gateway",{"type":49,"value":2414}," — SaaS connectors are pure serverless. Database connectors split into a customer-network gateway (classic) and a serverless ingestion pipeline (Delta-bound).",{"type":43,"tag":94,"props":2416,"children":2417},{},[2418,2423],{"type":43,"tag":61,"props":2419,"children":2420},{},[2421],{"type":49,"value":2422},"CDC and schema evolution are built in",{"type":49,"value":2424}," — for sources that support change tracking or CDC, the connector applies changes incrementally and evolves the target schema. Data-type changes typically require a full snapshot reload.",{"type":43,"tag":94,"props":2426,"children":2427},{},[2428,2433,2435,2441,2443,2449],{"type":43,"tag":61,"props":2429,"children":2430},{},[2431],{"type":49,"value":2432},"Streaming Delta output",{"type":49,"value":2434}," — destination tables are governed Delta tables; CDC sources are applied with change semantics (",{"type":43,"tag":841,"props":2436,"children":2438},{"className":2437},[],[2439],{"type":49,"value":2440},"APPLY CHANGES",{"type":49,"value":2442}," \u002F AUTO CDC, or ",{"type":43,"tag":841,"props":2444,"children":2446},{"className":2445},[],[2447],{"type":49,"value":2448},"apply_changes_from_snapshot",{"type":49,"value":2450}," for snapshot sources). Compatible with downstream materialized views and Spark streaming.",{"type":43,"tag":94,"props":2452,"children":2453},{},[2454,2459,2461,2467,2469,2475],{"type":43,"tag":61,"props":2455,"children":2456},{},[2457],{"type":49,"value":2458},"OAuth U2M is UI-only",{"type":49,"value":2460}," — DAB \u002F CLI cannot bootstrap OAuth U2M connections. Hand the one-time browser step to a human (Catalog Explorer > External Data > Connections > Create connection > pick the source > sign in), then resume once ",{"type":43,"tag":841,"props":2462,"children":2464},{"className":2463},[],[2465],{"type":49,"value":2466},"databricks connections get \u003Cconnection_name>",{"type":49,"value":2468}," reports ",{"type":43,"tag":841,"props":2470,"children":2472},{"className":2471},[],[2473],{"type":49,"value":2474},"READY",{"type":49,"value":344},{"type":43,"tag":69,"props":2477,"children":2478},{},[],{"type":43,"tag":73,"props":2480,"children":2482},{"id":2481},"common-issues",[2483],{"type":49,"value":2484},"Common Issues",{"type":43,"tag":52,"props":2486,"children":2487},{},[2488,2490,2496,2498,2502],{"type":49,"value":2489},"For common errors and their fixes — duplicate-key violations, watermark \u002F cursor problems, schema evolution, gateway region availability, the ",{"type":43,"tag":841,"props":2491,"children":2493},{"className":2492},[],[2494],{"type":49,"value":2495},"channel",{"type":49,"value":2497}," runtime-channel setting, and pipelines that run but land no data — see ",{"type":43,"tag":337,"props":2499,"children":2500},{"href":1568},[2501],{"type":49,"value":1571},{"type":49,"value":2503},", which also has the event-log queries to diagnose them.",{"type":43,"tag":69,"props":2505,"children":2506},{},[],{"type":43,"tag":73,"props":2508,"children":2510},{"id":2509},"related-skills",[2511],{"type":49,"value":2512},"Related Skills",{"type":43,"tag":90,"props":2514,"children":2515},{},[2516,2525,2534,2544,2561],{"type":43,"tag":94,"props":2517,"children":2518},{},[2519,2523],{"type":43,"tag":61,"props":2520,"children":2521},{},[2522],{"type":49,"value":267},{"type":49,"value":2524}," — the SDP runtime that Lakeflow Connect pipelines run on. For Auto Loader and downstream pipeline patterns.",{"type":43,"tag":94,"props":2526,"children":2527},{},[2528,2532],{"type":43,"tag":61,"props":2529,"children":2530},{},[2531],{"type":49,"value":313},{"type":49,"value":2533}," — push-based gRPC ingestion. Sibling to Lakeflow Connect's pull-based connectors.",{"type":43,"tag":94,"props":2535,"children":2536},{},[2537,2542],{"type":43,"tag":61,"props":2538,"children":2539},{},[2540],{"type":49,"value":2541},"databricks-dabs",{"type":49,"value":2543}," — author Lakeflow Connect pipelines as IaC.",{"type":43,"tag":94,"props":2545,"children":2546},{},[2547,2552,2554,2559],{"type":43,"tag":61,"props":2548,"children":2549},{},[2550],{"type":49,"value":2551},"databricks-unity-catalog",{"type":49,"value":2553}," — managing catalogs, schemas, and the UC ",{"type":43,"tag":841,"props":2555,"children":2557},{"className":2556},[],[2558],{"type":49,"value":983},{"type":49,"value":2560}," objects that LFC credentials live in.",{"type":43,"tag":94,"props":2562,"children":2563},{},[2564,2569,2571,2576],{"type":43,"tag":61,"props":2565,"children":2566},{},[2567],{"type":49,"value":2568},"databricks-jobs",{"type":49,"value":2570}," — schedule ingestion pipelines with ",{"type":43,"tag":841,"props":2572,"children":2574},{"className":2573},[],[2575],{"type":49,"value":1677},{"type":49,"value":344},{"type":43,"tag":69,"props":2578,"children":2579},{},[],{"type":43,"tag":73,"props":2581,"children":2583},{"id":2582},"resources",[2584],{"type":49,"value":2585},"Resources",{"type":43,"tag":90,"props":2587,"children":2588},{},[2589,2599,2608],{"type":43,"tag":94,"props":2590,"children":2591},{},[2592],{"type":43,"tag":337,"props":2593,"children":2596},{"href":2594,"rel":2595},"https:\u002F\u002Fdocs.databricks.com\u002Faws\u002Fen\u002Fingestion\u002Flakeflow-connect",[642],[2597],{"type":49,"value":2598},"Lakeflow Connect public docs hub",{"type":43,"tag":94,"props":2600,"children":2601},{},[2602],{"type":43,"tag":337,"props":2603,"children":2605},{"href":640,"rel":2604},[642],[2606],{"type":49,"value":2607},"Connector reference (per-connector setup)",{"type":43,"tag":94,"props":2609,"children":2610},{},[2611],{"type":43,"tag":337,"props":2612,"children":2614},{"href":2324,"rel":2613},[642],[2615],{"type":49,"value":2616},"Pricing",{"type":43,"tag":2618,"props":2619,"children":2620},"style",{},[2621],{"type":49,"value":2622},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2624,"total":2809},[2625,2642,2656,2673,2690,2710,2721,2744,2755,2771,2783,2796],{"slug":2626,"name":2626,"fn":2627,"description":2628,"org":2629,"tags":2630,"stars":23,"repoUrl":24,"updatedAt":2641},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2631,2634,2635,2638],{"name":2632,"slug":2633,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2636,"slug":2637,"type":15},"Knowledge Management","knowledge-management",{"name":2639,"slug":2640,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":2643,"name":2643,"fn":2644,"description":2645,"org":2646,"tags":2647,"stars":23,"repoUrl":24,"updatedAt":2655},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2648,2651,2652],{"name":2649,"slug":2650,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":2653,"slug":2654,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":2657,"name":2657,"fn":2658,"description":2659,"org":2660,"tags":2661,"stars":23,"repoUrl":24,"updatedAt":2672},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2662,2665,2666,2669],{"name":2663,"slug":2664,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":2667,"slug":2668,"type":15},"Docker","docker",{"name":2670,"slug":2671,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":2674,"name":2674,"fn":2675,"description":2676,"org":2677,"tags":2678,"stars":23,"repoUrl":24,"updatedAt":2689},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2679,2682,2685,2688],{"name":2680,"slug":2681,"type":15},"Analytics","analytics",{"name":2683,"slug":2684,"type":15},"Dashboards","dashboards",{"name":2686,"slug":2687,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":2691,"name":2691,"fn":2692,"description":2693,"org":2694,"tags":2695,"stars":23,"repoUrl":24,"updatedAt":2709},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2696,2697,2700,2703,2706],{"name":9,"slug":8,"type":15},{"name":2698,"slug":2699,"type":15},"Design","design",{"name":2701,"slug":2702,"type":15},"Frontend","frontend",{"name":2704,"slug":2705,"type":15},"React","react",{"name":2707,"slug":2708,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":2711,"name":2711,"fn":2712,"description":2713,"org":2714,"tags":2715,"stars":23,"repoUrl":24,"updatedAt":2720},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2716,2717,2718,2719],{"name":2680,"slug":2681,"type":15},{"name":2683,"slug":2684,"type":15},{"name":2649,"slug":2650,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":2722,"name":2722,"fn":2723,"description":2724,"org":2725,"tags":2726,"stars":23,"repoUrl":24,"updatedAt":2743},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2727,2728,2731,2734,2737,2740],{"name":9,"slug":8,"type":15},{"name":2729,"slug":2730,"type":15},"FastAPI","fastapi",{"name":2732,"slug":2733,"type":15},"Flask","flask",{"name":2735,"slug":2736,"type":15},"Gradio","gradio",{"name":2738,"slug":2739,"type":15},"Python","python",{"name":2741,"slug":2742,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",{"slug":38,"name":38,"fn":2745,"description":2746,"org":2747,"tags":2748,"stars":23,"repoUrl":24,"updatedAt":2754},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2749,2752,2753],{"name":2750,"slug":2751,"type":15},"Authentication","authentication",{"name":2663,"slug":2664,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":2541,"name":2541,"fn":2756,"description":2757,"org":2758,"tags":2759,"stars":23,"repoUrl":24,"updatedAt":2770},"manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2760,2763,2766,2767],{"name":2761,"slug":2762,"type":15},"Automation","automation",{"name":2764,"slug":2765,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":2768,"slug":2769,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":2772,"name":2772,"fn":2773,"description":2774,"org":2775,"tags":2776,"stars":23,"repoUrl":24,"updatedAt":2782},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2777,2778,2779,2780],{"name":2649,"slug":2650,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":2781,"slug":1722,"type":15},"SQL","2026-07-31T05:53:32.561877",{"slug":2784,"name":2784,"fn":2785,"description":2786,"org":2787,"tags":2788,"stars":23,"repoUrl":24,"updatedAt":2795},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2789,2790,2793,2794],{"name":2649,"slug":2650,"type":15},{"name":2791,"slug":2792,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":2781,"slug":1722,"type":15},"2026-07-12T08:04:08.678282",{"slug":2797,"name":2797,"fn":2798,"description":2799,"org":2800,"tags":2801,"stars":23,"repoUrl":24,"updatedAt":2808},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2802,2803,2806],{"name":9,"slug":8,"type":15},{"name":2804,"slug":2805,"type":15},"Documentation","documentation",{"name":426,"slug":2807,"type":15},"reference","2026-07-15T05:41:34.697746",31,{"items":2811,"total":2809},[2812,2819,2825,2832,2839,2847,2854],{"slug":2626,"name":2626,"fn":2627,"description":2628,"org":2813,"tags":2814,"stars":23,"repoUrl":24,"updatedAt":2641},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2815,2816,2817,2818],{"name":2632,"slug":2633,"type":15},{"name":9,"slug":8,"type":15},{"name":2636,"slug":2637,"type":15},{"name":2639,"slug":2640,"type":15},{"slug":2643,"name":2643,"fn":2644,"description":2645,"org":2820,"tags":2821,"stars":23,"repoUrl":24,"updatedAt":2655},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2822,2823,2824],{"name":2649,"slug":2650,"type":15},{"name":9,"slug":8,"type":15},{"name":2653,"slug":2654,"type":15},{"slug":2657,"name":2657,"fn":2658,"description":2659,"org":2826,"tags":2827,"stars":23,"repoUrl":24,"updatedAt":2672},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2828,2829,2830,2831],{"name":2663,"slug":2664,"type":15},{"name":9,"slug":8,"type":15},{"name":2667,"slug":2668,"type":15},{"name":2670,"slug":2671,"type":15},{"slug":2674,"name":2674,"fn":2675,"description":2676,"org":2833,"tags":2834,"stars":23,"repoUrl":24,"updatedAt":2689},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2835,2836,2837,2838],{"name":2680,"slug":2681,"type":15},{"name":2683,"slug":2684,"type":15},{"name":2686,"slug":2687,"type":15},{"name":9,"slug":8,"type":15},{"slug":2691,"name":2691,"fn":2692,"description":2693,"org":2840,"tags":2841,"stars":23,"repoUrl":24,"updatedAt":2709},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2842,2843,2844,2845,2846],{"name":9,"slug":8,"type":15},{"name":2698,"slug":2699,"type":15},{"name":2701,"slug":2702,"type":15},{"name":2704,"slug":2705,"type":15},{"name":2707,"slug":2708,"type":15},{"slug":2711,"name":2711,"fn":2712,"description":2713,"org":2848,"tags":2849,"stars":23,"repoUrl":24,"updatedAt":2720},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2850,2851,2852,2853],{"name":2680,"slug":2681,"type":15},{"name":2683,"slug":2684,"type":15},{"name":2649,"slug":2650,"type":15},{"name":9,"slug":8,"type":15},{"slug":2722,"name":2722,"fn":2723,"description":2724,"org":2855,"tags":2856,"stars":23,"repoUrl":24,"updatedAt":2743},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2857,2858,2859,2860,2861,2862],{"name":9,"slug":8,"type":15},{"name":2729,"slug":2730,"type":15},{"name":2732,"slug":2733,"type":15},{"name":2735,"slug":2736,"type":15},{"name":2738,"slug":2739,"type":15},{"name":2741,"slug":2742,"type":15}]