[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-zerobus-ingest":3,"mdc--twevyg-key":33,"related-org-databricks-databricks-zerobus-ingest":2087,"related-repo-databricks-databricks-zerobus-ingest":2272},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"databricks-zerobus-ingest","ingest data into Databricks via gRPC","Build Zerobus Ingest clients for near real-time data ingestion into Databricks Delta tables via gRPC. Use when creating producers that write directly to Unity Catalog tables without a message bus, working with the Zerobus Ingest SDK in Python\u002FJava\u002FGo\u002FTypeScript\u002FRust, generating Protobuf schemas from UC tables, or implementing stream-based ingestion with ACK handling and retry logic.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"Data Pipeline","data-pipeline",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Engineering","engineering",204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-12T08:04:18.42609",null,60,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-zerobus-ingest","---\nname: databricks-zerobus-ingest\ndescription: \"Build Zerobus Ingest clients for near real-time data ingestion into Databricks Delta tables via gRPC. Use when creating producers that write directly to Unity Catalog tables without a message bus, working with the Zerobus Ingest SDK in Python\u002FJava\u002FGo\u002FTypeScript\u002FRust, generating Protobuf schemas from UC tables, or implementing stream-based ingestion with ACK handling and retry logic.\"\ncompatibility: Requires databricks CLI (>= v1.0.0)\nmetadata:\n  version: \"0.1.0\"\n---\n\n# Zerobus Ingest\n\nBuild clients that ingest data directly into Databricks Delta tables via the Zerobus gRPC API.\n\n**Status:** Generally Available. Charges are billed against the Jobs Serverless SKU. Check the [Zerobus overview](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-overview) for the current status of specific features (some, such as Streaming-table targets and Arrow Flight, may be in Beta).\n\n**Documentation:**\n- [Zerobus Overview](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-overview)\n- [Zerobus Ingest SDK](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-ingest)\n- [Zerobus Limits](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-limits)\n\n---\n\n## What Is Zerobus Ingest?\n\nZerobus Ingest is a serverless connector that enables direct, record-by-record data ingestion into Delta tables via gRPC. It eliminates the need for message bus infrastructure (Kafka, Kinesis, Event Hub) for lakehouse-bound data. The service validates schemas, materializes data to target tables, and sends durability acknowledgments back to the client.\n\n**Core pattern:** SDK init -> create stream -> ingest records -> handle ACKs -> flush -> close\n\n---\n\n## Quick Decision: What Are You Building?\n\n| Scenario | Language | Serialization | Reference |\n|----------|----------|---------------|-----------|\n| Quick prototype \u002F test harness | Python | JSON | [references\u002F2-python-client.md](references\u002F2-python-client.md) |\n| Production Python producer | Python | Protobuf | [references\u002F2-python-client.md](references\u002F2-python-client.md) + [references\u002F4-protobuf-schema.md](references\u002F4-protobuf-schema.md) |\n| JVM microservice | Java | Protobuf | [references\u002F3-multilanguage-clients.md](references\u002F3-multilanguage-clients.md) |\n| Go service | Go | JSON or Protobuf | [references\u002F3-multilanguage-clients.md](references\u002F3-multilanguage-clients.md) |\n| Node.js \u002F TypeScript app | TypeScript | JSON | [references\u002F3-multilanguage-clients.md](references\u002F3-multilanguage-clients.md) |\n| High-performance system service | Rust | JSON or Protobuf | [references\u002F3-multilanguage-clients.md](references\u002F3-multilanguage-clients.md) |\n| Schema generation from UC table | Any | Protobuf | [references\u002F4-protobuf-schema.md](references\u002F4-protobuf-schema.md) |\n| Retry \u002F reconnection logic | Any | Any | [references\u002F5-operations-and-limits.md](references\u002F5-operations-and-limits.md) |\n\nIf not specified, default to python.\n\n---\n\n## Common Libraries\n\nThese libraries are essential for Zerobus data ingestion and are typically NOT pre-installed on Databricks:\n\n- **databricks-zerobus-ingest-sdk**: Zerobus SDK for high-performance streaming ingestion\n- **databricks-sdk**: Databricks workspace client for authentication and metadata\n- **grpcio-tools**: only needed to compile a `.proto` for Protobuf serialization\n\nInstall them through the **job\u002Fcluster library configuration** (see [Installing Libraries](#installing-libraries) below) rather than pip-installing at runtime — the SDK cannot pip-install on serverless compute.\n\n### grpcio-tools and protobuf compatibility\n\n`grpcio-tools` must match the runtime's `protobuf` version. If proto compilation fails with a version error, pin a compatible build (for older `protobuf` 5.26\u002F5.29 runtimes, `grpcio-tools==1.62.0` works); otherwise use the latest release.\n\n---\n\n## Prerequisites\n\nYou must never execute the skill without confirming the below objects are valid: \n\n1. **A Unity Catalog managed Delta table** to ingest into\n2. **A service principal id and secret** with `MODIFY` and `SELECT` on the target table\n3. **The Zerobus server endpoint** for your workspace region\n4. **The Zerobus Ingest SDK** installed for your target language\n\nSee [references\u002F1-setup-and-authentication.md](references\u002F1-setup-and-authentication.md) for complete setup instructions.\n\n---\n\n## Minimal Python Example (JSON)\n\n```python\nfrom zerobus.sdk.sync import ZerobusSdk\nfrom zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties\n\nsdk = ZerobusSdk(server_endpoint, workspace_url)\noptions = StreamConfigurationOptions(record_type=RecordType.JSON)\ntable_props = TableProperties(table_name)\n\nstream = sdk.create_stream(client_id, client_secret, table_props, options)\ntry:\n    # Pass a dict for JSON streams; the SDK serializes it.\n    record = {\"device_name\": \"sensor-1\", \"temp\": 22, \"humidity\": 55}\n    offset = stream.ingest_record_offset(record)\n    stream.wait_for_offset(offset)  # Block until durably written\nfinally:\n    stream.close()\n```\n\n---\n\n## Detailed guides\n\n| Topic | File | When to Read |\n|-------|------|--------------|\n| Setup & Auth | [references\u002F1-setup-and-authentication.md](references\u002F1-setup-and-authentication.md) | Endpoint formats, service principals, SDK install |\n| Python Client | [references\u002F2-python-client.md](references\u002F2-python-client.md) | Sync\u002Fasync Python, JSON and Protobuf flows, reusable client class |\n| Multi-Language | [references\u002F3-multilanguage-clients.md](references\u002F3-multilanguage-clients.md) | Java, Go, TypeScript, Rust SDK examples |\n| Protobuf Schema | [references\u002F4-protobuf-schema.md](references\u002F4-protobuf-schema.md) | Generate .proto from UC table, compile, type mappings |\n| Operations & Limits | [references\u002F5-operations-and-limits.md](references\u002F5-operations-and-limits.md) | ACK handling, retries, reconnection, throughput limits, constraints |\n\n---\n\nYou must always follow all the steps in the Workflow\n\n## Workflow\n0. **Display the plan of your execution**\n1. **Determine the type of client** and serialization (JSON for prototypes\u002Fsimple schemas; Protobuf for production\u002Ftype safety)\n2. **Get schema**: for Protobuf, generate the `.proto` per [references\u002F4-protobuf-schema.md](references\u002F4-protobuf-schema.md); for JSON, ensure record keys match the target table columns\n3. **Write Python code to a local file** following the instructions in the relevant guide (e.g., `scripts\u002Fzerobus_ingest.py`)\n4. **Upload to workspace**: `databricks workspace import-dir .\u002Fscripts \u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fscripts`\n5. **Execute on Databricks** using a job or notebook\n6. **If execution fails**: Edit the local file, re-upload, and re-execute\n\n---\n\n## Important\n- Install the SDK through the job\u002Fcluster library configuration, not by pip-installing at runtime in a notebook.\n- **Serverless limitation**: The Zerobus SDK cannot pip-install on serverless compute. Use classic compute clusters, or use the [Zerobus REST API](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-rest-api) (Beta) for notebook-based ingestion without the SDK.\n- **Explicit table grants**: Service principals need explicit `MODIFY` and `SELECT` grants on the target table. Schema-level inherited permissions may not be sufficient for the `authorization_details` OAuth flow.\n\n---\n\n### Execution Workflow\n\n**Step 1: Upload code to workspace**\n\nGet your workspace username for the `\u003Cuser>` path segment, then upload:\n```bash\nUSER=$(databricks current-user me --output json | jq -r .userName)\ndatabricks workspace import-dir .\u002Fscripts \"\u002FWorkspace\u002FUsers\u002F$USER\u002Fscripts\"\n```\n\n**Step 2: Create and run a job**\n```bash\ndatabricks jobs create --json '{\n  \"name\": \"zerobus-ingest\",\n  \"tasks\": [{\n    \"task_key\": \"ingest\",\n    \"spark_python_task\": {\n      \"python_file\": \"\u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fscripts\u002Fzerobus_ingest.py\"\n    },\n    \"new_cluster\": {\n      \"spark_version\": \"16.1.x-scala2.12\",\n      \"node_type_id\": \"i3.xlarge\",\n      \"num_workers\": 0\n    }\n  }]\n}'\n\ndatabricks jobs run-now JOB_ID\n```\n\n**If execution fails:**\n1. Read the error from the job run output\n2. Edit the local Python file to fix the issue\n3. Re-upload: `databricks workspace import-dir .\u002Fscripts \u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fscripts`\n4. Re-run: `databricks jobs run-now JOB_ID`\n\n---\n\n### Installing Libraries\n\nDatabricks provides Spark, pandas, numpy, and common data libraries by default, but the Zerobus SDK is **never pre-installed** — always add `databricks-zerobus-ingest-sdk` to the job\u002Fcluster library config. Only add other libraries if you hit an import error.\n\nAdd to the job configuration:\n```json\n\"libraries\": [\n  {\"pypi\": {\"package\": \"databricks-zerobus-ingest-sdk>=1.0.0\"}}\n]\n```\n\nOr use init scripts in the cluster configuration.\n\n## Timestamp Format\n\nA Delta `TIMESTAMP` column maps to a Protobuf `int64` of **epoch microseconds** (see the type mappings in [references\u002F4-protobuf-schema.md](references\u002F4-protobuf-schema.md)) — supply an integer, not a string:\n\n```python\nfrom datetime import datetime, timezone\n\nevent_time = int(datetime.now(timezone.utc).timestamp() * 1_000_000)  # epoch microseconds\n```\n\n---\n\n## Key Concepts\n\n- **gRPC + Protobuf**: Zerobus uses gRPC as its transport protocol. Any application that can communicate via gRPC and construct Protobuf messages can produce to Zerobus.\n- **JSON or Protobuf serialization**: JSON for quick starts; Protobuf for type safety, forward compatibility, and performance.\n- **At-least-once delivery**: The connector provides at-least-once guarantees. Design consumers to handle duplicates.\n- **Durability ACKs**: Ingestion is acknowledged when records are durably written. Use `ingest_record_offset()` + `wait_for_offset(offset)` for offset-based tracking, an `AckCallback` for asynchronous confirmation, or `flush()` to ensure all buffered records are durably written.\n- **No table management**: Zerobus does not create or alter tables. You must pre-create your target table and manage schema evolution yourself.\n- **Single-AZ durability**: The service runs in a single availability zone. Plan for potential zone outages.\n\n---\n\n## Common Issues\n\n| Issue | Solution |\n|-------|----------|\n| **Connection refused** | Verify server endpoint format matches your cloud (AWS vs Azure). Check firewall allowlists. |\n| **Authentication failed** | Confirm service principal client_id\u002Fsecret. Verify GRANT statements on the target table. |\n| **Schema mismatch** | Ensure record fields match the target table schema exactly. Regenerate .proto if table changed. |\n| **Stream closed unexpectedly** | Implement retry with exponential backoff and stream reinitialization. See [references\u002F5-operations-and-limits.md](references\u002F5-operations-and-limits.md). |\n| **Throughput limits hit** | Max 100 MB\u002Fs and 15,000 rows\u002Fs per stream. Open multiple streams or contact Databricks. |\n| **Region not supported** | Check supported regions in [references\u002F5-operations-and-limits.md](references\u002F5-operations-and-limits.md). |\n| **Table not found** | Ensure table is a managed Delta table in a supported region with correct three-part name. |\n| **SDK install fails on serverless** | The Zerobus SDK cannot be pip-installed on serverless compute. Use classic compute clusters or the REST API (Beta) from notebooks. |\n| **Error 4024 \u002F authorization_details** | Service principal lacks explicit table-level grants. Grant `MODIFY` and `SELECT` directly on the target table — schema-level inherited grants may be insufficient. |\n\n---\n\n## Related Skills\n\n- **[databricks-python-sdk](..\u002Fdatabricks-python-sdk\u002FSKILL.md)** - General SDK patterns and WorkspaceClient for table\u002Fschema management\n- **databricks-pipelines** - Downstream pipeline processing of ingested data\n- **[databricks-unity-catalog](..\u002Fdatabricks-unity-catalog\u002FSKILL.md)** - Managing catalogs, schemas, and tables that Zerobus writes to\n- **[databricks-synthetic-data-gen](..\u002Fdatabricks-synthetic-data-gen\u002FSKILL.md)** - Generate test data to feed into Zerobus producers\n- **databricks-core** - CLI install, profile selection, authentication\n\n## Resources\n\n- [Zerobus Overview](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-overview)\n- [Zerobus Ingest SDK](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-ingest)\n- [Zerobus Limits](https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-limits)\n",{"data":34,"body":38},{"name":4,"description":6,"compatibility":35,"metadata":36},"Requires databricks CLI (>= v1.0.0)",{"version":37},"0.1.0",{"type":39,"children":40},"root",[41,50,56,78,86,120,124,131,136,146,149,155,396,401,404,410,415,457,477,484,517,520,526,531,591,603,606,612,760,763,769,896,899,904,910,1009,1012,1018,1076,1079,1085,1093,1106,1228,1236,1413,1421,1455,1458,1463,1482,1487,1597,1602,1608,1642,1672,1675,1681,1775,1778,1784,1974,1977,1983,2048,2054,2081],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"zerobus-ingest",[47],{"type":48,"value":49},"text","Zerobus Ingest",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54],{"type":48,"value":55},"Build clients that ingest data directly into Databricks Delta tables via the Zerobus gRPC API.",{"type":42,"tag":51,"props":57,"children":58},{},[59,65,67,76],{"type":42,"tag":60,"props":61,"children":62},"strong",{},[63],{"type":48,"value":64},"Status:",{"type":48,"value":66}," Generally Available. Charges are billed against the Jobs Serverless SKU. Check the ",{"type":42,"tag":68,"props":69,"children":73},"a",{"href":70,"rel":71},"https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-overview",[72],"nofollow",[74],{"type":48,"value":75},"Zerobus overview",{"type":48,"value":77}," for the current status of specific features (some, such as Streaming-table targets and Arrow Flight, may be in Beta).",{"type":42,"tag":51,"props":79,"children":80},{},[81],{"type":42,"tag":60,"props":82,"children":83},{},[84],{"type":48,"value":85},"Documentation:",{"type":42,"tag":87,"props":88,"children":89},"ul",{},[90,100,110],{"type":42,"tag":91,"props":92,"children":93},"li",{},[94],{"type":42,"tag":68,"props":95,"children":97},{"href":70,"rel":96},[72],[98],{"type":48,"value":99},"Zerobus Overview",{"type":42,"tag":91,"props":101,"children":102},{},[103],{"type":42,"tag":68,"props":104,"children":107},{"href":105,"rel":106},"https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-ingest",[72],[108],{"type":48,"value":109},"Zerobus Ingest SDK",{"type":42,"tag":91,"props":111,"children":112},{},[113],{"type":42,"tag":68,"props":114,"children":117},{"href":115,"rel":116},"https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-limits",[72],[118],{"type":48,"value":119},"Zerobus Limits",{"type":42,"tag":121,"props":122,"children":123},"hr",{},[],{"type":42,"tag":125,"props":126,"children":128},"h2",{"id":127},"what-is-zerobus-ingest",[129],{"type":48,"value":130},"What Is Zerobus Ingest?",{"type":42,"tag":51,"props":132,"children":133},{},[134],{"type":48,"value":135},"Zerobus Ingest is a serverless connector that enables direct, record-by-record data ingestion into Delta tables via gRPC. It eliminates the need for message bus infrastructure (Kafka, Kinesis, Event Hub) for lakehouse-bound data. The service validates schemas, materializes data to target tables, and sends durability acknowledgments back to the client.",{"type":42,"tag":51,"props":137,"children":138},{},[139,144],{"type":42,"tag":60,"props":140,"children":141},{},[142],{"type":48,"value":143},"Core pattern:",{"type":48,"value":145}," SDK init -> create stream -> ingest records -> handle ACKs -> flush -> close",{"type":42,"tag":121,"props":147,"children":148},{},[],{"type":42,"tag":125,"props":150,"children":152},{"id":151},"quick-decision-what-are-you-building",[153],{"type":48,"value":154},"Quick Decision: What Are You Building?",{"type":42,"tag":156,"props":157,"children":158},"table",{},[159,188],{"type":42,"tag":160,"props":161,"children":162},"thead",{},[163],{"type":42,"tag":164,"props":165,"children":166},"tr",{},[167,173,178,183],{"type":42,"tag":168,"props":169,"children":170},"th",{},[171],{"type":48,"value":172},"Scenario",{"type":42,"tag":168,"props":174,"children":175},{},[176],{"type":48,"value":177},"Language",{"type":42,"tag":168,"props":179,"children":180},{},[181],{"type":48,"value":182},"Serialization",{"type":42,"tag":168,"props":184,"children":185},{},[186],{"type":48,"value":187},"Reference",{"type":42,"tag":189,"props":190,"children":191},"tbody",{},[192,219,250,275,300,324,348,372],{"type":42,"tag":164,"props":193,"children":194},{},[195,201,206,211],{"type":42,"tag":196,"props":197,"children":198},"td",{},[199],{"type":48,"value":200},"Quick prototype \u002F test harness",{"type":42,"tag":196,"props":202,"children":203},{},[204],{"type":48,"value":205},"Python",{"type":42,"tag":196,"props":207,"children":208},{},[209],{"type":48,"value":210},"JSON",{"type":42,"tag":196,"props":212,"children":213},{},[214],{"type":42,"tag":68,"props":215,"children":217},{"href":216},"references\u002F2-python-client.md",[218],{"type":48,"value":216},{"type":42,"tag":164,"props":220,"children":221},{},[222,227,231,236],{"type":42,"tag":196,"props":223,"children":224},{},[225],{"type":48,"value":226},"Production Python producer",{"type":42,"tag":196,"props":228,"children":229},{},[230],{"type":48,"value":205},{"type":42,"tag":196,"props":232,"children":233},{},[234],{"type":48,"value":235},"Protobuf",{"type":42,"tag":196,"props":237,"children":238},{},[239,243,245],{"type":42,"tag":68,"props":240,"children":241},{"href":216},[242],{"type":48,"value":216},{"type":48,"value":244}," + ",{"type":42,"tag":68,"props":246,"children":248},{"href":247},"references\u002F4-protobuf-schema.md",[249],{"type":48,"value":247},{"type":42,"tag":164,"props":251,"children":252},{},[253,258,263,267],{"type":42,"tag":196,"props":254,"children":255},{},[256],{"type":48,"value":257},"JVM microservice",{"type":42,"tag":196,"props":259,"children":260},{},[261],{"type":48,"value":262},"Java",{"type":42,"tag":196,"props":264,"children":265},{},[266],{"type":48,"value":235},{"type":42,"tag":196,"props":268,"children":269},{},[270],{"type":42,"tag":68,"props":271,"children":273},{"href":272},"references\u002F3-multilanguage-clients.md",[274],{"type":48,"value":272},{"type":42,"tag":164,"props":276,"children":277},{},[278,283,288,293],{"type":42,"tag":196,"props":279,"children":280},{},[281],{"type":48,"value":282},"Go service",{"type":42,"tag":196,"props":284,"children":285},{},[286],{"type":48,"value":287},"Go",{"type":42,"tag":196,"props":289,"children":290},{},[291],{"type":48,"value":292},"JSON or Protobuf",{"type":42,"tag":196,"props":294,"children":295},{},[296],{"type":42,"tag":68,"props":297,"children":298},{"href":272},[299],{"type":48,"value":272},{"type":42,"tag":164,"props":301,"children":302},{},[303,308,313,317],{"type":42,"tag":196,"props":304,"children":305},{},[306],{"type":48,"value":307},"Node.js \u002F TypeScript app",{"type":42,"tag":196,"props":309,"children":310},{},[311],{"type":48,"value":312},"TypeScript",{"type":42,"tag":196,"props":314,"children":315},{},[316],{"type":48,"value":210},{"type":42,"tag":196,"props":318,"children":319},{},[320],{"type":42,"tag":68,"props":321,"children":322},{"href":272},[323],{"type":48,"value":272},{"type":42,"tag":164,"props":325,"children":326},{},[327,332,337,341],{"type":42,"tag":196,"props":328,"children":329},{},[330],{"type":48,"value":331},"High-performance system service",{"type":42,"tag":196,"props":333,"children":334},{},[335],{"type":48,"value":336},"Rust",{"type":42,"tag":196,"props":338,"children":339},{},[340],{"type":48,"value":292},{"type":42,"tag":196,"props":342,"children":343},{},[344],{"type":42,"tag":68,"props":345,"children":346},{"href":272},[347],{"type":48,"value":272},{"type":42,"tag":164,"props":349,"children":350},{},[351,356,361,365],{"type":42,"tag":196,"props":352,"children":353},{},[354],{"type":48,"value":355},"Schema generation from UC table",{"type":42,"tag":196,"props":357,"children":358},{},[359],{"type":48,"value":360},"Any",{"type":42,"tag":196,"props":362,"children":363},{},[364],{"type":48,"value":235},{"type":42,"tag":196,"props":366,"children":367},{},[368],{"type":42,"tag":68,"props":369,"children":370},{"href":247},[371],{"type":48,"value":247},{"type":42,"tag":164,"props":373,"children":374},{},[375,380,384,388],{"type":42,"tag":196,"props":376,"children":377},{},[378],{"type":48,"value":379},"Retry \u002F reconnection logic",{"type":42,"tag":196,"props":381,"children":382},{},[383],{"type":48,"value":360},{"type":42,"tag":196,"props":385,"children":386},{},[387],{"type":48,"value":360},{"type":42,"tag":196,"props":389,"children":390},{},[391],{"type":42,"tag":68,"props":392,"children":394},{"href":393},"references\u002F5-operations-and-limits.md",[395],{"type":48,"value":393},{"type":42,"tag":51,"props":397,"children":398},{},[399],{"type":48,"value":400},"If not specified, default to python.",{"type":42,"tag":121,"props":402,"children":403},{},[],{"type":42,"tag":125,"props":405,"children":407},{"id":406},"common-libraries",[408],{"type":48,"value":409},"Common Libraries",{"type":42,"tag":51,"props":411,"children":412},{},[413],{"type":48,"value":414},"These libraries are essential for Zerobus data ingestion and are typically NOT pre-installed on Databricks:",{"type":42,"tag":87,"props":416,"children":417},{},[418,428,438],{"type":42,"tag":91,"props":419,"children":420},{},[421,426],{"type":42,"tag":60,"props":422,"children":423},{},[424],{"type":48,"value":425},"databricks-zerobus-ingest-sdk",{"type":48,"value":427},": Zerobus SDK for high-performance streaming ingestion",{"type":42,"tag":91,"props":429,"children":430},{},[431,436],{"type":42,"tag":60,"props":432,"children":433},{},[434],{"type":48,"value":435},"databricks-sdk",{"type":48,"value":437},": Databricks workspace client for authentication and metadata",{"type":42,"tag":91,"props":439,"children":440},{},[441,446,448,455],{"type":42,"tag":60,"props":442,"children":443},{},[444],{"type":48,"value":445},"grpcio-tools",{"type":48,"value":447},": only needed to compile a ",{"type":42,"tag":449,"props":450,"children":452},"code",{"className":451},[],[453],{"type":48,"value":454},".proto",{"type":48,"value":456}," for Protobuf serialization",{"type":42,"tag":51,"props":458,"children":459},{},[460,462,467,469,475],{"type":48,"value":461},"Install them through the ",{"type":42,"tag":60,"props":463,"children":464},{},[465],{"type":48,"value":466},"job\u002Fcluster library configuration",{"type":48,"value":468}," (see ",{"type":42,"tag":68,"props":470,"children":472},{"href":471},"#installing-libraries",[473],{"type":48,"value":474},"Installing Libraries",{"type":48,"value":476}," below) rather than pip-installing at runtime — the SDK cannot pip-install on serverless compute.",{"type":42,"tag":478,"props":479,"children":481},"h3",{"id":480},"grpcio-tools-and-protobuf-compatibility",[482],{"type":48,"value":483},"grpcio-tools and protobuf compatibility",{"type":42,"tag":51,"props":485,"children":486},{},[487,492,494,500,502,507,509,515],{"type":42,"tag":449,"props":488,"children":490},{"className":489},[],[491],{"type":48,"value":445},{"type":48,"value":493}," must match the runtime's ",{"type":42,"tag":449,"props":495,"children":497},{"className":496},[],[498],{"type":48,"value":499},"protobuf",{"type":48,"value":501}," version. If proto compilation fails with a version error, pin a compatible build (for older ",{"type":42,"tag":449,"props":503,"children":505},{"className":504},[],[506],{"type":48,"value":499},{"type":48,"value":508}," 5.26\u002F5.29 runtimes, ",{"type":42,"tag":449,"props":510,"children":512},{"className":511},[],[513],{"type":48,"value":514},"grpcio-tools==1.62.0",{"type":48,"value":516}," works); otherwise use the latest release.",{"type":42,"tag":121,"props":518,"children":519},{},[],{"type":42,"tag":125,"props":521,"children":523},{"id":522},"prerequisites",[524],{"type":48,"value":525},"Prerequisites",{"type":42,"tag":51,"props":527,"children":528},{},[529],{"type":48,"value":530},"You must never execute the skill without confirming the below objects are valid:",{"type":42,"tag":532,"props":533,"children":534},"ol",{},[535,545,571,581],{"type":42,"tag":91,"props":536,"children":537},{},[538,543],{"type":42,"tag":60,"props":539,"children":540},{},[541],{"type":48,"value":542},"A Unity Catalog managed Delta table",{"type":48,"value":544}," to ingest into",{"type":42,"tag":91,"props":546,"children":547},{},[548,553,555,561,563,569],{"type":42,"tag":60,"props":549,"children":550},{},[551],{"type":48,"value":552},"A service principal id and secret",{"type":48,"value":554}," with ",{"type":42,"tag":449,"props":556,"children":558},{"className":557},[],[559],{"type":48,"value":560},"MODIFY",{"type":48,"value":562}," and ",{"type":42,"tag":449,"props":564,"children":566},{"className":565},[],[567],{"type":48,"value":568},"SELECT",{"type":48,"value":570}," on the target table",{"type":42,"tag":91,"props":572,"children":573},{},[574,579],{"type":42,"tag":60,"props":575,"children":576},{},[577],{"type":48,"value":578},"The Zerobus server endpoint",{"type":48,"value":580}," for your workspace region",{"type":42,"tag":91,"props":582,"children":583},{},[584,589],{"type":42,"tag":60,"props":585,"children":586},{},[587],{"type":48,"value":588},"The Zerobus Ingest SDK",{"type":48,"value":590}," installed for your target language",{"type":42,"tag":51,"props":592,"children":593},{},[594,596,601],{"type":48,"value":595},"See ",{"type":42,"tag":68,"props":597,"children":599},{"href":598},"references\u002F1-setup-and-authentication.md",[600],{"type":48,"value":598},{"type":48,"value":602}," for complete setup instructions.",{"type":42,"tag":121,"props":604,"children":605},{},[],{"type":42,"tag":125,"props":607,"children":609},{"id":608},"minimal-python-example-json",[610],{"type":48,"value":611},"Minimal Python Example (JSON)",{"type":42,"tag":613,"props":614,"children":619},"pre",{"className":615,"code":616,"language":617,"meta":618,"style":618},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from zerobus.sdk.sync import ZerobusSdk\nfrom zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties\n\nsdk = ZerobusSdk(server_endpoint, workspace_url)\noptions = StreamConfigurationOptions(record_type=RecordType.JSON)\ntable_props = TableProperties(table_name)\n\nstream = sdk.create_stream(client_id, client_secret, table_props, options)\ntry:\n    # Pass a dict for JSON streams; the SDK serializes it.\n    record = {\"device_name\": \"sensor-1\", \"temp\": 22, \"humidity\": 55}\n    offset = stream.ingest_record_offset(record)\n    stream.wait_for_offset(offset)  # Block until durably written\nfinally:\n    stream.close()\n","python","",[620],{"type":42,"tag":449,"props":621,"children":622},{"__ignoreMap":618},[623,634,643,653,662,671,680,688,697,706,715,724,733,742,751],{"type":42,"tag":624,"props":625,"children":628},"span",{"class":626,"line":627},"line",1,[629],{"type":42,"tag":624,"props":630,"children":631},{},[632],{"type":48,"value":633},"from zerobus.sdk.sync import ZerobusSdk\n",{"type":42,"tag":624,"props":635,"children":637},{"class":626,"line":636},2,[638],{"type":42,"tag":624,"props":639,"children":640},{},[641],{"type":48,"value":642},"from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties\n",{"type":42,"tag":624,"props":644,"children":646},{"class":626,"line":645},3,[647],{"type":42,"tag":624,"props":648,"children":650},{"emptyLinePlaceholder":649},true,[651],{"type":48,"value":652},"\n",{"type":42,"tag":624,"props":654,"children":656},{"class":626,"line":655},4,[657],{"type":42,"tag":624,"props":658,"children":659},{},[660],{"type":48,"value":661},"sdk = ZerobusSdk(server_endpoint, workspace_url)\n",{"type":42,"tag":624,"props":663,"children":665},{"class":626,"line":664},5,[666],{"type":42,"tag":624,"props":667,"children":668},{},[669],{"type":48,"value":670},"options = StreamConfigurationOptions(record_type=RecordType.JSON)\n",{"type":42,"tag":624,"props":672,"children":674},{"class":626,"line":673},6,[675],{"type":42,"tag":624,"props":676,"children":677},{},[678],{"type":48,"value":679},"table_props = TableProperties(table_name)\n",{"type":42,"tag":624,"props":681,"children":683},{"class":626,"line":682},7,[684],{"type":42,"tag":624,"props":685,"children":686},{"emptyLinePlaceholder":649},[687],{"type":48,"value":652},{"type":42,"tag":624,"props":689,"children":691},{"class":626,"line":690},8,[692],{"type":42,"tag":624,"props":693,"children":694},{},[695],{"type":48,"value":696},"stream = sdk.create_stream(client_id, client_secret, table_props, options)\n",{"type":42,"tag":624,"props":698,"children":700},{"class":626,"line":699},9,[701],{"type":42,"tag":624,"props":702,"children":703},{},[704],{"type":48,"value":705},"try:\n",{"type":42,"tag":624,"props":707,"children":709},{"class":626,"line":708},10,[710],{"type":42,"tag":624,"props":711,"children":712},{},[713],{"type":48,"value":714},"    # Pass a dict for JSON streams; the SDK serializes it.\n",{"type":42,"tag":624,"props":716,"children":718},{"class":626,"line":717},11,[719],{"type":42,"tag":624,"props":720,"children":721},{},[722],{"type":48,"value":723},"    record = {\"device_name\": \"sensor-1\", \"temp\": 22, \"humidity\": 55}\n",{"type":42,"tag":624,"props":725,"children":727},{"class":626,"line":726},12,[728],{"type":42,"tag":624,"props":729,"children":730},{},[731],{"type":48,"value":732},"    offset = stream.ingest_record_offset(record)\n",{"type":42,"tag":624,"props":734,"children":736},{"class":626,"line":735},13,[737],{"type":42,"tag":624,"props":738,"children":739},{},[740],{"type":48,"value":741},"    stream.wait_for_offset(offset)  # Block until durably written\n",{"type":42,"tag":624,"props":743,"children":745},{"class":626,"line":744},14,[746],{"type":42,"tag":624,"props":747,"children":748},{},[749],{"type":48,"value":750},"finally:\n",{"type":42,"tag":624,"props":752,"children":754},{"class":626,"line":753},15,[755],{"type":42,"tag":624,"props":756,"children":757},{},[758],{"type":48,"value":759},"    stream.close()\n",{"type":42,"tag":121,"props":761,"children":762},{},[],{"type":42,"tag":125,"props":764,"children":766},{"id":765},"detailed-guides",[767],{"type":48,"value":768},"Detailed guides",{"type":42,"tag":156,"props":770,"children":771},{},[772,793],{"type":42,"tag":160,"props":773,"children":774},{},[775],{"type":42,"tag":164,"props":776,"children":777},{},[778,783,788],{"type":42,"tag":168,"props":779,"children":780},{},[781],{"type":48,"value":782},"Topic",{"type":42,"tag":168,"props":784,"children":785},{},[786],{"type":48,"value":787},"File",{"type":42,"tag":168,"props":789,"children":790},{},[791],{"type":48,"value":792},"When to Read",{"type":42,"tag":189,"props":794,"children":795},{},[796,816,836,856,876],{"type":42,"tag":164,"props":797,"children":798},{},[799,804,811],{"type":42,"tag":196,"props":800,"children":801},{},[802],{"type":48,"value":803},"Setup & Auth",{"type":42,"tag":196,"props":805,"children":806},{},[807],{"type":42,"tag":68,"props":808,"children":809},{"href":598},[810],{"type":48,"value":598},{"type":42,"tag":196,"props":812,"children":813},{},[814],{"type":48,"value":815},"Endpoint formats, service principals, SDK install",{"type":42,"tag":164,"props":817,"children":818},{},[819,824,831],{"type":42,"tag":196,"props":820,"children":821},{},[822],{"type":48,"value":823},"Python Client",{"type":42,"tag":196,"props":825,"children":826},{},[827],{"type":42,"tag":68,"props":828,"children":829},{"href":216},[830],{"type":48,"value":216},{"type":42,"tag":196,"props":832,"children":833},{},[834],{"type":48,"value":835},"Sync\u002Fasync Python, JSON and Protobuf flows, reusable client class",{"type":42,"tag":164,"props":837,"children":838},{},[839,844,851],{"type":42,"tag":196,"props":840,"children":841},{},[842],{"type":48,"value":843},"Multi-Language",{"type":42,"tag":196,"props":845,"children":846},{},[847],{"type":42,"tag":68,"props":848,"children":849},{"href":272},[850],{"type":48,"value":272},{"type":42,"tag":196,"props":852,"children":853},{},[854],{"type":48,"value":855},"Java, Go, TypeScript, Rust SDK examples",{"type":42,"tag":164,"props":857,"children":858},{},[859,864,871],{"type":42,"tag":196,"props":860,"children":861},{},[862],{"type":48,"value":863},"Protobuf Schema",{"type":42,"tag":196,"props":865,"children":866},{},[867],{"type":42,"tag":68,"props":868,"children":869},{"href":247},[870],{"type":48,"value":247},{"type":42,"tag":196,"props":872,"children":873},{},[874],{"type":48,"value":875},"Generate .proto from UC table, compile, type mappings",{"type":42,"tag":164,"props":877,"children":878},{},[879,884,891],{"type":42,"tag":196,"props":880,"children":881},{},[882],{"type":48,"value":883},"Operations & Limits",{"type":42,"tag":196,"props":885,"children":886},{},[887],{"type":42,"tag":68,"props":888,"children":889},{"href":393},[890],{"type":48,"value":393},{"type":42,"tag":196,"props":892,"children":893},{},[894],{"type":48,"value":895},"ACK handling, retries, reconnection, throughput limits, constraints",{"type":42,"tag":121,"props":897,"children":898},{},[],{"type":42,"tag":51,"props":900,"children":901},{},[902],{"type":48,"value":903},"You must always follow all the steps in the Workflow",{"type":42,"tag":125,"props":905,"children":907},{"id":906},"workflow",[908],{"type":48,"value":909},"Workflow",{"type":42,"tag":532,"props":911,"children":913},{"start":912},0,[914,922,932,955,973,989,999],{"type":42,"tag":91,"props":915,"children":916},{},[917],{"type":42,"tag":60,"props":918,"children":919},{},[920],{"type":48,"value":921},"Display the plan of your execution",{"type":42,"tag":91,"props":923,"children":924},{},[925,930],{"type":42,"tag":60,"props":926,"children":927},{},[928],{"type":48,"value":929},"Determine the type of client",{"type":48,"value":931}," and serialization (JSON for prototypes\u002Fsimple schemas; Protobuf for production\u002Ftype safety)",{"type":42,"tag":91,"props":933,"children":934},{},[935,940,942,947,949,953],{"type":42,"tag":60,"props":936,"children":937},{},[938],{"type":48,"value":939},"Get schema",{"type":48,"value":941},": for Protobuf, generate the ",{"type":42,"tag":449,"props":943,"children":945},{"className":944},[],[946],{"type":48,"value":454},{"type":48,"value":948}," per ",{"type":42,"tag":68,"props":950,"children":951},{"href":247},[952],{"type":48,"value":247},{"type":48,"value":954},"; for JSON, ensure record keys match the target table columns",{"type":42,"tag":91,"props":956,"children":957},{},[958,963,965,971],{"type":42,"tag":60,"props":959,"children":960},{},[961],{"type":48,"value":962},"Write Python code to a local file",{"type":48,"value":964}," following the instructions in the relevant guide (e.g., ",{"type":42,"tag":449,"props":966,"children":968},{"className":967},[],[969],{"type":48,"value":970},"scripts\u002Fzerobus_ingest.py",{"type":48,"value":972},")",{"type":42,"tag":91,"props":974,"children":975},{},[976,981,983],{"type":42,"tag":60,"props":977,"children":978},{},[979],{"type":48,"value":980},"Upload to workspace",{"type":48,"value":982},": ",{"type":42,"tag":449,"props":984,"children":986},{"className":985},[],[987],{"type":48,"value":988},"databricks workspace import-dir .\u002Fscripts \u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fscripts",{"type":42,"tag":91,"props":990,"children":991},{},[992,997],{"type":42,"tag":60,"props":993,"children":994},{},[995],{"type":48,"value":996},"Execute on Databricks",{"type":48,"value":998}," using a job or notebook",{"type":42,"tag":91,"props":1000,"children":1001},{},[1002,1007],{"type":42,"tag":60,"props":1003,"children":1004},{},[1005],{"type":48,"value":1006},"If execution fails",{"type":48,"value":1008},": Edit the local file, re-upload, and re-execute",{"type":42,"tag":121,"props":1010,"children":1011},{},[],{"type":42,"tag":125,"props":1013,"children":1015},{"id":1014},"important",[1016],{"type":48,"value":1017},"Important",{"type":42,"tag":87,"props":1019,"children":1020},{},[1021,1026,1045],{"type":42,"tag":91,"props":1022,"children":1023},{},[1024],{"type":48,"value":1025},"Install the SDK through the job\u002Fcluster library configuration, not by pip-installing at runtime in a notebook.",{"type":42,"tag":91,"props":1027,"children":1028},{},[1029,1034,1036,1043],{"type":42,"tag":60,"props":1030,"children":1031},{},[1032],{"type":48,"value":1033},"Serverless limitation",{"type":48,"value":1035},": The Zerobus SDK cannot pip-install on serverless compute. Use classic compute clusters, or use the ",{"type":42,"tag":68,"props":1037,"children":1040},{"href":1038,"rel":1039},"https:\u002F\u002Fdocs.databricks.com\u002Fingestion\u002Fzerobus-rest-api",[72],[1041],{"type":48,"value":1042},"Zerobus REST API",{"type":48,"value":1044}," (Beta) for notebook-based ingestion without the SDK.",{"type":42,"tag":91,"props":1046,"children":1047},{},[1048,1053,1055,1060,1061,1066,1068,1074],{"type":42,"tag":60,"props":1049,"children":1050},{},[1051],{"type":48,"value":1052},"Explicit table grants",{"type":48,"value":1054},": Service principals need explicit ",{"type":42,"tag":449,"props":1056,"children":1058},{"className":1057},[],[1059],{"type":48,"value":560},{"type":48,"value":562},{"type":42,"tag":449,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":48,"value":568},{"type":48,"value":1067}," grants on the target table. Schema-level inherited permissions may not be sufficient for the ",{"type":42,"tag":449,"props":1069,"children":1071},{"className":1070},[],[1072],{"type":48,"value":1073},"authorization_details",{"type":48,"value":1075}," OAuth flow.",{"type":42,"tag":121,"props":1077,"children":1078},{},[],{"type":42,"tag":478,"props":1080,"children":1082},{"id":1081},"execution-workflow",[1083],{"type":48,"value":1084},"Execution Workflow",{"type":42,"tag":51,"props":1086,"children":1087},{},[1088],{"type":42,"tag":60,"props":1089,"children":1090},{},[1091],{"type":48,"value":1092},"Step 1: Upload code to workspace",{"type":42,"tag":51,"props":1094,"children":1095},{},[1096,1098,1104],{"type":48,"value":1097},"Get your workspace username for the ",{"type":42,"tag":449,"props":1099,"children":1101},{"className":1100},[],[1102],{"type":48,"value":1103},"\u003Cuser>",{"type":48,"value":1105}," path segment, then upload:",{"type":42,"tag":613,"props":1107,"children":1111},{"className":1108,"code":1109,"language":1110,"meta":618,"style":618},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","USER=$(databricks current-user me --output json | jq -r .userName)\ndatabricks workspace import-dir .\u002Fscripts \"\u002FWorkspace\u002FUsers\u002F$USER\u002Fscripts\"\n","bash",[1112],{"type":42,"tag":449,"props":1113,"children":1114},{"__ignoreMap":618},[1115,1181],{"type":42,"tag":624,"props":1116,"children":1117},{"class":626,"line":627},[1118,1124,1130,1135,1141,1146,1151,1156,1161,1166,1171,1176],{"type":42,"tag":624,"props":1119,"children":1121},{"style":1120},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1122],{"type":48,"value":1123},"USER",{"type":42,"tag":624,"props":1125,"children":1127},{"style":1126},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1128],{"type":48,"value":1129},"=$(",{"type":42,"tag":624,"props":1131,"children":1133},{"style":1132},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1134],{"type":48,"value":8},{"type":42,"tag":624,"props":1136,"children":1138},{"style":1137},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1139],{"type":48,"value":1140}," current-user",{"type":42,"tag":624,"props":1142,"children":1143},{"style":1137},[1144],{"type":48,"value":1145}," me",{"type":42,"tag":624,"props":1147,"children":1148},{"style":1137},[1149],{"type":48,"value":1150}," --output",{"type":42,"tag":624,"props":1152,"children":1153},{"style":1137},[1154],{"type":48,"value":1155}," json",{"type":42,"tag":624,"props":1157,"children":1158},{"style":1126},[1159],{"type":48,"value":1160}," |",{"type":42,"tag":624,"props":1162,"children":1163},{"style":1132},[1164],{"type":48,"value":1165}," jq",{"type":42,"tag":624,"props":1167,"children":1168},{"style":1137},[1169],{"type":48,"value":1170}," -r",{"type":42,"tag":624,"props":1172,"children":1173},{"style":1137},[1174],{"type":48,"value":1175}," .userName",{"type":42,"tag":624,"props":1177,"children":1178},{"style":1126},[1179],{"type":48,"value":1180},")\n",{"type":42,"tag":624,"props":1182,"children":1183},{"class":626,"line":636},[1184,1188,1193,1198,1203,1208,1213,1218,1223],{"type":42,"tag":624,"props":1185,"children":1186},{"style":1132},[1187],{"type":48,"value":8},{"type":42,"tag":624,"props":1189,"children":1190},{"style":1137},[1191],{"type":48,"value":1192}," workspace",{"type":42,"tag":624,"props":1194,"children":1195},{"style":1137},[1196],{"type":48,"value":1197}," import-dir",{"type":42,"tag":624,"props":1199,"children":1200},{"style":1137},[1201],{"type":48,"value":1202}," .\u002Fscripts",{"type":42,"tag":624,"props":1204,"children":1205},{"style":1126},[1206],{"type":48,"value":1207}," \"",{"type":42,"tag":624,"props":1209,"children":1210},{"style":1137},[1211],{"type":48,"value":1212},"\u002FWorkspace\u002FUsers\u002F",{"type":42,"tag":624,"props":1214,"children":1215},{"style":1120},[1216],{"type":48,"value":1217},"$USER",{"type":42,"tag":624,"props":1219,"children":1220},{"style":1137},[1221],{"type":48,"value":1222},"\u002Fscripts",{"type":42,"tag":624,"props":1224,"children":1225},{"style":1126},[1226],{"type":48,"value":1227},"\"\n",{"type":42,"tag":51,"props":1229,"children":1230},{},[1231],{"type":42,"tag":60,"props":1232,"children":1233},{},[1234],{"type":48,"value":1235},"Step 2: Create and run a job",{"type":42,"tag":613,"props":1237,"children":1239},{"className":1108,"code":1238,"language":1110,"meta":618,"style":618},"databricks jobs create --json '{\n  \"name\": \"zerobus-ingest\",\n  \"tasks\": [{\n    \"task_key\": \"ingest\",\n    \"spark_python_task\": {\n      \"python_file\": \"\u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fscripts\u002Fzerobus_ingest.py\"\n    },\n    \"new_cluster\": {\n      \"spark_version\": \"16.1.x-scala2.12\",\n      \"node_type_id\": \"i3.xlarge\",\n      \"num_workers\": 0\n    }\n  }]\n}'\n\ndatabricks jobs run-now JOB_ID\n",[1240],{"type":42,"tag":449,"props":1241,"children":1242},{"__ignoreMap":618},[1243,1275,1283,1291,1299,1307,1315,1323,1331,1339,1347,1355,1363,1371,1384,1391],{"type":42,"tag":624,"props":1244,"children":1245},{"class":626,"line":627},[1246,1250,1255,1260,1265,1270],{"type":42,"tag":624,"props":1247,"children":1248},{"style":1132},[1249],{"type":48,"value":8},{"type":42,"tag":624,"props":1251,"children":1252},{"style":1137},[1253],{"type":48,"value":1254}," jobs",{"type":42,"tag":624,"props":1256,"children":1257},{"style":1137},[1258],{"type":48,"value":1259}," create",{"type":42,"tag":624,"props":1261,"children":1262},{"style":1137},[1263],{"type":48,"value":1264}," --json",{"type":42,"tag":624,"props":1266,"children":1267},{"style":1126},[1268],{"type":48,"value":1269}," '",{"type":42,"tag":624,"props":1271,"children":1272},{"style":1137},[1273],{"type":48,"value":1274},"{\n",{"type":42,"tag":624,"props":1276,"children":1277},{"class":626,"line":636},[1278],{"type":42,"tag":624,"props":1279,"children":1280},{"style":1137},[1281],{"type":48,"value":1282},"  \"name\": \"zerobus-ingest\",\n",{"type":42,"tag":624,"props":1284,"children":1285},{"class":626,"line":645},[1286],{"type":42,"tag":624,"props":1287,"children":1288},{"style":1137},[1289],{"type":48,"value":1290},"  \"tasks\": [{\n",{"type":42,"tag":624,"props":1292,"children":1293},{"class":626,"line":655},[1294],{"type":42,"tag":624,"props":1295,"children":1296},{"style":1137},[1297],{"type":48,"value":1298},"    \"task_key\": \"ingest\",\n",{"type":42,"tag":624,"props":1300,"children":1301},{"class":626,"line":664},[1302],{"type":42,"tag":624,"props":1303,"children":1304},{"style":1137},[1305],{"type":48,"value":1306},"    \"spark_python_task\": {\n",{"type":42,"tag":624,"props":1308,"children":1309},{"class":626,"line":673},[1310],{"type":42,"tag":624,"props":1311,"children":1312},{"style":1137},[1313],{"type":48,"value":1314},"      \"python_file\": \"\u002FWorkspace\u002FUsers\u002F\u003Cuser>\u002Fscripts\u002Fzerobus_ingest.py\"\n",{"type":42,"tag":624,"props":1316,"children":1317},{"class":626,"line":682},[1318],{"type":42,"tag":624,"props":1319,"children":1320},{"style":1137},[1321],{"type":48,"value":1322},"    },\n",{"type":42,"tag":624,"props":1324,"children":1325},{"class":626,"line":690},[1326],{"type":42,"tag":624,"props":1327,"children":1328},{"style":1137},[1329],{"type":48,"value":1330},"    \"new_cluster\": {\n",{"type":42,"tag":624,"props":1332,"children":1333},{"class":626,"line":699},[1334],{"type":42,"tag":624,"props":1335,"children":1336},{"style":1137},[1337],{"type":48,"value":1338},"      \"spark_version\": \"16.1.x-scala2.12\",\n",{"type":42,"tag":624,"props":1340,"children":1341},{"class":626,"line":708},[1342],{"type":42,"tag":624,"props":1343,"children":1344},{"style":1137},[1345],{"type":48,"value":1346},"      \"node_type_id\": \"i3.xlarge\",\n",{"type":42,"tag":624,"props":1348,"children":1349},{"class":626,"line":717},[1350],{"type":42,"tag":624,"props":1351,"children":1352},{"style":1137},[1353],{"type":48,"value":1354},"      \"num_workers\": 0\n",{"type":42,"tag":624,"props":1356,"children":1357},{"class":626,"line":726},[1358],{"type":42,"tag":624,"props":1359,"children":1360},{"style":1137},[1361],{"type":48,"value":1362},"    }\n",{"type":42,"tag":624,"props":1364,"children":1365},{"class":626,"line":735},[1366],{"type":42,"tag":624,"props":1367,"children":1368},{"style":1137},[1369],{"type":48,"value":1370},"  }]\n",{"type":42,"tag":624,"props":1372,"children":1373},{"class":626,"line":744},[1374,1379],{"type":42,"tag":624,"props":1375,"children":1376},{"style":1137},[1377],{"type":48,"value":1378},"}",{"type":42,"tag":624,"props":1380,"children":1381},{"style":1126},[1382],{"type":48,"value":1383},"'\n",{"type":42,"tag":624,"props":1385,"children":1386},{"class":626,"line":753},[1387],{"type":42,"tag":624,"props":1388,"children":1389},{"emptyLinePlaceholder":649},[1390],{"type":48,"value":652},{"type":42,"tag":624,"props":1392,"children":1394},{"class":626,"line":1393},16,[1395,1399,1403,1408],{"type":42,"tag":624,"props":1396,"children":1397},{"style":1132},[1398],{"type":48,"value":8},{"type":42,"tag":624,"props":1400,"children":1401},{"style":1137},[1402],{"type":48,"value":1254},{"type":42,"tag":624,"props":1404,"children":1405},{"style":1137},[1406],{"type":48,"value":1407}," run-now",{"type":42,"tag":624,"props":1409,"children":1410},{"style":1137},[1411],{"type":48,"value":1412}," JOB_ID\n",{"type":42,"tag":51,"props":1414,"children":1415},{},[1416],{"type":42,"tag":60,"props":1417,"children":1418},{},[1419],{"type":48,"value":1420},"If execution fails:",{"type":42,"tag":532,"props":1422,"children":1423},{},[1424,1429,1434,1444],{"type":42,"tag":91,"props":1425,"children":1426},{},[1427],{"type":48,"value":1428},"Read the error from the job run output",{"type":42,"tag":91,"props":1430,"children":1431},{},[1432],{"type":48,"value":1433},"Edit the local Python file to fix the issue",{"type":42,"tag":91,"props":1435,"children":1436},{},[1437,1439],{"type":48,"value":1438},"Re-upload: ",{"type":42,"tag":449,"props":1440,"children":1442},{"className":1441},[],[1443],{"type":48,"value":988},{"type":42,"tag":91,"props":1445,"children":1446},{},[1447,1449],{"type":48,"value":1448},"Re-run: ",{"type":42,"tag":449,"props":1450,"children":1452},{"className":1451},[],[1453],{"type":48,"value":1454},"databricks jobs run-now JOB_ID",{"type":42,"tag":121,"props":1456,"children":1457},{},[],{"type":42,"tag":478,"props":1459,"children":1461},{"id":1460},"installing-libraries",[1462],{"type":48,"value":474},{"type":42,"tag":51,"props":1464,"children":1465},{},[1466,1468,1473,1475,1480],{"type":48,"value":1467},"Databricks provides Spark, pandas, numpy, and common data libraries by default, but the Zerobus SDK is ",{"type":42,"tag":60,"props":1469,"children":1470},{},[1471],{"type":48,"value":1472},"never pre-installed",{"type":48,"value":1474}," — always add ",{"type":42,"tag":449,"props":1476,"children":1478},{"className":1477},[],[1479],{"type":48,"value":425},{"type":48,"value":1481}," to the job\u002Fcluster library config. Only add other libraries if you hit an import error.",{"type":42,"tag":51,"props":1483,"children":1484},{},[1485],{"type":48,"value":1486},"Add to the job configuration:",{"type":42,"tag":613,"props":1488,"children":1492},{"className":1489,"code":1490,"language":1491,"meta":618,"style":618},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\"libraries\": [\n  {\"pypi\": {\"package\": \"databricks-zerobus-ingest-sdk>=1.0.0\"}}\n]\n","json",[1493],{"type":42,"tag":449,"props":1494,"children":1495},{"__ignoreMap":618},[1496,1522,1589],{"type":42,"tag":624,"props":1497,"children":1498},{"class":626,"line":627},[1499,1504,1509,1513,1517],{"type":42,"tag":624,"props":1500,"children":1501},{"style":1126},[1502],{"type":48,"value":1503},"\"",{"type":42,"tag":624,"props":1505,"children":1506},{"style":1137},[1507],{"type":48,"value":1508},"libraries",{"type":42,"tag":624,"props":1510,"children":1511},{"style":1126},[1512],{"type":48,"value":1503},{"type":42,"tag":624,"props":1514,"children":1515},{"style":1120},[1516],{"type":48,"value":982},{"type":42,"tag":624,"props":1518,"children":1519},{"style":1126},[1520],{"type":48,"value":1521},"[\n",{"type":42,"tag":624,"props":1523,"children":1524},{"class":626,"line":636},[1525,1530,1534,1540,1544,1549,1554,1558,1563,1567,1571,1575,1580,1584],{"type":42,"tag":624,"props":1526,"children":1527},{"style":1126},[1528],{"type":48,"value":1529},"  {",{"type":42,"tag":624,"props":1531,"children":1532},{"style":1126},[1533],{"type":48,"value":1503},{"type":42,"tag":624,"props":1535,"children":1537},{"style":1536},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1538],{"type":48,"value":1539},"pypi",{"type":42,"tag":624,"props":1541,"children":1542},{"style":1126},[1543],{"type":48,"value":1503},{"type":42,"tag":624,"props":1545,"children":1546},{"style":1126},[1547],{"type":48,"value":1548},":",{"type":42,"tag":624,"props":1550,"children":1551},{"style":1126},[1552],{"type":48,"value":1553}," {",{"type":42,"tag":624,"props":1555,"children":1556},{"style":1126},[1557],{"type":48,"value":1503},{"type":42,"tag":624,"props":1559,"children":1560},{"style":1132},[1561],{"type":48,"value":1562},"package",{"type":42,"tag":624,"props":1564,"children":1565},{"style":1126},[1566],{"type":48,"value":1503},{"type":42,"tag":624,"props":1568,"children":1569},{"style":1126},[1570],{"type":48,"value":1548},{"type":42,"tag":624,"props":1572,"children":1573},{"style":1126},[1574],{"type":48,"value":1207},{"type":42,"tag":624,"props":1576,"children":1577},{"style":1137},[1578],{"type":48,"value":1579},"databricks-zerobus-ingest-sdk>=1.0.0",{"type":42,"tag":624,"props":1581,"children":1582},{"style":1126},[1583],{"type":48,"value":1503},{"type":42,"tag":624,"props":1585,"children":1586},{"style":1126},[1587],{"type":48,"value":1588},"}}\n",{"type":42,"tag":624,"props":1590,"children":1591},{"class":626,"line":645},[1592],{"type":42,"tag":624,"props":1593,"children":1594},{"style":1126},[1595],{"type":48,"value":1596},"]\n",{"type":42,"tag":51,"props":1598,"children":1599},{},[1600],{"type":48,"value":1601},"Or use init scripts in the cluster configuration.",{"type":42,"tag":125,"props":1603,"children":1605},{"id":1604},"timestamp-format",[1606],{"type":48,"value":1607},"Timestamp Format",{"type":42,"tag":51,"props":1609,"children":1610},{},[1611,1613,1619,1621,1627,1629,1634,1636,1640],{"type":48,"value":1612},"A Delta ",{"type":42,"tag":449,"props":1614,"children":1616},{"className":1615},[],[1617],{"type":48,"value":1618},"TIMESTAMP",{"type":48,"value":1620}," column maps to a Protobuf ",{"type":42,"tag":449,"props":1622,"children":1624},{"className":1623},[],[1625],{"type":48,"value":1626},"int64",{"type":48,"value":1628}," of ",{"type":42,"tag":60,"props":1630,"children":1631},{},[1632],{"type":48,"value":1633},"epoch microseconds",{"type":48,"value":1635}," (see the type mappings in ",{"type":42,"tag":68,"props":1637,"children":1638},{"href":247},[1639],{"type":48,"value":247},{"type":48,"value":1641},") — supply an integer, not a string:",{"type":42,"tag":613,"props":1643,"children":1645},{"className":615,"code":1644,"language":617,"meta":618,"style":618},"from datetime import datetime, timezone\n\nevent_time = int(datetime.now(timezone.utc).timestamp() * 1_000_000)  # epoch microseconds\n",[1646],{"type":42,"tag":449,"props":1647,"children":1648},{"__ignoreMap":618},[1649,1657,1664],{"type":42,"tag":624,"props":1650,"children":1651},{"class":626,"line":627},[1652],{"type":42,"tag":624,"props":1653,"children":1654},{},[1655],{"type":48,"value":1656},"from datetime import datetime, timezone\n",{"type":42,"tag":624,"props":1658,"children":1659},{"class":626,"line":636},[1660],{"type":42,"tag":624,"props":1661,"children":1662},{"emptyLinePlaceholder":649},[1663],{"type":48,"value":652},{"type":42,"tag":624,"props":1665,"children":1666},{"class":626,"line":645},[1667],{"type":42,"tag":624,"props":1668,"children":1669},{},[1670],{"type":48,"value":1671},"event_time = int(datetime.now(timezone.utc).timestamp() * 1_000_000)  # epoch microseconds\n",{"type":42,"tag":121,"props":1673,"children":1674},{},[],{"type":42,"tag":125,"props":1676,"children":1678},{"id":1677},"key-concepts",[1679],{"type":48,"value":1680},"Key Concepts",{"type":42,"tag":87,"props":1682,"children":1683},{},[1684,1694,1704,1714,1755,1765],{"type":42,"tag":91,"props":1685,"children":1686},{},[1687,1692],{"type":42,"tag":60,"props":1688,"children":1689},{},[1690],{"type":48,"value":1691},"gRPC + Protobuf",{"type":48,"value":1693},": Zerobus uses gRPC as its transport protocol. Any application that can communicate via gRPC and construct Protobuf messages can produce to Zerobus.",{"type":42,"tag":91,"props":1695,"children":1696},{},[1697,1702],{"type":42,"tag":60,"props":1698,"children":1699},{},[1700],{"type":48,"value":1701},"JSON or Protobuf serialization",{"type":48,"value":1703},": JSON for quick starts; Protobuf for type safety, forward compatibility, and performance.",{"type":42,"tag":91,"props":1705,"children":1706},{},[1707,1712],{"type":42,"tag":60,"props":1708,"children":1709},{},[1710],{"type":48,"value":1711},"At-least-once delivery",{"type":48,"value":1713},": The connector provides at-least-once guarantees. Design consumers to handle duplicates.",{"type":42,"tag":91,"props":1715,"children":1716},{},[1717,1722,1724,1730,1731,1737,1739,1745,1747,1753],{"type":42,"tag":60,"props":1718,"children":1719},{},[1720],{"type":48,"value":1721},"Durability ACKs",{"type":48,"value":1723},": Ingestion is acknowledged when records are durably written. Use ",{"type":42,"tag":449,"props":1725,"children":1727},{"className":1726},[],[1728],{"type":48,"value":1729},"ingest_record_offset()",{"type":48,"value":244},{"type":42,"tag":449,"props":1732,"children":1734},{"className":1733},[],[1735],{"type":48,"value":1736},"wait_for_offset(offset)",{"type":48,"value":1738}," for offset-based tracking, an ",{"type":42,"tag":449,"props":1740,"children":1742},{"className":1741},[],[1743],{"type":48,"value":1744},"AckCallback",{"type":48,"value":1746}," for asynchronous confirmation, or ",{"type":42,"tag":449,"props":1748,"children":1750},{"className":1749},[],[1751],{"type":48,"value":1752},"flush()",{"type":48,"value":1754}," to ensure all buffered records are durably written.",{"type":42,"tag":91,"props":1756,"children":1757},{},[1758,1763],{"type":42,"tag":60,"props":1759,"children":1760},{},[1761],{"type":48,"value":1762},"No table management",{"type":48,"value":1764},": Zerobus does not create or alter tables. You must pre-create your target table and manage schema evolution yourself.",{"type":42,"tag":91,"props":1766,"children":1767},{},[1768,1773],{"type":42,"tag":60,"props":1769,"children":1770},{},[1771],{"type":48,"value":1772},"Single-AZ durability",{"type":48,"value":1774},": The service runs in a single availability zone. Plan for potential zone outages.",{"type":42,"tag":121,"props":1776,"children":1777},{},[],{"type":42,"tag":125,"props":1779,"children":1781},{"id":1780},"common-issues",[1782],{"type":48,"value":1783},"Common Issues",{"type":42,"tag":156,"props":1785,"children":1786},{},[1787,1803],{"type":42,"tag":160,"props":1788,"children":1789},{},[1790],{"type":42,"tag":164,"props":1791,"children":1792},{},[1793,1798],{"type":42,"tag":168,"props":1794,"children":1795},{},[1796],{"type":48,"value":1797},"Issue",{"type":42,"tag":168,"props":1799,"children":1800},{},[1801],{"type":48,"value":1802},"Solution",{"type":42,"tag":189,"props":1804,"children":1805},{},[1806,1822,1838,1854,1876,1892,1913,1929,1945],{"type":42,"tag":164,"props":1807,"children":1808},{},[1809,1817],{"type":42,"tag":196,"props":1810,"children":1811},{},[1812],{"type":42,"tag":60,"props":1813,"children":1814},{},[1815],{"type":48,"value":1816},"Connection refused",{"type":42,"tag":196,"props":1818,"children":1819},{},[1820],{"type":48,"value":1821},"Verify server endpoint format matches your cloud (AWS vs Azure). Check firewall allowlists.",{"type":42,"tag":164,"props":1823,"children":1824},{},[1825,1833],{"type":42,"tag":196,"props":1826,"children":1827},{},[1828],{"type":42,"tag":60,"props":1829,"children":1830},{},[1831],{"type":48,"value":1832},"Authentication failed",{"type":42,"tag":196,"props":1834,"children":1835},{},[1836],{"type":48,"value":1837},"Confirm service principal client_id\u002Fsecret. Verify GRANT statements on the target table.",{"type":42,"tag":164,"props":1839,"children":1840},{},[1841,1849],{"type":42,"tag":196,"props":1842,"children":1843},{},[1844],{"type":42,"tag":60,"props":1845,"children":1846},{},[1847],{"type":48,"value":1848},"Schema mismatch",{"type":42,"tag":196,"props":1850,"children":1851},{},[1852],{"type":48,"value":1853},"Ensure record fields match the target table schema exactly. Regenerate .proto if table changed.",{"type":42,"tag":164,"props":1855,"children":1856},{},[1857,1865],{"type":42,"tag":196,"props":1858,"children":1859},{},[1860],{"type":42,"tag":60,"props":1861,"children":1862},{},[1863],{"type":48,"value":1864},"Stream closed unexpectedly",{"type":42,"tag":196,"props":1866,"children":1867},{},[1868,1870,1874],{"type":48,"value":1869},"Implement retry with exponential backoff and stream reinitialization. See ",{"type":42,"tag":68,"props":1871,"children":1872},{"href":393},[1873],{"type":48,"value":393},{"type":48,"value":1875},".",{"type":42,"tag":164,"props":1877,"children":1878},{},[1879,1887],{"type":42,"tag":196,"props":1880,"children":1881},{},[1882],{"type":42,"tag":60,"props":1883,"children":1884},{},[1885],{"type":48,"value":1886},"Throughput limits hit",{"type":42,"tag":196,"props":1888,"children":1889},{},[1890],{"type":48,"value":1891},"Max 100 MB\u002Fs and 15,000 rows\u002Fs per stream. Open multiple streams or contact Databricks.",{"type":42,"tag":164,"props":1893,"children":1894},{},[1895,1903],{"type":42,"tag":196,"props":1896,"children":1897},{},[1898],{"type":42,"tag":60,"props":1899,"children":1900},{},[1901],{"type":48,"value":1902},"Region not supported",{"type":42,"tag":196,"props":1904,"children":1905},{},[1906,1908,1912],{"type":48,"value":1907},"Check supported regions in ",{"type":42,"tag":68,"props":1909,"children":1910},{"href":393},[1911],{"type":48,"value":393},{"type":48,"value":1875},{"type":42,"tag":164,"props":1914,"children":1915},{},[1916,1924],{"type":42,"tag":196,"props":1917,"children":1918},{},[1919],{"type":42,"tag":60,"props":1920,"children":1921},{},[1922],{"type":48,"value":1923},"Table not found",{"type":42,"tag":196,"props":1925,"children":1926},{},[1927],{"type":48,"value":1928},"Ensure table is a managed Delta table in a supported region with correct three-part name.",{"type":42,"tag":164,"props":1930,"children":1931},{},[1932,1940],{"type":42,"tag":196,"props":1933,"children":1934},{},[1935],{"type":42,"tag":60,"props":1936,"children":1937},{},[1938],{"type":48,"value":1939},"SDK install fails on serverless",{"type":42,"tag":196,"props":1941,"children":1942},{},[1943],{"type":48,"value":1944},"The Zerobus SDK cannot be pip-installed on serverless compute. Use classic compute clusters or the REST API (Beta) from notebooks.",{"type":42,"tag":164,"props":1946,"children":1947},{},[1948,1956],{"type":42,"tag":196,"props":1949,"children":1950},{},[1951],{"type":42,"tag":60,"props":1952,"children":1953},{},[1954],{"type":48,"value":1955},"Error 4024 \u002F authorization_details",{"type":42,"tag":196,"props":1957,"children":1958},{},[1959,1961,1966,1967,1972],{"type":48,"value":1960},"Service principal lacks explicit table-level grants. Grant ",{"type":42,"tag":449,"props":1962,"children":1964},{"className":1963},[],[1965],{"type":48,"value":560},{"type":48,"value":562},{"type":42,"tag":449,"props":1968,"children":1970},{"className":1969},[],[1971],{"type":48,"value":568},{"type":48,"value":1973}," directly on the target table — schema-level inherited grants may be insufficient.",{"type":42,"tag":121,"props":1975,"children":1976},{},[],{"type":42,"tag":125,"props":1978,"children":1980},{"id":1979},"related-skills",[1981],{"type":48,"value":1982},"Related Skills",{"type":42,"tag":87,"props":1984,"children":1985},{},[1986,2000,2010,2024,2038],{"type":42,"tag":91,"props":1987,"children":1988},{},[1989,1998],{"type":42,"tag":60,"props":1990,"children":1991},{},[1992],{"type":42,"tag":68,"props":1993,"children":1995},{"href":1994},"..\u002Fdatabricks-python-sdk\u002FSKILL.md",[1996],{"type":48,"value":1997},"databricks-python-sdk",{"type":48,"value":1999}," - General SDK patterns and WorkspaceClient for table\u002Fschema management",{"type":42,"tag":91,"props":2001,"children":2002},{},[2003,2008],{"type":42,"tag":60,"props":2004,"children":2005},{},[2006],{"type":48,"value":2007},"databricks-pipelines",{"type":48,"value":2009}," - Downstream pipeline processing of ingested data",{"type":42,"tag":91,"props":2011,"children":2012},{},[2013,2022],{"type":42,"tag":60,"props":2014,"children":2015},{},[2016],{"type":42,"tag":68,"props":2017,"children":2019},{"href":2018},"..\u002Fdatabricks-unity-catalog\u002FSKILL.md",[2020],{"type":48,"value":2021},"databricks-unity-catalog",{"type":48,"value":2023}," - Managing catalogs, schemas, and tables that Zerobus writes to",{"type":42,"tag":91,"props":2025,"children":2026},{},[2027,2036],{"type":42,"tag":60,"props":2028,"children":2029},{},[2030],{"type":42,"tag":68,"props":2031,"children":2033},{"href":2032},"..\u002Fdatabricks-synthetic-data-gen\u002FSKILL.md",[2034],{"type":48,"value":2035},"databricks-synthetic-data-gen",{"type":48,"value":2037}," - Generate test data to feed into Zerobus producers",{"type":42,"tag":91,"props":2039,"children":2040},{},[2041,2046],{"type":42,"tag":60,"props":2042,"children":2043},{},[2044],{"type":48,"value":2045},"databricks-core",{"type":48,"value":2047}," - CLI install, profile selection, authentication",{"type":42,"tag":125,"props":2049,"children":2051},{"id":2050},"resources",[2052],{"type":48,"value":2053},"Resources",{"type":42,"tag":87,"props":2055,"children":2056},{},[2057,2065,2073],{"type":42,"tag":91,"props":2058,"children":2059},{},[2060],{"type":42,"tag":68,"props":2061,"children":2063},{"href":70,"rel":2062},[72],[2064],{"type":48,"value":99},{"type":42,"tag":91,"props":2066,"children":2067},{},[2068],{"type":42,"tag":68,"props":2069,"children":2071},{"href":105,"rel":2070},[72],[2072],{"type":48,"value":109},{"type":42,"tag":91,"props":2074,"children":2075},{},[2076],{"type":42,"tag":68,"props":2077,"children":2079},{"href":115,"rel":2078},[72],[2080],{"type":48,"value":119},{"type":42,"tag":2082,"props":2083,"children":2084},"style",{},[2085],{"type":48,"value":2086},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2088,"total":2271},[2089,2106,2120,2135,2152,2172,2183,2204,2215,2232,2245,2258],{"slug":2090,"name":2090,"fn":2091,"description":2092,"org":2093,"tags":2094,"stars":23,"repoUrl":24,"updatedAt":2105},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2095,2098,2099,2102],{"name":2096,"slug":2097,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2100,"slug":2101,"type":15},"Knowledge Management","knowledge-management",{"name":2103,"slug":2104,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":2107,"name":2107,"fn":2108,"description":2109,"org":2110,"tags":2111,"stars":23,"repoUrl":24,"updatedAt":2119},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2112,2115,2116],{"name":2113,"slug":2114,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":2117,"slug":2118,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":2121,"name":2121,"fn":2122,"description":2123,"org":2124,"tags":2125,"stars":23,"repoUrl":24,"updatedAt":2134},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2126,2129,2130,2133],{"name":2127,"slug":2128,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":2131,"slug":2132,"type":15},"Docker","docker",{"name":21,"slug":22,"type":15},"2026-07-12T08:04:55.843982",{"slug":2136,"name":2136,"fn":2137,"description":2138,"org":2139,"tags":2140,"stars":23,"repoUrl":24,"updatedAt":2151},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2141,2144,2147,2150],{"name":2142,"slug":2143,"type":15},"Analytics","analytics",{"name":2145,"slug":2146,"type":15},"Dashboards","dashboards",{"name":2148,"slug":2149,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":2153,"name":2153,"fn":2154,"description":2155,"org":2156,"tags":2157,"stars":23,"repoUrl":24,"updatedAt":2171},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2158,2159,2162,2165,2168],{"name":9,"slug":8,"type":15},{"name":2160,"slug":2161,"type":15},"Design","design",{"name":2163,"slug":2164,"type":15},"Frontend","frontend",{"name":2166,"slug":2167,"type":15},"React","react",{"name":2169,"slug":2170,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":2173,"name":2173,"fn":2174,"description":2175,"org":2176,"tags":2177,"stars":23,"repoUrl":24,"updatedAt":2182},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2178,2179,2180,2181],{"name":2142,"slug":2143,"type":15},{"name":2145,"slug":2146,"type":15},{"name":2113,"slug":2114,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":2184,"name":2184,"fn":2185,"description":2186,"org":2187,"tags":2188,"stars":23,"repoUrl":24,"updatedAt":2203},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2189,2190,2193,2196,2199,2200],{"name":9,"slug":8,"type":15},{"name":2191,"slug":2192,"type":15},"FastAPI","fastapi",{"name":2194,"slug":2195,"type":15},"Flask","flask",{"name":2197,"slug":2198,"type":15},"Gradio","gradio",{"name":205,"slug":617,"type":15},{"name":2201,"slug":2202,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",{"slug":2045,"name":2045,"fn":2205,"description":2206,"org":2207,"tags":2208,"stars":23,"repoUrl":24,"updatedAt":2214},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2209,2212,2213],{"name":2210,"slug":2211,"type":15},"Authentication","authentication",{"name":2127,"slug":2128,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":2216,"name":2216,"fn":2217,"description":2218,"org":2219,"tags":2220,"stars":23,"repoUrl":24,"updatedAt":2231},"databricks-dabs","manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2221,2224,2227,2228],{"name":2222,"slug":2223,"type":15},"Automation","automation",{"name":2225,"slug":2226,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":2229,"slug":2230,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":2233,"name":2233,"fn":2234,"description":2235,"org":2236,"tags":2237,"stars":23,"repoUrl":24,"updatedAt":2244},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2238,2239,2240,2241],{"name":2113,"slug":2114,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":2242,"slug":2243,"type":15},"SQL","sql","2026-07-31T05:53:32.561877",{"slug":2246,"name":2246,"fn":2247,"description":2248,"org":2249,"tags":2250,"stars":23,"repoUrl":24,"updatedAt":2257},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2251,2252,2255,2256],{"name":2113,"slug":2114,"type":15},{"name":2253,"slug":2254,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":2242,"slug":2243,"type":15},"2026-07-12T08:04:08.678282",{"slug":2259,"name":2259,"fn":2260,"description":2261,"org":2262,"tags":2263,"stars":23,"repoUrl":24,"updatedAt":2270},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2264,2265,2268],{"name":9,"slug":8,"type":15},{"name":2266,"slug":2267,"type":15},"Documentation","documentation",{"name":187,"slug":2269,"type":15},"reference","2026-07-15T05:41:34.697746",31,{"items":2273,"total":2271},[2274,2281,2287,2294,2301,2309,2316],{"slug":2090,"name":2090,"fn":2091,"description":2092,"org":2275,"tags":2276,"stars":23,"repoUrl":24,"updatedAt":2105},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2277,2278,2279,2280],{"name":2096,"slug":2097,"type":15},{"name":9,"slug":8,"type":15},{"name":2100,"slug":2101,"type":15},{"name":2103,"slug":2104,"type":15},{"slug":2107,"name":2107,"fn":2108,"description":2109,"org":2282,"tags":2283,"stars":23,"repoUrl":24,"updatedAt":2119},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2284,2285,2286],{"name":2113,"slug":2114,"type":15},{"name":9,"slug":8,"type":15},{"name":2117,"slug":2118,"type":15},{"slug":2121,"name":2121,"fn":2122,"description":2123,"org":2288,"tags":2289,"stars":23,"repoUrl":24,"updatedAt":2134},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2290,2291,2292,2293],{"name":2127,"slug":2128,"type":15},{"name":9,"slug":8,"type":15},{"name":2131,"slug":2132,"type":15},{"name":21,"slug":22,"type":15},{"slug":2136,"name":2136,"fn":2137,"description":2138,"org":2295,"tags":2296,"stars":23,"repoUrl":24,"updatedAt":2151},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2297,2298,2299,2300],{"name":2142,"slug":2143,"type":15},{"name":2145,"slug":2146,"type":15},{"name":2148,"slug":2149,"type":15},{"name":9,"slug":8,"type":15},{"slug":2153,"name":2153,"fn":2154,"description":2155,"org":2302,"tags":2303,"stars":23,"repoUrl":24,"updatedAt":2171},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2304,2305,2306,2307,2308],{"name":9,"slug":8,"type":15},{"name":2160,"slug":2161,"type":15},{"name":2163,"slug":2164,"type":15},{"name":2166,"slug":2167,"type":15},{"name":2169,"slug":2170,"type":15},{"slug":2173,"name":2173,"fn":2174,"description":2175,"org":2310,"tags":2311,"stars":23,"repoUrl":24,"updatedAt":2182},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2312,2313,2314,2315],{"name":2142,"slug":2143,"type":15},{"name":2145,"slug":2146,"type":15},{"name":2113,"slug":2114,"type":15},{"name":9,"slug":8,"type":15},{"slug":2184,"name":2184,"fn":2185,"description":2186,"org":2317,"tags":2318,"stars":23,"repoUrl":24,"updatedAt":2203},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2319,2320,2321,2322,2323,2324],{"name":9,"slug":8,"type":15},{"name":2191,"slug":2192,"type":15},{"name":2194,"slug":2195,"type":15},{"name":2197,"slug":2198,"type":15},{"name":205,"slug":617,"type":15},{"name":2201,"slug":2202,"type":15}]