[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-spark-structured-streaming":3,"mdc--8uhy0n-key":33,"related-org-databricks-databricks-spark-structured-streaming":717,"related-repo-databricks-databricks-spark-structured-streaming":903},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"databricks-spark-structured-streaming","build Spark Structured Streaming pipelines","Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"Data Pipeline","data-pipeline",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Spark","spark",204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-12T08:04:16.449925",null,60,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-spark-structured-streaming","---\nname: databricks-spark-structured-streaming\ndescription: \"Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.\"\ncompatibility: Requires databricks CLI (>= v1.0.0)\nmetadata:\n  version: \"0.1.0\"\nparent: databricks-core\n---\n\n# Spark Structured Streaming\n\nProduction-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.\n\n## Quick Start\n\n```python\nfrom pyspark.sql.functions import col, from_json\n\n# Basic Kafka to Delta streaming\ndf = (spark\n    .readStream\n    .format(\"kafka\")\n    .option(\"kafka.bootstrap.servers\", \"broker:9092\")\n    .option(\"subscribe\", \"topic\")\n    .load()\n    .select(from_json(col(\"value\").cast(\"string\"), schema).alias(\"data\"))\n    .select(\"data.*\")\n)\n\ndf.writeStream \\\n    .format(\"delta\") \\\n    .outputMode(\"append\") \\\n    .option(\"checkpointLocation\", \"\u002FVolumes\u002Fcatalog\u002Fcheckpoints\u002Fstream\") \\\n    .trigger(processingTime=\"30 seconds\") \\\n    .start(\"\u002Fdelta\u002Ftarget_table\")\n```\n\n## Core Patterns\n\n| Pattern | Description | Reference |\n|---------|-------------|-----------|\n| **Kafka Streaming** | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See [references\u002Fkafka-streaming.md](references\u002Fkafka-streaming.md) |\n| **Real-Time Mode (RTM)** | Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics | See [references\u002Freal-time-mode.md](references\u002Freal-time-mode.md) |\n| **Lakebase Sink** | Write streaming records into Lakebase Postgres with transactional upserts. Native `format(\"postgresql\")` sink (DBR 18.3+) and manual `foreach` sink as a fallback | See [references\u002Flakebase-sink-python.md](references\u002Flakebase-sink-python.md) |\n| **Stream Joins** | Stream-stream joins, stream-static joins | See [references\u002Fstream-stream-joins.md](references\u002Fstream-stream-joins.md), [references\u002Fstream-static-joins.md](references\u002Fstream-static-joins.md) |\n| **Multi-Sink Writes** | Write to multiple tables, parallel merges | See [references\u002Fmulti-sink-writes.md](references\u002Fmulti-sink-writes.md) |\n| **Merge Operations** | MERGE performance, parallel merges, optimizations | See [references\u002Fmerge-operations.md](references\u002Fmerge-operations.md) |\n\n## Configuration\n\n| Topic | Description | Reference |\n|-------|-------------|-----------|\n| **Checkpoints** | Checkpoint management and best practices | See [references\u002Fcheckpoint-best-practices.md](references\u002Fcheckpoint-best-practices.md) |\n| **Stateful Operations** | Watermarks, state stores, RocksDB configuration | See [references\u002Fstateful-operations.md](references\u002Fstateful-operations.md) |\n| **Trigger & Cost** | Trigger selection, cost optimization, RTM | See [references\u002Ftrigger-and-cost-optimization.md](references\u002Ftrigger-and-cost-optimization.md) |\n\n## Best Practices\n\n| Topic | Description | Reference |\n|-------|-------------|-----------|\n| **Production Checklist** | Comprehensive best practices | See [references\u002Fstreaming-best-practices.md](references\u002Fstreaming-best-practices.md) |\n\n## Production Checklist\n\n- [ ] Checkpoint location is persistent (UC volumes, not DBFS)\n- [ ] Unique checkpoint per stream\n- [ ] Fixed-size cluster (no autoscaling for streaming)\n- [ ] Monitoring configured (input rate, lag, batch duration)\n- [ ] Exactly-once verified (txnVersion\u002FtxnAppId)\n- [ ] Watermark configured for stateful operations\n- [ ] Left joins for stream-static (not inner)\n",{"data":34,"body":39},{"name":4,"description":6,"compatibility":35,"metadata":36,"parent":38},"Requires databricks CLI (>= v1.0.0)",{"version":37},"0.1.0","databricks-core",{"type":40,"children":41},"root",[42,51,57,64,249,255,472,478,578,584,633,638,711],{"type":43,"tag":44,"props":45,"children":47},"element","h1",{"id":46},"spark-structured-streaming",[48],{"type":49,"value":50},"text","Spark Structured Streaming",{"type":43,"tag":52,"props":53,"children":54},"p",{},[55],{"type":49,"value":56},"Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.",{"type":43,"tag":58,"props":59,"children":61},"h2",{"id":60},"quick-start",[62],{"type":49,"value":63},"Quick Start",{"type":43,"tag":65,"props":66,"children":71},"pre",{"className":67,"code":68,"language":69,"meta":70,"style":70},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from pyspark.sql.functions import col, from_json\n\n# Basic Kafka to Delta streaming\ndf = (spark\n    .readStream\n    .format(\"kafka\")\n    .option(\"kafka.bootstrap.servers\", \"broker:9092\")\n    .option(\"subscribe\", \"topic\")\n    .load()\n    .select(from_json(col(\"value\").cast(\"string\"), schema).alias(\"data\"))\n    .select(\"data.*\")\n)\n\ndf.writeStream \\\n    .format(\"delta\") \\\n    .outputMode(\"append\") \\\n    .option(\"checkpointLocation\", \"\u002FVolumes\u002Fcatalog\u002Fcheckpoints\u002Fstream\") \\\n    .trigger(processingTime=\"30 seconds\") \\\n    .start(\"\u002Fdelta\u002Ftarget_table\")\n","python","",[72],{"type":43,"tag":73,"props":74,"children":75},"code",{"__ignoreMap":70},[76,87,97,106,115,124,133,142,151,160,169,178,187,195,204,213,222,231,240],{"type":43,"tag":77,"props":78,"children":81},"span",{"class":79,"line":80},"line",1,[82],{"type":43,"tag":77,"props":83,"children":84},{},[85],{"type":49,"value":86},"from pyspark.sql.functions import col, from_json\n",{"type":43,"tag":77,"props":88,"children":90},{"class":79,"line":89},2,[91],{"type":43,"tag":77,"props":92,"children":94},{"emptyLinePlaceholder":93},true,[95],{"type":49,"value":96},"\n",{"type":43,"tag":77,"props":98,"children":100},{"class":79,"line":99},3,[101],{"type":43,"tag":77,"props":102,"children":103},{},[104],{"type":49,"value":105},"# Basic Kafka to Delta streaming\n",{"type":43,"tag":77,"props":107,"children":109},{"class":79,"line":108},4,[110],{"type":43,"tag":77,"props":111,"children":112},{},[113],{"type":49,"value":114},"df = (spark\n",{"type":43,"tag":77,"props":116,"children":118},{"class":79,"line":117},5,[119],{"type":43,"tag":77,"props":120,"children":121},{},[122],{"type":49,"value":123},"    .readStream\n",{"type":43,"tag":77,"props":125,"children":127},{"class":79,"line":126},6,[128],{"type":43,"tag":77,"props":129,"children":130},{},[131],{"type":49,"value":132},"    .format(\"kafka\")\n",{"type":43,"tag":77,"props":134,"children":136},{"class":79,"line":135},7,[137],{"type":43,"tag":77,"props":138,"children":139},{},[140],{"type":49,"value":141},"    .option(\"kafka.bootstrap.servers\", \"broker:9092\")\n",{"type":43,"tag":77,"props":143,"children":145},{"class":79,"line":144},8,[146],{"type":43,"tag":77,"props":147,"children":148},{},[149],{"type":49,"value":150},"    .option(\"subscribe\", \"topic\")\n",{"type":43,"tag":77,"props":152,"children":154},{"class":79,"line":153},9,[155],{"type":43,"tag":77,"props":156,"children":157},{},[158],{"type":49,"value":159},"    .load()\n",{"type":43,"tag":77,"props":161,"children":163},{"class":79,"line":162},10,[164],{"type":43,"tag":77,"props":165,"children":166},{},[167],{"type":49,"value":168},"    .select(from_json(col(\"value\").cast(\"string\"), schema).alias(\"data\"))\n",{"type":43,"tag":77,"props":170,"children":172},{"class":79,"line":171},11,[173],{"type":43,"tag":77,"props":174,"children":175},{},[176],{"type":49,"value":177},"    .select(\"data.*\")\n",{"type":43,"tag":77,"props":179,"children":181},{"class":79,"line":180},12,[182],{"type":43,"tag":77,"props":183,"children":184},{},[185],{"type":49,"value":186},")\n",{"type":43,"tag":77,"props":188,"children":190},{"class":79,"line":189},13,[191],{"type":43,"tag":77,"props":192,"children":193},{"emptyLinePlaceholder":93},[194],{"type":49,"value":96},{"type":43,"tag":77,"props":196,"children":198},{"class":79,"line":197},14,[199],{"type":43,"tag":77,"props":200,"children":201},{},[202],{"type":49,"value":203},"df.writeStream \\\n",{"type":43,"tag":77,"props":205,"children":207},{"class":79,"line":206},15,[208],{"type":43,"tag":77,"props":209,"children":210},{},[211],{"type":49,"value":212},"    .format(\"delta\") \\\n",{"type":43,"tag":77,"props":214,"children":216},{"class":79,"line":215},16,[217],{"type":43,"tag":77,"props":218,"children":219},{},[220],{"type":49,"value":221},"    .outputMode(\"append\") \\\n",{"type":43,"tag":77,"props":223,"children":225},{"class":79,"line":224},17,[226],{"type":43,"tag":77,"props":227,"children":228},{},[229],{"type":49,"value":230},"    .option(\"checkpointLocation\", \"\u002FVolumes\u002Fcatalog\u002Fcheckpoints\u002Fstream\") \\\n",{"type":43,"tag":77,"props":232,"children":234},{"class":79,"line":233},18,[235],{"type":43,"tag":77,"props":236,"children":237},{},[238],{"type":49,"value":239},"    .trigger(processingTime=\"30 seconds\") \\\n",{"type":43,"tag":77,"props":241,"children":243},{"class":79,"line":242},19,[244],{"type":43,"tag":77,"props":245,"children":246},{},[247],{"type":49,"value":248},"    .start(\"\u002Fdelta\u002Ftarget_table\")\n",{"type":43,"tag":58,"props":250,"children":252},{"id":251},"core-patterns",[253],{"type":49,"value":254},"Core Patterns",{"type":43,"tag":256,"props":257,"children":258},"table",{},[259,283],{"type":43,"tag":260,"props":261,"children":262},"thead",{},[263],{"type":43,"tag":264,"props":265,"children":266},"tr",{},[267,273,278],{"type":43,"tag":268,"props":269,"children":270},"th",{},[271],{"type":49,"value":272},"Pattern",{"type":43,"tag":268,"props":274,"children":275},{},[276],{"type":49,"value":277},"Description",{"type":43,"tag":268,"props":279,"children":280},{},[281],{"type":49,"value":282},"Reference",{"type":43,"tag":284,"props":285,"children":286},"tbody",{},[287,316,349,390,422,447],{"type":43,"tag":264,"props":288,"children":289},{},[290,300,305],{"type":43,"tag":291,"props":292,"children":293},"td",{},[294],{"type":43,"tag":295,"props":296,"children":297},"strong",{},[298],{"type":49,"value":299},"Kafka Streaming",{"type":43,"tag":291,"props":301,"children":302},{},[303],{"type":49,"value":304},"Kafka to Delta, Kafka to Kafka, Real-Time Mode",{"type":43,"tag":291,"props":306,"children":307},{},[308,310],{"type":49,"value":309},"See ",{"type":43,"tag":311,"props":312,"children":314},"a",{"href":313},"references\u002Fkafka-streaming.md",[315],{"type":49,"value":313},{"type":43,"tag":264,"props":317,"children":318},{},[319,327,340],{"type":43,"tag":291,"props":320,"children":321},{},[322],{"type":43,"tag":295,"props":323,"children":324},{},[325],{"type":49,"value":326},"Real-Time Mode (RTM)",{"type":43,"tag":291,"props":328,"children":329},{},[330,332,338],{"type":49,"value":331},"Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), ",{"type":43,"tag":73,"props":333,"children":335},{"className":334},[],[336],{"type":49,"value":337},"transformWithState",{"type":49,"value":339},", observability, error classes, delivery semantics",{"type":43,"tag":291,"props":341,"children":342},{},[343,344],{"type":49,"value":309},{"type":43,"tag":311,"props":345,"children":347},{"href":346},"references\u002Freal-time-mode.md",[348],{"type":49,"value":346},{"type":43,"tag":264,"props":350,"children":351},{},[352,360,381],{"type":43,"tag":291,"props":353,"children":354},{},[355],{"type":43,"tag":295,"props":356,"children":357},{},[358],{"type":49,"value":359},"Lakebase Sink",{"type":43,"tag":291,"props":361,"children":362},{},[363,365,371,373,379],{"type":49,"value":364},"Write streaming records into Lakebase Postgres with transactional upserts. Native ",{"type":43,"tag":73,"props":366,"children":368},{"className":367},[],[369],{"type":49,"value":370},"format(\"postgresql\")",{"type":49,"value":372}," sink (DBR 18.3+) and manual ",{"type":43,"tag":73,"props":374,"children":376},{"className":375},[],[377],{"type":49,"value":378},"foreach",{"type":49,"value":380}," sink as a fallback",{"type":43,"tag":291,"props":382,"children":383},{},[384,385],{"type":49,"value":309},{"type":43,"tag":311,"props":386,"children":388},{"href":387},"references\u002Flakebase-sink-python.md",[389],{"type":49,"value":387},{"type":43,"tag":264,"props":391,"children":392},{},[393,401,406],{"type":43,"tag":291,"props":394,"children":395},{},[396],{"type":43,"tag":295,"props":397,"children":398},{},[399],{"type":49,"value":400},"Stream Joins",{"type":43,"tag":291,"props":402,"children":403},{},[404],{"type":49,"value":405},"Stream-stream joins, stream-static joins",{"type":43,"tag":291,"props":407,"children":408},{},[409,410,415,417],{"type":49,"value":309},{"type":43,"tag":311,"props":411,"children":413},{"href":412},"references\u002Fstream-stream-joins.md",[414],{"type":49,"value":412},{"type":49,"value":416},", ",{"type":43,"tag":311,"props":418,"children":420},{"href":419},"references\u002Fstream-static-joins.md",[421],{"type":49,"value":419},{"type":43,"tag":264,"props":423,"children":424},{},[425,433,438],{"type":43,"tag":291,"props":426,"children":427},{},[428],{"type":43,"tag":295,"props":429,"children":430},{},[431],{"type":49,"value":432},"Multi-Sink Writes",{"type":43,"tag":291,"props":434,"children":435},{},[436],{"type":49,"value":437},"Write to multiple tables, parallel merges",{"type":43,"tag":291,"props":439,"children":440},{},[441,442],{"type":49,"value":309},{"type":43,"tag":311,"props":443,"children":445},{"href":444},"references\u002Fmulti-sink-writes.md",[446],{"type":49,"value":444},{"type":43,"tag":264,"props":448,"children":449},{},[450,458,463],{"type":43,"tag":291,"props":451,"children":452},{},[453],{"type":43,"tag":295,"props":454,"children":455},{},[456],{"type":49,"value":457},"Merge Operations",{"type":43,"tag":291,"props":459,"children":460},{},[461],{"type":49,"value":462},"MERGE performance, parallel merges, optimizations",{"type":43,"tag":291,"props":464,"children":465},{},[466,467],{"type":49,"value":309},{"type":43,"tag":311,"props":468,"children":470},{"href":469},"references\u002Fmerge-operations.md",[471],{"type":49,"value":469},{"type":43,"tag":58,"props":473,"children":475},{"id":474},"configuration",[476],{"type":49,"value":477},"Configuration",{"type":43,"tag":256,"props":479,"children":480},{},[481,500],{"type":43,"tag":260,"props":482,"children":483},{},[484],{"type":43,"tag":264,"props":485,"children":486},{},[487,492,496],{"type":43,"tag":268,"props":488,"children":489},{},[490],{"type":49,"value":491},"Topic",{"type":43,"tag":268,"props":493,"children":494},{},[495],{"type":49,"value":277},{"type":43,"tag":268,"props":497,"children":498},{},[499],{"type":49,"value":282},{"type":43,"tag":284,"props":501,"children":502},{},[503,528,553],{"type":43,"tag":264,"props":504,"children":505},{},[506,514,519],{"type":43,"tag":291,"props":507,"children":508},{},[509],{"type":43,"tag":295,"props":510,"children":511},{},[512],{"type":49,"value":513},"Checkpoints",{"type":43,"tag":291,"props":515,"children":516},{},[517],{"type":49,"value":518},"Checkpoint management and best practices",{"type":43,"tag":291,"props":520,"children":521},{},[522,523],{"type":49,"value":309},{"type":43,"tag":311,"props":524,"children":526},{"href":525},"references\u002Fcheckpoint-best-practices.md",[527],{"type":49,"value":525},{"type":43,"tag":264,"props":529,"children":530},{},[531,539,544],{"type":43,"tag":291,"props":532,"children":533},{},[534],{"type":43,"tag":295,"props":535,"children":536},{},[537],{"type":49,"value":538},"Stateful Operations",{"type":43,"tag":291,"props":540,"children":541},{},[542],{"type":49,"value":543},"Watermarks, state stores, RocksDB configuration",{"type":43,"tag":291,"props":545,"children":546},{},[547,548],{"type":49,"value":309},{"type":43,"tag":311,"props":549,"children":551},{"href":550},"references\u002Fstateful-operations.md",[552],{"type":49,"value":550},{"type":43,"tag":264,"props":554,"children":555},{},[556,564,569],{"type":43,"tag":291,"props":557,"children":558},{},[559],{"type":43,"tag":295,"props":560,"children":561},{},[562],{"type":49,"value":563},"Trigger & Cost",{"type":43,"tag":291,"props":565,"children":566},{},[567],{"type":49,"value":568},"Trigger selection, cost optimization, RTM",{"type":43,"tag":291,"props":570,"children":571},{},[572,573],{"type":49,"value":309},{"type":43,"tag":311,"props":574,"children":576},{"href":575},"references\u002Ftrigger-and-cost-optimization.md",[577],{"type":49,"value":575},{"type":43,"tag":58,"props":579,"children":581},{"id":580},"best-practices",[582],{"type":49,"value":583},"Best Practices",{"type":43,"tag":256,"props":585,"children":586},{},[587,605],{"type":43,"tag":260,"props":588,"children":589},{},[590],{"type":43,"tag":264,"props":591,"children":592},{},[593,597,601],{"type":43,"tag":268,"props":594,"children":595},{},[596],{"type":49,"value":491},{"type":43,"tag":268,"props":598,"children":599},{},[600],{"type":49,"value":277},{"type":43,"tag":268,"props":602,"children":603},{},[604],{"type":49,"value":282},{"type":43,"tag":284,"props":606,"children":607},{},[608],{"type":43,"tag":264,"props":609,"children":610},{},[611,619,624],{"type":43,"tag":291,"props":612,"children":613},{},[614],{"type":43,"tag":295,"props":615,"children":616},{},[617],{"type":49,"value":618},"Production Checklist",{"type":43,"tag":291,"props":620,"children":621},{},[622],{"type":49,"value":623},"Comprehensive best practices",{"type":43,"tag":291,"props":625,"children":626},{},[627,628],{"type":49,"value":309},{"type":43,"tag":311,"props":629,"children":631},{"href":630},"references\u002Fstreaming-best-practices.md",[632],{"type":49,"value":630},{"type":43,"tag":58,"props":634,"children":636},{"id":635},"production-checklist",[637],{"type":49,"value":618},{"type":43,"tag":639,"props":640,"children":643},"ul",{"className":641},[642],"contains-task-list",[644,657,666,675,684,693,702],{"type":43,"tag":645,"props":646,"children":649},"li",{"className":647},[648],"task-list-item",[650,655],{"type":43,"tag":651,"props":652,"children":654},"input",{"disabled":93,"type":653},"checkbox",[],{"type":49,"value":656}," Checkpoint location is persistent (UC volumes, not DBFS)",{"type":43,"tag":645,"props":658,"children":660},{"className":659},[648],[661,664],{"type":43,"tag":651,"props":662,"children":663},{"disabled":93,"type":653},[],{"type":49,"value":665}," Unique checkpoint per stream",{"type":43,"tag":645,"props":667,"children":669},{"className":668},[648],[670,673],{"type":43,"tag":651,"props":671,"children":672},{"disabled":93,"type":653},[],{"type":49,"value":674}," Fixed-size cluster (no autoscaling for streaming)",{"type":43,"tag":645,"props":676,"children":678},{"className":677},[648],[679,682],{"type":43,"tag":651,"props":680,"children":681},{"disabled":93,"type":653},[],{"type":49,"value":683}," Monitoring configured (input rate, lag, batch duration)",{"type":43,"tag":645,"props":685,"children":687},{"className":686},[648],[688,691],{"type":43,"tag":651,"props":689,"children":690},{"disabled":93,"type":653},[],{"type":49,"value":692}," Exactly-once verified (txnVersion\u002FtxnAppId)",{"type":43,"tag":645,"props":694,"children":696},{"className":695},[648],[697,700],{"type":43,"tag":651,"props":698,"children":699},{"disabled":93,"type":653},[],{"type":49,"value":701}," Watermark configured for stateful operations",{"type":43,"tag":645,"props":703,"children":705},{"className":704},[648],[706,709],{"type":43,"tag":651,"props":707,"children":708},{"disabled":93,"type":653},[],{"type":49,"value":710}," Left joins for stream-static (not inner)",{"type":43,"tag":712,"props":713,"children":714},"style",{},[715],{"type":49,"value":716},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":718,"total":902},[719,736,750,767,784,804,815,837,848,863,876,889],{"slug":720,"name":720,"fn":721,"description":722,"org":723,"tags":724,"stars":23,"repoUrl":24,"updatedAt":735},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[725,728,729,732],{"name":726,"slug":727,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":730,"slug":731,"type":15},"Knowledge Management","knowledge-management",{"name":733,"slug":734,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":737,"name":737,"fn":738,"description":739,"org":740,"tags":741,"stars":23,"repoUrl":24,"updatedAt":749},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[742,745,746],{"name":743,"slug":744,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":747,"slug":748,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":751,"name":751,"fn":752,"description":753,"org":754,"tags":755,"stars":23,"repoUrl":24,"updatedAt":766},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[756,759,760,763],{"name":757,"slug":758,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":761,"slug":762,"type":15},"Docker","docker",{"name":764,"slug":765,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":768,"name":768,"fn":769,"description":770,"org":771,"tags":772,"stars":23,"repoUrl":24,"updatedAt":783},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[773,776,779,782],{"name":774,"slug":775,"type":15},"Analytics","analytics",{"name":777,"slug":778,"type":15},"Dashboards","dashboards",{"name":780,"slug":781,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":785,"name":785,"fn":786,"description":787,"org":788,"tags":789,"stars":23,"repoUrl":24,"updatedAt":803},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[790,791,794,797,800],{"name":9,"slug":8,"type":15},{"name":792,"slug":793,"type":15},"Design","design",{"name":795,"slug":796,"type":15},"Frontend","frontend",{"name":798,"slug":799,"type":15},"React","react",{"name":801,"slug":802,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":805,"name":805,"fn":806,"description":807,"org":808,"tags":809,"stars":23,"repoUrl":24,"updatedAt":814},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[810,811,812,813],{"name":774,"slug":775,"type":15},{"name":777,"slug":778,"type":15},{"name":743,"slug":744,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":816,"name":816,"fn":817,"description":818,"org":819,"tags":820,"stars":23,"repoUrl":24,"updatedAt":836},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[821,822,825,828,831,833],{"name":9,"slug":8,"type":15},{"name":823,"slug":824,"type":15},"FastAPI","fastapi",{"name":826,"slug":827,"type":15},"Flask","flask",{"name":829,"slug":830,"type":15},"Gradio","gradio",{"name":832,"slug":69,"type":15},"Python",{"name":834,"slug":835,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",{"slug":38,"name":38,"fn":838,"description":839,"org":840,"tags":841,"stars":23,"repoUrl":24,"updatedAt":847},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[842,845,846],{"name":843,"slug":844,"type":15},"Authentication","authentication",{"name":757,"slug":758,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":849,"name":849,"fn":850,"description":851,"org":852,"tags":853,"stars":23,"repoUrl":24,"updatedAt":862},"databricks-dabs","manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[854,857,858,859],{"name":855,"slug":856,"type":15},"Automation","automation",{"name":477,"slug":474,"type":15},{"name":9,"slug":8,"type":15},{"name":860,"slug":861,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":864,"name":864,"fn":865,"description":866,"org":867,"tags":868,"stars":23,"repoUrl":24,"updatedAt":875},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[869,870,871,872],{"name":743,"slug":744,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":873,"slug":874,"type":15},"SQL","sql","2026-07-31T05:53:32.561877",{"slug":877,"name":877,"fn":878,"description":879,"org":880,"tags":881,"stars":23,"repoUrl":24,"updatedAt":888},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[882,883,886,887],{"name":743,"slug":744,"type":15},{"name":884,"slug":885,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":873,"slug":874,"type":15},"2026-07-12T08:04:08.678282",{"slug":890,"name":890,"fn":891,"description":892,"org":893,"tags":894,"stars":23,"repoUrl":24,"updatedAt":901},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[895,896,899],{"name":9,"slug":8,"type":15},{"name":897,"slug":898,"type":15},"Documentation","documentation",{"name":282,"slug":900,"type":15},"reference","2026-07-15T05:41:34.697746",31,{"items":904,"total":902},[905,912,918,925,932,940,947],{"slug":720,"name":720,"fn":721,"description":722,"org":906,"tags":907,"stars":23,"repoUrl":24,"updatedAt":735},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[908,909,910,911],{"name":726,"slug":727,"type":15},{"name":9,"slug":8,"type":15},{"name":730,"slug":731,"type":15},{"name":733,"slug":734,"type":15},{"slug":737,"name":737,"fn":738,"description":739,"org":913,"tags":914,"stars":23,"repoUrl":24,"updatedAt":749},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[915,916,917],{"name":743,"slug":744,"type":15},{"name":9,"slug":8,"type":15},{"name":747,"slug":748,"type":15},{"slug":751,"name":751,"fn":752,"description":753,"org":919,"tags":920,"stars":23,"repoUrl":24,"updatedAt":766},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[921,922,923,924],{"name":757,"slug":758,"type":15},{"name":9,"slug":8,"type":15},{"name":761,"slug":762,"type":15},{"name":764,"slug":765,"type":15},{"slug":768,"name":768,"fn":769,"description":770,"org":926,"tags":927,"stars":23,"repoUrl":24,"updatedAt":783},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[928,929,930,931],{"name":774,"slug":775,"type":15},{"name":777,"slug":778,"type":15},{"name":780,"slug":781,"type":15},{"name":9,"slug":8,"type":15},{"slug":785,"name":785,"fn":786,"description":787,"org":933,"tags":934,"stars":23,"repoUrl":24,"updatedAt":803},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[935,936,937,938,939],{"name":9,"slug":8,"type":15},{"name":792,"slug":793,"type":15},{"name":795,"slug":796,"type":15},{"name":798,"slug":799,"type":15},{"name":801,"slug":802,"type":15},{"slug":805,"name":805,"fn":806,"description":807,"org":941,"tags":942,"stars":23,"repoUrl":24,"updatedAt":814},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[943,944,945,946],{"name":774,"slug":775,"type":15},{"name":777,"slug":778,"type":15},{"name":743,"slug":744,"type":15},{"name":9,"slug":8,"type":15},{"slug":816,"name":816,"fn":817,"description":818,"org":948,"tags":949,"stars":23,"repoUrl":24,"updatedAt":836},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[950,951,952,953,954,955],{"name":9,"slug":8,"type":15},{"name":823,"slug":824,"type":15},{"name":826,"slug":827,"type":15},{"name":829,"slug":830,"type":15},{"name":832,"slug":69,"type":15},{"name":834,"slug":835,"type":15}]