[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-spark-python-data-source":3,"mdc--k34n28-key":35,"related-org-databricks-spark-python-data-source":1101,"related-repo-databricks-spark-python-data-source":1289},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":33,"mdContent":34},"spark-python-data-source","build custom PySpark data sources","Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers\u002Fwriters for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say \"read from X in Spark\" or \"write DataFrame to Y\" and there's no native connector, this skill applies.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"PySpark","pyspark","tag",{"name":17,"slug":18,"type":15},"Data Engineering","data-engineering",{"name":20,"slug":21,"type":15},"Python","python",{"name":23,"slug":24,"type":15},"API Development","api-development",204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-15T05:41:39.79133",null,60,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":28},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fexperimental\u002Fspark-python-data-source","---\nname: spark-python-data-source\ndescription: Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers\u002Fwriters for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say \"read from X in Spark\" or \"write DataFrame to Y\" and there's no native connector, this skill applies.\ncompatibility: Requires databricks CLI (>= v1.0.0)\nmetadata:\n  version: \"0.1.0\"\n---\n\n# spark-python-data-source\n\nBuild custom Python data sources for Apache Spark 4.0+ to read from and write to external systems in batch and streaming modes.\n\n## Instructions\n\nYou are an experienced Spark developer building custom Python data sources using the PySpark DataSource API. Follow these principles and patterns.\n\n### Core Architecture\n\nEach data source follows a flat, single-level inheritance structure:\n\n1. **DataSource class** — entry point that returns readers\u002Fwriters\n2. **Base Reader\u002FWriter classes** — shared logic for options and data processing\n3. **Batch classes** — inherit from base + `DataSourceReader`\u002F`DataSourceWriter`\n4. **Stream classes** — inherit from base + `DataSourceStreamReader`\u002F`DataSourceStreamWriter`\n\nSee [implementation-template.md](references\u002Fimplementation-template.md) for the full annotated skeleton covering all four modes (batch read\u002Fwrite, stream read\u002Fwrite).\n\n### Spark-Specific Design Constraints\n\nThese are specific to the PySpark DataSource API and its driver\u002Fexecutor architecture — general Python best practices (clean code, minimal dependencies, no premature abstraction) still apply but aren't repeated here.\n\n**Flat single-level inheritance only.** PySpark serializes reader\u002Fwriter instances to ship them to executors. Complex inheritance hierarchies and abstract base classes break serialization and make cross-process debugging painful. Use one shared base class mixed with the PySpark interface (e.g., `class YourBatchWriter(YourWriter, DataSourceWriter)`).\n\n**Import third-party libraries inside executor methods.** The `read()` and `write()` methods run on remote executor processes that don't share the driver's Python environment. Top-level imports from the driver won't be available on executors — always import libraries like `requests` or database drivers inside the methods that run on workers.\n\n**Minimize dependencies.** Every package you add must be installed on all executor nodes in the cluster, not just the driver. Prefer the standard library; when external packages are needed, keep them few and well-known.\n\n**No async\u002Fawait** unless the external system's SDK is async-only. The PySpark DataSource API is synchronous, so async adds complexity with no benefit.\n\n### Project Setup\n\nCreate a Python project using a packaging tool such as `uv`, `poetry`, or `hatch`. Examples use `uv` (substitute your tool of choice):\n\n```bash\nuv init your-datasource\ncd your-datasource\nuv add pyspark pytest pytest-spark\n```\n\n```\nyour-datasource\u002F\n├── pyproject.toml\n├── src\u002F\n│   └── your_datasource\u002F\n│       ├── __init__.py\n│       └── datasource.py\n└── tests\u002F\n    ├── conftest.py\n    └── test_datasource.py\n```\n\nRun all commands through the packaging tool so they execute within the correct virtual environment:\n\n```bash\nuv run pytest                       # Run tests\nuv run ruff check src\u002F              # Lint\nuv run ruff format src\u002F             # Format\nuv build                            # Build wheel\n```\n\n### Key Implementation Decisions\n\n**Partitioning Strategy** — choose based on data source characteristics:\n- Time-based: for APIs with temporal data\n- Token-range: for distributed databases\n- ID-range: for paginated APIs\n- See [partitioning-patterns.md](references\u002Fpartitioning-patterns.md) for implementations of each strategy\n\n**Authentication** — support multiple methods in priority order:\n- Databricks Unity Catalog credentials\n- Cloud default credentials (managed identity)\n- Explicit credentials (service principal, API key, username\u002Fpassword)\n- See [authentication-patterns.md](references\u002Fauthentication-patterns.md) for patterns with fallback chains\n\n**Type Conversion** — map between Spark and external types:\n- Handle nulls, timestamps, UUIDs, collections\n- See [type-conversion.md](references\u002Ftype-conversion.md) for bidirectional mapping tables and helpers\n\n**Streaming Offsets** — design for exactly-once semantics:\n- JSON-serializable offset class\n- Non-overlapping partition boundaries\n- See [streaming-patterns.md](references\u002Fstreaming-patterns.md) for offset tracking and watermark patterns\n\n**Error Handling** — implement retries and resilience:\n- Exponential backoff for transient failures (network, rate limits)\n- Circuit breakers for cascading failures\n- See [error-handling.md](references\u002Ferror-handling.md) for retry decorators and failure classification\n\n### Testing\n\n```python\nimport pytest\nfrom unittest.mock import patch, Mock\n\n@pytest.fixture\ndef spark():\n    from pyspark.sql import SparkSession\n    return SparkSession.builder.master(\"local[2]\").getOrCreate()\n\ndef test_data_source_name():\n    assert YourDataSource.name() == \"your-format\"\n\ndef test_writer_sends_data(spark):\n    with patch('requests.post') as mock_post:\n        mock_post.return_value = Mock(status_code=200)\n\n        df = spark.createDataFrame([(1, \"test\")], [\"id\", \"value\"])\n        df.write.format(\"your-format\").option(\"url\", \"http:\u002F\u002Fapi\").save()\n\n        assert mock_post.called\n```\n\nSee [testing-patterns.md](references\u002Ftesting-patterns.md) for unit\u002Fintegration test patterns, fixtures, and running tests.\n\n### Reference Implementations\n\nStudy these for real-world patterns:\n- [cyber-spark-data-connectors](https:\u002F\u002Fgithub.com\u002Falexott\u002Fcyber-spark-data-connectors) — Sentinel, Splunk, REST\n- [spark-cassandra-data-source](https:\u002F\u002Fgithub.com\u002Falexott\u002Fspark-cassandra-data-source) — Token-range partitioning\n- [pyspark-hubspot](https:\u002F\u002Fgithub.com\u002Fdgomez04\u002Fpyspark-hubspot) — REST API pagination\n- [pyspark-mqtt](https:\u002F\u002Fgithub.com\u002Fdatabricks-industry-solutions\u002Fpython-data-sources\u002Ftree\u002Fmain\u002Fmqtt) — Streaming with TLS\n\n## Example Prompts\n\n```\nCreate a Spark data source for reading from MongoDB with sharding support\nBuild a streaming connector for RabbitMQ with at-least-once delivery\nImplement a batch writer for Snowflake with staged uploads\nWrite a data source for REST API with OAuth2 authentication and pagination\n```\n\n## Related\n\n- databricks-testing: Test data sources on Databricks clusters\n- databricks-spark-declarative-pipelines: Use custom sources in DLT pipelines\n- python-dev: Python development best practices\n\n## References\n\n- [implementation-template.md](references\u002Fimplementation-template.md) — Full annotated skeleton; read when starting a new data source\n- [partitioning-patterns.md](references\u002Fpartitioning-patterns.md) — Read when the source supports parallel reads and you need to split work across executors\n- [authentication-patterns.md](references\u002Fauthentication-patterns.md) — Read when the external system requires credentials or tokens\n- [type-conversion.md](references\u002Ftype-conversion.md) — Read when mapping between Spark types and the external system's type system\n- [streaming-patterns.md](references\u002Fstreaming-patterns.md) — Read when implementing `DataSourceStreamReader` or `DataSourceStreamWriter`\n- [error-handling.md](references\u002Ferror-handling.md) — Read when adding retry logic or handling transient failures\n- [testing-patterns.md](references\u002Ftesting-patterns.md) — Read when writing tests; covers unit, integration, and performance testing\n- [production-patterns.md](references\u002Fproduction-patterns.md) — Read when hardening for production: observability, security, input validation\n- [Official Databricks Documentation](https:\u002F\u002Fdocs.databricks.com\u002Fpyspark\u002Fdatasources)\n- [Apache Spark Python DataSource Tutorial](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fapi\u002Fpython\u002Ftutorial\u002Fsql\u002Fpython_data_source.html)\n- [awesome-python-datasources](https:\u002F\u002Fgithub.com\u002Fallisonwang-db\u002Fawesome-python-datasources) — Directory of community implementations\n",{"data":36,"body":40},{"name":4,"description":6,"compatibility":37,"metadata":38},"Requires databricks CLI (>= v1.0.0)",{"version":39},"0.1.0",{"type":41,"children":42},"root",[43,50,56,63,68,75,80,153,167,173,178,196,230,240,250,256,292,367,377,382,489,495,505,536,546,576,586,606,616,641,651,676,682,854,866,872,877,929,935,944,950,968,974,1095],{"type":44,"tag":45,"props":46,"children":47},"element","h1",{"id":4},[48],{"type":49,"value":4},"text",{"type":44,"tag":51,"props":52,"children":53},"p",{},[54],{"type":49,"value":55},"Build custom Python data sources for Apache Spark 4.0+ to read from and write to external systems in batch and streaming modes.",{"type":44,"tag":57,"props":58,"children":60},"h2",{"id":59},"instructions",[61],{"type":49,"value":62},"Instructions",{"type":44,"tag":51,"props":64,"children":65},{},[66],{"type":49,"value":67},"You are an experienced Spark developer building custom Python data sources using the PySpark DataSource API. Follow these principles and patterns.",{"type":44,"tag":69,"props":70,"children":72},"h3",{"id":71},"core-architecture",[73],{"type":49,"value":74},"Core Architecture",{"type":44,"tag":51,"props":76,"children":77},{},[78],{"type":49,"value":79},"Each data source follows a flat, single-level inheritance structure:",{"type":44,"tag":81,"props":82,"children":83},"ol",{},[84,96,106,131],{"type":44,"tag":85,"props":86,"children":87},"li",{},[88,94],{"type":44,"tag":89,"props":90,"children":91},"strong",{},[92],{"type":49,"value":93},"DataSource class",{"type":49,"value":95}," — entry point that returns readers\u002Fwriters",{"type":44,"tag":85,"props":97,"children":98},{},[99,104],{"type":44,"tag":89,"props":100,"children":101},{},[102],{"type":49,"value":103},"Base Reader\u002FWriter classes",{"type":49,"value":105}," — shared logic for options and data processing",{"type":44,"tag":85,"props":107,"children":108},{},[109,114,116,123,125],{"type":44,"tag":89,"props":110,"children":111},{},[112],{"type":49,"value":113},"Batch classes",{"type":49,"value":115}," — inherit from base + ",{"type":44,"tag":117,"props":118,"children":120},"code",{"className":119},[],[121],{"type":49,"value":122},"DataSourceReader",{"type":49,"value":124},"\u002F",{"type":44,"tag":117,"props":126,"children":128},{"className":127},[],[129],{"type":49,"value":130},"DataSourceWriter",{"type":44,"tag":85,"props":132,"children":133},{},[134,139,140,146,147],{"type":44,"tag":89,"props":135,"children":136},{},[137],{"type":49,"value":138},"Stream classes",{"type":49,"value":115},{"type":44,"tag":117,"props":141,"children":143},{"className":142},[],[144],{"type":49,"value":145},"DataSourceStreamReader",{"type":49,"value":124},{"type":44,"tag":117,"props":148,"children":150},{"className":149},[],[151],{"type":49,"value":152},"DataSourceStreamWriter",{"type":44,"tag":51,"props":154,"children":155},{},[156,158,165],{"type":49,"value":157},"See ",{"type":44,"tag":159,"props":160,"children":162},"a",{"href":161},"references\u002Fimplementation-template.md",[163],{"type":49,"value":164},"implementation-template.md",{"type":49,"value":166}," for the full annotated skeleton covering all four modes (batch read\u002Fwrite, stream read\u002Fwrite).",{"type":44,"tag":69,"props":168,"children":170},{"id":169},"spark-specific-design-constraints",[171],{"type":49,"value":172},"Spark-Specific Design Constraints",{"type":44,"tag":51,"props":174,"children":175},{},[176],{"type":49,"value":177},"These are specific to the PySpark DataSource API and its driver\u002Fexecutor architecture — general Python best practices (clean code, minimal dependencies, no premature abstraction) still apply but aren't repeated here.",{"type":44,"tag":51,"props":179,"children":180},{},[181,186,188,194],{"type":44,"tag":89,"props":182,"children":183},{},[184],{"type":49,"value":185},"Flat single-level inheritance only.",{"type":49,"value":187}," PySpark serializes reader\u002Fwriter instances to ship them to executors. Complex inheritance hierarchies and abstract base classes break serialization and make cross-process debugging painful. Use one shared base class mixed with the PySpark interface (e.g., ",{"type":44,"tag":117,"props":189,"children":191},{"className":190},[],[192],{"type":49,"value":193},"class YourBatchWriter(YourWriter, DataSourceWriter)",{"type":49,"value":195},").",{"type":44,"tag":51,"props":197,"children":198},{},[199,204,206,212,214,220,222,228],{"type":44,"tag":89,"props":200,"children":201},{},[202],{"type":49,"value":203},"Import third-party libraries inside executor methods.",{"type":49,"value":205}," The ",{"type":44,"tag":117,"props":207,"children":209},{"className":208},[],[210],{"type":49,"value":211},"read()",{"type":49,"value":213}," and ",{"type":44,"tag":117,"props":215,"children":217},{"className":216},[],[218],{"type":49,"value":219},"write()",{"type":49,"value":221}," methods run on remote executor processes that don't share the driver's Python environment. Top-level imports from the driver won't be available on executors — always import libraries like ",{"type":44,"tag":117,"props":223,"children":225},{"className":224},[],[226],{"type":49,"value":227},"requests",{"type":49,"value":229}," or database drivers inside the methods that run on workers.",{"type":44,"tag":51,"props":231,"children":232},{},[233,238],{"type":44,"tag":89,"props":234,"children":235},{},[236],{"type":49,"value":237},"Minimize dependencies.",{"type":49,"value":239}," Every package you add must be installed on all executor nodes in the cluster, not just the driver. Prefer the standard library; when external packages are needed, keep them few and well-known.",{"type":44,"tag":51,"props":241,"children":242},{},[243,248],{"type":44,"tag":89,"props":244,"children":245},{},[246],{"type":49,"value":247},"No async\u002Fawait",{"type":49,"value":249}," unless the external system's SDK is async-only. The PySpark DataSource API is synchronous, so async adds complexity with no benefit.",{"type":44,"tag":69,"props":251,"children":253},{"id":252},"project-setup",[254],{"type":49,"value":255},"Project Setup",{"type":44,"tag":51,"props":257,"children":258},{},[259,261,267,269,275,277,283,285,290],{"type":49,"value":260},"Create a Python project using a packaging tool such as ",{"type":44,"tag":117,"props":262,"children":264},{"className":263},[],[265],{"type":49,"value":266},"uv",{"type":49,"value":268},", ",{"type":44,"tag":117,"props":270,"children":272},{"className":271},[],[273],{"type":49,"value":274},"poetry",{"type":49,"value":276},", or ",{"type":44,"tag":117,"props":278,"children":280},{"className":279},[],[281],{"type":49,"value":282},"hatch",{"type":49,"value":284},". Examples use ",{"type":44,"tag":117,"props":286,"children":288},{"className":287},[],[289],{"type":49,"value":266},{"type":49,"value":291}," (substitute your tool of choice):",{"type":44,"tag":293,"props":294,"children":299},"pre",{"className":295,"code":296,"language":297,"meta":298,"style":298},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv init your-datasource\ncd your-datasource\nuv add pyspark pytest pytest-spark\n","bash","",[300],{"type":44,"tag":117,"props":301,"children":302},{"__ignoreMap":298},[303,325,339],{"type":44,"tag":304,"props":305,"children":308},"span",{"class":306,"line":307},"line",1,[309,314,320],{"type":44,"tag":304,"props":310,"children":312},{"style":311},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[313],{"type":49,"value":266},{"type":44,"tag":304,"props":315,"children":317},{"style":316},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[318],{"type":49,"value":319}," init",{"type":44,"tag":304,"props":321,"children":322},{"style":316},[323],{"type":49,"value":324}," your-datasource\n",{"type":44,"tag":304,"props":326,"children":328},{"class":306,"line":327},2,[329,335],{"type":44,"tag":304,"props":330,"children":332},{"style":331},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[333],{"type":49,"value":334},"cd",{"type":44,"tag":304,"props":336,"children":337},{"style":316},[338],{"type":49,"value":324},{"type":44,"tag":304,"props":340,"children":342},{"class":306,"line":341},3,[343,347,352,357,362],{"type":44,"tag":304,"props":344,"children":345},{"style":311},[346],{"type":49,"value":266},{"type":44,"tag":304,"props":348,"children":349},{"style":316},[350],{"type":49,"value":351}," add",{"type":44,"tag":304,"props":353,"children":354},{"style":316},[355],{"type":49,"value":356}," pyspark",{"type":44,"tag":304,"props":358,"children":359},{"style":316},[360],{"type":49,"value":361}," pytest",{"type":44,"tag":304,"props":363,"children":364},{"style":316},[365],{"type":49,"value":366}," pytest-spark\n",{"type":44,"tag":293,"props":368,"children":372},{"className":369,"code":371,"language":49},[370],"language-text","your-datasource\u002F\n├── pyproject.toml\n├── src\u002F\n│   └── your_datasource\u002F\n│       ├── __init__.py\n│       └── datasource.py\n└── tests\u002F\n    ├── conftest.py\n    └── test_datasource.py\n",[373],{"type":44,"tag":117,"props":374,"children":375},{"__ignoreMap":298},[376],{"type":49,"value":371},{"type":44,"tag":51,"props":378,"children":379},{},[380],{"type":49,"value":381},"Run all commands through the packaging tool so they execute within the correct virtual environment:",{"type":44,"tag":293,"props":383,"children":385},{"className":295,"code":384,"language":297,"meta":298,"style":298},"uv run pytest                       # Run tests\nuv run ruff check src\u002F              # Lint\nuv run ruff format src\u002F             # Format\nuv build                            # Build wheel\n",[386],{"type":44,"tag":117,"props":387,"children":388},{"__ignoreMap":298},[389,411,442,471],{"type":44,"tag":304,"props":390,"children":391},{"class":306,"line":307},[392,396,401,405],{"type":44,"tag":304,"props":393,"children":394},{"style":311},[395],{"type":49,"value":266},{"type":44,"tag":304,"props":397,"children":398},{"style":316},[399],{"type":49,"value":400}," run",{"type":44,"tag":304,"props":402,"children":403},{"style":316},[404],{"type":49,"value":361},{"type":44,"tag":304,"props":406,"children":408},{"style":407},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[409],{"type":49,"value":410},"                       # Run tests\n",{"type":44,"tag":304,"props":412,"children":413},{"class":306,"line":327},[414,418,422,427,432,437],{"type":44,"tag":304,"props":415,"children":416},{"style":311},[417],{"type":49,"value":266},{"type":44,"tag":304,"props":419,"children":420},{"style":316},[421],{"type":49,"value":400},{"type":44,"tag":304,"props":423,"children":424},{"style":316},[425],{"type":49,"value":426}," ruff",{"type":44,"tag":304,"props":428,"children":429},{"style":316},[430],{"type":49,"value":431}," check",{"type":44,"tag":304,"props":433,"children":434},{"style":316},[435],{"type":49,"value":436}," src\u002F",{"type":44,"tag":304,"props":438,"children":439},{"style":407},[440],{"type":49,"value":441},"              # Lint\n",{"type":44,"tag":304,"props":443,"children":444},{"class":306,"line":341},[445,449,453,457,462,466],{"type":44,"tag":304,"props":446,"children":447},{"style":311},[448],{"type":49,"value":266},{"type":44,"tag":304,"props":450,"children":451},{"style":316},[452],{"type":49,"value":400},{"type":44,"tag":304,"props":454,"children":455},{"style":316},[456],{"type":49,"value":426},{"type":44,"tag":304,"props":458,"children":459},{"style":316},[460],{"type":49,"value":461}," format",{"type":44,"tag":304,"props":463,"children":464},{"style":316},[465],{"type":49,"value":436},{"type":44,"tag":304,"props":467,"children":468},{"style":407},[469],{"type":49,"value":470},"             # Format\n",{"type":44,"tag":304,"props":472,"children":474},{"class":306,"line":473},4,[475,479,484],{"type":44,"tag":304,"props":476,"children":477},{"style":311},[478],{"type":49,"value":266},{"type":44,"tag":304,"props":480,"children":481},{"style":316},[482],{"type":49,"value":483}," build",{"type":44,"tag":304,"props":485,"children":486},{"style":407},[487],{"type":49,"value":488},"                            # Build wheel\n",{"type":44,"tag":69,"props":490,"children":492},{"id":491},"key-implementation-decisions",[493],{"type":49,"value":494},"Key Implementation Decisions",{"type":44,"tag":51,"props":496,"children":497},{},[498,503],{"type":44,"tag":89,"props":499,"children":500},{},[501],{"type":49,"value":502},"Partitioning Strategy",{"type":49,"value":504}," — choose based on data source characteristics:",{"type":44,"tag":506,"props":507,"children":508},"ul",{},[509,514,519,524],{"type":44,"tag":85,"props":510,"children":511},{},[512],{"type":49,"value":513},"Time-based: for APIs with temporal data",{"type":44,"tag":85,"props":515,"children":516},{},[517],{"type":49,"value":518},"Token-range: for distributed databases",{"type":44,"tag":85,"props":520,"children":521},{},[522],{"type":49,"value":523},"ID-range: for paginated APIs",{"type":44,"tag":85,"props":525,"children":526},{},[527,528,534],{"type":49,"value":157},{"type":44,"tag":159,"props":529,"children":531},{"href":530},"references\u002Fpartitioning-patterns.md",[532],{"type":49,"value":533},"partitioning-patterns.md",{"type":49,"value":535}," for implementations of each strategy",{"type":44,"tag":51,"props":537,"children":538},{},[539,544],{"type":44,"tag":89,"props":540,"children":541},{},[542],{"type":49,"value":543},"Authentication",{"type":49,"value":545}," — support multiple methods in priority order:",{"type":44,"tag":506,"props":547,"children":548},{},[549,554,559,564],{"type":44,"tag":85,"props":550,"children":551},{},[552],{"type":49,"value":553},"Databricks Unity Catalog credentials",{"type":44,"tag":85,"props":555,"children":556},{},[557],{"type":49,"value":558},"Cloud default credentials (managed identity)",{"type":44,"tag":85,"props":560,"children":561},{},[562],{"type":49,"value":563},"Explicit credentials (service principal, API key, username\u002Fpassword)",{"type":44,"tag":85,"props":565,"children":566},{},[567,568,574],{"type":49,"value":157},{"type":44,"tag":159,"props":569,"children":571},{"href":570},"references\u002Fauthentication-patterns.md",[572],{"type":49,"value":573},"authentication-patterns.md",{"type":49,"value":575}," for patterns with fallback chains",{"type":44,"tag":51,"props":577,"children":578},{},[579,584],{"type":44,"tag":89,"props":580,"children":581},{},[582],{"type":49,"value":583},"Type Conversion",{"type":49,"value":585}," — map between Spark and external types:",{"type":44,"tag":506,"props":587,"children":588},{},[589,594],{"type":44,"tag":85,"props":590,"children":591},{},[592],{"type":49,"value":593},"Handle nulls, timestamps, UUIDs, collections",{"type":44,"tag":85,"props":595,"children":596},{},[597,598,604],{"type":49,"value":157},{"type":44,"tag":159,"props":599,"children":601},{"href":600},"references\u002Ftype-conversion.md",[602],{"type":49,"value":603},"type-conversion.md",{"type":49,"value":605}," for bidirectional mapping tables and helpers",{"type":44,"tag":51,"props":607,"children":608},{},[609,614],{"type":44,"tag":89,"props":610,"children":611},{},[612],{"type":49,"value":613},"Streaming Offsets",{"type":49,"value":615}," — design for exactly-once semantics:",{"type":44,"tag":506,"props":617,"children":618},{},[619,624,629],{"type":44,"tag":85,"props":620,"children":621},{},[622],{"type":49,"value":623},"JSON-serializable offset class",{"type":44,"tag":85,"props":625,"children":626},{},[627],{"type":49,"value":628},"Non-overlapping partition boundaries",{"type":44,"tag":85,"props":630,"children":631},{},[632,633,639],{"type":49,"value":157},{"type":44,"tag":159,"props":634,"children":636},{"href":635},"references\u002Fstreaming-patterns.md",[637],{"type":49,"value":638},"streaming-patterns.md",{"type":49,"value":640}," for offset tracking and watermark patterns",{"type":44,"tag":51,"props":642,"children":643},{},[644,649],{"type":44,"tag":89,"props":645,"children":646},{},[647],{"type":49,"value":648},"Error Handling",{"type":49,"value":650}," — implement retries and resilience:",{"type":44,"tag":506,"props":652,"children":653},{},[654,659,664],{"type":44,"tag":85,"props":655,"children":656},{},[657],{"type":49,"value":658},"Exponential backoff for transient failures (network, rate limits)",{"type":44,"tag":85,"props":660,"children":661},{},[662],{"type":49,"value":663},"Circuit breakers for cascading failures",{"type":44,"tag":85,"props":665,"children":666},{},[667,668,674],{"type":49,"value":157},{"type":44,"tag":159,"props":669,"children":671},{"href":670},"references\u002Ferror-handling.md",[672],{"type":49,"value":673},"error-handling.md",{"type":49,"value":675}," for retry decorators and failure classification",{"type":44,"tag":69,"props":677,"children":679},{"id":678},"testing",[680],{"type":49,"value":681},"Testing",{"type":44,"tag":293,"props":683,"children":686},{"className":684,"code":685,"language":21,"meta":298,"style":298},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import pytest\nfrom unittest.mock import patch, Mock\n\n@pytest.fixture\ndef spark():\n    from pyspark.sql import SparkSession\n    return SparkSession.builder.master(\"local[2]\").getOrCreate()\n\ndef test_data_source_name():\n    assert YourDataSource.name() == \"your-format\"\n\ndef test_writer_sends_data(spark):\n    with patch('requests.post') as mock_post:\n        mock_post.return_value = Mock(status_code=200)\n\n        df = spark.createDataFrame([(1, \"test\")], [\"id\", \"value\"])\n        df.write.format(\"your-format\").option(\"url\", \"http:\u002F\u002Fapi\").save()\n\n        assert mock_post.called\n",[687],{"type":44,"tag":117,"props":688,"children":689},{"__ignoreMap":298},[690,698,706,715,723,732,741,750,758,767,776,784,793,802,811,819,828,837,845],{"type":44,"tag":304,"props":691,"children":692},{"class":306,"line":307},[693],{"type":44,"tag":304,"props":694,"children":695},{},[696],{"type":49,"value":697},"import pytest\n",{"type":44,"tag":304,"props":699,"children":700},{"class":306,"line":327},[701],{"type":44,"tag":304,"props":702,"children":703},{},[704],{"type":49,"value":705},"from unittest.mock import patch, Mock\n",{"type":44,"tag":304,"props":707,"children":708},{"class":306,"line":341},[709],{"type":44,"tag":304,"props":710,"children":712},{"emptyLinePlaceholder":711},true,[713],{"type":49,"value":714},"\n",{"type":44,"tag":304,"props":716,"children":717},{"class":306,"line":473},[718],{"type":44,"tag":304,"props":719,"children":720},{},[721],{"type":49,"value":722},"@pytest.fixture\n",{"type":44,"tag":304,"props":724,"children":726},{"class":306,"line":725},5,[727],{"type":44,"tag":304,"props":728,"children":729},{},[730],{"type":49,"value":731},"def spark():\n",{"type":44,"tag":304,"props":733,"children":735},{"class":306,"line":734},6,[736],{"type":44,"tag":304,"props":737,"children":738},{},[739],{"type":49,"value":740},"    from pyspark.sql import SparkSession\n",{"type":44,"tag":304,"props":742,"children":744},{"class":306,"line":743},7,[745],{"type":44,"tag":304,"props":746,"children":747},{},[748],{"type":49,"value":749},"    return SparkSession.builder.master(\"local[2]\").getOrCreate()\n",{"type":44,"tag":304,"props":751,"children":753},{"class":306,"line":752},8,[754],{"type":44,"tag":304,"props":755,"children":756},{"emptyLinePlaceholder":711},[757],{"type":49,"value":714},{"type":44,"tag":304,"props":759,"children":761},{"class":306,"line":760},9,[762],{"type":44,"tag":304,"props":763,"children":764},{},[765],{"type":49,"value":766},"def test_data_source_name():\n",{"type":44,"tag":304,"props":768,"children":770},{"class":306,"line":769},10,[771],{"type":44,"tag":304,"props":772,"children":773},{},[774],{"type":49,"value":775},"    assert YourDataSource.name() == \"your-format\"\n",{"type":44,"tag":304,"props":777,"children":779},{"class":306,"line":778},11,[780],{"type":44,"tag":304,"props":781,"children":782},{"emptyLinePlaceholder":711},[783],{"type":49,"value":714},{"type":44,"tag":304,"props":785,"children":787},{"class":306,"line":786},12,[788],{"type":44,"tag":304,"props":789,"children":790},{},[791],{"type":49,"value":792},"def test_writer_sends_data(spark):\n",{"type":44,"tag":304,"props":794,"children":796},{"class":306,"line":795},13,[797],{"type":44,"tag":304,"props":798,"children":799},{},[800],{"type":49,"value":801},"    with patch('requests.post') as mock_post:\n",{"type":44,"tag":304,"props":803,"children":805},{"class":306,"line":804},14,[806],{"type":44,"tag":304,"props":807,"children":808},{},[809],{"type":49,"value":810},"        mock_post.return_value = Mock(status_code=200)\n",{"type":44,"tag":304,"props":812,"children":814},{"class":306,"line":813},15,[815],{"type":44,"tag":304,"props":816,"children":817},{"emptyLinePlaceholder":711},[818],{"type":49,"value":714},{"type":44,"tag":304,"props":820,"children":822},{"class":306,"line":821},16,[823],{"type":44,"tag":304,"props":824,"children":825},{},[826],{"type":49,"value":827},"        df = spark.createDataFrame([(1, \"test\")], [\"id\", \"value\"])\n",{"type":44,"tag":304,"props":829,"children":831},{"class":306,"line":830},17,[832],{"type":44,"tag":304,"props":833,"children":834},{},[835],{"type":49,"value":836},"        df.write.format(\"your-format\").option(\"url\", \"http:\u002F\u002Fapi\").save()\n",{"type":44,"tag":304,"props":838,"children":840},{"class":306,"line":839},18,[841],{"type":44,"tag":304,"props":842,"children":843},{"emptyLinePlaceholder":711},[844],{"type":49,"value":714},{"type":44,"tag":304,"props":846,"children":848},{"class":306,"line":847},19,[849],{"type":44,"tag":304,"props":850,"children":851},{},[852],{"type":49,"value":853},"        assert mock_post.called\n",{"type":44,"tag":51,"props":855,"children":856},{},[857,858,864],{"type":49,"value":157},{"type":44,"tag":159,"props":859,"children":861},{"href":860},"references\u002Ftesting-patterns.md",[862],{"type":49,"value":863},"testing-patterns.md",{"type":49,"value":865}," for unit\u002Fintegration test patterns, fixtures, and running tests.",{"type":44,"tag":69,"props":867,"children":869},{"id":868},"reference-implementations",[870],{"type":49,"value":871},"Reference Implementations",{"type":44,"tag":51,"props":873,"children":874},{},[875],{"type":49,"value":876},"Study these for real-world patterns:",{"type":44,"tag":506,"props":878,"children":879},{},[880,893,905,917],{"type":44,"tag":85,"props":881,"children":882},{},[883,891],{"type":44,"tag":159,"props":884,"children":888},{"href":885,"rel":886},"https:\u002F\u002Fgithub.com\u002Falexott\u002Fcyber-spark-data-connectors",[887],"nofollow",[889],{"type":49,"value":890},"cyber-spark-data-connectors",{"type":49,"value":892}," — Sentinel, Splunk, REST",{"type":44,"tag":85,"props":894,"children":895},{},[896,903],{"type":44,"tag":159,"props":897,"children":900},{"href":898,"rel":899},"https:\u002F\u002Fgithub.com\u002Falexott\u002Fspark-cassandra-data-source",[887],[901],{"type":49,"value":902},"spark-cassandra-data-source",{"type":49,"value":904}," — Token-range partitioning",{"type":44,"tag":85,"props":906,"children":907},{},[908,915],{"type":44,"tag":159,"props":909,"children":912},{"href":910,"rel":911},"https:\u002F\u002Fgithub.com\u002Fdgomez04\u002Fpyspark-hubspot",[887],[913],{"type":49,"value":914},"pyspark-hubspot",{"type":49,"value":916}," — REST API pagination",{"type":44,"tag":85,"props":918,"children":919},{},[920,927],{"type":44,"tag":159,"props":921,"children":924},{"href":922,"rel":923},"https:\u002F\u002Fgithub.com\u002Fdatabricks-industry-solutions\u002Fpython-data-sources\u002Ftree\u002Fmain\u002Fmqtt",[887],[925],{"type":49,"value":926},"pyspark-mqtt",{"type":49,"value":928}," — Streaming with TLS",{"type":44,"tag":57,"props":930,"children":932},{"id":931},"example-prompts",[933],{"type":49,"value":934},"Example Prompts",{"type":44,"tag":293,"props":936,"children":939},{"className":937,"code":938,"language":49},[370],"Create a Spark data source for reading from MongoDB with sharding support\nBuild a streaming connector for RabbitMQ with at-least-once delivery\nImplement a batch writer for Snowflake with staged uploads\nWrite a data source for REST API with OAuth2 authentication and pagination\n",[940],{"type":44,"tag":117,"props":941,"children":942},{"__ignoreMap":298},[943],{"type":49,"value":938},{"type":44,"tag":57,"props":945,"children":947},{"id":946},"related",[948],{"type":49,"value":949},"Related",{"type":44,"tag":506,"props":951,"children":952},{},[953,958,963],{"type":44,"tag":85,"props":954,"children":955},{},[956],{"type":49,"value":957},"databricks-testing: Test data sources on Databricks clusters",{"type":44,"tag":85,"props":959,"children":960},{},[961],{"type":49,"value":962},"databricks-spark-declarative-pipelines: Use custom sources in DLT pipelines",{"type":44,"tag":85,"props":964,"children":965},{},[966],{"type":49,"value":967},"python-dev: Python development best practices",{"type":44,"tag":57,"props":969,"children":971},{"id":970},"references",[972],{"type":49,"value":973},"References",{"type":44,"tag":506,"props":975,"children":976},{},[977,986,995,1004,1013,1034,1043,1052,1063,1073,1083],{"type":44,"tag":85,"props":978,"children":979},{},[980,984],{"type":44,"tag":159,"props":981,"children":982},{"href":161},[983],{"type":49,"value":164},{"type":49,"value":985}," — Full annotated skeleton; read when starting a new data source",{"type":44,"tag":85,"props":987,"children":988},{},[989,993],{"type":44,"tag":159,"props":990,"children":991},{"href":530},[992],{"type":49,"value":533},{"type":49,"value":994}," — Read when the source supports parallel reads and you need to split work across executors",{"type":44,"tag":85,"props":996,"children":997},{},[998,1002],{"type":44,"tag":159,"props":999,"children":1000},{"href":570},[1001],{"type":49,"value":573},{"type":49,"value":1003}," — Read when the external system requires credentials or tokens",{"type":44,"tag":85,"props":1005,"children":1006},{},[1007,1011],{"type":44,"tag":159,"props":1008,"children":1009},{"href":600},[1010],{"type":49,"value":603},{"type":49,"value":1012}," — Read when mapping between Spark types and the external system's type system",{"type":44,"tag":85,"props":1014,"children":1015},{},[1016,1020,1022,1027,1029],{"type":44,"tag":159,"props":1017,"children":1018},{"href":635},[1019],{"type":49,"value":638},{"type":49,"value":1021}," — Read when implementing ",{"type":44,"tag":117,"props":1023,"children":1025},{"className":1024},[],[1026],{"type":49,"value":145},{"type":49,"value":1028}," or ",{"type":44,"tag":117,"props":1030,"children":1032},{"className":1031},[],[1033],{"type":49,"value":152},{"type":44,"tag":85,"props":1035,"children":1036},{},[1037,1041],{"type":44,"tag":159,"props":1038,"children":1039},{"href":670},[1040],{"type":49,"value":673},{"type":49,"value":1042}," — Read when adding retry logic or handling transient failures",{"type":44,"tag":85,"props":1044,"children":1045},{},[1046,1050],{"type":44,"tag":159,"props":1047,"children":1048},{"href":860},[1049],{"type":49,"value":863},{"type":49,"value":1051}," — Read when writing tests; covers unit, integration, and performance testing",{"type":44,"tag":85,"props":1053,"children":1054},{},[1055,1061],{"type":44,"tag":159,"props":1056,"children":1058},{"href":1057},"references\u002Fproduction-patterns.md",[1059],{"type":49,"value":1060},"production-patterns.md",{"type":49,"value":1062}," — Read when hardening for production: observability, security, input validation",{"type":44,"tag":85,"props":1064,"children":1065},{},[1066],{"type":44,"tag":159,"props":1067,"children":1070},{"href":1068,"rel":1069},"https:\u002F\u002Fdocs.databricks.com\u002Fpyspark\u002Fdatasources",[887],[1071],{"type":49,"value":1072},"Official Databricks Documentation",{"type":44,"tag":85,"props":1074,"children":1075},{},[1076],{"type":44,"tag":159,"props":1077,"children":1080},{"href":1078,"rel":1079},"https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fapi\u002Fpython\u002Ftutorial\u002Fsql\u002Fpython_data_source.html",[887],[1081],{"type":49,"value":1082},"Apache Spark Python DataSource Tutorial",{"type":44,"tag":85,"props":1084,"children":1085},{},[1086,1093],{"type":44,"tag":159,"props":1087,"children":1090},{"href":1088,"rel":1089},"https:\u002F\u002Fgithub.com\u002Fallisonwang-db\u002Fawesome-python-datasources",[887],[1091],{"type":49,"value":1092},"awesome-python-datasources",{"type":49,"value":1094}," — Directory of community implementations",{"type":44,"tag":1096,"props":1097,"children":1098},"style",{},[1099],{"type":49,"value":1100},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1102,"total":1288},[1103,1120,1134,1151,1168,1188,1199,1220,1231,1248,1261,1274],{"slug":1104,"name":1104,"fn":1105,"description":1106,"org":1107,"tags":1108,"stars":25,"repoUrl":26,"updatedAt":1119},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1109,1112,1113,1116],{"name":1110,"slug":1111,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1114,"slug":1115,"type":15},"Knowledge Management","knowledge-management",{"name":1117,"slug":1118,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":1121,"name":1121,"fn":1122,"description":1123,"org":1124,"tags":1125,"stars":25,"repoUrl":26,"updatedAt":1133},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1126,1129,1130],{"name":1127,"slug":1128,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":1131,"slug":1132,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":1135,"name":1135,"fn":1136,"description":1137,"org":1138,"tags":1139,"stars":25,"repoUrl":26,"updatedAt":1150},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1140,1143,1144,1147],{"name":1141,"slug":1142,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":1145,"slug":1146,"type":15},"Docker","docker",{"name":1148,"slug":1149,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":1152,"name":1152,"fn":1153,"description":1154,"org":1155,"tags":1156,"stars":25,"repoUrl":26,"updatedAt":1167},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1157,1160,1163,1166],{"name":1158,"slug":1159,"type":15},"Analytics","analytics",{"name":1161,"slug":1162,"type":15},"Dashboards","dashboards",{"name":1164,"slug":1165,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":1169,"name":1169,"fn":1170,"description":1171,"org":1172,"tags":1173,"stars":25,"repoUrl":26,"updatedAt":1187},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1174,1175,1178,1181,1184],{"name":9,"slug":8,"type":15},{"name":1176,"slug":1177,"type":15},"Design","design",{"name":1179,"slug":1180,"type":15},"Frontend","frontend",{"name":1182,"slug":1183,"type":15},"React","react",{"name":1185,"slug":1186,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":1189,"name":1189,"fn":1190,"description":1191,"org":1192,"tags":1193,"stars":25,"repoUrl":26,"updatedAt":1198},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1194,1195,1196,1197],{"name":1158,"slug":1159,"type":15},{"name":1161,"slug":1162,"type":15},{"name":1127,"slug":1128,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":1200,"name":1200,"fn":1201,"description":1202,"org":1203,"tags":1204,"stars":25,"repoUrl":26,"updatedAt":1219},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1205,1206,1209,1212,1215,1216],{"name":9,"slug":8,"type":15},{"name":1207,"slug":1208,"type":15},"FastAPI","fastapi",{"name":1210,"slug":1211,"type":15},"Flask","flask",{"name":1213,"slug":1214,"type":15},"Gradio","gradio",{"name":20,"slug":21,"type":15},{"name":1217,"slug":1218,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",{"slug":1221,"name":1221,"fn":1222,"description":1223,"org":1224,"tags":1225,"stars":25,"repoUrl":26,"updatedAt":1230},"databricks-core","configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1226,1228,1229],{"name":543,"slug":1227,"type":15},"authentication",{"name":1141,"slug":1142,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":1232,"name":1232,"fn":1233,"description":1234,"org":1235,"tags":1236,"stars":25,"repoUrl":26,"updatedAt":1247},"databricks-dabs","manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1237,1240,1243,1244],{"name":1238,"slug":1239,"type":15},"Automation","automation",{"name":1241,"slug":1242,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":1245,"slug":1246,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":1249,"name":1249,"fn":1250,"description":1251,"org":1252,"tags":1253,"stars":25,"repoUrl":26,"updatedAt":1260},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1254,1255,1256,1257],{"name":1127,"slug":1128,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":1258,"slug":1259,"type":15},"SQL","sql","2026-07-31T05:53:32.561877",{"slug":1262,"name":1262,"fn":1263,"description":1264,"org":1265,"tags":1266,"stars":25,"repoUrl":26,"updatedAt":1273},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1267,1268,1271,1272],{"name":1127,"slug":1128,"type":15},{"name":1269,"slug":1270,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":1258,"slug":1259,"type":15},"2026-07-12T08:04:08.678282",{"slug":1275,"name":1275,"fn":1276,"description":1277,"org":1278,"tags":1279,"stars":25,"repoUrl":26,"updatedAt":1287},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1280,1281,1284],{"name":9,"slug":8,"type":15},{"name":1282,"slug":1283,"type":15},"Documentation","documentation",{"name":1285,"slug":1286,"type":15},"Reference","reference","2026-07-15T05:41:34.697746",31,{"items":1290,"total":1288},[1291,1298,1304,1311,1318,1326,1333],{"slug":1104,"name":1104,"fn":1105,"description":1106,"org":1292,"tags":1293,"stars":25,"repoUrl":26,"updatedAt":1119},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1294,1295,1296,1297],{"name":1110,"slug":1111,"type":15},{"name":9,"slug":8,"type":15},{"name":1114,"slug":1115,"type":15},{"name":1117,"slug":1118,"type":15},{"slug":1121,"name":1121,"fn":1122,"description":1123,"org":1299,"tags":1300,"stars":25,"repoUrl":26,"updatedAt":1133},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1301,1302,1303],{"name":1127,"slug":1128,"type":15},{"name":9,"slug":8,"type":15},{"name":1131,"slug":1132,"type":15},{"slug":1135,"name":1135,"fn":1136,"description":1137,"org":1305,"tags":1306,"stars":25,"repoUrl":26,"updatedAt":1150},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1307,1308,1309,1310],{"name":1141,"slug":1142,"type":15},{"name":9,"slug":8,"type":15},{"name":1145,"slug":1146,"type":15},{"name":1148,"slug":1149,"type":15},{"slug":1152,"name":1152,"fn":1153,"description":1154,"org":1312,"tags":1313,"stars":25,"repoUrl":26,"updatedAt":1167},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1314,1315,1316,1317],{"name":1158,"slug":1159,"type":15},{"name":1161,"slug":1162,"type":15},{"name":1164,"slug":1165,"type":15},{"name":9,"slug":8,"type":15},{"slug":1169,"name":1169,"fn":1170,"description":1171,"org":1319,"tags":1320,"stars":25,"repoUrl":26,"updatedAt":1187},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1321,1322,1323,1324,1325],{"name":9,"slug":8,"type":15},{"name":1176,"slug":1177,"type":15},{"name":1179,"slug":1180,"type":15},{"name":1182,"slug":1183,"type":15},{"name":1185,"slug":1186,"type":15},{"slug":1189,"name":1189,"fn":1190,"description":1191,"org":1327,"tags":1328,"stars":25,"repoUrl":26,"updatedAt":1198},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1329,1330,1331,1332],{"name":1158,"slug":1159,"type":15},{"name":1161,"slug":1162,"type":15},{"name":1127,"slug":1128,"type":15},{"name":9,"slug":8,"type":15},{"slug":1200,"name":1200,"fn":1201,"description":1202,"org":1334,"tags":1335,"stars":25,"repoUrl":26,"updatedAt":1219},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1336,1337,1338,1339,1340,1341],{"name":9,"slug":8,"type":15},{"name":1207,"slug":1208,"type":15},{"name":1210,"slug":1211,"type":15},{"name":1213,"slug":1214,"type":15},{"name":20,"slug":21,"type":15},{"name":1217,"slug":1218,"type":15}]