[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-microsoft-spark-authoring-cli":3,"mdc--4757hy-key":39,"related-org-microsoft-spark-authoring-cli":4345,"related-repo-microsoft-spark-authoring-cli":4541},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":28,"repoUrl":29,"updatedAt":30,"license":31,"forks":32,"topics":33,"repo":34,"sourceUrl":37,"mdContent":38},"spark-authoring-cli","develop Microsoft Fabric Spark workflows","Author Fabric notebook cell code; run a notebook by name and report its status; and author\u002Fcreate Materialized Lake View (MLV) definitions (CREATE MATERIALIZED LAKE VIEW). Use for writing notebook cell code (PySpark, Scala, SparkR, %%sql, %%configure) — a %%sql cell that queries a lakehouse is authoring, not a T-SQL query — and running a notebook via the Jobs API (RunNotebook), including on the success path. Not for Livy sessions or ad-hoc calculations (use `spark-consumption-cli`); to refresh\u002Fschedule an EXISTING MLV, use `mlv-operations-cli`; for plain T-SQL, use `sqldw-consumption-cli`; for a FAILED notebook\u002Fjob, use `spark-operations-cli`. Triggers: \"write notebook code\", \"notebook cell code\", \"%%sql cell\", \"run the notebook named\", \"run notebook\", \"execute notebook\", \"notebookutils\", \"PySpark notebook\", \"%%configure\", \"create a materialized lake view\", \"create MLV\", \"materialized lake view\", \"MLV\", \"CREATE MATERIALIZED LAKE VIEW\", \"MLV incremental refresh\"\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"microsoft","Microsoft","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fmicrosoft.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"Data Pipeline","data-pipeline",{"name":20,"slug":21,"type":15},"CLI","cli",{"name":23,"slug":24,"type":15},"Microsoft Fabric","microsoft-fabric",{"name":26,"slug":27,"type":15},"Spark","spark",858,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric","2026-07-31T05:55:05.048008",null,237,[],{"repoUrl":29,"stars":28,"forks":32,"topics":35,"description":36},[],"A collection of skills and MCP systems to enable users of CLI, VSCode, Claude to operate over Microsoft Fabric","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Ftree\u002FHEAD\u002Fskills\u002Fspark-authoring-cli","---\nname: spark-authoring-cli\ndescription: >\n  Author Fabric notebook cell code; run a notebook by name and report its status; and author\u002Fcreate\n  Materialized Lake View (MLV) definitions (CREATE MATERIALIZED LAKE VIEW).\n  Use for writing notebook cell code (PySpark, Scala, SparkR, %%sql, %%configure) — a %%sql cell that\n  queries a lakehouse is authoring, not a T-SQL query — and running a notebook via the Jobs API\n  (RunNotebook), including on the success path.\n  Not for Livy sessions or ad-hoc calculations (use `spark-consumption-cli`); to refresh\u002Fschedule an\n  EXISTING MLV, use `mlv-operations-cli`; for plain T-SQL, use `sqldw-consumption-cli`; for a FAILED\n  notebook\u002Fjob, use `spark-operations-cli`.\n  Triggers: \"write notebook code\", \"notebook cell code\", \"%%sql cell\", \"run the notebook named\",\n  \"run notebook\", \"execute notebook\", \"notebookutils\", \"PySpark notebook\", \"%%configure\",\n  \"create a materialized lake view\", \"create MLV\", \"materialized lake view\", \"MLV\",\n  \"CREATE MATERIALIZED LAKE VIEW\", \"MLV incremental refresh\"\n---\n\n> **Update Check — ONCE PER SESSION (mandatory)**\n> The first time this skill is used in a session, run the **check-updates** skill before proceeding.\n> - **GitHub Copilot CLI \u002F VS Code**: invoke the `check-updates` skill.\n> - **Claude Code \u002F Cowork \u002F Cursor \u002F Windsurf \u002F Codex**: compare local vs remote package.json version.\n> - Skip if the check was already performed earlier in this session.\n\n> **CRITICAL NOTES**\n> 1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering\n> 2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering\n\n# Spark Authoring — CLI Skill\n\nThis skill covers two complementary areas: (1) **managing Fabric Spark artifacts via REST APIs** (workspaces, lakehouses, notebooks, jobs, pipelines) and (2) **writing code inside Fabric Notebook cells** (PySpark, Scala, SparkR, SQL with correct lakehouse access, notebookutils, and Spark configuration). For notebook code authoring fundamentals and shared modules, MUST see [SPARK-NOTEBOOK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-NOTEBOOK-AUTHORING-CORE.md).\n\n## Table of Contents\n\n| Task | Reference | Notes |\n|---|---|---|\n| RULES — Read these first, follow them always | [SKILL.md § RULES](#rules--read-these-first-follow-them-always) | **MUST read** — 4 rules for this skill |\n| Finding Workspaces and Items in Fabric | [COMMON-CLI.md § Finding Workspaces and Items in Fabric](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#finding-workspaces-and-items-in-fabric) | **Mandatory** — *READ link first* [needed for finding workspace id by its name or item id by its name, item type, and workspace id] |\n| Fabric Topology & Key Concepts | [COMMON-CORE.md § Fabric Topology & Key Concepts](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#fabric-topology--key-concepts) ||\n| Environment URLs | [COMMON-CORE.md § Environment URLs](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#environment-urls) ||\n| Authentication & Token Acquisition | [COMMON-CORE.md § Authentication & Token Acquisition](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#authentication--token-acquisition) | Wrong audience = 401; read before any auth issue |\n| Core Control-Plane REST APIs | [COMMON-CORE.md § Core Control-Plane REST APIs](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#core-control-plane-rest-apis) ||\n| Pagination | [COMMON-CORE.md § Pagination](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#pagination) ||\n| Long-Running Operations (LRO) | [COMMON-CORE.md § Long-Running Operations (LRO)](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#long-running-operations-lro) ||\n| Rate Limiting & Throttling | [COMMON-CORE.md § Rate Limiting & Throttling](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#rate-limiting--throttling) ||\n| OneLake Data Access | [COMMON-CORE.md § OneLake Data Access](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#onelake-data-access) | Requires `storage.azure.com` token, not Fabric token |\n| Definition Envelope | [ITEM-DEFINITIONS-CORE.md § Definition Envelope](..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#definition-envelope) | Definition payload structure |\n| Per-Item-Type Definitions | [ITEM-DEFINITIONS-CORE.md § Per-Item-Type Definitions](..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#per-item-type-definitions) | Support matrix, decoded content, part paths — [REST specs](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#item-creation), [CLI recipes](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#item-crud-operations) |\n| Job Execution | [COMMON-CORE.md § Job Execution](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#job-execution) ||\n| Capacity Management | [COMMON-CORE.md § Capacity Management](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#capacity-management) ||\n| Gotchas & Troubleshooting | [COMMON-CORE.md § Gotchas & Troubleshooting](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#gotchas--troubleshooting) ||\n| Best Practices | [COMMON-CORE.md § Best Practices](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#best-practices) ||\n| Tool Selection Rationale | [COMMON-CLI.md § Tool Selection Rationale](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#tool-selection-rationale) ||\n| Authentication Recipes | [COMMON-CLI.md § Authentication Recipes](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#authentication-recipes) | `az login` flows and token acquisition |\n| Fabric Control-Plane API via `az rest` | [COMMON-CLI.md § Fabric Control-Plane API via az rest](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#fabric-control-plane-api-via-az-rest) | **Always pass `--resource https:\u002F\u002Fapi.fabric.microsoft.com`** or `az rest` fails |\n| Pagination Pattern | [COMMON-CLI.md § Pagination Pattern](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#pagination-pattern) ||\n| Long-Running Operations (LRO) Pattern | [COMMON-CLI.md § Long-Running Operations (LRO) Pattern](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#long-running-operations-lro-pattern) ||\n| OneLake Data Access via `curl` | [COMMON-CLI.md § OneLake Data Access via curl](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#onelake-data-access-via-curl) | Use `curl` not `az rest` (different token audience) |\n| SQL \u002F TDS Data-Plane Access | [COMMON-CLI.md § SQL \u002F TDS Data-Plane Access](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#sql--tds-data-plane-access) ||\n| Job Execution (CLI) | [COMMON-CLI.md § Job Execution](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#job-execution) ||\n| Job Scheduling | [COMMON-CLI.md § Job Scheduling](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#job-scheduling) | URL is `\u002Fjobs\u002F{jobType}\u002Fschedules`; `endDateTime` required |\n| OneLake Shortcuts | [COMMON-CLI.md § OneLake Shortcuts](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#onelake-shortcuts) ||\n| Capacity Management (CLI) | [COMMON-CLI.md § Capacity Management](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#capacity-management) ||\n| Composite Recipes | [COMMON-CLI.md § Composite Recipes](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#composite-recipes) ||\n| Gotchas & Troubleshooting (CLI-Specific) | [COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific)](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#gotchas--troubleshooting-cli-specific) | `az rest` audience, shell escaping, token expiry |\n| Quick Reference: `az rest` Template | [COMMON-CLI.md § Quick Reference: az rest Template](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#quick-reference-az-rest-template) ||\n| Quick Reference: Token Audience \u002F CLI Tool Matrix | [COMMON-CLI.md § Quick Reference: Token Audience ↔ CLI Tool Matrix](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#quick-reference-token-audience--cli-tool-matrix) | Which `--resource` + tool for each service |\n| Relationship to SPARK-CONSUMPTION-CORE.md | [SPARK-AUTHORING-CORE.md § Relationship to SPARK-CONSUMPTION-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#relationship-to-spark-consumption-coremd) ||\n| Data Engineering Authoring Capability Matrix | [SPARK-AUTHORING-CORE.md § Data Engineering Authoring Capability Matrix](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#data-engineering-authoring-capability-matrix) ||\n| Lakehouse Management | [SPARK-AUTHORING-CORE.md § Lakehouse Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#lakehouse-management) ||\n| Notebook Management | [SPARK-AUTHORING-CORE.md § Notebook Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#notebook-management) ||\n| Notebook Execution & Job Management | [SPARK-AUTHORING-CORE.md § Notebook Execution & Job Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#notebook-execution--job-management) ||\n| CI\u002FCD & Automation Patterns | [SPARK-AUTHORING-CORE.md § CI\u002FCD & Automation Patterns](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#cicd--automation-patterns) ||\n| Infrastructure-as-Code | [SPARK-AUTHORING-CORE.md § Infrastructure-as-Code](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#infrastructure-as-code) ||\n| Performance Optimization & Resource Management | [SPARK-AUTHORING-CORE.md § Performance Optimization & Resource Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#performance-optimization--resource-management) ||\n| Runtime 2.0 Performance Features | [data-engineering-patterns.md § Runtime 2.0 Performance Features](resources\u002Fdata-engineering-patterns.md#runtime-20-performance-features-spark-41-delta-42) | NEE, Efficient Scaledown, Liquid Clustering, Spark 4.1 |\n| Authoring Gotchas and Troubleshooting | [SPARK-AUTHORING-CORE.md § Authoring Gotchas and Troubleshooting](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#authoring-gotchas-and-troubleshooting) ||\n| Quick Reference: Authoring Decision Guide | [SPARK-AUTHORING-CORE.md § Quick Reference: Authoring Decision Guide](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#quick-reference-authoring-decision-guide) ||\n| Recommended Patterns (Data Engineering) |[data-engineering-patterns.md § Recommended patterns](resources\u002Fdata-engineering-patterns.md#recommended-patterns) ||\n| Data Ingestion Principles | [data-engineering-patterns.md § Data Ingestion Principles](resources\u002Fdata-engineering-patterns.md#data-ingestion-principles) ||\n| Transformation Patterns | [data-engineering-patterns.md § Transformation Patterns](resources\u002Fdata-engineering-patterns.md#transformation-patterns) ||\n| Delta Lake Best Practices | [data-engineering-patterns.md § Delta Lake Best Practices](resources\u002Fdata-engineering-patterns.md#delta-lake-best-practices) ||\n| Quality Assurance Strategies | [data-engineering-patterns.md § Quality Assurance Strategies](resources\u002Fdata-engineering-patterns.md#quality-assurance-strategies) ||\n| Recommended Patterns (Development Workflow) | [development-workflow.md § Recommended patterns](resources\u002Fdevelopment-workflow.md#recommended-patterns) ||\n| Notebook Lifecycle | [development-workflow.md § Notebook Lifecycle](resources\u002Fdevelopment-workflow.md#notebook-lifecycle) ||\n| Parameterization Patterns | [development-workflow.md § Parameterization Patterns](resources\u002Fdevelopment-workflow.md#parameterization-patterns) ||\n| Variable Library (notebook + pipeline usage) | [development-workflow.md § Method 4: Variable Library](resources\u002Fdevelopment-workflow.md#parameterization-patterns) | `getLibrary()` + dot notation in notebooks; `libraryVariables` + `@pipeline().libraryVariables` in pipelines |\n| Variable Library Definition | [ITEM-DEFINITIONS-CORE.md § VariableLibrary](..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#variablelibrary) | Definition parts, decoded content, types, pipeline mappings, gotchas |\n| Local Testing Strategy | [development-workflow.md § Local Testing Strategy](resources\u002Fdevelopment-workflow.md#local-testing-strategy) ||\n| Debugging Patterns | [development-workflow.md § Debugging Patterns](resources\u002Fdevelopment-workflow.md#debugging-patterns) ||\n| Recommended Patterns (Infrastructure) | [infrastructure-orchestration.md § Recommended patterns](resources\u002Finfrastructure-orchestration.md#recommended-patterns) ||\n| Materialized Lake View patterns | [materialized-lake-view-patterns.md § Recommended patterns](resources\u002Fmaterialized-lake-view-patterns.md#recommended-patterns) | Spark Lakehouse authoring guidance for MLV design (when to use MLVs, layering patterns) |\n| MLV incremental refresh patterns | [mlv-incremental-refresh-patterns.md § IR-friendly syntax guide](resources\u002Fmlv-incremental-refresh-patterns.md#ir-friendly-syntax-guide) | Use for refresh-readiness review and safe non-breaking rewrites |\n| MLV schedule & job management | [mlv-operations-cli](..\u002Fmlv-operations-cli\u002FSKILL.md) | Route here when user asks to schedule, trigger, monitor, or cancel MLV refreshes (not authoring) |\n| Workspace Provisioning Principles | [infrastructure-orchestration.md § Workspace Provisioning Principles](resources\u002Finfrastructure-orchestration.md#workspace-provisioning-principles) ||\n| Lakehouse Configuration Guidance | [infrastructure-orchestration.md § Lakehouse Configuration Guidance](resources\u002Finfrastructure-orchestration.md#lakehouse-configuration-guidance) ||\n| Pipeline Design Patterns | [infrastructure-orchestration.md § Pipeline Design Patterns](resources\u002Finfrastructure-orchestration.md#pipeline-design-patterns) ||\n| CI\u002FCD Integration Strategy | [infrastructure-orchestration.md § CI\u002FCD Integration Strategy](resources\u002Finfrastructure-orchestration.md#cicd-integration-strategy) ||\n| Notebook API — Which Endpoint to Use | [notebook-api-operations.md § Quick Decision](resources\u002Fnotebook-api-operations.md#quick-decision-which-endpoint-to-use) | **Start here for remote notebook edits** — getDefinition vs updateDefinition |\n| Notebook Modification Workflow | [notebook-api-operations.md § Workflow](resources\u002Fnotebook-api-operations.md#workflow-get--decode--modify--encode--upload--verify) | Five-step flow: retrieve, decode, modify, encode, upload |\n| Notebook Orchestration (parallel + DAG) | [notebook-api-operations.md § Notebook Orchestration](resources\u002Fnotebook-api-operations.md#notebook-orchestration--parallel-run--dag-dependencies) | `notebookutils.notebook.runMultiple(DAG)` for parallel runs and run-order dependencies (fan-in\u002Ffan-out); use instead of hand-rolled threads |\n| Notebook API Error Reference | [notebook-api-operations.md § Error Reference](resources\u002Fnotebook-api-operations.md#error-reference) | 411, 400 (updateMetadata), 401, 403 explained |\n| Notebook API Gotchas | [notebook-api-operations.md § Gotchas](resources\u002Fnotebook-api-operations.md#gotchas) | `\u002Fresult` suffix, empty body, `\\n` per-line rule, `format=ipynb` |\n| Default Lakehouse Binding | [notebook-api-operations.md § Default Lakehouse Binding](resources\u002Fnotebook-api-operations.md#default-lakehouse-binding) | `.ipynb` metadata vs `.py` `# METADATA` block; discover IDs dynamically |\n| Public URL Data Ingestion | [notebook-api-operations.md § Public URL Data Ingestion](resources\u002Fnotebook-api-operations.md#public-url-data-ingestion-spark) | Use real source URL, stage into `Files\u002F`, then read with Spark |\n| getDefinition (read notebook content) | [notebook-api-operations.md § Step 1 — Retrieve Notebook Content](resources\u002Fnotebook-api-operations.md#step-1--retrieve-notebook-content-getdefinition) | LRO flow, `?format=ipynb`, empty body (`--body '{}'`) requirement |\n| Decode Base64 Notebook Payload | [notebook-api-operations.md § Step 2 — Decode the Notebook Content](resources\u002Fnotebook-api-operations.md#step-2--decode-the-notebook-content) | Extract payload, base64 decode, ipynb JSON structure |\n| Modify Notebook Cells | [notebook-api-operations.md § Step 3 — Modify the Notebook Content](resources\u002Fnotebook-api-operations.md#step-3--modify-the-notebook-content) | Find cell, insert\u002Freplace lines, `\\n` per-line rule |\n| updateDefinition (write notebook content) | [notebook-api-operations.md § Step 4 — Re-encode and Upload](resources\u002Fnotebook-api-operations.md#step-4--re-encode-and-upload-updatedefinition) | Re-encode, upload, LRO poll, updateMetadata flag pitfall |\n| Verify Notebook Update (Optional) | [notebook-api-operations.md § Step 5 — Verify the Update](resources\u002Fnotebook-api-operations.md#step-5--verify-the-update) | Skip unless you suspect a silent failure — `Succeeded` from updateDefinition is sufficient (see Rule 2) |\n| Notebook API Error Reference | [notebook-api-operations.md § Error Reference](resources\u002Fnotebook-api-operations.md#error-reference) | 411, 400 (updateMetadata), 401, 403 explained |\n| Notebook API End-to-End Script | [notebook-api-operations.md § Complete End-to-End Script](resources\u002Fnotebook-api-operations.md#complete-end-to-end-script) | Full bash: get → decode → modify → encode → update → verify |\n| Quick Start Examples | [SKILL.md § Quick Start Examples](#quick-start-examples) | Minimal examples for common operations |\n| **— Notebook Code Authoring (shared modules) —** | | |\n| Notebook Authoring Core | [SPARK-NOTEBOOK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-NOTEBOOK-AUTHORING-CORE.md) | **READ FIRST for notebook code tasks** — fundamentals, code gen approach, module index |\n\n---\n\n## Must\u002FPrefer\u002FAvoid\n\n### MUST DO\n- **Check for recent jobs BEFORE creating new notebook runs** — Query job instances from last 5 minutes; if recent job exists, monitor it instead of creating duplicate\n- **Capture job instance ID immediately after POST** — Store job ID before any other operations to enable proper monitoring\n- **Verify workspace capacity assignment** before operations — Workspace must have capacity assigned and active\n- **When user provides a public data URL, follow the Public URL Data Ingestion policy** — keep detailed behavior in the linked resource section to avoid drift\u002Fduplication\n- **Format notebook cells correctly** — Each line in cell source array MUST end with `\\n` to prevent code merging\n- **Use correct Lakehouse Livy session body format** — Send a FLAT JSON with `name`, `driverMemory`, `driverCores`, `executorMemory`, `executorCores`. Do NOT wrap in `{\"payload\": ...}` or send only `{\"kind\": \"pyspark\"}` — that causes HTTP 500. Use valid memory values (28g, 56g, 112g, 224g). See Create Lakehouse Livy Session example below and SPARK-CONSUMPTION-CORE.md.\n\n### PREFER\n- **Poll job status with proper intervals** — 10-30 seconds between polls; timeout after reasonable duration (e.g., 30 minutes)\n- **Check job history when POST response is unreadable** — If POST returns \"No Content\" or unreadable response, query recent jobs (last 1 minute) before retrying\n- **Use Starter Pool for development** — Development\u002Ftesting workloads should use `useStarterPool: true`\n- **Use Workspace Pool for production** — Production workloads need consistent performance with `useWorkspacePool: true`\n- **Enable lakehouse schemas** during creation — Set `creationPayload.enableSchemas: true` for better table organization\n- **Implement idempotency checks** — Prevent duplicate operations by checking existing state first\n\n### AVOID\n- **Never retry POST with same parameters** — If you have a job ID, only use GET to check status; don't create duplicate job instances\n- **Don't skip capacity verification** — Operations will fail if workspace capacity is paused or unassigned\n- **Avoid immediate POST retries on failures** — Check for existing\u002Factive jobs first to prevent duplicates\n- **Don't create new runs if monitoring existing job** — One job at a time; wait for completion before submitting new runs\n- **Don't hardcode workspace\u002Flakehouse IDs** — Discover dynamically via item listing or catalog search APIs\n- **Own `%%sql` notebook cells here, do not defer to `sqldw-consumption-cli`** — a request to \"write a %%sql cell\" (or any notebook magic cell) is notebook-cell authoring even when the cell queries a lakehouse table. Only route to `sqldw-consumption-cli` when the user wants a plain T-SQL query executed against a SQL endpoint, not a notebook cell.\n- **Own \"run\u002Fexecute the notebook named X\" here** — running a Fabric notebook by name is notebook execution via the Jobs API (`RunNotebook`), which belongs to this skill; do not defer to `spark-consumption-cli` (that skill is only for ad-hoc Livy session code execution).\n- **Do NOT use Lakehouse Livy sessions to run a Fabric notebook** — Lakehouse Livy sessions (the public Livy API) are for ad-hoc interactive Spark code execution. To run a notebook as a job, use the Jobs API (`RunNotebook`) which creates a Notebook Spark session internally. See SPARK-AUTHORING-CORE.md § Notebook Execution & Job Management\n- **Do NOT schedule MLV refreshes from notebooks** — If the user asks to \"schedule MLV refresh\", route to [mlv-operations-cli](..\u002Fmlv-operations-cli\u002FSKILL.md) which uses the REST API. Notebook-based `REFRESH MATERIALIZED LAKE VIEW ... FULL` is for one-time manual refresh only, not recurring schedules.\n\n---\n\n\n## RULES — Read these first, follow them always\n\n> **Rule 1 — Validate prerequisites before operations.**\n> Verify workspace has capacity assigned (see COMMON-CORE.md Create Workspace and Capacity Management) and resource IDs exist before attempting operations.\n>\n> **Rule 2 — Trust updateDefinition success.**\n> A `Succeeded` poll result from `updateDefinition` is sufficient confirmation that content and lakehouse bindings persisted. Do NOT call `getDefinition` after every upload — it is an async LRO that adds significant latency. Only use `getDefinition` for its intended purpose: reading current notebook content before making modifications.\n>\n> **Rule 3 — Prevent duplicate jobs and monitor execution properly.**\n> Before submitting new notebook run, ALWAYS check for recent job instances first (last 5 minutes). If recent job exists, monitor it instead of creating duplicate. After submission, capture job instance ID immediately and poll status - never retry POST. See SPARK-AUTHORING-CORE.md Job Monitoring for patterns.\n>\n> **Rule 4 — For notebook code authoring, MUST follow SPARK-NOTEBOOK-AUTHORING-CORE.md.**\n> When writing code inside notebook cells, MUST read [SPARK-NOTEBOOK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-NOTEBOOK-AUTHORING-CORE.md) first — it defines the code generation approach, rules, and a Module Index linking to detailed guides (lakehouse paths, connections, context, orchestration, etc.). Use the Spark-specific resources in this skill ([data-engineering-patterns.md](resources\u002Fdata-engineering-patterns.md), [development-workflow.md](resources\u002Fdevelopment-workflow.md)) for Spark-only implementation details. When the task is about Materialized Lake Views, read [materialized-lake-view-patterns.md](resources\u002Fmaterialized-lake-view-patterns.md) for authoring\u002Fdesign guidance and [mlv-incremental-refresh-patterns.md](resources\u002Fmlv-incremental-refresh-patterns.md) for refresh-readiness analysis.\n\n---\n\n## Notebook Codegen Quick Rules\n\nQuick reference for common notebook-authoring tasks. The shared `common\u002Fnotebook-authoring\u002F` core (see Rule 4 \u002F [SPARK-NOTEBOOK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-NOTEBOOK-AUTHORING-CORE.md)) is authoritative; if these ever differ, follow the common core.\n\n| User asks for | Required output pattern |\n|---|---|\n| `%%sql` \u002F cross-lakehouse query cell | Return a Fabric notebook `%%sql` cell. Include the named workspace\u002Flakehouse\u002Fschema\u002Ftable in the code or explanatory note. This is notebook authoring, not interactive Spark consumption. |\n| Pipeline context detection | Use `notebookutils.runtime.context`; include `isForPipeline = context[\"isForPipeline\"]` and read `context[\"currentWorkspaceId\"]`. |\n| Built-in notebook resource files with Spark | Use `notebookutils.nbResPath`, `builtin\u002F`, and the `file:` prefix: `spark.read.json(f\"file:{notebookutils.nbResPath}\u002Fbuiltin\u002Fconfig.json\")`. Spark and `notebookutils.fs` require `file:` for resource-folder local paths. |\n| Fabric connections | Use `notebookutils.connections.getCredential(\"{connectionId}\")` directly and show a compact code sample. Do not web-search; keep the answer under the tool\u002Fturn budget. |\n\n### Minimal snippets\n\n```python\n# Pipeline context\ncontext = notebookutils.runtime.context\nisForPipeline = context[\"isForPipeline\"]\nworkspace_id = context[\"currentWorkspaceId\"]\nprint(f\"isForPipeline={isForPipeline}, workspace_id={workspace_id}\")\n```\n\n```python\n# Spark read from notebook built-in resources\nconfig_df = spark.read.option(\"multiline\", \"true\").json(\n    f\"file:{notebookutils.nbResPath}\u002Fbuiltin\u002Fconfig.json\"\n)\ndisplay(config_df)\n```\n\n```python\n# PostgreSQL through a Fabric connection (fill in connectionId and host\u002Fdb)\nimport psycopg2\n\ncredential = notebookutils.connections.getCredential(\"{connectionId}\")\nconn = psycopg2.connect(\n    host=\"\u003Cpostgres-host>\",\n    database=\"\u003Cdatabase>\",\n    user=credential[\"username\"],\n    password=credential[\"password\"],\n)\n```\n\n---\n\n## Quick Start Examples\n\nFor detailed patterns, authentication, and comprehensive API usage, see:\n- **COMMON-CORE.md** — Fabric REST API patterns, authentication, item discovery\n- **COMMON-CLI.md** — `az rest` usage, environment detection, token acquisition\n- **SPARK-AUTHORING-CORE.md** — Notebook deployment, lakehouse creation, job execution\n\nBelow are minimal quick-start examples. **Always reference the COMMON-* files for production use.**\n\n### Create Workspace & Lakehouse\n```bash\n# See COMMON-CORE.md Environment URLs and SPARK-AUTHORING-CORE.md for full patterns\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"DataEng-Dev\"}\nEOF\nworkspace_id=$(az rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\" \\\n  --body @\u002Ftmp\u002Fbody.json --query \"id\" --output tsv)\n\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"DevLakehouse\", \"type\": \"Lakehouse\", \"creationPayload\": {\"enableSchemas\": true}}\nEOF\nlakehouse_id=$(az rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F$workspace_id\u002Fitems\" \\\n  --body @\u002Ftmp\u002Fbody.json --query \"id\" --output tsv)\n```\n\n### Organize Lakehouse Tables with Schemas\n```python\n# See SPARK-AUTHORING-CORE.md Lakehouse Schema Organization for table organization patterns\n#\n# IMPORTANT: `CREATE SCHEMA` is only supported on schema-enabled lakehouses. On a\n# lakehouse created WITHOUT `enableSchemas`, it fails with\n# \"Feature not supported on Apache Spark in Microsoft Fabric\". Always detect schema\n# support first, then branch: create schemas only when enabled, otherwise skip.\n# Schema names are configurable. If your workspace standard is `raw`\u002F`curated`\n# instead of `bronze`\u002F`silver`\u002F`gold`, substitute those names consistently.\nimport uuid\n\ndef lakehouse_is_schema_enabled():\n    # Probe schema support by attempting CREATE SCHEMA and catching the\n    # platform-blocked error. Use a unique probe schema name to avoid\n    # accidentally dropping a real pre-existing schema.\n    probe = f\"_schema_probe_{uuid.uuid4().hex}\"\n    try:\n        spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {probe}\")\n        spark.sql(f\"DROP SCHEMA IF EXISTS {probe}\")\n        return True\n    except Exception as e:\n        if \"Feature not supported on Apache Spark in Microsoft Fabric\" in str(e):\n            return False\n        raise\n\nif lakehouse_is_schema_enabled():\n    # Schema-enabled: create medallion schemas and write tables as `schema.table`.\n    spark.sql(\"CREATE SCHEMA IF NOT EXISTS bronze\")\n    spark.sql(\"CREATE SCHEMA IF NOT EXISTS silver\")\n    spark.sql(\"CREATE SCHEMA IF NOT EXISTS gold\")\nelse:\n    # Not schema-enabled: skip CREATE SCHEMA (unsupported). Organize tables by name\n    # prefix instead (e.g. `bronze_orders` \u002F `raw_orders`, `silver_orders` \u002F\n    # `curated_orders`), or recreate the lakehouse with\n    # `creationPayload.enableSchemas: true` if schemas are required.\n    print(\"Lakehouse is not schema-enabled; skipping schema creation and using name prefixes.\")\n```\n\n### Create and Refresh a Materialized Lake View (MLV)\n```sql\n-- See resources\u002Fmaterialized-lake-view-patterns.md for design guidance\n-- and resources\u002Fmlv-incremental-refresh-patterns.md for refresh-readiness review.\n\n-- Bronze\u002FSilver\u002FGold schemas in a Lakehouse with schemas enabled\nCREATE SCHEMA IF NOT EXISTS bronze;\nCREATE SCHEMA IF NOT EXISTS silver;\nCREATE SCHEMA IF NOT EXISTS gold;\n\n-- A simple Silver MLV with data quality constraints\n--\n-- Prerequisite for incremental refresh: enable Change Data Feed (CDF) on every\n-- source table the MLV reads from. Without CDF, optimal refresh can only choose\n-- between no refresh (sources unchanged) and full refresh — never incremental.\n-- See resources\u002Fmlv-incremental-refresh-patterns.md.\nALTER TABLE bronze.orders_raw SET TBLPROPERTIES (delta.enableChangeDataFeed = true);\n\nCREATE OR REPLACE MATERIALIZED LAKE VIEW silver.orders_clean\n(\n    CONSTRAINT valid_order_id CHECK (order_id IS NOT NULL) ON MISMATCH DROP\n)\nAS\nSELECT\n  order_id,\n  customer_id,\n  CAST(order_ts AS TIMESTAMP) AS order_ts,\n  amount\nFROM bronze.orders_raw;\n\n-- Routine refresh is handled by the lakehouse Materialized lake views → Manage\n-- schedule\u002Flineage view; don't orchestrate from notebooks. The SQL form below is\n-- documented only for forcing a one-time FULL recompute (troubleshooting \u002F after\n-- a correction). There is no documented SQL form for triggering incremental refresh.\nREFRESH MATERIALIZED LAKE VIEW silver.orders_clean FULL;\n```\n\n### Create Lakehouse Livy Session\n```bash\n# See SPARK-CONSUMPTION-CORE.md for Lakehouse Livy session configuration and management\n# IMPORTANT: Body MUST be flat JSON with memory\u002Fcores — do NOT wrap in {\"payload\": ...}\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"name\": \"dev-session\", \"driverMemory\": \"56g\", \"driverCores\": 8, \"executorMemory\": \"56g\", \"executorCores\": 8, \"conf\": {\"spark.dynamicAllocation.enabled\": \"true\", \"spark.fabric.pool.name\": \"Starter Pool\"}}\nEOF\naz rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F$workspace_id\u002Flakehouses\u002F$lakehouse_id\u002Flivyapi\u002Fversions\u002F2023-12-01\u002Fsessions\" \\\n  --body @\u002Ftmp\u002Fbody.json\n```\n\n> **Lakehouse Livy Session Body — Common Mistakes**\n> - ❌ `{\"payload\": {\"kind\": \"pyspark\"}}` → HTTP 500 (wrong wrapper, missing required fields)\n> - ❌ `{\"kind\": \"pyspark\"}` → HTTP 500 (missing `driverMemory`, `executorMemory`, etc.)\n> - ✅ Flat JSON with `name`, `driverMemory`, `driverCores`, `executorMemory`, `executorCores` (and optionally `conf` with Starter Pool)\n\n### Spark Performance Configs\n**For detailed workload-specific configurations, see data-engineering-patterns.md Delta Lake Best Practices.**\n\nQuick reference:\n```python\n# Write-heavy (Bronze): Disable V-Order, enable autoCompact\n# Balanced (Silver): Enable V-Order, adaptive execution  \n# Read-heavy (Gold): Vectorized reads, optimal parallelism\n# See data-engineering-patterns.md for complete config tables\n```\n\n---\n\n**Focus**: Essential CLI patterns for Spark\u002Fdata engineering development and notebook code authoring, with intelligent routing to specialized resources. For comprehensive patterns, always reference COMMON-* files and resource documents.\n",{"data":40,"body":41},{"name":4,"description":6},{"type":42,"children":43},"root",[44,107,132,139,167,174,2095,2099,2105,2112,2234,2240,2323,2329,2480,2483,2488,2598,2601,2607,2626,2786,2792,2849,2896,2988,2991,2996,3001,3040,3057,3063,3427,3433,3739,3745,4013,4019,4173,4268,4274,4282,4287,4326,4329,4339],{"type":45,"tag":46,"props":47,"children":48},"element","blockquote",{},[49,69],{"type":45,"tag":50,"props":51,"children":52},"p",{},[53,60,62,67],{"type":45,"tag":54,"props":55,"children":56},"strong",{},[57],{"type":58,"value":59},"text","Update Check — ONCE PER SESSION (mandatory)",{"type":58,"value":61},"\nThe first time this skill is used in a session, run the ",{"type":45,"tag":54,"props":63,"children":64},{},[65],{"type":58,"value":66},"check-updates",{"type":58,"value":68}," skill before proceeding.",{"type":45,"tag":70,"props":71,"children":72},"ul",{},[73,92,102],{"type":45,"tag":74,"props":75,"children":76},"li",{},[77,82,84,90],{"type":45,"tag":54,"props":78,"children":79},{},[80],{"type":58,"value":81},"GitHub Copilot CLI \u002F VS Code",{"type":58,"value":83},": invoke the ",{"type":45,"tag":85,"props":86,"children":88},"code",{"className":87},[],[89],{"type":58,"value":66},{"type":58,"value":91}," skill.",{"type":45,"tag":74,"props":93,"children":94},{},[95,100],{"type":45,"tag":54,"props":96,"children":97},{},[98],{"type":58,"value":99},"Claude Code \u002F Cowork \u002F Cursor \u002F Windsurf \u002F Codex",{"type":58,"value":101},": compare local vs remote package.json version.",{"type":45,"tag":74,"props":103,"children":104},{},[105],{"type":58,"value":106},"Skip if the check was already performed earlier in this session.",{"type":45,"tag":46,"props":108,"children":109},{},[110,118],{"type":45,"tag":50,"props":111,"children":112},{},[113],{"type":45,"tag":54,"props":114,"children":115},{},[116],{"type":58,"value":117},"CRITICAL NOTES",{"type":45,"tag":119,"props":120,"children":121},"ol",{},[122,127],{"type":45,"tag":74,"props":123,"children":124},{},[125],{"type":58,"value":126},"To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering",{"type":45,"tag":74,"props":128,"children":129},{},[130],{"type":58,"value":131},"To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering",{"type":45,"tag":133,"props":134,"children":136},"h1",{"id":135},"spark-authoring-cli-skill",[137],{"type":58,"value":138},"Spark Authoring — CLI Skill",{"type":45,"tag":50,"props":140,"children":141},{},[142,144,149,151,156,158,165],{"type":58,"value":143},"This skill covers two complementary areas: (1) ",{"type":45,"tag":54,"props":145,"children":146},{},[147],{"type":58,"value":148},"managing Fabric Spark artifacts via REST APIs",{"type":58,"value":150}," (workspaces, lakehouses, notebooks, jobs, pipelines) and (2) ",{"type":45,"tag":54,"props":152,"children":153},{},[154],{"type":58,"value":155},"writing code inside Fabric Notebook cells",{"type":58,"value":157}," (PySpark, Scala, SparkR, SQL with correct lakehouse access, notebookutils, and Spark configuration). For notebook code authoring fundamentals and shared modules, MUST see ",{"type":45,"tag":159,"props":160,"children":162},"a",{"href":161},"..\u002F..\u002Fcommon\u002FSPARK-NOTEBOOK-AUTHORING-CORE.md",[163],{"type":58,"value":164},"SPARK-NOTEBOOK-AUTHORING-CORE.md",{"type":58,"value":166},".",{"type":45,"tag":168,"props":169,"children":171},"h2",{"id":170},"table-of-contents",[172],{"type":58,"value":173},"Table of Contents",{"type":45,"tag":175,"props":176,"children":177},"table",{},[178,202],{"type":45,"tag":179,"props":180,"children":181},"thead",{},[182],{"type":45,"tag":183,"props":184,"children":185},"tr",{},[186,192,197],{"type":45,"tag":187,"props":188,"children":189},"th",{},[190],{"type":58,"value":191},"Task",{"type":45,"tag":187,"props":193,"children":194},{},[195],{"type":58,"value":196},"Reference",{"type":45,"tag":187,"props":198,"children":199},{},[200],{"type":58,"value":201},"Notes",{"type":45,"tag":203,"props":204,"children":205},"tbody",{},[206,234,275,295,315,337,357,377,397,417,447,469,505,525,545,565,585,605,633,679,699,719,761,781,801,839,859,879,899,926,953,983,1003,1023,1043,1063,1083,1103,1123,1143,1165,1185,1205,1225,1245,1265,1285,1305,1325,1345,1365,1408,1430,1450,1470,1490,1512,1534,1556,1576,1596,1616,1636,1663,1685,1713,1735,1777,1820,1850,1888,1910,1939,1961,1991,2009,2031,2053,2070],{"type":45,"tag":183,"props":207,"children":208},{},[209,215,224],{"type":45,"tag":210,"props":211,"children":212},"td",{},[213],{"type":58,"value":214},"RULES — Read these first, follow them always",{"type":45,"tag":210,"props":216,"children":217},{},[218],{"type":45,"tag":159,"props":219,"children":221},{"href":220},"#rules--read-these-first-follow-them-always",[222],{"type":58,"value":223},"SKILL.md § RULES",{"type":45,"tag":210,"props":225,"children":226},{},[227,232],{"type":45,"tag":54,"props":228,"children":229},{},[230],{"type":58,"value":231},"MUST read",{"type":58,"value":233}," — 4 rules for this skill",{"type":45,"tag":183,"props":235,"children":236},{},[237,242,251],{"type":45,"tag":210,"props":238,"children":239},{},[240],{"type":58,"value":241},"Finding Workspaces and Items in Fabric",{"type":45,"tag":210,"props":243,"children":244},{},[245],{"type":45,"tag":159,"props":246,"children":248},{"href":247},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#finding-workspaces-and-items-in-fabric",[249],{"type":58,"value":250},"COMMON-CLI.md § Finding Workspaces and Items in Fabric",{"type":45,"tag":210,"props":252,"children":253},{},[254,259,261,267,269],{"type":45,"tag":54,"props":255,"children":256},{},[257],{"type":58,"value":258},"Mandatory",{"type":58,"value":260}," — ",{"type":45,"tag":262,"props":263,"children":264},"em",{},[265],{"type":58,"value":266},"READ link first",{"type":58,"value":268}," ",{"type":45,"tag":270,"props":271,"children":272},"span",{},[273],{"type":58,"value":274},"needed for finding workspace id by its name or item id by its name, item type, and workspace id",{"type":45,"tag":183,"props":276,"children":277},{},[278,283,292],{"type":45,"tag":210,"props":279,"children":280},{},[281],{"type":58,"value":282},"Fabric Topology & Key Concepts",{"type":45,"tag":210,"props":284,"children":285},{},[286],{"type":45,"tag":159,"props":287,"children":289},{"href":288},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#fabric-topology--key-concepts",[290],{"type":58,"value":291},"COMMON-CORE.md § Fabric Topology & Key Concepts",{"type":45,"tag":210,"props":293,"children":294},{},[],{"type":45,"tag":183,"props":296,"children":297},{},[298,303,312],{"type":45,"tag":210,"props":299,"children":300},{},[301],{"type":58,"value":302},"Environment URLs",{"type":45,"tag":210,"props":304,"children":305},{},[306],{"type":45,"tag":159,"props":307,"children":309},{"href":308},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#environment-urls",[310],{"type":58,"value":311},"COMMON-CORE.md § Environment URLs",{"type":45,"tag":210,"props":313,"children":314},{},[],{"type":45,"tag":183,"props":316,"children":317},{},[318,323,332],{"type":45,"tag":210,"props":319,"children":320},{},[321],{"type":58,"value":322},"Authentication & Token Acquisition",{"type":45,"tag":210,"props":324,"children":325},{},[326],{"type":45,"tag":159,"props":327,"children":329},{"href":328},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#authentication--token-acquisition",[330],{"type":58,"value":331},"COMMON-CORE.md § Authentication & Token Acquisition",{"type":45,"tag":210,"props":333,"children":334},{},[335],{"type":58,"value":336},"Wrong audience = 401; read before any auth issue",{"type":45,"tag":183,"props":338,"children":339},{},[340,345,354],{"type":45,"tag":210,"props":341,"children":342},{},[343],{"type":58,"value":344},"Core Control-Plane REST APIs",{"type":45,"tag":210,"props":346,"children":347},{},[348],{"type":45,"tag":159,"props":349,"children":351},{"href":350},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#core-control-plane-rest-apis",[352],{"type":58,"value":353},"COMMON-CORE.md § Core Control-Plane REST APIs",{"type":45,"tag":210,"props":355,"children":356},{},[],{"type":45,"tag":183,"props":358,"children":359},{},[360,365,374],{"type":45,"tag":210,"props":361,"children":362},{},[363],{"type":58,"value":364},"Pagination",{"type":45,"tag":210,"props":366,"children":367},{},[368],{"type":45,"tag":159,"props":369,"children":371},{"href":370},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#pagination",[372],{"type":58,"value":373},"COMMON-CORE.md § Pagination",{"type":45,"tag":210,"props":375,"children":376},{},[],{"type":45,"tag":183,"props":378,"children":379},{},[380,385,394],{"type":45,"tag":210,"props":381,"children":382},{},[383],{"type":58,"value":384},"Long-Running Operations (LRO)",{"type":45,"tag":210,"props":386,"children":387},{},[388],{"type":45,"tag":159,"props":389,"children":391},{"href":390},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#long-running-operations-lro",[392],{"type":58,"value":393},"COMMON-CORE.md § Long-Running Operations (LRO)",{"type":45,"tag":210,"props":395,"children":396},{},[],{"type":45,"tag":183,"props":398,"children":399},{},[400,405,414],{"type":45,"tag":210,"props":401,"children":402},{},[403],{"type":58,"value":404},"Rate Limiting & Throttling",{"type":45,"tag":210,"props":406,"children":407},{},[408],{"type":45,"tag":159,"props":409,"children":411},{"href":410},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#rate-limiting--throttling",[412],{"type":58,"value":413},"COMMON-CORE.md § Rate Limiting & Throttling",{"type":45,"tag":210,"props":415,"children":416},{},[],{"type":45,"tag":183,"props":418,"children":419},{},[420,425,434],{"type":45,"tag":210,"props":421,"children":422},{},[423],{"type":58,"value":424},"OneLake Data Access",{"type":45,"tag":210,"props":426,"children":427},{},[428],{"type":45,"tag":159,"props":429,"children":431},{"href":430},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#onelake-data-access",[432],{"type":58,"value":433},"COMMON-CORE.md § OneLake Data Access",{"type":45,"tag":210,"props":435,"children":436},{},[437,439,445],{"type":58,"value":438},"Requires ",{"type":45,"tag":85,"props":440,"children":442},{"className":441},[],[443],{"type":58,"value":444},"storage.azure.com",{"type":58,"value":446}," token, not Fabric token",{"type":45,"tag":183,"props":448,"children":449},{},[450,455,464],{"type":45,"tag":210,"props":451,"children":452},{},[453],{"type":58,"value":454},"Definition Envelope",{"type":45,"tag":210,"props":456,"children":457},{},[458],{"type":45,"tag":159,"props":459,"children":461},{"href":460},"..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#definition-envelope",[462],{"type":58,"value":463},"ITEM-DEFINITIONS-CORE.md § Definition Envelope",{"type":45,"tag":210,"props":465,"children":466},{},[467],{"type":58,"value":468},"Definition payload structure",{"type":45,"tag":183,"props":470,"children":471},{},[472,477,486],{"type":45,"tag":210,"props":473,"children":474},{},[475],{"type":58,"value":476},"Per-Item-Type Definitions",{"type":45,"tag":210,"props":478,"children":479},{},[480],{"type":45,"tag":159,"props":481,"children":483},{"href":482},"..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#per-item-type-definitions",[484],{"type":58,"value":485},"ITEM-DEFINITIONS-CORE.md § Per-Item-Type Definitions",{"type":45,"tag":210,"props":487,"children":488},{},[489,491,497,499],{"type":58,"value":490},"Support matrix, decoded content, part paths — ",{"type":45,"tag":159,"props":492,"children":494},{"href":493},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#item-creation",[495],{"type":58,"value":496},"REST specs",{"type":58,"value":498},", ",{"type":45,"tag":159,"props":500,"children":502},{"href":501},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#item-crud-operations",[503],{"type":58,"value":504},"CLI recipes",{"type":45,"tag":183,"props":506,"children":507},{},[508,513,522],{"type":45,"tag":210,"props":509,"children":510},{},[511],{"type":58,"value":512},"Job Execution",{"type":45,"tag":210,"props":514,"children":515},{},[516],{"type":45,"tag":159,"props":517,"children":519},{"href":518},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#job-execution",[520],{"type":58,"value":521},"COMMON-CORE.md § Job Execution",{"type":45,"tag":210,"props":523,"children":524},{},[],{"type":45,"tag":183,"props":526,"children":527},{},[528,533,542],{"type":45,"tag":210,"props":529,"children":530},{},[531],{"type":58,"value":532},"Capacity Management",{"type":45,"tag":210,"props":534,"children":535},{},[536],{"type":45,"tag":159,"props":537,"children":539},{"href":538},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#capacity-management",[540],{"type":58,"value":541},"COMMON-CORE.md § Capacity Management",{"type":45,"tag":210,"props":543,"children":544},{},[],{"type":45,"tag":183,"props":546,"children":547},{},[548,553,562],{"type":45,"tag":210,"props":549,"children":550},{},[551],{"type":58,"value":552},"Gotchas & Troubleshooting",{"type":45,"tag":210,"props":554,"children":555},{},[556],{"type":45,"tag":159,"props":557,"children":559},{"href":558},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#gotchas--troubleshooting",[560],{"type":58,"value":561},"COMMON-CORE.md § Gotchas & Troubleshooting",{"type":45,"tag":210,"props":563,"children":564},{},[],{"type":45,"tag":183,"props":566,"children":567},{},[568,573,582],{"type":45,"tag":210,"props":569,"children":570},{},[571],{"type":58,"value":572},"Best Practices",{"type":45,"tag":210,"props":574,"children":575},{},[576],{"type":45,"tag":159,"props":577,"children":579},{"href":578},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#best-practices",[580],{"type":58,"value":581},"COMMON-CORE.md § Best Practices",{"type":45,"tag":210,"props":583,"children":584},{},[],{"type":45,"tag":183,"props":586,"children":587},{},[588,593,602],{"type":45,"tag":210,"props":589,"children":590},{},[591],{"type":58,"value":592},"Tool Selection Rationale",{"type":45,"tag":210,"props":594,"children":595},{},[596],{"type":45,"tag":159,"props":597,"children":599},{"href":598},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#tool-selection-rationale",[600],{"type":58,"value":601},"COMMON-CLI.md § Tool Selection Rationale",{"type":45,"tag":210,"props":603,"children":604},{},[],{"type":45,"tag":183,"props":606,"children":607},{},[608,613,622],{"type":45,"tag":210,"props":609,"children":610},{},[611],{"type":58,"value":612},"Authentication Recipes",{"type":45,"tag":210,"props":614,"children":615},{},[616],{"type":45,"tag":159,"props":617,"children":619},{"href":618},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#authentication-recipes",[620],{"type":58,"value":621},"COMMON-CLI.md § Authentication Recipes",{"type":45,"tag":210,"props":623,"children":624},{},[625,631],{"type":45,"tag":85,"props":626,"children":628},{"className":627},[],[629],{"type":58,"value":630},"az login",{"type":58,"value":632}," flows and token acquisition",{"type":45,"tag":183,"props":634,"children":635},{},[636,647,656],{"type":45,"tag":210,"props":637,"children":638},{},[639,641],{"type":58,"value":640},"Fabric Control-Plane API via ",{"type":45,"tag":85,"props":642,"children":644},{"className":643},[],[645],{"type":58,"value":646},"az rest",{"type":45,"tag":210,"props":648,"children":649},{},[650],{"type":45,"tag":159,"props":651,"children":653},{"href":652},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#fabric-control-plane-api-via-az-rest",[654],{"type":58,"value":655},"COMMON-CLI.md § Fabric Control-Plane API via az rest",{"type":45,"tag":210,"props":657,"children":658},{},[659,670,672,677],{"type":45,"tag":54,"props":660,"children":661},{},[662,664],{"type":58,"value":663},"Always pass ",{"type":45,"tag":85,"props":665,"children":667},{"className":666},[],[668],{"type":58,"value":669},"--resource https:\u002F\u002Fapi.fabric.microsoft.com",{"type":58,"value":671}," or ",{"type":45,"tag":85,"props":673,"children":675},{"className":674},[],[676],{"type":58,"value":646},{"type":58,"value":678}," fails",{"type":45,"tag":183,"props":680,"children":681},{},[682,687,696],{"type":45,"tag":210,"props":683,"children":684},{},[685],{"type":58,"value":686},"Pagination Pattern",{"type":45,"tag":210,"props":688,"children":689},{},[690],{"type":45,"tag":159,"props":691,"children":693},{"href":692},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#pagination-pattern",[694],{"type":58,"value":695},"COMMON-CLI.md § Pagination Pattern",{"type":45,"tag":210,"props":697,"children":698},{},[],{"type":45,"tag":183,"props":700,"children":701},{},[702,707,716],{"type":45,"tag":210,"props":703,"children":704},{},[705],{"type":58,"value":706},"Long-Running Operations (LRO) Pattern",{"type":45,"tag":210,"props":708,"children":709},{},[710],{"type":45,"tag":159,"props":711,"children":713},{"href":712},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#long-running-operations-lro-pattern",[714],{"type":58,"value":715},"COMMON-CLI.md § Long-Running Operations (LRO) Pattern",{"type":45,"tag":210,"props":717,"children":718},{},[],{"type":45,"tag":183,"props":720,"children":721},{},[722,733,742],{"type":45,"tag":210,"props":723,"children":724},{},[725,727],{"type":58,"value":726},"OneLake Data Access via ",{"type":45,"tag":85,"props":728,"children":730},{"className":729},[],[731],{"type":58,"value":732},"curl",{"type":45,"tag":210,"props":734,"children":735},{},[736],{"type":45,"tag":159,"props":737,"children":739},{"href":738},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#onelake-data-access-via-curl",[740],{"type":58,"value":741},"COMMON-CLI.md § OneLake Data Access via curl",{"type":45,"tag":210,"props":743,"children":744},{},[745,747,752,754,759],{"type":58,"value":746},"Use ",{"type":45,"tag":85,"props":748,"children":750},{"className":749},[],[751],{"type":58,"value":732},{"type":58,"value":753}," not ",{"type":45,"tag":85,"props":755,"children":757},{"className":756},[],[758],{"type":58,"value":646},{"type":58,"value":760}," (different token audience)",{"type":45,"tag":183,"props":762,"children":763},{},[764,769,778],{"type":45,"tag":210,"props":765,"children":766},{},[767],{"type":58,"value":768},"SQL \u002F TDS Data-Plane Access",{"type":45,"tag":210,"props":770,"children":771},{},[772],{"type":45,"tag":159,"props":773,"children":775},{"href":774},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#sql--tds-data-plane-access",[776],{"type":58,"value":777},"COMMON-CLI.md § SQL \u002F TDS Data-Plane Access",{"type":45,"tag":210,"props":779,"children":780},{},[],{"type":45,"tag":183,"props":782,"children":783},{},[784,789,798],{"type":45,"tag":210,"props":785,"children":786},{},[787],{"type":58,"value":788},"Job Execution (CLI)",{"type":45,"tag":210,"props":790,"children":791},{},[792],{"type":45,"tag":159,"props":793,"children":795},{"href":794},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#job-execution",[796],{"type":58,"value":797},"COMMON-CLI.md § Job Execution",{"type":45,"tag":210,"props":799,"children":800},{},[],{"type":45,"tag":183,"props":802,"children":803},{},[804,809,818],{"type":45,"tag":210,"props":805,"children":806},{},[807],{"type":58,"value":808},"Job Scheduling",{"type":45,"tag":210,"props":810,"children":811},{},[812],{"type":45,"tag":159,"props":813,"children":815},{"href":814},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#job-scheduling",[816],{"type":58,"value":817},"COMMON-CLI.md § Job Scheduling",{"type":45,"tag":210,"props":819,"children":820},{},[821,823,829,831,837],{"type":58,"value":822},"URL is ",{"type":45,"tag":85,"props":824,"children":826},{"className":825},[],[827],{"type":58,"value":828},"\u002Fjobs\u002F{jobType}\u002Fschedules",{"type":58,"value":830},"; ",{"type":45,"tag":85,"props":832,"children":834},{"className":833},[],[835],{"type":58,"value":836},"endDateTime",{"type":58,"value":838}," required",{"type":45,"tag":183,"props":840,"children":841},{},[842,847,856],{"type":45,"tag":210,"props":843,"children":844},{},[845],{"type":58,"value":846},"OneLake Shortcuts",{"type":45,"tag":210,"props":848,"children":849},{},[850],{"type":45,"tag":159,"props":851,"children":853},{"href":852},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#onelake-shortcuts",[854],{"type":58,"value":855},"COMMON-CLI.md § OneLake Shortcuts",{"type":45,"tag":210,"props":857,"children":858},{},[],{"type":45,"tag":183,"props":860,"children":861},{},[862,867,876],{"type":45,"tag":210,"props":863,"children":864},{},[865],{"type":58,"value":866},"Capacity Management (CLI)",{"type":45,"tag":210,"props":868,"children":869},{},[870],{"type":45,"tag":159,"props":871,"children":873},{"href":872},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#capacity-management",[874],{"type":58,"value":875},"COMMON-CLI.md § Capacity Management",{"type":45,"tag":210,"props":877,"children":878},{},[],{"type":45,"tag":183,"props":880,"children":881},{},[882,887,896],{"type":45,"tag":210,"props":883,"children":884},{},[885],{"type":58,"value":886},"Composite Recipes",{"type":45,"tag":210,"props":888,"children":889},{},[890],{"type":45,"tag":159,"props":891,"children":893},{"href":892},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#composite-recipes",[894],{"type":58,"value":895},"COMMON-CLI.md § Composite Recipes",{"type":45,"tag":210,"props":897,"children":898},{},[],{"type":45,"tag":183,"props":900,"children":901},{},[902,907,916],{"type":45,"tag":210,"props":903,"children":904},{},[905],{"type":58,"value":906},"Gotchas & Troubleshooting (CLI-Specific)",{"type":45,"tag":210,"props":908,"children":909},{},[910],{"type":45,"tag":159,"props":911,"children":913},{"href":912},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#gotchas--troubleshooting-cli-specific",[914],{"type":58,"value":915},"COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific)",{"type":45,"tag":210,"props":917,"children":918},{},[919,924],{"type":45,"tag":85,"props":920,"children":922},{"className":921},[],[923],{"type":58,"value":646},{"type":58,"value":925}," audience, shell escaping, token expiry",{"type":45,"tag":183,"props":927,"children":928},{},[929,941,950],{"type":45,"tag":210,"props":930,"children":931},{},[932,934,939],{"type":58,"value":933},"Quick Reference: ",{"type":45,"tag":85,"props":935,"children":937},{"className":936},[],[938],{"type":58,"value":646},{"type":58,"value":940}," Template",{"type":45,"tag":210,"props":942,"children":943},{},[944],{"type":45,"tag":159,"props":945,"children":947},{"href":946},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#quick-reference-az-rest-template",[948],{"type":58,"value":949},"COMMON-CLI.md § Quick Reference: az rest Template",{"type":45,"tag":210,"props":951,"children":952},{},[],{"type":45,"tag":183,"props":954,"children":955},{},[956,961,970],{"type":45,"tag":210,"props":957,"children":958},{},[959],{"type":58,"value":960},"Quick Reference: Token Audience \u002F CLI Tool Matrix",{"type":45,"tag":210,"props":962,"children":963},{},[964],{"type":45,"tag":159,"props":965,"children":967},{"href":966},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#quick-reference-token-audience--cli-tool-matrix",[968],{"type":58,"value":969},"COMMON-CLI.md § Quick Reference: Token Audience ↔ CLI Tool Matrix",{"type":45,"tag":210,"props":971,"children":972},{},[973,975,981],{"type":58,"value":974},"Which ",{"type":45,"tag":85,"props":976,"children":978},{"className":977},[],[979],{"type":58,"value":980},"--resource",{"type":58,"value":982}," + tool for each service",{"type":45,"tag":183,"props":984,"children":985},{},[986,991,1000],{"type":45,"tag":210,"props":987,"children":988},{},[989],{"type":58,"value":990},"Relationship to SPARK-CONSUMPTION-CORE.md",{"type":45,"tag":210,"props":992,"children":993},{},[994],{"type":45,"tag":159,"props":995,"children":997},{"href":996},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#relationship-to-spark-consumption-coremd",[998],{"type":58,"value":999},"SPARK-AUTHORING-CORE.md § Relationship to SPARK-CONSUMPTION-CORE.md",{"type":45,"tag":210,"props":1001,"children":1002},{},[],{"type":45,"tag":183,"props":1004,"children":1005},{},[1006,1011,1020],{"type":45,"tag":210,"props":1007,"children":1008},{},[1009],{"type":58,"value":1010},"Data Engineering Authoring Capability Matrix",{"type":45,"tag":210,"props":1012,"children":1013},{},[1014],{"type":45,"tag":159,"props":1015,"children":1017},{"href":1016},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#data-engineering-authoring-capability-matrix",[1018],{"type":58,"value":1019},"SPARK-AUTHORING-CORE.md § Data Engineering Authoring Capability Matrix",{"type":45,"tag":210,"props":1021,"children":1022},{},[],{"type":45,"tag":183,"props":1024,"children":1025},{},[1026,1031,1040],{"type":45,"tag":210,"props":1027,"children":1028},{},[1029],{"type":58,"value":1030},"Lakehouse Management",{"type":45,"tag":210,"props":1032,"children":1033},{},[1034],{"type":45,"tag":159,"props":1035,"children":1037},{"href":1036},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#lakehouse-management",[1038],{"type":58,"value":1039},"SPARK-AUTHORING-CORE.md § Lakehouse Management",{"type":45,"tag":210,"props":1041,"children":1042},{},[],{"type":45,"tag":183,"props":1044,"children":1045},{},[1046,1051,1060],{"type":45,"tag":210,"props":1047,"children":1048},{},[1049],{"type":58,"value":1050},"Notebook Management",{"type":45,"tag":210,"props":1052,"children":1053},{},[1054],{"type":45,"tag":159,"props":1055,"children":1057},{"href":1056},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#notebook-management",[1058],{"type":58,"value":1059},"SPARK-AUTHORING-CORE.md § Notebook Management",{"type":45,"tag":210,"props":1061,"children":1062},{},[],{"type":45,"tag":183,"props":1064,"children":1065},{},[1066,1071,1080],{"type":45,"tag":210,"props":1067,"children":1068},{},[1069],{"type":58,"value":1070},"Notebook Execution & Job Management",{"type":45,"tag":210,"props":1072,"children":1073},{},[1074],{"type":45,"tag":159,"props":1075,"children":1077},{"href":1076},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#notebook-execution--job-management",[1078],{"type":58,"value":1079},"SPARK-AUTHORING-CORE.md § Notebook Execution & Job Management",{"type":45,"tag":210,"props":1081,"children":1082},{},[],{"type":45,"tag":183,"props":1084,"children":1085},{},[1086,1091,1100],{"type":45,"tag":210,"props":1087,"children":1088},{},[1089],{"type":58,"value":1090},"CI\u002FCD & Automation Patterns",{"type":45,"tag":210,"props":1092,"children":1093},{},[1094],{"type":45,"tag":159,"props":1095,"children":1097},{"href":1096},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#cicd--automation-patterns",[1098],{"type":58,"value":1099},"SPARK-AUTHORING-CORE.md § CI\u002FCD & Automation Patterns",{"type":45,"tag":210,"props":1101,"children":1102},{},[],{"type":45,"tag":183,"props":1104,"children":1105},{},[1106,1111,1120],{"type":45,"tag":210,"props":1107,"children":1108},{},[1109],{"type":58,"value":1110},"Infrastructure-as-Code",{"type":45,"tag":210,"props":1112,"children":1113},{},[1114],{"type":45,"tag":159,"props":1115,"children":1117},{"href":1116},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#infrastructure-as-code",[1118],{"type":58,"value":1119},"SPARK-AUTHORING-CORE.md § Infrastructure-as-Code",{"type":45,"tag":210,"props":1121,"children":1122},{},[],{"type":45,"tag":183,"props":1124,"children":1125},{},[1126,1131,1140],{"type":45,"tag":210,"props":1127,"children":1128},{},[1129],{"type":58,"value":1130},"Performance Optimization & Resource Management",{"type":45,"tag":210,"props":1132,"children":1133},{},[1134],{"type":45,"tag":159,"props":1135,"children":1137},{"href":1136},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#performance-optimization--resource-management",[1138],{"type":58,"value":1139},"SPARK-AUTHORING-CORE.md § Performance Optimization & Resource Management",{"type":45,"tag":210,"props":1141,"children":1142},{},[],{"type":45,"tag":183,"props":1144,"children":1145},{},[1146,1151,1160],{"type":45,"tag":210,"props":1147,"children":1148},{},[1149],{"type":58,"value":1150},"Runtime 2.0 Performance Features",{"type":45,"tag":210,"props":1152,"children":1153},{},[1154],{"type":45,"tag":159,"props":1155,"children":1157},{"href":1156},"resources\u002Fdata-engineering-patterns.md#runtime-20-performance-features-spark-41-delta-42",[1158],{"type":58,"value":1159},"data-engineering-patterns.md § Runtime 2.0 Performance Features",{"type":45,"tag":210,"props":1161,"children":1162},{},[1163],{"type":58,"value":1164},"NEE, Efficient Scaledown, Liquid Clustering, Spark 4.1",{"type":45,"tag":183,"props":1166,"children":1167},{},[1168,1173,1182],{"type":45,"tag":210,"props":1169,"children":1170},{},[1171],{"type":58,"value":1172},"Authoring Gotchas and Troubleshooting",{"type":45,"tag":210,"props":1174,"children":1175},{},[1176],{"type":45,"tag":159,"props":1177,"children":1179},{"href":1178},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#authoring-gotchas-and-troubleshooting",[1180],{"type":58,"value":1181},"SPARK-AUTHORING-CORE.md § Authoring Gotchas and Troubleshooting",{"type":45,"tag":210,"props":1183,"children":1184},{},[],{"type":45,"tag":183,"props":1186,"children":1187},{},[1188,1193,1202],{"type":45,"tag":210,"props":1189,"children":1190},{},[1191],{"type":58,"value":1192},"Quick Reference: Authoring Decision Guide",{"type":45,"tag":210,"props":1194,"children":1195},{},[1196],{"type":45,"tag":159,"props":1197,"children":1199},{"href":1198},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#quick-reference-authoring-decision-guide",[1200],{"type":58,"value":1201},"SPARK-AUTHORING-CORE.md § Quick Reference: Authoring Decision Guide",{"type":45,"tag":210,"props":1203,"children":1204},{},[],{"type":45,"tag":183,"props":1206,"children":1207},{},[1208,1213,1222],{"type":45,"tag":210,"props":1209,"children":1210},{},[1211],{"type":58,"value":1212},"Recommended Patterns (Data Engineering)",{"type":45,"tag":210,"props":1214,"children":1215},{},[1216],{"type":45,"tag":159,"props":1217,"children":1219},{"href":1218},"resources\u002Fdata-engineering-patterns.md#recommended-patterns",[1220],{"type":58,"value":1221},"data-engineering-patterns.md § Recommended patterns",{"type":45,"tag":210,"props":1223,"children":1224},{},[],{"type":45,"tag":183,"props":1226,"children":1227},{},[1228,1233,1242],{"type":45,"tag":210,"props":1229,"children":1230},{},[1231],{"type":58,"value":1232},"Data Ingestion Principles",{"type":45,"tag":210,"props":1234,"children":1235},{},[1236],{"type":45,"tag":159,"props":1237,"children":1239},{"href":1238},"resources\u002Fdata-engineering-patterns.md#data-ingestion-principles",[1240],{"type":58,"value":1241},"data-engineering-patterns.md § Data Ingestion Principles",{"type":45,"tag":210,"props":1243,"children":1244},{},[],{"type":45,"tag":183,"props":1246,"children":1247},{},[1248,1253,1262],{"type":45,"tag":210,"props":1249,"children":1250},{},[1251],{"type":58,"value":1252},"Transformation Patterns",{"type":45,"tag":210,"props":1254,"children":1255},{},[1256],{"type":45,"tag":159,"props":1257,"children":1259},{"href":1258},"resources\u002Fdata-engineering-patterns.md#transformation-patterns",[1260],{"type":58,"value":1261},"data-engineering-patterns.md § Transformation Patterns",{"type":45,"tag":210,"props":1263,"children":1264},{},[],{"type":45,"tag":183,"props":1266,"children":1267},{},[1268,1273,1282],{"type":45,"tag":210,"props":1269,"children":1270},{},[1271],{"type":58,"value":1272},"Delta Lake Best Practices",{"type":45,"tag":210,"props":1274,"children":1275},{},[1276],{"type":45,"tag":159,"props":1277,"children":1279},{"href":1278},"resources\u002Fdata-engineering-patterns.md#delta-lake-best-practices",[1280],{"type":58,"value":1281},"data-engineering-patterns.md § Delta Lake Best Practices",{"type":45,"tag":210,"props":1283,"children":1284},{},[],{"type":45,"tag":183,"props":1286,"children":1287},{},[1288,1293,1302],{"type":45,"tag":210,"props":1289,"children":1290},{},[1291],{"type":58,"value":1292},"Quality Assurance Strategies",{"type":45,"tag":210,"props":1294,"children":1295},{},[1296],{"type":45,"tag":159,"props":1297,"children":1299},{"href":1298},"resources\u002Fdata-engineering-patterns.md#quality-assurance-strategies",[1300],{"type":58,"value":1301},"data-engineering-patterns.md § Quality Assurance Strategies",{"type":45,"tag":210,"props":1303,"children":1304},{},[],{"type":45,"tag":183,"props":1306,"children":1307},{},[1308,1313,1322],{"type":45,"tag":210,"props":1309,"children":1310},{},[1311],{"type":58,"value":1312},"Recommended Patterns (Development Workflow)",{"type":45,"tag":210,"props":1314,"children":1315},{},[1316],{"type":45,"tag":159,"props":1317,"children":1319},{"href":1318},"resources\u002Fdevelopment-workflow.md#recommended-patterns",[1320],{"type":58,"value":1321},"development-workflow.md § Recommended patterns",{"type":45,"tag":210,"props":1323,"children":1324},{},[],{"type":45,"tag":183,"props":1326,"children":1327},{},[1328,1333,1342],{"type":45,"tag":210,"props":1329,"children":1330},{},[1331],{"type":58,"value":1332},"Notebook Lifecycle",{"type":45,"tag":210,"props":1334,"children":1335},{},[1336],{"type":45,"tag":159,"props":1337,"children":1339},{"href":1338},"resources\u002Fdevelopment-workflow.md#notebook-lifecycle",[1340],{"type":58,"value":1341},"development-workflow.md § Notebook Lifecycle",{"type":45,"tag":210,"props":1343,"children":1344},{},[],{"type":45,"tag":183,"props":1346,"children":1347},{},[1348,1353,1362],{"type":45,"tag":210,"props":1349,"children":1350},{},[1351],{"type":58,"value":1352},"Parameterization Patterns",{"type":45,"tag":210,"props":1354,"children":1355},{},[1356],{"type":45,"tag":159,"props":1357,"children":1359},{"href":1358},"resources\u002Fdevelopment-workflow.md#parameterization-patterns",[1360],{"type":58,"value":1361},"development-workflow.md § Parameterization Patterns",{"type":45,"tag":210,"props":1363,"children":1364},{},[],{"type":45,"tag":183,"props":1366,"children":1367},{},[1368,1373,1381],{"type":45,"tag":210,"props":1369,"children":1370},{},[1371],{"type":58,"value":1372},"Variable Library (notebook + pipeline usage)",{"type":45,"tag":210,"props":1374,"children":1375},{},[1376],{"type":45,"tag":159,"props":1377,"children":1378},{"href":1358},[1379],{"type":58,"value":1380},"development-workflow.md § Method 4: Variable Library",{"type":45,"tag":210,"props":1382,"children":1383},{},[1384,1390,1392,1398,1400,1406],{"type":45,"tag":85,"props":1385,"children":1387},{"className":1386},[],[1388],{"type":58,"value":1389},"getLibrary()",{"type":58,"value":1391}," + dot notation in notebooks; ",{"type":45,"tag":85,"props":1393,"children":1395},{"className":1394},[],[1396],{"type":58,"value":1397},"libraryVariables",{"type":58,"value":1399}," + ",{"type":45,"tag":85,"props":1401,"children":1403},{"className":1402},[],[1404],{"type":58,"value":1405},"@pipeline().libraryVariables",{"type":58,"value":1407}," in pipelines",{"type":45,"tag":183,"props":1409,"children":1410},{},[1411,1416,1425],{"type":45,"tag":210,"props":1412,"children":1413},{},[1414],{"type":58,"value":1415},"Variable Library Definition",{"type":45,"tag":210,"props":1417,"children":1418},{},[1419],{"type":45,"tag":159,"props":1420,"children":1422},{"href":1421},"..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#variablelibrary",[1423],{"type":58,"value":1424},"ITEM-DEFINITIONS-CORE.md § VariableLibrary",{"type":45,"tag":210,"props":1426,"children":1427},{},[1428],{"type":58,"value":1429},"Definition parts, decoded content, types, pipeline mappings, gotchas",{"type":45,"tag":183,"props":1431,"children":1432},{},[1433,1438,1447],{"type":45,"tag":210,"props":1434,"children":1435},{},[1436],{"type":58,"value":1437},"Local Testing Strategy",{"type":45,"tag":210,"props":1439,"children":1440},{},[1441],{"type":45,"tag":159,"props":1442,"children":1444},{"href":1443},"resources\u002Fdevelopment-workflow.md#local-testing-strategy",[1445],{"type":58,"value":1446},"development-workflow.md § Local Testing Strategy",{"type":45,"tag":210,"props":1448,"children":1449},{},[],{"type":45,"tag":183,"props":1451,"children":1452},{},[1453,1458,1467],{"type":45,"tag":210,"props":1454,"children":1455},{},[1456],{"type":58,"value":1457},"Debugging Patterns",{"type":45,"tag":210,"props":1459,"children":1460},{},[1461],{"type":45,"tag":159,"props":1462,"children":1464},{"href":1463},"resources\u002Fdevelopment-workflow.md#debugging-patterns",[1465],{"type":58,"value":1466},"development-workflow.md § Debugging Patterns",{"type":45,"tag":210,"props":1468,"children":1469},{},[],{"type":45,"tag":183,"props":1471,"children":1472},{},[1473,1478,1487],{"type":45,"tag":210,"props":1474,"children":1475},{},[1476],{"type":58,"value":1477},"Recommended Patterns (Infrastructure)",{"type":45,"tag":210,"props":1479,"children":1480},{},[1481],{"type":45,"tag":159,"props":1482,"children":1484},{"href":1483},"resources\u002Finfrastructure-orchestration.md#recommended-patterns",[1485],{"type":58,"value":1486},"infrastructure-orchestration.md § Recommended patterns",{"type":45,"tag":210,"props":1488,"children":1489},{},[],{"type":45,"tag":183,"props":1491,"children":1492},{},[1493,1498,1507],{"type":45,"tag":210,"props":1494,"children":1495},{},[1496],{"type":58,"value":1497},"Materialized Lake View patterns",{"type":45,"tag":210,"props":1499,"children":1500},{},[1501],{"type":45,"tag":159,"props":1502,"children":1504},{"href":1503},"resources\u002Fmaterialized-lake-view-patterns.md#recommended-patterns",[1505],{"type":58,"value":1506},"materialized-lake-view-patterns.md § Recommended patterns",{"type":45,"tag":210,"props":1508,"children":1509},{},[1510],{"type":58,"value":1511},"Spark Lakehouse authoring guidance for MLV design (when to use MLVs, layering patterns)",{"type":45,"tag":183,"props":1513,"children":1514},{},[1515,1520,1529],{"type":45,"tag":210,"props":1516,"children":1517},{},[1518],{"type":58,"value":1519},"MLV incremental refresh patterns",{"type":45,"tag":210,"props":1521,"children":1522},{},[1523],{"type":45,"tag":159,"props":1524,"children":1526},{"href":1525},"resources\u002Fmlv-incremental-refresh-patterns.md#ir-friendly-syntax-guide",[1527],{"type":58,"value":1528},"mlv-incremental-refresh-patterns.md § IR-friendly syntax guide",{"type":45,"tag":210,"props":1530,"children":1531},{},[1532],{"type":58,"value":1533},"Use for refresh-readiness review and safe non-breaking rewrites",{"type":45,"tag":183,"props":1535,"children":1536},{},[1537,1542,1551],{"type":45,"tag":210,"props":1538,"children":1539},{},[1540],{"type":58,"value":1541},"MLV schedule & job management",{"type":45,"tag":210,"props":1543,"children":1544},{},[1545],{"type":45,"tag":159,"props":1546,"children":1548},{"href":1547},"..\u002Fmlv-operations-cli\u002FSKILL.md",[1549],{"type":58,"value":1550},"mlv-operations-cli",{"type":45,"tag":210,"props":1552,"children":1553},{},[1554],{"type":58,"value":1555},"Route here when user asks to schedule, trigger, monitor, or cancel MLV refreshes (not authoring)",{"type":45,"tag":183,"props":1557,"children":1558},{},[1559,1564,1573],{"type":45,"tag":210,"props":1560,"children":1561},{},[1562],{"type":58,"value":1563},"Workspace Provisioning Principles",{"type":45,"tag":210,"props":1565,"children":1566},{},[1567],{"type":45,"tag":159,"props":1568,"children":1570},{"href":1569},"resources\u002Finfrastructure-orchestration.md#workspace-provisioning-principles",[1571],{"type":58,"value":1572},"infrastructure-orchestration.md § Workspace Provisioning Principles",{"type":45,"tag":210,"props":1574,"children":1575},{},[],{"type":45,"tag":183,"props":1577,"children":1578},{},[1579,1584,1593],{"type":45,"tag":210,"props":1580,"children":1581},{},[1582],{"type":58,"value":1583},"Lakehouse Configuration Guidance",{"type":45,"tag":210,"props":1585,"children":1586},{},[1587],{"type":45,"tag":159,"props":1588,"children":1590},{"href":1589},"resources\u002Finfrastructure-orchestration.md#lakehouse-configuration-guidance",[1591],{"type":58,"value":1592},"infrastructure-orchestration.md § Lakehouse Configuration Guidance",{"type":45,"tag":210,"props":1594,"children":1595},{},[],{"type":45,"tag":183,"props":1597,"children":1598},{},[1599,1604,1613],{"type":45,"tag":210,"props":1600,"children":1601},{},[1602],{"type":58,"value":1603},"Pipeline Design Patterns",{"type":45,"tag":210,"props":1605,"children":1606},{},[1607],{"type":45,"tag":159,"props":1608,"children":1610},{"href":1609},"resources\u002Finfrastructure-orchestration.md#pipeline-design-patterns",[1611],{"type":58,"value":1612},"infrastructure-orchestration.md § Pipeline Design Patterns",{"type":45,"tag":210,"props":1614,"children":1615},{},[],{"type":45,"tag":183,"props":1617,"children":1618},{},[1619,1624,1633],{"type":45,"tag":210,"props":1620,"children":1621},{},[1622],{"type":58,"value":1623},"CI\u002FCD Integration Strategy",{"type":45,"tag":210,"props":1625,"children":1626},{},[1627],{"type":45,"tag":159,"props":1628,"children":1630},{"href":1629},"resources\u002Finfrastructure-orchestration.md#cicd-integration-strategy",[1631],{"type":58,"value":1632},"infrastructure-orchestration.md § CI\u002FCD Integration Strategy",{"type":45,"tag":210,"props":1634,"children":1635},{},[],{"type":45,"tag":183,"props":1637,"children":1638},{},[1639,1644,1653],{"type":45,"tag":210,"props":1640,"children":1641},{},[1642],{"type":58,"value":1643},"Notebook API — Which Endpoint to Use",{"type":45,"tag":210,"props":1645,"children":1646},{},[1647],{"type":45,"tag":159,"props":1648,"children":1650},{"href":1649},"resources\u002Fnotebook-api-operations.md#quick-decision-which-endpoint-to-use",[1651],{"type":58,"value":1652},"notebook-api-operations.md § Quick Decision",{"type":45,"tag":210,"props":1654,"children":1655},{},[1656,1661],{"type":45,"tag":54,"props":1657,"children":1658},{},[1659],{"type":58,"value":1660},"Start here for remote notebook edits",{"type":58,"value":1662}," — getDefinition vs updateDefinition",{"type":45,"tag":183,"props":1664,"children":1665},{},[1666,1671,1680],{"type":45,"tag":210,"props":1667,"children":1668},{},[1669],{"type":58,"value":1670},"Notebook Modification Workflow",{"type":45,"tag":210,"props":1672,"children":1673},{},[1674],{"type":45,"tag":159,"props":1675,"children":1677},{"href":1676},"resources\u002Fnotebook-api-operations.md#workflow-get--decode--modify--encode--upload--verify",[1678],{"type":58,"value":1679},"notebook-api-operations.md § Workflow",{"type":45,"tag":210,"props":1681,"children":1682},{},[1683],{"type":58,"value":1684},"Five-step flow: retrieve, decode, modify, encode, upload",{"type":45,"tag":183,"props":1686,"children":1687},{},[1688,1693,1702],{"type":45,"tag":210,"props":1689,"children":1690},{},[1691],{"type":58,"value":1692},"Notebook Orchestration (parallel + DAG)",{"type":45,"tag":210,"props":1694,"children":1695},{},[1696],{"type":45,"tag":159,"props":1697,"children":1699},{"href":1698},"resources\u002Fnotebook-api-operations.md#notebook-orchestration--parallel-run--dag-dependencies",[1700],{"type":58,"value":1701},"notebook-api-operations.md § Notebook Orchestration",{"type":45,"tag":210,"props":1703,"children":1704},{},[1705,1711],{"type":45,"tag":85,"props":1706,"children":1708},{"className":1707},[],[1709],{"type":58,"value":1710},"notebookutils.notebook.runMultiple(DAG)",{"type":58,"value":1712}," for parallel runs and run-order dependencies (fan-in\u002Ffan-out); use instead of hand-rolled threads",{"type":45,"tag":183,"props":1714,"children":1715},{},[1716,1721,1730],{"type":45,"tag":210,"props":1717,"children":1718},{},[1719],{"type":58,"value":1720},"Notebook API Error Reference",{"type":45,"tag":210,"props":1722,"children":1723},{},[1724],{"type":45,"tag":159,"props":1725,"children":1727},{"href":1726},"resources\u002Fnotebook-api-operations.md#error-reference",[1728],{"type":58,"value":1729},"notebook-api-operations.md § Error Reference",{"type":45,"tag":210,"props":1731,"children":1732},{},[1733],{"type":58,"value":1734},"411, 400 (updateMetadata), 401, 403 explained",{"type":45,"tag":183,"props":1736,"children":1737},{},[1738,1743,1752],{"type":45,"tag":210,"props":1739,"children":1740},{},[1741],{"type":58,"value":1742},"Notebook API Gotchas",{"type":45,"tag":210,"props":1744,"children":1745},{},[1746],{"type":45,"tag":159,"props":1747,"children":1749},{"href":1748},"resources\u002Fnotebook-api-operations.md#gotchas",[1750],{"type":58,"value":1751},"notebook-api-operations.md § Gotchas",{"type":45,"tag":210,"props":1753,"children":1754},{},[1755,1761,1763,1769,1771],{"type":45,"tag":85,"props":1756,"children":1758},{"className":1757},[],[1759],{"type":58,"value":1760},"\u002Fresult",{"type":58,"value":1762}," suffix, empty body, ",{"type":45,"tag":85,"props":1764,"children":1766},{"className":1765},[],[1767],{"type":58,"value":1768},"\\n",{"type":58,"value":1770}," per-line rule, ",{"type":45,"tag":85,"props":1772,"children":1774},{"className":1773},[],[1775],{"type":58,"value":1776},"format=ipynb",{"type":45,"tag":183,"props":1778,"children":1779},{},[1780,1785,1794],{"type":45,"tag":210,"props":1781,"children":1782},{},[1783],{"type":58,"value":1784},"Default Lakehouse Binding",{"type":45,"tag":210,"props":1786,"children":1787},{},[1788],{"type":45,"tag":159,"props":1789,"children":1791},{"href":1790},"resources\u002Fnotebook-api-operations.md#default-lakehouse-binding",[1792],{"type":58,"value":1793},"notebook-api-operations.md § Default Lakehouse Binding",{"type":45,"tag":210,"props":1795,"children":1796},{},[1797,1803,1805,1811,1812,1818],{"type":45,"tag":85,"props":1798,"children":1800},{"className":1799},[],[1801],{"type":58,"value":1802},".ipynb",{"type":58,"value":1804}," metadata vs ",{"type":45,"tag":85,"props":1806,"children":1808},{"className":1807},[],[1809],{"type":58,"value":1810},".py",{"type":58,"value":268},{"type":45,"tag":85,"props":1813,"children":1815},{"className":1814},[],[1816],{"type":58,"value":1817},"# METADATA",{"type":58,"value":1819}," block; discover IDs dynamically",{"type":45,"tag":183,"props":1821,"children":1822},{},[1823,1828,1837],{"type":45,"tag":210,"props":1824,"children":1825},{},[1826],{"type":58,"value":1827},"Public URL Data Ingestion",{"type":45,"tag":210,"props":1829,"children":1830},{},[1831],{"type":45,"tag":159,"props":1832,"children":1834},{"href":1833},"resources\u002Fnotebook-api-operations.md#public-url-data-ingestion-spark",[1835],{"type":58,"value":1836},"notebook-api-operations.md § Public URL Data Ingestion",{"type":45,"tag":210,"props":1838,"children":1839},{},[1840,1842,1848],{"type":58,"value":1841},"Use real source URL, stage into ",{"type":45,"tag":85,"props":1843,"children":1845},{"className":1844},[],[1846],{"type":58,"value":1847},"Files\u002F",{"type":58,"value":1849},", then read with Spark",{"type":45,"tag":183,"props":1851,"children":1852},{},[1853,1858,1867],{"type":45,"tag":210,"props":1854,"children":1855},{},[1856],{"type":58,"value":1857},"getDefinition (read notebook content)",{"type":45,"tag":210,"props":1859,"children":1860},{},[1861],{"type":45,"tag":159,"props":1862,"children":1864},{"href":1863},"resources\u002Fnotebook-api-operations.md#step-1--retrieve-notebook-content-getdefinition",[1865],{"type":58,"value":1866},"notebook-api-operations.md § Step 1 — Retrieve Notebook Content",{"type":45,"tag":210,"props":1868,"children":1869},{},[1870,1872,1878,1880,1886],{"type":58,"value":1871},"LRO flow, ",{"type":45,"tag":85,"props":1873,"children":1875},{"className":1874},[],[1876],{"type":58,"value":1877},"?format=ipynb",{"type":58,"value":1879},", empty body (",{"type":45,"tag":85,"props":1881,"children":1883},{"className":1882},[],[1884],{"type":58,"value":1885},"--body '{}'",{"type":58,"value":1887},") requirement",{"type":45,"tag":183,"props":1889,"children":1890},{},[1891,1896,1905],{"type":45,"tag":210,"props":1892,"children":1893},{},[1894],{"type":58,"value":1895},"Decode Base64 Notebook Payload",{"type":45,"tag":210,"props":1897,"children":1898},{},[1899],{"type":45,"tag":159,"props":1900,"children":1902},{"href":1901},"resources\u002Fnotebook-api-operations.md#step-2--decode-the-notebook-content",[1903],{"type":58,"value":1904},"notebook-api-operations.md § Step 2 — Decode the Notebook Content",{"type":45,"tag":210,"props":1906,"children":1907},{},[1908],{"type":58,"value":1909},"Extract payload, base64 decode, ipynb JSON structure",{"type":45,"tag":183,"props":1911,"children":1912},{},[1913,1918,1927],{"type":45,"tag":210,"props":1914,"children":1915},{},[1916],{"type":58,"value":1917},"Modify Notebook Cells",{"type":45,"tag":210,"props":1919,"children":1920},{},[1921],{"type":45,"tag":159,"props":1922,"children":1924},{"href":1923},"resources\u002Fnotebook-api-operations.md#step-3--modify-the-notebook-content",[1925],{"type":58,"value":1926},"notebook-api-operations.md § Step 3 — Modify the Notebook Content",{"type":45,"tag":210,"props":1928,"children":1929},{},[1930,1932,1937],{"type":58,"value":1931},"Find cell, insert\u002Freplace lines, ",{"type":45,"tag":85,"props":1933,"children":1935},{"className":1934},[],[1936],{"type":58,"value":1768},{"type":58,"value":1938}," per-line rule",{"type":45,"tag":183,"props":1940,"children":1941},{},[1942,1947,1956],{"type":45,"tag":210,"props":1943,"children":1944},{},[1945],{"type":58,"value":1946},"updateDefinition (write notebook content)",{"type":45,"tag":210,"props":1948,"children":1949},{},[1950],{"type":45,"tag":159,"props":1951,"children":1953},{"href":1952},"resources\u002Fnotebook-api-operations.md#step-4--re-encode-and-upload-updatedefinition",[1954],{"type":58,"value":1955},"notebook-api-operations.md § Step 4 — Re-encode and Upload",{"type":45,"tag":210,"props":1957,"children":1958},{},[1959],{"type":58,"value":1960},"Re-encode, upload, LRO poll, updateMetadata flag pitfall",{"type":45,"tag":183,"props":1962,"children":1963},{},[1964,1969,1978],{"type":45,"tag":210,"props":1965,"children":1966},{},[1967],{"type":58,"value":1968},"Verify Notebook Update (Optional)",{"type":45,"tag":210,"props":1970,"children":1971},{},[1972],{"type":45,"tag":159,"props":1973,"children":1975},{"href":1974},"resources\u002Fnotebook-api-operations.md#step-5--verify-the-update",[1976],{"type":58,"value":1977},"notebook-api-operations.md § Step 5 — Verify the Update",{"type":45,"tag":210,"props":1979,"children":1980},{},[1981,1983,1989],{"type":58,"value":1982},"Skip unless you suspect a silent failure — ",{"type":45,"tag":85,"props":1984,"children":1986},{"className":1985},[],[1987],{"type":58,"value":1988},"Succeeded",{"type":58,"value":1990}," from updateDefinition is sufficient (see Rule 2)",{"type":45,"tag":183,"props":1992,"children":1993},{},[1994,1998,2005],{"type":45,"tag":210,"props":1995,"children":1996},{},[1997],{"type":58,"value":1720},{"type":45,"tag":210,"props":1999,"children":2000},{},[2001],{"type":45,"tag":159,"props":2002,"children":2003},{"href":1726},[2004],{"type":58,"value":1729},{"type":45,"tag":210,"props":2006,"children":2007},{},[2008],{"type":58,"value":1734},{"type":45,"tag":183,"props":2010,"children":2011},{},[2012,2017,2026],{"type":45,"tag":210,"props":2013,"children":2014},{},[2015],{"type":58,"value":2016},"Notebook API End-to-End Script",{"type":45,"tag":210,"props":2018,"children":2019},{},[2020],{"type":45,"tag":159,"props":2021,"children":2023},{"href":2022},"resources\u002Fnotebook-api-operations.md#complete-end-to-end-script",[2024],{"type":58,"value":2025},"notebook-api-operations.md § Complete End-to-End Script",{"type":45,"tag":210,"props":2027,"children":2028},{},[2029],{"type":58,"value":2030},"Full bash: get → decode → modify → encode → update → verify",{"type":45,"tag":183,"props":2032,"children":2033},{},[2034,2039,2048],{"type":45,"tag":210,"props":2035,"children":2036},{},[2037],{"type":58,"value":2038},"Quick Start Examples",{"type":45,"tag":210,"props":2040,"children":2041},{},[2042],{"type":45,"tag":159,"props":2043,"children":2045},{"href":2044},"#quick-start-examples",[2046],{"type":58,"value":2047},"SKILL.md § Quick Start Examples",{"type":45,"tag":210,"props":2049,"children":2050},{},[2051],{"type":58,"value":2052},"Minimal examples for common operations",{"type":45,"tag":183,"props":2054,"children":2055},{},[2056,2064,2067],{"type":45,"tag":210,"props":2057,"children":2058},{},[2059],{"type":45,"tag":54,"props":2060,"children":2061},{},[2062],{"type":58,"value":2063},"— Notebook Code Authoring (shared modules) —",{"type":45,"tag":210,"props":2065,"children":2066},{},[],{"type":45,"tag":210,"props":2068,"children":2069},{},[],{"type":45,"tag":183,"props":2071,"children":2072},{},[2073,2078,2085],{"type":45,"tag":210,"props":2074,"children":2075},{},[2076],{"type":58,"value":2077},"Notebook Authoring Core",{"type":45,"tag":210,"props":2079,"children":2080},{},[2081],{"type":45,"tag":159,"props":2082,"children":2083},{"href":161},[2084],{"type":58,"value":164},{"type":45,"tag":210,"props":2086,"children":2087},{},[2088,2093],{"type":45,"tag":54,"props":2089,"children":2090},{},[2091],{"type":58,"value":2092},"READ FIRST for notebook code tasks",{"type":58,"value":2094}," — fundamentals, code gen approach, module index",{"type":45,"tag":2096,"props":2097,"children":2098},"hr",{},[],{"type":45,"tag":168,"props":2100,"children":2102},{"id":2101},"mustpreferavoid",[2103],{"type":58,"value":2104},"Must\u002FPrefer\u002FAvoid",{"type":45,"tag":2106,"props":2107,"children":2109},"h3",{"id":2108},"must-do",[2110],{"type":58,"value":2111},"MUST DO",{"type":45,"tag":70,"props":2113,"children":2114},{},[2115,2125,2135,2145,2155,2172],{"type":45,"tag":74,"props":2116,"children":2117},{},[2118,2123],{"type":45,"tag":54,"props":2119,"children":2120},{},[2121],{"type":58,"value":2122},"Check for recent jobs BEFORE creating new notebook runs",{"type":58,"value":2124}," — Query job instances from last 5 minutes; if recent job exists, monitor it instead of creating duplicate",{"type":45,"tag":74,"props":2126,"children":2127},{},[2128,2133],{"type":45,"tag":54,"props":2129,"children":2130},{},[2131],{"type":58,"value":2132},"Capture job instance ID immediately after POST",{"type":58,"value":2134}," — Store job ID before any other operations to enable proper monitoring",{"type":45,"tag":74,"props":2136,"children":2137},{},[2138,2143],{"type":45,"tag":54,"props":2139,"children":2140},{},[2141],{"type":58,"value":2142},"Verify workspace capacity assignment",{"type":58,"value":2144}," before operations — Workspace must have capacity assigned and active",{"type":45,"tag":74,"props":2146,"children":2147},{},[2148,2153],{"type":45,"tag":54,"props":2149,"children":2150},{},[2151],{"type":58,"value":2152},"When user provides a public data URL, follow the Public URL Data Ingestion policy",{"type":58,"value":2154}," — keep detailed behavior in the linked resource section to avoid drift\u002Fduplication",{"type":45,"tag":74,"props":2156,"children":2157},{},[2158,2163,2165,2170],{"type":45,"tag":54,"props":2159,"children":2160},{},[2161],{"type":58,"value":2162},"Format notebook cells correctly",{"type":58,"value":2164}," — Each line in cell source array MUST end with ",{"type":45,"tag":85,"props":2166,"children":2168},{"className":2167},[],[2169],{"type":58,"value":1768},{"type":58,"value":2171}," to prevent code merging",{"type":45,"tag":74,"props":2173,"children":2174},{},[2175,2180,2182,2188,2189,2195,2196,2202,2203,2209,2210,2216,2218,2224,2226,2232],{"type":45,"tag":54,"props":2176,"children":2177},{},[2178],{"type":58,"value":2179},"Use correct Lakehouse Livy session body format",{"type":58,"value":2181}," — Send a FLAT JSON with ",{"type":45,"tag":85,"props":2183,"children":2185},{"className":2184},[],[2186],{"type":58,"value":2187},"name",{"type":58,"value":498},{"type":45,"tag":85,"props":2190,"children":2192},{"className":2191},[],[2193],{"type":58,"value":2194},"driverMemory",{"type":58,"value":498},{"type":45,"tag":85,"props":2197,"children":2199},{"className":2198},[],[2200],{"type":58,"value":2201},"driverCores",{"type":58,"value":498},{"type":45,"tag":85,"props":2204,"children":2206},{"className":2205},[],[2207],{"type":58,"value":2208},"executorMemory",{"type":58,"value":498},{"type":45,"tag":85,"props":2211,"children":2213},{"className":2212},[],[2214],{"type":58,"value":2215},"executorCores",{"type":58,"value":2217},". Do NOT wrap in ",{"type":45,"tag":85,"props":2219,"children":2221},{"className":2220},[],[2222],{"type":58,"value":2223},"{\"payload\": ...}",{"type":58,"value":2225}," or send only ",{"type":45,"tag":85,"props":2227,"children":2229},{"className":2228},[],[2230],{"type":58,"value":2231},"{\"kind\": \"pyspark\"}",{"type":58,"value":2233}," — that causes HTTP 500. Use valid memory values (28g, 56g, 112g, 224g). See Create Lakehouse Livy Session example below and SPARK-CONSUMPTION-CORE.md.",{"type":45,"tag":2106,"props":2235,"children":2237},{"id":2236},"prefer",[2238],{"type":58,"value":2239},"PREFER",{"type":45,"tag":70,"props":2241,"children":2242},{},[2243,2253,2263,2279,2295,2313],{"type":45,"tag":74,"props":2244,"children":2245},{},[2246,2251],{"type":45,"tag":54,"props":2247,"children":2248},{},[2249],{"type":58,"value":2250},"Poll job status with proper intervals",{"type":58,"value":2252}," — 10-30 seconds between polls; timeout after reasonable duration (e.g., 30 minutes)",{"type":45,"tag":74,"props":2254,"children":2255},{},[2256,2261],{"type":45,"tag":54,"props":2257,"children":2258},{},[2259],{"type":58,"value":2260},"Check job history when POST response is unreadable",{"type":58,"value":2262}," — If POST returns \"No Content\" or unreadable response, query recent jobs (last 1 minute) before retrying",{"type":45,"tag":74,"props":2264,"children":2265},{},[2266,2271,2273],{"type":45,"tag":54,"props":2267,"children":2268},{},[2269],{"type":58,"value":2270},"Use Starter Pool for development",{"type":58,"value":2272}," — Development\u002Ftesting workloads should use ",{"type":45,"tag":85,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":58,"value":2278},"useStarterPool: true",{"type":45,"tag":74,"props":2280,"children":2281},{},[2282,2287,2289],{"type":45,"tag":54,"props":2283,"children":2284},{},[2285],{"type":58,"value":2286},"Use Workspace Pool for production",{"type":58,"value":2288}," — Production workloads need consistent performance with ",{"type":45,"tag":85,"props":2290,"children":2292},{"className":2291},[],[2293],{"type":58,"value":2294},"useWorkspacePool: true",{"type":45,"tag":74,"props":2296,"children":2297},{},[2298,2303,2305,2311],{"type":45,"tag":54,"props":2299,"children":2300},{},[2301],{"type":58,"value":2302},"Enable lakehouse schemas",{"type":58,"value":2304}," during creation — Set ",{"type":45,"tag":85,"props":2306,"children":2308},{"className":2307},[],[2309],{"type":58,"value":2310},"creationPayload.enableSchemas: true",{"type":58,"value":2312}," for better table organization",{"type":45,"tag":74,"props":2314,"children":2315},{},[2316,2321],{"type":45,"tag":54,"props":2317,"children":2318},{},[2319],{"type":58,"value":2320},"Implement idempotency checks",{"type":58,"value":2322}," — Prevent duplicate operations by checking existing state first",{"type":45,"tag":2106,"props":2324,"children":2326},{"id":2325},"avoid",[2327],{"type":58,"value":2328},"AVOID",{"type":45,"tag":70,"props":2330,"children":2331},{},[2332,2342,2352,2362,2372,2382,2413,2439,2456],{"type":45,"tag":74,"props":2333,"children":2334},{},[2335,2340],{"type":45,"tag":54,"props":2336,"children":2337},{},[2338],{"type":58,"value":2339},"Never retry POST with same parameters",{"type":58,"value":2341}," — If you have a job ID, only use GET to check status; don't create duplicate job instances",{"type":45,"tag":74,"props":2343,"children":2344},{},[2345,2350],{"type":45,"tag":54,"props":2346,"children":2347},{},[2348],{"type":58,"value":2349},"Don't skip capacity verification",{"type":58,"value":2351}," — Operations will fail if workspace capacity is paused or unassigned",{"type":45,"tag":74,"props":2353,"children":2354},{},[2355,2360],{"type":45,"tag":54,"props":2356,"children":2357},{},[2358],{"type":58,"value":2359},"Avoid immediate POST retries on failures",{"type":58,"value":2361}," — Check for existing\u002Factive jobs first to prevent duplicates",{"type":45,"tag":74,"props":2363,"children":2364},{},[2365,2370],{"type":45,"tag":54,"props":2366,"children":2367},{},[2368],{"type":58,"value":2369},"Don't create new runs if monitoring existing job",{"type":58,"value":2371}," — One job at a time; wait for completion before submitting new runs",{"type":45,"tag":74,"props":2373,"children":2374},{},[2375,2380],{"type":45,"tag":54,"props":2376,"children":2377},{},[2378],{"type":58,"value":2379},"Don't hardcode workspace\u002Flakehouse IDs",{"type":58,"value":2381}," — Discover dynamically via item listing or catalog search APIs",{"type":45,"tag":74,"props":2383,"children":2384},{},[2385,2404,2406,2411],{"type":45,"tag":54,"props":2386,"children":2387},{},[2388,2390,2396,2398],{"type":58,"value":2389},"Own ",{"type":45,"tag":85,"props":2391,"children":2393},{"className":2392},[],[2394],{"type":58,"value":2395},"%%sql",{"type":58,"value":2397}," notebook cells here, do not defer to ",{"type":45,"tag":85,"props":2399,"children":2401},{"className":2400},[],[2402],{"type":58,"value":2403},"sqldw-consumption-cli",{"type":58,"value":2405}," — a request to \"write a %%sql cell\" (or any notebook magic cell) is notebook-cell authoring even when the cell queries a lakehouse table. Only route to ",{"type":45,"tag":85,"props":2407,"children":2409},{"className":2408},[],[2410],{"type":58,"value":2403},{"type":58,"value":2412}," when the user wants a plain T-SQL query executed against a SQL endpoint, not a notebook cell.",{"type":45,"tag":74,"props":2414,"children":2415},{},[2416,2421,2423,2429,2431,2437],{"type":45,"tag":54,"props":2417,"children":2418},{},[2419],{"type":58,"value":2420},"Own \"run\u002Fexecute the notebook named X\" here",{"type":58,"value":2422}," — running a Fabric notebook by name is notebook execution via the Jobs API (",{"type":45,"tag":85,"props":2424,"children":2426},{"className":2425},[],[2427],{"type":58,"value":2428},"RunNotebook",{"type":58,"value":2430},"), which belongs to this skill; do not defer to ",{"type":45,"tag":85,"props":2432,"children":2434},{"className":2433},[],[2435],{"type":58,"value":2436},"spark-consumption-cli",{"type":58,"value":2438}," (that skill is only for ad-hoc Livy session code execution).",{"type":45,"tag":74,"props":2440,"children":2441},{},[2442,2447,2449,2454],{"type":45,"tag":54,"props":2443,"children":2444},{},[2445],{"type":58,"value":2446},"Do NOT use Lakehouse Livy sessions to run a Fabric notebook",{"type":58,"value":2448}," — Lakehouse Livy sessions (the public Livy API) are for ad-hoc interactive Spark code execution. To run a notebook as a job, use the Jobs API (",{"type":45,"tag":85,"props":2450,"children":2452},{"className":2451},[],[2453],{"type":58,"value":2428},{"type":58,"value":2455},") which creates a Notebook Spark session internally. See SPARK-AUTHORING-CORE.md § Notebook Execution & Job Management",{"type":45,"tag":74,"props":2457,"children":2458},{},[2459,2464,2466,2470,2472,2478],{"type":45,"tag":54,"props":2460,"children":2461},{},[2462],{"type":58,"value":2463},"Do NOT schedule MLV refreshes from notebooks",{"type":58,"value":2465}," — If the user asks to \"schedule MLV refresh\", route to ",{"type":45,"tag":159,"props":2467,"children":2468},{"href":1547},[2469],{"type":58,"value":1550},{"type":58,"value":2471}," which uses the REST API. Notebook-based ",{"type":45,"tag":85,"props":2473,"children":2475},{"className":2474},[],[2476],{"type":58,"value":2477},"REFRESH MATERIALIZED LAKE VIEW ... FULL",{"type":58,"value":2479}," is for one-time manual refresh only, not recurring schedules.",{"type":45,"tag":2096,"props":2481,"children":2482},{},[],{"type":45,"tag":168,"props":2484,"children":2486},{"id":2485},"rules-read-these-first-follow-them-always",[2487],{"type":58,"value":214},{"type":45,"tag":46,"props":2489,"children":2490},{},[2491,2501,2541,2551],{"type":45,"tag":50,"props":2492,"children":2493},{},[2494,2499],{"type":45,"tag":54,"props":2495,"children":2496},{},[2497],{"type":58,"value":2498},"Rule 1 — Validate prerequisites before operations.",{"type":58,"value":2500},"\nVerify workspace has capacity assigned (see COMMON-CORE.md Create Workspace and Capacity Management) and resource IDs exist before attempting operations.",{"type":45,"tag":50,"props":2502,"children":2503},{},[2504,2509,2511,2516,2518,2524,2526,2532,2534,2539],{"type":45,"tag":54,"props":2505,"children":2506},{},[2507],{"type":58,"value":2508},"Rule 2 — Trust updateDefinition success.",{"type":58,"value":2510},"\nA ",{"type":45,"tag":85,"props":2512,"children":2514},{"className":2513},[],[2515],{"type":58,"value":1988},{"type":58,"value":2517}," poll result from ",{"type":45,"tag":85,"props":2519,"children":2521},{"className":2520},[],[2522],{"type":58,"value":2523},"updateDefinition",{"type":58,"value":2525}," is sufficient confirmation that content and lakehouse bindings persisted. Do NOT call ",{"type":45,"tag":85,"props":2527,"children":2529},{"className":2528},[],[2530],{"type":58,"value":2531},"getDefinition",{"type":58,"value":2533}," after every upload — it is an async LRO that adds significant latency. Only use ",{"type":45,"tag":85,"props":2535,"children":2537},{"className":2536},[],[2538],{"type":58,"value":2531},{"type":58,"value":2540}," for its intended purpose: reading current notebook content before making modifications.",{"type":45,"tag":50,"props":2542,"children":2543},{},[2544,2549],{"type":45,"tag":54,"props":2545,"children":2546},{},[2547],{"type":58,"value":2548},"Rule 3 — Prevent duplicate jobs and monitor execution properly.",{"type":58,"value":2550},"\nBefore submitting new notebook run, ALWAYS check for recent job instances first (last 5 minutes). If recent job exists, monitor it instead of creating duplicate. After submission, capture job instance ID immediately and poll status - never retry POST. See SPARK-AUTHORING-CORE.md Job Monitoring for patterns.",{"type":45,"tag":50,"props":2552,"children":2553},{},[2554,2559,2561,2565,2567,2573,2574,2580,2582,2588,2590,2596],{"type":45,"tag":54,"props":2555,"children":2556},{},[2557],{"type":58,"value":2558},"Rule 4 — For notebook code authoring, MUST follow SPARK-NOTEBOOK-AUTHORING-CORE.md.",{"type":58,"value":2560},"\nWhen writing code inside notebook cells, MUST read ",{"type":45,"tag":159,"props":2562,"children":2563},{"href":161},[2564],{"type":58,"value":164},{"type":58,"value":2566}," first — it defines the code generation approach, rules, and a Module Index linking to detailed guides (lakehouse paths, connections, context, orchestration, etc.). Use the Spark-specific resources in this skill (",{"type":45,"tag":159,"props":2568,"children":2570},{"href":2569},"resources\u002Fdata-engineering-patterns.md",[2571],{"type":58,"value":2572},"data-engineering-patterns.md",{"type":58,"value":498},{"type":45,"tag":159,"props":2575,"children":2577},{"href":2576},"resources\u002Fdevelopment-workflow.md",[2578],{"type":58,"value":2579},"development-workflow.md",{"type":58,"value":2581},") for Spark-only implementation details. When the task is about Materialized Lake Views, read ",{"type":45,"tag":159,"props":2583,"children":2585},{"href":2584},"resources\u002Fmaterialized-lake-view-patterns.md",[2586],{"type":58,"value":2587},"materialized-lake-view-patterns.md",{"type":58,"value":2589}," for authoring\u002Fdesign guidance and ",{"type":45,"tag":159,"props":2591,"children":2593},{"href":2592},"resources\u002Fmlv-incremental-refresh-patterns.md",[2594],{"type":58,"value":2595},"mlv-incremental-refresh-patterns.md",{"type":58,"value":2597}," for refresh-readiness analysis.",{"type":45,"tag":2096,"props":2599,"children":2600},{},[],{"type":45,"tag":168,"props":2602,"children":2604},{"id":2603},"notebook-codegen-quick-rules",[2605],{"type":58,"value":2606},"Notebook Codegen Quick Rules",{"type":45,"tag":50,"props":2608,"children":2609},{},[2610,2612,2618,2620,2624],{"type":58,"value":2611},"Quick reference for common notebook-authoring tasks. The shared ",{"type":45,"tag":85,"props":2613,"children":2615},{"className":2614},[],[2616],{"type":58,"value":2617},"common\u002Fnotebook-authoring\u002F",{"type":58,"value":2619}," core (see Rule 4 \u002F ",{"type":45,"tag":159,"props":2621,"children":2622},{"href":161},[2623],{"type":58,"value":164},{"type":58,"value":2625},") is authoritative; if these ever differ, follow the common core.",{"type":45,"tag":175,"props":2627,"children":2628},{},[2629,2645],{"type":45,"tag":179,"props":2630,"children":2631},{},[2632],{"type":45,"tag":183,"props":2633,"children":2634},{},[2635,2640],{"type":45,"tag":187,"props":2636,"children":2637},{},[2638],{"type":58,"value":2639},"User asks for",{"type":45,"tag":187,"props":2641,"children":2642},{},[2643],{"type":58,"value":2644},"Required output pattern",{"type":45,"tag":203,"props":2646,"children":2647},{},[2648,2673,2708,2766],{"type":45,"tag":183,"props":2649,"children":2650},{},[2651,2661],{"type":45,"tag":210,"props":2652,"children":2653},{},[2654,2659],{"type":45,"tag":85,"props":2655,"children":2657},{"className":2656},[],[2658],{"type":58,"value":2395},{"type":58,"value":2660}," \u002F cross-lakehouse query cell",{"type":45,"tag":210,"props":2662,"children":2663},{},[2664,2666,2671],{"type":58,"value":2665},"Return a Fabric notebook ",{"type":45,"tag":85,"props":2667,"children":2669},{"className":2668},[],[2670],{"type":58,"value":2395},{"type":58,"value":2672}," cell. Include the named workspace\u002Flakehouse\u002Fschema\u002Ftable in the code or explanatory note. This is notebook authoring, not interactive Spark consumption.",{"type":45,"tag":183,"props":2674,"children":2675},{},[2676,2681],{"type":45,"tag":210,"props":2677,"children":2678},{},[2679],{"type":58,"value":2680},"Pipeline context detection",{"type":45,"tag":210,"props":2682,"children":2683},{},[2684,2685,2691,2693,2699,2701,2707],{"type":58,"value":746},{"type":45,"tag":85,"props":2686,"children":2688},{"className":2687},[],[2689],{"type":58,"value":2690},"notebookutils.runtime.context",{"type":58,"value":2692},"; include ",{"type":45,"tag":85,"props":2694,"children":2696},{"className":2695},[],[2697],{"type":58,"value":2698},"isForPipeline = context[\"isForPipeline\"]",{"type":58,"value":2700}," and read ",{"type":45,"tag":85,"props":2702,"children":2704},{"className":2703},[],[2705],{"type":58,"value":2706},"context[\"currentWorkspaceId\"]",{"type":58,"value":166},{"type":45,"tag":183,"props":2709,"children":2710},{},[2711,2716],{"type":45,"tag":210,"props":2712,"children":2713},{},[2714],{"type":58,"value":2715},"Built-in notebook resource files with Spark",{"type":45,"tag":210,"props":2717,"children":2718},{},[2719,2720,2726,2727,2733,2735,2741,2743,2749,2751,2757,2759,2764],{"type":58,"value":746},{"type":45,"tag":85,"props":2721,"children":2723},{"className":2722},[],[2724],{"type":58,"value":2725},"notebookutils.nbResPath",{"type":58,"value":498},{"type":45,"tag":85,"props":2728,"children":2730},{"className":2729},[],[2731],{"type":58,"value":2732},"builtin\u002F",{"type":58,"value":2734},", and the ",{"type":45,"tag":85,"props":2736,"children":2738},{"className":2737},[],[2739],{"type":58,"value":2740},"file:",{"type":58,"value":2742}," prefix: ",{"type":45,"tag":85,"props":2744,"children":2746},{"className":2745},[],[2747],{"type":58,"value":2748},"spark.read.json(f\"file:{notebookutils.nbResPath}\u002Fbuiltin\u002Fconfig.json\")",{"type":58,"value":2750},". Spark and ",{"type":45,"tag":85,"props":2752,"children":2754},{"className":2753},[],[2755],{"type":58,"value":2756},"notebookutils.fs",{"type":58,"value":2758}," require ",{"type":45,"tag":85,"props":2760,"children":2762},{"className":2761},[],[2763],{"type":58,"value":2740},{"type":58,"value":2765}," for resource-folder local paths.",{"type":45,"tag":183,"props":2767,"children":2768},{},[2769,2774],{"type":45,"tag":210,"props":2770,"children":2771},{},[2772],{"type":58,"value":2773},"Fabric connections",{"type":45,"tag":210,"props":2775,"children":2776},{},[2777,2778,2784],{"type":58,"value":746},{"type":45,"tag":85,"props":2779,"children":2781},{"className":2780},[],[2782],{"type":58,"value":2783},"notebookutils.connections.getCredential(\"{connectionId}\")",{"type":58,"value":2785}," directly and show a compact code sample. Do not web-search; keep the answer under the tool\u002Fturn budget.",{"type":45,"tag":2106,"props":2787,"children":2789},{"id":2788},"minimal-snippets",[2790],{"type":58,"value":2791},"Minimal snippets",{"type":45,"tag":2793,"props":2794,"children":2799},"pre",{"className":2795,"code":2796,"language":2797,"meta":2798,"style":2798},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Pipeline context\ncontext = notebookutils.runtime.context\nisForPipeline = context[\"isForPipeline\"]\nworkspace_id = context[\"currentWorkspaceId\"]\nprint(f\"isForPipeline={isForPipeline}, workspace_id={workspace_id}\")\n","python","",[2800],{"type":45,"tag":85,"props":2801,"children":2802},{"__ignoreMap":2798},[2803,2813,2822,2831,2840],{"type":45,"tag":270,"props":2804,"children":2807},{"class":2805,"line":2806},"line",1,[2808],{"type":45,"tag":270,"props":2809,"children":2810},{},[2811],{"type":58,"value":2812},"# Pipeline context\n",{"type":45,"tag":270,"props":2814,"children":2816},{"class":2805,"line":2815},2,[2817],{"type":45,"tag":270,"props":2818,"children":2819},{},[2820],{"type":58,"value":2821},"context = notebookutils.runtime.context\n",{"type":45,"tag":270,"props":2823,"children":2825},{"class":2805,"line":2824},3,[2826],{"type":45,"tag":270,"props":2827,"children":2828},{},[2829],{"type":58,"value":2830},"isForPipeline = context[\"isForPipeline\"]\n",{"type":45,"tag":270,"props":2832,"children":2834},{"class":2805,"line":2833},4,[2835],{"type":45,"tag":270,"props":2836,"children":2837},{},[2838],{"type":58,"value":2839},"workspace_id = context[\"currentWorkspaceId\"]\n",{"type":45,"tag":270,"props":2841,"children":2843},{"class":2805,"line":2842},5,[2844],{"type":45,"tag":270,"props":2845,"children":2846},{},[2847],{"type":58,"value":2848},"print(f\"isForPipeline={isForPipeline}, workspace_id={workspace_id}\")\n",{"type":45,"tag":2793,"props":2850,"children":2852},{"className":2795,"code":2851,"language":2797,"meta":2798,"style":2798},"# Spark read from notebook built-in resources\nconfig_df = spark.read.option(\"multiline\", \"true\").json(\n    f\"file:{notebookutils.nbResPath}\u002Fbuiltin\u002Fconfig.json\"\n)\ndisplay(config_df)\n",[2853],{"type":45,"tag":85,"props":2854,"children":2855},{"__ignoreMap":2798},[2856,2864,2872,2880,2888],{"type":45,"tag":270,"props":2857,"children":2858},{"class":2805,"line":2806},[2859],{"type":45,"tag":270,"props":2860,"children":2861},{},[2862],{"type":58,"value":2863},"# Spark read from notebook built-in resources\n",{"type":45,"tag":270,"props":2865,"children":2866},{"class":2805,"line":2815},[2867],{"type":45,"tag":270,"props":2868,"children":2869},{},[2870],{"type":58,"value":2871},"config_df = spark.read.option(\"multiline\", \"true\").json(\n",{"type":45,"tag":270,"props":2873,"children":2874},{"class":2805,"line":2824},[2875],{"type":45,"tag":270,"props":2876,"children":2877},{},[2878],{"type":58,"value":2879},"    f\"file:{notebookutils.nbResPath}\u002Fbuiltin\u002Fconfig.json\"\n",{"type":45,"tag":270,"props":2881,"children":2882},{"class":2805,"line":2833},[2883],{"type":45,"tag":270,"props":2884,"children":2885},{},[2886],{"type":58,"value":2887},")\n",{"type":45,"tag":270,"props":2889,"children":2890},{"class":2805,"line":2842},[2891],{"type":45,"tag":270,"props":2892,"children":2893},{},[2894],{"type":58,"value":2895},"display(config_df)\n",{"type":45,"tag":2793,"props":2897,"children":2899},{"className":2795,"code":2898,"language":2797,"meta":2798,"style":2798},"# PostgreSQL through a Fabric connection (fill in connectionId and host\u002Fdb)\nimport psycopg2\n\ncredential = notebookutils.connections.getCredential(\"{connectionId}\")\nconn = psycopg2.connect(\n    host=\"\u003Cpostgres-host>\",\n    database=\"\u003Cdatabase>\",\n    user=credential[\"username\"],\n    password=credential[\"password\"],\n)\n",[2900],{"type":45,"tag":85,"props":2901,"children":2902},{"__ignoreMap":2798},[2903,2911,2919,2928,2936,2944,2953,2962,2971,2980],{"type":45,"tag":270,"props":2904,"children":2905},{"class":2805,"line":2806},[2906],{"type":45,"tag":270,"props":2907,"children":2908},{},[2909],{"type":58,"value":2910},"# PostgreSQL through a Fabric connection (fill in connectionId and host\u002Fdb)\n",{"type":45,"tag":270,"props":2912,"children":2913},{"class":2805,"line":2815},[2914],{"type":45,"tag":270,"props":2915,"children":2916},{},[2917],{"type":58,"value":2918},"import psycopg2\n",{"type":45,"tag":270,"props":2920,"children":2921},{"class":2805,"line":2824},[2922],{"type":45,"tag":270,"props":2923,"children":2925},{"emptyLinePlaceholder":2924},true,[2926],{"type":58,"value":2927},"\n",{"type":45,"tag":270,"props":2929,"children":2930},{"class":2805,"line":2833},[2931],{"type":45,"tag":270,"props":2932,"children":2933},{},[2934],{"type":58,"value":2935},"credential = notebookutils.connections.getCredential(\"{connectionId}\")\n",{"type":45,"tag":270,"props":2937,"children":2938},{"class":2805,"line":2842},[2939],{"type":45,"tag":270,"props":2940,"children":2941},{},[2942],{"type":58,"value":2943},"conn = psycopg2.connect(\n",{"type":45,"tag":270,"props":2945,"children":2947},{"class":2805,"line":2946},6,[2948],{"type":45,"tag":270,"props":2949,"children":2950},{},[2951],{"type":58,"value":2952},"    host=\"\u003Cpostgres-host>\",\n",{"type":45,"tag":270,"props":2954,"children":2956},{"class":2805,"line":2955},7,[2957],{"type":45,"tag":270,"props":2958,"children":2959},{},[2960],{"type":58,"value":2961},"    database=\"\u003Cdatabase>\",\n",{"type":45,"tag":270,"props":2963,"children":2965},{"class":2805,"line":2964},8,[2966],{"type":45,"tag":270,"props":2967,"children":2968},{},[2969],{"type":58,"value":2970},"    user=credential[\"username\"],\n",{"type":45,"tag":270,"props":2972,"children":2974},{"class":2805,"line":2973},9,[2975],{"type":45,"tag":270,"props":2976,"children":2977},{},[2978],{"type":58,"value":2979},"    password=credential[\"password\"],\n",{"type":45,"tag":270,"props":2981,"children":2983},{"class":2805,"line":2982},10,[2984],{"type":45,"tag":270,"props":2985,"children":2986},{},[2987],{"type":58,"value":2887},{"type":45,"tag":2096,"props":2989,"children":2990},{},[],{"type":45,"tag":168,"props":2992,"children":2994},{"id":2993},"quick-start-examples",[2995],{"type":58,"value":2038},{"type":45,"tag":50,"props":2997,"children":2998},{},[2999],{"type":58,"value":3000},"For detailed patterns, authentication, and comprehensive API usage, see:",{"type":45,"tag":70,"props":3002,"children":3003},{},[3004,3014,3030],{"type":45,"tag":74,"props":3005,"children":3006},{},[3007,3012],{"type":45,"tag":54,"props":3008,"children":3009},{},[3010],{"type":58,"value":3011},"COMMON-CORE.md",{"type":58,"value":3013}," — Fabric REST API patterns, authentication, item discovery",{"type":45,"tag":74,"props":3015,"children":3016},{},[3017,3022,3023,3028],{"type":45,"tag":54,"props":3018,"children":3019},{},[3020],{"type":58,"value":3021},"COMMON-CLI.md",{"type":58,"value":260},{"type":45,"tag":85,"props":3024,"children":3026},{"className":3025},[],[3027],{"type":58,"value":646},{"type":58,"value":3029}," usage, environment detection, token acquisition",{"type":45,"tag":74,"props":3031,"children":3032},{},[3033,3038],{"type":45,"tag":54,"props":3034,"children":3035},{},[3036],{"type":58,"value":3037},"SPARK-AUTHORING-CORE.md",{"type":58,"value":3039}," — Notebook deployment, lakehouse creation, job execution",{"type":45,"tag":50,"props":3041,"children":3042},{},[3043,3045,3055],{"type":58,"value":3044},"Below are minimal quick-start examples. ",{"type":45,"tag":262,"props":3046,"children":3047},{},[3048,3053],{"type":45,"tag":262,"props":3049,"children":3050},{},[3051],{"type":58,"value":3052},"Always reference the COMMON-",{"type":58,"value":3054}," files for production use.",{"type":58,"value":3056},"*",{"type":45,"tag":2106,"props":3058,"children":3060},{"id":3059},"create-workspace-lakehouse",[3061],{"type":58,"value":3062},"Create Workspace & Lakehouse",{"type":45,"tag":2793,"props":3064,"children":3068},{"className":3065,"code":3066,"language":3067,"meta":2798,"style":2798},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# See COMMON-CORE.md Environment URLs and SPARK-AUTHORING-CORE.md for full patterns\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"DataEng-Dev\"}\nEOF\nworkspace_id=$(az rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\" \\\n  --body @\u002Ftmp\u002Fbody.json --query \"id\" --output tsv)\n\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"DevLakehouse\", \"type\": \"Lakehouse\", \"creationPayload\": {\"enableSchemas\": true}}\nEOF\nlakehouse_id=$(az rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F$workspace_id\u002Fitems\" \\\n  --body @\u002Ftmp\u002Fbody.json --query \"id\" --output tsv)\n","bash",[3069],{"type":45,"tag":85,"props":3070,"children":3071},{"__ignoreMap":2798},[3072,3081,3112,3120,3128,3187,3212,3257,3264,3287,3295,3303,3352,3387],{"type":45,"tag":270,"props":3073,"children":3074},{"class":2805,"line":2806},[3075],{"type":45,"tag":270,"props":3076,"children":3078},{"style":3077},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[3079],{"type":58,"value":3080},"# See COMMON-CORE.md Environment URLs and SPARK-AUTHORING-CORE.md for full patterns\n",{"type":45,"tag":270,"props":3082,"children":3083},{"class":2805,"line":2815},[3084,3090,3096,3102,3107],{"type":45,"tag":270,"props":3085,"children":3087},{"style":3086},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[3088],{"type":58,"value":3089},"cat",{"type":45,"tag":270,"props":3091,"children":3093},{"style":3092},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[3094],{"type":58,"value":3095}," >",{"type":45,"tag":270,"props":3097,"children":3099},{"style":3098},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[3100],{"type":58,"value":3101}," \u002Ftmp\u002Fbody.json",{"type":45,"tag":270,"props":3103,"children":3104},{"style":3092},[3105],{"type":58,"value":3106}," \u003C\u003C",{"type":45,"tag":270,"props":3108,"children":3109},{"style":3092},[3110],{"type":58,"value":3111}," 'EOF'\n",{"type":45,"tag":270,"props":3113,"children":3114},{"class":2805,"line":2824},[3115],{"type":45,"tag":270,"props":3116,"children":3117},{"style":3098},[3118],{"type":58,"value":3119},"{\"displayName\": \"DataEng-Dev\"}\n",{"type":45,"tag":270,"props":3121,"children":3122},{"class":2805,"line":2833},[3123],{"type":45,"tag":270,"props":3124,"children":3125},{"style":3092},[3126],{"type":58,"value":3127},"EOF\n",{"type":45,"tag":270,"props":3129,"children":3130},{"class":2805,"line":2842},[3131,3137,3142,3147,3152,3157,3162,3167,3172,3177,3182],{"type":45,"tag":270,"props":3132,"children":3134},{"style":3133},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[3135],{"type":58,"value":3136},"workspace_id",{"type":45,"tag":270,"props":3138,"children":3139},{"style":3092},[3140],{"type":58,"value":3141},"=$(",{"type":45,"tag":270,"props":3143,"children":3144},{"style":3086},[3145],{"type":58,"value":3146},"az",{"type":45,"tag":270,"props":3148,"children":3149},{"style":3098},[3150],{"type":58,"value":3151}," rest",{"type":45,"tag":270,"props":3153,"children":3154},{"style":3098},[3155],{"type":58,"value":3156}," --method",{"type":45,"tag":270,"props":3158,"children":3159},{"style":3098},[3160],{"type":58,"value":3161}," post",{"type":45,"tag":270,"props":3163,"children":3164},{"style":3098},[3165],{"type":58,"value":3166}," --resource",{"type":45,"tag":270,"props":3168,"children":3169},{"style":3092},[3170],{"type":58,"value":3171}," \"",{"type":45,"tag":270,"props":3173,"children":3174},{"style":3098},[3175],{"type":58,"value":3176},"https:\u002F\u002Fapi.fabric.microsoft.com",{"type":45,"tag":270,"props":3178,"children":3179},{"style":3092},[3180],{"type":58,"value":3181},"\"",{"type":45,"tag":270,"props":3183,"children":3184},{"style":3133},[3185],{"type":58,"value":3186}," \\\n",{"type":45,"tag":270,"props":3188,"children":3189},{"class":2805,"line":2946},[3190,3195,3199,3204,3208],{"type":45,"tag":270,"props":3191,"children":3192},{"style":3098},[3193],{"type":58,"value":3194},"  --url",{"type":45,"tag":270,"props":3196,"children":3197},{"style":3092},[3198],{"type":58,"value":3171},{"type":45,"tag":270,"props":3200,"children":3201},{"style":3098},[3202],{"type":58,"value":3203},"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces",{"type":45,"tag":270,"props":3205,"children":3206},{"style":3092},[3207],{"type":58,"value":3181},{"type":45,"tag":270,"props":3209,"children":3210},{"style":3133},[3211],{"type":58,"value":3186},{"type":45,"tag":270,"props":3213,"children":3214},{"class":2805,"line":2955},[3215,3220,3225,3230,3234,3239,3243,3248,3253],{"type":45,"tag":270,"props":3216,"children":3217},{"style":3098},[3218],{"type":58,"value":3219},"  --body",{"type":45,"tag":270,"props":3221,"children":3222},{"style":3098},[3223],{"type":58,"value":3224}," @\u002Ftmp\u002Fbody.json",{"type":45,"tag":270,"props":3226,"children":3227},{"style":3098},[3228],{"type":58,"value":3229}," --query",{"type":45,"tag":270,"props":3231,"children":3232},{"style":3092},[3233],{"type":58,"value":3171},{"type":45,"tag":270,"props":3235,"children":3236},{"style":3098},[3237],{"type":58,"value":3238},"id",{"type":45,"tag":270,"props":3240,"children":3241},{"style":3092},[3242],{"type":58,"value":3181},{"type":45,"tag":270,"props":3244,"children":3245},{"style":3098},[3246],{"type":58,"value":3247}," --output",{"type":45,"tag":270,"props":3249,"children":3250},{"style":3098},[3251],{"type":58,"value":3252}," tsv",{"type":45,"tag":270,"props":3254,"children":3255},{"style":3092},[3256],{"type":58,"value":2887},{"type":45,"tag":270,"props":3258,"children":3259},{"class":2805,"line":2964},[3260],{"type":45,"tag":270,"props":3261,"children":3262},{"emptyLinePlaceholder":2924},[3263],{"type":58,"value":2927},{"type":45,"tag":270,"props":3265,"children":3266},{"class":2805,"line":2973},[3267,3271,3275,3279,3283],{"type":45,"tag":270,"props":3268,"children":3269},{"style":3086},[3270],{"type":58,"value":3089},{"type":45,"tag":270,"props":3272,"children":3273},{"style":3092},[3274],{"type":58,"value":3095},{"type":45,"tag":270,"props":3276,"children":3277},{"style":3098},[3278],{"type":58,"value":3101},{"type":45,"tag":270,"props":3280,"children":3281},{"style":3092},[3282],{"type":58,"value":3106},{"type":45,"tag":270,"props":3284,"children":3285},{"style":3092},[3286],{"type":58,"value":3111},{"type":45,"tag":270,"props":3288,"children":3289},{"class":2805,"line":2982},[3290],{"type":45,"tag":270,"props":3291,"children":3292},{"style":3098},[3293],{"type":58,"value":3294},"{\"displayName\": \"DevLakehouse\", \"type\": \"Lakehouse\", \"creationPayload\": {\"enableSchemas\": true}}\n",{"type":45,"tag":270,"props":3296,"children":3298},{"class":2805,"line":3297},11,[3299],{"type":45,"tag":270,"props":3300,"children":3301},{"style":3092},[3302],{"type":58,"value":3127},{"type":45,"tag":270,"props":3304,"children":3306},{"class":2805,"line":3305},12,[3307,3312,3316,3320,3324,3328,3332,3336,3340,3344,3348],{"type":45,"tag":270,"props":3308,"children":3309},{"style":3133},[3310],{"type":58,"value":3311},"lakehouse_id",{"type":45,"tag":270,"props":3313,"children":3314},{"style":3092},[3315],{"type":58,"value":3141},{"type":45,"tag":270,"props":3317,"children":3318},{"style":3086},[3319],{"type":58,"value":3146},{"type":45,"tag":270,"props":3321,"children":3322},{"style":3098},[3323],{"type":58,"value":3151},{"type":45,"tag":270,"props":3325,"children":3326},{"style":3098},[3327],{"type":58,"value":3156},{"type":45,"tag":270,"props":3329,"children":3330},{"style":3098},[3331],{"type":58,"value":3161},{"type":45,"tag":270,"props":3333,"children":3334},{"style":3098},[3335],{"type":58,"value":3166},{"type":45,"tag":270,"props":3337,"children":3338},{"style":3092},[3339],{"type":58,"value":3171},{"type":45,"tag":270,"props":3341,"children":3342},{"style":3098},[3343],{"type":58,"value":3176},{"type":45,"tag":270,"props":3345,"children":3346},{"style":3092},[3347],{"type":58,"value":3181},{"type":45,"tag":270,"props":3349,"children":3350},{"style":3133},[3351],{"type":58,"value":3186},{"type":45,"tag":270,"props":3353,"children":3355},{"class":2805,"line":3354},13,[3356,3360,3364,3369,3374,3379,3383],{"type":45,"tag":270,"props":3357,"children":3358},{"style":3098},[3359],{"type":58,"value":3194},{"type":45,"tag":270,"props":3361,"children":3362},{"style":3092},[3363],{"type":58,"value":3171},{"type":45,"tag":270,"props":3365,"children":3366},{"style":3098},[3367],{"type":58,"value":3368},"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F",{"type":45,"tag":270,"props":3370,"children":3371},{"style":3133},[3372],{"type":58,"value":3373},"$workspace_id",{"type":45,"tag":270,"props":3375,"children":3376},{"style":3098},[3377],{"type":58,"value":3378},"\u002Fitems",{"type":45,"tag":270,"props":3380,"children":3381},{"style":3092},[3382],{"type":58,"value":3181},{"type":45,"tag":270,"props":3384,"children":3385},{"style":3133},[3386],{"type":58,"value":3186},{"type":45,"tag":270,"props":3388,"children":3390},{"class":2805,"line":3389},14,[3391,3395,3399,3403,3407,3411,3415,3419,3423],{"type":45,"tag":270,"props":3392,"children":3393},{"style":3098},[3394],{"type":58,"value":3219},{"type":45,"tag":270,"props":3396,"children":3397},{"style":3098},[3398],{"type":58,"value":3224},{"type":45,"tag":270,"props":3400,"children":3401},{"style":3098},[3402],{"type":58,"value":3229},{"type":45,"tag":270,"props":3404,"children":3405},{"style":3092},[3406],{"type":58,"value":3171},{"type":45,"tag":270,"props":3408,"children":3409},{"style":3098},[3410],{"type":58,"value":3238},{"type":45,"tag":270,"props":3412,"children":3413},{"style":3092},[3414],{"type":58,"value":3181},{"type":45,"tag":270,"props":3416,"children":3417},{"style":3098},[3418],{"type":58,"value":3247},{"type":45,"tag":270,"props":3420,"children":3421},{"style":3098},[3422],{"type":58,"value":3252},{"type":45,"tag":270,"props":3424,"children":3425},{"style":3092},[3426],{"type":58,"value":2887},{"type":45,"tag":2106,"props":3428,"children":3430},{"id":3429},"organize-lakehouse-tables-with-schemas",[3431],{"type":58,"value":3432},"Organize Lakehouse Tables with Schemas",{"type":45,"tag":2793,"props":3434,"children":3436},{"className":2795,"code":3435,"language":2797,"meta":2798,"style":2798},"# See SPARK-AUTHORING-CORE.md Lakehouse Schema Organization for table organization patterns\n#\n# IMPORTANT: `CREATE SCHEMA` is only supported on schema-enabled lakehouses. On a\n# lakehouse created WITHOUT `enableSchemas`, it fails with\n# \"Feature not supported on Apache Spark in Microsoft Fabric\". Always detect schema\n# support first, then branch: create schemas only when enabled, otherwise skip.\n# Schema names are configurable. If your workspace standard is `raw`\u002F`curated`\n# instead of `bronze`\u002F`silver`\u002F`gold`, substitute those names consistently.\nimport uuid\n\ndef lakehouse_is_schema_enabled():\n    # Probe schema support by attempting CREATE SCHEMA and catching the\n    # platform-blocked error. Use a unique probe schema name to avoid\n    # accidentally dropping a real pre-existing schema.\n    probe = f\"_schema_probe_{uuid.uuid4().hex}\"\n    try:\n        spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {probe}\")\n        spark.sql(f\"DROP SCHEMA IF EXISTS {probe}\")\n        return True\n    except Exception as e:\n        if \"Feature not supported on Apache Spark in Microsoft Fabric\" in str(e):\n            return False\n        raise\n\nif lakehouse_is_schema_enabled():\n    # Schema-enabled: create medallion schemas and write tables as `schema.table`.\n    spark.sql(\"CREATE SCHEMA IF NOT EXISTS bronze\")\n    spark.sql(\"CREATE SCHEMA IF NOT EXISTS silver\")\n    spark.sql(\"CREATE SCHEMA IF NOT EXISTS gold\")\nelse:\n    # Not schema-enabled: skip CREATE SCHEMA (unsupported). Organize tables by name\n    # prefix instead (e.g. `bronze_orders` \u002F `raw_orders`, `silver_orders` \u002F\n    # `curated_orders`), or recreate the lakehouse with\n    # `creationPayload.enableSchemas: true` if schemas are required.\n    print(\"Lakehouse is not schema-enabled; skipping schema creation and using name prefixes.\")\n",[3437],{"type":45,"tag":85,"props":3438,"children":3439},{"__ignoreMap":2798},[3440,3448,3456,3464,3472,3480,3488,3496,3504,3512,3519,3527,3535,3543,3551,3560,3569,3578,3587,3596,3605,3614,3623,3632,3640,3649,3658,3667,3676,3685,3694,3703,3712,3721,3730],{"type":45,"tag":270,"props":3441,"children":3442},{"class":2805,"line":2806},[3443],{"type":45,"tag":270,"props":3444,"children":3445},{},[3446],{"type":58,"value":3447},"# See SPARK-AUTHORING-CORE.md Lakehouse Schema Organization for table organization patterns\n",{"type":45,"tag":270,"props":3449,"children":3450},{"class":2805,"line":2815},[3451],{"type":45,"tag":270,"props":3452,"children":3453},{},[3454],{"type":58,"value":3455},"#\n",{"type":45,"tag":270,"props":3457,"children":3458},{"class":2805,"line":2824},[3459],{"type":45,"tag":270,"props":3460,"children":3461},{},[3462],{"type":58,"value":3463},"# IMPORTANT: `CREATE SCHEMA` is only supported on schema-enabled lakehouses. On a\n",{"type":45,"tag":270,"props":3465,"children":3466},{"class":2805,"line":2833},[3467],{"type":45,"tag":270,"props":3468,"children":3469},{},[3470],{"type":58,"value":3471},"# lakehouse created WITHOUT `enableSchemas`, it fails with\n",{"type":45,"tag":270,"props":3473,"children":3474},{"class":2805,"line":2842},[3475],{"type":45,"tag":270,"props":3476,"children":3477},{},[3478],{"type":58,"value":3479},"# \"Feature not supported on Apache Spark in Microsoft Fabric\". Always detect schema\n",{"type":45,"tag":270,"props":3481,"children":3482},{"class":2805,"line":2946},[3483],{"type":45,"tag":270,"props":3484,"children":3485},{},[3486],{"type":58,"value":3487},"# support first, then branch: create schemas only when enabled, otherwise skip.\n",{"type":45,"tag":270,"props":3489,"children":3490},{"class":2805,"line":2955},[3491],{"type":45,"tag":270,"props":3492,"children":3493},{},[3494],{"type":58,"value":3495},"# Schema names are configurable. If your workspace standard is `raw`\u002F`curated`\n",{"type":45,"tag":270,"props":3497,"children":3498},{"class":2805,"line":2964},[3499],{"type":45,"tag":270,"props":3500,"children":3501},{},[3502],{"type":58,"value":3503},"# instead of `bronze`\u002F`silver`\u002F`gold`, substitute those names consistently.\n",{"type":45,"tag":270,"props":3505,"children":3506},{"class":2805,"line":2973},[3507],{"type":45,"tag":270,"props":3508,"children":3509},{},[3510],{"type":58,"value":3511},"import uuid\n",{"type":45,"tag":270,"props":3513,"children":3514},{"class":2805,"line":2982},[3515],{"type":45,"tag":270,"props":3516,"children":3517},{"emptyLinePlaceholder":2924},[3518],{"type":58,"value":2927},{"type":45,"tag":270,"props":3520,"children":3521},{"class":2805,"line":3297},[3522],{"type":45,"tag":270,"props":3523,"children":3524},{},[3525],{"type":58,"value":3526},"def lakehouse_is_schema_enabled():\n",{"type":45,"tag":270,"props":3528,"children":3529},{"class":2805,"line":3305},[3530],{"type":45,"tag":270,"props":3531,"children":3532},{},[3533],{"type":58,"value":3534},"    # Probe schema support by attempting CREATE SCHEMA and catching the\n",{"type":45,"tag":270,"props":3536,"children":3537},{"class":2805,"line":3354},[3538],{"type":45,"tag":270,"props":3539,"children":3540},{},[3541],{"type":58,"value":3542},"    # platform-blocked error. Use a unique probe schema name to avoid\n",{"type":45,"tag":270,"props":3544,"children":3545},{"class":2805,"line":3389},[3546],{"type":45,"tag":270,"props":3547,"children":3548},{},[3549],{"type":58,"value":3550},"    # accidentally dropping a real pre-existing schema.\n",{"type":45,"tag":270,"props":3552,"children":3554},{"class":2805,"line":3553},15,[3555],{"type":45,"tag":270,"props":3556,"children":3557},{},[3558],{"type":58,"value":3559},"    probe = f\"_schema_probe_{uuid.uuid4().hex}\"\n",{"type":45,"tag":270,"props":3561,"children":3563},{"class":2805,"line":3562},16,[3564],{"type":45,"tag":270,"props":3565,"children":3566},{},[3567],{"type":58,"value":3568},"    try:\n",{"type":45,"tag":270,"props":3570,"children":3572},{"class":2805,"line":3571},17,[3573],{"type":45,"tag":270,"props":3574,"children":3575},{},[3576],{"type":58,"value":3577},"        spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {probe}\")\n",{"type":45,"tag":270,"props":3579,"children":3581},{"class":2805,"line":3580},18,[3582],{"type":45,"tag":270,"props":3583,"children":3584},{},[3585],{"type":58,"value":3586},"        spark.sql(f\"DROP SCHEMA IF EXISTS {probe}\")\n",{"type":45,"tag":270,"props":3588,"children":3590},{"class":2805,"line":3589},19,[3591],{"type":45,"tag":270,"props":3592,"children":3593},{},[3594],{"type":58,"value":3595},"        return True\n",{"type":45,"tag":270,"props":3597,"children":3599},{"class":2805,"line":3598},20,[3600],{"type":45,"tag":270,"props":3601,"children":3602},{},[3603],{"type":58,"value":3604},"    except Exception as e:\n",{"type":45,"tag":270,"props":3606,"children":3608},{"class":2805,"line":3607},21,[3609],{"type":45,"tag":270,"props":3610,"children":3611},{},[3612],{"type":58,"value":3613},"        if \"Feature not supported on Apache Spark in Microsoft Fabric\" in str(e):\n",{"type":45,"tag":270,"props":3615,"children":3617},{"class":2805,"line":3616},22,[3618],{"type":45,"tag":270,"props":3619,"children":3620},{},[3621],{"type":58,"value":3622},"            return False\n",{"type":45,"tag":270,"props":3624,"children":3626},{"class":2805,"line":3625},23,[3627],{"type":45,"tag":270,"props":3628,"children":3629},{},[3630],{"type":58,"value":3631},"        raise\n",{"type":45,"tag":270,"props":3633,"children":3635},{"class":2805,"line":3634},24,[3636],{"type":45,"tag":270,"props":3637,"children":3638},{"emptyLinePlaceholder":2924},[3639],{"type":58,"value":2927},{"type":45,"tag":270,"props":3641,"children":3643},{"class":2805,"line":3642},25,[3644],{"type":45,"tag":270,"props":3645,"children":3646},{},[3647],{"type":58,"value":3648},"if lakehouse_is_schema_enabled():\n",{"type":45,"tag":270,"props":3650,"children":3652},{"class":2805,"line":3651},26,[3653],{"type":45,"tag":270,"props":3654,"children":3655},{},[3656],{"type":58,"value":3657},"    # Schema-enabled: create medallion schemas and write tables as `schema.table`.\n",{"type":45,"tag":270,"props":3659,"children":3661},{"class":2805,"line":3660},27,[3662],{"type":45,"tag":270,"props":3663,"children":3664},{},[3665],{"type":58,"value":3666},"    spark.sql(\"CREATE SCHEMA IF NOT EXISTS bronze\")\n",{"type":45,"tag":270,"props":3668,"children":3670},{"class":2805,"line":3669},28,[3671],{"type":45,"tag":270,"props":3672,"children":3673},{},[3674],{"type":58,"value":3675},"    spark.sql(\"CREATE SCHEMA IF NOT EXISTS silver\")\n",{"type":45,"tag":270,"props":3677,"children":3679},{"class":2805,"line":3678},29,[3680],{"type":45,"tag":270,"props":3681,"children":3682},{},[3683],{"type":58,"value":3684},"    spark.sql(\"CREATE SCHEMA IF NOT EXISTS gold\")\n",{"type":45,"tag":270,"props":3686,"children":3688},{"class":2805,"line":3687},30,[3689],{"type":45,"tag":270,"props":3690,"children":3691},{},[3692],{"type":58,"value":3693},"else:\n",{"type":45,"tag":270,"props":3695,"children":3697},{"class":2805,"line":3696},31,[3698],{"type":45,"tag":270,"props":3699,"children":3700},{},[3701],{"type":58,"value":3702},"    # Not schema-enabled: skip CREATE SCHEMA (unsupported). Organize tables by name\n",{"type":45,"tag":270,"props":3704,"children":3706},{"class":2805,"line":3705},32,[3707],{"type":45,"tag":270,"props":3708,"children":3709},{},[3710],{"type":58,"value":3711},"    # prefix instead (e.g. `bronze_orders` \u002F `raw_orders`, `silver_orders` \u002F\n",{"type":45,"tag":270,"props":3713,"children":3715},{"class":2805,"line":3714},33,[3716],{"type":45,"tag":270,"props":3717,"children":3718},{},[3719],{"type":58,"value":3720},"    # `curated_orders`), or recreate the lakehouse with\n",{"type":45,"tag":270,"props":3722,"children":3724},{"class":2805,"line":3723},34,[3725],{"type":45,"tag":270,"props":3726,"children":3727},{},[3728],{"type":58,"value":3729},"    # `creationPayload.enableSchemas: true` if schemas are required.\n",{"type":45,"tag":270,"props":3731,"children":3733},{"class":2805,"line":3732},35,[3734],{"type":45,"tag":270,"props":3735,"children":3736},{},[3737],{"type":58,"value":3738},"    print(\"Lakehouse is not schema-enabled; skipping schema creation and using name prefixes.\")\n",{"type":45,"tag":2106,"props":3740,"children":3742},{"id":3741},"create-and-refresh-a-materialized-lake-view-mlv",[3743],{"type":58,"value":3744},"Create and Refresh a Materialized Lake View (MLV)",{"type":45,"tag":2793,"props":3746,"children":3750},{"className":3747,"code":3748,"language":3749,"meta":2798,"style":2798},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","-- See resources\u002Fmaterialized-lake-view-patterns.md for design guidance\n-- and resources\u002Fmlv-incremental-refresh-patterns.md for refresh-readiness review.\n\n-- Bronze\u002FSilver\u002FGold schemas in a Lakehouse with schemas enabled\nCREATE SCHEMA IF NOT EXISTS bronze;\nCREATE SCHEMA IF NOT EXISTS silver;\nCREATE SCHEMA IF NOT EXISTS gold;\n\n-- A simple Silver MLV with data quality constraints\n--\n-- Prerequisite for incremental refresh: enable Change Data Feed (CDF) on every\n-- source table the MLV reads from. Without CDF, optimal refresh can only choose\n-- between no refresh (sources unchanged) and full refresh — never incremental.\n-- See resources\u002Fmlv-incremental-refresh-patterns.md.\nALTER TABLE bronze.orders_raw SET TBLPROPERTIES (delta.enableChangeDataFeed = true);\n\nCREATE OR REPLACE MATERIALIZED LAKE VIEW silver.orders_clean\n(\n    CONSTRAINT valid_order_id CHECK (order_id IS NOT NULL) ON MISMATCH DROP\n)\nAS\nSELECT\n  order_id,\n  customer_id,\n  CAST(order_ts AS TIMESTAMP) AS order_ts,\n  amount\nFROM bronze.orders_raw;\n\n-- Routine refresh is handled by the lakehouse Materialized lake views → Manage\n-- schedule\u002Flineage view; don't orchestrate from notebooks. The SQL form below is\n-- documented only for forcing a one-time FULL recompute (troubleshooting \u002F after\n-- a correction). There is no documented SQL form for triggering incremental refresh.\nREFRESH MATERIALIZED LAKE VIEW silver.orders_clean FULL;\n","sql",[3751],{"type":45,"tag":85,"props":3752,"children":3753},{"__ignoreMap":2798},[3754,3762,3770,3777,3785,3793,3801,3809,3816,3824,3832,3840,3848,3856,3864,3872,3879,3887,3895,3903,3910,3918,3926,3934,3942,3950,3958,3966,3973,3981,3989,3997,4005],{"type":45,"tag":270,"props":3755,"children":3756},{"class":2805,"line":2806},[3757],{"type":45,"tag":270,"props":3758,"children":3759},{},[3760],{"type":58,"value":3761},"-- See resources\u002Fmaterialized-lake-view-patterns.md for design guidance\n",{"type":45,"tag":270,"props":3763,"children":3764},{"class":2805,"line":2815},[3765],{"type":45,"tag":270,"props":3766,"children":3767},{},[3768],{"type":58,"value":3769},"-- and resources\u002Fmlv-incremental-refresh-patterns.md for refresh-readiness review.\n",{"type":45,"tag":270,"props":3771,"children":3772},{"class":2805,"line":2824},[3773],{"type":45,"tag":270,"props":3774,"children":3775},{"emptyLinePlaceholder":2924},[3776],{"type":58,"value":2927},{"type":45,"tag":270,"props":3778,"children":3779},{"class":2805,"line":2833},[3780],{"type":45,"tag":270,"props":3781,"children":3782},{},[3783],{"type":58,"value":3784},"-- Bronze\u002FSilver\u002FGold schemas in a Lakehouse with schemas enabled\n",{"type":45,"tag":270,"props":3786,"children":3787},{"class":2805,"line":2842},[3788],{"type":45,"tag":270,"props":3789,"children":3790},{},[3791],{"type":58,"value":3792},"CREATE SCHEMA IF NOT EXISTS bronze;\n",{"type":45,"tag":270,"props":3794,"children":3795},{"class":2805,"line":2946},[3796],{"type":45,"tag":270,"props":3797,"children":3798},{},[3799],{"type":58,"value":3800},"CREATE SCHEMA IF NOT EXISTS silver;\n",{"type":45,"tag":270,"props":3802,"children":3803},{"class":2805,"line":2955},[3804],{"type":45,"tag":270,"props":3805,"children":3806},{},[3807],{"type":58,"value":3808},"CREATE SCHEMA IF NOT EXISTS gold;\n",{"type":45,"tag":270,"props":3810,"children":3811},{"class":2805,"line":2964},[3812],{"type":45,"tag":270,"props":3813,"children":3814},{"emptyLinePlaceholder":2924},[3815],{"type":58,"value":2927},{"type":45,"tag":270,"props":3817,"children":3818},{"class":2805,"line":2973},[3819],{"type":45,"tag":270,"props":3820,"children":3821},{},[3822],{"type":58,"value":3823},"-- A simple Silver MLV with data quality constraints\n",{"type":45,"tag":270,"props":3825,"children":3826},{"class":2805,"line":2982},[3827],{"type":45,"tag":270,"props":3828,"children":3829},{},[3830],{"type":58,"value":3831},"--\n",{"type":45,"tag":270,"props":3833,"children":3834},{"class":2805,"line":3297},[3835],{"type":45,"tag":270,"props":3836,"children":3837},{},[3838],{"type":58,"value":3839},"-- Prerequisite for incremental refresh: enable Change Data Feed (CDF) on every\n",{"type":45,"tag":270,"props":3841,"children":3842},{"class":2805,"line":3305},[3843],{"type":45,"tag":270,"props":3844,"children":3845},{},[3846],{"type":58,"value":3847},"-- source table the MLV reads from. Without CDF, optimal refresh can only choose\n",{"type":45,"tag":270,"props":3849,"children":3850},{"class":2805,"line":3354},[3851],{"type":45,"tag":270,"props":3852,"children":3853},{},[3854],{"type":58,"value":3855},"-- between no refresh (sources unchanged) and full refresh — never incremental.\n",{"type":45,"tag":270,"props":3857,"children":3858},{"class":2805,"line":3389},[3859],{"type":45,"tag":270,"props":3860,"children":3861},{},[3862],{"type":58,"value":3863},"-- See resources\u002Fmlv-incremental-refresh-patterns.md.\n",{"type":45,"tag":270,"props":3865,"children":3866},{"class":2805,"line":3553},[3867],{"type":45,"tag":270,"props":3868,"children":3869},{},[3870],{"type":58,"value":3871},"ALTER TABLE bronze.orders_raw SET TBLPROPERTIES (delta.enableChangeDataFeed = true);\n",{"type":45,"tag":270,"props":3873,"children":3874},{"class":2805,"line":3562},[3875],{"type":45,"tag":270,"props":3876,"children":3877},{"emptyLinePlaceholder":2924},[3878],{"type":58,"value":2927},{"type":45,"tag":270,"props":3880,"children":3881},{"class":2805,"line":3571},[3882],{"type":45,"tag":270,"props":3883,"children":3884},{},[3885],{"type":58,"value":3886},"CREATE OR REPLACE MATERIALIZED LAKE VIEW silver.orders_clean\n",{"type":45,"tag":270,"props":3888,"children":3889},{"class":2805,"line":3580},[3890],{"type":45,"tag":270,"props":3891,"children":3892},{},[3893],{"type":58,"value":3894},"(\n",{"type":45,"tag":270,"props":3896,"children":3897},{"class":2805,"line":3589},[3898],{"type":45,"tag":270,"props":3899,"children":3900},{},[3901],{"type":58,"value":3902},"    CONSTRAINT valid_order_id CHECK (order_id IS NOT NULL) ON MISMATCH DROP\n",{"type":45,"tag":270,"props":3904,"children":3905},{"class":2805,"line":3598},[3906],{"type":45,"tag":270,"props":3907,"children":3908},{},[3909],{"type":58,"value":2887},{"type":45,"tag":270,"props":3911,"children":3912},{"class":2805,"line":3607},[3913],{"type":45,"tag":270,"props":3914,"children":3915},{},[3916],{"type":58,"value":3917},"AS\n",{"type":45,"tag":270,"props":3919,"children":3920},{"class":2805,"line":3616},[3921],{"type":45,"tag":270,"props":3922,"children":3923},{},[3924],{"type":58,"value":3925},"SELECT\n",{"type":45,"tag":270,"props":3927,"children":3928},{"class":2805,"line":3625},[3929],{"type":45,"tag":270,"props":3930,"children":3931},{},[3932],{"type":58,"value":3933},"  order_id,\n",{"type":45,"tag":270,"props":3935,"children":3936},{"class":2805,"line":3634},[3937],{"type":45,"tag":270,"props":3938,"children":3939},{},[3940],{"type":58,"value":3941},"  customer_id,\n",{"type":45,"tag":270,"props":3943,"children":3944},{"class":2805,"line":3642},[3945],{"type":45,"tag":270,"props":3946,"children":3947},{},[3948],{"type":58,"value":3949},"  CAST(order_ts AS TIMESTAMP) AS order_ts,\n",{"type":45,"tag":270,"props":3951,"children":3952},{"class":2805,"line":3651},[3953],{"type":45,"tag":270,"props":3954,"children":3955},{},[3956],{"type":58,"value":3957},"  amount\n",{"type":45,"tag":270,"props":3959,"children":3960},{"class":2805,"line":3660},[3961],{"type":45,"tag":270,"props":3962,"children":3963},{},[3964],{"type":58,"value":3965},"FROM bronze.orders_raw;\n",{"type":45,"tag":270,"props":3967,"children":3968},{"class":2805,"line":3669},[3969],{"type":45,"tag":270,"props":3970,"children":3971},{"emptyLinePlaceholder":2924},[3972],{"type":58,"value":2927},{"type":45,"tag":270,"props":3974,"children":3975},{"class":2805,"line":3678},[3976],{"type":45,"tag":270,"props":3977,"children":3978},{},[3979],{"type":58,"value":3980},"-- Routine refresh is handled by the lakehouse Materialized lake views → Manage\n",{"type":45,"tag":270,"props":3982,"children":3983},{"class":2805,"line":3687},[3984],{"type":45,"tag":270,"props":3985,"children":3986},{},[3987],{"type":58,"value":3988},"-- schedule\u002Flineage view; don't orchestrate from notebooks. The SQL form below is\n",{"type":45,"tag":270,"props":3990,"children":3991},{"class":2805,"line":3696},[3992],{"type":45,"tag":270,"props":3993,"children":3994},{},[3995],{"type":58,"value":3996},"-- documented only for forcing a one-time FULL recompute (troubleshooting \u002F after\n",{"type":45,"tag":270,"props":3998,"children":3999},{"class":2805,"line":3705},[4000],{"type":45,"tag":270,"props":4001,"children":4002},{},[4003],{"type":58,"value":4004},"-- a correction). There is no documented SQL form for triggering incremental refresh.\n",{"type":45,"tag":270,"props":4006,"children":4007},{"class":2805,"line":3714},[4008],{"type":45,"tag":270,"props":4009,"children":4010},{},[4011],{"type":58,"value":4012},"REFRESH MATERIALIZED LAKE VIEW silver.orders_clean FULL;\n",{"type":45,"tag":2106,"props":4014,"children":4016},{"id":4015},"create-lakehouse-livy-session",[4017],{"type":58,"value":4018},"Create Lakehouse Livy Session",{"type":45,"tag":2793,"props":4020,"children":4022},{"className":3065,"code":4021,"language":3067,"meta":2798,"style":2798},"# See SPARK-CONSUMPTION-CORE.md for Lakehouse Livy session configuration and management\n# IMPORTANT: Body MUST be flat JSON with memory\u002Fcores — do NOT wrap in {\"payload\": ...}\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"name\": \"dev-session\", \"driverMemory\": \"56g\", \"driverCores\": 8, \"executorMemory\": \"56g\", \"executorCores\": 8, \"conf\": {\"spark.dynamicAllocation.enabled\": \"true\", \"spark.fabric.pool.name\": \"Starter Pool\"}}\nEOF\naz rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F$workspace_id\u002Flakehouses\u002F$lakehouse_id\u002Flivyapi\u002Fversions\u002F2023-12-01\u002Fsessions\" \\\n  --body @\u002Ftmp\u002Fbody.json\n",[4023],{"type":45,"tag":85,"props":4024,"children":4025},{"__ignoreMap":2798},[4026,4034,4042,4065,4073,4080,4119,4161],{"type":45,"tag":270,"props":4027,"children":4028},{"class":2805,"line":2806},[4029],{"type":45,"tag":270,"props":4030,"children":4031},{"style":3077},[4032],{"type":58,"value":4033},"# See SPARK-CONSUMPTION-CORE.md for Lakehouse Livy session configuration and management\n",{"type":45,"tag":270,"props":4035,"children":4036},{"class":2805,"line":2815},[4037],{"type":45,"tag":270,"props":4038,"children":4039},{"style":3077},[4040],{"type":58,"value":4041},"# IMPORTANT: Body MUST be flat JSON with memory\u002Fcores — do NOT wrap in {\"payload\": ...}\n",{"type":45,"tag":270,"props":4043,"children":4044},{"class":2805,"line":2824},[4045,4049,4053,4057,4061],{"type":45,"tag":270,"props":4046,"children":4047},{"style":3086},[4048],{"type":58,"value":3089},{"type":45,"tag":270,"props":4050,"children":4051},{"style":3092},[4052],{"type":58,"value":3095},{"type":45,"tag":270,"props":4054,"children":4055},{"style":3098},[4056],{"type":58,"value":3101},{"type":45,"tag":270,"props":4058,"children":4059},{"style":3092},[4060],{"type":58,"value":3106},{"type":45,"tag":270,"props":4062,"children":4063},{"style":3092},[4064],{"type":58,"value":3111},{"type":45,"tag":270,"props":4066,"children":4067},{"class":2805,"line":2833},[4068],{"type":45,"tag":270,"props":4069,"children":4070},{"style":3098},[4071],{"type":58,"value":4072},"{\"name\": \"dev-session\", \"driverMemory\": \"56g\", \"driverCores\": 8, \"executorMemory\": \"56g\", \"executorCores\": 8, \"conf\": {\"spark.dynamicAllocation.enabled\": \"true\", \"spark.fabric.pool.name\": \"Starter Pool\"}}\n",{"type":45,"tag":270,"props":4074,"children":4075},{"class":2805,"line":2842},[4076],{"type":45,"tag":270,"props":4077,"children":4078},{"style":3092},[4079],{"type":58,"value":3127},{"type":45,"tag":270,"props":4081,"children":4082},{"class":2805,"line":2946},[4083,4087,4091,4095,4099,4103,4107,4111,4115],{"type":45,"tag":270,"props":4084,"children":4085},{"style":3086},[4086],{"type":58,"value":3146},{"type":45,"tag":270,"props":4088,"children":4089},{"style":3098},[4090],{"type":58,"value":3151},{"type":45,"tag":270,"props":4092,"children":4093},{"style":3098},[4094],{"type":58,"value":3156},{"type":45,"tag":270,"props":4096,"children":4097},{"style":3098},[4098],{"type":58,"value":3161},{"type":45,"tag":270,"props":4100,"children":4101},{"style":3098},[4102],{"type":58,"value":3166},{"type":45,"tag":270,"props":4104,"children":4105},{"style":3092},[4106],{"type":58,"value":3171},{"type":45,"tag":270,"props":4108,"children":4109},{"style":3098},[4110],{"type":58,"value":3176},{"type":45,"tag":270,"props":4112,"children":4113},{"style":3092},[4114],{"type":58,"value":3181},{"type":45,"tag":270,"props":4116,"children":4117},{"style":3133},[4118],{"type":58,"value":3186},{"type":45,"tag":270,"props":4120,"children":4121},{"class":2805,"line":2955},[4122,4126,4130,4134,4138,4143,4148,4153,4157],{"type":45,"tag":270,"props":4123,"children":4124},{"style":3098},[4125],{"type":58,"value":3194},{"type":45,"tag":270,"props":4127,"children":4128},{"style":3092},[4129],{"type":58,"value":3171},{"type":45,"tag":270,"props":4131,"children":4132},{"style":3098},[4133],{"type":58,"value":3368},{"type":45,"tag":270,"props":4135,"children":4136},{"style":3133},[4137],{"type":58,"value":3373},{"type":45,"tag":270,"props":4139,"children":4140},{"style":3098},[4141],{"type":58,"value":4142},"\u002Flakehouses\u002F",{"type":45,"tag":270,"props":4144,"children":4145},{"style":3133},[4146],{"type":58,"value":4147},"$lakehouse_id",{"type":45,"tag":270,"props":4149,"children":4150},{"style":3098},[4151],{"type":58,"value":4152},"\u002Flivyapi\u002Fversions\u002F2023-12-01\u002Fsessions",{"type":45,"tag":270,"props":4154,"children":4155},{"style":3092},[4156],{"type":58,"value":3181},{"type":45,"tag":270,"props":4158,"children":4159},{"style":3133},[4160],{"type":58,"value":3186},{"type":45,"tag":270,"props":4162,"children":4163},{"class":2805,"line":2964},[4164,4168],{"type":45,"tag":270,"props":4165,"children":4166},{"style":3098},[4167],{"type":58,"value":3219},{"type":45,"tag":270,"props":4169,"children":4170},{"style":3098},[4171],{"type":58,"value":4172}," @\u002Ftmp\u002Fbody.json\n",{"type":45,"tag":46,"props":4174,"children":4175},{},[4176,4184],{"type":45,"tag":50,"props":4177,"children":4178},{},[4179],{"type":45,"tag":54,"props":4180,"children":4181},{},[4182],{"type":58,"value":4183},"Lakehouse Livy Session Body — Common Mistakes",{"type":45,"tag":70,"props":4185,"children":4186},{},[4187,4200,4224],{"type":45,"tag":74,"props":4188,"children":4189},{},[4190,4192,4198],{"type":58,"value":4191},"❌ ",{"type":45,"tag":85,"props":4193,"children":4195},{"className":4194},[],[4196],{"type":58,"value":4197},"{\"payload\": {\"kind\": \"pyspark\"}}",{"type":58,"value":4199}," → HTTP 500 (wrong wrapper, missing required fields)",{"type":45,"tag":74,"props":4201,"children":4202},{},[4203,4204,4209,4211,4216,4217,4222],{"type":58,"value":4191},{"type":45,"tag":85,"props":4205,"children":4207},{"className":4206},[],[4208],{"type":58,"value":2231},{"type":58,"value":4210}," → HTTP 500 (missing ",{"type":45,"tag":85,"props":4212,"children":4214},{"className":4213},[],[4215],{"type":58,"value":2194},{"type":58,"value":498},{"type":45,"tag":85,"props":4218,"children":4220},{"className":4219},[],[4221],{"type":58,"value":2208},{"type":58,"value":4223},", etc.)",{"type":45,"tag":74,"props":4225,"children":4226},{},[4227,4229,4234,4235,4240,4241,4246,4247,4252,4253,4258,4260,4266],{"type":58,"value":4228},"✅ Flat JSON with ",{"type":45,"tag":85,"props":4230,"children":4232},{"className":4231},[],[4233],{"type":58,"value":2187},{"type":58,"value":498},{"type":45,"tag":85,"props":4236,"children":4238},{"className":4237},[],[4239],{"type":58,"value":2194},{"type":58,"value":498},{"type":45,"tag":85,"props":4242,"children":4244},{"className":4243},[],[4245],{"type":58,"value":2201},{"type":58,"value":498},{"type":45,"tag":85,"props":4248,"children":4250},{"className":4249},[],[4251],{"type":58,"value":2208},{"type":58,"value":498},{"type":45,"tag":85,"props":4254,"children":4256},{"className":4255},[],[4257],{"type":58,"value":2215},{"type":58,"value":4259}," (and optionally ",{"type":45,"tag":85,"props":4261,"children":4263},{"className":4262},[],[4264],{"type":58,"value":4265},"conf",{"type":58,"value":4267}," with Starter Pool)",{"type":45,"tag":2106,"props":4269,"children":4271},{"id":4270},"spark-performance-configs",[4272],{"type":58,"value":4273},"Spark Performance Configs",{"type":45,"tag":50,"props":4275,"children":4276},{},[4277],{"type":45,"tag":54,"props":4278,"children":4279},{},[4280],{"type":58,"value":4281},"For detailed workload-specific configurations, see data-engineering-patterns.md Delta Lake Best Practices.",{"type":45,"tag":50,"props":4283,"children":4284},{},[4285],{"type":58,"value":4286},"Quick reference:",{"type":45,"tag":2793,"props":4288,"children":4290},{"className":2795,"code":4289,"language":2797,"meta":2798,"style":2798},"# Write-heavy (Bronze): Disable V-Order, enable autoCompact\n# Balanced (Silver): Enable V-Order, adaptive execution  \n# Read-heavy (Gold): Vectorized reads, optimal parallelism\n# See data-engineering-patterns.md for complete config tables\n",[4291],{"type":45,"tag":85,"props":4292,"children":4293},{"__ignoreMap":2798},[4294,4302,4310,4318],{"type":45,"tag":270,"props":4295,"children":4296},{"class":2805,"line":2806},[4297],{"type":45,"tag":270,"props":4298,"children":4299},{},[4300],{"type":58,"value":4301},"# Write-heavy (Bronze): Disable V-Order, enable autoCompact\n",{"type":45,"tag":270,"props":4303,"children":4304},{"class":2805,"line":2815},[4305],{"type":45,"tag":270,"props":4306,"children":4307},{},[4308],{"type":58,"value":4309},"# Balanced (Silver): Enable V-Order, adaptive execution  \n",{"type":45,"tag":270,"props":4311,"children":4312},{"class":2805,"line":2824},[4313],{"type":45,"tag":270,"props":4314,"children":4315},{},[4316],{"type":58,"value":4317},"# Read-heavy (Gold): Vectorized reads, optimal parallelism\n",{"type":45,"tag":270,"props":4319,"children":4320},{"class":2805,"line":2833},[4321],{"type":45,"tag":270,"props":4322,"children":4323},{},[4324],{"type":58,"value":4325},"# See data-engineering-patterns.md for complete config tables\n",{"type":45,"tag":2096,"props":4327,"children":4328},{},[],{"type":45,"tag":50,"props":4330,"children":4331},{},[4332,4337],{"type":45,"tag":54,"props":4333,"children":4334},{},[4335],{"type":58,"value":4336},"Focus",{"type":58,"value":4338},": Essential CLI patterns for Spark\u002Fdata engineering development and notebook code authoring, with intelligent routing to specialized resources. For comprehensive patterns, always reference COMMON-* files and resource documents.",{"type":45,"tag":4340,"props":4341,"children":4342},"style",{},[4343],{"type":58,"value":4344},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4346,"total":4540},[4347,4369,4390,4411,4426,4442,4453,4466,4481,4496,4515,4528],{"slug":4348,"name":4348,"fn":4349,"description":4350,"org":4351,"tags":4352,"stars":4366,"repoUrl":4367,"updatedAt":4368},"rushstack-best-practices","manage Rush monorepos with best practices","Provides best practices and guidance for working with Rush monorepos. Use when the user is working in a Rush-based repository, asks about Rush commands (install, update, build, rebuild), needs help with project selection, dependency management, build caching, subspace configuration, or troubleshooting Rush-specific issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4353,4356,4359,4360,4363],{"name":4354,"slug":4355,"type":15},"Engineering","engineering",{"name":4357,"slug":4358,"type":15},"Local Development","local-development",{"name":9,"slug":8,"type":15},{"name":4361,"slug":4362,"type":15},"Project Management","project-management",{"name":4364,"slug":4365,"type":15},"Rush","rush",6484,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Frushstack","2026-04-06T18:34:44.965032",{"slug":4370,"name":4370,"fn":4371,"description":4372,"org":4373,"tags":4374,"stars":4387,"repoUrl":4388,"updatedAt":4389},"azure-ai-agents-persistent-dotnet","build AI agents with Azure .NET SDK","Azure AI Agents Persistent SDK for .NET. Low-level SDK for creating and managing AI agents with threads, messages, runs, and tools. Use for agent CRUD, conversation threads, streaming responses, function calling, file search, and code interpreter. Triggers: \"PersistentAgentsClient\", \"persistent agents\", \"agent threads\", \"agent runs\", \"streaming agents\", \"function calling agents .NET\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4375,4378,4381,4384],{"name":4376,"slug":4377,"type":15},".NET","net",{"name":4379,"slug":4380,"type":15},"Agents","agents",{"name":4382,"slug":4383,"type":15},"Azure","azure",{"name":4385,"slug":4386,"type":15},"LLM","llm",2804,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills","2026-07-03T16:32:10.297433",{"slug":4391,"name":4391,"fn":4392,"description":4393,"org":4394,"tags":4395,"stars":4387,"repoUrl":4388,"updatedAt":4410},"azure-ai-anomalydetector-java","build anomaly detection applications with Java","Build anomaly detection applications with Azure AI Anomaly Detector SDK for Java. Use when implementing univariate\u002Fmultivariate anomaly detection, time-series analysis, or AI-powered monitoring.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4396,4399,4400,4403,4406,4407],{"name":4397,"slug":4398,"type":15},"Analytics","analytics",{"name":4382,"slug":4383,"type":15},{"name":4401,"slug":4402,"type":15},"Data Analysis","data-analysis",{"name":4404,"slug":4405,"type":15},"Java","java",{"name":9,"slug":8,"type":15},{"name":4408,"slug":4409,"type":15},"Monitoring","monitoring","2026-05-13T06:14:16.261754",{"slug":4412,"name":4412,"fn":4413,"description":4414,"org":4415,"tags":4416,"stars":4387,"repoUrl":4388,"updatedAt":4425},"azure-ai-contentsafety-java","build content moderation applications with Azure AI","Build content moderation applications with Azure AI Content Safety SDK for Java. Use when implementing text\u002Fimage analysis, blocklist management, or harm detection for hate, violence, sexual content, and self-harm.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4417,4420,4421,4422],{"name":4418,"slug":4419,"type":15},"AI Infrastructure","ai-infrastructure",{"name":4382,"slug":4383,"type":15},{"name":4404,"slug":4405,"type":15},{"name":4423,"slug":4424,"type":15},"Security","security","2026-07-07T06:53:31.293235",{"slug":4427,"name":4427,"fn":4428,"description":4429,"org":4430,"tags":4431,"stars":4387,"repoUrl":4388,"updatedAt":4441},"azure-ai-contentsafety-py","detect harmful content with Azure AI Content Safety","Azure AI Content Safety SDK for Python. Use for detecting harmful content in text and images with multi-severity classification.\nTriggers: \"azure-ai-contentsafety\", \"ContentSafetyClient\", \"content moderation\", \"harmful content\", \"text analysis\", \"image analysis\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4432,4433,4436,4437,4438,4440],{"name":4382,"slug":4383,"type":15},{"name":4434,"slug":4435,"type":15},"Compliance","compliance",{"name":4385,"slug":4386,"type":15},{"name":9,"slug":8,"type":15},{"name":4439,"slug":2797,"type":15},"Python",{"name":4423,"slug":4424,"type":15},"2026-07-18T05:14:23.017504",{"slug":4443,"name":4443,"fn":4444,"description":4445,"org":4446,"tags":4447,"stars":4387,"repoUrl":4388,"updatedAt":4452},"azure-ai-language-conversations-py","implement conversational language understanding with Python","Implement Conversational Language Understanding (CLU) using the azure-ai-language-conversations Python SDK. Use when working with ConversationAnalysisClient to analyze conversation intent and entities, building NLP features, or integrating language understanding into applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4448,4449,4450,4451],{"name":4397,"slug":4398,"type":15},{"name":4382,"slug":4383,"type":15},{"name":4385,"slug":4386,"type":15},{"name":4439,"slug":2797,"type":15},"2026-07-31T05:54:29.068751",{"slug":4454,"name":4454,"fn":4455,"description":4456,"org":4457,"tags":4458,"stars":4387,"repoUrl":4388,"updatedAt":4465},"azure-ai-translation-text-py","translate text using Azure AI services","Azure AI Text Translation SDK for real-time text translation, transliteration, language detection, and dictionary lookup. Use for translating text content in applications.\nTriggers: \"text translation\", \"translator\", \"translate text\", \"transliterate\", \"TextTranslationClient\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4459,4462,4463,4464],{"name":4460,"slug":4461,"type":15},"API Development","api-development",{"name":4382,"slug":4383,"type":15},{"name":9,"slug":8,"type":15},{"name":4439,"slug":2797,"type":15},"2026-07-18T05:14:16.988376",{"slug":4467,"name":4467,"fn":4468,"description":4469,"org":4470,"tags":4471,"stars":4387,"repoUrl":4388,"updatedAt":4480},"azure-ai-vision-imageanalysis-py","analyze images with Azure AI Vision","Azure AI Vision Image Analysis SDK for captions, tags, objects, OCR, people detection, and smart cropping. Use for computer vision and image understanding tasks.\nTriggers: \"image analysis\", \"computer vision\", \"OCR\", \"object detection\", \"ImageAnalysisClient\", \"image caption\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4472,4473,4476,4479],{"name":4382,"slug":4383,"type":15},{"name":4474,"slug":4475,"type":15},"Computer Vision","computer-vision",{"name":4477,"slug":4478,"type":15},"Images","images",{"name":4439,"slug":2797,"type":15},"2026-07-18T05:14:18.007737",{"slug":4482,"name":4482,"fn":4483,"description":4484,"org":4485,"tags":4486,"stars":4387,"repoUrl":4388,"updatedAt":4495},"azure-appconfiguration-java","manage configuration with Azure App Configuration","Azure App Configuration SDK for Java. Centralized application configuration management with key-value settings, feature flags, and snapshots.\nTriggers: \"ConfigurationClient java\", \"app configuration java\", \"feature flag java\", \"configuration setting java\", \"azure config java\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4487,4488,4491,4494],{"name":4382,"slug":4383,"type":15},{"name":4489,"slug":4490,"type":15},"Configuration","configuration",{"name":4492,"slug":4493,"type":15},"Feature Flags","feature-flags",{"name":4404,"slug":4405,"type":15},"2026-07-03T16:32:01.278468",{"slug":4497,"name":4497,"fn":4498,"description":4499,"org":4500,"tags":4501,"stars":4387,"repoUrl":4388,"updatedAt":4514},"azure-cosmos-rust","build applications with Azure Cosmos DB","Azure Cosmos DB library for Rust (NoSQL API). Document CRUD, containers, and globally distributed data.\nTriggers: \"cosmos db rust\", \"CosmosClient rust\", \"document crud rust\", \"NoSQL rust\", \"partition key rust\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4502,4505,4508,4511],{"name":4503,"slug":4504,"type":15},"Cosmos DB","cosmos-db",{"name":4506,"slug":4507,"type":15},"Database","database",{"name":4509,"slug":4510,"type":15},"NoSQL","nosql",{"name":4512,"slug":4513,"type":15},"Rust","rust","2026-07-31T05:54:27.021432",{"slug":4516,"name":4516,"fn":4498,"description":4517,"org":4518,"tags":4519,"stars":4387,"repoUrl":4388,"updatedAt":4527},"azure-cosmos-ts","Azure Cosmos DB JavaScript\u002FTypeScript SDK (@azure\u002Fcosmos) for data plane operations. Use for CRUD operations on documents, queries, bulk operations, and container management. Triggers: \"Cosmos DB\", \"@azure\u002Fcosmos\", \"CosmosClient\", \"document CRUD\", \"NoSQL queries\", \"bulk operations\", \"partition key\", \"container.items\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4520,4521,4522,4523,4524],{"name":4503,"slug":4504,"type":15},{"name":4506,"slug":4507,"type":15},{"name":9,"slug":8,"type":15},{"name":4509,"slug":4510,"type":15},{"name":4525,"slug":4526,"type":15},"TypeScript","typescript","2026-07-03T16:31:19.368382",{"slug":4529,"name":4529,"fn":4530,"description":4531,"org":4532,"tags":4533,"stars":4387,"repoUrl":4388,"updatedAt":4539},"azure-data-tables-java","build table storage applications with Java","Build table storage applications with Azure Tables SDK for Java. Use when working with Azure Table Storage or Cosmos DB Table API for NoSQL key-value data, schemaless storage, or structured data at scale.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4534,4535,4536,4537,4538],{"name":4382,"slug":4383,"type":15},{"name":4503,"slug":4504,"type":15},{"name":4506,"slug":4507,"type":15},{"name":4404,"slug":4405,"type":15},{"name":4509,"slug":4510,"type":15},"2026-05-13T06:14:17.582229",267,{"items":4542,"total":3589},[4543,4554,4573,4587,4603,4617,4631],{"slug":66,"name":66,"fn":4544,"description":4545,"org":4546,"tags":4547,"stars":28,"repoUrl":29,"updatedAt":4553},"check for Fabric skill marketplace updates","Check an installed skills-for-fabric plugin bundle or git clone for updates, show the matching changelog, and provide host-appropriate update guidance. Use when the user wants to: (1) check for skill updates, (2) see what changed, (3) verify the installed version. Triggers: \"check for updates\", \"am I up to date\", \"what version\", \"update skills\", \"show changelog\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4548,4549,4550],{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":4551,"slug":4552,"type":15},"Operations","operations","2026-07-24T05:41:25.892114",{"slug":4555,"name":4555,"fn":4556,"description":4557,"org":4558,"tags":4559,"stars":28,"repoUrl":29,"updatedAt":4572},"databricks-migration","migrate Databricks notebooks to Microsoft Fabric","Port Databricks notebooks and jobs to Microsoft Fabric. Provides an exhaustive dbutils to notebookutils substitution table: fs operations (runtime mounts or OneLake Shortcuts), secret scope to Key Vault URL conversion, notebook run and exit, widget replacement with parameter-tagged cells, and library install replacement with Fabric Environments. Covers Unity Catalog three-level namespace reduction to Lakehouse two-level schemas, DBFS path conversion to OneLake, Databricks Jobs to Spark Job Definitions, MLflow tracking URI removal, and Photon to Native Execution Engine substitution. Use when the user wants to: (1) replace dbutils with notebookutils, (2) collapse Unity Catalog namespaces to Lakehouse schemas, (3) convert Databricks Jobs or Delta Live Tables. Triggers: \"migrate from databricks\", \"databricks to fabric\", \"dbutils to notebookutils\", \"dbutils fabric\", \"unity catalog migration\", \"dbfs to onelake\", \"databricks notebook migration\", \"delta live tables fabric\", \"photon native execution\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4560,4561,4564,4565,4566,4569],{"name":13,"slug":14,"type":15},{"name":4562,"slug":4563,"type":15},"Databricks","databricks",{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":4567,"slug":4568,"type":15},"Migration","migration",{"name":4570,"slug":4571,"type":15},"Notebooks","notebooks","2026-07-24T05:41:22.914341",{"slug":4574,"name":4574,"fn":4575,"description":4576,"org":4577,"tags":4578,"stars":28,"repoUrl":29,"updatedAt":4586},"dataflows-authoring-cli","manage Microsoft Fabric Dataflows via CLI","Create, update, delete, and refresh Fabric Dataflows Gen2 with write-side CLI via Fabric APIs. Build mashup.pq and queryMetadata.json, preview candidate M with executeQuery\u002FcustomMashupDocument, bind connections, and configure output destinations. For saved query execution or refresh-status reads, use `dataflows-consumption-cli`. If a request explicitly insists on the Dataflows consumption or read-only path for a mutation, do not route here; let consumption refuse before any separately confirmed authoring handoff. Triggers: \"create dataflow\", \"update dataflow\", \"delete dataflow\", \"trigger dataflow refresh\", \"preview Power Query M\", \"preview before save\", \"customMashupDocument\", \"create Fabric data source connection\", \"create SQL Server source REST\", \"POST \u002Fv1\u002Fconnections\", \"supportedConnectionTypes\", \"passwordReference\", \"bind connection\", \"dataflow output destination\", \"dataflow write to lakehouse\", \"dataflow write to warehouse\", \"dataflow write to ADX\", \"DataDestinations annotation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4579,4580,4581,4582,4583],{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":4584,"slug":4585,"type":15},"REST API","rest-api","2026-07-31T05:54:59.059517",{"slug":4588,"name":4588,"fn":4589,"description":4590,"org":4591,"tags":4592,"stars":28,"repoUrl":29,"updatedAt":4602},"e2e-medallion-architecture","implement Medallion Architecture in Microsoft Fabric","Plan and implement end-to-end Microsoft Fabric data platforms and Medallion Architecture (Bronze\u002FSilver\u002FGold) lakehouse patterns using PySpark, Delta Lake, Lakehouse\u002FWarehouse items, Fabric Pipelines, and semantic-model handoff. Use when the user wants to: (1) design a Bronze\u002FSilver\u002FGold data lakehouse, (2) plan an ingestion-to-semantic-model Fabric data platform, (3) set up multi-layer workspaces with lakehouses for each tier, (4) build ingestion-to-analytics pipelines with data quality enforcement, (5) orchestrate Bronze-to-Silver-to-Gold flows via notebooks. For natural-language business questions over existing Power BI report data, use `fabriciq`. Triggers: \"medallion architecture\", \"bronze silver gold\", \"lakehouse layers\", \"e2e data pipeline\", \"end-to-end lakehouse\", \"data lakehouse pattern\", \"multi-layer lakehouse\", \"build medallion\", \"setup medallion\", \"end-to-end Fabric data platform\", \"bronze silver gold platform\", \"ingestion semantic model platform\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4593,4596,4597,4598,4599],{"name":4594,"slug":4595,"type":15},"Architecture","architecture",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":4600,"slug":4601,"type":15},"PySpark","pyspark","2026-07-31T05:54:57.038396",{"slug":4604,"name":4604,"fn":4605,"description":4606,"org":4607,"tags":4608,"stars":28,"repoUrl":29,"updatedAt":4616},"eventhouse-authoring-cli","manage Fabric Eventhouse and KQL databases","Execute KQL management commands (table management, ingestion, policies, functions, materialized views) against Fabric Eventhouse and KQL Databases via CLI. Use when the user wants to:\n  1. Create or alter KQL tables, columns, or functions\n  2. Ingest data into an Eventhouse (inline, from storage, streaming)\n  3. Configure retention, caching, or partitioning policies\n  4. Create or manage materialized views and update policies\n  5. Manage data mappings for ingestion pipelines\n  6. Deploy KQL schema via scripts\nTriggers: \"create kql table\", \"kql ingestion\", \"ingest into eventhouse\", \"kql function\", \"materialized view\", \"kql retention policy\", \"eventhouse schema\", \"kql authoring\", \"create eventhouse table\", \"kql mapping\"\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4609,4610,4613,4614,4615],{"name":4506,"slug":4507,"type":15},{"name":4611,"slug":4612,"type":15},"KQL","kql",{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":4551,"slug":4552,"type":15},"2026-04-06T18:36:51.340409",{"slug":4618,"name":4618,"fn":4619,"description":4620,"org":4621,"tags":4622,"stars":28,"repoUrl":29,"updatedAt":4630},"eventhouse-consumption-cli","query Fabric Eventhouse with KQL","Run KQL queries against Fabric Eventhouse for real-time intelligence and time-series analytics using `az rest` against the Kusto REST API. Covers KQL operators (where, summarize, join, render), Eventhouse schema discovery (.show tables), time-series patterns with bin(), and ingestion monitoring. Use when the user wants to:\n  1. Run read-only KQL queries against an Eventhouse or KQL Database\n  2. Discover Eventhouse table schema and metadata\n  3. Analyse real-time or time-series data with KQL operators\n  4. Monitor ingestion health and active KQL queries\n  5. Export KQL results to JSON\nTriggers: \"kql query\", \"kusto query\", \"eventhouse query\", \"kql database\", \"real-time intelligence\", \"time-series kql\", \"query eventhouse\", \"explore eventhouse\", \"show tables kql\"\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4623,4624,4625,4626,4627],{"name":4397,"slug":4398,"type":15},{"name":4611,"slug":4612,"type":15},{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":4628,"slug":4629,"type":15},"Real-Time Intelligence","real-time-intelligence","2026-04-06T18:36:45.018807",{"slug":4632,"name":4632,"fn":4633,"description":4634,"org":4635,"tags":4636,"stars":28,"repoUrl":29,"updatedAt":4641},"eventstream-consumption-cli","monitor Fabric Eventstream ingestion pipelines","List, inspect, and monitor Fabric Eventstream real-time ingestion pipelines via the Items REST API. Discover Eventstreams across workspaces, decode base64 graph topologies tracing event flow from source through operators to destination nodes. Validate connection IDs, wiring, retention policies (1-90 days), and throughput levels. Retrieve Custom Endpoint Kafka credentials via Topology API. **Invoke this skill** to: (1) list Eventstreams, (2) inspect Eventstream topology showing sources and destinations, (3) validate Eventstream configurations, (4) check Eventstream retention policy and throughput level, (5) get connection strings. Triggers: \"list eventstreams\", \"inspect eventstream\", \"describe eventstream topology\", \"eventstream operator nodes\", \"eventstream sources and destinations\", \"eventstream health\", \"eventstream status\", \"eventstream configuration\", \"eventstream retention\", \"eventstream throughput level\", \"eventstream connection string\", \"custom endpoint credentials\", \"check eventstream\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4637,4638,4639,4640],{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"name":4408,"slug":4409,"type":15},"2026-07-31T05:55:03.045486"]