[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-microsoft-e2e-medallion-architecture":3,"mdc--joyy06-key":37,"related-repo-microsoft-e2e-medallion-architecture":3453,"related-org-microsoft-e2e-medallion-architecture":3557},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"e2e-medallion-architecture","implement Medallion Architecture in Microsoft Fabric","Plan and implement end-to-end Microsoft Fabric data platforms and Medallion Architecture (Bronze\u002FSilver\u002FGold) lakehouse patterns using PySpark, Delta Lake, Lakehouse\u002FWarehouse items, Fabric Pipelines, and semantic-model handoff. Use when the user wants to: (1) design a Bronze\u002FSilver\u002FGold data lakehouse, (2) plan an ingestion-to-semantic-model Fabric data platform, (3) set up multi-layer workspaces with lakehouses for each tier, (4) build ingestion-to-analytics pipelines with data quality enforcement, (5) orchestrate Bronze-to-Silver-to-Gold flows via notebooks. For natural-language business questions over existing Power BI report data, use `fabriciq`. Triggers: \"medallion architecture\", \"bronze silver gold\", \"lakehouse layers\", \"e2e data pipeline\", \"end-to-end lakehouse\", \"data lakehouse pattern\", \"multi-layer lakehouse\", \"build medallion\", \"setup medallion\", \"end-to-end Fabric data platform\", \"bronze silver gold platform\", \"ingestion semantic model platform\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"microsoft","Microsoft","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fmicrosoft.png",[12,16,19,22,23],{"name":13,"slug":14,"type":15},"PySpark","pyspark","tag",{"name":17,"slug":18,"type":15},"Architecture","architecture",{"name":20,"slug":21,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},"Microsoft Fabric","microsoft-fabric",858,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric","2026-07-31T05:54:57.038396",null,237,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"A collection of skills and MCP systems to enable users of CLI, VSCode, Claude to operate over Microsoft Fabric","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Ftree\u002FHEAD\u002Fskills\u002Fe2e-medallion-architecture","---\nname: e2e-medallion-architecture\ndescription: >\n  Plan and implement end-to-end Microsoft Fabric data platforms and Medallion Architecture (Bronze\u002FSilver\u002FGold)\n  lakehouse patterns using PySpark, Delta Lake, Lakehouse\u002FWarehouse items, Fabric Pipelines, and semantic-model handoff.\n  Use when the user wants to: (1) design a Bronze\u002FSilver\u002FGold data lakehouse, (2) plan an ingestion-to-semantic-model Fabric data platform,\n  (3) set up multi-layer workspaces with lakehouses for each tier, (4) build ingestion-to-analytics pipelines with data quality enforcement,\n  (5) orchestrate Bronze-to-Silver-to-Gold flows via notebooks. For natural-language business questions over existing Power BI report data, use `fabriciq`.\n  Triggers: \"medallion architecture\",\n  \"bronze silver gold\", \"lakehouse layers\", \"e2e data pipeline\", \"end-to-end lakehouse\",\n  \"data lakehouse pattern\", \"multi-layer lakehouse\", \"build medallion\", \"setup medallion\",\n  \"end-to-end Fabric data platform\", \"bronze silver gold platform\",\n  \"ingestion semantic model platform\".\n---\n\n> **Update Check — ONCE PER SESSION (mandatory)**\n> The first time this skill is used in a session, run the **check-updates** skill before proceeding.\n> - **GitHub Copilot CLI \u002F VS Code**: invoke the `check-updates` skill.\n> - **Claude Code \u002F Cowork \u002F Cursor \u002F Windsurf \u002F Codex**: compare local vs remote package.json version.\n> - Skip if the check was already performed earlier in this session.\n\n> **CRITICAL NOTES**\n> 1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering\n> 2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering\n\n# End-to-End Medallion Architecture\n\n## Prerequisite Knowledge\n\nRead these companion documents — they contain the foundational context this skill depends on:\n\n- [COMMON-CORE.md](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery\n- [COMMON-CLI.md](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI\n- [SPARK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, job execution\n- [notebook-api-operations.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md) — **Required for notebook creation** — `.ipynb` structure requirements, cell format, `getDefinition`\u002F`updateDefinition` workflow\n\nFor Spark-specific optimization details, see [data-engineering-patterns.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fdata-engineering-patterns.md).\n\n---\n\n## Architecture Overview\n\n**Medallion Architecture** is a data lakehouse pattern with three progressive layers:\n\n| Layer | Purpose | Optimization Profile | Use Case |\n|-------|---------|---------------------|----------|\n| **Bronze** (Raw) | Land raw data exactly as received | Write-optimized, append-only, partitioned by ingestion date | Audit trail, reprocessing, lineage |\n| **Silver** (Cleaned) | Deduplicated, validated, conformed data | Balanced read\u002Fwrite, partitioned by business date | Feature engineering, operational reporting |\n| **Gold** (Aggregated) | Pre-calculated metrics for analytics | Read-optimized (ZORDER, compaction), partitioned by month\u002Fyear | Power BI reports, dashboards, ad-hoc analytics via SQL endpoint |\n\n- **Bronze**: Schema-on-read — flexible schema, Delta time travel supports audit and rollback\n- **Silver**: Schema enforcement — reject non-conforming writes; handle schema evolution with `mergeSchema` when sources change\n- **Gold**: Strict schema governance — curated, business-approved datasets only\n\n---\n\n## Must\u002FPrefer\u002FAvoid\n\n### MUST DO\n- **Choose lakehouse architecture** based on schema-enabled availability (see [infrastructure-orchestration.md](..\u002Fspark-authoring-cli\u002Fresources\u002Finfrastructure-orchestration.md)):\n  - **Preferred:** Schema-enabled lakehouse → create ONE workspace + ONE lakehouse with `bronze`, `silver`, `gold` schemas\n  - **Legacy:** Non-schema-enabled → create separate workspaces per layer (Bronze, Silver, Gold) for governance and access control\n- **Use Livy API for schema and table creation** — to create schemas and tables in a schema-enabled lakehouse, submit Spark SQL statements via Livy sessions (`POST \u002FlivyApi\u002Fversions\u002F2023-12-01\u002Fsessions` → `POST ...\u002Fstatements`). This is the only programmatic REST path for DDL operations (CREATE SCHEMA, CREATE TABLE) in Fabric lakehouses.\n- Add **metadata columns** in Bronze: ingestion timestamp, source file, batch ID\n- Apply **data quality rules** in the Bronze-to-Silver transformation (deduplication, null handling, range validation)\n- Use **Delta Lake format** for all medallion layer tables\n- Use **partition-aware overwrite** in Silver\u002FGold writes to avoid reprocessing unchanged data\n- Include **validation steps** after each layer (row counts, schema checks, anomaly detection)\n- Follow the **`.ipynb` validation + Fabric nuances** in [notebook-api-operations.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md#ipynb-validation--fabric-nuances) when creating notebooks via REST API — every code cell must include `\"outputs\": []` and `\"execution_count\": null`\n- **Complete the full end-to-end flow** — do not stop after creating notebooks; always bind lakehouses, execute notebooks sequentially (Bronze → Silver → Gold), verify results, and connect Power BI to the Gold layer unless the user explicitly requests a partial setup\n\n### PREFER\n- Incremental processing (watermark pattern) over full refresh\n- Separate notebooks per layer for independent testing and debugging\n- ZORDER on frequently filtered columns in Gold tables\n- Running OPTIMIZE after writes in Silver and Gold layers\n- Environment-specific Spark configs (write-heavy for Bronze, balanced for Silver, read-heavy for Gold)\n- OneLake shortcuts to expose Gold data to consumer workspaces without duplication\n- Clear layer ownership: engineers own Bronze\u002FSilver, analysts own Gold\n- Fabric Variable Libraries to centralize paths and configuration across layers\n- Multi-workspace deployment patterns for medium\u002Fhigh governance requirements (Bronze\u002FSilver\u002FGold in separate workspaces)\n- Use Materialized Lake Views (MLVs) for Silver\u002FGold tables when the transformation is expressible in Spark SQL and benefits from declarative refresh semantics. See [spark-authoring-cli — Materialized Lake View patterns](..\u002Fspark-authoring-cli\u002Fresources\u002Fmaterialized-lake-view-patterns.md) and [MLV incremental refresh patterns](..\u002Fspark-authoring-cli\u002Fresources\u002Fmlv-incremental-refresh-patterns.md).\n- For MLV refresh scheduling and monitoring (create\u002Fdelete schedules, trigger on-demand refresh, check job status), use [mlv-operations-cli](..\u002Fmlv-operations-cli\u002FSKILL.md). Note: \"materialized view\", \"spark materialized view\", and \"MLV\" all refer to the same Fabric feature.\n\n### AVOID\n- **Storing all layers in a single lakehouse WITHOUT schemas** — non-schema lakehouses require notebook init cells or Environment configuration to enable OneLake Spark Catalog for RLS\u002FCLS and MLVs. Use separate lakehouses for isolation if schemas aren't available.\n- **Creating 3 separate lakehouses when schema-enabled lakehouse is available** — use schemas within one lakehouse instead (cleaner, no boilerplate init cells, more efficient for MLV cross-schema transformations)\n- Skipping the Silver layer and going directly from Bronze to Gold\n- Hardcoded workspace IDs, lakehouse IDs, or FQDNs — discover via REST API\n- SELECT * without LIMIT on Bronze tables (they grow unboundedly)\n- Running VACUUM without checking downstream dependencies\n- Chaining OneLake shortcuts between medallion layers (Bronze→Silver→Gold) — each layer must be physically materialized for lineage and governance\n- Copying complete implementation code into skills — guide the LLM to generate instead\n- Reading from **external HTTP\u002FHTTPS URLs** directly in Spark — Fabric Spark cannot access arbitrary external URLs; land data in lakehouse `Files\u002F` first (via `curl`, OneLake API, or Fabric pipeline Copy activity), then read from the lakehouse path\n- Creating notebooks via REST API **without validating `.ipynb` structure** — missing `execution_count: null` or `outputs: []` on code cells causes silent failures or \"Job instance failed without detail error\"\n\n---\n\n## Workspace Setup Guidance\n\nWhen setting up a medallion workspace, choose your architecture pattern first (see [infrastructure-orchestration.md](..\u002Fspark-authoring-cli\u002Fresources\u002Finfrastructure-orchestration.md) for detailed guidance):\n\n### Option A: Schema-Enabled Lakehouse (Preferred)\n\n1. **Create single workspace**: `{project}-{env}`\n2. **Create one lakehouse** with schemas: `{project}_lakehouse`\n3. **Create schemas within the lakehouse**:\n   - `bronze` schema for raw ingestion\n   - `silver` schema for cleaned\u002Fvalidated data\n   - `gold` schema for aggregated analytics\n4. **Choose transformation approach**:\n   - **Option 4a:** Use notebooks for each layer (PySpark or Spark SQL transformations)\n   - **Option 4b:** Use Materialized Lake Views (Spark SQL) for declarative transformations with incremental refresh (when query is IR-eligible) — see [materialized-lake-view-patterns.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fmaterialized-lake-view-patterns.md) and [mlv-incremental-refresh-patterns.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fmlv-incremental-refresh-patterns.md)\n   - **Note:** PySpark MLVs exist but use full refresh only (no incremental) — use when you need UDFs\u002Fcomplex Python logic\n   - **MLV benefit:** OneLake Spark Catalog is **automatically enabled** for schema-enabled lakehouses — MLVs work out-of-box with no notebook init cells or Environment configuration required\n5. **RBAC** (optional): Use row-level security and column masking within schemas for fine-grained access control (also requires OneLake Spark Catalog)\n\n### Option B: Separate Lakehouses (Legacy)\n\n1. **Create three workspaces**:\n   - `{project}-bronze-{env}`\n   - `{project}-silver-{env}`\n   - `{project}-gold-{env}`\n2. **Create one lakehouse per workspace**:\n   - Bronze workspace → `{project}_bronze` lakehouse\n   - Silver workspace → `{project}_silver` lakehouse\n   - Gold workspace → `{project}_gold` lakehouse\n3. **Assign RBAC per layer workspace**:\n   - Bronze: ingestion\u002Fengineering write permissions\n   - Silver: engineering\u002Fdata quality permissions\n   - Gold: analytics\u002FBI consumer access with stricter curation controls\n4. **Enable OneLake Spark Catalog for non-schema lakehouses** (required for RLS\u002FCLS and catalog-backed access patterns):\n   - **Primary:** Set `spark.sql.fabric.catalog.enable-schemaless-lakehouses=true` in an Environment and attach it to notebooks.\n   - **Alternative:** Omit default lakehouse binding from notebooks. Use four-part fully-qualified references (`workspace.lakehouse.schema.table`). OneLake Spark Catalog auto-enables when no default lakehouse is set.\n   - **Alternative (internal\u002Funsupported):** Add this as the **first cell** in every notebook:\n   ```python\n   %%pyspark\n   !echo \"spark.sql.fabric.catalog.enable-schemaless-lakehouses=true\" >> \u002Fhome\u002Ftrusted-service-user\u002F.trident-context\n   ```\n   - **⚠️ Note:** This workaround uses an internal runtime configuration path that may change in future Fabric releases. **Prefer schema-enabled lakehouses** for stable, documented OneLake Spark Catalog support.\n   - With this configuration, non-schema lakehouses support:\n     - ✅ Row-level security (RLS) and column-level security (CLS)\n   - **Note:** MLVs require schema-enabled lakehouses (Option A). For non-schema lakehouses, use notebooks with Delta tables.\n\n### Common Steps (Both Options)\n\nAfter completing Option A or Option B above, perform these steps:\n\n1. **Create notebooks** for each layer (one per transformation stage) — follow `.ipynb` validation + Fabric nuances\n2. **Bind each notebook to its lakehouse** — set `metadata.dependencies.lakehouse` with the correct lakehouse ID (see [notebook-api-operations.md § Default Lakehouse Binding](..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md#default-lakehouse-binding)):\n   - Option A: All notebooks → same lakehouse, use schema prefixes (`bronze.table`, `silver.table`)\n   - Option B:\n     - Bronze notebook → Bronze workspace\u002Flakehouse\n     - Silver notebook → Silver workspace\u002Flakehouse (reads Bronze via cross-workspace OneLake access \u002F fully qualified references)\n     - Gold notebook → Gold workspace\u002Flakehouse (reads Silver via cross-workspace access)\n3. **Confirm notebook deployment** — check that `updateDefinition` returned `Succeeded`; this is sufficient confirmation that content and lakehouse binding persisted. Do NOT call `getDefinition` to re-verify — it is an async LRO and adds unnecessary latency.\n4. **Execute notebooks** sequentially — Bronze first, then Silver, then Gold — using `POST ...\u002Fjobs\u002Finstances?jobType=RunNotebook` with the correct `defaultLakehouse` in execution config (both `id` and `name` required)\n5. **Connect Power BI to Gold layer** — discover the Gold lakehouse SQL endpoint, create a Direct Lake semantic model, create a report with visuals on the Gold summary table (see [Gold Layer → Power BI Consumption](#gold-layer--power-bi-consumption))\n6. **Create pipeline** to orchestrate the Bronze → Silver → Gold flow for recurring execution\n\n### Explicit Override: Single Workspace\n\nIf the user explicitly asks for a single workspace deployment (for example, POC\u002Fsmall team\u002Fmonolithic pattern), keep the current approach:\n\n- One workspace with separate Bronze\u002FSilver\u002FGold lakehouses\n- Preserve layer separation logically even when workspace is shared\n- Call out governance trade-offs versus multi-workspace design\n\nParameterize by environment: workspace name suffix (`-dev`, `-prod`), data volume (sample vs full), capacity SKU, and Bronze retention period.\n\n---\n\n## Bronze Layer — Ingestion Patterns\n\nWhen a user requests data ingestion into the Bronze layer, guide LLM to:\n\n1. **Land data in lakehouse first**: External data must be staged into the lakehouse `Files\u002F` folder before Spark can read it — use one of:\n   - **Fabric Pipeline Copy activity** (preferred for recurring loads) — connects to external sources (HTTP, FTP, databases, cloud storage) and writes to OneLake\n   - **OneLake API \u002F `curl`** — upload files via REST API using `storage.azure.com` token (see COMMON-CLI.md § OneLake Data Access)\n   - **OneLake Shortcut** — for data already in Azure ADLS Gen2, S3, or another OneLake location\n   - **`notebookutils.fs`** — copy from mounted storage paths within a notebook\n   - ⚠️ **Fabric Spark cannot read from arbitrary HTTP\u002FHTTPS URLs** — `spark.read.format(\"csv\").load(\"https:\u002F\u002F...\")` will fail\n2. **Read from lakehouse path**: Once data is in `Files\u002F`, read using lakehouse-relative paths (e.g., `spark.read.format(\"csv\").load(\"Files\u002Flanding\u002Fdaily\u002F\")`)\n3. **Add metadata and write**: Tracking columns (ingestion timestamp, source file, batch ID), Delta table with descriptive name, partition by ingestion date, append mode\n4. **Validate**: Log row counts, validate schema structure, flag anomalies vs historical patterns\n\n---\n\n## Silver Layer — Transformation Patterns\n\nWhen a user requests Bronze-to-Silver transformation, guide LLM to:\n\n- **Quality rules**: Deduplicate on natural\u002Fcomposite key, filter invalid ranges, handle nulls (drop required, fill optional), validate logical constraints\n- **Schema conformance**: snake_case column names, standardized data types, derived columns (durations, percentages, categories)\n- **Schema evolution**: Use `mergeSchema` option when source schemas change; coordinate downstream updates to Gold tables and Power BI datasets\n- **Write strategy**: Partition by business date, partition-aware overwrite, run OPTIMIZE after write, log before\u002Fafter metrics\n\n---\n\n## Gold Layer — Aggregation Patterns\n\nWhen a user requests Gold analytics tables, guide LLM to generate:\n\n- **Common aggregates**: Daily\u002Fweekly\u002Fmonthly summaries, dimensional analysis (by location, category, type), trend breakdowns over time, demand patterns (hour-of-day, day-of-week)\n- **Spark session config** — set these properties in the Gold notebook **before** any write operations:\n  ```python\n  spark.conf.set(\"spark.sql.parquet.vorder.default\", \"true\")\n  spark.conf.set(\"spark.databricks.delta.optimizeWrite.enabled\", \"true\")\n  spark.conf.set(\"spark.databricks.delta.optimizeWrite.binSize\", \"1g\")\n  ```\n  - **V-Order** (`vorder.default`) — applies Fabric's columnar sort optimization to all Parquet files, dramatically improving Direct Lake and SQL endpoint read performance\n  - **Optimize Write** (`optimizeWrite.enabled`) — coalesces small partitions into optimally-sized files (target ~1 GB per `binSize`), reducing file count and improving scan efficiency\n- **Optimization**: ZORDER on filter columns, run OPTIMIZE after writes, pre-aggregate metrics to avoid runtime computation\n\n---\n\n## End-to-End Execution Flow\n\nWhen setting up medallion architecture end-to-end, the LLM **must not stop** after creating notebooks and deploying code. The complete lifecycle is:\n\n```\nCreate Resources → Deploy Content → Bind Lakehouses → Execute → Verify Results\n```\n\n### Step-by-Step\n\n1. **Create layer workspaces and lakehouses (default)** — one workspace and one lakehouse per layer (Bronze, Silver, Gold); capture workspace IDs and lakehouse IDs\n2. **Create notebooks** — one per layer, with valid `.ipynb` structure (see [notebook-api-operations.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md))\n3. **Bind lakehouse to each notebook** — include `metadata.dependencies.lakehouse` in the `.ipynb` payload with:\n   - `default_lakehouse`: the target lakehouse GUID\n   - `default_lakehouse_name`: the lakehouse display name\n   - `default_lakehouse_workspace_id`: the workspace GUID\n4. **Deploy notebook content** — `updateDefinition` with the Base64-encoded `.ipynb` payload (content + lakehouse binding together)\n5. **Confirm deployment** — check that each `updateDefinition` LRO returned `Succeeded`; that is sufficient. Do NOT call `getDefinition` to re-verify — it is an async LRO and adds significant latency per notebook.\n6. **Execute notebooks sequentially** — use `POST ...\u002Fjobs\u002Finstances?jobType=RunNotebook`:\n   - Pass `defaultLakehouse` with both `id` and `name` in `executionData.configuration`\n   - Run Bronze first → poll until `Completed` → run Silver → poll → run Gold → poll\n   - Check for recent jobs before submitting (prevent duplicates — see SPARK-AUTHORING-CORE.md)\n7. **Verify results** — after each notebook completes, confirm expected tables exist and row counts are reasonable\n8. **Connect Power BI to Gold** — create semantic model + report on Gold summary tables (see [Gold Layer → Power BI Consumption](#gold-layer--power-bi-consumption))\n\n### Common Failure: Stopping After Notebook Creation\n\nIf the flow stops after deploying notebook code without binding or executing:\n- Notebooks will have no lakehouse context → `spark.sql()` and relative paths (`Tables\u002F`, `Files\u002F`) fail at runtime\n- The user sees no output or results — the architecture is set up but never tested\n- **Always complete through step 7** unless the user explicitly asks to stop at a specific step\n\n---\n\n## Gold Layer → Power BI Consumption\n\nAfter Gold tables are populated, connect Power BI to surface the analytics. \nBuild a semantic model on top of the Gold lakehouse, using DirectLake. \n\n\n### Step-by-Step\n\n1. **Discover the Gold lakehouse SQL endpoint** — call `GET \u002Fv1\u002Fworkspaces\u002F{workspaceId}\u002Flakehouses\u002F{goldLakehouseId}` and extract `properties.sqlEndpointProperties.connectionString` and `provisioningStatus`; wait until status is `Success`\n2. **Verify Gold tables via SQL** — connect to the SQL endpoint using `sqlcmd` (see [COMMON-CLI.md § SQL \u002F TDS Data-Plane Access](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#sql--tds-data-plane-access)) and confirm the target table exists:\n   ```sql\n   SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 'nyc_taxi_daily_summary'\n   ```\n3. **Create a semantic model** — use the [semantic-model-authoring](..\u002Fsemantic-model-authoring\u002FSKILL.md) skill for semantic model creation and TMDL deployment. Create via `POST \u002Fv1\u002Fworkspaces\u002F{workspaceId}\u002Fitems` with `type: \"SemanticModel\"` then deploy definition via `updateDefinition` using TMDL format (see [ITEM-DEFINITIONS-CORE.md § SemanticModel](..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#semanticmodel)):\n   - The model must reference the Gold lakehouse SQL endpoint as its data source\n   - Define a table mapping to the Gold summary table (e.g., `nyc_taxi_daily_summary`)\n   - Use **Direct Lake** mode — this connects directly to Delta tables in OneLake without data import\n   - Include measures for key aggregations you find interesting (e.g., `Total Trips`, `Avg Fare`, `Total Revenue`, `Month over Month Growth`)\n4. **Create a Power BI report** — `POST \u002Fv1\u002Fworkspaces\u002F{workspaceId}\u002Fitems` with `type: \"Report\"` then deploy definition via `updateDefinition` using PBIR format (see [ITEM-DEFINITIONS-CORE.md § Report](..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#report)):\n   - Reference the semantic model created in step 3 via `definition.pbir`\n   - Define at least one page with visuals on the Gold summary table\n   - Suggested visuals: line chart (daily trend), card (KPI totals), bar chart (by category), table (detail view)\n5. **Verify end-to-end** — use the `semantic-model-authoring` skill for metadata discovery running DAX queries against the semantic model and confirm data flows from Gold tables through to the report\n\n### Principles\n\n- **Discover SQL endpoint dynamically** — the connection string is in `properties.sqlEndpointProperties.connectionString` on the lakehouse response; never hardcode it\n- **Wait for SQL endpoint provisioning** — status must be `Success` before connecting; newly created lakehouses may take minutes to provision\n- **Prefer Direct Lake mode** — avoids data duplication; semantic model reads directly from OneLake Delta tables\n- **Match table\u002Fcolumn names exactly** — the semantic model table definition must use the exact Delta table and column names from the Gold lakehouse\n- **For semantic model authoring** (TMDL, refresh, permissions), cross-reference the [semantic-model-authoring](..\u002Fsemantic-model-authoring\u002FSKILL.md) skill\n- **For DAX query validation**, cross-reference the [semantic-model-authoring](..\u002Fsemantic-model-authoring\u002FSKILL.md) skill for metadata discovery and DAX queries for validation.\n\n---\n\n## Pipeline Orchestration\n\nWhen a user requests a pipeline for the medallion flow, guide LLM to design with:\n\n- **Structure**: Sequential activities (Bronze → Silver → Gold), each waiting for previous success; independent Gold aggregations can run in parallel; include validation and notification activities\n- **Parameterization**: Pipeline-level processing date (defaults to yesterday), passed to all notebooks; dynamic date expressions\n- **Scheduling**: Daily aligned with source refresh, watermark-based incremental processing, periodic full refresh for corrections\n- **Error handling**: Retry with backoff for transient failures, alerting for persistent failures, graceful degradation (downstream uses previous data if upstream fails)\n\n---\n\n## Environment Optimization\n\n**For detailed Spark configurations and optimization strategies, see [data-engineering-patterns.md](..\u002Fspark-authoring-cli\u002Fresources\u002Fdata-engineering-patterns.md).**\n\n| Layer | Profile | Key Settings |\n|-------|---------|-------------|\n| Bronze | Write-heavy | Disable V-Order, enable autoCompact, large file targets, partition by ingestion_date |\n| Silver | Balanced | Enable V-Order, adaptive query execution, partition by business date, ZORDER on filtered columns |\n| Gold | Read-heavy | V-Order (`spark.sql.parquet.vorder.default=true`), Optimize Write (`optimizeWrite.enabled=true`, `binSize=1g`), vectorized readers, adaptive execution, ZORDER on all filter columns, pre-aggregate metrics |\n\n---\n\n## Examples\n\n### Example 1: Set Up Medallion Workspaces (Default)\n\n**Prompt**: \"Set up medallion architecture with separate Bronze, Silver, and Gold workspaces for sales analytics\"\n\n**What the LLM should generate**: REST API calls to:\n1. Create workspaces: `sales-bronze-dev`, `sales-silver-dev`, `sales-gold-dev`\n2. Create one lakehouse in each workspace: `sales_bronze`, `sales_silver`, `sales_gold`\n3. Assign RBAC roles per workspace\u002Flayer\n\n```bash\n# Workspace creation (see COMMON-CLI.md for full patterns)\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"sales-analytics-dev\"}\nEOF\nworkspace_id=$(az rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\" \\\n  --body @\u002Ftmp\u002Fbody.json --query \"id\" --output tsv)\n\n# Create Bronze lakehouse\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"sales_bronze\", \"type\": \"Lakehouse\"}\nEOF\naz rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F$workspace_id\u002Fitems\" \\\n  --body @\u002Ftmp\u002Fbody.json\n```\n\n### Example 2: Design Bronze Ingestion\n\n**Prompt**: \"Ingest daily CSV files into bronze lakehouse with metadata columns\"\n\n**What the LLM should generate**: PySpark notebook that:\n1. Reads source files with schema inference or explicit schema\n2. Adds `ingestion_timestamp`, `source_file`, `batch_id` columns\n3. Writes to Delta table partitioned by ingestion date\n4. Logs row count and validation metrics\n\n```python\n# Bronze ingestion pattern (guide LLM to generate full implementation)\nfrom pyspark.sql.functions import current_timestamp, input_file_name, lit\nimport uuid\n\nbatch_id = str(uuid.uuid4())\ndf = (spark.read.format(\"csv\").option(\"header\", True).load(\"\u002FFiles\u002Flanding\u002Fdaily\u002F\")\n      .withColumn(\"ingestion_timestamp\", current_timestamp())\n      .withColumn(\"source_file\", input_file_name())\n      .withColumn(\"batch_id\", lit(batch_id)))\ndf.write.mode(\"append\").partitionBy(\"ingestion_date\").format(\"delta\").saveAsTable(\"bronze.events_raw\")\n```\n\n### Example 3: Bronze-to-Silver Transformation\n\n**Prompt**: \"Clean bronze data: remove duplicates, filter invalid records, add derived columns, write to silver\"\n\n**What the LLM should generate**: PySpark notebook applying quality rules, schema conformance, and partitioned write with optimization.\n\n### Example 4: End-to-End Pipeline\n\n**Prompt**: \"Create a pipeline that runs bronze ingestion, then silver transformation, then gold aggregation daily at 2 AM\"\n\n**What the LLM should generate**: Pipeline JSON definition with sequential notebook activities, date parameter, retry logic, and schedule trigger.\n",{"data":38,"body":39},{"name":4,"description":6},{"type":40,"children":41},"root",[42,105,130,137,144,149,242,255,259,265,275,397,435,438,444,451,650,656,736,742,858,861,867,878,884,1042,1048,1305,1311,1316,1506,1512,1517,1535,1555,1558,1564,1569,1712,1715,1721,1726,1776,1779,1785,1790,1908,1911,1917,1929,1939,1945,2185,2191,2196,2241,2244,2249,2254,2259,2531,2537,2626,2629,2635,2640,2683,2686,2692,2705,2805,2808,2814,2820,2830,2840,2898,3244,3250,3259,3268,3313,3399,3405,3414,3423,3429,3438,3447],{"type":43,"tag":44,"props":45,"children":46},"element","blockquote",{},[47,67],{"type":43,"tag":48,"props":49,"children":50},"p",{},[51,58,60,65],{"type":43,"tag":52,"props":53,"children":54},"strong",{},[55],{"type":56,"value":57},"text","Update Check — ONCE PER SESSION (mandatory)",{"type":56,"value":59},"\nThe first time this skill is used in a session, run the ",{"type":43,"tag":52,"props":61,"children":62},{},[63],{"type":56,"value":64},"check-updates",{"type":56,"value":66}," skill before proceeding.",{"type":43,"tag":68,"props":69,"children":70},"ul",{},[71,90,100],{"type":43,"tag":72,"props":73,"children":74},"li",{},[75,80,82,88],{"type":43,"tag":52,"props":76,"children":77},{},[78],{"type":56,"value":79},"GitHub Copilot CLI \u002F VS Code",{"type":56,"value":81},": invoke the ",{"type":43,"tag":83,"props":84,"children":86},"code",{"className":85},[],[87],{"type":56,"value":64},{"type":56,"value":89}," skill.",{"type":43,"tag":72,"props":91,"children":92},{},[93,98],{"type":43,"tag":52,"props":94,"children":95},{},[96],{"type":56,"value":97},"Claude Code \u002F Cowork \u002F Cursor \u002F Windsurf \u002F Codex",{"type":56,"value":99},": compare local vs remote package.json version.",{"type":43,"tag":72,"props":101,"children":102},{},[103],{"type":56,"value":104},"Skip if the check was already performed earlier in this session.",{"type":43,"tag":44,"props":106,"children":107},{},[108,116],{"type":43,"tag":48,"props":109,"children":110},{},[111],{"type":43,"tag":52,"props":112,"children":113},{},[114],{"type":56,"value":115},"CRITICAL NOTES",{"type":43,"tag":117,"props":118,"children":119},"ol",{},[120,125],{"type":43,"tag":72,"props":121,"children":122},{},[123],{"type":56,"value":124},"To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering",{"type":43,"tag":72,"props":126,"children":127},{},[128],{"type":56,"value":129},"To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering",{"type":43,"tag":131,"props":132,"children":134},"h1",{"id":133},"end-to-end-medallion-architecture",[135],{"type":56,"value":136},"End-to-End Medallion Architecture",{"type":43,"tag":138,"props":139,"children":141},"h2",{"id":140},"prerequisite-knowledge",[142],{"type":56,"value":143},"Prerequisite Knowledge",{"type":43,"tag":48,"props":145,"children":146},{},[147],{"type":56,"value":148},"Read these companion documents — they contain the foundational context this skill depends on:",{"type":43,"tag":68,"props":150,"children":151},{},[152,164,191,202],{"type":43,"tag":72,"props":153,"children":154},{},[155,162],{"type":43,"tag":156,"props":157,"children":159},"a",{"href":158},"..\u002F..\u002Fcommon\u002FCOMMON-CORE.md",[160],{"type":56,"value":161},"COMMON-CORE.md",{"type":56,"value":163}," — Fabric REST API patterns, authentication, token audiences, item discovery",{"type":43,"tag":72,"props":165,"children":166},{},[167,173,175,181,183,189],{"type":43,"tag":156,"props":168,"children":170},{"href":169},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md",[171],{"type":56,"value":172},"COMMON-CLI.md",{"type":56,"value":174}," — ",{"type":43,"tag":83,"props":176,"children":178},{"className":177},[],[179],{"type":56,"value":180},"az rest",{"type":56,"value":182},", ",{"type":43,"tag":83,"props":184,"children":186},{"className":185},[],[187],{"type":56,"value":188},"az login",{"type":56,"value":190},", token acquisition, Fabric REST via CLI",{"type":43,"tag":72,"props":192,"children":193},{},[194,200],{"type":43,"tag":156,"props":195,"children":197},{"href":196},"..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md",[198],{"type":56,"value":199},"SPARK-AUTHORING-CORE.md",{"type":56,"value":201}," — Notebook deployment, lakehouse creation, job execution",{"type":43,"tag":72,"props":203,"children":204},{},[205,211,212,217,218,224,226,232,234,240],{"type":43,"tag":156,"props":206,"children":208},{"href":207},"..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md",[209],{"type":56,"value":210},"notebook-api-operations.md",{"type":56,"value":174},{"type":43,"tag":52,"props":213,"children":214},{},[215],{"type":56,"value":216},"Required for notebook creation",{"type":56,"value":174},{"type":43,"tag":83,"props":219,"children":221},{"className":220},[],[222],{"type":56,"value":223},".ipynb",{"type":56,"value":225}," structure requirements, cell format, ",{"type":43,"tag":83,"props":227,"children":229},{"className":228},[],[230],{"type":56,"value":231},"getDefinition",{"type":56,"value":233},"\u002F",{"type":43,"tag":83,"props":235,"children":237},{"className":236},[],[238],{"type":56,"value":239},"updateDefinition",{"type":56,"value":241}," workflow",{"type":43,"tag":48,"props":243,"children":244},{},[245,247,253],{"type":56,"value":246},"For Spark-specific optimization details, see ",{"type":43,"tag":156,"props":248,"children":250},{"href":249},"..\u002Fspark-authoring-cli\u002Fresources\u002Fdata-engineering-patterns.md",[251],{"type":56,"value":252},"data-engineering-patterns.md",{"type":56,"value":254},".",{"type":43,"tag":256,"props":257,"children":258},"hr",{},[],{"type":43,"tag":138,"props":260,"children":262},{"id":261},"architecture-overview",[263],{"type":56,"value":264},"Architecture Overview",{"type":43,"tag":48,"props":266,"children":267},{},[268,273],{"type":43,"tag":52,"props":269,"children":270},{},[271],{"type":56,"value":272},"Medallion Architecture",{"type":56,"value":274}," is a data lakehouse pattern with three progressive layers:",{"type":43,"tag":276,"props":277,"children":278},"table",{},[279,308],{"type":43,"tag":280,"props":281,"children":282},"thead",{},[283],{"type":43,"tag":284,"props":285,"children":286},"tr",{},[287,293,298,303],{"type":43,"tag":288,"props":289,"children":290},"th",{},[291],{"type":56,"value":292},"Layer",{"type":43,"tag":288,"props":294,"children":295},{},[296],{"type":56,"value":297},"Purpose",{"type":43,"tag":288,"props":299,"children":300},{},[301],{"type":56,"value":302},"Optimization Profile",{"type":43,"tag":288,"props":304,"children":305},{},[306],{"type":56,"value":307},"Use Case",{"type":43,"tag":309,"props":310,"children":311},"tbody",{},[312,341,369],{"type":43,"tag":284,"props":313,"children":314},{},[315,326,331,336],{"type":43,"tag":316,"props":317,"children":318},"td",{},[319,324],{"type":43,"tag":52,"props":320,"children":321},{},[322],{"type":56,"value":323},"Bronze",{"type":56,"value":325}," (Raw)",{"type":43,"tag":316,"props":327,"children":328},{},[329],{"type":56,"value":330},"Land raw data exactly as received",{"type":43,"tag":316,"props":332,"children":333},{},[334],{"type":56,"value":335},"Write-optimized, append-only, partitioned by ingestion date",{"type":43,"tag":316,"props":337,"children":338},{},[339],{"type":56,"value":340},"Audit trail, reprocessing, lineage",{"type":43,"tag":284,"props":342,"children":343},{},[344,354,359,364],{"type":43,"tag":316,"props":345,"children":346},{},[347,352],{"type":43,"tag":52,"props":348,"children":349},{},[350],{"type":56,"value":351},"Silver",{"type":56,"value":353}," (Cleaned)",{"type":43,"tag":316,"props":355,"children":356},{},[357],{"type":56,"value":358},"Deduplicated, validated, conformed data",{"type":43,"tag":316,"props":360,"children":361},{},[362],{"type":56,"value":363},"Balanced read\u002Fwrite, partitioned by business date",{"type":43,"tag":316,"props":365,"children":366},{},[367],{"type":56,"value":368},"Feature engineering, operational reporting",{"type":43,"tag":284,"props":370,"children":371},{},[372,382,387,392],{"type":43,"tag":316,"props":373,"children":374},{},[375,380],{"type":43,"tag":52,"props":376,"children":377},{},[378],{"type":56,"value":379},"Gold",{"type":56,"value":381}," (Aggregated)",{"type":43,"tag":316,"props":383,"children":384},{},[385],{"type":56,"value":386},"Pre-calculated metrics for analytics",{"type":43,"tag":316,"props":388,"children":389},{},[390],{"type":56,"value":391},"Read-optimized (ZORDER, compaction), partitioned by month\u002Fyear",{"type":43,"tag":316,"props":393,"children":394},{},[395],{"type":56,"value":396},"Power BI reports, dashboards, ad-hoc analytics via SQL endpoint",{"type":43,"tag":68,"props":398,"children":399},{},[400,409,426],{"type":43,"tag":72,"props":401,"children":402},{},[403,407],{"type":43,"tag":52,"props":404,"children":405},{},[406],{"type":56,"value":323},{"type":56,"value":408},": Schema-on-read — flexible schema, Delta time travel supports audit and rollback",{"type":43,"tag":72,"props":410,"children":411},{},[412,416,418,424],{"type":43,"tag":52,"props":413,"children":414},{},[415],{"type":56,"value":351},{"type":56,"value":417},": Schema enforcement — reject non-conforming writes; handle schema evolution with ",{"type":43,"tag":83,"props":419,"children":421},{"className":420},[],[422],{"type":56,"value":423},"mergeSchema",{"type":56,"value":425}," when sources change",{"type":43,"tag":72,"props":427,"children":428},{},[429,433],{"type":43,"tag":52,"props":430,"children":431},{},[432],{"type":56,"value":379},{"type":56,"value":434},": Strict schema governance — curated, business-approved datasets only",{"type":43,"tag":256,"props":436,"children":437},{},[],{"type":43,"tag":138,"props":439,"children":441},{"id":440},"mustpreferavoid",[442],{"type":56,"value":443},"Must\u002FPrefer\u002FAvoid",{"type":43,"tag":445,"props":446,"children":448},"h3",{"id":447},"must-do",[449],{"type":56,"value":450},"MUST DO",{"type":43,"tag":68,"props":452,"children":453},{},[454,517,543,555,567,579,590,602,640],{"type":43,"tag":72,"props":455,"children":456},{},[457,462,464,470,472],{"type":43,"tag":52,"props":458,"children":459},{},[460],{"type":56,"value":461},"Choose lakehouse architecture",{"type":56,"value":463}," based on schema-enabled availability (see ",{"type":43,"tag":156,"props":465,"children":467},{"href":466},"..\u002Fspark-authoring-cli\u002Fresources\u002Finfrastructure-orchestration.md",[468],{"type":56,"value":469},"infrastructure-orchestration.md",{"type":56,"value":471},"):\n",{"type":43,"tag":68,"props":473,"children":474},{},[475,507],{"type":43,"tag":72,"props":476,"children":477},{},[478,483,485,491,492,498,499,505],{"type":43,"tag":52,"props":479,"children":480},{},[481],{"type":56,"value":482},"Preferred:",{"type":56,"value":484}," Schema-enabled lakehouse → create ONE workspace + ONE lakehouse with ",{"type":43,"tag":83,"props":486,"children":488},{"className":487},[],[489],{"type":56,"value":490},"bronze",{"type":56,"value":182},{"type":43,"tag":83,"props":493,"children":495},{"className":494},[],[496],{"type":56,"value":497},"silver",{"type":56,"value":182},{"type":43,"tag":83,"props":500,"children":502},{"className":501},[],[503],{"type":56,"value":504},"gold",{"type":56,"value":506}," schemas",{"type":43,"tag":72,"props":508,"children":509},{},[510,515],{"type":43,"tag":52,"props":511,"children":512},{},[513],{"type":56,"value":514},"Legacy:",{"type":56,"value":516}," Non-schema-enabled → create separate workspaces per layer (Bronze, Silver, Gold) for governance and access control",{"type":43,"tag":72,"props":518,"children":519},{},[520,525,527,533,535,541],{"type":43,"tag":52,"props":521,"children":522},{},[523],{"type":56,"value":524},"Use Livy API for schema and table creation",{"type":56,"value":526}," — to create schemas and tables in a schema-enabled lakehouse, submit Spark SQL statements via Livy sessions (",{"type":43,"tag":83,"props":528,"children":530},{"className":529},[],[531],{"type":56,"value":532},"POST \u002FlivyApi\u002Fversions\u002F2023-12-01\u002Fsessions",{"type":56,"value":534}," → ",{"type":43,"tag":83,"props":536,"children":538},{"className":537},[],[539],{"type":56,"value":540},"POST ...\u002Fstatements",{"type":56,"value":542},"). This is the only programmatic REST path for DDL operations (CREATE SCHEMA, CREATE TABLE) in Fabric lakehouses.",{"type":43,"tag":72,"props":544,"children":545},{},[546,548,553],{"type":56,"value":547},"Add ",{"type":43,"tag":52,"props":549,"children":550},{},[551],{"type":56,"value":552},"metadata columns",{"type":56,"value":554}," in Bronze: ingestion timestamp, source file, batch ID",{"type":43,"tag":72,"props":556,"children":557},{},[558,560,565],{"type":56,"value":559},"Apply ",{"type":43,"tag":52,"props":561,"children":562},{},[563],{"type":56,"value":564},"data quality rules",{"type":56,"value":566}," in the Bronze-to-Silver transformation (deduplication, null handling, range validation)",{"type":43,"tag":72,"props":568,"children":569},{},[570,572,577],{"type":56,"value":571},"Use ",{"type":43,"tag":52,"props":573,"children":574},{},[575],{"type":56,"value":576},"Delta Lake format",{"type":56,"value":578}," for all medallion layer tables",{"type":43,"tag":72,"props":580,"children":581},{},[582,583,588],{"type":56,"value":571},{"type":43,"tag":52,"props":584,"children":585},{},[586],{"type":56,"value":587},"partition-aware overwrite",{"type":56,"value":589}," in Silver\u002FGold writes to avoid reprocessing unchanged data",{"type":43,"tag":72,"props":591,"children":592},{},[593,595,600],{"type":56,"value":594},"Include ",{"type":43,"tag":52,"props":596,"children":597},{},[598],{"type":56,"value":599},"validation steps",{"type":56,"value":601}," after each layer (row counts, schema checks, anomaly detection)",{"type":43,"tag":72,"props":603,"children":604},{},[605,607,617,619,624,626,632,634],{"type":56,"value":606},"Follow the ",{"type":43,"tag":52,"props":608,"children":609},{},[610,615],{"type":43,"tag":83,"props":611,"children":613},{"className":612},[],[614],{"type":56,"value":223},{"type":56,"value":616}," validation + Fabric nuances",{"type":56,"value":618}," in ",{"type":43,"tag":156,"props":620,"children":622},{"href":621},"..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md#ipynb-validation--fabric-nuances",[623],{"type":56,"value":210},{"type":56,"value":625}," when creating notebooks via REST API — every code cell must include ",{"type":43,"tag":83,"props":627,"children":629},{"className":628},[],[630],{"type":56,"value":631},"\"outputs\": []",{"type":56,"value":633}," and ",{"type":43,"tag":83,"props":635,"children":637},{"className":636},[],[638],{"type":56,"value":639},"\"execution_count\": null",{"type":43,"tag":72,"props":641,"children":642},{},[643,648],{"type":43,"tag":52,"props":644,"children":645},{},[646],{"type":56,"value":647},"Complete the full end-to-end flow",{"type":56,"value":649}," — do not stop after creating notebooks; always bind lakehouses, execute notebooks sequentially (Bronze → Silver → Gold), verify results, and connect Power BI to the Gold layer unless the user explicitly requests a partial setup",{"type":43,"tag":445,"props":651,"children":653},{"id":652},"prefer",[654],{"type":56,"value":655},"PREFER",{"type":43,"tag":68,"props":657,"children":658},{},[659,664,669,674,679,684,689,694,699,704,723],{"type":43,"tag":72,"props":660,"children":661},{},[662],{"type":56,"value":663},"Incremental processing (watermark pattern) over full refresh",{"type":43,"tag":72,"props":665,"children":666},{},[667],{"type":56,"value":668},"Separate notebooks per layer for independent testing and debugging",{"type":43,"tag":72,"props":670,"children":671},{},[672],{"type":56,"value":673},"ZORDER on frequently filtered columns in Gold tables",{"type":43,"tag":72,"props":675,"children":676},{},[677],{"type":56,"value":678},"Running OPTIMIZE after writes in Silver and Gold layers",{"type":43,"tag":72,"props":680,"children":681},{},[682],{"type":56,"value":683},"Environment-specific Spark configs (write-heavy for Bronze, balanced for Silver, read-heavy for Gold)",{"type":43,"tag":72,"props":685,"children":686},{},[687],{"type":56,"value":688},"OneLake shortcuts to expose Gold data to consumer workspaces without duplication",{"type":43,"tag":72,"props":690,"children":691},{},[692],{"type":56,"value":693},"Clear layer ownership: engineers own Bronze\u002FSilver, analysts own Gold",{"type":43,"tag":72,"props":695,"children":696},{},[697],{"type":56,"value":698},"Fabric Variable Libraries to centralize paths and configuration across layers",{"type":43,"tag":72,"props":700,"children":701},{},[702],{"type":56,"value":703},"Multi-workspace deployment patterns for medium\u002Fhigh governance requirements (Bronze\u002FSilver\u002FGold in separate workspaces)",{"type":43,"tag":72,"props":705,"children":706},{},[707,709,715,716,722],{"type":56,"value":708},"Use Materialized Lake Views (MLVs) for Silver\u002FGold tables when the transformation is expressible in Spark SQL and benefits from declarative refresh semantics. See ",{"type":43,"tag":156,"props":710,"children":712},{"href":711},"..\u002Fspark-authoring-cli\u002Fresources\u002Fmaterialized-lake-view-patterns.md",[713],{"type":56,"value":714},"spark-authoring-cli — Materialized Lake View patterns",{"type":56,"value":633},{"type":43,"tag":156,"props":717,"children":719},{"href":718},"..\u002Fspark-authoring-cli\u002Fresources\u002Fmlv-incremental-refresh-patterns.md",[720],{"type":56,"value":721},"MLV incremental refresh patterns",{"type":56,"value":254},{"type":43,"tag":72,"props":724,"children":725},{},[726,728,734],{"type":56,"value":727},"For MLV refresh scheduling and monitoring (create\u002Fdelete schedules, trigger on-demand refresh, check job status), use ",{"type":43,"tag":156,"props":729,"children":731},{"href":730},"..\u002Fmlv-operations-cli\u002FSKILL.md",[732],{"type":56,"value":733},"mlv-operations-cli",{"type":56,"value":735},". Note: \"materialized view\", \"spark materialized view\", and \"MLV\" all refer to the same Fabric feature.",{"type":43,"tag":445,"props":737,"children":739},{"id":738},"avoid",[740],{"type":56,"value":741},"AVOID",{"type":43,"tag":68,"props":743,"children":744},{},[745,755,765,770,775,780,785,790,795,823],{"type":43,"tag":72,"props":746,"children":747},{},[748,753],{"type":43,"tag":52,"props":749,"children":750},{},[751],{"type":56,"value":752},"Storing all layers in a single lakehouse WITHOUT schemas",{"type":56,"value":754}," — non-schema lakehouses require notebook init cells or Environment configuration to enable OneLake Spark Catalog for RLS\u002FCLS and MLVs. Use separate lakehouses for isolation if schemas aren't available.",{"type":43,"tag":72,"props":756,"children":757},{},[758,763],{"type":43,"tag":52,"props":759,"children":760},{},[761],{"type":56,"value":762},"Creating 3 separate lakehouses when schema-enabled lakehouse is available",{"type":56,"value":764}," — use schemas within one lakehouse instead (cleaner, no boilerplate init cells, more efficient for MLV cross-schema transformations)",{"type":43,"tag":72,"props":766,"children":767},{},[768],{"type":56,"value":769},"Skipping the Silver layer and going directly from Bronze to Gold",{"type":43,"tag":72,"props":771,"children":772},{},[773],{"type":56,"value":774},"Hardcoded workspace IDs, lakehouse IDs, or FQDNs — discover via REST API",{"type":43,"tag":72,"props":776,"children":777},{},[778],{"type":56,"value":779},"SELECT * without LIMIT on Bronze tables (they grow unboundedly)",{"type":43,"tag":72,"props":781,"children":782},{},[783],{"type":56,"value":784},"Running VACUUM without checking downstream dependencies",{"type":43,"tag":72,"props":786,"children":787},{},[788],{"type":56,"value":789},"Chaining OneLake shortcuts between medallion layers (Bronze→Silver→Gold) — each layer must be physically materialized for lineage and governance",{"type":43,"tag":72,"props":791,"children":792},{},[793],{"type":56,"value":794},"Copying complete implementation code into skills — guide the LLM to generate instead",{"type":43,"tag":72,"props":796,"children":797},{},[798,800,805,807,813,815,821],{"type":56,"value":799},"Reading from ",{"type":43,"tag":52,"props":801,"children":802},{},[803],{"type":56,"value":804},"external HTTP\u002FHTTPS URLs",{"type":56,"value":806}," directly in Spark — Fabric Spark cannot access arbitrary external URLs; land data in lakehouse ",{"type":43,"tag":83,"props":808,"children":810},{"className":809},[],[811],{"type":56,"value":812},"Files\u002F",{"type":56,"value":814}," first (via ",{"type":43,"tag":83,"props":816,"children":818},{"className":817},[],[819],{"type":56,"value":820},"curl",{"type":56,"value":822},", OneLake API, or Fabric pipeline Copy activity), then read from the lakehouse path",{"type":43,"tag":72,"props":824,"children":825},{},[826,828,840,842,848,850,856],{"type":56,"value":827},"Creating notebooks via REST API ",{"type":43,"tag":52,"props":829,"children":830},{},[831,833,838],{"type":56,"value":832},"without validating ",{"type":43,"tag":83,"props":834,"children":836},{"className":835},[],[837],{"type":56,"value":223},{"type":56,"value":839}," structure",{"type":56,"value":841}," — missing ",{"type":43,"tag":83,"props":843,"children":845},{"className":844},[],[846],{"type":56,"value":847},"execution_count: null",{"type":56,"value":849}," or ",{"type":43,"tag":83,"props":851,"children":853},{"className":852},[],[854],{"type":56,"value":855},"outputs: []",{"type":56,"value":857}," on code cells causes silent failures or \"Job instance failed without detail error\"",{"type":43,"tag":256,"props":859,"children":860},{},[],{"type":43,"tag":138,"props":862,"children":864},{"id":863},"workspace-setup-guidance",[865],{"type":56,"value":866},"Workspace Setup Guidance",{"type":43,"tag":48,"props":868,"children":869},{},[870,872,876],{"type":56,"value":871},"When setting up a medallion workspace, choose your architecture pattern first (see ",{"type":43,"tag":156,"props":873,"children":874},{"href":466},[875],{"type":56,"value":469},{"type":56,"value":877}," for detailed guidance):",{"type":43,"tag":445,"props":879,"children":881},{"id":880},"option-a-schema-enabled-lakehouse-preferred",[882],{"type":56,"value":883},"Option A: Schema-Enabled Lakehouse (Preferred)",{"type":43,"tag":117,"props":885,"children":886},{},[887,903,919,962,1032],{"type":43,"tag":72,"props":888,"children":889},{},[890,895,897],{"type":43,"tag":52,"props":891,"children":892},{},[893],{"type":56,"value":894},"Create single workspace",{"type":56,"value":896},": ",{"type":43,"tag":83,"props":898,"children":900},{"className":899},[],[901],{"type":56,"value":902},"{project}-{env}",{"type":43,"tag":72,"props":904,"children":905},{},[906,911,913],{"type":43,"tag":52,"props":907,"children":908},{},[909],{"type":56,"value":910},"Create one lakehouse",{"type":56,"value":912}," with schemas: ",{"type":43,"tag":83,"props":914,"children":916},{"className":915},[],[917],{"type":56,"value":918},"{project}_lakehouse",{"type":43,"tag":72,"props":920,"children":921},{},[922,927,929],{"type":43,"tag":52,"props":923,"children":924},{},[925],{"type":56,"value":926},"Create schemas within the lakehouse",{"type":56,"value":928},":\n",{"type":43,"tag":68,"props":930,"children":931},{},[932,942,952],{"type":43,"tag":72,"props":933,"children":934},{},[935,940],{"type":43,"tag":83,"props":936,"children":938},{"className":937},[],[939],{"type":56,"value":490},{"type":56,"value":941}," schema for raw ingestion",{"type":43,"tag":72,"props":943,"children":944},{},[945,950],{"type":43,"tag":83,"props":946,"children":948},{"className":947},[],[949],{"type":56,"value":497},{"type":56,"value":951}," schema for cleaned\u002Fvalidated data",{"type":43,"tag":72,"props":953,"children":954},{},[955,960],{"type":43,"tag":83,"props":956,"children":958},{"className":957},[],[959],{"type":56,"value":504},{"type":56,"value":961}," schema for aggregated analytics",{"type":43,"tag":72,"props":963,"children":964},{},[965,970,971],{"type":43,"tag":52,"props":966,"children":967},{},[968],{"type":56,"value":969},"Choose transformation approach",{"type":56,"value":928},{"type":43,"tag":68,"props":972,"children":973},{},[974,984,1005,1015],{"type":43,"tag":72,"props":975,"children":976},{},[977,982],{"type":43,"tag":52,"props":978,"children":979},{},[980],{"type":56,"value":981},"Option 4a:",{"type":56,"value":983}," Use notebooks for each layer (PySpark or Spark SQL transformations)",{"type":43,"tag":72,"props":985,"children":986},{},[987,992,994,999,1000],{"type":43,"tag":52,"props":988,"children":989},{},[990],{"type":56,"value":991},"Option 4b:",{"type":56,"value":993}," Use Materialized Lake Views (Spark SQL) for declarative transformations with incremental refresh (when query is IR-eligible) — see ",{"type":43,"tag":156,"props":995,"children":996},{"href":711},[997],{"type":56,"value":998},"materialized-lake-view-patterns.md",{"type":56,"value":633},{"type":43,"tag":156,"props":1001,"children":1002},{"href":718},[1003],{"type":56,"value":1004},"mlv-incremental-refresh-patterns.md",{"type":43,"tag":72,"props":1006,"children":1007},{},[1008,1013],{"type":43,"tag":52,"props":1009,"children":1010},{},[1011],{"type":56,"value":1012},"Note:",{"type":56,"value":1014}," PySpark MLVs exist but use full refresh only (no incremental) — use when you need UDFs\u002Fcomplex Python logic",{"type":43,"tag":72,"props":1016,"children":1017},{},[1018,1023,1025,1030],{"type":43,"tag":52,"props":1019,"children":1020},{},[1021],{"type":56,"value":1022},"MLV benefit:",{"type":56,"value":1024}," OneLake Spark Catalog is ",{"type":43,"tag":52,"props":1026,"children":1027},{},[1028],{"type":56,"value":1029},"automatically enabled",{"type":56,"value":1031}," for schema-enabled lakehouses — MLVs work out-of-box with no notebook init cells or Environment configuration required",{"type":43,"tag":72,"props":1033,"children":1034},{},[1035,1040],{"type":43,"tag":52,"props":1036,"children":1037},{},[1038],{"type":56,"value":1039},"RBAC",{"type":56,"value":1041}," (optional): Use row-level security and column masking within schemas for fine-grained access control (also requires OneLake Spark Catalog)",{"type":43,"tag":445,"props":1043,"children":1045},{"id":1044},"option-b-separate-lakehouses-legacy",[1046],{"type":56,"value":1047},"Option B: Separate Lakehouses (Legacy)",{"type":43,"tag":117,"props":1049,"children":1050},{},[1051,1090,1139,1166],{"type":43,"tag":72,"props":1052,"children":1053},{},[1054,1059,1060],{"type":43,"tag":52,"props":1055,"children":1056},{},[1057],{"type":56,"value":1058},"Create three workspaces",{"type":56,"value":928},{"type":43,"tag":68,"props":1061,"children":1062},{},[1063,1072,1081],{"type":43,"tag":72,"props":1064,"children":1065},{},[1066],{"type":43,"tag":83,"props":1067,"children":1069},{"className":1068},[],[1070],{"type":56,"value":1071},"{project}-bronze-{env}",{"type":43,"tag":72,"props":1073,"children":1074},{},[1075],{"type":43,"tag":83,"props":1076,"children":1078},{"className":1077},[],[1079],{"type":56,"value":1080},"{project}-silver-{env}",{"type":43,"tag":72,"props":1082,"children":1083},{},[1084],{"type":43,"tag":83,"props":1085,"children":1087},{"className":1086},[],[1088],{"type":56,"value":1089},"{project}-gold-{env}",{"type":43,"tag":72,"props":1091,"children":1092},{},[1093,1098,1099],{"type":43,"tag":52,"props":1094,"children":1095},{},[1096],{"type":56,"value":1097},"Create one lakehouse per workspace",{"type":56,"value":928},{"type":43,"tag":68,"props":1100,"children":1101},{},[1102,1115,1127],{"type":43,"tag":72,"props":1103,"children":1104},{},[1105,1107,1113],{"type":56,"value":1106},"Bronze workspace → ",{"type":43,"tag":83,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":56,"value":1112},"{project}_bronze",{"type":56,"value":1114}," lakehouse",{"type":43,"tag":72,"props":1116,"children":1117},{},[1118,1120,1126],{"type":56,"value":1119},"Silver workspace → ",{"type":43,"tag":83,"props":1121,"children":1123},{"className":1122},[],[1124],{"type":56,"value":1125},"{project}_silver",{"type":56,"value":1114},{"type":43,"tag":72,"props":1128,"children":1129},{},[1130,1132,1138],{"type":56,"value":1131},"Gold workspace → ",{"type":43,"tag":83,"props":1133,"children":1135},{"className":1134},[],[1136],{"type":56,"value":1137},"{project}_gold",{"type":56,"value":1114},{"type":43,"tag":72,"props":1140,"children":1141},{},[1142,1147,1148],{"type":43,"tag":52,"props":1143,"children":1144},{},[1145],{"type":56,"value":1146},"Assign RBAC per layer workspace",{"type":56,"value":928},{"type":43,"tag":68,"props":1149,"children":1150},{},[1151,1156,1161],{"type":43,"tag":72,"props":1152,"children":1153},{},[1154],{"type":56,"value":1155},"Bronze: ingestion\u002Fengineering write permissions",{"type":43,"tag":72,"props":1157,"children":1158},{},[1159],{"type":56,"value":1160},"Silver: engineering\u002Fdata quality permissions",{"type":43,"tag":72,"props":1162,"children":1163},{},[1164],{"type":56,"value":1165},"Gold: analytics\u002FBI consumer access with stricter curation controls",{"type":43,"tag":72,"props":1167,"children":1168},{},[1169,1174,1176,1232,1263],{"type":43,"tag":52,"props":1170,"children":1171},{},[1172],{"type":56,"value":1173},"Enable OneLake Spark Catalog for non-schema lakehouses",{"type":56,"value":1175}," (required for RLS\u002FCLS and catalog-backed access patterns):\n",{"type":43,"tag":68,"props":1177,"children":1178},{},[1179,1197,1215],{"type":43,"tag":72,"props":1180,"children":1181},{},[1182,1187,1189,1195],{"type":43,"tag":52,"props":1183,"children":1184},{},[1185],{"type":56,"value":1186},"Primary:",{"type":56,"value":1188}," Set ",{"type":43,"tag":83,"props":1190,"children":1192},{"className":1191},[],[1193],{"type":56,"value":1194},"spark.sql.fabric.catalog.enable-schemaless-lakehouses=true",{"type":56,"value":1196}," in an Environment and attach it to notebooks.",{"type":43,"tag":72,"props":1198,"children":1199},{},[1200,1205,1207,1213],{"type":43,"tag":52,"props":1201,"children":1202},{},[1203],{"type":56,"value":1204},"Alternative:",{"type":56,"value":1206}," Omit default lakehouse binding from notebooks. Use four-part fully-qualified references (",{"type":43,"tag":83,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":56,"value":1212},"workspace.lakehouse.schema.table",{"type":56,"value":1214},"). OneLake Spark Catalog auto-enables when no default lakehouse is set.",{"type":43,"tag":72,"props":1216,"children":1217},{},[1218,1223,1225,1230],{"type":43,"tag":52,"props":1219,"children":1220},{},[1221],{"type":56,"value":1222},"Alternative (internal\u002Funsupported):",{"type":56,"value":1224}," Add this as the ",{"type":43,"tag":52,"props":1226,"children":1227},{},[1228],{"type":56,"value":1229},"first cell",{"type":56,"value":1231}," in every notebook:",{"type":43,"tag":1233,"props":1234,"children":1239},"pre",{"className":1235,"code":1236,"language":1237,"meta":1238,"style":1238},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","%%pyspark\n!echo \"spark.sql.fabric.catalog.enable-schemaless-lakehouses=true\" >> \u002Fhome\u002Ftrusted-service-user\u002F.trident-context\n","python","",[1240],{"type":43,"tag":83,"props":1241,"children":1242},{"__ignoreMap":1238},[1243,1254],{"type":43,"tag":1244,"props":1245,"children":1248},"span",{"class":1246,"line":1247},"line",1,[1249],{"type":43,"tag":1244,"props":1250,"children":1251},{},[1252],{"type":56,"value":1253},"%%pyspark\n",{"type":43,"tag":1244,"props":1255,"children":1257},{"class":1246,"line":1256},2,[1258],{"type":43,"tag":1244,"props":1259,"children":1260},{},[1261],{"type":56,"value":1262},"!echo \"spark.sql.fabric.catalog.enable-schemaless-lakehouses=true\" >> \u002Fhome\u002Ftrusted-service-user\u002F.trident-context\n",{"type":43,"tag":68,"props":1264,"children":1265},{},[1266,1283,1296],{"type":43,"tag":72,"props":1267,"children":1268},{},[1269,1274,1276,1281],{"type":43,"tag":52,"props":1270,"children":1271},{},[1272],{"type":56,"value":1273},"⚠️ Note:",{"type":56,"value":1275}," This workaround uses an internal runtime configuration path that may change in future Fabric releases. ",{"type":43,"tag":52,"props":1277,"children":1278},{},[1279],{"type":56,"value":1280},"Prefer schema-enabled lakehouses",{"type":56,"value":1282}," for stable, documented OneLake Spark Catalog support.",{"type":43,"tag":72,"props":1284,"children":1285},{},[1286,1288],{"type":56,"value":1287},"With this configuration, non-schema lakehouses support:\n",{"type":43,"tag":68,"props":1289,"children":1290},{},[1291],{"type":43,"tag":72,"props":1292,"children":1293},{},[1294],{"type":56,"value":1295},"✅ Row-level security (RLS) and column-level security (CLS)",{"type":43,"tag":72,"props":1297,"children":1298},{},[1299,1303],{"type":43,"tag":52,"props":1300,"children":1301},{},[1302],{"type":56,"value":1012},{"type":56,"value":1304}," MLVs require schema-enabled lakehouses (Option A). For non-schema lakehouses, use notebooks with Delta tables.",{"type":43,"tag":445,"props":1306,"children":1308},{"id":1307},"common-steps-both-options",[1309],{"type":56,"value":1310},"Common Steps (Both Options)",{"type":43,"tag":48,"props":1312,"children":1313},{},[1314],{"type":56,"value":1315},"After completing Option A or Option B above, perform these steps:",{"type":43,"tag":117,"props":1317,"children":1318},{},[1319,1335,1406,1438,1479,1496],{"type":43,"tag":72,"props":1320,"children":1321},{},[1322,1327,1329,1334],{"type":43,"tag":52,"props":1323,"children":1324},{},[1325],{"type":56,"value":1326},"Create notebooks",{"type":56,"value":1328}," for each layer (one per transformation stage) — follow ",{"type":43,"tag":83,"props":1330,"children":1332},{"className":1331},[],[1333],{"type":56,"value":223},{"type":56,"value":616},{"type":43,"tag":72,"props":1336,"children":1337},{},[1338,1343,1345,1351,1353,1359,1360],{"type":43,"tag":52,"props":1339,"children":1340},{},[1341],{"type":56,"value":1342},"Bind each notebook to its lakehouse",{"type":56,"value":1344}," — set ",{"type":43,"tag":83,"props":1346,"children":1348},{"className":1347},[],[1349],{"type":56,"value":1350},"metadata.dependencies.lakehouse",{"type":56,"value":1352}," with the correct lakehouse ID (see ",{"type":43,"tag":156,"props":1354,"children":1356},{"href":1355},"..\u002Fspark-authoring-cli\u002Fresources\u002Fnotebook-api-operations.md#default-lakehouse-binding",[1357],{"type":56,"value":1358},"notebook-api-operations.md § Default Lakehouse Binding",{"type":56,"value":471},{"type":43,"tag":68,"props":1361,"children":1362},{},[1363,1383],{"type":43,"tag":72,"props":1364,"children":1365},{},[1366,1368,1374,1375,1381],{"type":56,"value":1367},"Option A: All notebooks → same lakehouse, use schema prefixes (",{"type":43,"tag":83,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":56,"value":1373},"bronze.table",{"type":56,"value":182},{"type":43,"tag":83,"props":1376,"children":1378},{"className":1377},[],[1379],{"type":56,"value":1380},"silver.table",{"type":56,"value":1382},")",{"type":43,"tag":72,"props":1384,"children":1385},{},[1386,1388],{"type":56,"value":1387},"Option B:\n",{"type":43,"tag":68,"props":1389,"children":1390},{},[1391,1396,1401],{"type":43,"tag":72,"props":1392,"children":1393},{},[1394],{"type":56,"value":1395},"Bronze notebook → Bronze workspace\u002Flakehouse",{"type":43,"tag":72,"props":1397,"children":1398},{},[1399],{"type":56,"value":1400},"Silver notebook → Silver workspace\u002Flakehouse (reads Bronze via cross-workspace OneLake access \u002F fully qualified references)",{"type":43,"tag":72,"props":1402,"children":1403},{},[1404],{"type":56,"value":1405},"Gold notebook → Gold workspace\u002Flakehouse (reads Silver via cross-workspace access)",{"type":43,"tag":72,"props":1407,"children":1408},{},[1409,1414,1416,1421,1423,1429,1431,1436],{"type":43,"tag":52,"props":1410,"children":1411},{},[1412],{"type":56,"value":1413},"Confirm notebook deployment",{"type":56,"value":1415}," — check that ",{"type":43,"tag":83,"props":1417,"children":1419},{"className":1418},[],[1420],{"type":56,"value":239},{"type":56,"value":1422}," returned ",{"type":43,"tag":83,"props":1424,"children":1426},{"className":1425},[],[1427],{"type":56,"value":1428},"Succeeded",{"type":56,"value":1430},"; this is sufficient confirmation that content and lakehouse binding persisted. Do NOT call ",{"type":43,"tag":83,"props":1432,"children":1434},{"className":1433},[],[1435],{"type":56,"value":231},{"type":56,"value":1437}," to re-verify — it is an async LRO and adds unnecessary latency.",{"type":43,"tag":72,"props":1439,"children":1440},{},[1441,1446,1448,1454,1456,1462,1464,1470,1471,1477],{"type":43,"tag":52,"props":1442,"children":1443},{},[1444],{"type":56,"value":1445},"Execute notebooks",{"type":56,"value":1447}," sequentially — Bronze first, then Silver, then Gold — using ",{"type":43,"tag":83,"props":1449,"children":1451},{"className":1450},[],[1452],{"type":56,"value":1453},"POST ...\u002Fjobs\u002Finstances?jobType=RunNotebook",{"type":56,"value":1455}," with the correct ",{"type":43,"tag":83,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":56,"value":1461},"defaultLakehouse",{"type":56,"value":1463}," in execution config (both ",{"type":43,"tag":83,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":56,"value":1469},"id",{"type":56,"value":633},{"type":43,"tag":83,"props":1472,"children":1474},{"className":1473},[],[1475],{"type":56,"value":1476},"name",{"type":56,"value":1478}," required)",{"type":43,"tag":72,"props":1480,"children":1481},{},[1482,1487,1489,1495],{"type":43,"tag":52,"props":1483,"children":1484},{},[1485],{"type":56,"value":1486},"Connect Power BI to Gold layer",{"type":56,"value":1488}," — discover the Gold lakehouse SQL endpoint, create a Direct Lake semantic model, create a report with visuals on the Gold summary table (see ",{"type":43,"tag":156,"props":1490,"children":1492},{"href":1491},"#gold-layer--power-bi-consumption",[1493],{"type":56,"value":1494},"Gold Layer → Power BI Consumption",{"type":56,"value":1382},{"type":43,"tag":72,"props":1497,"children":1498},{},[1499,1504],{"type":43,"tag":52,"props":1500,"children":1501},{},[1502],{"type":56,"value":1503},"Create pipeline",{"type":56,"value":1505}," to orchestrate the Bronze → Silver → Gold flow for recurring execution",{"type":43,"tag":445,"props":1507,"children":1509},{"id":1508},"explicit-override-single-workspace",[1510],{"type":56,"value":1511},"Explicit Override: Single Workspace",{"type":43,"tag":48,"props":1513,"children":1514},{},[1515],{"type":56,"value":1516},"If the user explicitly asks for a single workspace deployment (for example, POC\u002Fsmall team\u002Fmonolithic pattern), keep the current approach:",{"type":43,"tag":68,"props":1518,"children":1519},{},[1520,1525,1530],{"type":43,"tag":72,"props":1521,"children":1522},{},[1523],{"type":56,"value":1524},"One workspace with separate Bronze\u002FSilver\u002FGold lakehouses",{"type":43,"tag":72,"props":1526,"children":1527},{},[1528],{"type":56,"value":1529},"Preserve layer separation logically even when workspace is shared",{"type":43,"tag":72,"props":1531,"children":1532},{},[1533],{"type":56,"value":1534},"Call out governance trade-offs versus multi-workspace design",{"type":43,"tag":48,"props":1536,"children":1537},{},[1538,1540,1546,1547,1553],{"type":56,"value":1539},"Parameterize by environment: workspace name suffix (",{"type":43,"tag":83,"props":1541,"children":1543},{"className":1542},[],[1544],{"type":56,"value":1545},"-dev",{"type":56,"value":182},{"type":43,"tag":83,"props":1548,"children":1550},{"className":1549},[],[1551],{"type":56,"value":1552},"-prod",{"type":56,"value":1554},"), data volume (sample vs full), capacity SKU, and Bronze retention period.",{"type":43,"tag":256,"props":1556,"children":1557},{},[],{"type":43,"tag":138,"props":1559,"children":1561},{"id":1560},"bronze-layer-ingestion-patterns",[1562],{"type":56,"value":1563},"Bronze Layer — Ingestion Patterns",{"type":43,"tag":48,"props":1565,"children":1566},{},[1567],{"type":56,"value":1568},"When a user requests data ingestion into the Bronze layer, guide LLM to:",{"type":43,"tag":117,"props":1570,"children":1571},{},[1572,1668,1692,1702],{"type":43,"tag":72,"props":1573,"children":1574},{},[1575,1580,1582,1587,1589],{"type":43,"tag":52,"props":1576,"children":1577},{},[1578],{"type":56,"value":1579},"Land data in lakehouse first",{"type":56,"value":1581},": External data must be staged into the lakehouse ",{"type":43,"tag":83,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":56,"value":812},{"type":56,"value":1588}," folder before Spark can read it — use one of:\n",{"type":43,"tag":68,"props":1590,"children":1591},{},[1592,1602,1625,1635,1649],{"type":43,"tag":72,"props":1593,"children":1594},{},[1595,1600],{"type":43,"tag":52,"props":1596,"children":1597},{},[1598],{"type":56,"value":1599},"Fabric Pipeline Copy activity",{"type":56,"value":1601}," (preferred for recurring loads) — connects to external sources (HTTP, FTP, databases, cloud storage) and writes to OneLake",{"type":43,"tag":72,"props":1603,"children":1604},{},[1605,1615,1617,1623],{"type":43,"tag":52,"props":1606,"children":1607},{},[1608,1610],{"type":56,"value":1609},"OneLake API \u002F ",{"type":43,"tag":83,"props":1611,"children":1613},{"className":1612},[],[1614],{"type":56,"value":820},{"type":56,"value":1616}," — upload files via REST API using ",{"type":43,"tag":83,"props":1618,"children":1620},{"className":1619},[],[1621],{"type":56,"value":1622},"storage.azure.com",{"type":56,"value":1624}," token (see COMMON-CLI.md § OneLake Data Access)",{"type":43,"tag":72,"props":1626,"children":1627},{},[1628,1633],{"type":43,"tag":52,"props":1629,"children":1630},{},[1631],{"type":56,"value":1632},"OneLake Shortcut",{"type":56,"value":1634}," — for data already in Azure ADLS Gen2, S3, or another OneLake location",{"type":43,"tag":72,"props":1636,"children":1637},{},[1638,1647],{"type":43,"tag":52,"props":1639,"children":1640},{},[1641],{"type":43,"tag":83,"props":1642,"children":1644},{"className":1643},[],[1645],{"type":56,"value":1646},"notebookutils.fs",{"type":56,"value":1648}," — copy from mounted storage paths within a notebook",{"type":43,"tag":72,"props":1650,"children":1651},{},[1652,1654,1659,1660,1666],{"type":56,"value":1653},"⚠️ ",{"type":43,"tag":52,"props":1655,"children":1656},{},[1657],{"type":56,"value":1658},"Fabric Spark cannot read from arbitrary HTTP\u002FHTTPS URLs",{"type":56,"value":174},{"type":43,"tag":83,"props":1661,"children":1663},{"className":1662},[],[1664],{"type":56,"value":1665},"spark.read.format(\"csv\").load(\"https:\u002F\u002F...\")",{"type":56,"value":1667}," will fail",{"type":43,"tag":72,"props":1669,"children":1670},{},[1671,1676,1678,1683,1685,1691],{"type":43,"tag":52,"props":1672,"children":1673},{},[1674],{"type":56,"value":1675},"Read from lakehouse path",{"type":56,"value":1677},": Once data is in ",{"type":43,"tag":83,"props":1679,"children":1681},{"className":1680},[],[1682],{"type":56,"value":812},{"type":56,"value":1684},", read using lakehouse-relative paths (e.g., ",{"type":43,"tag":83,"props":1686,"children":1688},{"className":1687},[],[1689],{"type":56,"value":1690},"spark.read.format(\"csv\").load(\"Files\u002Flanding\u002Fdaily\u002F\")",{"type":56,"value":1382},{"type":43,"tag":72,"props":1693,"children":1694},{},[1695,1700],{"type":43,"tag":52,"props":1696,"children":1697},{},[1698],{"type":56,"value":1699},"Add metadata and write",{"type":56,"value":1701},": Tracking columns (ingestion timestamp, source file, batch ID), Delta table with descriptive name, partition by ingestion date, append mode",{"type":43,"tag":72,"props":1703,"children":1704},{},[1705,1710],{"type":43,"tag":52,"props":1706,"children":1707},{},[1708],{"type":56,"value":1709},"Validate",{"type":56,"value":1711},": Log row counts, validate schema structure, flag anomalies vs historical patterns",{"type":43,"tag":256,"props":1713,"children":1714},{},[],{"type":43,"tag":138,"props":1716,"children":1718},{"id":1717},"silver-layer-transformation-patterns",[1719],{"type":56,"value":1720},"Silver Layer — Transformation Patterns",{"type":43,"tag":48,"props":1722,"children":1723},{},[1724],{"type":56,"value":1725},"When a user requests Bronze-to-Silver transformation, guide LLM to:",{"type":43,"tag":68,"props":1727,"children":1728},{},[1729,1739,1749,1766],{"type":43,"tag":72,"props":1730,"children":1731},{},[1732,1737],{"type":43,"tag":52,"props":1733,"children":1734},{},[1735],{"type":56,"value":1736},"Quality rules",{"type":56,"value":1738},": Deduplicate on natural\u002Fcomposite key, filter invalid ranges, handle nulls (drop required, fill optional), validate logical constraints",{"type":43,"tag":72,"props":1740,"children":1741},{},[1742,1747],{"type":43,"tag":52,"props":1743,"children":1744},{},[1745],{"type":56,"value":1746},"Schema conformance",{"type":56,"value":1748},": snake_case column names, standardized data types, derived columns (durations, percentages, categories)",{"type":43,"tag":72,"props":1750,"children":1751},{},[1752,1757,1759,1764],{"type":43,"tag":52,"props":1753,"children":1754},{},[1755],{"type":56,"value":1756},"Schema evolution",{"type":56,"value":1758},": Use ",{"type":43,"tag":83,"props":1760,"children":1762},{"className":1761},[],[1763],{"type":56,"value":423},{"type":56,"value":1765}," option when source schemas change; coordinate downstream updates to Gold tables and Power BI datasets",{"type":43,"tag":72,"props":1767,"children":1768},{},[1769,1774],{"type":43,"tag":52,"props":1770,"children":1771},{},[1772],{"type":56,"value":1773},"Write strategy",{"type":56,"value":1775},": Partition by business date, partition-aware overwrite, run OPTIMIZE after write, log before\u002Fafter metrics",{"type":43,"tag":256,"props":1777,"children":1778},{},[],{"type":43,"tag":138,"props":1780,"children":1782},{"id":1781},"gold-layer-aggregation-patterns",[1783],{"type":56,"value":1784},"Gold Layer — Aggregation Patterns",{"type":43,"tag":48,"props":1786,"children":1787},{},[1788],{"type":56,"value":1789},"When a user requests Gold analytics tables, guide LLM to generate:",{"type":43,"tag":68,"props":1791,"children":1792},{},[1793,1803,1898],{"type":43,"tag":72,"props":1794,"children":1795},{},[1796,1801],{"type":43,"tag":52,"props":1797,"children":1798},{},[1799],{"type":56,"value":1800},"Common aggregates",{"type":56,"value":1802},": Daily\u002Fweekly\u002Fmonthly summaries, dimensional analysis (by location, category, type), trend breakdowns over time, demand patterns (hour-of-day, day-of-week)",{"type":43,"tag":72,"props":1804,"children":1805},{},[1806,1811,1813,1818,1820,1852],{"type":43,"tag":52,"props":1807,"children":1808},{},[1809],{"type":56,"value":1810},"Spark session config",{"type":56,"value":1812}," — set these properties in the Gold notebook ",{"type":43,"tag":52,"props":1814,"children":1815},{},[1816],{"type":56,"value":1817},"before",{"type":56,"value":1819}," any write operations:\n",{"type":43,"tag":1233,"props":1821,"children":1823},{"className":1235,"code":1822,"language":1237,"meta":1238,"style":1238},"spark.conf.set(\"spark.sql.parquet.vorder.default\", \"true\")\nspark.conf.set(\"spark.databricks.delta.optimizeWrite.enabled\", \"true\")\nspark.conf.set(\"spark.databricks.delta.optimizeWrite.binSize\", \"1g\")\n",[1824],{"type":43,"tag":83,"props":1825,"children":1826},{"__ignoreMap":1238},[1827,1835,1843],{"type":43,"tag":1244,"props":1828,"children":1829},{"class":1246,"line":1247},[1830],{"type":43,"tag":1244,"props":1831,"children":1832},{},[1833],{"type":56,"value":1834},"spark.conf.set(\"spark.sql.parquet.vorder.default\", \"true\")\n",{"type":43,"tag":1244,"props":1836,"children":1837},{"class":1246,"line":1256},[1838],{"type":43,"tag":1244,"props":1839,"children":1840},{},[1841],{"type":56,"value":1842},"spark.conf.set(\"spark.databricks.delta.optimizeWrite.enabled\", \"true\")\n",{"type":43,"tag":1244,"props":1844,"children":1846},{"class":1246,"line":1845},3,[1847],{"type":43,"tag":1244,"props":1848,"children":1849},{},[1850],{"type":56,"value":1851},"spark.conf.set(\"spark.databricks.delta.optimizeWrite.binSize\", \"1g\")\n",{"type":43,"tag":68,"props":1853,"children":1854},{},[1855,1873],{"type":43,"tag":72,"props":1856,"children":1857},{},[1858,1863,1865,1871],{"type":43,"tag":52,"props":1859,"children":1860},{},[1861],{"type":56,"value":1862},"V-Order",{"type":56,"value":1864}," (",{"type":43,"tag":83,"props":1866,"children":1868},{"className":1867},[],[1869],{"type":56,"value":1870},"vorder.default",{"type":56,"value":1872},") — applies Fabric's columnar sort optimization to all Parquet files, dramatically improving Direct Lake and SQL endpoint read performance",{"type":43,"tag":72,"props":1874,"children":1875},{},[1876,1881,1882,1888,1890,1896],{"type":43,"tag":52,"props":1877,"children":1878},{},[1879],{"type":56,"value":1880},"Optimize Write",{"type":56,"value":1864},{"type":43,"tag":83,"props":1883,"children":1885},{"className":1884},[],[1886],{"type":56,"value":1887},"optimizeWrite.enabled",{"type":56,"value":1889},") — coalesces small partitions into optimally-sized files (target ~1 GB per ",{"type":43,"tag":83,"props":1891,"children":1893},{"className":1892},[],[1894],{"type":56,"value":1895},"binSize",{"type":56,"value":1897},"), reducing file count and improving scan efficiency",{"type":43,"tag":72,"props":1899,"children":1900},{},[1901,1906],{"type":43,"tag":52,"props":1902,"children":1903},{},[1904],{"type":56,"value":1905},"Optimization",{"type":56,"value":1907},": ZORDER on filter columns, run OPTIMIZE after writes, pre-aggregate metrics to avoid runtime computation",{"type":43,"tag":256,"props":1909,"children":1910},{},[],{"type":43,"tag":138,"props":1912,"children":1914},{"id":1913},"end-to-end-execution-flow",[1915],{"type":56,"value":1916},"End-to-End Execution Flow",{"type":43,"tag":48,"props":1918,"children":1919},{},[1920,1922,1927],{"type":56,"value":1921},"When setting up medallion architecture end-to-end, the LLM ",{"type":43,"tag":52,"props":1923,"children":1924},{},[1925],{"type":56,"value":1926},"must not stop",{"type":56,"value":1928}," after creating notebooks and deploying code. The complete lifecycle is:",{"type":43,"tag":1233,"props":1930,"children":1934},{"className":1931,"code":1933,"language":56},[1932],"language-text","Create Resources → Deploy Content → Bind Lakehouses → Execute → Verify Results\n",[1935],{"type":43,"tag":83,"props":1936,"children":1937},{"__ignoreMap":1238},[1938],{"type":56,"value":1933},{"type":43,"tag":445,"props":1940,"children":1942},{"id":1941},"step-by-step",[1943],{"type":56,"value":1944},"Step-by-Step",{"type":43,"tag":117,"props":1946,"children":1947},{},[1948,1958,1979,2039,2062,2093,2160,2170],{"type":43,"tag":72,"props":1949,"children":1950},{},[1951,1956],{"type":43,"tag":52,"props":1952,"children":1953},{},[1954],{"type":56,"value":1955},"Create layer workspaces and lakehouses (default)",{"type":56,"value":1957}," — one workspace and one lakehouse per layer (Bronze, Silver, Gold); capture workspace IDs and lakehouse IDs",{"type":43,"tag":72,"props":1959,"children":1960},{},[1961,1965,1967,1972,1974,1978],{"type":43,"tag":52,"props":1962,"children":1963},{},[1964],{"type":56,"value":1326},{"type":56,"value":1966}," — one per layer, with valid ",{"type":43,"tag":83,"props":1968,"children":1970},{"className":1969},[],[1971],{"type":56,"value":223},{"type":56,"value":1973}," structure (see ",{"type":43,"tag":156,"props":1975,"children":1976},{"href":207},[1977],{"type":56,"value":210},{"type":56,"value":1382},{"type":43,"tag":72,"props":1980,"children":1981},{},[1982,1987,1989,1994,1996,2001,2003],{"type":43,"tag":52,"props":1983,"children":1984},{},[1985],{"type":56,"value":1986},"Bind lakehouse to each notebook",{"type":56,"value":1988}," — include ",{"type":43,"tag":83,"props":1990,"children":1992},{"className":1991},[],[1993],{"type":56,"value":1350},{"type":56,"value":1995}," in the ",{"type":43,"tag":83,"props":1997,"children":1999},{"className":1998},[],[2000],{"type":56,"value":223},{"type":56,"value":2002}," payload with:\n",{"type":43,"tag":68,"props":2004,"children":2005},{},[2006,2017,2028],{"type":43,"tag":72,"props":2007,"children":2008},{},[2009,2015],{"type":43,"tag":83,"props":2010,"children":2012},{"className":2011},[],[2013],{"type":56,"value":2014},"default_lakehouse",{"type":56,"value":2016},": the target lakehouse GUID",{"type":43,"tag":72,"props":2018,"children":2019},{},[2020,2026],{"type":43,"tag":83,"props":2021,"children":2023},{"className":2022},[],[2024],{"type":56,"value":2025},"default_lakehouse_name",{"type":56,"value":2027},": the lakehouse display name",{"type":43,"tag":72,"props":2029,"children":2030},{},[2031,2037],{"type":43,"tag":83,"props":2032,"children":2034},{"className":2033},[],[2035],{"type":56,"value":2036},"default_lakehouse_workspace_id",{"type":56,"value":2038},": the workspace GUID",{"type":43,"tag":72,"props":2040,"children":2041},{},[2042,2047,2048,2053,2055,2060],{"type":43,"tag":52,"props":2043,"children":2044},{},[2045],{"type":56,"value":2046},"Deploy notebook content",{"type":56,"value":174},{"type":43,"tag":83,"props":2049,"children":2051},{"className":2050},[],[2052],{"type":56,"value":239},{"type":56,"value":2054}," with the Base64-encoded ",{"type":43,"tag":83,"props":2056,"children":2058},{"className":2057},[],[2059],{"type":56,"value":223},{"type":56,"value":2061}," payload (content + lakehouse binding together)",{"type":43,"tag":72,"props":2063,"children":2064},{},[2065,2070,2072,2077,2079,2084,2086,2091],{"type":43,"tag":52,"props":2066,"children":2067},{},[2068],{"type":56,"value":2069},"Confirm deployment",{"type":56,"value":2071}," — check that each ",{"type":43,"tag":83,"props":2073,"children":2075},{"className":2074},[],[2076],{"type":56,"value":239},{"type":56,"value":2078}," LRO returned ",{"type":43,"tag":83,"props":2080,"children":2082},{"className":2081},[],[2083],{"type":56,"value":1428},{"type":56,"value":2085},"; that is sufficient. Do NOT call ",{"type":43,"tag":83,"props":2087,"children":2089},{"className":2088},[],[2090],{"type":56,"value":231},{"type":56,"value":2092}," to re-verify — it is an async LRO and adds significant latency per notebook.",{"type":43,"tag":72,"props":2094,"children":2095},{},[2096,2101,2103,2108,2109],{"type":43,"tag":52,"props":2097,"children":2098},{},[2099],{"type":56,"value":2100},"Execute notebooks sequentially",{"type":56,"value":2102}," — use ",{"type":43,"tag":83,"props":2104,"children":2106},{"className":2105},[],[2107],{"type":56,"value":1453},{"type":56,"value":928},{"type":43,"tag":68,"props":2110,"children":2111},{},[2112,2142,2155],{"type":43,"tag":72,"props":2113,"children":2114},{},[2115,2117,2122,2124,2129,2130,2135,2136],{"type":56,"value":2116},"Pass ",{"type":43,"tag":83,"props":2118,"children":2120},{"className":2119},[],[2121],{"type":56,"value":1461},{"type":56,"value":2123}," with both ",{"type":43,"tag":83,"props":2125,"children":2127},{"className":2126},[],[2128],{"type":56,"value":1469},{"type":56,"value":633},{"type":43,"tag":83,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":56,"value":1476},{"type":56,"value":618},{"type":43,"tag":83,"props":2137,"children":2139},{"className":2138},[],[2140],{"type":56,"value":2141},"executionData.configuration",{"type":43,"tag":72,"props":2143,"children":2144},{},[2145,2147,2153],{"type":56,"value":2146},"Run Bronze first → poll until ",{"type":43,"tag":83,"props":2148,"children":2150},{"className":2149},[],[2151],{"type":56,"value":2152},"Completed",{"type":56,"value":2154}," → run Silver → poll → run Gold → poll",{"type":43,"tag":72,"props":2156,"children":2157},{},[2158],{"type":56,"value":2159},"Check for recent jobs before submitting (prevent duplicates — see SPARK-AUTHORING-CORE.md)",{"type":43,"tag":72,"props":2161,"children":2162},{},[2163,2168],{"type":43,"tag":52,"props":2164,"children":2165},{},[2166],{"type":56,"value":2167},"Verify results",{"type":56,"value":2169}," — after each notebook completes, confirm expected tables exist and row counts are reasonable",{"type":43,"tag":72,"props":2171,"children":2172},{},[2173,2178,2180,2184],{"type":43,"tag":52,"props":2174,"children":2175},{},[2176],{"type":56,"value":2177},"Connect Power BI to Gold",{"type":56,"value":2179}," — create semantic model + report on Gold summary tables (see ",{"type":43,"tag":156,"props":2181,"children":2182},{"href":1491},[2183],{"type":56,"value":1494},{"type":56,"value":1382},{"type":43,"tag":445,"props":2186,"children":2188},{"id":2187},"common-failure-stopping-after-notebook-creation",[2189],{"type":56,"value":2190},"Common Failure: Stopping After Notebook Creation",{"type":43,"tag":48,"props":2192,"children":2193},{},[2194],{"type":56,"value":2195},"If the flow stops after deploying notebook code without binding or executing:",{"type":43,"tag":68,"props":2197,"children":2198},{},[2199,2226,2231],{"type":43,"tag":72,"props":2200,"children":2201},{},[2202,2204,2210,2212,2218,2219,2224],{"type":56,"value":2203},"Notebooks will have no lakehouse context → ",{"type":43,"tag":83,"props":2205,"children":2207},{"className":2206},[],[2208],{"type":56,"value":2209},"spark.sql()",{"type":56,"value":2211}," and relative paths (",{"type":43,"tag":83,"props":2213,"children":2215},{"className":2214},[],[2216],{"type":56,"value":2217},"Tables\u002F",{"type":56,"value":182},{"type":43,"tag":83,"props":2220,"children":2222},{"className":2221},[],[2223],{"type":56,"value":812},{"type":56,"value":2225},") fail at runtime",{"type":43,"tag":72,"props":2227,"children":2228},{},[2229],{"type":56,"value":2230},"The user sees no output or results — the architecture is set up but never tested",{"type":43,"tag":72,"props":2232,"children":2233},{},[2234,2239],{"type":43,"tag":52,"props":2235,"children":2236},{},[2237],{"type":56,"value":2238},"Always complete through step 7",{"type":56,"value":2240}," unless the user explicitly asks to stop at a specific step",{"type":43,"tag":256,"props":2242,"children":2243},{},[],{"type":43,"tag":138,"props":2245,"children":2247},{"id":2246},"gold-layer-power-bi-consumption",[2248],{"type":56,"value":1494},{"type":43,"tag":48,"props":2250,"children":2251},{},[2252],{"type":56,"value":2253},"After Gold tables are populated, connect Power BI to surface the analytics.\nBuild a semantic model on top of the Gold lakehouse, using DirectLake.",{"type":43,"tag":445,"props":2255,"children":2257},{"id":2256},"step-by-step-1",[2258],{"type":56,"value":1944},{"type":43,"tag":117,"props":2260,"children":2261},{},[2262,2301,2343,2455,2515],{"type":43,"tag":72,"props":2263,"children":2264},{},[2265,2270,2272,2278,2280,2286,2287,2293,2295],{"type":43,"tag":52,"props":2266,"children":2267},{},[2268],{"type":56,"value":2269},"Discover the Gold lakehouse SQL endpoint",{"type":56,"value":2271}," — call ",{"type":43,"tag":83,"props":2273,"children":2275},{"className":2274},[],[2276],{"type":56,"value":2277},"GET \u002Fv1\u002Fworkspaces\u002F{workspaceId}\u002Flakehouses\u002F{goldLakehouseId}",{"type":56,"value":2279}," and extract ",{"type":43,"tag":83,"props":2281,"children":2283},{"className":2282},[],[2284],{"type":56,"value":2285},"properties.sqlEndpointProperties.connectionString",{"type":56,"value":633},{"type":43,"tag":83,"props":2288,"children":2290},{"className":2289},[],[2291],{"type":56,"value":2292},"provisioningStatus",{"type":56,"value":2294},"; wait until status is ",{"type":43,"tag":83,"props":2296,"children":2298},{"className":2297},[],[2299],{"type":56,"value":2300},"Success",{"type":43,"tag":72,"props":2302,"children":2303},{},[2304,2309,2311,2317,2319,2325,2327],{"type":43,"tag":52,"props":2305,"children":2306},{},[2307],{"type":56,"value":2308},"Verify Gold tables via SQL",{"type":56,"value":2310}," — connect to the SQL endpoint using ",{"type":43,"tag":83,"props":2312,"children":2314},{"className":2313},[],[2315],{"type":56,"value":2316},"sqlcmd",{"type":56,"value":2318}," (see ",{"type":43,"tag":156,"props":2320,"children":2322},{"href":2321},"..\u002F..\u002Fcommon\u002FCOMMON-CLI.md#sql--tds-data-plane-access",[2323],{"type":56,"value":2324},"COMMON-CLI.md § SQL \u002F TDS Data-Plane Access",{"type":56,"value":2326},") and confirm the target table exists:\n",{"type":43,"tag":1233,"props":2328,"children":2332},{"className":2329,"code":2330,"language":2331,"meta":1238,"style":1238},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 'nyc_taxi_daily_summary'\n","sql",[2333],{"type":43,"tag":83,"props":2334,"children":2335},{"__ignoreMap":1238},[2336],{"type":43,"tag":1244,"props":2337,"children":2338},{"class":1246,"line":1247},[2339],{"type":43,"tag":1244,"props":2340,"children":2341},{},[2342],{"type":56,"value":2330},{"type":43,"tag":72,"props":2344,"children":2345},{},[2346,2351,2353,2359,2361,2367,2369,2375,2377,2382,2384,2390,2391],{"type":43,"tag":52,"props":2347,"children":2348},{},[2349],{"type":56,"value":2350},"Create a semantic model",{"type":56,"value":2352}," — use the ",{"type":43,"tag":156,"props":2354,"children":2356},{"href":2355},"..\u002Fsemantic-model-authoring\u002FSKILL.md",[2357],{"type":56,"value":2358},"semantic-model-authoring",{"type":56,"value":2360}," skill for semantic model creation and TMDL deployment. Create via ",{"type":43,"tag":83,"props":2362,"children":2364},{"className":2363},[],[2365],{"type":56,"value":2366},"POST \u002Fv1\u002Fworkspaces\u002F{workspaceId}\u002Fitems",{"type":56,"value":2368}," with ",{"type":43,"tag":83,"props":2370,"children":2372},{"className":2371},[],[2373],{"type":56,"value":2374},"type: \"SemanticModel\"",{"type":56,"value":2376}," then deploy definition via ",{"type":43,"tag":83,"props":2378,"children":2380},{"className":2379},[],[2381],{"type":56,"value":239},{"type":56,"value":2383}," using TMDL format (see ",{"type":43,"tag":156,"props":2385,"children":2387},{"href":2386},"..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#semanticmodel",[2388],{"type":56,"value":2389},"ITEM-DEFINITIONS-CORE.md § SemanticModel",{"type":56,"value":471},{"type":43,"tag":68,"props":2392,"children":2393},{},[2394,2399,2411,2422],{"type":43,"tag":72,"props":2395,"children":2396},{},[2397],{"type":56,"value":2398},"The model must reference the Gold lakehouse SQL endpoint as its data source",{"type":43,"tag":72,"props":2400,"children":2401},{},[2402,2404,2410],{"type":56,"value":2403},"Define a table mapping to the Gold summary table (e.g., ",{"type":43,"tag":83,"props":2405,"children":2407},{"className":2406},[],[2408],{"type":56,"value":2409},"nyc_taxi_daily_summary",{"type":56,"value":1382},{"type":43,"tag":72,"props":2412,"children":2413},{},[2414,2415,2420],{"type":56,"value":571},{"type":43,"tag":52,"props":2416,"children":2417},{},[2418],{"type":56,"value":2419},"Direct Lake",{"type":56,"value":2421}," mode — this connects directly to Delta tables in OneLake without data import",{"type":43,"tag":72,"props":2423,"children":2424},{},[2425,2427,2433,2434,2440,2441,2447,2448,2454],{"type":56,"value":2426},"Include measures for key aggregations you find interesting (e.g., ",{"type":43,"tag":83,"props":2428,"children":2430},{"className":2429},[],[2431],{"type":56,"value":2432},"Total Trips",{"type":56,"value":182},{"type":43,"tag":83,"props":2435,"children":2437},{"className":2436},[],[2438],{"type":56,"value":2439},"Avg Fare",{"type":56,"value":182},{"type":43,"tag":83,"props":2442,"children":2444},{"className":2443},[],[2445],{"type":56,"value":2446},"Total Revenue",{"type":56,"value":182},{"type":43,"tag":83,"props":2449,"children":2451},{"className":2450},[],[2452],{"type":56,"value":2453},"Month over Month Growth",{"type":56,"value":1382},{"type":43,"tag":72,"props":2456,"children":2457},{},[2458,2463,2464,2469,2470,2476,2477,2482,2484,2490,2491],{"type":43,"tag":52,"props":2459,"children":2460},{},[2461],{"type":56,"value":2462},"Create a Power BI report",{"type":56,"value":174},{"type":43,"tag":83,"props":2465,"children":2467},{"className":2466},[],[2468],{"type":56,"value":2366},{"type":56,"value":2368},{"type":43,"tag":83,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":56,"value":2475},"type: \"Report\"",{"type":56,"value":2376},{"type":43,"tag":83,"props":2478,"children":2480},{"className":2479},[],[2481],{"type":56,"value":239},{"type":56,"value":2483}," using PBIR format (see ",{"type":43,"tag":156,"props":2485,"children":2487},{"href":2486},"..\u002F..\u002Fcommon\u002FITEM-DEFINITIONS-CORE.md#report",[2488],{"type":56,"value":2489},"ITEM-DEFINITIONS-CORE.md § Report",{"type":56,"value":471},{"type":43,"tag":68,"props":2492,"children":2493},{},[2494,2505,2510],{"type":43,"tag":72,"props":2495,"children":2496},{},[2497,2499],{"type":56,"value":2498},"Reference the semantic model created in step 3 via ",{"type":43,"tag":83,"props":2500,"children":2502},{"className":2501},[],[2503],{"type":56,"value":2504},"definition.pbir",{"type":43,"tag":72,"props":2506,"children":2507},{},[2508],{"type":56,"value":2509},"Define at least one page with visuals on the Gold summary table",{"type":43,"tag":72,"props":2511,"children":2512},{},[2513],{"type":56,"value":2514},"Suggested visuals: line chart (daily trend), card (KPI totals), bar chart (by category), table (detail view)",{"type":43,"tag":72,"props":2516,"children":2517},{},[2518,2523,2524,2529],{"type":43,"tag":52,"props":2519,"children":2520},{},[2521],{"type":56,"value":2522},"Verify end-to-end",{"type":56,"value":2352},{"type":43,"tag":83,"props":2525,"children":2527},{"className":2526},[],[2528],{"type":56,"value":2358},{"type":56,"value":2530}," skill for metadata discovery running DAX queries against the semantic model and confirm data flows from Gold tables through to the report",{"type":43,"tag":445,"props":2532,"children":2534},{"id":2533},"principles",[2535],{"type":56,"value":2536},"Principles",{"type":43,"tag":68,"props":2538,"children":2539},{},[2540,2557,2574,2584,2594,2610],{"type":43,"tag":72,"props":2541,"children":2542},{},[2543,2548,2550,2555],{"type":43,"tag":52,"props":2544,"children":2545},{},[2546],{"type":56,"value":2547},"Discover SQL endpoint dynamically",{"type":56,"value":2549}," — the connection string is in ",{"type":43,"tag":83,"props":2551,"children":2553},{"className":2552},[],[2554],{"type":56,"value":2285},{"type":56,"value":2556}," on the lakehouse response; never hardcode it",{"type":43,"tag":72,"props":2558,"children":2559},{},[2560,2565,2567,2572],{"type":43,"tag":52,"props":2561,"children":2562},{},[2563],{"type":56,"value":2564},"Wait for SQL endpoint provisioning",{"type":56,"value":2566}," — status must be ",{"type":43,"tag":83,"props":2568,"children":2570},{"className":2569},[],[2571],{"type":56,"value":2300},{"type":56,"value":2573}," before connecting; newly created lakehouses may take minutes to provision",{"type":43,"tag":72,"props":2575,"children":2576},{},[2577,2582],{"type":43,"tag":52,"props":2578,"children":2579},{},[2580],{"type":56,"value":2581},"Prefer Direct Lake mode",{"type":56,"value":2583}," — avoids data duplication; semantic model reads directly from OneLake Delta tables",{"type":43,"tag":72,"props":2585,"children":2586},{},[2587,2592],{"type":43,"tag":52,"props":2588,"children":2589},{},[2590],{"type":56,"value":2591},"Match table\u002Fcolumn names exactly",{"type":56,"value":2593}," — the semantic model table definition must use the exact Delta table and column names from the Gold lakehouse",{"type":43,"tag":72,"props":2595,"children":2596},{},[2597,2602,2604,2608],{"type":43,"tag":52,"props":2598,"children":2599},{},[2600],{"type":56,"value":2601},"For semantic model authoring",{"type":56,"value":2603}," (TMDL, refresh, permissions), cross-reference the ",{"type":43,"tag":156,"props":2605,"children":2606},{"href":2355},[2607],{"type":56,"value":2358},{"type":56,"value":2609}," skill",{"type":43,"tag":72,"props":2611,"children":2612},{},[2613,2618,2620,2624],{"type":43,"tag":52,"props":2614,"children":2615},{},[2616],{"type":56,"value":2617},"For DAX query validation",{"type":56,"value":2619},", cross-reference the ",{"type":43,"tag":156,"props":2621,"children":2622},{"href":2355},[2623],{"type":56,"value":2358},{"type":56,"value":2625}," skill for metadata discovery and DAX queries for validation.",{"type":43,"tag":256,"props":2627,"children":2628},{},[],{"type":43,"tag":138,"props":2630,"children":2632},{"id":2631},"pipeline-orchestration",[2633],{"type":56,"value":2634},"Pipeline Orchestration",{"type":43,"tag":48,"props":2636,"children":2637},{},[2638],{"type":56,"value":2639},"When a user requests a pipeline for the medallion flow, guide LLM to design with:",{"type":43,"tag":68,"props":2641,"children":2642},{},[2643,2653,2663,2673],{"type":43,"tag":72,"props":2644,"children":2645},{},[2646,2651],{"type":43,"tag":52,"props":2647,"children":2648},{},[2649],{"type":56,"value":2650},"Structure",{"type":56,"value":2652},": Sequential activities (Bronze → Silver → Gold), each waiting for previous success; independent Gold aggregations can run in parallel; include validation and notification activities",{"type":43,"tag":72,"props":2654,"children":2655},{},[2656,2661],{"type":43,"tag":52,"props":2657,"children":2658},{},[2659],{"type":56,"value":2660},"Parameterization",{"type":56,"value":2662},": Pipeline-level processing date (defaults to yesterday), passed to all notebooks; dynamic date expressions",{"type":43,"tag":72,"props":2664,"children":2665},{},[2666,2671],{"type":43,"tag":52,"props":2667,"children":2668},{},[2669],{"type":56,"value":2670},"Scheduling",{"type":56,"value":2672},": Daily aligned with source refresh, watermark-based incremental processing, periodic full refresh for corrections",{"type":43,"tag":72,"props":2674,"children":2675},{},[2676,2681],{"type":43,"tag":52,"props":2677,"children":2678},{},[2679],{"type":56,"value":2680},"Error handling",{"type":56,"value":2682},": Retry with backoff for transient failures, alerting for persistent failures, graceful degradation (downstream uses previous data if upstream fails)",{"type":43,"tag":256,"props":2684,"children":2685},{},[],{"type":43,"tag":138,"props":2687,"children":2689},{"id":2688},"environment-optimization",[2690],{"type":56,"value":2691},"Environment Optimization",{"type":43,"tag":48,"props":2693,"children":2694},{},[2695],{"type":43,"tag":52,"props":2696,"children":2697},{},[2698,2700,2704],{"type":56,"value":2699},"For detailed Spark configurations and optimization strategies, see ",{"type":43,"tag":156,"props":2701,"children":2702},{"href":249},[2703],{"type":56,"value":252},{"type":56,"value":254},{"type":43,"tag":276,"props":2706,"children":2707},{},[2708,2728],{"type":43,"tag":280,"props":2709,"children":2710},{},[2711],{"type":43,"tag":284,"props":2712,"children":2713},{},[2714,2718,2723],{"type":43,"tag":288,"props":2715,"children":2716},{},[2717],{"type":56,"value":292},{"type":43,"tag":288,"props":2719,"children":2720},{},[2721],{"type":56,"value":2722},"Profile",{"type":43,"tag":288,"props":2724,"children":2725},{},[2726],{"type":56,"value":2727},"Key Settings",{"type":43,"tag":309,"props":2729,"children":2730},{},[2731,2748,2765],{"type":43,"tag":284,"props":2732,"children":2733},{},[2734,2738,2743],{"type":43,"tag":316,"props":2735,"children":2736},{},[2737],{"type":56,"value":323},{"type":43,"tag":316,"props":2739,"children":2740},{},[2741],{"type":56,"value":2742},"Write-heavy",{"type":43,"tag":316,"props":2744,"children":2745},{},[2746],{"type":56,"value":2747},"Disable V-Order, enable autoCompact, large file targets, partition by ingestion_date",{"type":43,"tag":284,"props":2749,"children":2750},{},[2751,2755,2760],{"type":43,"tag":316,"props":2752,"children":2753},{},[2754],{"type":56,"value":351},{"type":43,"tag":316,"props":2756,"children":2757},{},[2758],{"type":56,"value":2759},"Balanced",{"type":43,"tag":316,"props":2761,"children":2762},{},[2763],{"type":56,"value":2764},"Enable V-Order, adaptive query execution, partition by business date, ZORDER on filtered columns",{"type":43,"tag":284,"props":2766,"children":2767},{},[2768,2772,2777],{"type":43,"tag":316,"props":2769,"children":2770},{},[2771],{"type":56,"value":379},{"type":43,"tag":316,"props":2773,"children":2774},{},[2775],{"type":56,"value":2776},"Read-heavy",{"type":43,"tag":316,"props":2778,"children":2779},{},[2780,2782,2788,2790,2796,2797,2803],{"type":56,"value":2781},"V-Order (",{"type":43,"tag":83,"props":2783,"children":2785},{"className":2784},[],[2786],{"type":56,"value":2787},"spark.sql.parquet.vorder.default=true",{"type":56,"value":2789},"), Optimize Write (",{"type":43,"tag":83,"props":2791,"children":2793},{"className":2792},[],[2794],{"type":56,"value":2795},"optimizeWrite.enabled=true",{"type":56,"value":182},{"type":43,"tag":83,"props":2798,"children":2800},{"className":2799},[],[2801],{"type":56,"value":2802},"binSize=1g",{"type":56,"value":2804},"), vectorized readers, adaptive execution, ZORDER on all filter columns, pre-aggregate metrics",{"type":43,"tag":256,"props":2806,"children":2807},{},[],{"type":43,"tag":138,"props":2809,"children":2811},{"id":2810},"examples",[2812],{"type":56,"value":2813},"Examples",{"type":43,"tag":445,"props":2815,"children":2817},{"id":2816},"example-1-set-up-medallion-workspaces-default",[2818],{"type":56,"value":2819},"Example 1: Set Up Medallion Workspaces (Default)",{"type":43,"tag":48,"props":2821,"children":2822},{},[2823,2828],{"type":43,"tag":52,"props":2824,"children":2825},{},[2826],{"type":56,"value":2827},"Prompt",{"type":56,"value":2829},": \"Set up medallion architecture with separate Bronze, Silver, and Gold workspaces for sales analytics\"",{"type":43,"tag":48,"props":2831,"children":2832},{},[2833,2838],{"type":43,"tag":52,"props":2834,"children":2835},{},[2836],{"type":56,"value":2837},"What the LLM should generate",{"type":56,"value":2839},": REST API calls to:",{"type":43,"tag":117,"props":2841,"children":2842},{},[2843,2868,2893],{"type":43,"tag":72,"props":2844,"children":2845},{},[2846,2848,2854,2855,2861,2862],{"type":56,"value":2847},"Create workspaces: ",{"type":43,"tag":83,"props":2849,"children":2851},{"className":2850},[],[2852],{"type":56,"value":2853},"sales-bronze-dev",{"type":56,"value":182},{"type":43,"tag":83,"props":2856,"children":2858},{"className":2857},[],[2859],{"type":56,"value":2860},"sales-silver-dev",{"type":56,"value":182},{"type":43,"tag":83,"props":2863,"children":2865},{"className":2864},[],[2866],{"type":56,"value":2867},"sales-gold-dev",{"type":43,"tag":72,"props":2869,"children":2870},{},[2871,2873,2879,2880,2886,2887],{"type":56,"value":2872},"Create one lakehouse in each workspace: ",{"type":43,"tag":83,"props":2874,"children":2876},{"className":2875},[],[2877],{"type":56,"value":2878},"sales_bronze",{"type":56,"value":182},{"type":43,"tag":83,"props":2881,"children":2883},{"className":2882},[],[2884],{"type":56,"value":2885},"sales_silver",{"type":56,"value":182},{"type":43,"tag":83,"props":2888,"children":2890},{"className":2889},[],[2891],{"type":56,"value":2892},"sales_gold",{"type":43,"tag":72,"props":2894,"children":2895},{},[2896],{"type":56,"value":2897},"Assign RBAC roles per workspace\u002Flayer",{"type":43,"tag":1233,"props":2899,"children":2903},{"className":2900,"code":2901,"language":2902,"meta":1238,"style":1238},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Workspace creation (see COMMON-CLI.md for full patterns)\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"sales-analytics-dev\"}\nEOF\nworkspace_id=$(az rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\" \\\n  --body @\u002Ftmp\u002Fbody.json --query \"id\" --output tsv)\n\n# Create Bronze lakehouse\ncat > \u002Ftmp\u002Fbody.json \u003C\u003C 'EOF'\n{\"displayName\": \"sales_bronze\", \"type\": \"Lakehouse\"}\nEOF\naz rest --method post --resource \"https:\u002F\u002Fapi.fabric.microsoft.com\" \\\n  --url \"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F$workspace_id\u002Fitems\" \\\n  --body @\u002Ftmp\u002Fbody.json\n","bash",[2904],{"type":43,"tag":83,"props":2905,"children":2906},{"__ignoreMap":1238},[2907,2916,2947,2955,2964,3024,3050,3096,3106,3115,3139,3148,3156,3196,3231],{"type":43,"tag":1244,"props":2908,"children":2909},{"class":1246,"line":1247},[2910],{"type":43,"tag":1244,"props":2911,"children":2913},{"style":2912},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[2914],{"type":56,"value":2915},"# Workspace creation (see COMMON-CLI.md for full patterns)\n",{"type":43,"tag":1244,"props":2917,"children":2918},{"class":1246,"line":1256},[2919,2925,2931,2937,2942],{"type":43,"tag":1244,"props":2920,"children":2922},{"style":2921},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[2923],{"type":56,"value":2924},"cat",{"type":43,"tag":1244,"props":2926,"children":2928},{"style":2927},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[2929],{"type":56,"value":2930}," >",{"type":43,"tag":1244,"props":2932,"children":2934},{"style":2933},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[2935],{"type":56,"value":2936}," \u002Ftmp\u002Fbody.json",{"type":43,"tag":1244,"props":2938,"children":2939},{"style":2927},[2940],{"type":56,"value":2941}," \u003C\u003C",{"type":43,"tag":1244,"props":2943,"children":2944},{"style":2927},[2945],{"type":56,"value":2946}," 'EOF'\n",{"type":43,"tag":1244,"props":2948,"children":2949},{"class":1246,"line":1845},[2950],{"type":43,"tag":1244,"props":2951,"children":2952},{"style":2933},[2953],{"type":56,"value":2954},"{\"displayName\": \"sales-analytics-dev\"}\n",{"type":43,"tag":1244,"props":2956,"children":2958},{"class":1246,"line":2957},4,[2959],{"type":43,"tag":1244,"props":2960,"children":2961},{"style":2927},[2962],{"type":56,"value":2963},"EOF\n",{"type":43,"tag":1244,"props":2965,"children":2967},{"class":1246,"line":2966},5,[2968,2974,2979,2984,2989,2994,2999,3004,3009,3014,3019],{"type":43,"tag":1244,"props":2969,"children":2971},{"style":2970},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[2972],{"type":56,"value":2973},"workspace_id",{"type":43,"tag":1244,"props":2975,"children":2976},{"style":2927},[2977],{"type":56,"value":2978},"=$(",{"type":43,"tag":1244,"props":2980,"children":2981},{"style":2921},[2982],{"type":56,"value":2983},"az",{"type":43,"tag":1244,"props":2985,"children":2986},{"style":2933},[2987],{"type":56,"value":2988}," rest",{"type":43,"tag":1244,"props":2990,"children":2991},{"style":2933},[2992],{"type":56,"value":2993}," --method",{"type":43,"tag":1244,"props":2995,"children":2996},{"style":2933},[2997],{"type":56,"value":2998}," post",{"type":43,"tag":1244,"props":3000,"children":3001},{"style":2933},[3002],{"type":56,"value":3003}," --resource",{"type":43,"tag":1244,"props":3005,"children":3006},{"style":2927},[3007],{"type":56,"value":3008}," \"",{"type":43,"tag":1244,"props":3010,"children":3011},{"style":2933},[3012],{"type":56,"value":3013},"https:\u002F\u002Fapi.fabric.microsoft.com",{"type":43,"tag":1244,"props":3015,"children":3016},{"style":2927},[3017],{"type":56,"value":3018},"\"",{"type":43,"tag":1244,"props":3020,"children":3021},{"style":2970},[3022],{"type":56,"value":3023}," \\\n",{"type":43,"tag":1244,"props":3025,"children":3027},{"class":1246,"line":3026},6,[3028,3033,3037,3042,3046],{"type":43,"tag":1244,"props":3029,"children":3030},{"style":2933},[3031],{"type":56,"value":3032},"  --url",{"type":43,"tag":1244,"props":3034,"children":3035},{"style":2927},[3036],{"type":56,"value":3008},{"type":43,"tag":1244,"props":3038,"children":3039},{"style":2933},[3040],{"type":56,"value":3041},"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces",{"type":43,"tag":1244,"props":3043,"children":3044},{"style":2927},[3045],{"type":56,"value":3018},{"type":43,"tag":1244,"props":3047,"children":3048},{"style":2970},[3049],{"type":56,"value":3023},{"type":43,"tag":1244,"props":3051,"children":3053},{"class":1246,"line":3052},7,[3054,3059,3064,3069,3073,3077,3081,3086,3091],{"type":43,"tag":1244,"props":3055,"children":3056},{"style":2933},[3057],{"type":56,"value":3058},"  --body",{"type":43,"tag":1244,"props":3060,"children":3061},{"style":2933},[3062],{"type":56,"value":3063}," @\u002Ftmp\u002Fbody.json",{"type":43,"tag":1244,"props":3065,"children":3066},{"style":2933},[3067],{"type":56,"value":3068}," --query",{"type":43,"tag":1244,"props":3070,"children":3071},{"style":2927},[3072],{"type":56,"value":3008},{"type":43,"tag":1244,"props":3074,"children":3075},{"style":2933},[3076],{"type":56,"value":1469},{"type":43,"tag":1244,"props":3078,"children":3079},{"style":2927},[3080],{"type":56,"value":3018},{"type":43,"tag":1244,"props":3082,"children":3083},{"style":2933},[3084],{"type":56,"value":3085}," --output",{"type":43,"tag":1244,"props":3087,"children":3088},{"style":2933},[3089],{"type":56,"value":3090}," tsv",{"type":43,"tag":1244,"props":3092,"children":3093},{"style":2927},[3094],{"type":56,"value":3095},")\n",{"type":43,"tag":1244,"props":3097,"children":3099},{"class":1246,"line":3098},8,[3100],{"type":43,"tag":1244,"props":3101,"children":3103},{"emptyLinePlaceholder":3102},true,[3104],{"type":56,"value":3105},"\n",{"type":43,"tag":1244,"props":3107,"children":3109},{"class":1246,"line":3108},9,[3110],{"type":43,"tag":1244,"props":3111,"children":3112},{"style":2912},[3113],{"type":56,"value":3114},"# Create Bronze lakehouse\n",{"type":43,"tag":1244,"props":3116,"children":3118},{"class":1246,"line":3117},10,[3119,3123,3127,3131,3135],{"type":43,"tag":1244,"props":3120,"children":3121},{"style":2921},[3122],{"type":56,"value":2924},{"type":43,"tag":1244,"props":3124,"children":3125},{"style":2927},[3126],{"type":56,"value":2930},{"type":43,"tag":1244,"props":3128,"children":3129},{"style":2933},[3130],{"type":56,"value":2936},{"type":43,"tag":1244,"props":3132,"children":3133},{"style":2927},[3134],{"type":56,"value":2941},{"type":43,"tag":1244,"props":3136,"children":3137},{"style":2927},[3138],{"type":56,"value":2946},{"type":43,"tag":1244,"props":3140,"children":3142},{"class":1246,"line":3141},11,[3143],{"type":43,"tag":1244,"props":3144,"children":3145},{"style":2933},[3146],{"type":56,"value":3147},"{\"displayName\": \"sales_bronze\", \"type\": \"Lakehouse\"}\n",{"type":43,"tag":1244,"props":3149,"children":3151},{"class":1246,"line":3150},12,[3152],{"type":43,"tag":1244,"props":3153,"children":3154},{"style":2927},[3155],{"type":56,"value":2963},{"type":43,"tag":1244,"props":3157,"children":3159},{"class":1246,"line":3158},13,[3160,3164,3168,3172,3176,3180,3184,3188,3192],{"type":43,"tag":1244,"props":3161,"children":3162},{"style":2921},[3163],{"type":56,"value":2983},{"type":43,"tag":1244,"props":3165,"children":3166},{"style":2933},[3167],{"type":56,"value":2988},{"type":43,"tag":1244,"props":3169,"children":3170},{"style":2933},[3171],{"type":56,"value":2993},{"type":43,"tag":1244,"props":3173,"children":3174},{"style":2933},[3175],{"type":56,"value":2998},{"type":43,"tag":1244,"props":3177,"children":3178},{"style":2933},[3179],{"type":56,"value":3003},{"type":43,"tag":1244,"props":3181,"children":3182},{"style":2927},[3183],{"type":56,"value":3008},{"type":43,"tag":1244,"props":3185,"children":3186},{"style":2933},[3187],{"type":56,"value":3013},{"type":43,"tag":1244,"props":3189,"children":3190},{"style":2927},[3191],{"type":56,"value":3018},{"type":43,"tag":1244,"props":3193,"children":3194},{"style":2970},[3195],{"type":56,"value":3023},{"type":43,"tag":1244,"props":3197,"children":3199},{"class":1246,"line":3198},14,[3200,3204,3208,3213,3218,3223,3227],{"type":43,"tag":1244,"props":3201,"children":3202},{"style":2933},[3203],{"type":56,"value":3032},{"type":43,"tag":1244,"props":3205,"children":3206},{"style":2927},[3207],{"type":56,"value":3008},{"type":43,"tag":1244,"props":3209,"children":3210},{"style":2933},[3211],{"type":56,"value":3212},"https:\u002F\u002Fapi.fabric.microsoft.com\u002Fv1\u002Fworkspaces\u002F",{"type":43,"tag":1244,"props":3214,"children":3215},{"style":2970},[3216],{"type":56,"value":3217},"$workspace_id",{"type":43,"tag":1244,"props":3219,"children":3220},{"style":2933},[3221],{"type":56,"value":3222},"\u002Fitems",{"type":43,"tag":1244,"props":3224,"children":3225},{"style":2927},[3226],{"type":56,"value":3018},{"type":43,"tag":1244,"props":3228,"children":3229},{"style":2970},[3230],{"type":56,"value":3023},{"type":43,"tag":1244,"props":3232,"children":3234},{"class":1246,"line":3233},15,[3235,3239],{"type":43,"tag":1244,"props":3236,"children":3237},{"style":2933},[3238],{"type":56,"value":3058},{"type":43,"tag":1244,"props":3240,"children":3241},{"style":2933},[3242],{"type":56,"value":3243}," @\u002Ftmp\u002Fbody.json\n",{"type":43,"tag":445,"props":3245,"children":3247},{"id":3246},"example-2-design-bronze-ingestion",[3248],{"type":56,"value":3249},"Example 2: Design Bronze Ingestion",{"type":43,"tag":48,"props":3251,"children":3252},{},[3253,3257],{"type":43,"tag":52,"props":3254,"children":3255},{},[3256],{"type":56,"value":2827},{"type":56,"value":3258},": \"Ingest daily CSV files into bronze lakehouse with metadata columns\"",{"type":43,"tag":48,"props":3260,"children":3261},{},[3262,3266],{"type":43,"tag":52,"props":3263,"children":3264},{},[3265],{"type":56,"value":2837},{"type":56,"value":3267},": PySpark notebook that:",{"type":43,"tag":117,"props":3269,"children":3270},{},[3271,3276,3303,3308],{"type":43,"tag":72,"props":3272,"children":3273},{},[3274],{"type":56,"value":3275},"Reads source files with schema inference or explicit schema",{"type":43,"tag":72,"props":3277,"children":3278},{},[3279,3281,3287,3288,3294,3295,3301],{"type":56,"value":3280},"Adds ",{"type":43,"tag":83,"props":3282,"children":3284},{"className":3283},[],[3285],{"type":56,"value":3286},"ingestion_timestamp",{"type":56,"value":182},{"type":43,"tag":83,"props":3289,"children":3291},{"className":3290},[],[3292],{"type":56,"value":3293},"source_file",{"type":56,"value":182},{"type":43,"tag":83,"props":3296,"children":3298},{"className":3297},[],[3299],{"type":56,"value":3300},"batch_id",{"type":56,"value":3302}," columns",{"type":43,"tag":72,"props":3304,"children":3305},{},[3306],{"type":56,"value":3307},"Writes to Delta table partitioned by ingestion date",{"type":43,"tag":72,"props":3309,"children":3310},{},[3311],{"type":56,"value":3312},"Logs row count and validation metrics",{"type":43,"tag":1233,"props":3314,"children":3316},{"className":1235,"code":3315,"language":1237,"meta":1238,"style":1238},"# Bronze ingestion pattern (guide LLM to generate full implementation)\nfrom pyspark.sql.functions import current_timestamp, input_file_name, lit\nimport uuid\n\nbatch_id = str(uuid.uuid4())\ndf = (spark.read.format(\"csv\").option(\"header\", True).load(\"\u002FFiles\u002Flanding\u002Fdaily\u002F\")\n      .withColumn(\"ingestion_timestamp\", current_timestamp())\n      .withColumn(\"source_file\", input_file_name())\n      .withColumn(\"batch_id\", lit(batch_id)))\ndf.write.mode(\"append\").partitionBy(\"ingestion_date\").format(\"delta\").saveAsTable(\"bronze.events_raw\")\n",[3317],{"type":43,"tag":83,"props":3318,"children":3319},{"__ignoreMap":1238},[3320,3328,3336,3344,3351,3359,3367,3375,3383,3391],{"type":43,"tag":1244,"props":3321,"children":3322},{"class":1246,"line":1247},[3323],{"type":43,"tag":1244,"props":3324,"children":3325},{},[3326],{"type":56,"value":3327},"# Bronze ingestion pattern (guide LLM to generate full implementation)\n",{"type":43,"tag":1244,"props":3329,"children":3330},{"class":1246,"line":1256},[3331],{"type":43,"tag":1244,"props":3332,"children":3333},{},[3334],{"type":56,"value":3335},"from pyspark.sql.functions import current_timestamp, input_file_name, lit\n",{"type":43,"tag":1244,"props":3337,"children":3338},{"class":1246,"line":1845},[3339],{"type":43,"tag":1244,"props":3340,"children":3341},{},[3342],{"type":56,"value":3343},"import uuid\n",{"type":43,"tag":1244,"props":3345,"children":3346},{"class":1246,"line":2957},[3347],{"type":43,"tag":1244,"props":3348,"children":3349},{"emptyLinePlaceholder":3102},[3350],{"type":56,"value":3105},{"type":43,"tag":1244,"props":3352,"children":3353},{"class":1246,"line":2966},[3354],{"type":43,"tag":1244,"props":3355,"children":3356},{},[3357],{"type":56,"value":3358},"batch_id = str(uuid.uuid4())\n",{"type":43,"tag":1244,"props":3360,"children":3361},{"class":1246,"line":3026},[3362],{"type":43,"tag":1244,"props":3363,"children":3364},{},[3365],{"type":56,"value":3366},"df = (spark.read.format(\"csv\").option(\"header\", True).load(\"\u002FFiles\u002Flanding\u002Fdaily\u002F\")\n",{"type":43,"tag":1244,"props":3368,"children":3369},{"class":1246,"line":3052},[3370],{"type":43,"tag":1244,"props":3371,"children":3372},{},[3373],{"type":56,"value":3374},"      .withColumn(\"ingestion_timestamp\", current_timestamp())\n",{"type":43,"tag":1244,"props":3376,"children":3377},{"class":1246,"line":3098},[3378],{"type":43,"tag":1244,"props":3379,"children":3380},{},[3381],{"type":56,"value":3382},"      .withColumn(\"source_file\", input_file_name())\n",{"type":43,"tag":1244,"props":3384,"children":3385},{"class":1246,"line":3108},[3386],{"type":43,"tag":1244,"props":3387,"children":3388},{},[3389],{"type":56,"value":3390},"      .withColumn(\"batch_id\", lit(batch_id)))\n",{"type":43,"tag":1244,"props":3392,"children":3393},{"class":1246,"line":3117},[3394],{"type":43,"tag":1244,"props":3395,"children":3396},{},[3397],{"type":56,"value":3398},"df.write.mode(\"append\").partitionBy(\"ingestion_date\").format(\"delta\").saveAsTable(\"bronze.events_raw\")\n",{"type":43,"tag":445,"props":3400,"children":3402},{"id":3401},"example-3-bronze-to-silver-transformation",[3403],{"type":56,"value":3404},"Example 3: Bronze-to-Silver Transformation",{"type":43,"tag":48,"props":3406,"children":3407},{},[3408,3412],{"type":43,"tag":52,"props":3409,"children":3410},{},[3411],{"type":56,"value":2827},{"type":56,"value":3413},": \"Clean bronze data: remove duplicates, filter invalid records, add derived columns, write to silver\"",{"type":43,"tag":48,"props":3415,"children":3416},{},[3417,3421],{"type":43,"tag":52,"props":3418,"children":3419},{},[3420],{"type":56,"value":2837},{"type":56,"value":3422},": PySpark notebook applying quality rules, schema conformance, and partitioned write with optimization.",{"type":43,"tag":445,"props":3424,"children":3426},{"id":3425},"example-4-end-to-end-pipeline",[3427],{"type":56,"value":3428},"Example 4: End-to-End Pipeline",{"type":43,"tag":48,"props":3430,"children":3431},{},[3432,3436],{"type":43,"tag":52,"props":3433,"children":3434},{},[3435],{"type":56,"value":2827},{"type":56,"value":3437},": \"Create a pipeline that runs bronze ingestion, then silver transformation, then gold aggregation daily at 2 AM\"",{"type":43,"tag":48,"props":3439,"children":3440},{},[3441,3445],{"type":43,"tag":52,"props":3442,"children":3443},{},[3444],{"type":56,"value":2837},{"type":56,"value":3446},": Pipeline JSON definition with sequential notebook activities, date parameter, retry logic, and schedule trigger.",{"type":43,"tag":3448,"props":3449,"children":3450},"style",{},[3451],{"type":56,"value":3452},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3454,"total":3556},[3455,3466,3485,3503,3511,3527,3543],{"slug":64,"name":64,"fn":3456,"description":3457,"org":3458,"tags":3459,"stars":26,"repoUrl":27,"updatedAt":3465},"check for Fabric skill marketplace updates","Check an installed skills-for-fabric plugin bundle or git clone for updates, show the matching changelog, and provide host-appropriate update guidance. Use when the user wants to: (1) check for skill updates, (2) see what changed, (3) verify the installed version. Triggers: \"check for updates\", \"am I up to date\", \"what version\", \"update skills\", \"show changelog\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3460,3461,3462],{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":3463,"slug":3464,"type":15},"Operations","operations","2026-07-24T05:41:25.892114",{"slug":3467,"name":3467,"fn":3468,"description":3469,"org":3470,"tags":3471,"stars":26,"repoUrl":27,"updatedAt":3484},"databricks-migration","migrate Databricks notebooks to Microsoft Fabric","Port Databricks notebooks and jobs to Microsoft Fabric. Provides an exhaustive dbutils to notebookutils substitution table: fs operations (runtime mounts or OneLake Shortcuts), secret scope to Key Vault URL conversion, notebook run and exit, widget replacement with parameter-tagged cells, and library install replacement with Fabric Environments. Covers Unity Catalog three-level namespace reduction to Lakehouse two-level schemas, DBFS path conversion to OneLake, Databricks Jobs to Spark Job Definitions, MLflow tracking URI removal, and Photon to Native Execution Engine substitution. Use when the user wants to: (1) replace dbutils with notebookutils, (2) collapse Unity Catalog namespaces to Lakehouse schemas, (3) convert Databricks Jobs or Delta Live Tables. Triggers: \"migrate from databricks\", \"databricks to fabric\", \"dbutils to notebookutils\", \"dbutils fabric\", \"unity catalog migration\", \"dbfs to onelake\", \"databricks notebook migration\", \"delta live tables fabric\", \"photon native execution\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3472,3473,3476,3477,3478,3481],{"name":20,"slug":21,"type":15},{"name":3474,"slug":3475,"type":15},"Databricks","databricks",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":3479,"slug":3480,"type":15},"Migration","migration",{"name":3482,"slug":3483,"type":15},"Notebooks","notebooks","2026-07-24T05:41:22.914341",{"slug":3486,"name":3486,"fn":3487,"description":3488,"org":3489,"tags":3490,"stars":26,"repoUrl":27,"updatedAt":3502},"dataflows-authoring-cli","manage Microsoft Fabric Dataflows via CLI","Create, update, delete, and refresh Fabric Dataflows Gen2 with write-side CLI via Fabric APIs. Build mashup.pq and queryMetadata.json, preview candidate M with executeQuery\u002FcustomMashupDocument, bind connections, and configure output destinations. For saved query execution or refresh-status reads, use `dataflows-consumption-cli`. If a request explicitly insists on the Dataflows consumption or read-only path for a mutation, do not route here; let consumption refuse before any separately confirmed authoring handoff. Triggers: \"create dataflow\", \"update dataflow\", \"delete dataflow\", \"trigger dataflow refresh\", \"preview Power Query M\", \"preview before save\", \"customMashupDocument\", \"create Fabric data source connection\", \"create SQL Server source REST\", \"POST \u002Fv1\u002Fconnections\", \"supportedConnectionTypes\", \"passwordReference\", \"bind connection\", \"dataflow output destination\", \"dataflow write to lakehouse\", \"dataflow write to warehouse\", \"dataflow write to ADX\", \"DataDestinations annotation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3491,3494,3495,3498,3499],{"name":3492,"slug":3493,"type":15},"CLI","cli",{"name":20,"slug":21,"type":15},{"name":3496,"slug":3497,"type":15},"Data Pipeline","data-pipeline",{"name":24,"slug":25,"type":15},{"name":3500,"slug":3501,"type":15},"REST API","rest-api","2026-07-31T05:54:59.059517",{"slug":4,"name":4,"fn":5,"description":6,"org":3504,"tags":3505,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3506,3507,3508,3509,3510],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":13,"slug":14,"type":15},{"slug":3512,"name":3512,"fn":3513,"description":3514,"org":3515,"tags":3516,"stars":26,"repoUrl":27,"updatedAt":3526},"eventhouse-authoring-cli","manage Fabric Eventhouse and KQL databases","Execute KQL management commands (table management, ingestion, policies, functions, materialized views) against Fabric Eventhouse and KQL Databases via CLI. Use when the user wants to:\n  1. Create or alter KQL tables, columns, or functions\n  2. Ingest data into an Eventhouse (inline, from storage, streaming)\n  3. Configure retention, caching, or partitioning policies\n  4. Create or manage materialized views and update policies\n  5. Manage data mappings for ingestion pipelines\n  6. Deploy KQL schema via scripts\nTriggers: \"create kql table\", \"kql ingestion\", \"ingest into eventhouse\", \"kql function\", \"materialized view\", \"kql retention policy\", \"eventhouse schema\", \"kql authoring\", \"create eventhouse table\", \"kql mapping\"\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3517,3520,3523,3524,3525],{"name":3518,"slug":3519,"type":15},"Database","database",{"name":3521,"slug":3522,"type":15},"KQL","kql",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":3463,"slug":3464,"type":15},"2026-04-06T18:36:51.340409",{"slug":3528,"name":3528,"fn":3529,"description":3530,"org":3531,"tags":3532,"stars":26,"repoUrl":27,"updatedAt":3542},"eventhouse-consumption-cli","query Fabric Eventhouse with KQL","Run KQL queries against Fabric Eventhouse for real-time intelligence and time-series analytics using `az rest` against the Kusto REST API. Covers KQL operators (where, summarize, join, render), Eventhouse schema discovery (.show tables), time-series patterns with bin(), and ingestion monitoring. Use when the user wants to:\n  1. Run read-only KQL queries against an Eventhouse or KQL Database\n  2. Discover Eventhouse table schema and metadata\n  3. Analyse real-time or time-series data with KQL operators\n  4. Monitor ingestion health and active KQL queries\n  5. Export KQL results to JSON\nTriggers: \"kql query\", \"kusto query\", \"eventhouse query\", \"kql database\", \"real-time intelligence\", \"time-series kql\", \"query eventhouse\", \"explore eventhouse\", \"show tables kql\"\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3533,3536,3537,3538,3539],{"name":3534,"slug":3535,"type":15},"Analytics","analytics",{"name":3521,"slug":3522,"type":15},{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":3540,"slug":3541,"type":15},"Real-Time Intelligence","real-time-intelligence","2026-04-06T18:36:45.018807",{"slug":3544,"name":3544,"fn":3545,"description":3546,"org":3547,"tags":3548,"stars":26,"repoUrl":27,"updatedAt":3555},"eventstream-consumption-cli","monitor Fabric Eventstream ingestion pipelines","List, inspect, and monitor Fabric Eventstream real-time ingestion pipelines via the Items REST API. Discover Eventstreams across workspaces, decode base64 graph topologies tracing event flow from source through operators to destination nodes. Validate connection IDs, wiring, retention policies (1-90 days), and throughput levels. Retrieve Custom Endpoint Kafka credentials via Topology API. **Invoke this skill** to: (1) list Eventstreams, (2) inspect Eventstream topology showing sources and destinations, (3) validate Eventstream configurations, (4) check Eventstream retention policy and throughput level, (5) get connection strings. Triggers: \"list eventstreams\", \"inspect eventstream\", \"describe eventstream topology\", \"eventstream operator nodes\", \"eventstream sources and destinations\", \"eventstream health\", \"eventstream status\", \"eventstream configuration\", \"eventstream retention\", \"eventstream throughput level\", \"eventstream connection string\", \"custom endpoint credentials\", \"check eventstream\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3549,3550,3551,3552],{"name":3492,"slug":3493,"type":15},{"name":20,"slug":21,"type":15},{"name":24,"slug":25,"type":15},{"name":3553,"slug":3554,"type":15},"Monitoring","monitoring","2026-07-31T05:55:03.045486",19,{"items":3558,"total":3746},[3559,3581,3602,3619,3634,3650,3661,3674,3689,3704,3721,3734],{"slug":3560,"name":3560,"fn":3561,"description":3562,"org":3563,"tags":3564,"stars":3578,"repoUrl":3579,"updatedAt":3580},"rushstack-best-practices","manage Rush monorepos with best practices","Provides best practices and guidance for working with Rush monorepos. Use when the user is working in a Rush-based repository, asks about Rush commands (install, update, build, rebuild), needs help with project selection, dependency management, build caching, subspace configuration, or troubleshooting Rush-specific issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3565,3568,3571,3572,3575],{"name":3566,"slug":3567,"type":15},"Engineering","engineering",{"name":3569,"slug":3570,"type":15},"Local Development","local-development",{"name":9,"slug":8,"type":15},{"name":3573,"slug":3574,"type":15},"Project Management","project-management",{"name":3576,"slug":3577,"type":15},"Rush","rush",6484,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Frushstack","2026-04-06T18:34:44.965032",{"slug":3582,"name":3582,"fn":3583,"description":3584,"org":3585,"tags":3586,"stars":3599,"repoUrl":3600,"updatedAt":3601},"azure-ai-agents-persistent-dotnet","build AI agents with Azure .NET SDK","Azure AI Agents Persistent SDK for .NET. Low-level SDK for creating and managing AI agents with threads, messages, runs, and tools. Use for agent CRUD, conversation threads, streaming responses, function calling, file search, and code interpreter. Triggers: \"PersistentAgentsClient\", \"persistent agents\", \"agent threads\", \"agent runs\", \"streaming agents\", \"function calling agents .NET\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3587,3590,3593,3596],{"name":3588,"slug":3589,"type":15},".NET","net",{"name":3591,"slug":3592,"type":15},"Agents","agents",{"name":3594,"slug":3595,"type":15},"Azure","azure",{"name":3597,"slug":3598,"type":15},"LLM","llm",2804,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills","2026-07-03T16:32:10.297433",{"slug":3603,"name":3603,"fn":3604,"description":3605,"org":3606,"tags":3607,"stars":3599,"repoUrl":3600,"updatedAt":3618},"azure-ai-anomalydetector-java","build anomaly detection applications with Java","Build anomaly detection applications with Azure AI Anomaly Detector SDK for Java. Use when implementing univariate\u002Fmultivariate anomaly detection, time-series analysis, or AI-powered monitoring.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3608,3609,3610,3613,3616,3617],{"name":3534,"slug":3535,"type":15},{"name":3594,"slug":3595,"type":15},{"name":3611,"slug":3612,"type":15},"Data Analysis","data-analysis",{"name":3614,"slug":3615,"type":15},"Java","java",{"name":9,"slug":8,"type":15},{"name":3553,"slug":3554,"type":15},"2026-05-13T06:14:16.261754",{"slug":3620,"name":3620,"fn":3621,"description":3622,"org":3623,"tags":3624,"stars":3599,"repoUrl":3600,"updatedAt":3633},"azure-ai-contentsafety-java","build content moderation applications with Azure AI","Build content moderation applications with Azure AI Content Safety SDK for Java. Use when implementing text\u002Fimage analysis, blocklist management, or harm detection for hate, violence, sexual content, and self-harm.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3625,3628,3629,3630],{"name":3626,"slug":3627,"type":15},"AI Infrastructure","ai-infrastructure",{"name":3594,"slug":3595,"type":15},{"name":3614,"slug":3615,"type":15},{"name":3631,"slug":3632,"type":15},"Security","security","2026-07-07T06:53:31.293235",{"slug":3635,"name":3635,"fn":3636,"description":3637,"org":3638,"tags":3639,"stars":3599,"repoUrl":3600,"updatedAt":3649},"azure-ai-contentsafety-py","detect harmful content with Azure AI Content Safety","Azure AI Content Safety SDK for Python. Use for detecting harmful content in text and images with multi-severity classification.\nTriggers: \"azure-ai-contentsafety\", \"ContentSafetyClient\", \"content moderation\", \"harmful content\", \"text analysis\", \"image analysis\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3640,3641,3644,3645,3646,3648],{"name":3594,"slug":3595,"type":15},{"name":3642,"slug":3643,"type":15},"Compliance","compliance",{"name":3597,"slug":3598,"type":15},{"name":9,"slug":8,"type":15},{"name":3647,"slug":1237,"type":15},"Python",{"name":3631,"slug":3632,"type":15},"2026-07-18T05:14:23.017504",{"slug":3651,"name":3651,"fn":3652,"description":3653,"org":3654,"tags":3655,"stars":3599,"repoUrl":3600,"updatedAt":3660},"azure-ai-language-conversations-py","implement conversational language understanding with Python","Implement Conversational Language Understanding (CLU) using the azure-ai-language-conversations Python SDK. Use when working with ConversationAnalysisClient to analyze conversation intent and entities, building NLP features, or integrating language understanding into applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3656,3657,3658,3659],{"name":3534,"slug":3535,"type":15},{"name":3594,"slug":3595,"type":15},{"name":3597,"slug":3598,"type":15},{"name":3647,"slug":1237,"type":15},"2026-07-31T05:54:29.068751",{"slug":3662,"name":3662,"fn":3663,"description":3664,"org":3665,"tags":3666,"stars":3599,"repoUrl":3600,"updatedAt":3673},"azure-ai-translation-text-py","translate text using Azure AI services","Azure AI Text Translation SDK for real-time text translation, transliteration, language detection, and dictionary lookup. Use for translating text content in applications.\nTriggers: \"text translation\", \"translator\", \"translate text\", \"transliterate\", \"TextTranslationClient\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3667,3670,3671,3672],{"name":3668,"slug":3669,"type":15},"API Development","api-development",{"name":3594,"slug":3595,"type":15},{"name":9,"slug":8,"type":15},{"name":3647,"slug":1237,"type":15},"2026-07-18T05:14:16.988376",{"slug":3675,"name":3675,"fn":3676,"description":3677,"org":3678,"tags":3679,"stars":3599,"repoUrl":3600,"updatedAt":3688},"azure-ai-vision-imageanalysis-py","analyze images with Azure AI Vision","Azure AI Vision Image Analysis SDK for captions, tags, objects, OCR, people detection, and smart cropping. Use for computer vision and image understanding tasks.\nTriggers: \"image analysis\", \"computer vision\", \"OCR\", \"object detection\", \"ImageAnalysisClient\", \"image caption\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3680,3681,3684,3687],{"name":3594,"slug":3595,"type":15},{"name":3682,"slug":3683,"type":15},"Computer Vision","computer-vision",{"name":3685,"slug":3686,"type":15},"Images","images",{"name":3647,"slug":1237,"type":15},"2026-07-18T05:14:18.007737",{"slug":3690,"name":3690,"fn":3691,"description":3692,"org":3693,"tags":3694,"stars":3599,"repoUrl":3600,"updatedAt":3703},"azure-appconfiguration-java","manage configuration with Azure App Configuration","Azure App Configuration SDK for Java. Centralized application configuration management with key-value settings, feature flags, and snapshots.\nTriggers: \"ConfigurationClient java\", \"app configuration java\", \"feature flag java\", \"configuration setting java\", \"azure config java\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3695,3696,3699,3702],{"name":3594,"slug":3595,"type":15},{"name":3697,"slug":3698,"type":15},"Configuration","configuration",{"name":3700,"slug":3701,"type":15},"Feature Flags","feature-flags",{"name":3614,"slug":3615,"type":15},"2026-07-03T16:32:01.278468",{"slug":3705,"name":3705,"fn":3706,"description":3707,"org":3708,"tags":3709,"stars":3599,"repoUrl":3600,"updatedAt":3720},"azure-cosmos-rust","build applications with Azure Cosmos DB","Azure Cosmos DB library for Rust (NoSQL API). Document CRUD, containers, and globally distributed data.\nTriggers: \"cosmos db rust\", \"CosmosClient rust\", \"document crud rust\", \"NoSQL rust\", \"partition key rust\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3710,3713,3714,3717],{"name":3711,"slug":3712,"type":15},"Cosmos DB","cosmos-db",{"name":3518,"slug":3519,"type":15},{"name":3715,"slug":3716,"type":15},"NoSQL","nosql",{"name":3718,"slug":3719,"type":15},"Rust","rust","2026-07-31T05:54:27.021432",{"slug":3722,"name":3722,"fn":3706,"description":3723,"org":3724,"tags":3725,"stars":3599,"repoUrl":3600,"updatedAt":3733},"azure-cosmos-ts","Azure Cosmos DB JavaScript\u002FTypeScript SDK (@azure\u002Fcosmos) for data plane operations. Use for CRUD operations on documents, queries, bulk operations, and container management. Triggers: \"Cosmos DB\", \"@azure\u002Fcosmos\", \"CosmosClient\", \"document CRUD\", \"NoSQL queries\", \"bulk operations\", \"partition key\", \"container.items\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3726,3727,3728,3729,3730],{"name":3711,"slug":3712,"type":15},{"name":3518,"slug":3519,"type":15},{"name":9,"slug":8,"type":15},{"name":3715,"slug":3716,"type":15},{"name":3731,"slug":3732,"type":15},"TypeScript","typescript","2026-07-03T16:31:19.368382",{"slug":3735,"name":3735,"fn":3736,"description":3737,"org":3738,"tags":3739,"stars":3599,"repoUrl":3600,"updatedAt":3745},"azure-data-tables-java","build table storage applications with Java","Build table storage applications with Azure Tables SDK for Java. Use when working with Azure Table Storage or Cosmos DB Table API for NoSQL key-value data, schemaless storage, or structured data at scale.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3740,3741,3742,3743,3744],{"name":3594,"slug":3595,"type":15},{"name":3711,"slug":3712,"type":15},{"name":3518,"slug":3519,"type":15},{"name":3614,"slug":3615,"type":15},{"name":3715,"slug":3716,"type":15},"2026-05-13T06:14:17.582229",267]