[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-synthetic-data-gen":3,"mdc--y3js6e-key":33,"related-repo-databricks-databricks-synthetic-data-gen":2156,"related-org-databricks-databricks-synthetic-data-gen":2275},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"databricks-synthetic-data-gen","generate synthetic data with Spark","Generate realistic synthetic data using Spark + Faker (strongly recommended). Supports serverless execution, multiple output formats (Parquet\u002FJSON\u002FCSV\u002FDelta), and scales from thousands to millions of rows. For small datasets (\u003C10K rows), can optionally generate locally and upload to volumes. Use when user mentions 'synthetic data', 'test data', 'generate data', 'demo dataset', 'Faker', or 'sample data'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Spark","spark",204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-12T08:04:34.505856",null,60,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-synthetic-data-gen","---\nname: databricks-synthetic-data-gen\ndescription: \"Generate realistic synthetic data using Spark + Faker (strongly recommended). Supports serverless execution, multiple output formats (Parquet\u002FJSON\u002FCSV\u002FDelta), and scales from thousands to millions of rows. For small datasets (\u003C10K rows), can optionally generate locally and upload to volumes. Use when user mentions 'synthetic data', 'test data', 'generate data', 'demo dataset', 'Faker', or 'sample data'.\"\ncompatibility: Requires databricks CLI (>= v1.0.0)\nmetadata:\n  version: \"0.1.0\"\nparent: databricks-core\n---\n\n> Catalog and schema are **always user-supplied** — never default to any value. If the user hasn't provided them, ask. For any UC write, **always create the schema if it doesn't exist** before writing data.\n\n# Databricks Synthetic Data Generation\n\nGenerate realistic, story-driven synthetic data for Databricks using **Spark + Faker + Pandas UDFs** (strongly recommended).\n\n## Data Must Tell a Business Story\n\nSynthetic data should demonstrate how Databricks helps solve real business problems.\n\n**The pattern:** Something goes wrong → business impact ($) → analyze root cause → identify affected customers → fix and prevent.\n\n**Key principles:**\n- **Problem → Impact → Analysis → Solution** — Include an incident, anomaly, or issue that causes measurable business impact. The data lets you find the root cause and act on it.\n- **Industry-relevant but simple** — Use domain terms (e.g., \"SLA breach\", \"churn\", \"stockout\") but keep the schema easy to understand. A few tables, clear relationships.\n- **Business metrics with $ impact** — Revenue, MRR, cost, conversion rate. Every story needs a dollar sign to show why it matters.\n- **Tables explain each other** — Ticket spike? Incident table shows the outage. Revenue drop? Churn table shows who left and why. All data connects.\n- **Actionable insights** — Data should answer: What happened? Who's affected? How much did it cost? How do we prevent it?\n\n**Why no flat distributions:** Uniform data has no story — no spikes, no anomalies, no cohort, no 20\u002F80, no skew, nothing to investigate. It can't show Databricks' value for root cause analysis.\n\n## References\n\n| When | Guide |\n|------|-------|\n| User mentions **ML model training** or complex time patterns | [references\u002F1-data-patterns.md](references\u002F1-data-patterns.md) — ML-ready data, time multipliers, row coherence |\n| Errors during generation | [references\u002F2-troubleshooting.md](references\u002F2-troubleshooting.md) — Fixing common issues |\n\n## Critical Rules\n\n1. **Data tells a story** — Something goes wrong, impacts $, can be analyzed and fixed. Show Databricks value.\n2. **All data serves the story** — Every table and column must be coherent and usable in dashboards or ML models. No orphan data, no random noise — if it doesn't help explain or plot a futur dashboard or predict, don't generate it.\n3. **Industry terms, simple schema** — Use domain-specific vocabulary but keep it easy to understand (few tables, clear relationships)\n4. **Never uniform distributions** — Skewed categories, log-normal amounts, 80\u002F20 patterns. Flat = no story = useless\n5. **Enough data for trends** — ~100K+ rows for main tables so patterns survive aggregation\n6. **Ask for catalog\u002Fschema** — Never default, always confirm before generating\n7. **Present plan for approval** — Show tables, distributions, assumptions before writing code\n8. **Master tables first** — Generate parent tables, write to Delta, then create children with valid FKs\n9. **Use Spark + Faker + Pandas UDFs** — Scalable, parallel. Polars only if user explicitly wants local + \u003C30K rows\n10. **Use Databricks Connect Serverless by default to generate data** — Update databricks-connect on python 3.12 if required (avoid using execute_code unless instructed to not use Databricks Connect)\n11. **No `.cache()` or `.persist()`** — Not supported on serverless. Write to Delta, read back for joins\n12. **No Python loops or `.collect()`** — Use Spark parallelism. No driver-side iteration, avoid Pandas↔Spark conversions\n\n## Generation Planning Workflow\n\n**Before generating any code, you MUST present a plan for user approval.**\n\n### ⚠️ MUST DO: Confirm Catalog Before Proceeding\n\n**You MUST explicitly ask the user which catalog to use.** Do not assume or proceed without confirmation.\n\nExample prompt to user:\n> \"Which Unity Catalog should I use for this data?\"\n\nWhen presenting your plan, always show the selected catalog prominently:\n```\n📍 Output Location: catalog_name.schema_name\n   Volume: \u002FVolumes\u002Fcatalog_name\u002Fschema_name\u002Fraw_data\u002F\n```\n\nThis makes it easy for the user to spot and correct if needed.\n\n### Step 1: Gather Requirements\n\nAsk the user about:\n- **Catalog\u002FSchema** — Which catalog to use?\n- **Domain** — E-commerce, support tickets, IoT, financial? (Use industry terms)\n\n**If user doesn't specify a story:** Propose one. Don't generate bland data — suggest an incident, anomaly, or trend that shows Databricks value (e.g., \"I'll include a system outage that causes ticket spike and churn — this lets you demo root cause analysis\").\n\n### Step 2: Present Plan with Story\n\nShow a clear specification with **the business story and your assumptions surfaced**:\n\n```\n📍 Output Location: {user_catalog}.support_demo\n   Volume: \u002FVolumes\u002F{user_catalog}\u002Fsupport_demo\u002Fraw_data\u002F\n\n📖 Story: A payment system outage causes support ticket spike. Resolution times\n   degrade, enterprise customers churn, revenue drops $2.3M. With Databricks we\n   identify the root cause, affected customers, and prevent future impact.\n```\n\n| Table | Description | Rows | Key Assumptions |\n|-------|-------------|------|-----------------|\n| customers | Customer profiles with tier, MRR | 10,000 | Enterprise 10% but 60% of revenue |\n| tickets | Support tickets with priority, resolution_time | 80,000 | Spike during outage, SLA breaches |\n| incidents | System events (outages, deployments) | 50 | Payment outage mid-month |\n| churn_events | Customer cancellations with reason | 500 | Spike after poor support experience |\n\n**Business metrics:**\n- `customers.mrr` — Revenue at risk ($)\n- `tickets.resolution_hours` — SLA performance\n- `churn_events.lost_mrr` — Churn impact ($)\n\n**The story this data tells:**\n- Incident table shows payment outage on March 15\n- Tickets spike 5x during outage, resolution time degrades from 4h → 18h\n- Enterprise customers with SLA breaches churn 3 weeks later\n- Total impact: $2.3M lost MRR, traceable to one incident\n- **Databricks value:** Root cause analysis, identify at-risk customers, build alerting\n\n**Ask user**: \"Does this story work? Any adjustments?\"\n\n### Step 3: Ask About Data Features\n\n- [x] Skew (non-uniform distributions) - **Enabled by default**\n- [x] Joins (referential integrity) - **Enabled by default**\n- [ ] Bad data injection (for data quality testing)\n- [ ] Multi-language text\n- [ ] Incremental mode (append instead of overwrite)\n\n### Pre-Generation Checklist\n\n- [ ] **Catalog confirmed** - User explicitly approved which catalog to use\n- [ ] Output location shown prominently in plan (easy to spot\u002Fchange)\n- [ ] Table specification shown and approved\n- [ ] Assumptions about distributions confirmed\n- [ ] User confirmed compute preference (Databricks Connect on serverless recommended)\n- [ ] Data features selected\n\n**Do NOT proceed to code generation until user approves the plan, including the catalog.**\n\n### Post-Generation Validation\n\nUse `databricks experimental aitools tools query` to validate generated data (row counts, distributions, referential integrity). Query parquet files directly:\n\n```bash\ndatabricks experimental aitools tools query --warehouse $WAREHOUSE_ID \"\nSELECT COUNT(*) FROM parquet.\\`\u002FVolumes\u002FCATALOG\u002FSCHEMA\u002Fraw_data\u002Fcustomers\\`\n\"\n```\n\nSee [references\u002F2-troubleshooting.md](references\u002F2-troubleshooting.md) for full validation examples.\n\n## Use Databricks Connect Spark + Faker Pattern\n\n```python\nfrom databricks.connect import DatabricksSession\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import StringType\nimport pandas as pd\n\n# Setup serverless Spark session\nspark = DatabricksSession.builder.serverless(True).getOrCreate()\n\n# Pandas UDF pattern - import lib INSIDE the function (libs must be installed locally)\n@F.pandas_udf(StringType())\ndef fake_name(ids: pd.Series) -> pd.Series:\n    from faker import Faker  # Import inside UDF\n    fake = Faker()\n    return pd.Series([fake.name() for _ in range(len(ids))])\n\n# Generate with spark.range, apply UDFs\ncustomers_df = spark.range(0, 10000, numPartitions=16).select(\n    F.concat(F.lit(\"CUST-\"), F.lpad(F.col(\"id\").cast(\"string\"), 5, \"0\")).alias(\"customer_id\"),\n    fake_name(F.col(\"id\")).alias(\"name\"),\n)\n\n# Write to Volume as Parquet (default for raw data)\n# Path is a folder with table name: \u002FVolumes\u002Fcatalog\u002Fschema\u002Fraw_data\u002Fcustomers\u002F\nspark.sql(f\"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}\")\nspark.sql(f\"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data\")\ncustomers_df.write.mode(\"overwrite\").parquet(f\"\u002FVolumes\u002F{CATALOG}\u002F{SCHEMA}\u002Fraw_data\u002Fcustomers\")\n```\n\n**Partitions by scale:** `spark.range(N, numPartitions=P)`\n- \u003C100K rows: 8 partitions\n- 100K-500K: 16 partitions\n- 500K-1M: 32 partitions\n- 1M+: 64+ partitions\n\n**Output formats:**\n- **Parquet to Volume** (default): `df.write.parquet(\"\u002FVolumes\u002F...\u002Fraw_data\u002Ftable\")` — raw data for pipelines\n- **Delta Table**: `df.write.saveAsTable(\"catalog.schema.table\")` — if user wants queryable tables\n- **JSON\u002FCSV**: small dimension tables, replicate legacy systems\n\n## Performance Rules\n\nGenerated scripts must be highly performant. **Never** do these:\n\n| Anti-Pattern | Why It's Slow | Do This Instead |\n|--------------|---------------|-----------------|\n| Python loops on driver | Single-threaded, no parallelism | Use `spark.range()` + Spark operations |\n| `.collect()` then iterate | Brings all data to driver memory | Keep data in Spark, use DataFrame ops |\n| Pandas → Spark → Pandas | Serialization overhead, defeats distribution | Stay in Spark, use `pandas_udf` only for UDFs |\n| Read\u002Fwrite temp files | Unnecessary I\u002FO | Chain DataFrame transformations |\n| Scalar UDFs | Row-by-row processing | Use `pandas_udf` for batch processing |\n\n**Good pattern:** `spark.range()` → Spark transforms → `pandas_udf` for Faker → write directly\n\n## Common Patterns\n\n### Weighted Categories (never uniform)\n```python\nF.when(F.rand() \u003C 0.6, \"Free\").when(F.rand() \u003C 0.9, \"Pro\").otherwise(\"Enterprise\")\n```\n\n### Log-Normal Amounts (in a pandas UDF)\nUse `np.random.lognormal(mean, sigma)` — always positive, long tail:\n- Enterprise: `lognormal(7.5, 0.8)` → ~$1800 median\n- Pro: `lognormal(5.5, 0.7)` → ~$245 median\n- Free: `lognormal(4.0, 0.6)` → ~$55 median\n\n### Date Range (Last 6 Months)\n```python\nEND_DATE = datetime.now()\nSTART_DATE = END_DATE - timedelta(days=180)\n```\n\n### Infrastructure (always create in script)\n```python\nspark.sql(f\"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}\")\nspark.sql(f\"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data\")\n```\n\n### Referential Integrity (FK pattern)\nWrite master table to Delta first, then read back for FK joins (no `.cache()` on serverless):\n```python\n# 1. Write master table\ncustomers_df.write.mode(\"overwrite\").saveAsTable(f\"{CATALOG}.{SCHEMA}.customers\")\n\n# 2. Read back for FK lookup\ncustomer_lookup = spark.table(f\"{CATALOG}.{SCHEMA}.customers\").select(\"customer_idx\", \"customer_id\")\n\n# 3. Generate child table with valid FKs via join\norders_df = spark.range(N_ORDERS).select(\n    (F.abs(F.hash(F.col(\"id\"))) % N_CUSTOMERS).alias(\"customer_idx\")\n)\norders_with_fk = orders_df.join(customer_lookup, on=\"customer_idx\")\n```\n\n## Setup\n\nRequires Python 3.12 and databricks-connect>=16.4. Use `uv`:\n\n```bash\nuv pip install \"databricks-connect>=16.4,\u003C17.4\" faker numpy pandas holidays\n```\n\n## Related Skills\n\n- **databricks-unity-catalog** — Managing catalogs, schemas, and volumes\n- **databricks-dabs** — DABs for production deployment\n\n## Common Issues\n\n| Issue | Solution |\n|-------|----------|\n| `ModuleNotFoundError: faker` | Install locally: `uv pip install faker`, import inside UDF |\n| Faker UDF is slow | Use `pandas_udf` for batch processing |\n| Out of memory | Increase `numPartitions` in `spark.range()` |\n| Referential integrity errors | Write master table to Delta first, read back for FK joins |\n| `PERSIST TABLE is not supported on serverless` | **NEVER use `.cache()` or `.persist()` with serverless** - write to Delta table first, then read back |\n| `F.window` vs `Window` confusion | Use `from pyspark.sql.window import Window` for `row_number()`, `rank()`, etc. `F.window` is for streaming only. |\n| Broadcast variables not supported | **NEVER use `spark.sparkContext.broadcast()` with serverless** |\n\nSee [references\u002F2-troubleshooting.md](references\u002F2-troubleshooting.md) for full troubleshooting guide.\n",{"data":34,"body":39},{"name":4,"description":6,"compatibility":35,"metadata":36,"parent":38},"Requires databricks CLI (>= v1.0.0)",{"version":37},"0.1.0","databricks-core",{"type":40,"children":41},"root",[42,69,76,88,95,100,110,118,173,183,189,261,267,412,418,426,433,443,448,456,461,473,478,484,489,512,522,528,540,549,673,681,717,725,758,768,774,837,843,908,916,922,935,1025,1036,1042,1279,1294,1317,1325,1374,1380,1392,1535,1558,1564,1570,1584,1590,1602,1644,1650,1673,1679,1700,1706,1718,1810,1816,1828,1887,1893,1916,1922,2140,2150],{"type":43,"tag":44,"props":45,"children":46},"element","blockquote",{},[47],{"type":43,"tag":48,"props":49,"children":50},"p",{},[51,54,60,62,67],{"type":52,"value":53},"text","Catalog and schema are ",{"type":43,"tag":55,"props":56,"children":57},"strong",{},[58],{"type":52,"value":59},"always user-supplied",{"type":52,"value":61}," — never default to any value. If the user hasn't provided them, ask. For any UC write, ",{"type":43,"tag":55,"props":63,"children":64},{},[65],{"type":52,"value":66},"always create the schema if it doesn't exist",{"type":52,"value":68}," before writing data.",{"type":43,"tag":70,"props":71,"children":73},"h1",{"id":72},"databricks-synthetic-data-generation",[74],{"type":52,"value":75},"Databricks Synthetic Data Generation",{"type":43,"tag":48,"props":77,"children":78},{},[79,81,86],{"type":52,"value":80},"Generate realistic, story-driven synthetic data for Databricks using ",{"type":43,"tag":55,"props":82,"children":83},{},[84],{"type":52,"value":85},"Spark + Faker + Pandas UDFs",{"type":52,"value":87}," (strongly recommended).",{"type":43,"tag":89,"props":90,"children":92},"h2",{"id":91},"data-must-tell-a-business-story",[93],{"type":52,"value":94},"Data Must Tell a Business Story",{"type":43,"tag":48,"props":96,"children":97},{},[98],{"type":52,"value":99},"Synthetic data should demonstrate how Databricks helps solve real business problems.",{"type":43,"tag":48,"props":101,"children":102},{},[103,108],{"type":43,"tag":55,"props":104,"children":105},{},[106],{"type":52,"value":107},"The pattern:",{"type":52,"value":109}," Something goes wrong → business impact ($) → analyze root cause → identify affected customers → fix and prevent.",{"type":43,"tag":48,"props":111,"children":112},{},[113],{"type":43,"tag":55,"props":114,"children":115},{},[116],{"type":52,"value":117},"Key principles:",{"type":43,"tag":119,"props":120,"children":121},"ul",{},[122,133,143,153,163],{"type":43,"tag":123,"props":124,"children":125},"li",{},[126,131],{"type":43,"tag":55,"props":127,"children":128},{},[129],{"type":52,"value":130},"Problem → Impact → Analysis → Solution",{"type":52,"value":132}," — Include an incident, anomaly, or issue that causes measurable business impact. The data lets you find the root cause and act on it.",{"type":43,"tag":123,"props":134,"children":135},{},[136,141],{"type":43,"tag":55,"props":137,"children":138},{},[139],{"type":52,"value":140},"Industry-relevant but simple",{"type":52,"value":142}," — Use domain terms (e.g., \"SLA breach\", \"churn\", \"stockout\") but keep the schema easy to understand. A few tables, clear relationships.",{"type":43,"tag":123,"props":144,"children":145},{},[146,151],{"type":43,"tag":55,"props":147,"children":148},{},[149],{"type":52,"value":150},"Business metrics with $ impact",{"type":52,"value":152}," — Revenue, MRR, cost, conversion rate. Every story needs a dollar sign to show why it matters.",{"type":43,"tag":123,"props":154,"children":155},{},[156,161],{"type":43,"tag":55,"props":157,"children":158},{},[159],{"type":52,"value":160},"Tables explain each other",{"type":52,"value":162}," — Ticket spike? Incident table shows the outage. Revenue drop? Churn table shows who left and why. All data connects.",{"type":43,"tag":123,"props":164,"children":165},{},[166,171],{"type":43,"tag":55,"props":167,"children":168},{},[169],{"type":52,"value":170},"Actionable insights",{"type":52,"value":172}," — Data should answer: What happened? Who's affected? How much did it cost? How do we prevent it?",{"type":43,"tag":48,"props":174,"children":175},{},[176,181],{"type":43,"tag":55,"props":177,"children":178},{},[179],{"type":52,"value":180},"Why no flat distributions:",{"type":52,"value":182}," Uniform data has no story — no spikes, no anomalies, no cohort, no 20\u002F80, no skew, nothing to investigate. It can't show Databricks' value for root cause analysis.",{"type":43,"tag":89,"props":184,"children":186},{"id":185},"references",[187],{"type":52,"value":188},"References",{"type":43,"tag":190,"props":191,"children":192},"table",{},[193,212],{"type":43,"tag":194,"props":195,"children":196},"thead",{},[197],{"type":43,"tag":198,"props":199,"children":200},"tr",{},[201,207],{"type":43,"tag":202,"props":203,"children":204},"th",{},[205],{"type":52,"value":206},"When",{"type":43,"tag":202,"props":208,"children":209},{},[210],{"type":52,"value":211},"Guide",{"type":43,"tag":213,"props":214,"children":215},"tbody",{},[216,243],{"type":43,"tag":198,"props":217,"children":218},{},[219,232],{"type":43,"tag":220,"props":221,"children":222},"td",{},[223,225,230],{"type":52,"value":224},"User mentions ",{"type":43,"tag":55,"props":226,"children":227},{},[228],{"type":52,"value":229},"ML model training",{"type":52,"value":231}," or complex time patterns",{"type":43,"tag":220,"props":233,"children":234},{},[235,241],{"type":43,"tag":236,"props":237,"children":239},"a",{"href":238},"references\u002F1-data-patterns.md",[240],{"type":52,"value":238},{"type":52,"value":242}," — ML-ready data, time multipliers, row coherence",{"type":43,"tag":198,"props":244,"children":245},{},[246,251],{"type":43,"tag":220,"props":247,"children":248},{},[249],{"type":52,"value":250},"Errors during generation",{"type":43,"tag":220,"props":252,"children":253},{},[254,259],{"type":43,"tag":236,"props":255,"children":257},{"href":256},"references\u002F2-troubleshooting.md",[258],{"type":52,"value":256},{"type":52,"value":260}," — Fixing common issues",{"type":43,"tag":89,"props":262,"children":264},{"id":263},"critical-rules",[265],{"type":52,"value":266},"Critical Rules",{"type":43,"tag":268,"props":269,"children":270},"ol",{},[271,281,291,301,311,321,331,341,351,361,371,396],{"type":43,"tag":123,"props":272,"children":273},{},[274,279],{"type":43,"tag":55,"props":275,"children":276},{},[277],{"type":52,"value":278},"Data tells a story",{"type":52,"value":280}," — Something goes wrong, impacts $, can be analyzed and fixed. Show Databricks value.",{"type":43,"tag":123,"props":282,"children":283},{},[284,289],{"type":43,"tag":55,"props":285,"children":286},{},[287],{"type":52,"value":288},"All data serves the story",{"type":52,"value":290}," — Every table and column must be coherent and usable in dashboards or ML models. No orphan data, no random noise — if it doesn't help explain or plot a futur dashboard or predict, don't generate it.",{"type":43,"tag":123,"props":292,"children":293},{},[294,299],{"type":43,"tag":55,"props":295,"children":296},{},[297],{"type":52,"value":298},"Industry terms, simple schema",{"type":52,"value":300}," — Use domain-specific vocabulary but keep it easy to understand (few tables, clear relationships)",{"type":43,"tag":123,"props":302,"children":303},{},[304,309],{"type":43,"tag":55,"props":305,"children":306},{},[307],{"type":52,"value":308},"Never uniform distributions",{"type":52,"value":310}," — Skewed categories, log-normal amounts, 80\u002F20 patterns. Flat = no story = useless",{"type":43,"tag":123,"props":312,"children":313},{},[314,319],{"type":43,"tag":55,"props":315,"children":316},{},[317],{"type":52,"value":318},"Enough data for trends",{"type":52,"value":320}," — ~100K+ rows for main tables so patterns survive aggregation",{"type":43,"tag":123,"props":322,"children":323},{},[324,329],{"type":43,"tag":55,"props":325,"children":326},{},[327],{"type":52,"value":328},"Ask for catalog\u002Fschema",{"type":52,"value":330}," — Never default, always confirm before generating",{"type":43,"tag":123,"props":332,"children":333},{},[334,339],{"type":43,"tag":55,"props":335,"children":336},{},[337],{"type":52,"value":338},"Present plan for approval",{"type":52,"value":340}," — Show tables, distributions, assumptions before writing code",{"type":43,"tag":123,"props":342,"children":343},{},[344,349],{"type":43,"tag":55,"props":345,"children":346},{},[347],{"type":52,"value":348},"Master tables first",{"type":52,"value":350}," — Generate parent tables, write to Delta, then create children with valid FKs",{"type":43,"tag":123,"props":352,"children":353},{},[354,359],{"type":43,"tag":55,"props":355,"children":356},{},[357],{"type":52,"value":358},"Use Spark + Faker + Pandas UDFs",{"type":52,"value":360}," — Scalable, parallel. Polars only if user explicitly wants local + \u003C30K rows",{"type":43,"tag":123,"props":362,"children":363},{},[364,369],{"type":43,"tag":55,"props":365,"children":366},{},[367],{"type":52,"value":368},"Use Databricks Connect Serverless by default to generate data",{"type":52,"value":370}," — Update databricks-connect on python 3.12 if required (avoid using execute_code unless instructed to not use Databricks Connect)",{"type":43,"tag":123,"props":372,"children":373},{},[374,394],{"type":43,"tag":55,"props":375,"children":376},{},[377,379,386,388],{"type":52,"value":378},"No ",{"type":43,"tag":380,"props":381,"children":383},"code",{"className":382},[],[384],{"type":52,"value":385},".cache()",{"type":52,"value":387}," or ",{"type":43,"tag":380,"props":389,"children":391},{"className":390},[],[392],{"type":52,"value":393},".persist()",{"type":52,"value":395}," — Not supported on serverless. Write to Delta, read back for joins",{"type":43,"tag":123,"props":397,"children":398},{},[399,410],{"type":43,"tag":55,"props":400,"children":401},{},[402,404],{"type":52,"value":403},"No Python loops or ",{"type":43,"tag":380,"props":405,"children":407},{"className":406},[],[408],{"type":52,"value":409},".collect()",{"type":52,"value":411}," — Use Spark parallelism. No driver-side iteration, avoid Pandas↔Spark conversions",{"type":43,"tag":89,"props":413,"children":415},{"id":414},"generation-planning-workflow",[416],{"type":52,"value":417},"Generation Planning Workflow",{"type":43,"tag":48,"props":419,"children":420},{},[421],{"type":43,"tag":55,"props":422,"children":423},{},[424],{"type":52,"value":425},"Before generating any code, you MUST present a plan for user approval.",{"type":43,"tag":427,"props":428,"children":430},"h3",{"id":429},"️-must-do-confirm-catalog-before-proceeding",[431],{"type":52,"value":432},"⚠️ MUST DO: Confirm Catalog Before Proceeding",{"type":43,"tag":48,"props":434,"children":435},{},[436,441],{"type":43,"tag":55,"props":437,"children":438},{},[439],{"type":52,"value":440},"You MUST explicitly ask the user which catalog to use.",{"type":52,"value":442}," Do not assume or proceed without confirmation.",{"type":43,"tag":48,"props":444,"children":445},{},[446],{"type":52,"value":447},"Example prompt to user:",{"type":43,"tag":44,"props":449,"children":450},{},[451],{"type":43,"tag":48,"props":452,"children":453},{},[454],{"type":52,"value":455},"\"Which Unity Catalog should I use for this data?\"",{"type":43,"tag":48,"props":457,"children":458},{},[459],{"type":52,"value":460},"When presenting your plan, always show the selected catalog prominently:",{"type":43,"tag":462,"props":463,"children":467},"pre",{"className":464,"code":466,"language":52},[465],"language-text","📍 Output Location: catalog_name.schema_name\n   Volume: \u002FVolumes\u002Fcatalog_name\u002Fschema_name\u002Fraw_data\u002F\n",[468],{"type":43,"tag":380,"props":469,"children":471},{"__ignoreMap":470},"",[472],{"type":52,"value":466},{"type":43,"tag":48,"props":474,"children":475},{},[476],{"type":52,"value":477},"This makes it easy for the user to spot and correct if needed.",{"type":43,"tag":427,"props":479,"children":481},{"id":480},"step-1-gather-requirements",[482],{"type":52,"value":483},"Step 1: Gather Requirements",{"type":43,"tag":48,"props":485,"children":486},{},[487],{"type":52,"value":488},"Ask the user about:",{"type":43,"tag":119,"props":490,"children":491},{},[492,502],{"type":43,"tag":123,"props":493,"children":494},{},[495,500],{"type":43,"tag":55,"props":496,"children":497},{},[498],{"type":52,"value":499},"Catalog\u002FSchema",{"type":52,"value":501}," — Which catalog to use?",{"type":43,"tag":123,"props":503,"children":504},{},[505,510],{"type":43,"tag":55,"props":506,"children":507},{},[508],{"type":52,"value":509},"Domain",{"type":52,"value":511}," — E-commerce, support tickets, IoT, financial? (Use industry terms)",{"type":43,"tag":48,"props":513,"children":514},{},[515,520],{"type":43,"tag":55,"props":516,"children":517},{},[518],{"type":52,"value":519},"If user doesn't specify a story:",{"type":52,"value":521}," Propose one. Don't generate bland data — suggest an incident, anomaly, or trend that shows Databricks value (e.g., \"I'll include a system outage that causes ticket spike and churn — this lets you demo root cause analysis\").",{"type":43,"tag":427,"props":523,"children":525},{"id":524},"step-2-present-plan-with-story",[526],{"type":52,"value":527},"Step 2: Present Plan with Story",{"type":43,"tag":48,"props":529,"children":530},{},[531,533,538],{"type":52,"value":532},"Show a clear specification with ",{"type":43,"tag":55,"props":534,"children":535},{},[536],{"type":52,"value":537},"the business story and your assumptions surfaced",{"type":52,"value":539},":",{"type":43,"tag":462,"props":541,"children":544},{"className":542,"code":543,"language":52},[465],"📍 Output Location: {user_catalog}.support_demo\n   Volume: \u002FVolumes\u002F{user_catalog}\u002Fsupport_demo\u002Fraw_data\u002F\n\n📖 Story: A payment system outage causes support ticket spike. Resolution times\n   degrade, enterprise customers churn, revenue drops $2.3M. With Databricks we\n   identify the root cause, affected customers, and prevent future impact.\n",[545],{"type":43,"tag":380,"props":546,"children":547},{"__ignoreMap":470},[548],{"type":52,"value":543},{"type":43,"tag":190,"props":550,"children":551},{},[552,578],{"type":43,"tag":194,"props":553,"children":554},{},[555],{"type":43,"tag":198,"props":556,"children":557},{},[558,563,568,573],{"type":43,"tag":202,"props":559,"children":560},{},[561],{"type":52,"value":562},"Table",{"type":43,"tag":202,"props":564,"children":565},{},[566],{"type":52,"value":567},"Description",{"type":43,"tag":202,"props":569,"children":570},{},[571],{"type":52,"value":572},"Rows",{"type":43,"tag":202,"props":574,"children":575},{},[576],{"type":52,"value":577},"Key Assumptions",{"type":43,"tag":213,"props":579,"children":580},{},[581,604,627,650],{"type":43,"tag":198,"props":582,"children":583},{},[584,589,594,599],{"type":43,"tag":220,"props":585,"children":586},{},[587],{"type":52,"value":588},"customers",{"type":43,"tag":220,"props":590,"children":591},{},[592],{"type":52,"value":593},"Customer profiles with tier, MRR",{"type":43,"tag":220,"props":595,"children":596},{},[597],{"type":52,"value":598},"10,000",{"type":43,"tag":220,"props":600,"children":601},{},[602],{"type":52,"value":603},"Enterprise 10% but 60% of revenue",{"type":43,"tag":198,"props":605,"children":606},{},[607,612,617,622],{"type":43,"tag":220,"props":608,"children":609},{},[610],{"type":52,"value":611},"tickets",{"type":43,"tag":220,"props":613,"children":614},{},[615],{"type":52,"value":616},"Support tickets with priority, resolution_time",{"type":43,"tag":220,"props":618,"children":619},{},[620],{"type":52,"value":621},"80,000",{"type":43,"tag":220,"props":623,"children":624},{},[625],{"type":52,"value":626},"Spike during outage, SLA breaches",{"type":43,"tag":198,"props":628,"children":629},{},[630,635,640,645],{"type":43,"tag":220,"props":631,"children":632},{},[633],{"type":52,"value":634},"incidents",{"type":43,"tag":220,"props":636,"children":637},{},[638],{"type":52,"value":639},"System events (outages, deployments)",{"type":43,"tag":220,"props":641,"children":642},{},[643],{"type":52,"value":644},"50",{"type":43,"tag":220,"props":646,"children":647},{},[648],{"type":52,"value":649},"Payment outage mid-month",{"type":43,"tag":198,"props":651,"children":652},{},[653,658,663,668],{"type":43,"tag":220,"props":654,"children":655},{},[656],{"type":52,"value":657},"churn_events",{"type":43,"tag":220,"props":659,"children":660},{},[661],{"type":52,"value":662},"Customer cancellations with reason",{"type":43,"tag":220,"props":664,"children":665},{},[666],{"type":52,"value":667},"500",{"type":43,"tag":220,"props":669,"children":670},{},[671],{"type":52,"value":672},"Spike after poor support experience",{"type":43,"tag":48,"props":674,"children":675},{},[676],{"type":43,"tag":55,"props":677,"children":678},{},[679],{"type":52,"value":680},"Business metrics:",{"type":43,"tag":119,"props":682,"children":683},{},[684,695,706],{"type":43,"tag":123,"props":685,"children":686},{},[687,693],{"type":43,"tag":380,"props":688,"children":690},{"className":689},[],[691],{"type":52,"value":692},"customers.mrr",{"type":52,"value":694}," — Revenue at risk ($)",{"type":43,"tag":123,"props":696,"children":697},{},[698,704],{"type":43,"tag":380,"props":699,"children":701},{"className":700},[],[702],{"type":52,"value":703},"tickets.resolution_hours",{"type":52,"value":705}," — SLA performance",{"type":43,"tag":123,"props":707,"children":708},{},[709,715],{"type":43,"tag":380,"props":710,"children":712},{"className":711},[],[713],{"type":52,"value":714},"churn_events.lost_mrr",{"type":52,"value":716}," — Churn impact ($)",{"type":43,"tag":48,"props":718,"children":719},{},[720],{"type":43,"tag":55,"props":721,"children":722},{},[723],{"type":52,"value":724},"The story this data tells:",{"type":43,"tag":119,"props":726,"children":727},{},[728,733,738,743,748],{"type":43,"tag":123,"props":729,"children":730},{},[731],{"type":52,"value":732},"Incident table shows payment outage on March 15",{"type":43,"tag":123,"props":734,"children":735},{},[736],{"type":52,"value":737},"Tickets spike 5x during outage, resolution time degrades from 4h → 18h",{"type":43,"tag":123,"props":739,"children":740},{},[741],{"type":52,"value":742},"Enterprise customers with SLA breaches churn 3 weeks later",{"type":43,"tag":123,"props":744,"children":745},{},[746],{"type":52,"value":747},"Total impact: $2.3M lost MRR, traceable to one incident",{"type":43,"tag":123,"props":749,"children":750},{},[751,756],{"type":43,"tag":55,"props":752,"children":753},{},[754],{"type":52,"value":755},"Databricks value:",{"type":52,"value":757}," Root cause analysis, identify at-risk customers, build alerting",{"type":43,"tag":48,"props":759,"children":760},{},[761,766],{"type":43,"tag":55,"props":762,"children":763},{},[764],{"type":52,"value":765},"Ask user",{"type":52,"value":767},": \"Does this story work? Any adjustments?\"",{"type":43,"tag":427,"props":769,"children":771},{"id":770},"step-3-ask-about-data-features",[772],{"type":52,"value":773},"Step 3: Ask About Data Features",{"type":43,"tag":119,"props":775,"children":778},{"className":776},[777],"contains-task-list",[779,797,810,819,828],{"type":43,"tag":123,"props":780,"children":783},{"className":781},[782],"task-list-item",[784,790,792],{"type":43,"tag":785,"props":786,"children":789},"input",{"checked":787,"disabled":787,"type":788},true,"checkbox",[],{"type":52,"value":791}," Skew (non-uniform distributions) - ",{"type":43,"tag":55,"props":793,"children":794},{},[795],{"type":52,"value":796},"Enabled by default",{"type":43,"tag":123,"props":798,"children":800},{"className":799},[782],[801,804,806],{"type":43,"tag":785,"props":802,"children":803},{"checked":787,"disabled":787,"type":788},[],{"type":52,"value":805}," Joins (referential integrity) - ",{"type":43,"tag":55,"props":807,"children":808},{},[809],{"type":52,"value":796},{"type":43,"tag":123,"props":811,"children":813},{"className":812},[782],[814,817],{"type":43,"tag":785,"props":815,"children":816},{"disabled":787,"type":788},[],{"type":52,"value":818}," Bad data injection (for data quality testing)",{"type":43,"tag":123,"props":820,"children":822},{"className":821},[782],[823,826],{"type":43,"tag":785,"props":824,"children":825},{"disabled":787,"type":788},[],{"type":52,"value":827}," Multi-language text",{"type":43,"tag":123,"props":829,"children":831},{"className":830},[782],[832,835],{"type":43,"tag":785,"props":833,"children":834},{"disabled":787,"type":788},[],{"type":52,"value":836}," Incremental mode (append instead of overwrite)",{"type":43,"tag":427,"props":838,"children":840},{"id":839},"pre-generation-checklist",[841],{"type":52,"value":842},"Pre-Generation Checklist",{"type":43,"tag":119,"props":844,"children":846},{"className":845},[777],[847,863,872,881,890,899],{"type":43,"tag":123,"props":848,"children":850},{"className":849},[782],[851,854,856,861],{"type":43,"tag":785,"props":852,"children":853},{"disabled":787,"type":788},[],{"type":52,"value":855}," ",{"type":43,"tag":55,"props":857,"children":858},{},[859],{"type":52,"value":860},"Catalog confirmed",{"type":52,"value":862}," - User explicitly approved which catalog to use",{"type":43,"tag":123,"props":864,"children":866},{"className":865},[782],[867,870],{"type":43,"tag":785,"props":868,"children":869},{"disabled":787,"type":788},[],{"type":52,"value":871}," Output location shown prominently in plan (easy to spot\u002Fchange)",{"type":43,"tag":123,"props":873,"children":875},{"className":874},[782],[876,879],{"type":43,"tag":785,"props":877,"children":878},{"disabled":787,"type":788},[],{"type":52,"value":880}," Table specification shown and approved",{"type":43,"tag":123,"props":882,"children":884},{"className":883},[782],[885,888],{"type":43,"tag":785,"props":886,"children":887},{"disabled":787,"type":788},[],{"type":52,"value":889}," Assumptions about distributions confirmed",{"type":43,"tag":123,"props":891,"children":893},{"className":892},[782],[894,897],{"type":43,"tag":785,"props":895,"children":896},{"disabled":787,"type":788},[],{"type":52,"value":898}," User confirmed compute preference (Databricks Connect on serverless recommended)",{"type":43,"tag":123,"props":900,"children":902},{"className":901},[782],[903,906],{"type":43,"tag":785,"props":904,"children":905},{"disabled":787,"type":788},[],{"type":52,"value":907}," Data features selected",{"type":43,"tag":48,"props":909,"children":910},{},[911],{"type":43,"tag":55,"props":912,"children":913},{},[914],{"type":52,"value":915},"Do NOT proceed to code generation until user approves the plan, including the catalog.",{"type":43,"tag":427,"props":917,"children":919},{"id":918},"post-generation-validation",[920],{"type":52,"value":921},"Post-Generation Validation",{"type":43,"tag":48,"props":923,"children":924},{},[925,927,933],{"type":52,"value":926},"Use ",{"type":43,"tag":380,"props":928,"children":930},{"className":929},[],[931],{"type":52,"value":932},"databricks experimental aitools tools query",{"type":52,"value":934}," to validate generated data (row counts, distributions, referential integrity). Query parquet files directly:",{"type":43,"tag":462,"props":936,"children":940},{"className":937,"code":938,"language":939,"meta":470,"style":470},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","databricks experimental aitools tools query --warehouse $WAREHOUSE_ID \"\nSELECT COUNT(*) FROM parquet.\\`\u002FVolumes\u002FCATALOG\u002FSCHEMA\u002Fraw_data\u002Fcustomers\\`\n\"\n","bash",[941],{"type":43,"tag":380,"props":942,"children":943},{"__ignoreMap":470},[944,993,1017],{"type":43,"tag":945,"props":946,"children":949},"span",{"class":947,"line":948},"line",1,[950,955,961,966,971,976,981,987],{"type":43,"tag":945,"props":951,"children":953},{"style":952},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[954],{"type":52,"value":8},{"type":43,"tag":945,"props":956,"children":958},{"style":957},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[959],{"type":52,"value":960}," experimental",{"type":43,"tag":945,"props":962,"children":963},{"style":957},[964],{"type":52,"value":965}," aitools",{"type":43,"tag":945,"props":967,"children":968},{"style":957},[969],{"type":52,"value":970}," tools",{"type":43,"tag":945,"props":972,"children":973},{"style":957},[974],{"type":52,"value":975}," query",{"type":43,"tag":945,"props":977,"children":978},{"style":957},[979],{"type":52,"value":980}," --warehouse",{"type":43,"tag":945,"props":982,"children":984},{"style":983},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[985],{"type":52,"value":986}," $WAREHOUSE_ID ",{"type":43,"tag":945,"props":988,"children":990},{"style":989},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[991],{"type":52,"value":992},"\"\n",{"type":43,"tag":945,"props":994,"children":996},{"class":947,"line":995},2,[997,1002,1007,1012],{"type":43,"tag":945,"props":998,"children":999},{"style":957},[1000],{"type":52,"value":1001},"SELECT COUNT(*) FROM parquet.",{"type":43,"tag":945,"props":1003,"children":1004},{"style":983},[1005],{"type":52,"value":1006},"\\`",{"type":43,"tag":945,"props":1008,"children":1009},{"style":957},[1010],{"type":52,"value":1011},"\u002FVolumes\u002FCATALOG\u002FSCHEMA\u002Fraw_data\u002Fcustomers",{"type":43,"tag":945,"props":1013,"children":1014},{"style":983},[1015],{"type":52,"value":1016},"\\`\n",{"type":43,"tag":945,"props":1018,"children":1020},{"class":947,"line":1019},3,[1021],{"type":43,"tag":945,"props":1022,"children":1023},{"style":989},[1024],{"type":52,"value":992},{"type":43,"tag":48,"props":1026,"children":1027},{},[1028,1030,1034],{"type":52,"value":1029},"See ",{"type":43,"tag":236,"props":1031,"children":1032},{"href":256},[1033],{"type":52,"value":256},{"type":52,"value":1035}," for full validation examples.",{"type":43,"tag":89,"props":1037,"children":1039},{"id":1038},"use-databricks-connect-spark-faker-pattern",[1040],{"type":52,"value":1041},"Use Databricks Connect Spark + Faker Pattern",{"type":43,"tag":462,"props":1043,"children":1047},{"className":1044,"code":1045,"language":1046,"meta":470,"style":470},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from databricks.connect import DatabricksSession\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import StringType\nimport pandas as pd\n\n# Setup serverless Spark session\nspark = DatabricksSession.builder.serverless(True).getOrCreate()\n\n# Pandas UDF pattern - import lib INSIDE the function (libs must be installed locally)\n@F.pandas_udf(StringType())\ndef fake_name(ids: pd.Series) -> pd.Series:\n    from faker import Faker  # Import inside UDF\n    fake = Faker()\n    return pd.Series([fake.name() for _ in range(len(ids))])\n\n# Generate with spark.range, apply UDFs\ncustomers_df = spark.range(0, 10000, numPartitions=16).select(\n    F.concat(F.lit(\"CUST-\"), F.lpad(F.col(\"id\").cast(\"string\"), 5, \"0\")).alias(\"customer_id\"),\n    fake_name(F.col(\"id\")).alias(\"name\"),\n)\n\n# Write to Volume as Parquet (default for raw data)\n# Path is a folder with table name: \u002FVolumes\u002Fcatalog\u002Fschema\u002Fraw_data\u002Fcustomers\u002F\nspark.sql(f\"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}\")\nspark.sql(f\"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data\")\ncustomers_df.write.mode(\"overwrite\").parquet(f\"\u002FVolumes\u002F{CATALOG}\u002F{SCHEMA}\u002Fraw_data\u002Fcustomers\")\n","python",[1048],{"type":43,"tag":380,"props":1049,"children":1050},{"__ignoreMap":470},[1051,1059,1067,1075,1084,1093,1102,1111,1119,1128,1137,1146,1155,1164,1173,1181,1190,1199,1208,1217,1226,1234,1243,1252,1261,1270],{"type":43,"tag":945,"props":1052,"children":1053},{"class":947,"line":948},[1054],{"type":43,"tag":945,"props":1055,"children":1056},{},[1057],{"type":52,"value":1058},"from databricks.connect import DatabricksSession\n",{"type":43,"tag":945,"props":1060,"children":1061},{"class":947,"line":995},[1062],{"type":43,"tag":945,"props":1063,"children":1064},{},[1065],{"type":52,"value":1066},"from pyspark.sql import functions as F\n",{"type":43,"tag":945,"props":1068,"children":1069},{"class":947,"line":1019},[1070],{"type":43,"tag":945,"props":1071,"children":1072},{},[1073],{"type":52,"value":1074},"from pyspark.sql.types import StringType\n",{"type":43,"tag":945,"props":1076,"children":1078},{"class":947,"line":1077},4,[1079],{"type":43,"tag":945,"props":1080,"children":1081},{},[1082],{"type":52,"value":1083},"import pandas as pd\n",{"type":43,"tag":945,"props":1085,"children":1087},{"class":947,"line":1086},5,[1088],{"type":43,"tag":945,"props":1089,"children":1090},{"emptyLinePlaceholder":787},[1091],{"type":52,"value":1092},"\n",{"type":43,"tag":945,"props":1094,"children":1096},{"class":947,"line":1095},6,[1097],{"type":43,"tag":945,"props":1098,"children":1099},{},[1100],{"type":52,"value":1101},"# Setup serverless Spark session\n",{"type":43,"tag":945,"props":1103,"children":1105},{"class":947,"line":1104},7,[1106],{"type":43,"tag":945,"props":1107,"children":1108},{},[1109],{"type":52,"value":1110},"spark = DatabricksSession.builder.serverless(True).getOrCreate()\n",{"type":43,"tag":945,"props":1112,"children":1114},{"class":947,"line":1113},8,[1115],{"type":43,"tag":945,"props":1116,"children":1117},{"emptyLinePlaceholder":787},[1118],{"type":52,"value":1092},{"type":43,"tag":945,"props":1120,"children":1122},{"class":947,"line":1121},9,[1123],{"type":43,"tag":945,"props":1124,"children":1125},{},[1126],{"type":52,"value":1127},"# Pandas UDF pattern - import lib INSIDE the function (libs must be installed locally)\n",{"type":43,"tag":945,"props":1129,"children":1131},{"class":947,"line":1130},10,[1132],{"type":43,"tag":945,"props":1133,"children":1134},{},[1135],{"type":52,"value":1136},"@F.pandas_udf(StringType())\n",{"type":43,"tag":945,"props":1138,"children":1140},{"class":947,"line":1139},11,[1141],{"type":43,"tag":945,"props":1142,"children":1143},{},[1144],{"type":52,"value":1145},"def fake_name(ids: pd.Series) -> pd.Series:\n",{"type":43,"tag":945,"props":1147,"children":1149},{"class":947,"line":1148},12,[1150],{"type":43,"tag":945,"props":1151,"children":1152},{},[1153],{"type":52,"value":1154},"    from faker import Faker  # Import inside UDF\n",{"type":43,"tag":945,"props":1156,"children":1158},{"class":947,"line":1157},13,[1159],{"type":43,"tag":945,"props":1160,"children":1161},{},[1162],{"type":52,"value":1163},"    fake = Faker()\n",{"type":43,"tag":945,"props":1165,"children":1167},{"class":947,"line":1166},14,[1168],{"type":43,"tag":945,"props":1169,"children":1170},{},[1171],{"type":52,"value":1172},"    return pd.Series([fake.name() for _ in range(len(ids))])\n",{"type":43,"tag":945,"props":1174,"children":1176},{"class":947,"line":1175},15,[1177],{"type":43,"tag":945,"props":1178,"children":1179},{"emptyLinePlaceholder":787},[1180],{"type":52,"value":1092},{"type":43,"tag":945,"props":1182,"children":1184},{"class":947,"line":1183},16,[1185],{"type":43,"tag":945,"props":1186,"children":1187},{},[1188],{"type":52,"value":1189},"# Generate with spark.range, apply UDFs\n",{"type":43,"tag":945,"props":1191,"children":1193},{"class":947,"line":1192},17,[1194],{"type":43,"tag":945,"props":1195,"children":1196},{},[1197],{"type":52,"value":1198},"customers_df = spark.range(0, 10000, numPartitions=16).select(\n",{"type":43,"tag":945,"props":1200,"children":1202},{"class":947,"line":1201},18,[1203],{"type":43,"tag":945,"props":1204,"children":1205},{},[1206],{"type":52,"value":1207},"    F.concat(F.lit(\"CUST-\"), F.lpad(F.col(\"id\").cast(\"string\"), 5, \"0\")).alias(\"customer_id\"),\n",{"type":43,"tag":945,"props":1209,"children":1211},{"class":947,"line":1210},19,[1212],{"type":43,"tag":945,"props":1213,"children":1214},{},[1215],{"type":52,"value":1216},"    fake_name(F.col(\"id\")).alias(\"name\"),\n",{"type":43,"tag":945,"props":1218,"children":1220},{"class":947,"line":1219},20,[1221],{"type":43,"tag":945,"props":1222,"children":1223},{},[1224],{"type":52,"value":1225},")\n",{"type":43,"tag":945,"props":1227,"children":1229},{"class":947,"line":1228},21,[1230],{"type":43,"tag":945,"props":1231,"children":1232},{"emptyLinePlaceholder":787},[1233],{"type":52,"value":1092},{"type":43,"tag":945,"props":1235,"children":1237},{"class":947,"line":1236},22,[1238],{"type":43,"tag":945,"props":1239,"children":1240},{},[1241],{"type":52,"value":1242},"# Write to Volume as Parquet (default for raw data)\n",{"type":43,"tag":945,"props":1244,"children":1246},{"class":947,"line":1245},23,[1247],{"type":43,"tag":945,"props":1248,"children":1249},{},[1250],{"type":52,"value":1251},"# Path is a folder with table name: \u002FVolumes\u002Fcatalog\u002Fschema\u002Fraw_data\u002Fcustomers\u002F\n",{"type":43,"tag":945,"props":1253,"children":1255},{"class":947,"line":1254},24,[1256],{"type":43,"tag":945,"props":1257,"children":1258},{},[1259],{"type":52,"value":1260},"spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}\")\n",{"type":43,"tag":945,"props":1262,"children":1264},{"class":947,"line":1263},25,[1265],{"type":43,"tag":945,"props":1266,"children":1267},{},[1268],{"type":52,"value":1269},"spark.sql(f\"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data\")\n",{"type":43,"tag":945,"props":1271,"children":1273},{"class":947,"line":1272},26,[1274],{"type":43,"tag":945,"props":1275,"children":1276},{},[1277],{"type":52,"value":1278},"customers_df.write.mode(\"overwrite\").parquet(f\"\u002FVolumes\u002F{CATALOG}\u002F{SCHEMA}\u002Fraw_data\u002Fcustomers\")\n",{"type":43,"tag":48,"props":1280,"children":1281},{},[1282,1287,1288],{"type":43,"tag":55,"props":1283,"children":1284},{},[1285],{"type":52,"value":1286},"Partitions by scale:",{"type":52,"value":855},{"type":43,"tag":380,"props":1289,"children":1291},{"className":1290},[],[1292],{"type":52,"value":1293},"spark.range(N, numPartitions=P)",{"type":43,"tag":119,"props":1295,"children":1296},{},[1297,1302,1307,1312],{"type":43,"tag":123,"props":1298,"children":1299},{},[1300],{"type":52,"value":1301},"\u003C100K rows: 8 partitions",{"type":43,"tag":123,"props":1303,"children":1304},{},[1305],{"type":52,"value":1306},"100K-500K: 16 partitions",{"type":43,"tag":123,"props":1308,"children":1309},{},[1310],{"type":52,"value":1311},"500K-1M: 32 partitions",{"type":43,"tag":123,"props":1313,"children":1314},{},[1315],{"type":52,"value":1316},"1M+: 64+ partitions",{"type":43,"tag":48,"props":1318,"children":1319},{},[1320],{"type":43,"tag":55,"props":1321,"children":1322},{},[1323],{"type":52,"value":1324},"Output formats:",{"type":43,"tag":119,"props":1326,"children":1327},{},[1328,1346,1364],{"type":43,"tag":123,"props":1329,"children":1330},{},[1331,1336,1338,1344],{"type":43,"tag":55,"props":1332,"children":1333},{},[1334],{"type":52,"value":1335},"Parquet to Volume",{"type":52,"value":1337}," (default): ",{"type":43,"tag":380,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":52,"value":1343},"df.write.parquet(\"\u002FVolumes\u002F...\u002Fraw_data\u002Ftable\")",{"type":52,"value":1345}," — raw data for pipelines",{"type":43,"tag":123,"props":1347,"children":1348},{},[1349,1354,1356,1362],{"type":43,"tag":55,"props":1350,"children":1351},{},[1352],{"type":52,"value":1353},"Delta Table",{"type":52,"value":1355},": ",{"type":43,"tag":380,"props":1357,"children":1359},{"className":1358},[],[1360],{"type":52,"value":1361},"df.write.saveAsTable(\"catalog.schema.table\")",{"type":52,"value":1363}," — if user wants queryable tables",{"type":43,"tag":123,"props":1365,"children":1366},{},[1367,1372],{"type":43,"tag":55,"props":1368,"children":1369},{},[1370],{"type":52,"value":1371},"JSON\u002FCSV",{"type":52,"value":1373},": small dimension tables, replicate legacy systems",{"type":43,"tag":89,"props":1375,"children":1377},{"id":1376},"performance-rules",[1378],{"type":52,"value":1379},"Performance Rules",{"type":43,"tag":48,"props":1381,"children":1382},{},[1383,1385,1390],{"type":52,"value":1384},"Generated scripts must be highly performant. ",{"type":43,"tag":55,"props":1386,"children":1387},{},[1388],{"type":52,"value":1389},"Never",{"type":52,"value":1391}," do these:",{"type":43,"tag":190,"props":1393,"children":1394},{},[1395,1416],{"type":43,"tag":194,"props":1396,"children":1397},{},[1398],{"type":43,"tag":198,"props":1399,"children":1400},{},[1401,1406,1411],{"type":43,"tag":202,"props":1402,"children":1403},{},[1404],{"type":52,"value":1405},"Anti-Pattern",{"type":43,"tag":202,"props":1407,"children":1408},{},[1409],{"type":52,"value":1410},"Why It's Slow",{"type":43,"tag":202,"props":1412,"children":1413},{},[1414],{"type":52,"value":1415},"Do This Instead",{"type":43,"tag":213,"props":1417,"children":1418},{},[1419,1444,1467,1493,1511],{"type":43,"tag":198,"props":1420,"children":1421},{},[1422,1427,1432],{"type":43,"tag":220,"props":1423,"children":1424},{},[1425],{"type":52,"value":1426},"Python loops on driver",{"type":43,"tag":220,"props":1428,"children":1429},{},[1430],{"type":52,"value":1431},"Single-threaded, no parallelism",{"type":43,"tag":220,"props":1433,"children":1434},{},[1435,1436,1442],{"type":52,"value":926},{"type":43,"tag":380,"props":1437,"children":1439},{"className":1438},[],[1440],{"type":52,"value":1441},"spark.range()",{"type":52,"value":1443}," + Spark operations",{"type":43,"tag":198,"props":1445,"children":1446},{},[1447,1457,1462],{"type":43,"tag":220,"props":1448,"children":1449},{},[1450,1455],{"type":43,"tag":380,"props":1451,"children":1453},{"className":1452},[],[1454],{"type":52,"value":409},{"type":52,"value":1456}," then iterate",{"type":43,"tag":220,"props":1458,"children":1459},{},[1460],{"type":52,"value":1461},"Brings all data to driver memory",{"type":43,"tag":220,"props":1463,"children":1464},{},[1465],{"type":52,"value":1466},"Keep data in Spark, use DataFrame ops",{"type":43,"tag":198,"props":1468,"children":1469},{},[1470,1475,1480],{"type":43,"tag":220,"props":1471,"children":1472},{},[1473],{"type":52,"value":1474},"Pandas → Spark → Pandas",{"type":43,"tag":220,"props":1476,"children":1477},{},[1478],{"type":52,"value":1479},"Serialization overhead, defeats distribution",{"type":43,"tag":220,"props":1481,"children":1482},{},[1483,1485,1491],{"type":52,"value":1484},"Stay in Spark, use ",{"type":43,"tag":380,"props":1486,"children":1488},{"className":1487},[],[1489],{"type":52,"value":1490},"pandas_udf",{"type":52,"value":1492}," only for UDFs",{"type":43,"tag":198,"props":1494,"children":1495},{},[1496,1501,1506],{"type":43,"tag":220,"props":1497,"children":1498},{},[1499],{"type":52,"value":1500},"Read\u002Fwrite temp files",{"type":43,"tag":220,"props":1502,"children":1503},{},[1504],{"type":52,"value":1505},"Unnecessary I\u002FO",{"type":43,"tag":220,"props":1507,"children":1508},{},[1509],{"type":52,"value":1510},"Chain DataFrame transformations",{"type":43,"tag":198,"props":1512,"children":1513},{},[1514,1519,1524],{"type":43,"tag":220,"props":1515,"children":1516},{},[1517],{"type":52,"value":1518},"Scalar UDFs",{"type":43,"tag":220,"props":1520,"children":1521},{},[1522],{"type":52,"value":1523},"Row-by-row processing",{"type":43,"tag":220,"props":1525,"children":1526},{},[1527,1528,1533],{"type":52,"value":926},{"type":43,"tag":380,"props":1529,"children":1531},{"className":1530},[],[1532],{"type":52,"value":1490},{"type":52,"value":1534}," for batch processing",{"type":43,"tag":48,"props":1536,"children":1537},{},[1538,1543,1544,1549,1551,1556],{"type":43,"tag":55,"props":1539,"children":1540},{},[1541],{"type":52,"value":1542},"Good pattern:",{"type":52,"value":855},{"type":43,"tag":380,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":52,"value":1441},{"type":52,"value":1550}," → Spark transforms → ",{"type":43,"tag":380,"props":1552,"children":1554},{"className":1553},[],[1555],{"type":52,"value":1490},{"type":52,"value":1557}," for Faker → write directly",{"type":43,"tag":89,"props":1559,"children":1561},{"id":1560},"common-patterns",[1562],{"type":52,"value":1563},"Common Patterns",{"type":43,"tag":427,"props":1565,"children":1567},{"id":1566},"weighted-categories-never-uniform",[1568],{"type":52,"value":1569},"Weighted Categories (never uniform)",{"type":43,"tag":462,"props":1571,"children":1573},{"className":1044,"code":1572,"language":1046,"meta":470,"style":470},"F.when(F.rand() \u003C 0.6, \"Free\").when(F.rand() \u003C 0.9, \"Pro\").otherwise(\"Enterprise\")\n",[1574],{"type":43,"tag":380,"props":1575,"children":1576},{"__ignoreMap":470},[1577],{"type":43,"tag":945,"props":1578,"children":1579},{"class":947,"line":948},[1580],{"type":43,"tag":945,"props":1581,"children":1582},{},[1583],{"type":52,"value":1572},{"type":43,"tag":427,"props":1585,"children":1587},{"id":1586},"log-normal-amounts-in-a-pandas-udf",[1588],{"type":52,"value":1589},"Log-Normal Amounts (in a pandas UDF)",{"type":43,"tag":48,"props":1591,"children":1592},{},[1593,1594,1600],{"type":52,"value":926},{"type":43,"tag":380,"props":1595,"children":1597},{"className":1596},[],[1598],{"type":52,"value":1599},"np.random.lognormal(mean, sigma)",{"type":52,"value":1601}," — always positive, long tail:",{"type":43,"tag":119,"props":1603,"children":1604},{},[1605,1618,1631],{"type":43,"tag":123,"props":1606,"children":1607},{},[1608,1610,1616],{"type":52,"value":1609},"Enterprise: ",{"type":43,"tag":380,"props":1611,"children":1613},{"className":1612},[],[1614],{"type":52,"value":1615},"lognormal(7.5, 0.8)",{"type":52,"value":1617}," → ~$1800 median",{"type":43,"tag":123,"props":1619,"children":1620},{},[1621,1623,1629],{"type":52,"value":1622},"Pro: ",{"type":43,"tag":380,"props":1624,"children":1626},{"className":1625},[],[1627],{"type":52,"value":1628},"lognormal(5.5, 0.7)",{"type":52,"value":1630}," → ~$245 median",{"type":43,"tag":123,"props":1632,"children":1633},{},[1634,1636,1642],{"type":52,"value":1635},"Free: ",{"type":43,"tag":380,"props":1637,"children":1639},{"className":1638},[],[1640],{"type":52,"value":1641},"lognormal(4.0, 0.6)",{"type":52,"value":1643}," → ~$55 median",{"type":43,"tag":427,"props":1645,"children":1647},{"id":1646},"date-range-last-6-months",[1648],{"type":52,"value":1649},"Date Range (Last 6 Months)",{"type":43,"tag":462,"props":1651,"children":1653},{"className":1044,"code":1652,"language":1046,"meta":470,"style":470},"END_DATE = datetime.now()\nSTART_DATE = END_DATE - timedelta(days=180)\n",[1654],{"type":43,"tag":380,"props":1655,"children":1656},{"__ignoreMap":470},[1657,1665],{"type":43,"tag":945,"props":1658,"children":1659},{"class":947,"line":948},[1660],{"type":43,"tag":945,"props":1661,"children":1662},{},[1663],{"type":52,"value":1664},"END_DATE = datetime.now()\n",{"type":43,"tag":945,"props":1666,"children":1667},{"class":947,"line":995},[1668],{"type":43,"tag":945,"props":1669,"children":1670},{},[1671],{"type":52,"value":1672},"START_DATE = END_DATE - timedelta(days=180)\n",{"type":43,"tag":427,"props":1674,"children":1676},{"id":1675},"infrastructure-always-create-in-script",[1677],{"type":52,"value":1678},"Infrastructure (always create in script)",{"type":43,"tag":462,"props":1680,"children":1682},{"className":1044,"code":1681,"language":1046,"meta":470,"style":470},"spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}\")\nspark.sql(f\"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data\")\n",[1683],{"type":43,"tag":380,"props":1684,"children":1685},{"__ignoreMap":470},[1686,1693],{"type":43,"tag":945,"props":1687,"children":1688},{"class":947,"line":948},[1689],{"type":43,"tag":945,"props":1690,"children":1691},{},[1692],{"type":52,"value":1260},{"type":43,"tag":945,"props":1694,"children":1695},{"class":947,"line":995},[1696],{"type":43,"tag":945,"props":1697,"children":1698},{},[1699],{"type":52,"value":1269},{"type":43,"tag":427,"props":1701,"children":1703},{"id":1702},"referential-integrity-fk-pattern",[1704],{"type":52,"value":1705},"Referential Integrity (FK pattern)",{"type":43,"tag":48,"props":1707,"children":1708},{},[1709,1711,1716],{"type":52,"value":1710},"Write master table to Delta first, then read back for FK joins (no ",{"type":43,"tag":380,"props":1712,"children":1714},{"className":1713},[],[1715],{"type":52,"value":385},{"type":52,"value":1717}," on serverless):",{"type":43,"tag":462,"props":1719,"children":1721},{"className":1044,"code":1720,"language":1046,"meta":470,"style":470},"# 1. Write master table\ncustomers_df.write.mode(\"overwrite\").saveAsTable(f\"{CATALOG}.{SCHEMA}.customers\")\n\n# 2. Read back for FK lookup\ncustomer_lookup = spark.table(f\"{CATALOG}.{SCHEMA}.customers\").select(\"customer_idx\", \"customer_id\")\n\n# 3. Generate child table with valid FKs via join\norders_df = spark.range(N_ORDERS).select(\n    (F.abs(F.hash(F.col(\"id\"))) % N_CUSTOMERS).alias(\"customer_idx\")\n)\norders_with_fk = orders_df.join(customer_lookup, on=\"customer_idx\")\n",[1722],{"type":43,"tag":380,"props":1723,"children":1724},{"__ignoreMap":470},[1725,1733,1741,1748,1756,1764,1771,1779,1787,1795,1802],{"type":43,"tag":945,"props":1726,"children":1727},{"class":947,"line":948},[1728],{"type":43,"tag":945,"props":1729,"children":1730},{},[1731],{"type":52,"value":1732},"# 1. Write master table\n",{"type":43,"tag":945,"props":1734,"children":1735},{"class":947,"line":995},[1736],{"type":43,"tag":945,"props":1737,"children":1738},{},[1739],{"type":52,"value":1740},"customers_df.write.mode(\"overwrite\").saveAsTable(f\"{CATALOG}.{SCHEMA}.customers\")\n",{"type":43,"tag":945,"props":1742,"children":1743},{"class":947,"line":1019},[1744],{"type":43,"tag":945,"props":1745,"children":1746},{"emptyLinePlaceholder":787},[1747],{"type":52,"value":1092},{"type":43,"tag":945,"props":1749,"children":1750},{"class":947,"line":1077},[1751],{"type":43,"tag":945,"props":1752,"children":1753},{},[1754],{"type":52,"value":1755},"# 2. Read back for FK lookup\n",{"type":43,"tag":945,"props":1757,"children":1758},{"class":947,"line":1086},[1759],{"type":43,"tag":945,"props":1760,"children":1761},{},[1762],{"type":52,"value":1763},"customer_lookup = spark.table(f\"{CATALOG}.{SCHEMA}.customers\").select(\"customer_idx\", \"customer_id\")\n",{"type":43,"tag":945,"props":1765,"children":1766},{"class":947,"line":1095},[1767],{"type":43,"tag":945,"props":1768,"children":1769},{"emptyLinePlaceholder":787},[1770],{"type":52,"value":1092},{"type":43,"tag":945,"props":1772,"children":1773},{"class":947,"line":1104},[1774],{"type":43,"tag":945,"props":1775,"children":1776},{},[1777],{"type":52,"value":1778},"# 3. Generate child table with valid FKs via join\n",{"type":43,"tag":945,"props":1780,"children":1781},{"class":947,"line":1113},[1782],{"type":43,"tag":945,"props":1783,"children":1784},{},[1785],{"type":52,"value":1786},"orders_df = spark.range(N_ORDERS).select(\n",{"type":43,"tag":945,"props":1788,"children":1789},{"class":947,"line":1121},[1790],{"type":43,"tag":945,"props":1791,"children":1792},{},[1793],{"type":52,"value":1794},"    (F.abs(F.hash(F.col(\"id\"))) % N_CUSTOMERS).alias(\"customer_idx\")\n",{"type":43,"tag":945,"props":1796,"children":1797},{"class":947,"line":1130},[1798],{"type":43,"tag":945,"props":1799,"children":1800},{},[1801],{"type":52,"value":1225},{"type":43,"tag":945,"props":1803,"children":1804},{"class":947,"line":1139},[1805],{"type":43,"tag":945,"props":1806,"children":1807},{},[1808],{"type":52,"value":1809},"orders_with_fk = orders_df.join(customer_lookup, on=\"customer_idx\")\n",{"type":43,"tag":89,"props":1811,"children":1813},{"id":1812},"setup",[1814],{"type":52,"value":1815},"Setup",{"type":43,"tag":48,"props":1817,"children":1818},{},[1819,1821,1827],{"type":52,"value":1820},"Requires Python 3.12 and databricks-connect>=16.4. Use ",{"type":43,"tag":380,"props":1822,"children":1824},{"className":1823},[],[1825],{"type":52,"value":1826},"uv",{"type":52,"value":539},{"type":43,"tag":462,"props":1829,"children":1831},{"className":937,"code":1830,"language":939,"meta":470,"style":470},"uv pip install \"databricks-connect>=16.4,\u003C17.4\" faker numpy pandas holidays\n",[1832],{"type":43,"tag":380,"props":1833,"children":1834},{"__ignoreMap":470},[1835],{"type":43,"tag":945,"props":1836,"children":1837},{"class":947,"line":948},[1838,1842,1847,1852,1857,1862,1867,1872,1877,1882],{"type":43,"tag":945,"props":1839,"children":1840},{"style":952},[1841],{"type":52,"value":1826},{"type":43,"tag":945,"props":1843,"children":1844},{"style":957},[1845],{"type":52,"value":1846}," pip",{"type":43,"tag":945,"props":1848,"children":1849},{"style":957},[1850],{"type":52,"value":1851}," install",{"type":43,"tag":945,"props":1853,"children":1854},{"style":989},[1855],{"type":52,"value":1856}," \"",{"type":43,"tag":945,"props":1858,"children":1859},{"style":957},[1860],{"type":52,"value":1861},"databricks-connect>=16.4,\u003C17.4",{"type":43,"tag":945,"props":1863,"children":1864},{"style":989},[1865],{"type":52,"value":1866},"\"",{"type":43,"tag":945,"props":1868,"children":1869},{"style":957},[1870],{"type":52,"value":1871}," faker",{"type":43,"tag":945,"props":1873,"children":1874},{"style":957},[1875],{"type":52,"value":1876}," numpy",{"type":43,"tag":945,"props":1878,"children":1879},{"style":957},[1880],{"type":52,"value":1881}," pandas",{"type":43,"tag":945,"props":1883,"children":1884},{"style":957},[1885],{"type":52,"value":1886}," holidays\n",{"type":43,"tag":89,"props":1888,"children":1890},{"id":1889},"related-skills",[1891],{"type":52,"value":1892},"Related Skills",{"type":43,"tag":119,"props":1894,"children":1895},{},[1896,1906],{"type":43,"tag":123,"props":1897,"children":1898},{},[1899,1904],{"type":43,"tag":55,"props":1900,"children":1901},{},[1902],{"type":52,"value":1903},"databricks-unity-catalog",{"type":52,"value":1905}," — Managing catalogs, schemas, and volumes",{"type":43,"tag":123,"props":1907,"children":1908},{},[1909,1914],{"type":43,"tag":55,"props":1910,"children":1911},{},[1912],{"type":52,"value":1913},"databricks-dabs",{"type":52,"value":1915}," — DABs for production deployment",{"type":43,"tag":89,"props":1917,"children":1919},{"id":1918},"common-issues",[1920],{"type":52,"value":1921},"Common Issues",{"type":43,"tag":190,"props":1923,"children":1924},{},[1925,1941],{"type":43,"tag":194,"props":1926,"children":1927},{},[1928],{"type":43,"tag":198,"props":1929,"children":1930},{},[1931,1936],{"type":43,"tag":202,"props":1932,"children":1933},{},[1934],{"type":52,"value":1935},"Issue",{"type":43,"tag":202,"props":1937,"children":1938},{},[1939],{"type":52,"value":1940},"Solution",{"type":43,"tag":213,"props":1942,"children":1943},{},[1944,1969,1987,2013,2026,2061,2118],{"type":43,"tag":198,"props":1945,"children":1946},{},[1947,1956],{"type":43,"tag":220,"props":1948,"children":1949},{},[1950],{"type":43,"tag":380,"props":1951,"children":1953},{"className":1952},[],[1954],{"type":52,"value":1955},"ModuleNotFoundError: faker",{"type":43,"tag":220,"props":1957,"children":1958},{},[1959,1961,1967],{"type":52,"value":1960},"Install locally: ",{"type":43,"tag":380,"props":1962,"children":1964},{"className":1963},[],[1965],{"type":52,"value":1966},"uv pip install faker",{"type":52,"value":1968},", import inside UDF",{"type":43,"tag":198,"props":1970,"children":1971},{},[1972,1977],{"type":43,"tag":220,"props":1973,"children":1974},{},[1975],{"type":52,"value":1976},"Faker UDF is slow",{"type":43,"tag":220,"props":1978,"children":1979},{},[1980,1981,1986],{"type":52,"value":926},{"type":43,"tag":380,"props":1982,"children":1984},{"className":1983},[],[1985],{"type":52,"value":1490},{"type":52,"value":1534},{"type":43,"tag":198,"props":1988,"children":1989},{},[1990,1995],{"type":43,"tag":220,"props":1991,"children":1992},{},[1993],{"type":52,"value":1994},"Out of memory",{"type":43,"tag":220,"props":1996,"children":1997},{},[1998,2000,2006,2008],{"type":52,"value":1999},"Increase ",{"type":43,"tag":380,"props":2001,"children":2003},{"className":2002},[],[2004],{"type":52,"value":2005},"numPartitions",{"type":52,"value":2007}," in ",{"type":43,"tag":380,"props":2009,"children":2011},{"className":2010},[],[2012],{"type":52,"value":1441},{"type":43,"tag":198,"props":2014,"children":2015},{},[2016,2021],{"type":43,"tag":220,"props":2017,"children":2018},{},[2019],{"type":52,"value":2020},"Referential integrity errors",{"type":43,"tag":220,"props":2022,"children":2023},{},[2024],{"type":52,"value":2025},"Write master table to Delta first, read back for FK joins",{"type":43,"tag":198,"props":2027,"children":2028},{},[2029,2038],{"type":43,"tag":220,"props":2030,"children":2031},{},[2032],{"type":43,"tag":380,"props":2033,"children":2035},{"className":2034},[],[2036],{"type":52,"value":2037},"PERSIST TABLE is not supported on serverless",{"type":43,"tag":220,"props":2039,"children":2040},{},[2041,2059],{"type":43,"tag":55,"props":2042,"children":2043},{},[2044,2046,2051,2052,2057],{"type":52,"value":2045},"NEVER use ",{"type":43,"tag":380,"props":2047,"children":2049},{"className":2048},[],[2050],{"type":52,"value":385},{"type":52,"value":387},{"type":43,"tag":380,"props":2053,"children":2055},{"className":2054},[],[2056],{"type":52,"value":393},{"type":52,"value":2058}," with serverless",{"type":52,"value":2060}," - write to Delta table first, then read back",{"type":43,"tag":198,"props":2062,"children":2063},{},[2064,2083],{"type":43,"tag":220,"props":2065,"children":2066},{},[2067,2073,2075,2081],{"type":43,"tag":380,"props":2068,"children":2070},{"className":2069},[],[2071],{"type":52,"value":2072},"F.window",{"type":52,"value":2074}," vs ",{"type":43,"tag":380,"props":2076,"children":2078},{"className":2077},[],[2079],{"type":52,"value":2080},"Window",{"type":52,"value":2082}," confusion",{"type":43,"tag":220,"props":2084,"children":2085},{},[2086,2087,2093,2095,2101,2103,2109,2111,2116],{"type":52,"value":926},{"type":43,"tag":380,"props":2088,"children":2090},{"className":2089},[],[2091],{"type":52,"value":2092},"from pyspark.sql.window import Window",{"type":52,"value":2094}," for ",{"type":43,"tag":380,"props":2096,"children":2098},{"className":2097},[],[2099],{"type":52,"value":2100},"row_number()",{"type":52,"value":2102},", ",{"type":43,"tag":380,"props":2104,"children":2106},{"className":2105},[],[2107],{"type":52,"value":2108},"rank()",{"type":52,"value":2110},", etc. ",{"type":43,"tag":380,"props":2112,"children":2114},{"className":2113},[],[2115],{"type":52,"value":2072},{"type":52,"value":2117}," is for streaming only.",{"type":43,"tag":198,"props":2119,"children":2120},{},[2121,2126],{"type":43,"tag":220,"props":2122,"children":2123},{},[2124],{"type":52,"value":2125},"Broadcast variables not supported",{"type":43,"tag":220,"props":2127,"children":2128},{},[2129],{"type":43,"tag":55,"props":2130,"children":2131},{},[2132,2133,2139],{"type":52,"value":2045},{"type":43,"tag":380,"props":2134,"children":2136},{"className":2135},[],[2137],{"type":52,"value":2138},"spark.sparkContext.broadcast()",{"type":52,"value":2058},{"type":43,"tag":48,"props":2141,"children":2142},{},[2143,2144,2148],{"type":52,"value":1029},{"type":43,"tag":236,"props":2145,"children":2146},{"href":256},[2147],{"type":52,"value":256},{"type":52,"value":2149}," for full troubleshooting guide.",{"type":43,"tag":2151,"props":2152,"children":2153},"style",{},[2154],{"type":52,"value":2155},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2157,"total":2274},[2158,2175,2187,2204,2221,2241,2252],{"slug":2159,"name":2159,"fn":2160,"description":2161,"org":2162,"tags":2163,"stars":23,"repoUrl":24,"updatedAt":2174},"databricks-agent-bricks","create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2164,2167,2168,2171],{"name":2165,"slug":2166,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2169,"slug":2170,"type":15},"Knowledge Management","knowledge-management",{"name":2172,"slug":2173,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":2176,"name":2176,"fn":2177,"description":2178,"org":2179,"tags":2180,"stars":23,"repoUrl":24,"updatedAt":2186},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2181,2182,2183],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":2184,"slug":2185,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":2188,"name":2188,"fn":2189,"description":2190,"org":2191,"tags":2192,"stars":23,"repoUrl":24,"updatedAt":2203},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2193,2196,2197,2200],{"name":2194,"slug":2195,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":2198,"slug":2199,"type":15},"Docker","docker",{"name":2201,"slug":2202,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":2205,"name":2205,"fn":2206,"description":2207,"org":2208,"tags":2209,"stars":23,"repoUrl":24,"updatedAt":2220},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2210,2213,2216,2219],{"name":2211,"slug":2212,"type":15},"Analytics","analytics",{"name":2214,"slug":2215,"type":15},"Dashboards","dashboards",{"name":2217,"slug":2218,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":2222,"name":2222,"fn":2223,"description":2224,"org":2225,"tags":2226,"stars":23,"repoUrl":24,"updatedAt":2240},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2227,2228,2231,2234,2237],{"name":9,"slug":8,"type":15},{"name":2229,"slug":2230,"type":15},"Design","design",{"name":2232,"slug":2233,"type":15},"Frontend","frontend",{"name":2235,"slug":2236,"type":15},"React","react",{"name":2238,"slug":2239,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":2242,"name":2242,"fn":2243,"description":2244,"org":2245,"tags":2246,"stars":23,"repoUrl":24,"updatedAt":2251},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2247,2248,2249,2250],{"name":2211,"slug":2212,"type":15},{"name":2214,"slug":2215,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":2253,"name":2253,"fn":2254,"description":2255,"org":2256,"tags":2257,"stars":23,"repoUrl":24,"updatedAt":2273},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2258,2259,2262,2265,2268,2270],{"name":9,"slug":8,"type":15},{"name":2260,"slug":2261,"type":15},"FastAPI","fastapi",{"name":2263,"slug":2264,"type":15},"Flask","flask",{"name":2266,"slug":2267,"type":15},"Gradio","gradio",{"name":2269,"slug":1046,"type":15},"Python",{"name":2271,"slug":2272,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",31,{"items":2276,"total":2274},[2277,2284,2290,2297,2304,2312,2319,2328,2339,2355,2368,2381],{"slug":2159,"name":2159,"fn":2160,"description":2161,"org":2278,"tags":2279,"stars":23,"repoUrl":24,"updatedAt":2174},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2280,2281,2282,2283],{"name":2165,"slug":2166,"type":15},{"name":9,"slug":8,"type":15},{"name":2169,"slug":2170,"type":15},{"name":2172,"slug":2173,"type":15},{"slug":2176,"name":2176,"fn":2177,"description":2178,"org":2285,"tags":2286,"stars":23,"repoUrl":24,"updatedAt":2186},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2287,2288,2289],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":2184,"slug":2185,"type":15},{"slug":2188,"name":2188,"fn":2189,"description":2190,"org":2291,"tags":2292,"stars":23,"repoUrl":24,"updatedAt":2203},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2293,2294,2295,2296],{"name":2194,"slug":2195,"type":15},{"name":9,"slug":8,"type":15},{"name":2198,"slug":2199,"type":15},{"name":2201,"slug":2202,"type":15},{"slug":2205,"name":2205,"fn":2206,"description":2207,"org":2298,"tags":2299,"stars":23,"repoUrl":24,"updatedAt":2220},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2300,2301,2302,2303],{"name":2211,"slug":2212,"type":15},{"name":2214,"slug":2215,"type":15},{"name":2217,"slug":2218,"type":15},{"name":9,"slug":8,"type":15},{"slug":2222,"name":2222,"fn":2223,"description":2224,"org":2305,"tags":2306,"stars":23,"repoUrl":24,"updatedAt":2240},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2307,2308,2309,2310,2311],{"name":9,"slug":8,"type":15},{"name":2229,"slug":2230,"type":15},{"name":2232,"slug":2233,"type":15},{"name":2235,"slug":2236,"type":15},{"name":2238,"slug":2239,"type":15},{"slug":2242,"name":2242,"fn":2243,"description":2244,"org":2313,"tags":2314,"stars":23,"repoUrl":24,"updatedAt":2251},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2315,2316,2317,2318],{"name":2211,"slug":2212,"type":15},{"name":2214,"slug":2215,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"slug":2253,"name":2253,"fn":2254,"description":2255,"org":2320,"tags":2321,"stars":23,"repoUrl":24,"updatedAt":2273},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2322,2323,2324,2325,2326,2327],{"name":9,"slug":8,"type":15},{"name":2260,"slug":2261,"type":15},{"name":2263,"slug":2264,"type":15},{"name":2266,"slug":2267,"type":15},{"name":2269,"slug":1046,"type":15},{"name":2271,"slug":2272,"type":15},{"slug":38,"name":38,"fn":2329,"description":2330,"org":2331,"tags":2332,"stars":23,"repoUrl":24,"updatedAt":2338},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2333,2336,2337],{"name":2334,"slug":2335,"type":15},"Authentication","authentication",{"name":2194,"slug":2195,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":1913,"name":1913,"fn":2340,"description":2341,"org":2342,"tags":2343,"stars":23,"repoUrl":24,"updatedAt":2354},"manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2344,2347,2350,2351],{"name":2345,"slug":2346,"type":15},"Automation","automation",{"name":2348,"slug":2349,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":2352,"slug":2353,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":2356,"name":2356,"fn":2357,"description":2358,"org":2359,"tags":2360,"stars":23,"repoUrl":24,"updatedAt":2367},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2361,2362,2363,2364],{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":2365,"slug":2366,"type":15},"SQL","sql","2026-07-31T05:53:32.561877",{"slug":2369,"name":2369,"fn":2370,"description":2371,"org":2372,"tags":2373,"stars":23,"repoUrl":24,"updatedAt":2380},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2374,2375,2378,2379],{"name":17,"slug":18,"type":15},{"name":2376,"slug":2377,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":2365,"slug":2366,"type":15},"2026-07-12T08:04:08.678282",{"slug":2382,"name":2382,"fn":2383,"description":2384,"org":2385,"tags":2386,"stars":23,"repoUrl":24,"updatedAt":2394},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2387,2388,2391],{"name":9,"slug":8,"type":15},{"name":2389,"slug":2390,"type":15},"Documentation","documentation",{"name":2392,"slug":2393,"type":15},"Reference","reference","2026-07-15T05:41:34.697746"]