[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-databricks-databricks-ml-training":3,"mdc-ixglg4-key":33,"related-repo-databricks-databricks-ml-training":3425,"related-org-databricks-databricks-ml-training":3544},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"databricks-ml-training","train machine learning models on Databricks","Train ML models on Databricks. Use for: classification\u002Fregression\u002Fdeep-learning (XGBoost, scikit-learn, LightGBM, PyTorch) with Optuna, @prod\u002F@challenger aliases, batch scoring (spark_udf for plain models, fe.score_batch for feature-store-backed), custom PyFunc, custom ResponsesAgent (LangGraph + UC Function\u002FVector Search); UC feature tables + FeatureLookup + point-in-time joins + Lakebase online store; declarative Feature Views (create_feature, DeltaTableSource, RollingWindow\u002FSlidingWindow\u002FTumblingWindow, materialize_features, streaming Kafka features). NOT for: endpoint ops (databricks-model-serving), MLflow evaluation (databricks-mlflow-evaluation).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"databricks","Databricks","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatabricks.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":17,"slug":18,"type":15},"Deep Learning","deep-learning",{"name":20,"slug":21,"type":15},"Machine Learning","machine-learning",{"name":9,"slug":8,"type":15},204,"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills","2026-07-12T08:04:42.666956",null,60,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fdatabricks\u002Fdatabricks-agent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdatabricks\u002Fcopilot\u002Fskills\u002Fdatabricks-ml-training","---\nname: databricks-ml-training\ndescription: \"Train ML models on Databricks. Use for: classification\u002Fregression\u002Fdeep-learning (XGBoost, scikit-learn, LightGBM, PyTorch) with Optuna, @prod\u002F@challenger aliases, batch scoring (spark_udf for plain models, fe.score_batch for feature-store-backed), custom PyFunc, custom ResponsesAgent (LangGraph + UC Function\u002FVector Search); UC feature tables + FeatureLookup + point-in-time joins + Lakebase online store; declarative Feature Views (create_feature, DeltaTableSource, RollingWindow\u002FSlidingWindow\u002FTumblingWindow, materialize_features, streaming Kafka features). NOT for: endpoint ops (databricks-model-serving), MLflow evaluation (databricks-mlflow-evaluation).\"\ncompatibility: Requires databricks CLI (>= v0.294.0)\nmetadata:\n  version: \"0.1.0\"\nparent: databricks-core\n---\n\n# ML Training on Databricks\n\n**FIRST**: Use the parent `databricks-core` skill for CLI basics, authentication, and profile selection.\n\nTrain with MLflow → register to Unity Catalog → consume the **same artifact** as either a batch Spark UDF over Delta or (when low-latency is required) a real-time serving endpoint.\n\n> **Always train on Databricks** (serverless job or notebook), never in the local Python process the agent is running in. Local training has no access to the silver tables, no MLflow tracking server, no UC registry path, and dies if the chat session drops — submit `databricks jobs submit --no-wait` (see \"Train + deploy as a serverless job\" below). Only fall back to local execution if the user explicitly asks for it.\n\nIf you need to deploy a real time model serving endpoint **after** the model is registered (creating endpoints, traffic config, version-swapping, querying, Foundation Model API endpoints), see [databricks-model-serving](..\u002Fdatabricks-model-serving\u002FSKILL.md).\n\n| Consumption | When | How |\n|---|---|---|\n| **Batch UDF** | Dashboards, daily\u002Fhourly scores, predictions read by Genie\u002FDashboards or an app (often synced to a Lakebase table) | `mlflow.pyfunc.spark_udf(...)` → `INSERT INTO gold_predictions`. **If the model was logged with `fe.log_model(training_set=...)`, use `fe.score_batch()` instead** — see the [Feature Engineering](#feature-engineering-feature-store--feature-views) section below. |\n| **Real-time endpoint** | Score on a user action (fraud at authorization, rec at page load) — sub-100ms | `mlflow.deployments.get_deploy_client()` (classical) \u002F `agents.deploy()` (agents). Endpoint lifecycle: see [databricks-model-serving](..\u002Fdatabricks-model-serving\u002FSKILL.md). |\n\n## Default Canonical flow\n\n```\nsilver_\u003Cfeatures>  +  silver_\u003Clabels>\n        ▼\n   notebook (as a serverless job):\n   ├── train with mlflow.autolog (XGBoost \u002F sklearn \u002F etc.)\n   ├── mlflow.register_model → UC: {catalog}.{schema}.{model}\n   ├── set_registered_model_alias(name, \"prod\", version)\n   └── spark_udf(@prod) over latest features → MERGE into gold_predictions\n        ▼\ngold_\u003Centity>_predictions   ◄── dashboards, apps, Genie read this\n```\n\n> **Feature-store-backed models diverge here.** If training used `fe.log_model(training_set=...)`, replace `spark_udf(@prod)` with `fe.score_batch(model_uri, df=\u003Ckeys_only>)` — it auto-joins features via the model's registered feature lineage. See the [Feature Engineering](#feature-engineering-feature-store--feature-views) section below.\n\nOne notebook, one artifact. Re-running = retraining. Gold is where truth lives — read paths never call the model directly. Keep label-window logic (`failure occurred within 7 days`) in the notebook during dev; once stable, promote to a silver materialized view in SDP.\n\n---\n\n## Train and register (the 90% case)\n\n`mlflow.autolog()` captures params, metrics, code, and the model artifact for every run; `registered_model_name=...` auto-registers the best run to UC (auto-incremented version). Wrap training with **Optuna** so each trial is a child run and the best one is what gets registered.\n\n**Always `mlflow.set_registry_uri(\"databricks-uc\")`** — without it, models land in the deprecated workspace registry. **The experiment's parent folder must exist** — `set_experiment` does NOT auto-create it (fails with `NOT_FOUND: Parent directory does not exist`). Pre-create it once with `databricks workspace mkdirs` before the job runs.\n\n```bash\n# Once per project — create the parent folder for the MLflow experiment.\ndatabricks workspace mkdirs \u002FUsers\u002Fme@example.com\u002Fturbine_project\n```\n\nUse the Databricks notebook source format (`# Databricks notebook source` header, `# COMMAND ----------` separators, `# MAGIC %md`\u002F`%sql` magics for markdown\u002FSQL cells):\n\n```python\n# Databricks notebook source\n# MAGIC %md\n# MAGIC # Turbine failure prediction\n# MAGIC\n# MAGIC Train an XGBoost classifier on engineered turbine telemetry features.\n# MAGIC ## Data exploration\n\n# COMMAND ----------\n\n# (basic data exploration — class balance, schema sanity, etc.)\n\n# COMMAND ----------\n# MAGIC %md\n# MAGIC ## Training the model\n\n# COMMAND ----------\n\nimport mlflow, mlflow.xgboost, optuna\nfrom mlflow.tracking import MlflowClient\nfrom xgboost import XGBClassifier\nfrom sklearn.metrics import roc_auc_score\n\nmlflow.set_registry_uri(\"databricks-uc\")\nmlflow.set_experiment(\"\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Fmlflow_experiment\")\n\nCATALOG, SCHEMA, NAME = \"ai_demo_gen\", \"wind_farm\", \"turbine_failure\"\nFULL_NAME = f\"{CATALOG}.{SCHEMA}.{NAME}\"\n\n# Autolog WITHOUT registered_model_name — otherwise every Optuna trial registers a new UC\n# version, and a max-by-version pick lands on the last trial to finish, not the best one.\nmlflow.xgboost.autolog(log_input_examples=True)\n\n# For imbalanced labels: stratify the split, set scale_pos_weight = neg\u002Fpos.\ndef objective(trial):\n    params = {\n        \"n_estimators\":  trial.suggest_int(\"n_estimators\", 100, 400),\n        \"max_depth\":     trial.suggest_int(\"max_depth\", 3, 10),\n        \"learning_rate\": trial.suggest_float(\"learning_rate\", 0.01, 0.3, log=True),\n    }\n    with mlflow.start_run(nested=True):\n        m = XGBClassifier(**params).fit(X_train, y_train)\n        return roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])\n\nwith mlflow.start_run(run_name=\"hpo\") as parent:\n    study = optuna.create_study(direction=\"maximize\")\n    study.optimize(objective, n_trials=20)\n\n# COMMAND ----------\n# MAGIC %md\n# MAGIC ## Retrain best params and register\n\n# COMMAND ----------\n# Retrain on the winning trial's params explicitly, then register that single model.\nwith mlflow.start_run(run_name=\"best\"):\n    best = XGBClassifier(**study.best_params).fit(X_train, y_train)\n    mlflow.log_metric(\"val_auc\", study.best_value)\n    info = mlflow.xgboost.log_model(best, name=\"model\", registered_model_name=FULL_NAME)\n\n# Stages are deprecated — UC uses movable aliases. Repoint @prod at the version we just registered.\nclient = MlflowClient(registry_uri=\"databricks-uc\")\nclient.set_registered_model_alias(FULL_NAME, \"prod\", info.registered_model_version)\n```\n\n**Framework autolog**: `mlflow.{sklearn,xgboost,lightgbm,pytorch,tensorflow,spark}.autolog()`.\n\n**Aliases, not stages**: UC dropped `Staging`\u002F`Production`. Use movable `@prod`\u002F`@challenger`; load with `models:\u002F{full_name}@prod`. Promoting a new version is one `set_registered_model_alias` call.\n\n---\n\n## Consume: batch scoring over Delta\n\nThe cheap, default path **for models NOT backed by feature tables**. Load the registered model as a Spark UDF and score a Delta table; write predictions to a gold table that downstream consumers read. **For feature-store-backed models** (logged with `fe.log_model(training_set=...)`), skip `spark_udf` entirely and use `fe.score_batch()` — see the [Feature Engineering](#feature-engineering-feature-store--feature-views) section.\n\n```python\n# COMMAND ----------\n# MAGIC %md\n# MAGIC ## Score and save to a gold predictions table\n\n# COMMAND ----------\nimport mlflow\nfrom pyspark.sql import functions as F\n\n# env_manager rules:\n#   \"local\"     → same runtime as training (same notebook\u002Fjob). Fastest, default in dev\u002Fdemo.\n#   \"virtualenv\"→ different runtime than training; rebuilds the model's env.\n#   \"uv\"        → same as virtualenv but faster (MLflow ≥ 2.22).\npredict = mlflow.pyfunc.spark_udf(\n    spark,\n    model_uri=f\"models:\u002F{FULL_NAME}@prod\",\n    env_manager=\"local\",\n)\n\nfeatures = spark.table(f\"{CATALOG}.{SCHEMA}.silver_turbine_features_latest\")\nfeature_cols = [c for c in features.columns if c != \"turbine_id\"]   # exclude the join key\nscored = features.withColumn(\"risk_score\", predict(*[features[c] for c in feature_cols]))\n\n# Overwrite-per-run pattern for \"latest score per entity\":\nscored.select(\"turbine_id\", \"risk_score\", F.current_timestamp().alias(\"scored_at\")) \\\n    .write.mode(\"overwrite\").saveAsTable(f\"{CATALOG}.{SCHEMA}.gold_turbine_predictions\")\n```\n\nFor incremental scoring with history, MERGE into the predictions table instead of overwrite.\n\n---\n\n## Real-time serving (when required)\n\nAfter registering a model to UC, deploy it behind a Model Serving endpoint. The dev-side call is `mlflow.deployments.get_deploy_client(\"databricks\").create_endpoint(...)` for classical ML or `agents.deploy(...)` for `ResponsesAgent`s. First deploy is ~5 min for classical ML.\n\nFor endpoint create \u002F update \u002F version-swap, traffic config, AI Gateway, querying, the `state.ready` + `state.config_update` two-field readiness check, and Foundation Model API endpoints, see **[databricks-model-serving](..\u002Fdatabricks-model-serving\u002FSKILL.md)**.\n\n---\n\n## Train + deploy as a serverless job\n\nTraining notebooks run a few minutes (Optuna + UC register; endpoint warmup adds 5–15 min if you also deploy). Submit as a serverless one-time run so the CLI doesn't block. The notebook ends with `dbutils.notebook.exit(json.dumps({...}))` so the structured result (`model_version`, `val_auc`, `endpoint_name`) reaches `.notebook_output.result`.\n\n```bash\n# 1. Upload the training notebook\ndatabricks workspace import \u002FWorkspace\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Ftrain \\\n  --file .\u002Ftrain_notebook.py --format SOURCE --language PYTHON --overwrite\n\n# 2. Submit as serverless one-time run (returns {\"run_id\": N} immediately with --no-wait)\nRUN_ID=$(databricks jobs submit --no-wait --json '{\n  \"run_name\": \"turbine-train-and-deploy\",\n  \"tasks\": [{\n    \"task_key\": \"train\",\n    \"notebook_task\": {\"notebook_path\": \"\u002FWorkspace\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Ftrain\"},\n    \"environment_key\": \"ml_env\"\n  }],\n  \"environments\": [{\n    \"environment_key\": \"ml_env\",\n    \"spec\": {\n      \"client\": \"4\",\n      \"dependencies\": [\"mlflow==3.1.0\", \"xgboost==2.1.3\", \"optuna==4.1.0\", \"scikit-learn==1.5.2\"]\n    }\n  }]\n}' | jq -r .run_id)\n\n# 3. Poll until a terminal life_cycle_state.\nfor _ in $(seq 60); do\n  STATE=$(databricks jobs get-run \"$RUN_ID\" | jq -r '.state.life_cycle_state \u002F\u002F \"UNKNOWN\"')\n  echo \"$(date +%H:%M:%S) $STATE\"\n  [[ \"$STATE\" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] && break\n  sleep 30\ndone\n[[ \"$STATE\" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] || { databricks jobs cancel-run \"$RUN_ID\"; exit 1; }\n\n# life_cycle_state TERMINATED only means \"the run ended\" — check result_state.\nRESULT=$(databricks jobs get-run \"$RUN_ID\" | jq -r '.state.result_state \u002F\u002F \"UNKNOWN\"')\necho \"result_state=$RESULT\"\n[[ \"$RESULT\" == \"SUCCESS\" ]] || { echo \"Run did not succeed\"; exit 1; }\n\n# 4. Pull structured output via the TASK run_id (NOT the submit run_id).\nTASK_RUN_ID=$(databricks jobs get-run \"$RUN_ID\" | jq -r '.tasks[0].run_id')\ndatabricks jobs get-run-output \"$TASK_RUN_ID\" | jq '.notebook_output.result'\n# → '{\"model_version\":\"3\",\"val_auc\":0.91,\"rows_scored\":124,\"endpoint\":\"turbine-risk-endpoint\"}'\n```\n\nCommon `jobs submit` traps to be aware of: `environments[].spec.client: \"4\"` is required on serverless notebook tasks; use the TASK run_id (`tasks[0].run_id`) — NOT the submit run_id — for `get-run-output`; `print()` is unreliable on serverless one-time runs (use `dbutils.notebook.exit(json.dumps(...))`); `jobs submit` rejects `tags`. For the broader `databricks-jobs` skill, see **[databricks-jobs](..\u002Fdatabricks-jobs\u002FSKILL.md)**.\n\n---\n\n## Custom pyfunc\n\nWhen sklearn\u002FXGBoost autolog isn't enough — custom preprocessing, multiple sub-models, external API calls, ensemble logic. See **[references\u002Fcustom-pyfunc.md](references\u002Fcustom-pyfunc.md)** for a full worked example. Two non-obvious things:\n\n- **`python_model=\"path\u002Fto\u002Ffile.py\"`** (file path, not class instance) + `mlflow.models.set_model(MyModel())` at the end of that file. This is the \"Models from Code\" pattern — the file is logged verbatim, no pickling of the class.\n- **`mlflow.models.predict(model_uri=..., input_data=..., env_manager=\"uv\")`** before deploying. Catches missing deps before the endpoint does.\n\n---\n\n## Custom GenAI agents\n\nHand-rolled `ResponsesAgent` (LangGraph + UC Function tools + Vector Search retrieval) — see **[references\u002Fgenai-agents.md](references\u002Fgenai-agents.md)**.\n\nPrefer no-code authoring via [databricks-agent-bricks](..\u002Fdatabricks-agent-bricks\u002FSKILL.md) (Knowledge Assistants, Supervisor Agents) unless the user explicitly needs a custom LangGraph agent.\n\n---\n\n## Feature Engineering: Feature Store & Feature Views\n\nWhen to reach for Feature Engineering (either flavor) instead of a plain Delta table: when the same feature must be computed identically at training and serving time (no training\u002Fserving skew), the feature is time-dependent and needs point-in-time joins against labels (no future leakage), the feature needs to be served with \u003C10ms latency via an online store, or the feature is shared across models with lineage tracked in UC. If none apply, a plain UC table is enough — the sections below can be skipped.\n\n**Default: don't use Feature Engineering unless one of the reasons above clearly applies or the user explicitly asked for it.** It adds build time and complexity (an extra Delta table layer for `FeatureLookup`; a materialization pipeline for Feature Views). If you're unsure, use plain UC tables and add Feature Engineering later when a concrete need surfaces.\n\nTwo flavors, one train\u002Fscore path (`create_training_set` → `fe.log_model` → `fe.score_batch`):\n\n- **`FeatureLookup` API** — you own the feature tables (compute, write, refresh); bind them to a training set via `FeatureLookup` + `FeatureEngineeringClient`. Reach for it when the features already exist as Delta tables or you want direct control over how they are computed. See **[references\u002Ffeature-store.md](references\u002Ffeature-store.md)**.\n- **Feature Views** (declarative, Public Preview, `databricks-feature-engineering>=0.16.0`) — Databricks owns compute, materialization (Delta offline + Lakebase online), and refresh from a spec you declare (`create_feature` over `DeltaTableSource`; `RollingWindow`\u002F`SlidingWindow`\u002F`TumblingWindow`; `create_stream` for Kafka features). Reach for it when the feature is a formula you want Databricks to keep fresh. See **[references\u002Ffeature-views.md](references\u002Ffeature-views.md)**.\n\n---\n\n## Gotchas (the ones that cost time)\n\n| Trap | Fix |\n|---|---|\n| Model lands in workspace registry, not UC | `mlflow.set_registry_uri(\"databricks-uc\")` *before* logging |\n| Endpoint returns PERMISSION_DENIED at first query | Pass `resources=[...]` to `log_model` (covers UC functions, VS indexes, other endpoints, Lakebase) — see [references\u002Fgenai-agents.md#resources-that-need-passthrough-auth](references\u002Fgenai-agents.md#resources-that-need-passthrough-auth) for the full list |\n| Used `transition_model_version_stage` | Stages are deprecated in UC. Use `client.set_registered_model_alias(name, \"prod\", version)` |\n| `spark_udf` rebuilds a virtualenv on every call | Pass `env_manager=\"local\"` when training+scoring share a runtime |\n| `pip_requirements` mismatch crashes endpoint at load | Pin exact versions; or pull live with `f\"mlflow=={get_distribution('mlflow').version}\"` |\n| `agents.deploy()` produced a weirdly-named endpoint | Pass `endpoint_name=...` explicitly. Auto-derived name is `agents_\u003Ccatalog>-\u003Cschema>-\u003Cmodel>` |\n\nEndpoint-lifecycle gotchas (readiness two-state, version-swap, Serving-UI SP filter) live in [databricks-model-serving](..\u002Fdatabricks-model-serving\u002FSKILL.md).\n\n---\n\n## Reference files\n\n| File | Contents |\n|---|---|\n| [references\u002Fcustom-pyfunc.md](references\u002Fcustom-pyfunc.md) | Single end-to-end custom pyfunc example: artifacts, signature, code_paths, log → register → deploy → query. |\n| [references\u002Fgenai-agents.md](references\u002Fgenai-agents.md) | Custom LangGraph `ResponsesAgent` with UC Function + Vector Search tools. `create_text_output_item` gotcha and the `resources=[...]` passthrough-auth list. For no-code agents prefer **databricks-agent-bricks**. |\n| [references\u002Ffeature-store.md](references\u002Ffeature-store.md) | Feature Engineering in Unity Catalog (standard `FeatureLookup` API): `FeatureEngineeringClient`, `create_table` with `timeseries_column`, `FeatureLookup` with point-in-time joins, `fe.log_model` with lineage, `fe.score_batch` (replaces `spark_udf` for FE models), and Lakebase online store via `publish_table`. Requires `databricks-feature-engineering>=0.16.0`. |\n| [references\u002Ffeature-views.md](references\u002Ffeature-views.md) | Feature Views (declarative Feature Engineering, Public Preview): `create_feature` over `DeltaTableSource`, `RollingWindow`\u002F`SlidingWindow`\u002F`TumblingWindow` aggregations, `materialize_features` (offline + online), streaming features off Kafka via `create_stream`, point-in-time training sets, and the Feature Serving Endpoint. Requires `databricks-feature-engineering>=0.16.0`. |\n\n## Related skills\n\n- **[databricks-model-serving](..\u002Fdatabricks-model-serving\u002FSKILL.md)** — serving-endpoint lifecycle (create, query, update-config, version-swap, AI Gateway, Foundation Model API endpoints).\n- **[databricks-agent-bricks](..\u002Fdatabricks-agent-bricks\u002FSKILL.md)** — no-code Knowledge Assistants and Supervisor Agents. Prefer this over hand-rolling agents.\n- **[databricks-mlflow-evaluation](..\u002Fdatabricks-mlflow-evaluation\u002FSKILL.md)** — evaluate model\u002Fagent quality before promoting `@prod`.\n- **[databricks-vector-search](..\u002Fdatabricks-vector-search\u002FSKILL.md)** — vector indexes used as retrieval tools in agents.\n- **[databricks-jobs](..\u002Fdatabricks-jobs\u002FSKILL.md)** — async deploy pattern (`--no-wait`, TASK run_id trap).\n- **[databricks-unity-catalog](..\u002Fdatabricks-unity-catalog\u002FSKILL.md)** — UC governs the registered model: permissions, lineage, audit.\n",{"data":34,"body":39},{"name":4,"description":6,"compatibility":35,"metadata":36,"parent":38},"Requires databricks CLI (>= v0.294.0)",{"version":37},"0.1.0","databricks-core",{"type":40,"children":41},"root",[42,51,71,83,105,126,265,272,284,325,338,342,348,374,421,467,504,1042,1059,1115,1118,1124,1170,1370,1375,1378,1384,1413,1442,1445,1451,1494,2506,2591,2594,2600,2615,2656,2659,2665,2686,2699,2702,2708,2713,2731,2758,2870,2873,2879,3069,3079,3082,3088,3318,3324,3419],{"type":43,"tag":44,"props":45,"children":47},"element","h1",{"id":46},"ml-training-on-databricks",[48],{"type":49,"value":50},"text","ML Training on Databricks",{"type":43,"tag":52,"props":53,"children":54},"p",{},[55,61,63,69],{"type":43,"tag":56,"props":57,"children":58},"strong",{},[59],{"type":49,"value":60},"FIRST",{"type":49,"value":62},": Use the parent ",{"type":43,"tag":64,"props":65,"children":67},"code",{"className":66},[],[68],{"type":49,"value":38},{"type":49,"value":70}," skill for CLI basics, authentication, and profile selection.",{"type":43,"tag":52,"props":72,"children":73},{},[74,76,81],{"type":49,"value":75},"Train with MLflow → register to Unity Catalog → consume the ",{"type":43,"tag":56,"props":77,"children":78},{},[79],{"type":49,"value":80},"same artifact",{"type":49,"value":82}," as either a batch Spark UDF over Delta or (when low-latency is required) a real-time serving endpoint.",{"type":43,"tag":84,"props":85,"children":86},"blockquote",{},[87],{"type":43,"tag":52,"props":88,"children":89},{},[90,95,97,103],{"type":43,"tag":56,"props":91,"children":92},{},[93],{"type":49,"value":94},"Always train on Databricks",{"type":49,"value":96}," (serverless job or notebook), never in the local Python process the agent is running in. Local training has no access to the silver tables, no MLflow tracking server, no UC registry path, and dies if the chat session drops — submit ",{"type":43,"tag":64,"props":98,"children":100},{"className":99},[],[101],{"type":49,"value":102},"databricks jobs submit --no-wait",{"type":49,"value":104}," (see \"Train + deploy as a serverless job\" below). Only fall back to local execution if the user explicitly asks for it.",{"type":43,"tag":52,"props":106,"children":107},{},[108,110,115,117,124],{"type":49,"value":109},"If you need to deploy a real time model serving endpoint ",{"type":43,"tag":56,"props":111,"children":112},{},[113],{"type":49,"value":114},"after",{"type":49,"value":116}," the model is registered (creating endpoints, traffic config, version-swapping, querying, Foundation Model API endpoints), see ",{"type":43,"tag":118,"props":119,"children":121},"a",{"href":120},"..\u002Fdatabricks-model-serving\u002FSKILL.md",[122],{"type":49,"value":123},"databricks-model-serving",{"type":49,"value":125},".",{"type":43,"tag":127,"props":128,"children":129},"table",{},[130,154],{"type":43,"tag":131,"props":132,"children":133},"thead",{},[134],{"type":43,"tag":135,"props":136,"children":137},"tr",{},[138,144,149],{"type":43,"tag":139,"props":140,"children":141},"th",{},[142],{"type":49,"value":143},"Consumption",{"type":43,"tag":139,"props":145,"children":146},{},[147],{"type":49,"value":148},"When",{"type":43,"tag":139,"props":150,"children":151},{},[152],{"type":49,"value":153},"How",{"type":43,"tag":155,"props":156,"children":157},"tbody",{},[158,225],{"type":43,"tag":135,"props":159,"children":160},{},[161,170,175],{"type":43,"tag":162,"props":163,"children":164},"td",{},[165],{"type":43,"tag":56,"props":166,"children":167},{},[168],{"type":49,"value":169},"Batch UDF",{"type":43,"tag":162,"props":171,"children":172},{},[173],{"type":49,"value":174},"Dashboards, daily\u002Fhourly scores, predictions read by Genie\u002FDashboards or an app (often synced to a Lakebase table)",{"type":43,"tag":162,"props":176,"children":177},{},[178,184,186,192,194,215,217,223],{"type":43,"tag":64,"props":179,"children":181},{"className":180},[],[182],{"type":49,"value":183},"mlflow.pyfunc.spark_udf(...)",{"type":49,"value":185}," → ",{"type":43,"tag":64,"props":187,"children":189},{"className":188},[],[190],{"type":49,"value":191},"INSERT INTO gold_predictions",{"type":49,"value":193},". ",{"type":43,"tag":56,"props":195,"children":196},{},[197,199,205,207,213],{"type":49,"value":198},"If the model was logged with ",{"type":43,"tag":64,"props":200,"children":202},{"className":201},[],[203],{"type":49,"value":204},"fe.log_model(training_set=...)",{"type":49,"value":206},", use ",{"type":43,"tag":64,"props":208,"children":210},{"className":209},[],[211],{"type":49,"value":212},"fe.score_batch()",{"type":49,"value":214}," instead",{"type":49,"value":216}," — see the ",{"type":43,"tag":118,"props":218,"children":220},{"href":219},"#feature-engineering-feature-store--feature-views",[221],{"type":49,"value":222},"Feature Engineering",{"type":49,"value":224}," section below.",{"type":43,"tag":135,"props":226,"children":227},{},[228,236,241],{"type":43,"tag":162,"props":229,"children":230},{},[231],{"type":43,"tag":56,"props":232,"children":233},{},[234],{"type":49,"value":235},"Real-time endpoint",{"type":43,"tag":162,"props":237,"children":238},{},[239],{"type":49,"value":240},"Score on a user action (fraud at authorization, rec at page load) — sub-100ms",{"type":43,"tag":162,"props":242,"children":243},{},[244,250,252,258,260,264],{"type":43,"tag":64,"props":245,"children":247},{"className":246},[],[248],{"type":49,"value":249},"mlflow.deployments.get_deploy_client()",{"type":49,"value":251}," (classical) \u002F ",{"type":43,"tag":64,"props":253,"children":255},{"className":254},[],[256],{"type":49,"value":257},"agents.deploy()",{"type":49,"value":259}," (agents). Endpoint lifecycle: see ",{"type":43,"tag":118,"props":261,"children":262},{"href":120},[263],{"type":49,"value":123},{"type":49,"value":125},{"type":43,"tag":266,"props":267,"children":269},"h2",{"id":268},"default-canonical-flow",[270],{"type":49,"value":271},"Default Canonical flow",{"type":43,"tag":273,"props":274,"children":278},"pre",{"className":275,"code":277,"language":49},[276],"language-text","silver_\u003Cfeatures>  +  silver_\u003Clabels>\n        ▼\n   notebook (as a serverless job):\n   ├── train with mlflow.autolog (XGBoost \u002F sklearn \u002F etc.)\n   ├── mlflow.register_model → UC: {catalog}.{schema}.{model}\n   ├── set_registered_model_alias(name, \"prod\", version)\n   └── spark_udf(@prod) over latest features → MERGE into gold_predictions\n        ▼\ngold_\u003Centity>_predictions   ◄── dashboards, apps, Genie read this\n",[279],{"type":43,"tag":64,"props":280,"children":282},{"__ignoreMap":281},"",[283],{"type":49,"value":277},{"type":43,"tag":84,"props":285,"children":286},{},[287],{"type":43,"tag":52,"props":288,"children":289},{},[290,295,297,302,304,310,312,318,320,324],{"type":43,"tag":56,"props":291,"children":292},{},[293],{"type":49,"value":294},"Feature-store-backed models diverge here.",{"type":49,"value":296}," If training used ",{"type":43,"tag":64,"props":298,"children":300},{"className":299},[],[301],{"type":49,"value":204},{"type":49,"value":303},", replace ",{"type":43,"tag":64,"props":305,"children":307},{"className":306},[],[308],{"type":49,"value":309},"spark_udf(@prod)",{"type":49,"value":311}," with ",{"type":43,"tag":64,"props":313,"children":315},{"className":314},[],[316],{"type":49,"value":317},"fe.score_batch(model_uri, df=\u003Ckeys_only>)",{"type":49,"value":319}," — it auto-joins features via the model's registered feature lineage. See the ",{"type":43,"tag":118,"props":321,"children":322},{"href":219},[323],{"type":49,"value":222},{"type":49,"value":224},{"type":43,"tag":52,"props":326,"children":327},{},[328,330,336],{"type":49,"value":329},"One notebook, one artifact. Re-running = retraining. Gold is where truth lives — read paths never call the model directly. Keep label-window logic (",{"type":43,"tag":64,"props":331,"children":333},{"className":332},[],[334],{"type":49,"value":335},"failure occurred within 7 days",{"type":49,"value":337},") in the notebook during dev; once stable, promote to a silver materialized view in SDP.",{"type":43,"tag":339,"props":340,"children":341},"hr",{},[],{"type":43,"tag":266,"props":343,"children":345},{"id":344},"train-and-register-the-90-case",[346],{"type":49,"value":347},"Train and register (the 90% case)",{"type":43,"tag":52,"props":349,"children":350},{},[351,357,359,365,367,372],{"type":43,"tag":64,"props":352,"children":354},{"className":353},[],[355],{"type":49,"value":356},"mlflow.autolog()",{"type":49,"value":358}," captures params, metrics, code, and the model artifact for every run; ",{"type":43,"tag":64,"props":360,"children":362},{"className":361},[],[363],{"type":49,"value":364},"registered_model_name=...",{"type":49,"value":366}," auto-registers the best run to UC (auto-incremented version). Wrap training with ",{"type":43,"tag":56,"props":368,"children":369},{},[370],{"type":49,"value":371},"Optuna",{"type":49,"value":373}," so each trial is a child run and the best one is what gets registered.",{"type":43,"tag":52,"props":375,"children":376},{},[377,388,390,395,397,403,405,411,413,419],{"type":43,"tag":56,"props":378,"children":379},{},[380,382],{"type":49,"value":381},"Always ",{"type":43,"tag":64,"props":383,"children":385},{"className":384},[],[386],{"type":49,"value":387},"mlflow.set_registry_uri(\"databricks-uc\")",{"type":49,"value":389}," — without it, models land in the deprecated workspace registry. ",{"type":43,"tag":56,"props":391,"children":392},{},[393],{"type":49,"value":394},"The experiment's parent folder must exist",{"type":49,"value":396}," — ",{"type":43,"tag":64,"props":398,"children":400},{"className":399},[],[401],{"type":49,"value":402},"set_experiment",{"type":49,"value":404}," does NOT auto-create it (fails with ",{"type":43,"tag":64,"props":406,"children":408},{"className":407},[],[409],{"type":49,"value":410},"NOT_FOUND: Parent directory does not exist",{"type":49,"value":412},"). Pre-create it once with ",{"type":43,"tag":64,"props":414,"children":416},{"className":415},[],[417],{"type":49,"value":418},"databricks workspace mkdirs",{"type":49,"value":420}," before the job runs.",{"type":43,"tag":273,"props":422,"children":426},{"className":423,"code":424,"language":425,"meta":281,"style":281},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Once per project — create the parent folder for the MLflow experiment.\ndatabricks workspace mkdirs \u002FUsers\u002Fme@example.com\u002Fturbine_project\n","bash",[427],{"type":43,"tag":64,"props":428,"children":429},{"__ignoreMap":281},[430,442],{"type":43,"tag":431,"props":432,"children":435},"span",{"class":433,"line":434},"line",1,[436],{"type":43,"tag":431,"props":437,"children":439},{"style":438},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[440],{"type":49,"value":441},"# Once per project — create the parent folder for the MLflow experiment.\n",{"type":43,"tag":431,"props":443,"children":445},{"class":433,"line":444},2,[446,451,457,462],{"type":43,"tag":431,"props":447,"children":449},{"style":448},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[450],{"type":49,"value":8},{"type":43,"tag":431,"props":452,"children":454},{"style":453},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[455],{"type":49,"value":456}," workspace",{"type":43,"tag":431,"props":458,"children":459},{"style":453},[460],{"type":49,"value":461}," mkdirs",{"type":43,"tag":431,"props":463,"children":464},{"style":453},[465],{"type":49,"value":466}," \u002FUsers\u002Fme@example.com\u002Fturbine_project\n",{"type":43,"tag":52,"props":468,"children":469},{},[470,472,478,480,486,488,494,496,502],{"type":49,"value":471},"Use the Databricks notebook source format (",{"type":43,"tag":64,"props":473,"children":475},{"className":474},[],[476],{"type":49,"value":477},"# Databricks notebook source",{"type":49,"value":479}," header, ",{"type":43,"tag":64,"props":481,"children":483},{"className":482},[],[484],{"type":49,"value":485},"# COMMAND ----------",{"type":49,"value":487}," separators, ",{"type":43,"tag":64,"props":489,"children":491},{"className":490},[],[492],{"type":49,"value":493},"# MAGIC %md",{"type":49,"value":495},"\u002F",{"type":43,"tag":64,"props":497,"children":499},{"className":498},[],[500],{"type":49,"value":501},"%sql",{"type":49,"value":503}," magics for markdown\u002FSQL cells):",{"type":43,"tag":273,"props":505,"children":509},{"className":506,"code":507,"language":508,"meta":281,"style":281},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Databricks notebook source\n# MAGIC %md\n# MAGIC # Turbine failure prediction\n# MAGIC\n# MAGIC Train an XGBoost classifier on engineered turbine telemetry features.\n# MAGIC ## Data exploration\n\n# COMMAND ----------\n\n# (basic data exploration — class balance, schema sanity, etc.)\n\n# COMMAND ----------\n# MAGIC %md\n# MAGIC ## Training the model\n\n# COMMAND ----------\n\nimport mlflow, mlflow.xgboost, optuna\nfrom mlflow.tracking import MlflowClient\nfrom xgboost import XGBClassifier\nfrom sklearn.metrics import roc_auc_score\n\nmlflow.set_registry_uri(\"databricks-uc\")\nmlflow.set_experiment(\"\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Fmlflow_experiment\")\n\nCATALOG, SCHEMA, NAME = \"ai_demo_gen\", \"wind_farm\", \"turbine_failure\"\nFULL_NAME = f\"{CATALOG}.{SCHEMA}.{NAME}\"\n\n# Autolog WITHOUT registered_model_name — otherwise every Optuna trial registers a new UC\n# version, and a max-by-version pick lands on the last trial to finish, not the best one.\nmlflow.xgboost.autolog(log_input_examples=True)\n\n# For imbalanced labels: stratify the split, set scale_pos_weight = neg\u002Fpos.\ndef objective(trial):\n    params = {\n        \"n_estimators\":  trial.suggest_int(\"n_estimators\", 100, 400),\n        \"max_depth\":     trial.suggest_int(\"max_depth\", 3, 10),\n        \"learning_rate\": trial.suggest_float(\"learning_rate\", 0.01, 0.3, log=True),\n    }\n    with mlflow.start_run(nested=True):\n        m = XGBClassifier(**params).fit(X_train, y_train)\n        return roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])\n\nwith mlflow.start_run(run_name=\"hpo\") as parent:\n    study = optuna.create_study(direction=\"maximize\")\n    study.optimize(objective, n_trials=20)\n\n# COMMAND ----------\n# MAGIC %md\n# MAGIC ## Retrain best params and register\n\n# COMMAND ----------\n# Retrain on the winning trial's params explicitly, then register that single model.\nwith mlflow.start_run(run_name=\"best\"):\n    best = XGBClassifier(**study.best_params).fit(X_train, y_train)\n    mlflow.log_metric(\"val_auc\", study.best_value)\n    info = mlflow.xgboost.log_model(best, name=\"model\", registered_model_name=FULL_NAME)\n\n# Stages are deprecated — UC uses movable aliases. Repoint @prod at the version we just registered.\nclient = MlflowClient(registry_uri=\"databricks-uc\")\nclient.set_registered_model_alias(FULL_NAME, \"prod\", info.registered_model_version)\n","python",[510],{"type":43,"tag":64,"props":511,"children":512},{"__ignoreMap":281},[513,521,529,538,547,556,565,575,584,592,601,609,617,625,634,642,650,658,667,676,685,694,702,711,720,728,737,746,754,763,772,781,789,798,807,816,825,834,843,852,861,870,879,887,896,905,914,922,930,938,947,955,963,972,981,990,999,1008,1016,1025,1033],{"type":43,"tag":431,"props":514,"children":515},{"class":433,"line":434},[516],{"type":43,"tag":431,"props":517,"children":518},{},[519],{"type":49,"value":520},"# Databricks notebook source\n",{"type":43,"tag":431,"props":522,"children":523},{"class":433,"line":444},[524],{"type":43,"tag":431,"props":525,"children":526},{},[527],{"type":49,"value":528},"# MAGIC %md\n",{"type":43,"tag":431,"props":530,"children":532},{"class":433,"line":531},3,[533],{"type":43,"tag":431,"props":534,"children":535},{},[536],{"type":49,"value":537},"# MAGIC # Turbine failure prediction\n",{"type":43,"tag":431,"props":539,"children":541},{"class":433,"line":540},4,[542],{"type":43,"tag":431,"props":543,"children":544},{},[545],{"type":49,"value":546},"# MAGIC\n",{"type":43,"tag":431,"props":548,"children":550},{"class":433,"line":549},5,[551],{"type":43,"tag":431,"props":552,"children":553},{},[554],{"type":49,"value":555},"# MAGIC Train an XGBoost classifier on engineered turbine telemetry features.\n",{"type":43,"tag":431,"props":557,"children":559},{"class":433,"line":558},6,[560],{"type":43,"tag":431,"props":561,"children":562},{},[563],{"type":49,"value":564},"# MAGIC ## Data exploration\n",{"type":43,"tag":431,"props":566,"children":568},{"class":433,"line":567},7,[569],{"type":43,"tag":431,"props":570,"children":572},{"emptyLinePlaceholder":571},true,[573],{"type":49,"value":574},"\n",{"type":43,"tag":431,"props":576,"children":578},{"class":433,"line":577},8,[579],{"type":43,"tag":431,"props":580,"children":581},{},[582],{"type":49,"value":583},"# COMMAND ----------\n",{"type":43,"tag":431,"props":585,"children":587},{"class":433,"line":586},9,[588],{"type":43,"tag":431,"props":589,"children":590},{"emptyLinePlaceholder":571},[591],{"type":49,"value":574},{"type":43,"tag":431,"props":593,"children":595},{"class":433,"line":594},10,[596],{"type":43,"tag":431,"props":597,"children":598},{},[599],{"type":49,"value":600},"# (basic data exploration — class balance, schema sanity, etc.)\n",{"type":43,"tag":431,"props":602,"children":604},{"class":433,"line":603},11,[605],{"type":43,"tag":431,"props":606,"children":607},{"emptyLinePlaceholder":571},[608],{"type":49,"value":574},{"type":43,"tag":431,"props":610,"children":612},{"class":433,"line":611},12,[613],{"type":43,"tag":431,"props":614,"children":615},{},[616],{"type":49,"value":583},{"type":43,"tag":431,"props":618,"children":620},{"class":433,"line":619},13,[621],{"type":43,"tag":431,"props":622,"children":623},{},[624],{"type":49,"value":528},{"type":43,"tag":431,"props":626,"children":628},{"class":433,"line":627},14,[629],{"type":43,"tag":431,"props":630,"children":631},{},[632],{"type":49,"value":633},"# MAGIC ## Training the model\n",{"type":43,"tag":431,"props":635,"children":637},{"class":433,"line":636},15,[638],{"type":43,"tag":431,"props":639,"children":640},{"emptyLinePlaceholder":571},[641],{"type":49,"value":574},{"type":43,"tag":431,"props":643,"children":645},{"class":433,"line":644},16,[646],{"type":43,"tag":431,"props":647,"children":648},{},[649],{"type":49,"value":583},{"type":43,"tag":431,"props":651,"children":653},{"class":433,"line":652},17,[654],{"type":43,"tag":431,"props":655,"children":656},{"emptyLinePlaceholder":571},[657],{"type":49,"value":574},{"type":43,"tag":431,"props":659,"children":661},{"class":433,"line":660},18,[662],{"type":43,"tag":431,"props":663,"children":664},{},[665],{"type":49,"value":666},"import mlflow, mlflow.xgboost, optuna\n",{"type":43,"tag":431,"props":668,"children":670},{"class":433,"line":669},19,[671],{"type":43,"tag":431,"props":672,"children":673},{},[674],{"type":49,"value":675},"from mlflow.tracking import MlflowClient\n",{"type":43,"tag":431,"props":677,"children":679},{"class":433,"line":678},20,[680],{"type":43,"tag":431,"props":681,"children":682},{},[683],{"type":49,"value":684},"from xgboost import XGBClassifier\n",{"type":43,"tag":431,"props":686,"children":688},{"class":433,"line":687},21,[689],{"type":43,"tag":431,"props":690,"children":691},{},[692],{"type":49,"value":693},"from sklearn.metrics import roc_auc_score\n",{"type":43,"tag":431,"props":695,"children":697},{"class":433,"line":696},22,[698],{"type":43,"tag":431,"props":699,"children":700},{"emptyLinePlaceholder":571},[701],{"type":49,"value":574},{"type":43,"tag":431,"props":703,"children":705},{"class":433,"line":704},23,[706],{"type":43,"tag":431,"props":707,"children":708},{},[709],{"type":49,"value":710},"mlflow.set_registry_uri(\"databricks-uc\")\n",{"type":43,"tag":431,"props":712,"children":714},{"class":433,"line":713},24,[715],{"type":43,"tag":431,"props":716,"children":717},{},[718],{"type":49,"value":719},"mlflow.set_experiment(\"\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Fmlflow_experiment\")\n",{"type":43,"tag":431,"props":721,"children":723},{"class":433,"line":722},25,[724],{"type":43,"tag":431,"props":725,"children":726},{"emptyLinePlaceholder":571},[727],{"type":49,"value":574},{"type":43,"tag":431,"props":729,"children":731},{"class":433,"line":730},26,[732],{"type":43,"tag":431,"props":733,"children":734},{},[735],{"type":49,"value":736},"CATALOG, SCHEMA, NAME = \"ai_demo_gen\", \"wind_farm\", \"turbine_failure\"\n",{"type":43,"tag":431,"props":738,"children":740},{"class":433,"line":739},27,[741],{"type":43,"tag":431,"props":742,"children":743},{},[744],{"type":49,"value":745},"FULL_NAME = f\"{CATALOG}.{SCHEMA}.{NAME}\"\n",{"type":43,"tag":431,"props":747,"children":749},{"class":433,"line":748},28,[750],{"type":43,"tag":431,"props":751,"children":752},{"emptyLinePlaceholder":571},[753],{"type":49,"value":574},{"type":43,"tag":431,"props":755,"children":757},{"class":433,"line":756},29,[758],{"type":43,"tag":431,"props":759,"children":760},{},[761],{"type":49,"value":762},"# Autolog WITHOUT registered_model_name — otherwise every Optuna trial registers a new UC\n",{"type":43,"tag":431,"props":764,"children":766},{"class":433,"line":765},30,[767],{"type":43,"tag":431,"props":768,"children":769},{},[770],{"type":49,"value":771},"# version, and a max-by-version pick lands on the last trial to finish, not the best one.\n",{"type":43,"tag":431,"props":773,"children":775},{"class":433,"line":774},31,[776],{"type":43,"tag":431,"props":777,"children":778},{},[779],{"type":49,"value":780},"mlflow.xgboost.autolog(log_input_examples=True)\n",{"type":43,"tag":431,"props":782,"children":784},{"class":433,"line":783},32,[785],{"type":43,"tag":431,"props":786,"children":787},{"emptyLinePlaceholder":571},[788],{"type":49,"value":574},{"type":43,"tag":431,"props":790,"children":792},{"class":433,"line":791},33,[793],{"type":43,"tag":431,"props":794,"children":795},{},[796],{"type":49,"value":797},"# For imbalanced labels: stratify the split, set scale_pos_weight = neg\u002Fpos.\n",{"type":43,"tag":431,"props":799,"children":801},{"class":433,"line":800},34,[802],{"type":43,"tag":431,"props":803,"children":804},{},[805],{"type":49,"value":806},"def objective(trial):\n",{"type":43,"tag":431,"props":808,"children":810},{"class":433,"line":809},35,[811],{"type":43,"tag":431,"props":812,"children":813},{},[814],{"type":49,"value":815},"    params = {\n",{"type":43,"tag":431,"props":817,"children":819},{"class":433,"line":818},36,[820],{"type":43,"tag":431,"props":821,"children":822},{},[823],{"type":49,"value":824},"        \"n_estimators\":  trial.suggest_int(\"n_estimators\", 100, 400),\n",{"type":43,"tag":431,"props":826,"children":828},{"class":433,"line":827},37,[829],{"type":43,"tag":431,"props":830,"children":831},{},[832],{"type":49,"value":833},"        \"max_depth\":     trial.suggest_int(\"max_depth\", 3, 10),\n",{"type":43,"tag":431,"props":835,"children":837},{"class":433,"line":836},38,[838],{"type":43,"tag":431,"props":839,"children":840},{},[841],{"type":49,"value":842},"        \"learning_rate\": trial.suggest_float(\"learning_rate\", 0.01, 0.3, log=True),\n",{"type":43,"tag":431,"props":844,"children":846},{"class":433,"line":845},39,[847],{"type":43,"tag":431,"props":848,"children":849},{},[850],{"type":49,"value":851},"    }\n",{"type":43,"tag":431,"props":853,"children":855},{"class":433,"line":854},40,[856],{"type":43,"tag":431,"props":857,"children":858},{},[859],{"type":49,"value":860},"    with mlflow.start_run(nested=True):\n",{"type":43,"tag":431,"props":862,"children":864},{"class":433,"line":863},41,[865],{"type":43,"tag":431,"props":866,"children":867},{},[868],{"type":49,"value":869},"        m = XGBClassifier(**params).fit(X_train, y_train)\n",{"type":43,"tag":431,"props":871,"children":873},{"class":433,"line":872},42,[874],{"type":43,"tag":431,"props":875,"children":876},{},[877],{"type":49,"value":878},"        return roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])\n",{"type":43,"tag":431,"props":880,"children":882},{"class":433,"line":881},43,[883],{"type":43,"tag":431,"props":884,"children":885},{"emptyLinePlaceholder":571},[886],{"type":49,"value":574},{"type":43,"tag":431,"props":888,"children":890},{"class":433,"line":889},44,[891],{"type":43,"tag":431,"props":892,"children":893},{},[894],{"type":49,"value":895},"with mlflow.start_run(run_name=\"hpo\") as parent:\n",{"type":43,"tag":431,"props":897,"children":899},{"class":433,"line":898},45,[900],{"type":43,"tag":431,"props":901,"children":902},{},[903],{"type":49,"value":904},"    study = optuna.create_study(direction=\"maximize\")\n",{"type":43,"tag":431,"props":906,"children":908},{"class":433,"line":907},46,[909],{"type":43,"tag":431,"props":910,"children":911},{},[912],{"type":49,"value":913},"    study.optimize(objective, n_trials=20)\n",{"type":43,"tag":431,"props":915,"children":917},{"class":433,"line":916},47,[918],{"type":43,"tag":431,"props":919,"children":920},{"emptyLinePlaceholder":571},[921],{"type":49,"value":574},{"type":43,"tag":431,"props":923,"children":925},{"class":433,"line":924},48,[926],{"type":43,"tag":431,"props":927,"children":928},{},[929],{"type":49,"value":583},{"type":43,"tag":431,"props":931,"children":933},{"class":433,"line":932},49,[934],{"type":43,"tag":431,"props":935,"children":936},{},[937],{"type":49,"value":528},{"type":43,"tag":431,"props":939,"children":941},{"class":433,"line":940},50,[942],{"type":43,"tag":431,"props":943,"children":944},{},[945],{"type":49,"value":946},"# MAGIC ## Retrain best params and register\n",{"type":43,"tag":431,"props":948,"children":950},{"class":433,"line":949},51,[951],{"type":43,"tag":431,"props":952,"children":953},{"emptyLinePlaceholder":571},[954],{"type":49,"value":574},{"type":43,"tag":431,"props":956,"children":958},{"class":433,"line":957},52,[959],{"type":43,"tag":431,"props":960,"children":961},{},[962],{"type":49,"value":583},{"type":43,"tag":431,"props":964,"children":966},{"class":433,"line":965},53,[967],{"type":43,"tag":431,"props":968,"children":969},{},[970],{"type":49,"value":971},"# Retrain on the winning trial's params explicitly, then register that single model.\n",{"type":43,"tag":431,"props":973,"children":975},{"class":433,"line":974},54,[976],{"type":43,"tag":431,"props":977,"children":978},{},[979],{"type":49,"value":980},"with mlflow.start_run(run_name=\"best\"):\n",{"type":43,"tag":431,"props":982,"children":984},{"class":433,"line":983},55,[985],{"type":43,"tag":431,"props":986,"children":987},{},[988],{"type":49,"value":989},"    best = XGBClassifier(**study.best_params).fit(X_train, y_train)\n",{"type":43,"tag":431,"props":991,"children":993},{"class":433,"line":992},56,[994],{"type":43,"tag":431,"props":995,"children":996},{},[997],{"type":49,"value":998},"    mlflow.log_metric(\"val_auc\", study.best_value)\n",{"type":43,"tag":431,"props":1000,"children":1002},{"class":433,"line":1001},57,[1003],{"type":43,"tag":431,"props":1004,"children":1005},{},[1006],{"type":49,"value":1007},"    info = mlflow.xgboost.log_model(best, name=\"model\", registered_model_name=FULL_NAME)\n",{"type":43,"tag":431,"props":1009,"children":1011},{"class":433,"line":1010},58,[1012],{"type":43,"tag":431,"props":1013,"children":1014},{"emptyLinePlaceholder":571},[1015],{"type":49,"value":574},{"type":43,"tag":431,"props":1017,"children":1019},{"class":433,"line":1018},59,[1020],{"type":43,"tag":431,"props":1021,"children":1022},{},[1023],{"type":49,"value":1024},"# Stages are deprecated — UC uses movable aliases. Repoint @prod at the version we just registered.\n",{"type":43,"tag":431,"props":1026,"children":1027},{"class":433,"line":27},[1028],{"type":43,"tag":431,"props":1029,"children":1030},{},[1031],{"type":49,"value":1032},"client = MlflowClient(registry_uri=\"databricks-uc\")\n",{"type":43,"tag":431,"props":1034,"children":1036},{"class":433,"line":1035},61,[1037],{"type":43,"tag":431,"props":1038,"children":1039},{},[1040],{"type":49,"value":1041},"client.set_registered_model_alias(FULL_NAME, \"prod\", info.registered_model_version)\n",{"type":43,"tag":52,"props":1043,"children":1044},{},[1045,1050,1052,1058],{"type":43,"tag":56,"props":1046,"children":1047},{},[1048],{"type":49,"value":1049},"Framework autolog",{"type":49,"value":1051},": ",{"type":43,"tag":64,"props":1053,"children":1055},{"className":1054},[],[1056],{"type":49,"value":1057},"mlflow.{sklearn,xgboost,lightgbm,pytorch,tensorflow,spark}.autolog()",{"type":49,"value":125},{"type":43,"tag":52,"props":1060,"children":1061},{},[1062,1067,1069,1075,1076,1082,1084,1090,1091,1097,1099,1105,1107,1113],{"type":43,"tag":56,"props":1063,"children":1064},{},[1065],{"type":49,"value":1066},"Aliases, not stages",{"type":49,"value":1068},": UC dropped ",{"type":43,"tag":64,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":49,"value":1074},"Staging",{"type":49,"value":495},{"type":43,"tag":64,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":49,"value":1081},"Production",{"type":49,"value":1083},". Use movable ",{"type":43,"tag":64,"props":1085,"children":1087},{"className":1086},[],[1088],{"type":49,"value":1089},"@prod",{"type":49,"value":495},{"type":43,"tag":64,"props":1092,"children":1094},{"className":1093},[],[1095],{"type":49,"value":1096},"@challenger",{"type":49,"value":1098},"; load with ",{"type":43,"tag":64,"props":1100,"children":1102},{"className":1101},[],[1103],{"type":49,"value":1104},"models:\u002F{full_name}@prod",{"type":49,"value":1106},". Promoting a new version is one ",{"type":43,"tag":64,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":49,"value":1112},"set_registered_model_alias",{"type":49,"value":1114}," call.",{"type":43,"tag":339,"props":1116,"children":1117},{},[],{"type":43,"tag":266,"props":1119,"children":1121},{"id":1120},"consume-batch-scoring-over-delta",[1122],{"type":49,"value":1123},"Consume: batch scoring over Delta",{"type":43,"tag":52,"props":1125,"children":1126},{},[1127,1129,1134,1136,1141,1143,1148,1150,1156,1158,1163,1164,1168],{"type":49,"value":1128},"The cheap, default path ",{"type":43,"tag":56,"props":1130,"children":1131},{},[1132],{"type":49,"value":1133},"for models NOT backed by feature tables",{"type":49,"value":1135},". Load the registered model as a Spark UDF and score a Delta table; write predictions to a gold table that downstream consumers read. ",{"type":43,"tag":56,"props":1137,"children":1138},{},[1139],{"type":49,"value":1140},"For feature-store-backed models",{"type":49,"value":1142}," (logged with ",{"type":43,"tag":64,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":49,"value":204},{"type":49,"value":1149},"), skip ",{"type":43,"tag":64,"props":1151,"children":1153},{"className":1152},[],[1154],{"type":49,"value":1155},"spark_udf",{"type":49,"value":1157}," entirely and use ",{"type":43,"tag":64,"props":1159,"children":1161},{"className":1160},[],[1162],{"type":49,"value":212},{"type":49,"value":216},{"type":43,"tag":118,"props":1165,"children":1166},{"href":219},[1167],{"type":49,"value":222},{"type":49,"value":1169}," section.",{"type":43,"tag":273,"props":1171,"children":1173},{"className":506,"code":1172,"language":508,"meta":281,"style":281},"# COMMAND ----------\n# MAGIC %md\n# MAGIC ## Score and save to a gold predictions table\n\n# COMMAND ----------\nimport mlflow\nfrom pyspark.sql import functions as F\n\n# env_manager rules:\n#   \"local\"     → same runtime as training (same notebook\u002Fjob). Fastest, default in dev\u002Fdemo.\n#   \"virtualenv\"→ different runtime than training; rebuilds the model's env.\n#   \"uv\"        → same as virtualenv but faster (MLflow ≥ 2.22).\npredict = mlflow.pyfunc.spark_udf(\n    spark,\n    model_uri=f\"models:\u002F{FULL_NAME}@prod\",\n    env_manager=\"local\",\n)\n\nfeatures = spark.table(f\"{CATALOG}.{SCHEMA}.silver_turbine_features_latest\")\nfeature_cols = [c for c in features.columns if c != \"turbine_id\"]   # exclude the join key\nscored = features.withColumn(\"risk_score\", predict(*[features[c] for c in feature_cols]))\n\n# Overwrite-per-run pattern for \"latest score per entity\":\nscored.select(\"turbine_id\", \"risk_score\", F.current_timestamp().alias(\"scored_at\")) \\\n    .write.mode(\"overwrite\").saveAsTable(f\"{CATALOG}.{SCHEMA}.gold_turbine_predictions\")\n",[1174],{"type":43,"tag":64,"props":1175,"children":1176},{"__ignoreMap":281},[1177,1184,1191,1199,1206,1213,1221,1229,1236,1244,1252,1260,1268,1276,1284,1292,1300,1308,1315,1323,1331,1339,1346,1354,1362],{"type":43,"tag":431,"props":1178,"children":1179},{"class":433,"line":434},[1180],{"type":43,"tag":431,"props":1181,"children":1182},{},[1183],{"type":49,"value":583},{"type":43,"tag":431,"props":1185,"children":1186},{"class":433,"line":444},[1187],{"type":43,"tag":431,"props":1188,"children":1189},{},[1190],{"type":49,"value":528},{"type":43,"tag":431,"props":1192,"children":1193},{"class":433,"line":531},[1194],{"type":43,"tag":431,"props":1195,"children":1196},{},[1197],{"type":49,"value":1198},"# MAGIC ## Score and save to a gold predictions table\n",{"type":43,"tag":431,"props":1200,"children":1201},{"class":433,"line":540},[1202],{"type":43,"tag":431,"props":1203,"children":1204},{"emptyLinePlaceholder":571},[1205],{"type":49,"value":574},{"type":43,"tag":431,"props":1207,"children":1208},{"class":433,"line":549},[1209],{"type":43,"tag":431,"props":1210,"children":1211},{},[1212],{"type":49,"value":583},{"type":43,"tag":431,"props":1214,"children":1215},{"class":433,"line":558},[1216],{"type":43,"tag":431,"props":1217,"children":1218},{},[1219],{"type":49,"value":1220},"import mlflow\n",{"type":43,"tag":431,"props":1222,"children":1223},{"class":433,"line":567},[1224],{"type":43,"tag":431,"props":1225,"children":1226},{},[1227],{"type":49,"value":1228},"from pyspark.sql import functions as F\n",{"type":43,"tag":431,"props":1230,"children":1231},{"class":433,"line":577},[1232],{"type":43,"tag":431,"props":1233,"children":1234},{"emptyLinePlaceholder":571},[1235],{"type":49,"value":574},{"type":43,"tag":431,"props":1237,"children":1238},{"class":433,"line":586},[1239],{"type":43,"tag":431,"props":1240,"children":1241},{},[1242],{"type":49,"value":1243},"# env_manager rules:\n",{"type":43,"tag":431,"props":1245,"children":1246},{"class":433,"line":594},[1247],{"type":43,"tag":431,"props":1248,"children":1249},{},[1250],{"type":49,"value":1251},"#   \"local\"     → same runtime as training (same notebook\u002Fjob). Fastest, default in dev\u002Fdemo.\n",{"type":43,"tag":431,"props":1253,"children":1254},{"class":433,"line":603},[1255],{"type":43,"tag":431,"props":1256,"children":1257},{},[1258],{"type":49,"value":1259},"#   \"virtualenv\"→ different runtime than training; rebuilds the model's env.\n",{"type":43,"tag":431,"props":1261,"children":1262},{"class":433,"line":611},[1263],{"type":43,"tag":431,"props":1264,"children":1265},{},[1266],{"type":49,"value":1267},"#   \"uv\"        → same as virtualenv but faster (MLflow ≥ 2.22).\n",{"type":43,"tag":431,"props":1269,"children":1270},{"class":433,"line":619},[1271],{"type":43,"tag":431,"props":1272,"children":1273},{},[1274],{"type":49,"value":1275},"predict = mlflow.pyfunc.spark_udf(\n",{"type":43,"tag":431,"props":1277,"children":1278},{"class":433,"line":627},[1279],{"type":43,"tag":431,"props":1280,"children":1281},{},[1282],{"type":49,"value":1283},"    spark,\n",{"type":43,"tag":431,"props":1285,"children":1286},{"class":433,"line":636},[1287],{"type":43,"tag":431,"props":1288,"children":1289},{},[1290],{"type":49,"value":1291},"    model_uri=f\"models:\u002F{FULL_NAME}@prod\",\n",{"type":43,"tag":431,"props":1293,"children":1294},{"class":433,"line":644},[1295],{"type":43,"tag":431,"props":1296,"children":1297},{},[1298],{"type":49,"value":1299},"    env_manager=\"local\",\n",{"type":43,"tag":431,"props":1301,"children":1302},{"class":433,"line":652},[1303],{"type":43,"tag":431,"props":1304,"children":1305},{},[1306],{"type":49,"value":1307},")\n",{"type":43,"tag":431,"props":1309,"children":1310},{"class":433,"line":660},[1311],{"type":43,"tag":431,"props":1312,"children":1313},{"emptyLinePlaceholder":571},[1314],{"type":49,"value":574},{"type":43,"tag":431,"props":1316,"children":1317},{"class":433,"line":669},[1318],{"type":43,"tag":431,"props":1319,"children":1320},{},[1321],{"type":49,"value":1322},"features = spark.table(f\"{CATALOG}.{SCHEMA}.silver_turbine_features_latest\")\n",{"type":43,"tag":431,"props":1324,"children":1325},{"class":433,"line":678},[1326],{"type":43,"tag":431,"props":1327,"children":1328},{},[1329],{"type":49,"value":1330},"feature_cols = [c for c in features.columns if c != \"turbine_id\"]   # exclude the join key\n",{"type":43,"tag":431,"props":1332,"children":1333},{"class":433,"line":687},[1334],{"type":43,"tag":431,"props":1335,"children":1336},{},[1337],{"type":49,"value":1338},"scored = features.withColumn(\"risk_score\", predict(*[features[c] for c in feature_cols]))\n",{"type":43,"tag":431,"props":1340,"children":1341},{"class":433,"line":696},[1342],{"type":43,"tag":431,"props":1343,"children":1344},{"emptyLinePlaceholder":571},[1345],{"type":49,"value":574},{"type":43,"tag":431,"props":1347,"children":1348},{"class":433,"line":704},[1349],{"type":43,"tag":431,"props":1350,"children":1351},{},[1352],{"type":49,"value":1353},"# Overwrite-per-run pattern for \"latest score per entity\":\n",{"type":43,"tag":431,"props":1355,"children":1356},{"class":433,"line":713},[1357],{"type":43,"tag":431,"props":1358,"children":1359},{},[1360],{"type":49,"value":1361},"scored.select(\"turbine_id\", \"risk_score\", F.current_timestamp().alias(\"scored_at\")) \\\n",{"type":43,"tag":431,"props":1363,"children":1364},{"class":433,"line":722},[1365],{"type":43,"tag":431,"props":1366,"children":1367},{},[1368],{"type":49,"value":1369},"    .write.mode(\"overwrite\").saveAsTable(f\"{CATALOG}.{SCHEMA}.gold_turbine_predictions\")\n",{"type":43,"tag":52,"props":1371,"children":1372},{},[1373],{"type":49,"value":1374},"For incremental scoring with history, MERGE into the predictions table instead of overwrite.",{"type":43,"tag":339,"props":1376,"children":1377},{},[],{"type":43,"tag":266,"props":1379,"children":1381},{"id":1380},"real-time-serving-when-required",[1382],{"type":49,"value":1383},"Real-time serving (when required)",{"type":43,"tag":52,"props":1385,"children":1386},{},[1387,1389,1395,1397,1403,1405,1411],{"type":49,"value":1388},"After registering a model to UC, deploy it behind a Model Serving endpoint. The dev-side call is ",{"type":43,"tag":64,"props":1390,"children":1392},{"className":1391},[],[1393],{"type":49,"value":1394},"mlflow.deployments.get_deploy_client(\"databricks\").create_endpoint(...)",{"type":49,"value":1396}," for classical ML or ",{"type":43,"tag":64,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":49,"value":1402},"agents.deploy(...)",{"type":49,"value":1404}," for ",{"type":43,"tag":64,"props":1406,"children":1408},{"className":1407},[],[1409],{"type":49,"value":1410},"ResponsesAgent",{"type":49,"value":1412},"s. First deploy is ~5 min for classical ML.",{"type":43,"tag":52,"props":1414,"children":1415},{},[1416,1418,1424,1426,1432,1434,1441],{"type":49,"value":1417},"For endpoint create \u002F update \u002F version-swap, traffic config, AI Gateway, querying, the ",{"type":43,"tag":64,"props":1419,"children":1421},{"className":1420},[],[1422],{"type":49,"value":1423},"state.ready",{"type":49,"value":1425}," + ",{"type":43,"tag":64,"props":1427,"children":1429},{"className":1428},[],[1430],{"type":49,"value":1431},"state.config_update",{"type":49,"value":1433}," two-field readiness check, and Foundation Model API endpoints, see ",{"type":43,"tag":56,"props":1435,"children":1436},{},[1437],{"type":43,"tag":118,"props":1438,"children":1439},{"href":120},[1440],{"type":49,"value":123},{"type":49,"value":125},{"type":43,"tag":339,"props":1443,"children":1444},{},[],{"type":43,"tag":266,"props":1446,"children":1448},{"id":1447},"train-deploy-as-a-serverless-job",[1449],{"type":49,"value":1450},"Train + deploy as a serverless job",{"type":43,"tag":52,"props":1452,"children":1453},{},[1454,1456,1462,1464,1470,1472,1478,1479,1485,1487,1493],{"type":49,"value":1455},"Training notebooks run a few minutes (Optuna + UC register; endpoint warmup adds 5–15 min if you also deploy). Submit as a serverless one-time run so the CLI doesn't block. The notebook ends with ",{"type":43,"tag":64,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":49,"value":1461},"dbutils.notebook.exit(json.dumps({...}))",{"type":49,"value":1463}," so the structured result (",{"type":43,"tag":64,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":49,"value":1469},"model_version",{"type":49,"value":1471},", ",{"type":43,"tag":64,"props":1473,"children":1475},{"className":1474},[],[1476],{"type":49,"value":1477},"val_auc",{"type":49,"value":1471},{"type":43,"tag":64,"props":1480,"children":1482},{"className":1481},[],[1483],{"type":49,"value":1484},"endpoint_name",{"type":49,"value":1486},") reaches ",{"type":43,"tag":64,"props":1488,"children":1490},{"className":1489},[],[1491],{"type":49,"value":1492},".notebook_output.result",{"type":49,"value":125},{"type":43,"tag":273,"props":1495,"children":1497},{"className":423,"code":1496,"language":425,"meta":281,"style":281},"# 1. Upload the training notebook\ndatabricks workspace import \u002FWorkspace\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Ftrain \\\n  --file .\u002Ftrain_notebook.py --format SOURCE --language PYTHON --overwrite\n\n# 2. Submit as serverless one-time run (returns {\"run_id\": N} immediately with --no-wait)\nRUN_ID=$(databricks jobs submit --no-wait --json '{\n  \"run_name\": \"turbine-train-and-deploy\",\n  \"tasks\": [{\n    \"task_key\": \"train\",\n    \"notebook_task\": {\"notebook_path\": \"\u002FWorkspace\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Ftrain\"},\n    \"environment_key\": \"ml_env\"\n  }],\n  \"environments\": [{\n    \"environment_key\": \"ml_env\",\n    \"spec\": {\n      \"client\": \"4\",\n      \"dependencies\": [\"mlflow==3.1.0\", \"xgboost==2.1.3\", \"optuna==4.1.0\", \"scikit-learn==1.5.2\"]\n    }\n  }]\n}' | jq -r .run_id)\n\n# 3. Poll until a terminal life_cycle_state.\nfor _ in $(seq 60); do\n  STATE=$(databricks jobs get-run \"$RUN_ID\" | jq -r '.state.life_cycle_state \u002F\u002F \"UNKNOWN\"')\n  echo \"$(date +%H:%M:%S) $STATE\"\n  [[ \"$STATE\" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] && break\n  sleep 30\ndone\n[[ \"$STATE\" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] || { databricks jobs cancel-run \"$RUN_ID\"; exit 1; }\n\n# life_cycle_state TERMINATED only means \"the run ended\" — check result_state.\nRESULT=$(databricks jobs get-run \"$RUN_ID\" | jq -r '.state.result_state \u002F\u002F \"UNKNOWN\"')\necho \"result_state=$RESULT\"\n[[ \"$RESULT\" == \"SUCCESS\" ]] || { echo \"Run did not succeed\"; exit 1; }\n\n# 4. Pull structured output via the TASK run_id (NOT the submit run_id).\nTASK_RUN_ID=$(databricks jobs get-run \"$RUN_ID\" | jq -r '.tasks[0].run_id')\ndatabricks jobs get-run-output \"$TASK_RUN_ID\" | jq '.notebook_output.result'\n# → '{\"model_version\":\"3\",\"val_auc\":0.91,\"rows_scored\":124,\"endpoint\":\"turbine-risk-endpoint\"}'\n",[1498],{"type":43,"tag":64,"props":1499,"children":1500},{"__ignoreMap":281},[1501,1509,1536,1574,1581,1589,1637,1645,1653,1661,1669,1677,1685,1693,1701,1709,1717,1725,1732,1740,1777,1784,1792,1837,1906,1945,2029,2042,2050,2174,2181,2189,2254,2280,2368,2375,2383,2448,2498],{"type":43,"tag":431,"props":1502,"children":1503},{"class":433,"line":434},[1504],{"type":43,"tag":431,"props":1505,"children":1506},{"style":438},[1507],{"type":49,"value":1508},"# 1. Upload the training notebook\n",{"type":43,"tag":431,"props":1510,"children":1511},{"class":433,"line":444},[1512,1516,1520,1525,1530],{"type":43,"tag":431,"props":1513,"children":1514},{"style":448},[1515],{"type":49,"value":8},{"type":43,"tag":431,"props":1517,"children":1518},{"style":453},[1519],{"type":49,"value":456},{"type":43,"tag":431,"props":1521,"children":1522},{"style":453},[1523],{"type":49,"value":1524}," import",{"type":43,"tag":431,"props":1526,"children":1527},{"style":453},[1528],{"type":49,"value":1529}," \u002FWorkspace\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Ftrain",{"type":43,"tag":431,"props":1531,"children":1533},{"style":1532},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1534],{"type":49,"value":1535}," \\\n",{"type":43,"tag":431,"props":1537,"children":1538},{"class":433,"line":531},[1539,1544,1549,1554,1559,1564,1569],{"type":43,"tag":431,"props":1540,"children":1541},{"style":453},[1542],{"type":49,"value":1543},"  --file",{"type":43,"tag":431,"props":1545,"children":1546},{"style":453},[1547],{"type":49,"value":1548}," .\u002Ftrain_notebook.py",{"type":43,"tag":431,"props":1550,"children":1551},{"style":453},[1552],{"type":49,"value":1553}," --format",{"type":43,"tag":431,"props":1555,"children":1556},{"style":453},[1557],{"type":49,"value":1558}," SOURCE",{"type":43,"tag":431,"props":1560,"children":1561},{"style":453},[1562],{"type":49,"value":1563}," --language",{"type":43,"tag":431,"props":1565,"children":1566},{"style":453},[1567],{"type":49,"value":1568}," PYTHON",{"type":43,"tag":431,"props":1570,"children":1571},{"style":453},[1572],{"type":49,"value":1573}," --overwrite\n",{"type":43,"tag":431,"props":1575,"children":1576},{"class":433,"line":540},[1577],{"type":43,"tag":431,"props":1578,"children":1579},{"emptyLinePlaceholder":571},[1580],{"type":49,"value":574},{"type":43,"tag":431,"props":1582,"children":1583},{"class":433,"line":549},[1584],{"type":43,"tag":431,"props":1585,"children":1586},{"style":438},[1587],{"type":49,"value":1588},"# 2. Submit as serverless one-time run (returns {\"run_id\": N} immediately with --no-wait)\n",{"type":43,"tag":431,"props":1590,"children":1591},{"class":433,"line":558},[1592,1597,1603,1607,1612,1617,1622,1627,1632],{"type":43,"tag":431,"props":1593,"children":1594},{"style":1532},[1595],{"type":49,"value":1596},"RUN_ID",{"type":43,"tag":431,"props":1598,"children":1600},{"style":1599},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1601],{"type":49,"value":1602},"=$(",{"type":43,"tag":431,"props":1604,"children":1605},{"style":448},[1606],{"type":49,"value":8},{"type":43,"tag":431,"props":1608,"children":1609},{"style":453},[1610],{"type":49,"value":1611}," jobs",{"type":43,"tag":431,"props":1613,"children":1614},{"style":453},[1615],{"type":49,"value":1616}," submit",{"type":43,"tag":431,"props":1618,"children":1619},{"style":453},[1620],{"type":49,"value":1621}," --no-wait",{"type":43,"tag":431,"props":1623,"children":1624},{"style":453},[1625],{"type":49,"value":1626}," --json",{"type":43,"tag":431,"props":1628,"children":1629},{"style":1599},[1630],{"type":49,"value":1631}," '",{"type":43,"tag":431,"props":1633,"children":1634},{"style":453},[1635],{"type":49,"value":1636},"{\n",{"type":43,"tag":431,"props":1638,"children":1639},{"class":433,"line":567},[1640],{"type":43,"tag":431,"props":1641,"children":1642},{"style":453},[1643],{"type":49,"value":1644},"  \"run_name\": \"turbine-train-and-deploy\",\n",{"type":43,"tag":431,"props":1646,"children":1647},{"class":433,"line":577},[1648],{"type":43,"tag":431,"props":1649,"children":1650},{"style":453},[1651],{"type":49,"value":1652},"  \"tasks\": [{\n",{"type":43,"tag":431,"props":1654,"children":1655},{"class":433,"line":586},[1656],{"type":43,"tag":431,"props":1657,"children":1658},{"style":453},[1659],{"type":49,"value":1660},"    \"task_key\": \"train\",\n",{"type":43,"tag":431,"props":1662,"children":1663},{"class":433,"line":594},[1664],{"type":43,"tag":431,"props":1665,"children":1666},{"style":453},[1667],{"type":49,"value":1668},"    \"notebook_task\": {\"notebook_path\": \"\u002FWorkspace\u002FUsers\u002Fme@example.com\u002Fturbine_project\u002Ftrain\"},\n",{"type":43,"tag":431,"props":1670,"children":1671},{"class":433,"line":603},[1672],{"type":43,"tag":431,"props":1673,"children":1674},{"style":453},[1675],{"type":49,"value":1676},"    \"environment_key\": \"ml_env\"\n",{"type":43,"tag":431,"props":1678,"children":1679},{"class":433,"line":611},[1680],{"type":43,"tag":431,"props":1681,"children":1682},{"style":453},[1683],{"type":49,"value":1684},"  }],\n",{"type":43,"tag":431,"props":1686,"children":1687},{"class":433,"line":619},[1688],{"type":43,"tag":431,"props":1689,"children":1690},{"style":453},[1691],{"type":49,"value":1692},"  \"environments\": [{\n",{"type":43,"tag":431,"props":1694,"children":1695},{"class":433,"line":627},[1696],{"type":43,"tag":431,"props":1697,"children":1698},{"style":453},[1699],{"type":49,"value":1700},"    \"environment_key\": \"ml_env\",\n",{"type":43,"tag":431,"props":1702,"children":1703},{"class":433,"line":636},[1704],{"type":43,"tag":431,"props":1705,"children":1706},{"style":453},[1707],{"type":49,"value":1708},"    \"spec\": {\n",{"type":43,"tag":431,"props":1710,"children":1711},{"class":433,"line":644},[1712],{"type":43,"tag":431,"props":1713,"children":1714},{"style":453},[1715],{"type":49,"value":1716},"      \"client\": \"4\",\n",{"type":43,"tag":431,"props":1718,"children":1719},{"class":433,"line":652},[1720],{"type":43,"tag":431,"props":1721,"children":1722},{"style":453},[1723],{"type":49,"value":1724},"      \"dependencies\": [\"mlflow==3.1.0\", \"xgboost==2.1.3\", \"optuna==4.1.0\", \"scikit-learn==1.5.2\"]\n",{"type":43,"tag":431,"props":1726,"children":1727},{"class":433,"line":660},[1728],{"type":43,"tag":431,"props":1729,"children":1730},{"style":453},[1731],{"type":49,"value":851},{"type":43,"tag":431,"props":1733,"children":1734},{"class":433,"line":669},[1735],{"type":43,"tag":431,"props":1736,"children":1737},{"style":453},[1738],{"type":49,"value":1739},"  }]\n",{"type":43,"tag":431,"props":1741,"children":1742},{"class":433,"line":678},[1743,1748,1753,1758,1763,1768,1773],{"type":43,"tag":431,"props":1744,"children":1745},{"style":453},[1746],{"type":49,"value":1747},"}",{"type":43,"tag":431,"props":1749,"children":1750},{"style":1599},[1751],{"type":49,"value":1752},"'",{"type":43,"tag":431,"props":1754,"children":1755},{"style":1599},[1756],{"type":49,"value":1757}," |",{"type":43,"tag":431,"props":1759,"children":1760},{"style":448},[1761],{"type":49,"value":1762}," jq",{"type":43,"tag":431,"props":1764,"children":1765},{"style":453},[1766],{"type":49,"value":1767}," -r",{"type":43,"tag":431,"props":1769,"children":1770},{"style":453},[1771],{"type":49,"value":1772}," .run_id",{"type":43,"tag":431,"props":1774,"children":1775},{"style":1599},[1776],{"type":49,"value":1307},{"type":43,"tag":431,"props":1778,"children":1779},{"class":433,"line":687},[1780],{"type":43,"tag":431,"props":1781,"children":1782},{"emptyLinePlaceholder":571},[1783],{"type":49,"value":574},{"type":43,"tag":431,"props":1785,"children":1786},{"class":433,"line":696},[1787],{"type":43,"tag":431,"props":1788,"children":1789},{"style":438},[1790],{"type":49,"value":1791},"# 3. Poll until a terminal life_cycle_state.\n",{"type":43,"tag":431,"props":1793,"children":1794},{"class":433,"line":704},[1795,1801,1806,1811,1816,1821,1827,1832],{"type":43,"tag":431,"props":1796,"children":1798},{"style":1797},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[1799],{"type":49,"value":1800},"for",{"type":43,"tag":431,"props":1802,"children":1803},{"style":1532},[1804],{"type":49,"value":1805}," _ ",{"type":43,"tag":431,"props":1807,"children":1808},{"style":1797},[1809],{"type":49,"value":1810},"in",{"type":43,"tag":431,"props":1812,"children":1813},{"style":1599},[1814],{"type":49,"value":1815}," $(",{"type":43,"tag":431,"props":1817,"children":1818},{"style":448},[1819],{"type":49,"value":1820},"seq",{"type":43,"tag":431,"props":1822,"children":1824},{"style":1823},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1825],{"type":49,"value":1826}," 60",{"type":43,"tag":431,"props":1828,"children":1829},{"style":1599},[1830],{"type":49,"value":1831},");",{"type":43,"tag":431,"props":1833,"children":1834},{"style":1797},[1835],{"type":49,"value":1836}," do\n",{"type":43,"tag":431,"props":1838,"children":1839},{"class":433,"line":713},[1840,1845,1849,1853,1857,1862,1867,1872,1877,1881,1885,1889,1893,1898,1902],{"type":43,"tag":431,"props":1841,"children":1842},{"style":1532},[1843],{"type":49,"value":1844},"  STATE",{"type":43,"tag":431,"props":1846,"children":1847},{"style":1599},[1848],{"type":49,"value":1602},{"type":43,"tag":431,"props":1850,"children":1851},{"style":448},[1852],{"type":49,"value":8},{"type":43,"tag":431,"props":1854,"children":1855},{"style":453},[1856],{"type":49,"value":1611},{"type":43,"tag":431,"props":1858,"children":1859},{"style":453},[1860],{"type":49,"value":1861}," get-run",{"type":43,"tag":431,"props":1863,"children":1864},{"style":1599},[1865],{"type":49,"value":1866}," \"",{"type":43,"tag":431,"props":1868,"children":1869},{"style":1532},[1870],{"type":49,"value":1871},"$RUN_ID",{"type":43,"tag":431,"props":1873,"children":1874},{"style":1599},[1875],{"type":49,"value":1876},"\"",{"type":43,"tag":431,"props":1878,"children":1879},{"style":1599},[1880],{"type":49,"value":1757},{"type":43,"tag":431,"props":1882,"children":1883},{"style":448},[1884],{"type":49,"value":1762},{"type":43,"tag":431,"props":1886,"children":1887},{"style":453},[1888],{"type":49,"value":1767},{"type":43,"tag":431,"props":1890,"children":1891},{"style":1599},[1892],{"type":49,"value":1631},{"type":43,"tag":431,"props":1894,"children":1895},{"style":453},[1896],{"type":49,"value":1897},".state.life_cycle_state \u002F\u002F \"UNKNOWN\"",{"type":43,"tag":431,"props":1899,"children":1900},{"style":1599},[1901],{"type":49,"value":1752},{"type":43,"tag":431,"props":1903,"children":1904},{"style":1599},[1905],{"type":49,"value":1307},{"type":43,"tag":431,"props":1907,"children":1908},{"class":433,"line":722},[1909,1915,1920,1925,1930,1935,1940],{"type":43,"tag":431,"props":1910,"children":1912},{"style":1911},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1913],{"type":49,"value":1914},"  echo",{"type":43,"tag":431,"props":1916,"children":1917},{"style":1599},[1918],{"type":49,"value":1919}," \"$(",{"type":43,"tag":431,"props":1921,"children":1922},{"style":448},[1923],{"type":49,"value":1924},"date",{"type":43,"tag":431,"props":1926,"children":1927},{"style":453},[1928],{"type":49,"value":1929}," +%H:%M:%S",{"type":43,"tag":431,"props":1931,"children":1932},{"style":1599},[1933],{"type":49,"value":1934},")",{"type":43,"tag":431,"props":1936,"children":1937},{"style":1532},[1938],{"type":49,"value":1939}," $STATE",{"type":43,"tag":431,"props":1941,"children":1942},{"style":1599},[1943],{"type":49,"value":1944},"\"\n",{"type":43,"tag":431,"props":1946,"children":1947},{"class":433,"line":730},[1948,1953,1957,1962,1966,1971,1976,1981,1986,1991,1996,2000,2005,2009,2014,2019,2024],{"type":43,"tag":431,"props":1949,"children":1950},{"style":1599},[1951],{"type":49,"value":1952},"  [[",{"type":43,"tag":431,"props":1954,"children":1955},{"style":1599},[1956],{"type":49,"value":1866},{"type":43,"tag":431,"props":1958,"children":1959},{"style":1532},[1960],{"type":49,"value":1961},"$STATE",{"type":43,"tag":431,"props":1963,"children":1964},{"style":1599},[1965],{"type":49,"value":1876},{"type":43,"tag":431,"props":1967,"children":1968},{"style":1599},[1969],{"type":49,"value":1970}," =~",{"type":43,"tag":431,"props":1972,"children":1973},{"style":1532},[1974],{"type":49,"value":1975}," ^",{"type":43,"tag":431,"props":1977,"children":1978},{"style":1599},[1979],{"type":49,"value":1980},"(",{"type":43,"tag":431,"props":1982,"children":1983},{"style":1532},[1984],{"type":49,"value":1985},"TERMINATED",{"type":43,"tag":431,"props":1987,"children":1988},{"style":1599},[1989],{"type":49,"value":1990},"|",{"type":43,"tag":431,"props":1992,"children":1993},{"style":1532},[1994],{"type":49,"value":1995},"SKIPPED",{"type":43,"tag":431,"props":1997,"children":1998},{"style":1599},[1999],{"type":49,"value":1990},{"type":43,"tag":431,"props":2001,"children":2002},{"style":1532},[2003],{"type":49,"value":2004},"INTERNAL_ERROR",{"type":43,"tag":431,"props":2006,"children":2007},{"style":1599},[2008],{"type":49,"value":1934},{"type":43,"tag":431,"props":2010,"children":2011},{"style":1532},[2012],{"type":49,"value":2013},"$ ",{"type":43,"tag":431,"props":2015,"children":2016},{"style":1599},[2017],{"type":49,"value":2018},"]]",{"type":43,"tag":431,"props":2020,"children":2021},{"style":1599},[2022],{"type":49,"value":2023}," &&",{"type":43,"tag":431,"props":2025,"children":2026},{"style":1797},[2027],{"type":49,"value":2028}," break\n",{"type":43,"tag":431,"props":2030,"children":2031},{"class":433,"line":739},[2032,2037],{"type":43,"tag":431,"props":2033,"children":2034},{"style":448},[2035],{"type":49,"value":2036},"  sleep",{"type":43,"tag":431,"props":2038,"children":2039},{"style":1823},[2040],{"type":49,"value":2041}," 30\n",{"type":43,"tag":431,"props":2043,"children":2044},{"class":433,"line":748},[2045],{"type":43,"tag":431,"props":2046,"children":2047},{"style":1797},[2048],{"type":49,"value":2049},"done\n",{"type":43,"tag":431,"props":2051,"children":2052},{"class":433,"line":756},[2053,2058,2062,2066,2070,2074,2078,2082,2086,2090,2094,2098,2102,2106,2110,2114,2119,2124,2129,2133,2138,2142,2146,2150,2155,2160,2165,2169],{"type":43,"tag":431,"props":2054,"children":2055},{"style":1599},[2056],{"type":49,"value":2057},"[[",{"type":43,"tag":431,"props":2059,"children":2060},{"style":1599},[2061],{"type":49,"value":1866},{"type":43,"tag":431,"props":2063,"children":2064},{"style":1532},[2065],{"type":49,"value":1961},{"type":43,"tag":431,"props":2067,"children":2068},{"style":1599},[2069],{"type":49,"value":1876},{"type":43,"tag":431,"props":2071,"children":2072},{"style":1599},[2073],{"type":49,"value":1970},{"type":43,"tag":431,"props":2075,"children":2076},{"style":1532},[2077],{"type":49,"value":1975},{"type":43,"tag":431,"props":2079,"children":2080},{"style":1599},[2081],{"type":49,"value":1980},{"type":43,"tag":431,"props":2083,"children":2084},{"style":1532},[2085],{"type":49,"value":1985},{"type":43,"tag":431,"props":2087,"children":2088},{"style":1599},[2089],{"type":49,"value":1990},{"type":43,"tag":431,"props":2091,"children":2092},{"style":1532},[2093],{"type":49,"value":1995},{"type":43,"tag":431,"props":2095,"children":2096},{"style":1599},[2097],{"type":49,"value":1990},{"type":43,"tag":431,"props":2099,"children":2100},{"style":1532},[2101],{"type":49,"value":2004},{"type":43,"tag":431,"props":2103,"children":2104},{"style":1599},[2105],{"type":49,"value":1934},{"type":43,"tag":431,"props":2107,"children":2108},{"style":1532},[2109],{"type":49,"value":2013},{"type":43,"tag":431,"props":2111,"children":2112},{"style":1599},[2113],{"type":49,"value":2018},{"type":43,"tag":431,"props":2115,"children":2116},{"style":1599},[2117],{"type":49,"value":2118}," ||",{"type":43,"tag":431,"props":2120,"children":2121},{"style":1599},[2122],{"type":49,"value":2123}," {",{"type":43,"tag":431,"props":2125,"children":2126},{"style":448},[2127],{"type":49,"value":2128}," databricks",{"type":43,"tag":431,"props":2130,"children":2131},{"style":453},[2132],{"type":49,"value":1611},{"type":43,"tag":431,"props":2134,"children":2135},{"style":453},[2136],{"type":49,"value":2137}," cancel-run",{"type":43,"tag":431,"props":2139,"children":2140},{"style":1599},[2141],{"type":49,"value":1866},{"type":43,"tag":431,"props":2143,"children":2144},{"style":1532},[2145],{"type":49,"value":1871},{"type":43,"tag":431,"props":2147,"children":2148},{"style":1599},[2149],{"type":49,"value":1876},{"type":43,"tag":431,"props":2151,"children":2152},{"style":1599},[2153],{"type":49,"value":2154},";",{"type":43,"tag":431,"props":2156,"children":2157},{"style":1911},[2158],{"type":49,"value":2159}," exit",{"type":43,"tag":431,"props":2161,"children":2162},{"style":1823},[2163],{"type":49,"value":2164}," 1",{"type":43,"tag":431,"props":2166,"children":2167},{"style":1599},[2168],{"type":49,"value":2154},{"type":43,"tag":431,"props":2170,"children":2171},{"style":1599},[2172],{"type":49,"value":2173}," }\n",{"type":43,"tag":431,"props":2175,"children":2176},{"class":433,"line":765},[2177],{"type":43,"tag":431,"props":2178,"children":2179},{"emptyLinePlaceholder":571},[2180],{"type":49,"value":574},{"type":43,"tag":431,"props":2182,"children":2183},{"class":433,"line":774},[2184],{"type":43,"tag":431,"props":2185,"children":2186},{"style":438},[2187],{"type":49,"value":2188},"# life_cycle_state TERMINATED only means \"the run ended\" — check result_state.\n",{"type":43,"tag":431,"props":2190,"children":2191},{"class":433,"line":783},[2192,2197,2201,2205,2209,2213,2217,2221,2225,2229,2233,2237,2241,2246,2250],{"type":43,"tag":431,"props":2193,"children":2194},{"style":1532},[2195],{"type":49,"value":2196},"RESULT",{"type":43,"tag":431,"props":2198,"children":2199},{"style":1599},[2200],{"type":49,"value":1602},{"type":43,"tag":431,"props":2202,"children":2203},{"style":448},[2204],{"type":49,"value":8},{"type":43,"tag":431,"props":2206,"children":2207},{"style":453},[2208],{"type":49,"value":1611},{"type":43,"tag":431,"props":2210,"children":2211},{"style":453},[2212],{"type":49,"value":1861},{"type":43,"tag":431,"props":2214,"children":2215},{"style":1599},[2216],{"type":49,"value":1866},{"type":43,"tag":431,"props":2218,"children":2219},{"style":1532},[2220],{"type":49,"value":1871},{"type":43,"tag":431,"props":2222,"children":2223},{"style":1599},[2224],{"type":49,"value":1876},{"type":43,"tag":431,"props":2226,"children":2227},{"style":1599},[2228],{"type":49,"value":1757},{"type":43,"tag":431,"props":2230,"children":2231},{"style":448},[2232],{"type":49,"value":1762},{"type":43,"tag":431,"props":2234,"children":2235},{"style":453},[2236],{"type":49,"value":1767},{"type":43,"tag":431,"props":2238,"children":2239},{"style":1599},[2240],{"type":49,"value":1631},{"type":43,"tag":431,"props":2242,"children":2243},{"style":453},[2244],{"type":49,"value":2245},".state.result_state \u002F\u002F \"UNKNOWN\"",{"type":43,"tag":431,"props":2247,"children":2248},{"style":1599},[2249],{"type":49,"value":1752},{"type":43,"tag":431,"props":2251,"children":2252},{"style":1599},[2253],{"type":49,"value":1307},{"type":43,"tag":431,"props":2255,"children":2256},{"class":433,"line":791},[2257,2262,2266,2271,2276],{"type":43,"tag":431,"props":2258,"children":2259},{"style":1911},[2260],{"type":49,"value":2261},"echo",{"type":43,"tag":431,"props":2263,"children":2264},{"style":1599},[2265],{"type":49,"value":1866},{"type":43,"tag":431,"props":2267,"children":2268},{"style":453},[2269],{"type":49,"value":2270},"result_state=",{"type":43,"tag":431,"props":2272,"children":2273},{"style":1532},[2274],{"type":49,"value":2275},"$RESULT",{"type":43,"tag":431,"props":2277,"children":2278},{"style":1599},[2279],{"type":49,"value":1944},{"type":43,"tag":431,"props":2281,"children":2282},{"class":433,"line":800},[2283,2287,2291,2295,2299,2304,2308,2313,2317,2322,2326,2330,2335,2339,2344,2348,2352,2356,2360,2364],{"type":43,"tag":431,"props":2284,"children":2285},{"style":1599},[2286],{"type":49,"value":2057},{"type":43,"tag":431,"props":2288,"children":2289},{"style":1599},[2290],{"type":49,"value":1866},{"type":43,"tag":431,"props":2292,"children":2293},{"style":1532},[2294],{"type":49,"value":2275},{"type":43,"tag":431,"props":2296,"children":2297},{"style":1599},[2298],{"type":49,"value":1876},{"type":43,"tag":431,"props":2300,"children":2301},{"style":1599},[2302],{"type":49,"value":2303}," ==",{"type":43,"tag":431,"props":2305,"children":2306},{"style":1599},[2307],{"type":49,"value":1866},{"type":43,"tag":431,"props":2309,"children":2310},{"style":453},[2311],{"type":49,"value":2312},"SUCCESS",{"type":43,"tag":431,"props":2314,"children":2315},{"style":1599},[2316],{"type":49,"value":1876},{"type":43,"tag":431,"props":2318,"children":2319},{"style":1599},[2320],{"type":49,"value":2321}," ]]",{"type":43,"tag":431,"props":2323,"children":2324},{"style":1599},[2325],{"type":49,"value":2118},{"type":43,"tag":431,"props":2327,"children":2328},{"style":1599},[2329],{"type":49,"value":2123},{"type":43,"tag":431,"props":2331,"children":2332},{"style":1911},[2333],{"type":49,"value":2334}," echo",{"type":43,"tag":431,"props":2336,"children":2337},{"style":1599},[2338],{"type":49,"value":1866},{"type":43,"tag":431,"props":2340,"children":2341},{"style":453},[2342],{"type":49,"value":2343},"Run did not succeed",{"type":43,"tag":431,"props":2345,"children":2346},{"style":1599},[2347],{"type":49,"value":1876},{"type":43,"tag":431,"props":2349,"children":2350},{"style":1599},[2351],{"type":49,"value":2154},{"type":43,"tag":431,"props":2353,"children":2354},{"style":1911},[2355],{"type":49,"value":2159},{"type":43,"tag":431,"props":2357,"children":2358},{"style":1823},[2359],{"type":49,"value":2164},{"type":43,"tag":431,"props":2361,"children":2362},{"style":1599},[2363],{"type":49,"value":2154},{"type":43,"tag":431,"props":2365,"children":2366},{"style":1599},[2367],{"type":49,"value":2173},{"type":43,"tag":431,"props":2369,"children":2370},{"class":433,"line":809},[2371],{"type":43,"tag":431,"props":2372,"children":2373},{"emptyLinePlaceholder":571},[2374],{"type":49,"value":574},{"type":43,"tag":431,"props":2376,"children":2377},{"class":433,"line":818},[2378],{"type":43,"tag":431,"props":2379,"children":2380},{"style":438},[2381],{"type":49,"value":2382},"# 4. Pull structured output via the TASK run_id (NOT the submit run_id).\n",{"type":43,"tag":431,"props":2384,"children":2385},{"class":433,"line":827},[2386,2391,2395,2399,2403,2407,2411,2415,2419,2423,2427,2431,2435,2440,2444],{"type":43,"tag":431,"props":2387,"children":2388},{"style":1532},[2389],{"type":49,"value":2390},"TASK_RUN_ID",{"type":43,"tag":431,"props":2392,"children":2393},{"style":1599},[2394],{"type":49,"value":1602},{"type":43,"tag":431,"props":2396,"children":2397},{"style":448},[2398],{"type":49,"value":8},{"type":43,"tag":431,"props":2400,"children":2401},{"style":453},[2402],{"type":49,"value":1611},{"type":43,"tag":431,"props":2404,"children":2405},{"style":453},[2406],{"type":49,"value":1861},{"type":43,"tag":431,"props":2408,"children":2409},{"style":1599},[2410],{"type":49,"value":1866},{"type":43,"tag":431,"props":2412,"children":2413},{"style":1532},[2414],{"type":49,"value":1871},{"type":43,"tag":431,"props":2416,"children":2417},{"style":1599},[2418],{"type":49,"value":1876},{"type":43,"tag":431,"props":2420,"children":2421},{"style":1599},[2422],{"type":49,"value":1757},{"type":43,"tag":431,"props":2424,"children":2425},{"style":448},[2426],{"type":49,"value":1762},{"type":43,"tag":431,"props":2428,"children":2429},{"style":453},[2430],{"type":49,"value":1767},{"type":43,"tag":431,"props":2432,"children":2433},{"style":1599},[2434],{"type":49,"value":1631},{"type":43,"tag":431,"props":2436,"children":2437},{"style":453},[2438],{"type":49,"value":2439},".tasks[0].run_id",{"type":43,"tag":431,"props":2441,"children":2442},{"style":1599},[2443],{"type":49,"value":1752},{"type":43,"tag":431,"props":2445,"children":2446},{"style":1599},[2447],{"type":49,"value":1307},{"type":43,"tag":431,"props":2449,"children":2450},{"class":433,"line":836},[2451,2455,2459,2464,2468,2473,2477,2481,2485,2489,2493],{"type":43,"tag":431,"props":2452,"children":2453},{"style":448},[2454],{"type":49,"value":8},{"type":43,"tag":431,"props":2456,"children":2457},{"style":453},[2458],{"type":49,"value":1611},{"type":43,"tag":431,"props":2460,"children":2461},{"style":453},[2462],{"type":49,"value":2463}," get-run-output",{"type":43,"tag":431,"props":2465,"children":2466},{"style":1599},[2467],{"type":49,"value":1866},{"type":43,"tag":431,"props":2469,"children":2470},{"style":1532},[2471],{"type":49,"value":2472},"$TASK_RUN_ID",{"type":43,"tag":431,"props":2474,"children":2475},{"style":1599},[2476],{"type":49,"value":1876},{"type":43,"tag":431,"props":2478,"children":2479},{"style":1599},[2480],{"type":49,"value":1757},{"type":43,"tag":431,"props":2482,"children":2483},{"style":448},[2484],{"type":49,"value":1762},{"type":43,"tag":431,"props":2486,"children":2487},{"style":1599},[2488],{"type":49,"value":1631},{"type":43,"tag":431,"props":2490,"children":2491},{"style":453},[2492],{"type":49,"value":1492},{"type":43,"tag":431,"props":2494,"children":2495},{"style":1599},[2496],{"type":49,"value":2497},"'\n",{"type":43,"tag":431,"props":2499,"children":2500},{"class":433,"line":845},[2501],{"type":43,"tag":431,"props":2502,"children":2503},{"style":438},[2504],{"type":49,"value":2505},"# → '{\"model_version\":\"3\",\"val_auc\":0.91,\"rows_scored\":124,\"endpoint\":\"turbine-risk-endpoint\"}'\n",{"type":43,"tag":52,"props":2507,"children":2508},{},[2509,2511,2517,2519,2525,2527,2533,2535,2541,2543,2549,2551,2557,2559,2564,2566,2572,2574,2580,2582,2590],{"type":49,"value":2510},"Common ",{"type":43,"tag":64,"props":2512,"children":2514},{"className":2513},[],[2515],{"type":49,"value":2516},"jobs submit",{"type":49,"value":2518}," traps to be aware of: ",{"type":43,"tag":64,"props":2520,"children":2522},{"className":2521},[],[2523],{"type":49,"value":2524},"environments[].spec.client: \"4\"",{"type":49,"value":2526}," is required on serverless notebook tasks; use the TASK run_id (",{"type":43,"tag":64,"props":2528,"children":2530},{"className":2529},[],[2531],{"type":49,"value":2532},"tasks[0].run_id",{"type":49,"value":2534},") — NOT the submit run_id — for ",{"type":43,"tag":64,"props":2536,"children":2538},{"className":2537},[],[2539],{"type":49,"value":2540},"get-run-output",{"type":49,"value":2542},"; ",{"type":43,"tag":64,"props":2544,"children":2546},{"className":2545},[],[2547],{"type":49,"value":2548},"print()",{"type":49,"value":2550}," is unreliable on serverless one-time runs (use ",{"type":43,"tag":64,"props":2552,"children":2554},{"className":2553},[],[2555],{"type":49,"value":2556},"dbutils.notebook.exit(json.dumps(...))",{"type":49,"value":2558},"); ",{"type":43,"tag":64,"props":2560,"children":2562},{"className":2561},[],[2563],{"type":49,"value":2516},{"type":49,"value":2565}," rejects ",{"type":43,"tag":64,"props":2567,"children":2569},{"className":2568},[],[2570],{"type":49,"value":2571},"tags",{"type":49,"value":2573},". For the broader ",{"type":43,"tag":64,"props":2575,"children":2577},{"className":2576},[],[2578],{"type":49,"value":2579},"databricks-jobs",{"type":49,"value":2581}," skill, see ",{"type":43,"tag":56,"props":2583,"children":2584},{},[2585],{"type":43,"tag":118,"props":2586,"children":2588},{"href":2587},"..\u002Fdatabricks-jobs\u002FSKILL.md",[2589],{"type":49,"value":2579},{"type":49,"value":125},{"type":43,"tag":339,"props":2592,"children":2593},{},[],{"type":43,"tag":266,"props":2595,"children":2597},{"id":2596},"custom-pyfunc",[2598],{"type":49,"value":2599},"Custom pyfunc",{"type":43,"tag":52,"props":2601,"children":2602},{},[2603,2605,2613],{"type":49,"value":2604},"When sklearn\u002FXGBoost autolog isn't enough — custom preprocessing, multiple sub-models, external API calls, ensemble logic. See ",{"type":43,"tag":56,"props":2606,"children":2607},{},[2608],{"type":43,"tag":118,"props":2609,"children":2611},{"href":2610},"references\u002Fcustom-pyfunc.md",[2612],{"type":49,"value":2610},{"type":49,"value":2614}," for a full worked example. Two non-obvious things:",{"type":43,"tag":2616,"props":2617,"children":2618},"ul",{},[2619,2642],{"type":43,"tag":2620,"props":2621,"children":2622},"li",{},[2623,2632,2634,2640],{"type":43,"tag":56,"props":2624,"children":2625},{},[2626],{"type":43,"tag":64,"props":2627,"children":2629},{"className":2628},[],[2630],{"type":49,"value":2631},"python_model=\"path\u002Fto\u002Ffile.py\"",{"type":49,"value":2633}," (file path, not class instance) + ",{"type":43,"tag":64,"props":2635,"children":2637},{"className":2636},[],[2638],{"type":49,"value":2639},"mlflow.models.set_model(MyModel())",{"type":49,"value":2641}," at the end of that file. This is the \"Models from Code\" pattern — the file is logged verbatim, no pickling of the class.",{"type":43,"tag":2620,"props":2643,"children":2644},{},[2645,2654],{"type":43,"tag":56,"props":2646,"children":2647},{},[2648],{"type":43,"tag":64,"props":2649,"children":2651},{"className":2650},[],[2652],{"type":49,"value":2653},"mlflow.models.predict(model_uri=..., input_data=..., env_manager=\"uv\")",{"type":49,"value":2655}," before deploying. Catches missing deps before the endpoint does.",{"type":43,"tag":339,"props":2657,"children":2658},{},[],{"type":43,"tag":266,"props":2660,"children":2662},{"id":2661},"custom-genai-agents",[2663],{"type":49,"value":2664},"Custom GenAI agents",{"type":43,"tag":52,"props":2666,"children":2667},{},[2668,2670,2675,2677,2685],{"type":49,"value":2669},"Hand-rolled ",{"type":43,"tag":64,"props":2671,"children":2673},{"className":2672},[],[2674],{"type":49,"value":1410},{"type":49,"value":2676}," (LangGraph + UC Function tools + Vector Search retrieval) — see ",{"type":43,"tag":56,"props":2678,"children":2679},{},[2680],{"type":43,"tag":118,"props":2681,"children":2683},{"href":2682},"references\u002Fgenai-agents.md",[2684],{"type":49,"value":2682},{"type":49,"value":125},{"type":43,"tag":52,"props":2687,"children":2688},{},[2689,2691,2697],{"type":49,"value":2690},"Prefer no-code authoring via ",{"type":43,"tag":118,"props":2692,"children":2694},{"href":2693},"..\u002Fdatabricks-agent-bricks\u002FSKILL.md",[2695],{"type":49,"value":2696},"databricks-agent-bricks",{"type":49,"value":2698}," (Knowledge Assistants, Supervisor Agents) unless the user explicitly needs a custom LangGraph agent.",{"type":43,"tag":339,"props":2700,"children":2701},{},[],{"type":43,"tag":266,"props":2703,"children":2705},{"id":2704},"feature-engineering-feature-store-feature-views",[2706],{"type":49,"value":2707},"Feature Engineering: Feature Store & Feature Views",{"type":43,"tag":52,"props":2709,"children":2710},{},[2711],{"type":49,"value":2712},"When to reach for Feature Engineering (either flavor) instead of a plain Delta table: when the same feature must be computed identically at training and serving time (no training\u002Fserving skew), the feature is time-dependent and needs point-in-time joins against labels (no future leakage), the feature needs to be served with \u003C10ms latency via an online store, or the feature is shared across models with lineage tracked in UC. If none apply, a plain UC table is enough — the sections below can be skipped.",{"type":43,"tag":52,"props":2714,"children":2715},{},[2716,2721,2723,2729],{"type":43,"tag":56,"props":2717,"children":2718},{},[2719],{"type":49,"value":2720},"Default: don't use Feature Engineering unless one of the reasons above clearly applies or the user explicitly asked for it.",{"type":49,"value":2722}," It adds build time and complexity (an extra Delta table layer for ",{"type":43,"tag":64,"props":2724,"children":2726},{"className":2725},[],[2727],{"type":49,"value":2728},"FeatureLookup",{"type":49,"value":2730},"; a materialization pipeline for Feature Views). If you're unsure, use plain UC tables and add Feature Engineering later when a concrete need surfaces.",{"type":43,"tag":52,"props":2732,"children":2733},{},[2734,2736,2742,2743,2749,2750,2756],{"type":49,"value":2735},"Two flavors, one train\u002Fscore path (",{"type":43,"tag":64,"props":2737,"children":2739},{"className":2738},[],[2740],{"type":49,"value":2741},"create_training_set",{"type":49,"value":185},{"type":43,"tag":64,"props":2744,"children":2746},{"className":2745},[],[2747],{"type":49,"value":2748},"fe.log_model",{"type":49,"value":185},{"type":43,"tag":64,"props":2751,"children":2753},{"className":2752},[],[2754],{"type":49,"value":2755},"fe.score_batch",{"type":49,"value":2757},"):",{"type":43,"tag":2616,"props":2759,"children":2760},{},[2761,2799],{"type":43,"tag":2620,"props":2762,"children":2763},{},[2764,2774,2776,2781,2782,2788,2790,2798],{"type":43,"tag":56,"props":2765,"children":2766},{},[2767,2772],{"type":43,"tag":64,"props":2768,"children":2770},{"className":2769},[],[2771],{"type":49,"value":2728},{"type":49,"value":2773}," API",{"type":49,"value":2775}," — you own the feature tables (compute, write, refresh); bind them to a training set via ",{"type":43,"tag":64,"props":2777,"children":2779},{"className":2778},[],[2780],{"type":49,"value":2728},{"type":49,"value":1425},{"type":43,"tag":64,"props":2783,"children":2785},{"className":2784},[],[2786],{"type":49,"value":2787},"FeatureEngineeringClient",{"type":49,"value":2789},". Reach for it when the features already exist as Delta tables or you want direct control over how they are computed. See ",{"type":43,"tag":56,"props":2791,"children":2792},{},[2793],{"type":43,"tag":118,"props":2794,"children":2796},{"href":2795},"references\u002Ffeature-store.md",[2797],{"type":49,"value":2795},{"type":49,"value":125},{"type":43,"tag":2620,"props":2800,"children":2801},{},[2802,2807,2809,2815,2817,2823,2825,2831,2832,2838,2839,2845,2846,2852,2853,2859,2861,2869],{"type":43,"tag":56,"props":2803,"children":2804},{},[2805],{"type":49,"value":2806},"Feature Views",{"type":49,"value":2808}," (declarative, Public Preview, ",{"type":43,"tag":64,"props":2810,"children":2812},{"className":2811},[],[2813],{"type":49,"value":2814},"databricks-feature-engineering>=0.16.0",{"type":49,"value":2816},") — Databricks owns compute, materialization (Delta offline + Lakebase online), and refresh from a spec you declare (",{"type":43,"tag":64,"props":2818,"children":2820},{"className":2819},[],[2821],{"type":49,"value":2822},"create_feature",{"type":49,"value":2824}," over ",{"type":43,"tag":64,"props":2826,"children":2828},{"className":2827},[],[2829],{"type":49,"value":2830},"DeltaTableSource",{"type":49,"value":2542},{"type":43,"tag":64,"props":2833,"children":2835},{"className":2834},[],[2836],{"type":49,"value":2837},"RollingWindow",{"type":49,"value":495},{"type":43,"tag":64,"props":2840,"children":2842},{"className":2841},[],[2843],{"type":49,"value":2844},"SlidingWindow",{"type":49,"value":495},{"type":43,"tag":64,"props":2847,"children":2849},{"className":2848},[],[2850],{"type":49,"value":2851},"TumblingWindow",{"type":49,"value":2542},{"type":43,"tag":64,"props":2854,"children":2856},{"className":2855},[],[2857],{"type":49,"value":2858},"create_stream",{"type":49,"value":2860}," for Kafka features). Reach for it when the feature is a formula you want Databricks to keep fresh. See ",{"type":43,"tag":56,"props":2862,"children":2863},{},[2864],{"type":43,"tag":118,"props":2865,"children":2867},{"href":2866},"references\u002Ffeature-views.md",[2868],{"type":49,"value":2866},{"type":49,"value":125},{"type":43,"tag":339,"props":2871,"children":2872},{},[],{"type":43,"tag":266,"props":2874,"children":2876},{"id":2875},"gotchas-the-ones-that-cost-time",[2877],{"type":49,"value":2878},"Gotchas (the ones that cost time)",{"type":43,"tag":127,"props":2880,"children":2881},{},[2882,2898],{"type":43,"tag":131,"props":2883,"children":2884},{},[2885],{"type":43,"tag":135,"props":2886,"children":2887},{},[2888,2893],{"type":43,"tag":139,"props":2889,"children":2890},{},[2891],{"type":49,"value":2892},"Trap",{"type":43,"tag":139,"props":2894,"children":2895},{},[2896],{"type":49,"value":2897},"Fix",{"type":43,"tag":155,"props":2899,"children":2900},{},[2901,2927,2963,2988,3013,3038],{"type":43,"tag":135,"props":2902,"children":2903},{},[2904,2909],{"type":43,"tag":162,"props":2905,"children":2906},{},[2907],{"type":49,"value":2908},"Model lands in workspace registry, not UC",{"type":43,"tag":162,"props":2910,"children":2911},{},[2912,2917,2919,2925],{"type":43,"tag":64,"props":2913,"children":2915},{"className":2914},[],[2916],{"type":49,"value":387},{"type":49,"value":2918}," ",{"type":43,"tag":2920,"props":2921,"children":2922},"em",{},[2923],{"type":49,"value":2924},"before",{"type":49,"value":2926}," logging",{"type":43,"tag":135,"props":2928,"children":2929},{},[2930,2935],{"type":43,"tag":162,"props":2931,"children":2932},{},[2933],{"type":49,"value":2934},"Endpoint returns PERMISSION_DENIED at first query",{"type":43,"tag":162,"props":2936,"children":2937},{},[2938,2940,2946,2948,2954,2956,2961],{"type":49,"value":2939},"Pass ",{"type":43,"tag":64,"props":2941,"children":2943},{"className":2942},[],[2944],{"type":49,"value":2945},"resources=[...]",{"type":49,"value":2947}," to ",{"type":43,"tag":64,"props":2949,"children":2951},{"className":2950},[],[2952],{"type":49,"value":2953},"log_model",{"type":49,"value":2955}," (covers UC functions, VS indexes, other endpoints, Lakebase) — see ",{"type":43,"tag":118,"props":2957,"children":2959},{"href":2958},"references\u002Fgenai-agents.md#resources-that-need-passthrough-auth",[2960],{"type":49,"value":2958},{"type":49,"value":2962}," for the full list",{"type":43,"tag":135,"props":2964,"children":2965},{},[2966,2977],{"type":43,"tag":162,"props":2967,"children":2968},{},[2969,2971],{"type":49,"value":2970},"Used ",{"type":43,"tag":64,"props":2972,"children":2974},{"className":2973},[],[2975],{"type":49,"value":2976},"transition_model_version_stage",{"type":43,"tag":162,"props":2978,"children":2979},{},[2980,2982],{"type":49,"value":2981},"Stages are deprecated in UC. Use ",{"type":43,"tag":64,"props":2983,"children":2985},{"className":2984},[],[2986],{"type":49,"value":2987},"client.set_registered_model_alias(name, \"prod\", version)",{"type":43,"tag":135,"props":2989,"children":2990},{},[2991,3001],{"type":43,"tag":162,"props":2992,"children":2993},{},[2994,2999],{"type":43,"tag":64,"props":2995,"children":2997},{"className":2996},[],[2998],{"type":49,"value":1155},{"type":49,"value":3000}," rebuilds a virtualenv on every call",{"type":43,"tag":162,"props":3002,"children":3003},{},[3004,3005,3011],{"type":49,"value":2939},{"type":43,"tag":64,"props":3006,"children":3008},{"className":3007},[],[3009],{"type":49,"value":3010},"env_manager=\"local\"",{"type":49,"value":3012}," when training+scoring share a runtime",{"type":43,"tag":135,"props":3014,"children":3015},{},[3016,3027],{"type":43,"tag":162,"props":3017,"children":3018},{},[3019,3025],{"type":43,"tag":64,"props":3020,"children":3022},{"className":3021},[],[3023],{"type":49,"value":3024},"pip_requirements",{"type":49,"value":3026}," mismatch crashes endpoint at load",{"type":43,"tag":162,"props":3028,"children":3029},{},[3030,3032],{"type":49,"value":3031},"Pin exact versions; or pull live with ",{"type":43,"tag":64,"props":3033,"children":3035},{"className":3034},[],[3036],{"type":49,"value":3037},"f\"mlflow=={get_distribution('mlflow').version}\"",{"type":43,"tag":135,"props":3039,"children":3040},{},[3041,3051],{"type":43,"tag":162,"props":3042,"children":3043},{},[3044,3049],{"type":43,"tag":64,"props":3045,"children":3047},{"className":3046},[],[3048],{"type":49,"value":257},{"type":49,"value":3050}," produced a weirdly-named endpoint",{"type":43,"tag":162,"props":3052,"children":3053},{},[3054,3055,3061,3063],{"type":49,"value":2939},{"type":43,"tag":64,"props":3056,"children":3058},{"className":3057},[],[3059],{"type":49,"value":3060},"endpoint_name=...",{"type":49,"value":3062}," explicitly. Auto-derived name is ",{"type":43,"tag":64,"props":3064,"children":3066},{"className":3065},[],[3067],{"type":49,"value":3068},"agents_\u003Ccatalog>-\u003Cschema>-\u003Cmodel>",{"type":43,"tag":52,"props":3070,"children":3071},{},[3072,3074,3078],{"type":49,"value":3073},"Endpoint-lifecycle gotchas (readiness two-state, version-swap, Serving-UI SP filter) live in ",{"type":43,"tag":118,"props":3075,"children":3076},{"href":120},[3077],{"type":49,"value":123},{"type":49,"value":125},{"type":43,"tag":339,"props":3080,"children":3081},{},[],{"type":43,"tag":266,"props":3083,"children":3085},{"id":3084},"reference-files",[3086],{"type":49,"value":3087},"Reference files",{"type":43,"tag":127,"props":3089,"children":3090},{},[3091,3107],{"type":43,"tag":131,"props":3092,"children":3093},{},[3094],{"type":43,"tag":135,"props":3095,"children":3096},{},[3097,3102],{"type":43,"tag":139,"props":3098,"children":3099},{},[3100],{"type":49,"value":3101},"File",{"type":43,"tag":139,"props":3103,"children":3104},{},[3105],{"type":49,"value":3106},"Contents",{"type":43,"tag":155,"props":3108,"children":3109},{},[3110,3125,3167,3251],{"type":43,"tag":135,"props":3111,"children":3112},{},[3113,3120],{"type":43,"tag":162,"props":3114,"children":3115},{},[3116],{"type":43,"tag":118,"props":3117,"children":3118},{"href":2610},[3119],{"type":49,"value":2610},{"type":43,"tag":162,"props":3121,"children":3122},{},[3123],{"type":49,"value":3124},"Single end-to-end custom pyfunc example: artifacts, signature, code_paths, log → register → deploy → query.",{"type":43,"tag":135,"props":3126,"children":3127},{},[3128,3135],{"type":43,"tag":162,"props":3129,"children":3130},{},[3131],{"type":43,"tag":118,"props":3132,"children":3133},{"href":2682},[3134],{"type":49,"value":2682},{"type":43,"tag":162,"props":3136,"children":3137},{},[3138,3140,3145,3147,3153,3155,3160,3162,3166],{"type":49,"value":3139},"Custom LangGraph ",{"type":43,"tag":64,"props":3141,"children":3143},{"className":3142},[],[3144],{"type":49,"value":1410},{"type":49,"value":3146}," with UC Function + Vector Search tools. ",{"type":43,"tag":64,"props":3148,"children":3150},{"className":3149},[],[3151],{"type":49,"value":3152},"create_text_output_item",{"type":49,"value":3154}," gotcha and the ",{"type":43,"tag":64,"props":3156,"children":3158},{"className":3157},[],[3159],{"type":49,"value":2945},{"type":49,"value":3161}," passthrough-auth list. For no-code agents prefer ",{"type":43,"tag":56,"props":3163,"children":3164},{},[3165],{"type":49,"value":2696},{"type":49,"value":125},{"type":43,"tag":135,"props":3168,"children":3169},{},[3170,3177],{"type":43,"tag":162,"props":3171,"children":3172},{},[3173],{"type":43,"tag":118,"props":3174,"children":3175},{"href":2795},[3176],{"type":49,"value":2795},{"type":43,"tag":162,"props":3178,"children":3179},{},[3180,3182,3187,3189,3194,3195,3201,3202,3208,3209,3214,3216,3221,3223,3228,3230,3235,3237,3243,3245,3250],{"type":49,"value":3181},"Feature Engineering in Unity Catalog (standard ",{"type":43,"tag":64,"props":3183,"children":3185},{"className":3184},[],[3186],{"type":49,"value":2728},{"type":49,"value":3188}," API): ",{"type":43,"tag":64,"props":3190,"children":3192},{"className":3191},[],[3193],{"type":49,"value":2787},{"type":49,"value":1471},{"type":43,"tag":64,"props":3196,"children":3198},{"className":3197},[],[3199],{"type":49,"value":3200},"create_table",{"type":49,"value":311},{"type":43,"tag":64,"props":3203,"children":3205},{"className":3204},[],[3206],{"type":49,"value":3207},"timeseries_column",{"type":49,"value":1471},{"type":43,"tag":64,"props":3210,"children":3212},{"className":3211},[],[3213],{"type":49,"value":2728},{"type":49,"value":3215}," with point-in-time joins, ",{"type":43,"tag":64,"props":3217,"children":3219},{"className":3218},[],[3220],{"type":49,"value":2748},{"type":49,"value":3222}," with lineage, ",{"type":43,"tag":64,"props":3224,"children":3226},{"className":3225},[],[3227],{"type":49,"value":2755},{"type":49,"value":3229}," (replaces ",{"type":43,"tag":64,"props":3231,"children":3233},{"className":3232},[],[3234],{"type":49,"value":1155},{"type":49,"value":3236}," for FE models), and Lakebase online store via ",{"type":43,"tag":64,"props":3238,"children":3240},{"className":3239},[],[3241],{"type":49,"value":3242},"publish_table",{"type":49,"value":3244},". Requires ",{"type":43,"tag":64,"props":3246,"children":3248},{"className":3247},[],[3249],{"type":49,"value":2814},{"type":49,"value":125},{"type":43,"tag":135,"props":3252,"children":3253},{},[3254,3261],{"type":43,"tag":162,"props":3255,"children":3256},{},[3257],{"type":43,"tag":118,"props":3258,"children":3259},{"href":2866},[3260],{"type":49,"value":2866},{"type":43,"tag":162,"props":3262,"children":3263},{},[3264,3266,3271,3272,3277,3278,3283,3284,3289,3290,3295,3297,3303,3305,3310,3312,3317],{"type":49,"value":3265},"Feature Views (declarative Feature Engineering, Public Preview): ",{"type":43,"tag":64,"props":3267,"children":3269},{"className":3268},[],[3270],{"type":49,"value":2822},{"type":49,"value":2824},{"type":43,"tag":64,"props":3273,"children":3275},{"className":3274},[],[3276],{"type":49,"value":2830},{"type":49,"value":1471},{"type":43,"tag":64,"props":3279,"children":3281},{"className":3280},[],[3282],{"type":49,"value":2837},{"type":49,"value":495},{"type":43,"tag":64,"props":3285,"children":3287},{"className":3286},[],[3288],{"type":49,"value":2844},{"type":49,"value":495},{"type":43,"tag":64,"props":3291,"children":3293},{"className":3292},[],[3294],{"type":49,"value":2851},{"type":49,"value":3296}," aggregations, ",{"type":43,"tag":64,"props":3298,"children":3300},{"className":3299},[],[3301],{"type":49,"value":3302},"materialize_features",{"type":49,"value":3304}," (offline + online), streaming features off Kafka via ",{"type":43,"tag":64,"props":3306,"children":3308},{"className":3307},[],[3309],{"type":49,"value":2858},{"type":49,"value":3311},", point-in-time training sets, and the Feature Serving Endpoint. Requires ",{"type":43,"tag":64,"props":3313,"children":3315},{"className":3314},[],[3316],{"type":49,"value":2814},{"type":49,"value":125},{"type":43,"tag":266,"props":3319,"children":3321},{"id":3320},"related-skills",[3322],{"type":49,"value":3323},"Related skills",{"type":43,"tag":2616,"props":3325,"children":3326},{},[3327,3339,3351,3371,3385,3405],{"type":43,"tag":2620,"props":3328,"children":3329},{},[3330,3337],{"type":43,"tag":56,"props":3331,"children":3332},{},[3333],{"type":43,"tag":118,"props":3334,"children":3335},{"href":120},[3336],{"type":49,"value":123},{"type":49,"value":3338}," — serving-endpoint lifecycle (create, query, update-config, version-swap, AI Gateway, Foundation Model API endpoints).",{"type":43,"tag":2620,"props":3340,"children":3341},{},[3342,3349],{"type":43,"tag":56,"props":3343,"children":3344},{},[3345],{"type":43,"tag":118,"props":3346,"children":3347},{"href":2693},[3348],{"type":49,"value":2696},{"type":49,"value":3350}," — no-code Knowledge Assistants and Supervisor Agents. Prefer this over hand-rolling agents.",{"type":43,"tag":2620,"props":3352,"children":3353},{},[3354,3363,3365,3370],{"type":43,"tag":56,"props":3355,"children":3356},{},[3357],{"type":43,"tag":118,"props":3358,"children":3360},{"href":3359},"..\u002Fdatabricks-mlflow-evaluation\u002FSKILL.md",[3361],{"type":49,"value":3362},"databricks-mlflow-evaluation",{"type":49,"value":3364}," — evaluate model\u002Fagent quality before promoting ",{"type":43,"tag":64,"props":3366,"children":3368},{"className":3367},[],[3369],{"type":49,"value":1089},{"type":49,"value":125},{"type":43,"tag":2620,"props":3372,"children":3373},{},[3374,3383],{"type":43,"tag":56,"props":3375,"children":3376},{},[3377],{"type":43,"tag":118,"props":3378,"children":3380},{"href":3379},"..\u002Fdatabricks-vector-search\u002FSKILL.md",[3381],{"type":49,"value":3382},"databricks-vector-search",{"type":49,"value":3384}," — vector indexes used as retrieval tools in agents.",{"type":43,"tag":2620,"props":3386,"children":3387},{},[3388,3395,3397,3403],{"type":43,"tag":56,"props":3389,"children":3390},{},[3391],{"type":43,"tag":118,"props":3392,"children":3393},{"href":2587},[3394],{"type":49,"value":2579},{"type":49,"value":3396}," — async deploy pattern (",{"type":43,"tag":64,"props":3398,"children":3400},{"className":3399},[],[3401],{"type":49,"value":3402},"--no-wait",{"type":49,"value":3404},", TASK run_id trap).",{"type":43,"tag":2620,"props":3406,"children":3407},{},[3408,3417],{"type":43,"tag":56,"props":3409,"children":3410},{},[3411],{"type":43,"tag":118,"props":3412,"children":3414},{"href":3413},"..\u002Fdatabricks-unity-catalog\u002FSKILL.md",[3415],{"type":49,"value":3416},"databricks-unity-catalog",{"type":49,"value":3418}," — UC governs the registered model: permissions, lineage, audit.",{"type":43,"tag":3420,"props":3421,"children":3422},"style",{},[3423],{"type":49,"value":3424},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3426,"total":774},[3427,3443,3457,3474,3491,3511,3522],{"slug":2696,"name":2696,"fn":3428,"description":3429,"org":3430,"tags":3431,"stars":23,"repoUrl":24,"updatedAt":3442},"create Databricks Agent Bricks","Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3432,3435,3436,3439],{"name":3433,"slug":3434,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":3437,"slug":3438,"type":15},"Knowledge Management","knowledge-management",{"name":3440,"slug":3441,"type":15},"Multi-Agent","multi-agent","2026-07-15T05:41:38.548954",{"slug":3444,"name":3444,"fn":3445,"description":3446,"org":3447,"tags":3448,"stars":23,"repoUrl":24,"updatedAt":3456},"databricks-ai-functions","use Databricks built-in AI functions","Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3449,3452,3453],{"name":3450,"slug":3451,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":3454,"slug":3455,"type":15},"LLM","llm","2026-07-31T05:53:33.562077",{"slug":3458,"name":3458,"fn":3459,"description":3460,"org":3461,"tags":3462,"stars":23,"repoUrl":24,"updatedAt":3473},"databricks-ai-runtime","submit and manage Databricks GPU workloads","Databricks AI Runtime (`air`) CLI — the command-line tool for submitting and managing GPU training workloads on Databricks serverless compute. Use for: running `air` workloads, custom Docker image setup, environment configuration, and troubleshooting `air` jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3463,3466,3467,3470],{"name":3464,"slug":3465,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":3468,"slug":3469,"type":15},"Docker","docker",{"name":3471,"slug":3472,"type":15},"Engineering","engineering","2026-07-12T08:04:55.843982",{"slug":3475,"name":3475,"fn":3476,"description":3477,"org":3478,"tags":3479,"stars":23,"repoUrl":24,"updatedAt":3490},"databricks-aibi-dashboards","create Databricks AI\u002FBI dashboards","Create Databricks AI\u002FBI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3480,3483,3486,3489],{"name":3481,"slug":3482,"type":15},"Analytics","analytics",{"name":3484,"slug":3485,"type":15},"Dashboards","dashboards",{"name":3487,"slug":3488,"type":15},"Data Visualization","data-visualization",{"name":9,"slug":8,"type":15},"2026-07-12T08:04:25.314591",{"slug":3492,"name":3492,"fn":3493,"description":3494,"org":3495,"tags":3496,"stars":23,"repoUrl":24,"updatedAt":3510},"databricks-app-design","design UX for Databricks AppKit applications","Design the UX of custom-code Databricks Apps (AppKit\u002FReact) data screens — KPI\u002Foverview pages, reports, charts, tables, and Genie\u002Fchat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit\u002FReact app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading\u002Fempty\u002Ferror), IBCS notation, and AI-result trust (showing generated SQL\u002Fsources for Genie\u002Fchat). A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding\u002Fbuild\u002Fdeploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie\u002Fchat\u002FAI surface.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3497,3498,3501,3504,3507],{"name":9,"slug":8,"type":15},{"name":3499,"slug":3500,"type":15},"Design","design",{"name":3502,"slug":3503,"type":15},"Frontend","frontend",{"name":3505,"slug":3506,"type":15},"React","react",{"name":3508,"slug":3509,"type":15},"UI Components","ui-components","2026-07-12T08:04:02.02398",{"slug":3512,"name":3512,"fn":3513,"description":3514,"org":3515,"tags":3516,"stars":23,"repoUrl":24,"updatedAt":3521},"databricks-apps","build applications on Databricks Apps","Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain \"create a dashboard\" request means a managed AI\u002FBI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3517,3518,3519,3520],{"name":3481,"slug":3482,"type":15},{"name":3484,"slug":3485,"type":15},{"name":3450,"slug":3451,"type":15},{"name":9,"slug":8,"type":15},"2026-07-12T08:03:59.061458",{"slug":3523,"name":3523,"fn":3524,"description":3525,"org":3526,"tags":3527,"stars":23,"repoUrl":24,"updatedAt":3543},"databricks-apps-python","build Python backends for Databricks Apps","Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex. **Default for a new Databricks App is `databricks-apps` (AppKit — Node\u002FTypeScript\u002FReact) — reach for it first.** Use this skill only when the user asks for a Python backend, extends an existing Python app, or the team is Python-only. Covers OAuth auth, app resources, SQL warehouse and Lakebase connectivity, foundation-model \u002F Vector Search \u002F model-serving APIs (via `databricks-python-sdk`), and deployment via CLI or DABs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3528,3529,3532,3535,3538,3540],{"name":9,"slug":8,"type":15},{"name":3530,"slug":3531,"type":15},"FastAPI","fastapi",{"name":3533,"slug":3534,"type":15},"Flask","flask",{"name":3536,"slug":3537,"type":15},"Gradio","gradio",{"name":3539,"slug":508,"type":15},"Python",{"name":3541,"slug":3542,"type":15},"Streamlit","streamlit","2026-07-12T08:04:10.970845",{"items":3545,"total":774},[3546,3553,3559,3566,3573,3581,3588,3597,3608,3625,3638,3651],{"slug":2696,"name":2696,"fn":3428,"description":3429,"org":3547,"tags":3548,"stars":23,"repoUrl":24,"updatedAt":3442},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3549,3550,3551,3552],{"name":3433,"slug":3434,"type":15},{"name":9,"slug":8,"type":15},{"name":3437,"slug":3438,"type":15},{"name":3440,"slug":3441,"type":15},{"slug":3444,"name":3444,"fn":3445,"description":3446,"org":3554,"tags":3555,"stars":23,"repoUrl":24,"updatedAt":3456},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3556,3557,3558],{"name":3450,"slug":3451,"type":15},{"name":9,"slug":8,"type":15},{"name":3454,"slug":3455,"type":15},{"slug":3458,"name":3458,"fn":3459,"description":3460,"org":3560,"tags":3561,"stars":23,"repoUrl":24,"updatedAt":3473},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3562,3563,3564,3565],{"name":3464,"slug":3465,"type":15},{"name":9,"slug":8,"type":15},{"name":3468,"slug":3469,"type":15},{"name":3471,"slug":3472,"type":15},{"slug":3475,"name":3475,"fn":3476,"description":3477,"org":3567,"tags":3568,"stars":23,"repoUrl":24,"updatedAt":3490},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3569,3570,3571,3572],{"name":3481,"slug":3482,"type":15},{"name":3484,"slug":3485,"type":15},{"name":3487,"slug":3488,"type":15},{"name":9,"slug":8,"type":15},{"slug":3492,"name":3492,"fn":3493,"description":3494,"org":3574,"tags":3575,"stars":23,"repoUrl":24,"updatedAt":3510},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3576,3577,3578,3579,3580],{"name":9,"slug":8,"type":15},{"name":3499,"slug":3500,"type":15},{"name":3502,"slug":3503,"type":15},{"name":3505,"slug":3506,"type":15},{"name":3508,"slug":3509,"type":15},{"slug":3512,"name":3512,"fn":3513,"description":3514,"org":3582,"tags":3583,"stars":23,"repoUrl":24,"updatedAt":3521},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3584,3585,3586,3587],{"name":3481,"slug":3482,"type":15},{"name":3484,"slug":3485,"type":15},{"name":3450,"slug":3451,"type":15},{"name":9,"slug":8,"type":15},{"slug":3523,"name":3523,"fn":3524,"description":3525,"org":3589,"tags":3590,"stars":23,"repoUrl":24,"updatedAt":3543},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3591,3592,3593,3594,3595,3596],{"name":9,"slug":8,"type":15},{"name":3530,"slug":3531,"type":15},{"name":3533,"slug":3534,"type":15},{"name":3536,"slug":3537,"type":15},{"name":3539,"slug":508,"type":15},{"name":3541,"slug":3542,"type":15},{"slug":38,"name":38,"fn":3598,"description":3599,"org":3600,"tags":3601,"stars":23,"repoUrl":24,"updatedAt":3607},"configure Databricks CLI and authentication","Databricks CLI operations and the parent\u002Fentry-point skill for Databricks CLI use: authentication, profile selection, and bundles. Load this first for CLI, auth, profile, and bundle tasks, then load the matching product skill. For finding or exploring data, answering questions about the data, or generating SQL, load the databricks-data-discovery skill (it routes to Genie One). Contains up-to-date guidelines for Databricks-related CLI tasks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3602,3605,3606],{"name":3603,"slug":3604,"type":15},"Authentication","authentication",{"name":3464,"slug":3465,"type":15},{"name":9,"slug":8,"type":15},"2026-07-18T05:11:05.45506",{"slug":3609,"name":3609,"fn":3610,"description":3611,"org":3612,"tags":3613,"stars":23,"repoUrl":24,"updatedAt":3624},"databricks-dabs","manage Databricks Declarative Automation Bundles","Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3614,3617,3620,3621],{"name":3615,"slug":3616,"type":15},"Automation","automation",{"name":3618,"slug":3619,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},{"name":3622,"slug":3623,"type":15},"Deployment","deployment","2026-07-15T05:41:35.930355",{"slug":3626,"name":3626,"fn":3627,"description":3628,"org":3629,"tags":3630,"stars":23,"repoUrl":24,"updatedAt":3637},"databricks-data-discovery","discover and query Databricks data","Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog\u002Fschema has Y'), answer a natural-language question about the data, or write a SQL query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3631,3632,3633,3634],{"name":3450,"slug":3451,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":3635,"slug":3636,"type":15},"SQL","sql","2026-07-31T05:53:32.561877",{"slug":3639,"name":3639,"fn":3640,"description":3641,"org":3642,"tags":3643,"stars":23,"repoUrl":24,"updatedAt":3650},"databricks-dbsql","query and script Databricks SQL warehouses","Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. This skill MUST be invoked when the user mentions: \"DBSQL\", \"Databricks SQL\", \"SQL warehouse\", \"SQL scripting\", \"stored procedure\", \"CALL procedure\", \"materialized view\", \"CREATE MATERIALIZED VIEW\", \"pipe syntax\", \"|>\", \"geospatial\", \"H3\", \"ST_\", \"spatial SQL\", \"collation\", \"COLLATE\", \"ai_query\", \"ai_classify\", \"ai_extract\", \"ai_gen\", \"AI function\", \"http_request\", \"remote_query\", \"read_files\", \"Lakehouse Federation\", \"recursive CTE\", \"WITH RECURSIVE\", \"multi-statement transaction\", \"temp table\", \"temporary view\", \"pipe operator\". SHOULD also invoke when the user asks about SQL best practices, data modeling patterns, or advanced SQL features on Databricks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3644,3645,3648,3649],{"name":3450,"slug":3451,"type":15},{"name":3646,"slug":3647,"type":15},"Database","database",{"name":9,"slug":8,"type":15},{"name":3635,"slug":3636,"type":15},"2026-07-12T08:04:08.678282",{"slug":3652,"name":3652,"fn":3653,"description":3654,"org":3655,"tags":3656,"stars":23,"repoUrl":24,"updatedAt":3664},"databricks-docs","search Databricks documentation","Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3657,3658,3661],{"name":9,"slug":8,"type":15},{"name":3659,"slug":3660,"type":15},"Documentation","documentation",{"name":3662,"slug":3663,"type":15},"Reference","reference","2026-07-15T05:41:34.697746"]