[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-explore-omni-omni-ai-eval":3,"mdc-8ebwm3-key":34,"related-repo-explore-omni-omni-ai-eval":3066,"related-org-explore-omni-omni-ai-eval":3159},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"omni-ai-eval","evaluate Omni AI query accuracy","Evaluate Omni AI accuracy using Omni's built-in eval system — define a prompt set, run a judged eval against a model (or branch), and read the accuracy-judge verdicts. Use this skill whenever someone wants to evaluate Omni AI, benchmark Blobby, run regression tests, compare AI output across branches or model-context changes, measure AI quality, run A\u002FB tests on model changes, assess the impact of an ai_context or modeling change, or any variant of \"run evals\", \"test Blobby\", \"benchmark query generation\", \"compare AI results\", \"regression test\", \"how accurate is the AI\", or \"measure the impact of my changes\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"explore-omni","Explore Omni","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fexplore-omni.png","exploreomni",[13,17,20],{"name":14,"slug":15,"type":16},"Omni","omni","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"Analytics","analytics",27,"https:\u002F\u002Fgithub.com\u002Fexploreomni\u002Fomni-agent-skills","2026-04-06T18:10:59.485921",null,3,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A collection of skill for working with Omni. These skills help AI agents understand and execute Omni workflows more effectively.","https:\u002F\u002Fgithub.com\u002Fexploreomni\u002Fomni-agent-skills\u002Ftree\u002FHEAD\u002Fskills\u002Fomni-ai-eval","---\nname: omni-ai-eval\ndescription: Evaluate Omni AI accuracy using Omni's built-in eval system — define a prompt set, run a judged eval against a model (or branch), and read the accuracy-judge verdicts. Use this skill whenever someone wants to evaluate Omni AI, benchmark Blobby, run regression tests, compare AI output across branches or model-context changes, measure AI quality, run A\u002FB tests on model changes, assess the impact of an ai_context or modeling change, or any variant of \"run evals\", \"test Blobby\", \"benchmark query generation\", \"compare AI results\", \"regression test\", \"how accurate is the AI\", or \"measure the impact of my changes\".\n---\n\n# Omni AI Eval\n\nOmni ships a first-class eval system (the **AI Hub** → **Prompt sets** and **Eval runs**). This skill drives it through the Omni CLI: define a reusable **prompt set**, start a judged **eval run** against a model or branch, and read per-prompt verdicts from Omni's built-in **accuracy judge**.\n\nPrefer this native system over building your own harness. The judge scores each answer *semantically* against the full agent conversation — it does not require golden query JSON, and it evaluates the whole agentic workflow (topic selection, queries, results, and the final written answer), not just generated query structure.\n\n> **Tip**: Use `omni-ai-optimizer` to improve scores after finding failures, `omni-model-builder` to apply context changes on a branch before A\u002FB testing, and `omni-model-explorer` to discover topics and fields when writing prompts.\n\n## Prerequisites\n\n```bash\n# Verify the Omni CLI is installed — if not, ask the user to install it.\n# See: https:\u002F\u002Fgithub.com\u002Fexploreomni\u002Fcli#readme\ncommand -v omni >\u002Fdev\u002Fnull || echo \"ERROR: Omni CLI is not installed.\"\n\n# Verify the CLI has the eval commands. If this errors with \"unknown command\",\n# the binary is stale — ask the user to update it (the ai-eval group is generated\n# from the bundled API spec).\nomni ai-eval --help >\u002Fdev\u002Fnull 2>&1 || echo \"ERROR: 'omni ai-eval' missing — update the Omni CLI.\"\n```\n\n```bash\n# Show available profiles and select the right one — running against the wrong\n# instance silently evaluates the wrong model.\nomni config show\n# If multiple profiles exist, ask the user which to use:\nomni config use \u003Cprofile-name>\n\n# Confirm the active profile is authenticated and inspect your permissions:\nomni whoami whoami\n```\n\n> **Auth**: a profile authenticates with an **API key** or **OAuth**. If `whoami` (or any call) returns **401**, hand off — ask the user to run `! omni config login \u003Cprofile>` (OAuth 2.1 browser flow; it blocks ~2 min on the browser). Don't run `config login` yourself in a headless\u002FCI session (no browser → timeout); on a local interactive machine you *may*. See the **`omni-api-conventions`** rule for profile setup (`omni config init --auth oauth`) and discovering request-body shapes with `--schema`.\n\nYou also need the **model ID** of a shared model to evaluate. Evals require at least **Querier** access on that model, and at least one topic optimized for AI. See the [Evals guide](https:\u002F\u002Fdocs.omni.co\u002Fai\u002Fevals) for concepts and prompt-set best practices.\n\n## How it works\n\n| Concept | What it is |\n|---------|-----------|\n| **Prompt set** | A reusable, named list of up to **25** natural-language prompts, scoped to one model. Each prompt may carry an optional `expectation` — a reference answer the judge scores against. Lives server-side; create one per topic, per release, or for regression coverage. |\n| **Eval run** | Executes a prompt set against a model branch (or `main`). Each prompt runs as a full async agentic AI job (the same engine as production Blobby), then the accuracy judge scores the result. |\n| **Accuracy judge** | A fixed judge model that reads the evaluated AI's full conversation and returns a **pass\u002Ffail** verdict per prompt, plus confidence and a rationale. It targets high-impact analysis errors (hallucinations, date\u002Ftime filtering, row-limit handling, mental math, period-over-period mistakes, wrong topic). It does **not** grade wording or formatting. |\n\nAll commands accept `-o json` (or `--compact`) to force structured output for parsing, and `--profile \u003Cname>` \u002F `--branch-id` style global flags. Run `omni ai-eval \u003Ccommand> --help` for the full flag list. For commands that take a `--body` (e.g. `prompt-sets-create`), run them with `--schema` to print the body's JSON schema and a filled example instead of guessing the JSON.\n\n## Step 1 — Build a prompt set\n\n```bash\nomni ai-eval prompt-sets-create --compact --body '{\n  \"model_id\": \"your-model-id\",\n  \"name\": \"Orders regression\",\n  \"slug\": \"orders-regression\",\n  \"description\": \"Core revenue + orders coverage\",\n  \"prompts\": [\n    { \"prompt_text\": \"Show me revenue by month\" },\n    { \"prompt_text\": \"What are the top 5 products by revenue?\",\n      \"expectation\": \"The top product by revenue should be Aniseed Syrup.\" },\n    { \"prompt_text\": \"How many orders were placed last week?\" }\n  ]\n}'\n```\n\nThe response includes the created `prompt_set.id` (a UUID) — capture it for the run.\n\n| Field | Required | Notes |\n|-------|----------|-------|\n| `model_id` | Yes | Shared model UUID the set is bound to |\n| `name` | Yes | ≤ 255 chars |\n| `slug` | Yes | Unique per `model_id`; must match `^[a-z][a-z0-9-]*$` |\n| `description` | No | ≤ 1024 chars |\n| `prompts[]` | No | ≤ 25; each needs `prompt_text` (≤ 8000 chars), optional `expectation` (≤ 16000 chars) |\n\n**Find or update an existing set** instead of recreating:\n\n```bash\nomni ai-eval prompt-sets-list --model-ids your-model-id --compact   # discover sets + ids\nomni ai-eval prompt-sets-get \u003CpromptSetId> --compact                # full set with prompts\n```\n\n`prompt-sets-update` replaces the **entire** `prompts` list: omitted prompts are deleted, entries with no `id` are created, entries with a matching `id` are updated in place. To add one prompt, send the full desired list (existing prompts carry their `id`).\n\n```bash\nomni ai-eval prompt-sets-update \u003CpromptSetId> --compact --body '{\n  \"prompts\": [\n    { \"id\": \"\u003Cexisting-prompt-id>\", \"prompt_text\": \"Show me revenue by month\" },\n    { \"prompt_text\": \"Revenue by month, last 12 months only\" }\n  ]\n}'\n```\n\n### Writing good prompts and expectations\n\n- **Mirror real user questions** — pull from the AI usage analytics dashboard rather than inventing phrasing that echoes field names.\n- **Favor breadth over depth** — one prompt across many topics yields more signal than ten on one topic.\n- **Add a regression prompt** whenever an answer turns out wrong, so the same failure is caught next time.\n- **Set an `expectation`** only when a prompt has a known answer worth pinning: a value or ranking (\"top product should be Aniseed Syrup\"), a direction (\"revenue should be up YoY\"), or a required breakdown. The judge treats a *material* divergence (wrong numbers, wrong direction, missing required result) as a failure but ignores wording\u002Fformatting differences. With no expectation, the judge decides whether the answer is correct on its own terms. The expectation is shown only to the judge — the evaluated AI never sees it.\n\n## Step 2 — Run an eval\n\n```bash\n# Against main:\nomni ai-eval runs-create --compact --body '{\n  \"prompt_set_id\": \"\u003CpromptSetId>\",\n  \"description\": \"Baseline on main\"\n}'\n\n# Against a branch (measures a model-context change before promotion):\nomni ai-eval runs-create --compact --body '{\n  \"prompt_set_id\": \"\u003CpromptSetId>\",\n  \"description\": \"After adding ai_context to order_items\",\n  \"run_config\": { \"branch_id\": \"\u003CbranchId>\" }\n}'\n```\n\nThe response returns `run.id` and `job_count` (one agentic job per prompt). Omit `run_config.branch_id` to run against the live shared model.\n\n> **Concurrency cap**: at most **2 eval runs in flight at once**. A `429` means the per-user active-run cap is reached — wait for an in-flight run to finish or cancel one. A `503` means eval is paused for the org. Check `runs-list` before launching.\n\n### Poll for completion\n\n```bash\nomni ai-eval runs-get \u003CrunId> --compact\n```\n\nPoll with backoff (e.g. 5s, 10s, 20s) until the run's `status` is terminal — `COMPLETE` or `CANCELLED`. Track progress with each result's `agentic_job.state` (`QUEUED` → `EXECUTING` → `COMPLETE`\u002F`FAILED`). Don't hammer the endpoint.\n\n## Step 3 — Read results\n\n`runs-get` returns `results[]`, one row per prompt:\n\n```json\n{\n  \"run\": {\n    \"status\": \"COMPLETE\",\n    \"branch_id\": null,\n    \"results\": [\n      {\n        \"prompt\": \"What are the top 5 products by revenue?\",\n        \"score\": 1,\n        \"error_reason\": null,\n        \"cost\": 0.0021,\n        \"scoring_cost\": 0.0004,\n        \"timing_ms\": 4321,\n        \"agentic_job\": { \"state\": \"COMPLETE\", \"conversation_id\": \"conv-uuid\", \"id\": \"job-uuid\" }\n      }\n    ]\n  }\n}\n```\n\n| Field | Meaning |\n|-------|---------|\n| `score` | Judge verdict for the prompt — pass = `1`, fail = `0` |\n| `error_reason` | Set when the underlying agentic job failed |\n| `cost` \u002F `scoring_cost` | LLM cost (USD) for the answer vs. for judging it |\n| `timing_ms` | Total **AI time** in ms — all LLM processing and tool calls (matches the \"AI time\" column in the UI), not wall-clock duration |\n| `agentic_job.conversation_id` | Open this chat to read the judge's full verdict, confidence, and rationale |\n\n**Overall accuracy = the pass rate** (mean of `score` across results). Report it with the per-prompt breakdown, and for any failure, point to the `conversation_id` so the user can read *why* the judge failed it — that rationale is where the actionable signal lives.\n\n```\nEval run \"Baseline on main\" — 9\u002F12 passed (75.0%)\n  ✗ \"Revenue by quarter\"        — judge: summed a row-limited result as a total\n  ✗ \"Top products this year\"    — judge: date filter used calendar instead of fiscal year\n  ✗ \"Churn rate by segment\"     — agentic job FAILED (error_reason)\n  (open each conversation_id for the full rationale)\n```\n\n## A\u002FB comparison: branch vs main\n\nThe core workflow for measuring whether a model change helps. Run the **same prompt set** twice — once on `main`, once on the branch — then compare.\n\n1. Create or identify the branch with the change (use `omni-model-builder` to apply `ai_context`, fields, joins, `ai_settings`, etc. on a branch).\n2. `runs-create` with no `run_config` → baseline run on main.\n3. `runs-create` with `run_config.branch_id` → branch run.\n4. Poll both to `COMPLETE`, then diff per-prompt verdicts.\n\n```\nA\u002FB: main vs branch\u002Fnew-context  (prompt set: orders-regression)\n                    main      branch     Δ\nAccuracy:           75.0%     91.7%     +16.7%\nPrompt credits:     0.024     0.026     +0.002\n\nRegressions (passed on main, failed on branch):\n  - rev-by-quarter\nImprovements (failed on main, passed on branch):\n  - top-products-this-year\n  - churn-rate-by-segment\n```\n\n> **Always check for regressions, not just net improvement.** A higher overall pass rate can still hide a prompt that newly broke — an `ai_context` change that helps most prompts may conflict with one. Call out any prompt that passed on main but fails on the branch.\n\nNotes for an auditable comparison:\n- Both runs must use the **same prompt set** so they evaluate identical prompts (the AI Hub comparison view enforces this too).\n- Record the full `branch_id` used and confirm the branch run's `run.branch_id` matches it — don't claim the branch was exercised without that.\n- Expectations are snapshotted per run, so editing a prompt's `expectation` later won't change how earlier runs were scored.\n\n## Managing prompt sets and runs\n\n```bash\nomni ai-eval runs-list --prompt-set-id \u003CpromptSetId> --compact   # runs for a set, newest first\nomni ai-eval runs-cancel \u003CrunId>                                 # cancel an in-flight run (also archives it)\nomni ai-eval runs-archive \u003CrunId>                                # archive a finished run\nomni ai-eval runs-unarchive \u003CrunId>                              # restore an archived run\n\nomni ai-eval prompt-sets-archive \u003CpromptSetId>                   # archive a set (cancels its in-flight jobs)\nomni ai-eval prompt-sets-unarchive \u003CpromptSetId>                 # restore an archived set\n```\n\nArchiving is a soft delete — sets and runs are preserved and can be restored. Cancelling a run marks it `CANCELLED` and archives it; use `runs-unarchive` to surface it in the default list again. Use `--archived true` on the `*-list` commands to see archived items.\n\n## Gotchas\n\n- **Right profile, right instance** — the eval runs against whatever model lives on the active profile's instance. Confirm the profile before creating sets or runs.\n- **Judge is binary and fixed** — a pass means the answer avoided the high-impact analysis errors, not that it's the single best possible answer. The judge model isn't configurable per run.\n- **Failed job ≠ failed judgment** — a result with `error_reason` set means the agentic job itself failed (it never produced an answer to score); treat that separately from a judge `score` of `0`.\n- **Snapshotting** — to keep runs reproducible, capture the model state alongside results: `omni models yaml-get \u003CmodelId> --compact` and `omni models validate \u003CmodelId>`. Branch runs already pin the change to a branch.\n\n## Docs reference\n\n- [Evals (concepts + judge)](https:\u002F\u002Fdocs.omni.co\u002Fai\u002Fevals) · [AI Eval API](https:\u002F\u002Fdocs.omni.co\u002Fapi\u002Fai-eval\u002Flist-eval-prompt-sets) · [Create AI job API](https:\u002F\u002Fdocs.omni.co\u002Fapi\u002Fai\u002Fcreate-ai-job) · [Optimizing models for AI](https:\u002F\u002Fdocs.omni.co\u002Fmodeling\u002Fdevelop\u002Fai-optimization)\n\n## Related skills\n\n- **omni-ai-optimizer** — improve AI accuracy based on eval findings (`ai_context`, `sample_queries`, field metadata)\n- **omni-model-builder** — apply context changes on a branch before A\u002FB testing\n- **omni-model-explorer** — discover topics and fields when writing prompts\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,47,96,109,148,155,333,450,541,571,577,690,758,764,904,917,1079,1089,1175,1223,1319,1326,1383,1389,1543,1571,1615,1621,1666,1731,1737,1756,2240,2369,2400,2410,2416,2435,2515,2524,2544,2549,2595,2601,2846,2882,2888,2966,2972,3009,3015,3060],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":46},"text","Omni AI Eval",{"type":40,"tag":48,"props":49,"children":50},"p",{},[51,53,59,61,66,68,73,75,80,82,87,89,94],{"type":45,"value":52},"Omni ships a first-class eval system (the ",{"type":40,"tag":54,"props":55,"children":56},"strong",{},[57],{"type":45,"value":58},"AI Hub",{"type":45,"value":60}," → ",{"type":40,"tag":54,"props":62,"children":63},{},[64],{"type":45,"value":65},"Prompt sets",{"type":45,"value":67}," and ",{"type":40,"tag":54,"props":69,"children":70},{},[71],{"type":45,"value":72},"Eval runs",{"type":45,"value":74},"). This skill drives it through the Omni CLI: define a reusable ",{"type":40,"tag":54,"props":76,"children":77},{},[78],{"type":45,"value":79},"prompt set",{"type":45,"value":81},", start a judged ",{"type":40,"tag":54,"props":83,"children":84},{},[85],{"type":45,"value":86},"eval run",{"type":45,"value":88}," against a model or branch, and read per-prompt verdicts from Omni's built-in ",{"type":40,"tag":54,"props":90,"children":91},{},[92],{"type":45,"value":93},"accuracy judge",{"type":45,"value":95},".",{"type":40,"tag":48,"props":97,"children":98},{},[99,101,107],{"type":45,"value":100},"Prefer this native system over building your own harness. The judge scores each answer ",{"type":40,"tag":102,"props":103,"children":104},"em",{},[105],{"type":45,"value":106},"semantically",{"type":45,"value":108}," against the full agent conversation — it does not require golden query JSON, and it evaluates the whole agentic workflow (topic selection, queries, results, and the final written answer), not just generated query structure.",{"type":40,"tag":110,"props":111,"children":112},"blockquote",{},[113],{"type":40,"tag":48,"props":114,"children":115},{},[116,121,123,130,132,138,140,146],{"type":40,"tag":54,"props":117,"children":118},{},[119],{"type":45,"value":120},"Tip",{"type":45,"value":122},": Use ",{"type":40,"tag":124,"props":125,"children":127},"code",{"className":126},[],[128],{"type":45,"value":129},"omni-ai-optimizer",{"type":45,"value":131}," to improve scores after finding failures, ",{"type":40,"tag":124,"props":133,"children":135},{"className":134},[],[136],{"type":45,"value":137},"omni-model-builder",{"type":45,"value":139}," to apply context changes on a branch before A\u002FB testing, and ",{"type":40,"tag":124,"props":141,"children":143},{"className":142},[],[144],{"type":45,"value":145},"omni-model-explorer",{"type":45,"value":147}," to discover topics and fields when writing prompts.",{"type":40,"tag":149,"props":150,"children":152},"h2",{"id":151},"prerequisites",[153],{"type":45,"value":154},"Prerequisites",{"type":40,"tag":156,"props":157,"children":162},"pre",{"className":158,"code":159,"language":160,"meta":161,"style":161},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Verify the Omni CLI is installed — if not, ask the user to install it.\n# See: https:\u002F\u002Fgithub.com\u002Fexploreomni\u002Fcli#readme\ncommand -v omni >\u002Fdev\u002Fnull || echo \"ERROR: Omni CLI is not installed.\"\n\n# Verify the CLI has the eval commands. If this errors with \"unknown command\",\n# the binary is stale — ask the user to update it (the ai-eval group is generated\n# from the bundled API spec).\nomni ai-eval --help >\u002Fdev\u002Fnull 2>&1 || echo \"ERROR: 'omni ai-eval' missing — update the Omni CLI.\"\n","bash","",[163],{"type":40,"tag":124,"props":164,"children":165},{"__ignoreMap":161},[166,178,187,243,253,262,271,280],{"type":40,"tag":167,"props":168,"children":171},"span",{"class":169,"line":170},"line",1,[172],{"type":40,"tag":167,"props":173,"children":175},{"style":174},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[176],{"type":45,"value":177},"# Verify the Omni CLI is installed — if not, ask the user to install it.\n",{"type":40,"tag":167,"props":179,"children":181},{"class":169,"line":180},2,[182],{"type":40,"tag":167,"props":183,"children":184},{"style":174},[185],{"type":45,"value":186},"# See: https:\u002F\u002Fgithub.com\u002Fexploreomni\u002Fcli#readme\n",{"type":40,"tag":167,"props":188,"children":189},{"class":169,"line":27},[190,196,202,207,213,218,223,228,233,238],{"type":40,"tag":167,"props":191,"children":193},{"style":192},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[194],{"type":45,"value":195},"command",{"type":40,"tag":167,"props":197,"children":199},{"style":198},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[200],{"type":45,"value":201}," -v",{"type":40,"tag":167,"props":203,"children":204},{"style":198},[205],{"type":45,"value":206}," omni",{"type":40,"tag":167,"props":208,"children":210},{"style":209},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[211],{"type":45,"value":212}," >",{"type":40,"tag":167,"props":214,"children":215},{"style":198},[216],{"type":45,"value":217},"\u002Fdev\u002Fnull",{"type":40,"tag":167,"props":219,"children":220},{"style":209},[221],{"type":45,"value":222}," ||",{"type":40,"tag":167,"props":224,"children":225},{"style":192},[226],{"type":45,"value":227}," echo",{"type":40,"tag":167,"props":229,"children":230},{"style":209},[231],{"type":45,"value":232}," \"",{"type":40,"tag":167,"props":234,"children":235},{"style":198},[236],{"type":45,"value":237},"ERROR: Omni CLI is not installed.",{"type":40,"tag":167,"props":239,"children":240},{"style":209},[241],{"type":45,"value":242},"\"\n",{"type":40,"tag":167,"props":244,"children":246},{"class":169,"line":245},4,[247],{"type":40,"tag":167,"props":248,"children":250},{"emptyLinePlaceholder":249},true,[251],{"type":45,"value":252},"\n",{"type":40,"tag":167,"props":254,"children":256},{"class":169,"line":255},5,[257],{"type":40,"tag":167,"props":258,"children":259},{"style":174},[260],{"type":45,"value":261},"# Verify the CLI has the eval commands. If this errors with \"unknown command\",\n",{"type":40,"tag":167,"props":263,"children":265},{"class":169,"line":264},6,[266],{"type":40,"tag":167,"props":267,"children":268},{"style":174},[269],{"type":45,"value":270},"# the binary is stale — ask the user to update it (the ai-eval group is generated\n",{"type":40,"tag":167,"props":272,"children":274},{"class":169,"line":273},7,[275],{"type":40,"tag":167,"props":276,"children":277},{"style":174},[278],{"type":45,"value":279},"# from the bundled API spec).\n",{"type":40,"tag":167,"props":281,"children":283},{"class":169,"line":282},8,[284,289,294,299,303,307,312,316,320,324,329],{"type":40,"tag":167,"props":285,"children":287},{"style":286},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[288],{"type":45,"value":15},{"type":40,"tag":167,"props":290,"children":291},{"style":198},[292],{"type":45,"value":293}," ai-eval",{"type":40,"tag":167,"props":295,"children":296},{"style":198},[297],{"type":45,"value":298}," --help",{"type":40,"tag":167,"props":300,"children":301},{"style":209},[302],{"type":45,"value":212},{"type":40,"tag":167,"props":304,"children":305},{"style":198},[306],{"type":45,"value":217},{"type":40,"tag":167,"props":308,"children":309},{"style":209},[310],{"type":45,"value":311}," 2>&1",{"type":40,"tag":167,"props":313,"children":314},{"style":209},[315],{"type":45,"value":222},{"type":40,"tag":167,"props":317,"children":318},{"style":192},[319],{"type":45,"value":227},{"type":40,"tag":167,"props":321,"children":322},{"style":209},[323],{"type":45,"value":232},{"type":40,"tag":167,"props":325,"children":326},{"style":198},[327],{"type":45,"value":328},"ERROR: 'omni ai-eval' missing — update the Omni CLI.",{"type":40,"tag":167,"props":330,"children":331},{"style":209},[332],{"type":45,"value":242},{"type":40,"tag":156,"props":334,"children":336},{"className":158,"code":335,"language":160,"meta":161,"style":161},"# Show available profiles and select the right one — running against the wrong\n# instance silently evaluates the wrong model.\nomni config show\n# If multiple profiles exist, ask the user which to use:\nomni config use \u003Cprofile-name>\n\n# Confirm the active profile is authenticated and inspect your permissions:\nomni whoami whoami\n",[337],{"type":40,"tag":124,"props":338,"children":339},{"__ignoreMap":161},[340,348,356,373,381,418,425,433],{"type":40,"tag":167,"props":341,"children":342},{"class":169,"line":170},[343],{"type":40,"tag":167,"props":344,"children":345},{"style":174},[346],{"type":45,"value":347},"# Show available profiles and select the right one — running against the wrong\n",{"type":40,"tag":167,"props":349,"children":350},{"class":169,"line":180},[351],{"type":40,"tag":167,"props":352,"children":353},{"style":174},[354],{"type":45,"value":355},"# instance silently evaluates the wrong model.\n",{"type":40,"tag":167,"props":357,"children":358},{"class":169,"line":27},[359,363,368],{"type":40,"tag":167,"props":360,"children":361},{"style":286},[362],{"type":45,"value":15},{"type":40,"tag":167,"props":364,"children":365},{"style":198},[366],{"type":45,"value":367}," config",{"type":40,"tag":167,"props":369,"children":370},{"style":198},[371],{"type":45,"value":372}," show\n",{"type":40,"tag":167,"props":374,"children":375},{"class":169,"line":245},[376],{"type":40,"tag":167,"props":377,"children":378},{"style":174},[379],{"type":45,"value":380},"# If multiple profiles exist, ask the user which to use:\n",{"type":40,"tag":167,"props":382,"children":383},{"class":169,"line":255},[384,388,392,397,402,407,413],{"type":40,"tag":167,"props":385,"children":386},{"style":286},[387],{"type":45,"value":15},{"type":40,"tag":167,"props":389,"children":390},{"style":198},[391],{"type":45,"value":367},{"type":40,"tag":167,"props":393,"children":394},{"style":198},[395],{"type":45,"value":396}," use",{"type":40,"tag":167,"props":398,"children":399},{"style":209},[400],{"type":45,"value":401}," \u003C",{"type":40,"tag":167,"props":403,"children":404},{"style":198},[405],{"type":45,"value":406},"profile-nam",{"type":40,"tag":167,"props":408,"children":410},{"style":409},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[411],{"type":45,"value":412},"e",{"type":40,"tag":167,"props":414,"children":415},{"style":209},[416],{"type":45,"value":417},">\n",{"type":40,"tag":167,"props":419,"children":420},{"class":169,"line":264},[421],{"type":40,"tag":167,"props":422,"children":423},{"emptyLinePlaceholder":249},[424],{"type":45,"value":252},{"type":40,"tag":167,"props":426,"children":427},{"class":169,"line":273},[428],{"type":40,"tag":167,"props":429,"children":430},{"style":174},[431],{"type":45,"value":432},"# Confirm the active profile is authenticated and inspect your permissions:\n",{"type":40,"tag":167,"props":434,"children":435},{"class":169,"line":282},[436,440,445],{"type":40,"tag":167,"props":437,"children":438},{"style":286},[439],{"type":45,"value":15},{"type":40,"tag":167,"props":441,"children":442},{"style":198},[443],{"type":45,"value":444}," whoami",{"type":40,"tag":167,"props":446,"children":447},{"style":198},[448],{"type":45,"value":449}," whoami\n",{"type":40,"tag":110,"props":451,"children":452},{},[453],{"type":40,"tag":48,"props":454,"children":455},{},[456,461,463,468,470,475,477,483,485,490,492,498,500,506,508,513,515,524,526,532,534,540],{"type":40,"tag":54,"props":457,"children":458},{},[459],{"type":45,"value":460},"Auth",{"type":45,"value":462},": a profile authenticates with an ",{"type":40,"tag":54,"props":464,"children":465},{},[466],{"type":45,"value":467},"API key",{"type":45,"value":469}," or ",{"type":40,"tag":54,"props":471,"children":472},{},[473],{"type":45,"value":474},"OAuth",{"type":45,"value":476},". If ",{"type":40,"tag":124,"props":478,"children":480},{"className":479},[],[481],{"type":45,"value":482},"whoami",{"type":45,"value":484}," (or any call) returns ",{"type":40,"tag":54,"props":486,"children":487},{},[488],{"type":45,"value":489},"401",{"type":45,"value":491},", hand off — ask the user to run ",{"type":40,"tag":124,"props":493,"children":495},{"className":494},[],[496],{"type":45,"value":497},"! omni config login \u003Cprofile>",{"type":45,"value":499}," (OAuth 2.1 browser flow; it blocks ~2 min on the browser). Don't run ",{"type":40,"tag":124,"props":501,"children":503},{"className":502},[],[504],{"type":45,"value":505},"config login",{"type":45,"value":507}," yourself in a headless\u002FCI session (no browser → timeout); on a local interactive machine you ",{"type":40,"tag":102,"props":509,"children":510},{},[511],{"type":45,"value":512},"may",{"type":45,"value":514},". See the ",{"type":40,"tag":54,"props":516,"children":517},{},[518],{"type":40,"tag":124,"props":519,"children":521},{"className":520},[],[522],{"type":45,"value":523},"omni-api-conventions",{"type":45,"value":525}," rule for profile setup (",{"type":40,"tag":124,"props":527,"children":529},{"className":528},[],[530],{"type":45,"value":531},"omni config init --auth oauth",{"type":45,"value":533},") and discovering request-body shapes with ",{"type":40,"tag":124,"props":535,"children":537},{"className":536},[],[538],{"type":45,"value":539},"--schema",{"type":45,"value":95},{"type":40,"tag":48,"props":542,"children":543},{},[544,546,551,553,558,560,569],{"type":45,"value":545},"You also need the ",{"type":40,"tag":54,"props":547,"children":548},{},[549],{"type":45,"value":550},"model ID",{"type":45,"value":552}," of a shared model to evaluate. Evals require at least ",{"type":40,"tag":54,"props":554,"children":555},{},[556],{"type":45,"value":557},"Querier",{"type":45,"value":559}," access on that model, and at least one topic optimized for AI. See the ",{"type":40,"tag":561,"props":562,"children":566},"a",{"href":563,"rel":564},"https:\u002F\u002Fdocs.omni.co\u002Fai\u002Fevals",[565],"nofollow",[567],{"type":45,"value":568},"Evals guide",{"type":45,"value":570}," for concepts and prompt-set best practices.",{"type":40,"tag":149,"props":572,"children":574},{"id":573},"how-it-works",[575],{"type":45,"value":576},"How it works",{"type":40,"tag":578,"props":579,"children":580},"table",{},[581,600],{"type":40,"tag":582,"props":583,"children":584},"thead",{},[585],{"type":40,"tag":586,"props":587,"children":588},"tr",{},[589,595],{"type":40,"tag":590,"props":591,"children":592},"th",{},[593],{"type":45,"value":594},"Concept",{"type":40,"tag":590,"props":596,"children":597},{},[598],{"type":45,"value":599},"What it is",{"type":40,"tag":601,"props":602,"children":603},"tbody",{},[604,636,660],{"type":40,"tag":586,"props":605,"children":606},{},[607,616],{"type":40,"tag":608,"props":609,"children":610},"td",{},[611],{"type":40,"tag":54,"props":612,"children":613},{},[614],{"type":45,"value":615},"Prompt set",{"type":40,"tag":608,"props":617,"children":618},{},[619,621,626,628,634],{"type":45,"value":620},"A reusable, named list of up to ",{"type":40,"tag":54,"props":622,"children":623},{},[624],{"type":45,"value":625},"25",{"type":45,"value":627}," natural-language prompts, scoped to one model. Each prompt may carry an optional ",{"type":40,"tag":124,"props":629,"children":631},{"className":630},[],[632],{"type":45,"value":633},"expectation",{"type":45,"value":635}," — a reference answer the judge scores against. Lives server-side; create one per topic, per release, or for regression coverage.",{"type":40,"tag":586,"props":637,"children":638},{},[639,647],{"type":40,"tag":608,"props":640,"children":641},{},[642],{"type":40,"tag":54,"props":643,"children":644},{},[645],{"type":45,"value":646},"Eval run",{"type":40,"tag":608,"props":648,"children":649},{},[650,652,658],{"type":45,"value":651},"Executes a prompt set against a model branch (or ",{"type":40,"tag":124,"props":653,"children":655},{"className":654},[],[656],{"type":45,"value":657},"main",{"type":45,"value":659},"). Each prompt runs as a full async agentic AI job (the same engine as production Blobby), then the accuracy judge scores the result.",{"type":40,"tag":586,"props":661,"children":662},{},[663,671],{"type":40,"tag":608,"props":664,"children":665},{},[666],{"type":40,"tag":54,"props":667,"children":668},{},[669],{"type":45,"value":670},"Accuracy judge",{"type":40,"tag":608,"props":672,"children":673},{},[674,676,681,683,688],{"type":45,"value":675},"A fixed judge model that reads the evaluated AI's full conversation and returns a ",{"type":40,"tag":54,"props":677,"children":678},{},[679],{"type":45,"value":680},"pass\u002Ffail",{"type":45,"value":682}," verdict per prompt, plus confidence and a rationale. It targets high-impact analysis errors (hallucinations, date\u002Ftime filtering, row-limit handling, mental math, period-over-period mistakes, wrong topic). It does ",{"type":40,"tag":54,"props":684,"children":685},{},[686],{"type":45,"value":687},"not",{"type":45,"value":689}," grade wording or formatting.",{"type":40,"tag":48,"props":691,"children":692},{},[693,695,701,703,709,711,717,719,725,727,733,735,741,743,749,751,756],{"type":45,"value":694},"All commands accept ",{"type":40,"tag":124,"props":696,"children":698},{"className":697},[],[699],{"type":45,"value":700},"-o json",{"type":45,"value":702}," (or ",{"type":40,"tag":124,"props":704,"children":706},{"className":705},[],[707],{"type":45,"value":708},"--compact",{"type":45,"value":710},") to force structured output for parsing, and ",{"type":40,"tag":124,"props":712,"children":714},{"className":713},[],[715],{"type":45,"value":716},"--profile \u003Cname>",{"type":45,"value":718}," \u002F ",{"type":40,"tag":124,"props":720,"children":722},{"className":721},[],[723],{"type":45,"value":724},"--branch-id",{"type":45,"value":726}," style global flags. Run ",{"type":40,"tag":124,"props":728,"children":730},{"className":729},[],[731],{"type":45,"value":732},"omni ai-eval \u003Ccommand> --help",{"type":45,"value":734}," for the full flag list. For commands that take a ",{"type":40,"tag":124,"props":736,"children":738},{"className":737},[],[739],{"type":45,"value":740},"--body",{"type":45,"value":742}," (e.g. ",{"type":40,"tag":124,"props":744,"children":746},{"className":745},[],[747],{"type":45,"value":748},"prompt-sets-create",{"type":45,"value":750},"), run them with ",{"type":40,"tag":124,"props":752,"children":754},{"className":753},[],[755],{"type":45,"value":539},{"type":45,"value":757}," to print the body's JSON schema and a filled example instead of guessing the JSON.",{"type":40,"tag":149,"props":759,"children":761},{"id":760},"step-1-build-a-prompt-set",[762],{"type":45,"value":763},"Step 1 — Build a prompt set",{"type":40,"tag":156,"props":765,"children":767},{"className":158,"code":766,"language":160,"meta":161,"style":161},"omni ai-eval prompt-sets-create --compact --body '{\n  \"model_id\": \"your-model-id\",\n  \"name\": \"Orders regression\",\n  \"slug\": \"orders-regression\",\n  \"description\": \"Core revenue + orders coverage\",\n  \"prompts\": [\n    { \"prompt_text\": \"Show me revenue by month\" },\n    { \"prompt_text\": \"What are the top 5 products by revenue?\",\n      \"expectation\": \"The top product by revenue should be Aniseed Syrup.\" },\n    { \"prompt_text\": \"How many orders were placed last week?\" }\n  ]\n}'\n",[768],{"type":40,"tag":124,"props":769,"children":770},{"__ignoreMap":161},[771,807,815,823,831,839,847,855,863,872,881,890],{"type":40,"tag":167,"props":772,"children":773},{"class":169,"line":170},[774,778,782,787,792,797,802],{"type":40,"tag":167,"props":775,"children":776},{"style":286},[777],{"type":45,"value":15},{"type":40,"tag":167,"props":779,"children":780},{"style":198},[781],{"type":45,"value":293},{"type":40,"tag":167,"props":783,"children":784},{"style":198},[785],{"type":45,"value":786}," prompt-sets-create",{"type":40,"tag":167,"props":788,"children":789},{"style":198},[790],{"type":45,"value":791}," --compact",{"type":40,"tag":167,"props":793,"children":794},{"style":198},[795],{"type":45,"value":796}," --body",{"type":40,"tag":167,"props":798,"children":799},{"style":209},[800],{"type":45,"value":801}," '",{"type":40,"tag":167,"props":803,"children":804},{"style":198},[805],{"type":45,"value":806},"{\n",{"type":40,"tag":167,"props":808,"children":809},{"class":169,"line":180},[810],{"type":40,"tag":167,"props":811,"children":812},{"style":198},[813],{"type":45,"value":814},"  \"model_id\": \"your-model-id\",\n",{"type":40,"tag":167,"props":816,"children":817},{"class":169,"line":27},[818],{"type":40,"tag":167,"props":819,"children":820},{"style":198},[821],{"type":45,"value":822},"  \"name\": \"Orders regression\",\n",{"type":40,"tag":167,"props":824,"children":825},{"class":169,"line":245},[826],{"type":40,"tag":167,"props":827,"children":828},{"style":198},[829],{"type":45,"value":830},"  \"slug\": \"orders-regression\",\n",{"type":40,"tag":167,"props":832,"children":833},{"class":169,"line":255},[834],{"type":40,"tag":167,"props":835,"children":836},{"style":198},[837],{"type":45,"value":838},"  \"description\": \"Core revenue + orders coverage\",\n",{"type":40,"tag":167,"props":840,"children":841},{"class":169,"line":264},[842],{"type":40,"tag":167,"props":843,"children":844},{"style":198},[845],{"type":45,"value":846},"  \"prompts\": [\n",{"type":40,"tag":167,"props":848,"children":849},{"class":169,"line":273},[850],{"type":40,"tag":167,"props":851,"children":852},{"style":198},[853],{"type":45,"value":854},"    { \"prompt_text\": \"Show me revenue by month\" },\n",{"type":40,"tag":167,"props":856,"children":857},{"class":169,"line":282},[858],{"type":40,"tag":167,"props":859,"children":860},{"style":198},[861],{"type":45,"value":862},"    { \"prompt_text\": \"What are the top 5 products by revenue?\",\n",{"type":40,"tag":167,"props":864,"children":866},{"class":169,"line":865},9,[867],{"type":40,"tag":167,"props":868,"children":869},{"style":198},[870],{"type":45,"value":871},"      \"expectation\": \"The top product by revenue should be Aniseed Syrup.\" },\n",{"type":40,"tag":167,"props":873,"children":875},{"class":169,"line":874},10,[876],{"type":40,"tag":167,"props":877,"children":878},{"style":198},[879],{"type":45,"value":880},"    { \"prompt_text\": \"How many orders were placed last week?\" }\n",{"type":40,"tag":167,"props":882,"children":884},{"class":169,"line":883},11,[885],{"type":40,"tag":167,"props":886,"children":887},{"style":198},[888],{"type":45,"value":889},"  ]\n",{"type":40,"tag":167,"props":891,"children":893},{"class":169,"line":892},12,[894,899],{"type":40,"tag":167,"props":895,"children":896},{"style":198},[897],{"type":45,"value":898},"}",{"type":40,"tag":167,"props":900,"children":901},{"style":209},[902],{"type":45,"value":903},"'\n",{"type":40,"tag":48,"props":905,"children":906},{},[907,909,915],{"type":45,"value":908},"The response includes the created ",{"type":40,"tag":124,"props":910,"children":912},{"className":911},[],[913],{"type":45,"value":914},"prompt_set.id",{"type":45,"value":916}," (a UUID) — capture it for the run.",{"type":40,"tag":578,"props":918,"children":919},{},[920,941],{"type":40,"tag":582,"props":921,"children":922},{},[923],{"type":40,"tag":586,"props":924,"children":925},{},[926,931,936],{"type":40,"tag":590,"props":927,"children":928},{},[929],{"type":45,"value":930},"Field",{"type":40,"tag":590,"props":932,"children":933},{},[934],{"type":45,"value":935},"Required",{"type":40,"tag":590,"props":937,"children":938},{},[939],{"type":45,"value":940},"Notes",{"type":40,"tag":601,"props":942,"children":943},{},[944,966,987,1021,1043],{"type":40,"tag":586,"props":945,"children":946},{},[947,956,961],{"type":40,"tag":608,"props":948,"children":949},{},[950],{"type":40,"tag":124,"props":951,"children":953},{"className":952},[],[954],{"type":45,"value":955},"model_id",{"type":40,"tag":608,"props":957,"children":958},{},[959],{"type":45,"value":960},"Yes",{"type":40,"tag":608,"props":962,"children":963},{},[964],{"type":45,"value":965},"Shared model UUID the set is bound to",{"type":40,"tag":586,"props":967,"children":968},{},[969,978,982],{"type":40,"tag":608,"props":970,"children":971},{},[972],{"type":40,"tag":124,"props":973,"children":975},{"className":974},[],[976],{"type":45,"value":977},"name",{"type":40,"tag":608,"props":979,"children":980},{},[981],{"type":45,"value":960},{"type":40,"tag":608,"props":983,"children":984},{},[985],{"type":45,"value":986},"≤ 255 chars",{"type":40,"tag":586,"props":988,"children":989},{},[990,999,1003],{"type":40,"tag":608,"props":991,"children":992},{},[993],{"type":40,"tag":124,"props":994,"children":996},{"className":995},[],[997],{"type":45,"value":998},"slug",{"type":40,"tag":608,"props":1000,"children":1001},{},[1002],{"type":45,"value":960},{"type":40,"tag":608,"props":1004,"children":1005},{},[1006,1008,1013,1015],{"type":45,"value":1007},"Unique per ",{"type":40,"tag":124,"props":1009,"children":1011},{"className":1010},[],[1012],{"type":45,"value":955},{"type":45,"value":1014},"; must match ",{"type":40,"tag":124,"props":1016,"children":1018},{"className":1017},[],[1019],{"type":45,"value":1020},"^[a-z][a-z0-9-]*$",{"type":40,"tag":586,"props":1022,"children":1023},{},[1024,1033,1038],{"type":40,"tag":608,"props":1025,"children":1026},{},[1027],{"type":40,"tag":124,"props":1028,"children":1030},{"className":1029},[],[1031],{"type":45,"value":1032},"description",{"type":40,"tag":608,"props":1034,"children":1035},{},[1036],{"type":45,"value":1037},"No",{"type":40,"tag":608,"props":1039,"children":1040},{},[1041],{"type":45,"value":1042},"≤ 1024 chars",{"type":40,"tag":586,"props":1044,"children":1045},{},[1046,1055,1059],{"type":40,"tag":608,"props":1047,"children":1048},{},[1049],{"type":40,"tag":124,"props":1050,"children":1052},{"className":1051},[],[1053],{"type":45,"value":1054},"prompts[]",{"type":40,"tag":608,"props":1056,"children":1057},{},[1058],{"type":45,"value":1037},{"type":40,"tag":608,"props":1060,"children":1061},{},[1062,1064,1070,1072,1077],{"type":45,"value":1063},"≤ 25; each needs ",{"type":40,"tag":124,"props":1065,"children":1067},{"className":1066},[],[1068],{"type":45,"value":1069},"prompt_text",{"type":45,"value":1071}," (≤ 8000 chars), optional ",{"type":40,"tag":124,"props":1073,"children":1075},{"className":1074},[],[1076],{"type":45,"value":633},{"type":45,"value":1078}," (≤ 16000 chars)",{"type":40,"tag":48,"props":1080,"children":1081},{},[1082,1087],{"type":40,"tag":54,"props":1083,"children":1084},{},[1085],{"type":45,"value":1086},"Find or update an existing set",{"type":45,"value":1088}," instead of recreating:",{"type":40,"tag":156,"props":1090,"children":1092},{"className":158,"code":1091,"language":160,"meta":161,"style":161},"omni ai-eval prompt-sets-list --model-ids your-model-id --compact   # discover sets + ids\nomni ai-eval prompt-sets-get \u003CpromptSetId> --compact                # full set with prompts\n",[1093],{"type":40,"tag":124,"props":1094,"children":1095},{"__ignoreMap":161},[1096,1131],{"type":40,"tag":167,"props":1097,"children":1098},{"class":169,"line":170},[1099,1103,1107,1112,1117,1122,1126],{"type":40,"tag":167,"props":1100,"children":1101},{"style":286},[1102],{"type":45,"value":15},{"type":40,"tag":167,"props":1104,"children":1105},{"style":198},[1106],{"type":45,"value":293},{"type":40,"tag":167,"props":1108,"children":1109},{"style":198},[1110],{"type":45,"value":1111}," prompt-sets-list",{"type":40,"tag":167,"props":1113,"children":1114},{"style":198},[1115],{"type":45,"value":1116}," --model-ids",{"type":40,"tag":167,"props":1118,"children":1119},{"style":198},[1120],{"type":45,"value":1121}," your-model-id",{"type":40,"tag":167,"props":1123,"children":1124},{"style":198},[1125],{"type":45,"value":791},{"type":40,"tag":167,"props":1127,"children":1128},{"style":174},[1129],{"type":45,"value":1130},"   # discover sets + ids\n",{"type":40,"tag":167,"props":1132,"children":1133},{"class":169,"line":180},[1134,1138,1142,1147,1151,1156,1161,1166,1170],{"type":40,"tag":167,"props":1135,"children":1136},{"style":286},[1137],{"type":45,"value":15},{"type":40,"tag":167,"props":1139,"children":1140},{"style":198},[1141],{"type":45,"value":293},{"type":40,"tag":167,"props":1143,"children":1144},{"style":198},[1145],{"type":45,"value":1146}," prompt-sets-get",{"type":40,"tag":167,"props":1148,"children":1149},{"style":209},[1150],{"type":45,"value":401},{"type":40,"tag":167,"props":1152,"children":1153},{"style":198},[1154],{"type":45,"value":1155},"promptSetI",{"type":40,"tag":167,"props":1157,"children":1158},{"style":409},[1159],{"type":45,"value":1160},"d",{"type":40,"tag":167,"props":1162,"children":1163},{"style":209},[1164],{"type":45,"value":1165},">",{"type":40,"tag":167,"props":1167,"children":1168},{"style":198},[1169],{"type":45,"value":791},{"type":40,"tag":167,"props":1171,"children":1172},{"style":174},[1173],{"type":45,"value":1174},"                # full set with prompts\n",{"type":40,"tag":48,"props":1176,"children":1177},{},[1178,1184,1186,1191,1193,1199,1201,1207,1209,1214,1216,1221],{"type":40,"tag":124,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":45,"value":1183},"prompt-sets-update",{"type":45,"value":1185}," replaces the ",{"type":40,"tag":54,"props":1187,"children":1188},{},[1189],{"type":45,"value":1190},"entire",{"type":45,"value":1192}," ",{"type":40,"tag":124,"props":1194,"children":1196},{"className":1195},[],[1197],{"type":45,"value":1198},"prompts",{"type":45,"value":1200}," list: omitted prompts are deleted, entries with no ",{"type":40,"tag":124,"props":1202,"children":1204},{"className":1203},[],[1205],{"type":45,"value":1206},"id",{"type":45,"value":1208}," are created, entries with a matching ",{"type":40,"tag":124,"props":1210,"children":1212},{"className":1211},[],[1213],{"type":45,"value":1206},{"type":45,"value":1215}," are updated in place. To add one prompt, send the full desired list (existing prompts carry their ",{"type":40,"tag":124,"props":1217,"children":1219},{"className":1218},[],[1220],{"type":45,"value":1206},{"type":45,"value":1222},").",{"type":40,"tag":156,"props":1224,"children":1226},{"className":158,"code":1225,"language":160,"meta":161,"style":161},"omni ai-eval prompt-sets-update \u003CpromptSetId> --compact --body '{\n  \"prompts\": [\n    { \"id\": \"\u003Cexisting-prompt-id>\", \"prompt_text\": \"Show me revenue by month\" },\n    { \"prompt_text\": \"Revenue by month, last 12 months only\" }\n  ]\n}'\n",[1227],{"type":40,"tag":124,"props":1228,"children":1229},{"__ignoreMap":161},[1230,1278,1285,1293,1301,1308],{"type":40,"tag":167,"props":1231,"children":1232},{"class":169,"line":170},[1233,1237,1241,1246,1250,1254,1258,1262,1266,1270,1274],{"type":40,"tag":167,"props":1234,"children":1235},{"style":286},[1236],{"type":45,"value":15},{"type":40,"tag":167,"props":1238,"children":1239},{"style":198},[1240],{"type":45,"value":293},{"type":40,"tag":167,"props":1242,"children":1243},{"style":198},[1244],{"type":45,"value":1245}," prompt-sets-update",{"type":40,"tag":167,"props":1247,"children":1248},{"style":209},[1249],{"type":45,"value":401},{"type":40,"tag":167,"props":1251,"children":1252},{"style":198},[1253],{"type":45,"value":1155},{"type":40,"tag":167,"props":1255,"children":1256},{"style":409},[1257],{"type":45,"value":1160},{"type":40,"tag":167,"props":1259,"children":1260},{"style":209},[1261],{"type":45,"value":1165},{"type":40,"tag":167,"props":1263,"children":1264},{"style":198},[1265],{"type":45,"value":791},{"type":40,"tag":167,"props":1267,"children":1268},{"style":198},[1269],{"type":45,"value":796},{"type":40,"tag":167,"props":1271,"children":1272},{"style":209},[1273],{"type":45,"value":801},{"type":40,"tag":167,"props":1275,"children":1276},{"style":198},[1277],{"type":45,"value":806},{"type":40,"tag":167,"props":1279,"children":1280},{"class":169,"line":180},[1281],{"type":40,"tag":167,"props":1282,"children":1283},{"style":198},[1284],{"type":45,"value":846},{"type":40,"tag":167,"props":1286,"children":1287},{"class":169,"line":27},[1288],{"type":40,"tag":167,"props":1289,"children":1290},{"style":198},[1291],{"type":45,"value":1292},"    { \"id\": \"\u003Cexisting-prompt-id>\", \"prompt_text\": \"Show me revenue by month\" },\n",{"type":40,"tag":167,"props":1294,"children":1295},{"class":169,"line":245},[1296],{"type":40,"tag":167,"props":1297,"children":1298},{"style":198},[1299],{"type":45,"value":1300},"    { \"prompt_text\": \"Revenue by month, last 12 months only\" }\n",{"type":40,"tag":167,"props":1302,"children":1303},{"class":169,"line":255},[1304],{"type":40,"tag":167,"props":1305,"children":1306},{"style":198},[1307],{"type":45,"value":889},{"type":40,"tag":167,"props":1309,"children":1310},{"class":169,"line":264},[1311,1315],{"type":40,"tag":167,"props":1312,"children":1313},{"style":198},[1314],{"type":45,"value":898},{"type":40,"tag":167,"props":1316,"children":1317},{"style":209},[1318],{"type":45,"value":903},{"type":40,"tag":1320,"props":1321,"children":1323},"h3",{"id":1322},"writing-good-prompts-and-expectations",[1324],{"type":45,"value":1325},"Writing good prompts and expectations",{"type":40,"tag":1327,"props":1328,"children":1329},"ul",{},[1330,1341,1351,1361],{"type":40,"tag":1331,"props":1332,"children":1333},"li",{},[1334,1339],{"type":40,"tag":54,"props":1335,"children":1336},{},[1337],{"type":45,"value":1338},"Mirror real user questions",{"type":45,"value":1340}," — pull from the AI usage analytics dashboard rather than inventing phrasing that echoes field names.",{"type":40,"tag":1331,"props":1342,"children":1343},{},[1344,1349],{"type":40,"tag":54,"props":1345,"children":1346},{},[1347],{"type":45,"value":1348},"Favor breadth over depth",{"type":45,"value":1350}," — one prompt across many topics yields more signal than ten on one topic.",{"type":40,"tag":1331,"props":1352,"children":1353},{},[1354,1359],{"type":40,"tag":54,"props":1355,"children":1356},{},[1357],{"type":45,"value":1358},"Add a regression prompt",{"type":45,"value":1360}," whenever an answer turns out wrong, so the same failure is caught next time.",{"type":40,"tag":1331,"props":1362,"children":1363},{},[1364,1374,1376,1381],{"type":40,"tag":54,"props":1365,"children":1366},{},[1367,1369],{"type":45,"value":1368},"Set an ",{"type":40,"tag":124,"props":1370,"children":1372},{"className":1371},[],[1373],{"type":45,"value":633},{"type":45,"value":1375}," only when a prompt has a known answer worth pinning: a value or ranking (\"top product should be Aniseed Syrup\"), a direction (\"revenue should be up YoY\"), or a required breakdown. The judge treats a ",{"type":40,"tag":102,"props":1377,"children":1378},{},[1379],{"type":45,"value":1380},"material",{"type":45,"value":1382}," divergence (wrong numbers, wrong direction, missing required result) as a failure but ignores wording\u002Fformatting differences. With no expectation, the judge decides whether the answer is correct on its own terms. The expectation is shown only to the judge — the evaluated AI never sees it.",{"type":40,"tag":149,"props":1384,"children":1386},{"id":1385},"step-2-run-an-eval",[1387],{"type":45,"value":1388},"Step 2 — Run an eval",{"type":40,"tag":156,"props":1390,"children":1392},{"className":158,"code":1391,"language":160,"meta":161,"style":161},"# Against main:\nomni ai-eval runs-create --compact --body '{\n  \"prompt_set_id\": \"\u003CpromptSetId>\",\n  \"description\": \"Baseline on main\"\n}'\n\n# Against a branch (measures a model-context change before promotion):\nomni ai-eval runs-create --compact --body '{\n  \"prompt_set_id\": \"\u003CpromptSetId>\",\n  \"description\": \"After adding ai_context to order_items\",\n  \"run_config\": { \"branch_id\": \"\u003CbranchId>\" }\n}'\n",[1393],{"type":40,"tag":124,"props":1394,"children":1395},{"__ignoreMap":161},[1396,1404,1436,1444,1452,1463,1470,1478,1509,1516,1524,1532],{"type":40,"tag":167,"props":1397,"children":1398},{"class":169,"line":170},[1399],{"type":40,"tag":167,"props":1400,"children":1401},{"style":174},[1402],{"type":45,"value":1403},"# Against main:\n",{"type":40,"tag":167,"props":1405,"children":1406},{"class":169,"line":180},[1407,1411,1415,1420,1424,1428,1432],{"type":40,"tag":167,"props":1408,"children":1409},{"style":286},[1410],{"type":45,"value":15},{"type":40,"tag":167,"props":1412,"children":1413},{"style":198},[1414],{"type":45,"value":293},{"type":40,"tag":167,"props":1416,"children":1417},{"style":198},[1418],{"type":45,"value":1419}," runs-create",{"type":40,"tag":167,"props":1421,"children":1422},{"style":198},[1423],{"type":45,"value":791},{"type":40,"tag":167,"props":1425,"children":1426},{"style":198},[1427],{"type":45,"value":796},{"type":40,"tag":167,"props":1429,"children":1430},{"style":209},[1431],{"type":45,"value":801},{"type":40,"tag":167,"props":1433,"children":1434},{"style":198},[1435],{"type":45,"value":806},{"type":40,"tag":167,"props":1437,"children":1438},{"class":169,"line":27},[1439],{"type":40,"tag":167,"props":1440,"children":1441},{"style":198},[1442],{"type":45,"value":1443},"  \"prompt_set_id\": \"\u003CpromptSetId>\",\n",{"type":40,"tag":167,"props":1445,"children":1446},{"class":169,"line":245},[1447],{"type":40,"tag":167,"props":1448,"children":1449},{"style":198},[1450],{"type":45,"value":1451},"  \"description\": \"Baseline on main\"\n",{"type":40,"tag":167,"props":1453,"children":1454},{"class":169,"line":255},[1455,1459],{"type":40,"tag":167,"props":1456,"children":1457},{"style":198},[1458],{"type":45,"value":898},{"type":40,"tag":167,"props":1460,"children":1461},{"style":209},[1462],{"type":45,"value":903},{"type":40,"tag":167,"props":1464,"children":1465},{"class":169,"line":264},[1466],{"type":40,"tag":167,"props":1467,"children":1468},{"emptyLinePlaceholder":249},[1469],{"type":45,"value":252},{"type":40,"tag":167,"props":1471,"children":1472},{"class":169,"line":273},[1473],{"type":40,"tag":167,"props":1474,"children":1475},{"style":174},[1476],{"type":45,"value":1477},"# Against a branch (measures a model-context change before promotion):\n",{"type":40,"tag":167,"props":1479,"children":1480},{"class":169,"line":282},[1481,1485,1489,1493,1497,1501,1505],{"type":40,"tag":167,"props":1482,"children":1483},{"style":286},[1484],{"type":45,"value":15},{"type":40,"tag":167,"props":1486,"children":1487},{"style":198},[1488],{"type":45,"value":293},{"type":40,"tag":167,"props":1490,"children":1491},{"style":198},[1492],{"type":45,"value":1419},{"type":40,"tag":167,"props":1494,"children":1495},{"style":198},[1496],{"type":45,"value":791},{"type":40,"tag":167,"props":1498,"children":1499},{"style":198},[1500],{"type":45,"value":796},{"type":40,"tag":167,"props":1502,"children":1503},{"style":209},[1504],{"type":45,"value":801},{"type":40,"tag":167,"props":1506,"children":1507},{"style":198},[1508],{"type":45,"value":806},{"type":40,"tag":167,"props":1510,"children":1511},{"class":169,"line":865},[1512],{"type":40,"tag":167,"props":1513,"children":1514},{"style":198},[1515],{"type":45,"value":1443},{"type":40,"tag":167,"props":1517,"children":1518},{"class":169,"line":874},[1519],{"type":40,"tag":167,"props":1520,"children":1521},{"style":198},[1522],{"type":45,"value":1523},"  \"description\": \"After adding ai_context to order_items\",\n",{"type":40,"tag":167,"props":1525,"children":1526},{"class":169,"line":883},[1527],{"type":40,"tag":167,"props":1528,"children":1529},{"style":198},[1530],{"type":45,"value":1531},"  \"run_config\": { \"branch_id\": \"\u003CbranchId>\" }\n",{"type":40,"tag":167,"props":1533,"children":1534},{"class":169,"line":892},[1535,1539],{"type":40,"tag":167,"props":1536,"children":1537},{"style":198},[1538],{"type":45,"value":898},{"type":40,"tag":167,"props":1540,"children":1541},{"style":209},[1542],{"type":45,"value":903},{"type":40,"tag":48,"props":1544,"children":1545},{},[1546,1548,1554,1555,1561,1563,1569],{"type":45,"value":1547},"The response returns ",{"type":40,"tag":124,"props":1549,"children":1551},{"className":1550},[],[1552],{"type":45,"value":1553},"run.id",{"type":45,"value":67},{"type":40,"tag":124,"props":1556,"children":1558},{"className":1557},[],[1559],{"type":45,"value":1560},"job_count",{"type":45,"value":1562}," (one agentic job per prompt). Omit ",{"type":40,"tag":124,"props":1564,"children":1566},{"className":1565},[],[1567],{"type":45,"value":1568},"run_config.branch_id",{"type":45,"value":1570}," to run against the live shared model.",{"type":40,"tag":110,"props":1572,"children":1573},{},[1574],{"type":40,"tag":48,"props":1575,"children":1576},{},[1577,1582,1584,1589,1591,1597,1599,1605,1607,1613],{"type":40,"tag":54,"props":1578,"children":1579},{},[1580],{"type":45,"value":1581},"Concurrency cap",{"type":45,"value":1583},": at most ",{"type":40,"tag":54,"props":1585,"children":1586},{},[1587],{"type":45,"value":1588},"2 eval runs in flight at once",{"type":45,"value":1590},". A ",{"type":40,"tag":124,"props":1592,"children":1594},{"className":1593},[],[1595],{"type":45,"value":1596},"429",{"type":45,"value":1598}," means the per-user active-run cap is reached — wait for an in-flight run to finish or cancel one. A ",{"type":40,"tag":124,"props":1600,"children":1602},{"className":1601},[],[1603],{"type":45,"value":1604},"503",{"type":45,"value":1606}," means eval is paused for the org. Check ",{"type":40,"tag":124,"props":1608,"children":1610},{"className":1609},[],[1611],{"type":45,"value":1612},"runs-list",{"type":45,"value":1614}," before launching.",{"type":40,"tag":1320,"props":1616,"children":1618},{"id":1617},"poll-for-completion",[1619],{"type":45,"value":1620},"Poll for completion",{"type":40,"tag":156,"props":1622,"children":1624},{"className":158,"code":1623,"language":160,"meta":161,"style":161},"omni ai-eval runs-get \u003CrunId> --compact\n",[1625],{"type":40,"tag":124,"props":1626,"children":1627},{"__ignoreMap":161},[1628],{"type":40,"tag":167,"props":1629,"children":1630},{"class":169,"line":170},[1631,1635,1639,1644,1648,1653,1657,1661],{"type":40,"tag":167,"props":1632,"children":1633},{"style":286},[1634],{"type":45,"value":15},{"type":40,"tag":167,"props":1636,"children":1637},{"style":198},[1638],{"type":45,"value":293},{"type":40,"tag":167,"props":1640,"children":1641},{"style":198},[1642],{"type":45,"value":1643}," runs-get",{"type":40,"tag":167,"props":1645,"children":1646},{"style":209},[1647],{"type":45,"value":401},{"type":40,"tag":167,"props":1649,"children":1650},{"style":198},[1651],{"type":45,"value":1652},"runI",{"type":40,"tag":167,"props":1654,"children":1655},{"style":409},[1656],{"type":45,"value":1160},{"type":40,"tag":167,"props":1658,"children":1659},{"style":209},[1660],{"type":45,"value":1165},{"type":40,"tag":167,"props":1662,"children":1663},{"style":198},[1664],{"type":45,"value":1665}," --compact\n",{"type":40,"tag":48,"props":1667,"children":1668},{},[1669,1671,1677,1679,1685,1686,1692,1694,1700,1702,1708,1709,1715,1716,1721,1723,1729],{"type":45,"value":1670},"Poll with backoff (e.g. 5s, 10s, 20s) until the run's ",{"type":40,"tag":124,"props":1672,"children":1674},{"className":1673},[],[1675],{"type":45,"value":1676},"status",{"type":45,"value":1678}," is terminal — ",{"type":40,"tag":124,"props":1680,"children":1682},{"className":1681},[],[1683],{"type":45,"value":1684},"COMPLETE",{"type":45,"value":469},{"type":40,"tag":124,"props":1687,"children":1689},{"className":1688},[],[1690],{"type":45,"value":1691},"CANCELLED",{"type":45,"value":1693},". Track progress with each result's ",{"type":40,"tag":124,"props":1695,"children":1697},{"className":1696},[],[1698],{"type":45,"value":1699},"agentic_job.state",{"type":45,"value":1701}," (",{"type":40,"tag":124,"props":1703,"children":1705},{"className":1704},[],[1706],{"type":45,"value":1707},"QUEUED",{"type":45,"value":60},{"type":40,"tag":124,"props":1710,"children":1712},{"className":1711},[],[1713],{"type":45,"value":1714},"EXECUTING",{"type":45,"value":60},{"type":40,"tag":124,"props":1717,"children":1719},{"className":1718},[],[1720],{"type":45,"value":1684},{"type":45,"value":1722},"\u002F",{"type":40,"tag":124,"props":1724,"children":1726},{"className":1725},[],[1727],{"type":45,"value":1728},"FAILED",{"type":45,"value":1730},"). Don't hammer the endpoint.",{"type":40,"tag":149,"props":1732,"children":1734},{"id":1733},"step-3-read-results",[1735],{"type":45,"value":1736},"Step 3 — Read results",{"type":40,"tag":48,"props":1738,"children":1739},{},[1740,1746,1748,1754],{"type":40,"tag":124,"props":1741,"children":1743},{"className":1742},[],[1744],{"type":45,"value":1745},"runs-get",{"type":45,"value":1747}," returns ",{"type":40,"tag":124,"props":1749,"children":1751},{"className":1750},[],[1752],{"type":45,"value":1753},"results[]",{"type":45,"value":1755},", one row per prompt:",{"type":40,"tag":156,"props":1757,"children":1761},{"className":1758,"code":1759,"language":1760,"meta":161,"style":161},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"run\": {\n    \"status\": \"COMPLETE\",\n    \"branch_id\": null,\n    \"results\": [\n      {\n        \"prompt\": \"What are the top 5 products by revenue?\",\n        \"score\": 1,\n        \"error_reason\": null,\n        \"cost\": 0.0021,\n        \"scoring_cost\": 0.0004,\n        \"timing_ms\": 4321,\n        \"agentic_job\": { \"state\": \"COMPLETE\", \"conversation_id\": \"conv-uuid\", \"id\": \"job-uuid\" }\n      }\n    ]\n  }\n}\n","json",[1762],{"type":40,"tag":124,"props":1763,"children":1764},{"__ignoreMap":161},[1765,1772,1801,1838,1863,1888,1896,1935,1964,1988,2017,2046,2075,2204,2213,2222,2231],{"type":40,"tag":167,"props":1766,"children":1767},{"class":169,"line":170},[1768],{"type":40,"tag":167,"props":1769,"children":1770},{"style":209},[1771],{"type":45,"value":806},{"type":40,"tag":167,"props":1773,"children":1774},{"class":169,"line":180},[1775,1780,1786,1791,1796],{"type":40,"tag":167,"props":1776,"children":1777},{"style":209},[1778],{"type":45,"value":1779},"  \"",{"type":40,"tag":167,"props":1781,"children":1783},{"style":1782},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1784],{"type":45,"value":1785},"run",{"type":40,"tag":167,"props":1787,"children":1788},{"style":209},[1789],{"type":45,"value":1790},"\"",{"type":40,"tag":167,"props":1792,"children":1793},{"style":209},[1794],{"type":45,"value":1795},":",{"type":40,"tag":167,"props":1797,"children":1798},{"style":209},[1799],{"type":45,"value":1800}," {\n",{"type":40,"tag":167,"props":1802,"children":1803},{"class":169,"line":27},[1804,1809,1813,1817,1821,1825,1829,1833],{"type":40,"tag":167,"props":1805,"children":1806},{"style":209},[1807],{"type":45,"value":1808},"    \"",{"type":40,"tag":167,"props":1810,"children":1811},{"style":286},[1812],{"type":45,"value":1676},{"type":40,"tag":167,"props":1814,"children":1815},{"style":209},[1816],{"type":45,"value":1790},{"type":40,"tag":167,"props":1818,"children":1819},{"style":209},[1820],{"type":45,"value":1795},{"type":40,"tag":167,"props":1822,"children":1823},{"style":209},[1824],{"type":45,"value":232},{"type":40,"tag":167,"props":1826,"children":1827},{"style":198},[1828],{"type":45,"value":1684},{"type":40,"tag":167,"props":1830,"children":1831},{"style":209},[1832],{"type":45,"value":1790},{"type":40,"tag":167,"props":1834,"children":1835},{"style":209},[1836],{"type":45,"value":1837},",\n",{"type":40,"tag":167,"props":1839,"children":1840},{"class":169,"line":245},[1841,1845,1850,1854,1858],{"type":40,"tag":167,"props":1842,"children":1843},{"style":209},[1844],{"type":45,"value":1808},{"type":40,"tag":167,"props":1846,"children":1847},{"style":286},[1848],{"type":45,"value":1849},"branch_id",{"type":40,"tag":167,"props":1851,"children":1852},{"style":209},[1853],{"type":45,"value":1790},{"type":40,"tag":167,"props":1855,"children":1856},{"style":209},[1857],{"type":45,"value":1795},{"type":40,"tag":167,"props":1859,"children":1860},{"style":209},[1861],{"type":45,"value":1862}," null,\n",{"type":40,"tag":167,"props":1864,"children":1865},{"class":169,"line":255},[1866,1870,1875,1879,1883],{"type":40,"tag":167,"props":1867,"children":1868},{"style":209},[1869],{"type":45,"value":1808},{"type":40,"tag":167,"props":1871,"children":1872},{"style":286},[1873],{"type":45,"value":1874},"results",{"type":40,"tag":167,"props":1876,"children":1877},{"style":209},[1878],{"type":45,"value":1790},{"type":40,"tag":167,"props":1880,"children":1881},{"style":209},[1882],{"type":45,"value":1795},{"type":40,"tag":167,"props":1884,"children":1885},{"style":209},[1886],{"type":45,"value":1887}," [\n",{"type":40,"tag":167,"props":1889,"children":1890},{"class":169,"line":264},[1891],{"type":40,"tag":167,"props":1892,"children":1893},{"style":209},[1894],{"type":45,"value":1895},"      {\n",{"type":40,"tag":167,"props":1897,"children":1898},{"class":169,"line":273},[1899,1904,1910,1914,1918,1922,1927,1931],{"type":40,"tag":167,"props":1900,"children":1901},{"style":209},[1902],{"type":45,"value":1903},"        \"",{"type":40,"tag":167,"props":1905,"children":1907},{"style":1906},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1908],{"type":45,"value":1909},"prompt",{"type":40,"tag":167,"props":1911,"children":1912},{"style":209},[1913],{"type":45,"value":1790},{"type":40,"tag":167,"props":1915,"children":1916},{"style":209},[1917],{"type":45,"value":1795},{"type":40,"tag":167,"props":1919,"children":1920},{"style":209},[1921],{"type":45,"value":232},{"type":40,"tag":167,"props":1923,"children":1924},{"style":198},[1925],{"type":45,"value":1926},"What are the top 5 products by revenue?",{"type":40,"tag":167,"props":1928,"children":1929},{"style":209},[1930],{"type":45,"value":1790},{"type":40,"tag":167,"props":1932,"children":1933},{"style":209},[1934],{"type":45,"value":1837},{"type":40,"tag":167,"props":1936,"children":1937},{"class":169,"line":282},[1938,1942,1947,1951,1955,1960],{"type":40,"tag":167,"props":1939,"children":1940},{"style":209},[1941],{"type":45,"value":1903},{"type":40,"tag":167,"props":1943,"children":1944},{"style":1906},[1945],{"type":45,"value":1946},"score",{"type":40,"tag":167,"props":1948,"children":1949},{"style":209},[1950],{"type":45,"value":1790},{"type":40,"tag":167,"props":1952,"children":1953},{"style":209},[1954],{"type":45,"value":1795},{"type":40,"tag":167,"props":1956,"children":1957},{"style":1906},[1958],{"type":45,"value":1959}," 1",{"type":40,"tag":167,"props":1961,"children":1962},{"style":209},[1963],{"type":45,"value":1837},{"type":40,"tag":167,"props":1965,"children":1966},{"class":169,"line":865},[1967,1971,1976,1980,1984],{"type":40,"tag":167,"props":1968,"children":1969},{"style":209},[1970],{"type":45,"value":1903},{"type":40,"tag":167,"props":1972,"children":1973},{"style":1906},[1974],{"type":45,"value":1975},"error_reason",{"type":40,"tag":167,"props":1977,"children":1978},{"style":209},[1979],{"type":45,"value":1790},{"type":40,"tag":167,"props":1981,"children":1982},{"style":209},[1983],{"type":45,"value":1795},{"type":40,"tag":167,"props":1985,"children":1986},{"style":209},[1987],{"type":45,"value":1862},{"type":40,"tag":167,"props":1989,"children":1990},{"class":169,"line":874},[1991,1995,2000,2004,2008,2013],{"type":40,"tag":167,"props":1992,"children":1993},{"style":209},[1994],{"type":45,"value":1903},{"type":40,"tag":167,"props":1996,"children":1997},{"style":1906},[1998],{"type":45,"value":1999},"cost",{"type":40,"tag":167,"props":2001,"children":2002},{"style":209},[2003],{"type":45,"value":1790},{"type":40,"tag":167,"props":2005,"children":2006},{"style":209},[2007],{"type":45,"value":1795},{"type":40,"tag":167,"props":2009,"children":2010},{"style":1906},[2011],{"type":45,"value":2012}," 0.0021",{"type":40,"tag":167,"props":2014,"children":2015},{"style":209},[2016],{"type":45,"value":1837},{"type":40,"tag":167,"props":2018,"children":2019},{"class":169,"line":883},[2020,2024,2029,2033,2037,2042],{"type":40,"tag":167,"props":2021,"children":2022},{"style":209},[2023],{"type":45,"value":1903},{"type":40,"tag":167,"props":2025,"children":2026},{"style":1906},[2027],{"type":45,"value":2028},"scoring_cost",{"type":40,"tag":167,"props":2030,"children":2031},{"style":209},[2032],{"type":45,"value":1790},{"type":40,"tag":167,"props":2034,"children":2035},{"style":209},[2036],{"type":45,"value":1795},{"type":40,"tag":167,"props":2038,"children":2039},{"style":1906},[2040],{"type":45,"value":2041}," 0.0004",{"type":40,"tag":167,"props":2043,"children":2044},{"style":209},[2045],{"type":45,"value":1837},{"type":40,"tag":167,"props":2047,"children":2048},{"class":169,"line":892},[2049,2053,2058,2062,2066,2071],{"type":40,"tag":167,"props":2050,"children":2051},{"style":209},[2052],{"type":45,"value":1903},{"type":40,"tag":167,"props":2054,"children":2055},{"style":1906},[2056],{"type":45,"value":2057},"timing_ms",{"type":40,"tag":167,"props":2059,"children":2060},{"style":209},[2061],{"type":45,"value":1790},{"type":40,"tag":167,"props":2063,"children":2064},{"style":209},[2065],{"type":45,"value":1795},{"type":40,"tag":167,"props":2067,"children":2068},{"style":1906},[2069],{"type":45,"value":2070}," 4321",{"type":40,"tag":167,"props":2072,"children":2073},{"style":209},[2074],{"type":45,"value":1837},{"type":40,"tag":167,"props":2076,"children":2078},{"class":169,"line":2077},13,[2079,2083,2088,2092,2096,2101,2105,2111,2115,2119,2123,2127,2131,2136,2140,2145,2149,2153,2157,2162,2166,2170,2174,2178,2182,2186,2190,2195,2199],{"type":40,"tag":167,"props":2080,"children":2081},{"style":209},[2082],{"type":45,"value":1903},{"type":40,"tag":167,"props":2084,"children":2085},{"style":1906},[2086],{"type":45,"value":2087},"agentic_job",{"type":40,"tag":167,"props":2089,"children":2090},{"style":209},[2091],{"type":45,"value":1790},{"type":40,"tag":167,"props":2093,"children":2094},{"style":209},[2095],{"type":45,"value":1795},{"type":40,"tag":167,"props":2097,"children":2098},{"style":209},[2099],{"type":45,"value":2100}," {",{"type":40,"tag":167,"props":2102,"children":2103},{"style":209},[2104],{"type":45,"value":232},{"type":40,"tag":167,"props":2106,"children":2108},{"style":2107},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[2109],{"type":45,"value":2110},"state",{"type":40,"tag":167,"props":2112,"children":2113},{"style":209},[2114],{"type":45,"value":1790},{"type":40,"tag":167,"props":2116,"children":2117},{"style":209},[2118],{"type":45,"value":1795},{"type":40,"tag":167,"props":2120,"children":2121},{"style":209},[2122],{"type":45,"value":232},{"type":40,"tag":167,"props":2124,"children":2125},{"style":198},[2126],{"type":45,"value":1684},{"type":40,"tag":167,"props":2128,"children":2129},{"style":209},[2130],{"type":45,"value":1790},{"type":40,"tag":167,"props":2132,"children":2133},{"style":209},[2134],{"type":45,"value":2135},",",{"type":40,"tag":167,"props":2137,"children":2138},{"style":209},[2139],{"type":45,"value":232},{"type":40,"tag":167,"props":2141,"children":2142},{"style":2107},[2143],{"type":45,"value":2144},"conversation_id",{"type":40,"tag":167,"props":2146,"children":2147},{"style":209},[2148],{"type":45,"value":1790},{"type":40,"tag":167,"props":2150,"children":2151},{"style":209},[2152],{"type":45,"value":1795},{"type":40,"tag":167,"props":2154,"children":2155},{"style":209},[2156],{"type":45,"value":232},{"type":40,"tag":167,"props":2158,"children":2159},{"style":198},[2160],{"type":45,"value":2161},"conv-uuid",{"type":40,"tag":167,"props":2163,"children":2164},{"style":209},[2165],{"type":45,"value":1790},{"type":40,"tag":167,"props":2167,"children":2168},{"style":209},[2169],{"type":45,"value":2135},{"type":40,"tag":167,"props":2171,"children":2172},{"style":209},[2173],{"type":45,"value":232},{"type":40,"tag":167,"props":2175,"children":2176},{"style":2107},[2177],{"type":45,"value":1206},{"type":40,"tag":167,"props":2179,"children":2180},{"style":209},[2181],{"type":45,"value":1790},{"type":40,"tag":167,"props":2183,"children":2184},{"style":209},[2185],{"type":45,"value":1795},{"type":40,"tag":167,"props":2187,"children":2188},{"style":209},[2189],{"type":45,"value":232},{"type":40,"tag":167,"props":2191,"children":2192},{"style":198},[2193],{"type":45,"value":2194},"job-uuid",{"type":40,"tag":167,"props":2196,"children":2197},{"style":209},[2198],{"type":45,"value":1790},{"type":40,"tag":167,"props":2200,"children":2201},{"style":209},[2202],{"type":45,"value":2203}," }\n",{"type":40,"tag":167,"props":2205,"children":2207},{"class":169,"line":2206},14,[2208],{"type":40,"tag":167,"props":2209,"children":2210},{"style":209},[2211],{"type":45,"value":2212},"      }\n",{"type":40,"tag":167,"props":2214,"children":2216},{"class":169,"line":2215},15,[2217],{"type":40,"tag":167,"props":2218,"children":2219},{"style":209},[2220],{"type":45,"value":2221},"    ]\n",{"type":40,"tag":167,"props":2223,"children":2225},{"class":169,"line":2224},16,[2226],{"type":40,"tag":167,"props":2227,"children":2228},{"style":209},[2229],{"type":45,"value":2230},"  }\n",{"type":40,"tag":167,"props":2232,"children":2234},{"class":169,"line":2233},17,[2235],{"type":40,"tag":167,"props":2236,"children":2237},{"style":209},[2238],{"type":45,"value":2239},"}\n",{"type":40,"tag":578,"props":2241,"children":2242},{},[2243,2258],{"type":40,"tag":582,"props":2244,"children":2245},{},[2246],{"type":40,"tag":586,"props":2247,"children":2248},{},[2249,2253],{"type":40,"tag":590,"props":2250,"children":2251},{},[2252],{"type":45,"value":930},{"type":40,"tag":590,"props":2254,"children":2255},{},[2256],{"type":45,"value":2257},"Meaning",{"type":40,"tag":601,"props":2259,"children":2260},{},[2261,2291,2307,2329,2352],{"type":40,"tag":586,"props":2262,"children":2263},{},[2264,2272],{"type":40,"tag":608,"props":2265,"children":2266},{},[2267],{"type":40,"tag":124,"props":2268,"children":2270},{"className":2269},[],[2271],{"type":45,"value":1946},{"type":40,"tag":608,"props":2273,"children":2274},{},[2275,2277,2283,2285],{"type":45,"value":2276},"Judge verdict for the prompt — pass = ",{"type":40,"tag":124,"props":2278,"children":2280},{"className":2279},[],[2281],{"type":45,"value":2282},"1",{"type":45,"value":2284},", fail = ",{"type":40,"tag":124,"props":2286,"children":2288},{"className":2287},[],[2289],{"type":45,"value":2290},"0",{"type":40,"tag":586,"props":2292,"children":2293},{},[2294,2302],{"type":40,"tag":608,"props":2295,"children":2296},{},[2297],{"type":40,"tag":124,"props":2298,"children":2300},{"className":2299},[],[2301],{"type":45,"value":1975},{"type":40,"tag":608,"props":2303,"children":2304},{},[2305],{"type":45,"value":2306},"Set when the underlying agentic job failed",{"type":40,"tag":586,"props":2308,"children":2309},{},[2310,2324],{"type":40,"tag":608,"props":2311,"children":2312},{},[2313,2318,2319],{"type":40,"tag":124,"props":2314,"children":2316},{"className":2315},[],[2317],{"type":45,"value":1999},{"type":45,"value":718},{"type":40,"tag":124,"props":2320,"children":2322},{"className":2321},[],[2323],{"type":45,"value":2028},{"type":40,"tag":608,"props":2325,"children":2326},{},[2327],{"type":45,"value":2328},"LLM cost (USD) for the answer vs. for judging it",{"type":40,"tag":586,"props":2330,"children":2331},{},[2332,2340],{"type":40,"tag":608,"props":2333,"children":2334},{},[2335],{"type":40,"tag":124,"props":2336,"children":2338},{"className":2337},[],[2339],{"type":45,"value":2057},{"type":40,"tag":608,"props":2341,"children":2342},{},[2343,2345,2350],{"type":45,"value":2344},"Total ",{"type":40,"tag":54,"props":2346,"children":2347},{},[2348],{"type":45,"value":2349},"AI time",{"type":45,"value":2351}," in ms — all LLM processing and tool calls (matches the \"AI time\" column in the UI), not wall-clock duration",{"type":40,"tag":586,"props":2353,"children":2354},{},[2355,2364],{"type":40,"tag":608,"props":2356,"children":2357},{},[2358],{"type":40,"tag":124,"props":2359,"children":2361},{"className":2360},[],[2362],{"type":45,"value":2363},"agentic_job.conversation_id",{"type":40,"tag":608,"props":2365,"children":2366},{},[2367],{"type":45,"value":2368},"Open this chat to read the judge's full verdict, confidence, and rationale",{"type":40,"tag":48,"props":2370,"children":2371},{},[2372,2377,2379,2384,2386,2391,2393,2398],{"type":40,"tag":54,"props":2373,"children":2374},{},[2375],{"type":45,"value":2376},"Overall accuracy = the pass rate",{"type":45,"value":2378}," (mean of ",{"type":40,"tag":124,"props":2380,"children":2382},{"className":2381},[],[2383],{"type":45,"value":1946},{"type":45,"value":2385}," across results). Report it with the per-prompt breakdown, and for any failure, point to the ",{"type":40,"tag":124,"props":2387,"children":2389},{"className":2388},[],[2390],{"type":45,"value":2144},{"type":45,"value":2392}," so the user can read ",{"type":40,"tag":102,"props":2394,"children":2395},{},[2396],{"type":45,"value":2397},"why",{"type":45,"value":2399}," the judge failed it — that rationale is where the actionable signal lives.",{"type":40,"tag":156,"props":2401,"children":2405},{"className":2402,"code":2404,"language":45},[2403],"language-text","Eval run \"Baseline on main\" — 9\u002F12 passed (75.0%)\n  ✗ \"Revenue by quarter\"        — judge: summed a row-limited result as a total\n  ✗ \"Top products this year\"    — judge: date filter used calendar instead of fiscal year\n  ✗ \"Churn rate by segment\"     — agentic job FAILED (error_reason)\n  (open each conversation_id for the full rationale)\n",[2406],{"type":40,"tag":124,"props":2407,"children":2408},{"__ignoreMap":161},[2409],{"type":45,"value":2404},{"type":40,"tag":149,"props":2411,"children":2413},{"id":2412},"ab-comparison-branch-vs-main",[2414],{"type":45,"value":2415},"A\u002FB comparison: branch vs main",{"type":40,"tag":48,"props":2417,"children":2418},{},[2419,2421,2426,2428,2433],{"type":45,"value":2420},"The core workflow for measuring whether a model change helps. Run the ",{"type":40,"tag":54,"props":2422,"children":2423},{},[2424],{"type":45,"value":2425},"same prompt set",{"type":45,"value":2427}," twice — once on ",{"type":40,"tag":124,"props":2429,"children":2431},{"className":2430},[],[2432],{"type":45,"value":657},{"type":45,"value":2434},", once on the branch — then compare.",{"type":40,"tag":2436,"props":2437,"children":2438},"ol",{},[2439,2467,2486,2503],{"type":40,"tag":1331,"props":2440,"children":2441},{},[2442,2444,2449,2451,2457,2459,2465],{"type":45,"value":2443},"Create or identify the branch with the change (use ",{"type":40,"tag":124,"props":2445,"children":2447},{"className":2446},[],[2448],{"type":45,"value":137},{"type":45,"value":2450}," to apply ",{"type":40,"tag":124,"props":2452,"children":2454},{"className":2453},[],[2455],{"type":45,"value":2456},"ai_context",{"type":45,"value":2458},", fields, joins, ",{"type":40,"tag":124,"props":2460,"children":2462},{"className":2461},[],[2463],{"type":45,"value":2464},"ai_settings",{"type":45,"value":2466},", etc. on a branch).",{"type":40,"tag":1331,"props":2468,"children":2469},{},[2470,2476,2478,2484],{"type":40,"tag":124,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":45,"value":2475},"runs-create",{"type":45,"value":2477}," with no ",{"type":40,"tag":124,"props":2479,"children":2481},{"className":2480},[],[2482],{"type":45,"value":2483},"run_config",{"type":45,"value":2485}," → baseline run on main.",{"type":40,"tag":1331,"props":2487,"children":2488},{},[2489,2494,2496,2501],{"type":40,"tag":124,"props":2490,"children":2492},{"className":2491},[],[2493],{"type":45,"value":2475},{"type":45,"value":2495}," with ",{"type":40,"tag":124,"props":2497,"children":2499},{"className":2498},[],[2500],{"type":45,"value":1568},{"type":45,"value":2502}," → branch run.",{"type":40,"tag":1331,"props":2504,"children":2505},{},[2506,2508,2513],{"type":45,"value":2507},"Poll both to ",{"type":40,"tag":124,"props":2509,"children":2511},{"className":2510},[],[2512],{"type":45,"value":1684},{"type":45,"value":2514},", then diff per-prompt verdicts.",{"type":40,"tag":156,"props":2516,"children":2519},{"className":2517,"code":2518,"language":45},[2403],"A\u002FB: main vs branch\u002Fnew-context  (prompt set: orders-regression)\n                    main      branch     Δ\nAccuracy:           75.0%     91.7%     +16.7%\nPrompt credits:     0.024     0.026     +0.002\n\nRegressions (passed on main, failed on branch):\n  - rev-by-quarter\nImprovements (failed on main, passed on branch):\n  - top-products-this-year\n  - churn-rate-by-segment\n",[2520],{"type":40,"tag":124,"props":2521,"children":2522},{"__ignoreMap":161},[2523],{"type":45,"value":2518},{"type":40,"tag":110,"props":2525,"children":2526},{},[2527],{"type":40,"tag":48,"props":2528,"children":2529},{},[2530,2535,2537,2542],{"type":40,"tag":54,"props":2531,"children":2532},{},[2533],{"type":45,"value":2534},"Always check for regressions, not just net improvement.",{"type":45,"value":2536}," A higher overall pass rate can still hide a prompt that newly broke — an ",{"type":40,"tag":124,"props":2538,"children":2540},{"className":2539},[],[2541],{"type":45,"value":2456},{"type":45,"value":2543}," change that helps most prompts may conflict with one. Call out any prompt that passed on main but fails on the branch.",{"type":40,"tag":48,"props":2545,"children":2546},{},[2547],{"type":45,"value":2548},"Notes for an auditable comparison:",{"type":40,"tag":1327,"props":2550,"children":2551},{},[2552,2563,2583],{"type":40,"tag":1331,"props":2553,"children":2554},{},[2555,2557,2561],{"type":45,"value":2556},"Both runs must use the ",{"type":40,"tag":54,"props":2558,"children":2559},{},[2560],{"type":45,"value":2425},{"type":45,"value":2562}," so they evaluate identical prompts (the AI Hub comparison view enforces this too).",{"type":40,"tag":1331,"props":2564,"children":2565},{},[2566,2568,2573,2575,2581],{"type":45,"value":2567},"Record the full ",{"type":40,"tag":124,"props":2569,"children":2571},{"className":2570},[],[2572],{"type":45,"value":1849},{"type":45,"value":2574}," used and confirm the branch run's ",{"type":40,"tag":124,"props":2576,"children":2578},{"className":2577},[],[2579],{"type":45,"value":2580},"run.branch_id",{"type":45,"value":2582}," matches it — don't claim the branch was exercised without that.",{"type":40,"tag":1331,"props":2584,"children":2585},{},[2586,2588,2593],{"type":45,"value":2587},"Expectations are snapshotted per run, so editing a prompt's ",{"type":40,"tag":124,"props":2589,"children":2591},{"className":2590},[],[2592],{"type":45,"value":633},{"type":45,"value":2594}," later won't change how earlier runs were scored.",{"type":40,"tag":149,"props":2596,"children":2598},{"id":2597},"managing-prompt-sets-and-runs",[2599],{"type":45,"value":2600},"Managing prompt sets and runs",{"type":40,"tag":156,"props":2602,"children":2604},{"className":158,"code":2603,"language":160,"meta":161,"style":161},"omni ai-eval runs-list --prompt-set-id \u003CpromptSetId> --compact   # runs for a set, newest first\nomni ai-eval runs-cancel \u003CrunId>                                 # cancel an in-flight run (also archives it)\nomni ai-eval runs-archive \u003CrunId>                                # archive a finished run\nomni ai-eval runs-unarchive \u003CrunId>                              # restore an archived run\n\nomni ai-eval prompt-sets-archive \u003CpromptSetId>                   # archive a set (cancels its in-flight jobs)\nomni ai-eval prompt-sets-unarchive \u003CpromptSetId>                 # restore an archived set\n",[2605],{"type":40,"tag":124,"props":2606,"children":2607},{"__ignoreMap":161},[2608,2654,2691,2728,2765,2772,2809],{"type":40,"tag":167,"props":2609,"children":2610},{"class":169,"line":170},[2611,2615,2619,2624,2629,2633,2637,2641,2645,2649],{"type":40,"tag":167,"props":2612,"children":2613},{"style":286},[2614],{"type":45,"value":15},{"type":40,"tag":167,"props":2616,"children":2617},{"style":198},[2618],{"type":45,"value":293},{"type":40,"tag":167,"props":2620,"children":2621},{"style":198},[2622],{"type":45,"value":2623}," runs-list",{"type":40,"tag":167,"props":2625,"children":2626},{"style":198},[2627],{"type":45,"value":2628}," --prompt-set-id",{"type":40,"tag":167,"props":2630,"children":2631},{"style":209},[2632],{"type":45,"value":401},{"type":40,"tag":167,"props":2634,"children":2635},{"style":198},[2636],{"type":45,"value":1155},{"type":40,"tag":167,"props":2638,"children":2639},{"style":409},[2640],{"type":45,"value":1160},{"type":40,"tag":167,"props":2642,"children":2643},{"style":209},[2644],{"type":45,"value":1165},{"type":40,"tag":167,"props":2646,"children":2647},{"style":198},[2648],{"type":45,"value":791},{"type":40,"tag":167,"props":2650,"children":2651},{"style":174},[2652],{"type":45,"value":2653},"   # runs for a set, newest first\n",{"type":40,"tag":167,"props":2655,"children":2656},{"class":169,"line":180},[2657,2661,2665,2670,2674,2678,2682,2686],{"type":40,"tag":167,"props":2658,"children":2659},{"style":286},[2660],{"type":45,"value":15},{"type":40,"tag":167,"props":2662,"children":2663},{"style":198},[2664],{"type":45,"value":293},{"type":40,"tag":167,"props":2666,"children":2667},{"style":198},[2668],{"type":45,"value":2669}," runs-cancel",{"type":40,"tag":167,"props":2671,"children":2672},{"style":209},[2673],{"type":45,"value":401},{"type":40,"tag":167,"props":2675,"children":2676},{"style":198},[2677],{"type":45,"value":1652},{"type":40,"tag":167,"props":2679,"children":2680},{"style":409},[2681],{"type":45,"value":1160},{"type":40,"tag":167,"props":2683,"children":2684},{"style":209},[2685],{"type":45,"value":1165},{"type":40,"tag":167,"props":2687,"children":2688},{"style":174},[2689],{"type":45,"value":2690},"                                 # cancel an in-flight run (also archives it)\n",{"type":40,"tag":167,"props":2692,"children":2693},{"class":169,"line":27},[2694,2698,2702,2707,2711,2715,2719,2723],{"type":40,"tag":167,"props":2695,"children":2696},{"style":286},[2697],{"type":45,"value":15},{"type":40,"tag":167,"props":2699,"children":2700},{"style":198},[2701],{"type":45,"value":293},{"type":40,"tag":167,"props":2703,"children":2704},{"style":198},[2705],{"type":45,"value":2706}," runs-archive",{"type":40,"tag":167,"props":2708,"children":2709},{"style":209},[2710],{"type":45,"value":401},{"type":40,"tag":167,"props":2712,"children":2713},{"style":198},[2714],{"type":45,"value":1652},{"type":40,"tag":167,"props":2716,"children":2717},{"style":409},[2718],{"type":45,"value":1160},{"type":40,"tag":167,"props":2720,"children":2721},{"style":209},[2722],{"type":45,"value":1165},{"type":40,"tag":167,"props":2724,"children":2725},{"style":174},[2726],{"type":45,"value":2727},"                                # archive a finished run\n",{"type":40,"tag":167,"props":2729,"children":2730},{"class":169,"line":245},[2731,2735,2739,2744,2748,2752,2756,2760],{"type":40,"tag":167,"props":2732,"children":2733},{"style":286},[2734],{"type":45,"value":15},{"type":40,"tag":167,"props":2736,"children":2737},{"style":198},[2738],{"type":45,"value":293},{"type":40,"tag":167,"props":2740,"children":2741},{"style":198},[2742],{"type":45,"value":2743}," runs-unarchive",{"type":40,"tag":167,"props":2745,"children":2746},{"style":209},[2747],{"type":45,"value":401},{"type":40,"tag":167,"props":2749,"children":2750},{"style":198},[2751],{"type":45,"value":1652},{"type":40,"tag":167,"props":2753,"children":2754},{"style":409},[2755],{"type":45,"value":1160},{"type":40,"tag":167,"props":2757,"children":2758},{"style":209},[2759],{"type":45,"value":1165},{"type":40,"tag":167,"props":2761,"children":2762},{"style":174},[2763],{"type":45,"value":2764},"                              # restore an archived run\n",{"type":40,"tag":167,"props":2766,"children":2767},{"class":169,"line":255},[2768],{"type":40,"tag":167,"props":2769,"children":2770},{"emptyLinePlaceholder":249},[2771],{"type":45,"value":252},{"type":40,"tag":167,"props":2773,"children":2774},{"class":169,"line":264},[2775,2779,2783,2788,2792,2796,2800,2804],{"type":40,"tag":167,"props":2776,"children":2777},{"style":286},[2778],{"type":45,"value":15},{"type":40,"tag":167,"props":2780,"children":2781},{"style":198},[2782],{"type":45,"value":293},{"type":40,"tag":167,"props":2784,"children":2785},{"style":198},[2786],{"type":45,"value":2787}," prompt-sets-archive",{"type":40,"tag":167,"props":2789,"children":2790},{"style":209},[2791],{"type":45,"value":401},{"type":40,"tag":167,"props":2793,"children":2794},{"style":198},[2795],{"type":45,"value":1155},{"type":40,"tag":167,"props":2797,"children":2798},{"style":409},[2799],{"type":45,"value":1160},{"type":40,"tag":167,"props":2801,"children":2802},{"style":209},[2803],{"type":45,"value":1165},{"type":40,"tag":167,"props":2805,"children":2806},{"style":174},[2807],{"type":45,"value":2808},"                   # archive a set (cancels its in-flight jobs)\n",{"type":40,"tag":167,"props":2810,"children":2811},{"class":169,"line":273},[2812,2816,2820,2825,2829,2833,2837,2841],{"type":40,"tag":167,"props":2813,"children":2814},{"style":286},[2815],{"type":45,"value":15},{"type":40,"tag":167,"props":2817,"children":2818},{"style":198},[2819],{"type":45,"value":293},{"type":40,"tag":167,"props":2821,"children":2822},{"style":198},[2823],{"type":45,"value":2824}," prompt-sets-unarchive",{"type":40,"tag":167,"props":2826,"children":2827},{"style":209},[2828],{"type":45,"value":401},{"type":40,"tag":167,"props":2830,"children":2831},{"style":198},[2832],{"type":45,"value":1155},{"type":40,"tag":167,"props":2834,"children":2835},{"style":409},[2836],{"type":45,"value":1160},{"type":40,"tag":167,"props":2838,"children":2839},{"style":209},[2840],{"type":45,"value":1165},{"type":40,"tag":167,"props":2842,"children":2843},{"style":174},[2844],{"type":45,"value":2845},"                 # restore an archived set\n",{"type":40,"tag":48,"props":2847,"children":2848},{},[2849,2851,2856,2858,2864,2866,2872,2874,2880],{"type":45,"value":2850},"Archiving is a soft delete — sets and runs are preserved and can be restored. Cancelling a run marks it ",{"type":40,"tag":124,"props":2852,"children":2854},{"className":2853},[],[2855],{"type":45,"value":1691},{"type":45,"value":2857}," and archives it; use ",{"type":40,"tag":124,"props":2859,"children":2861},{"className":2860},[],[2862],{"type":45,"value":2863},"runs-unarchive",{"type":45,"value":2865}," to surface it in the default list again. Use ",{"type":40,"tag":124,"props":2867,"children":2869},{"className":2868},[],[2870],{"type":45,"value":2871},"--archived true",{"type":45,"value":2873}," on the ",{"type":40,"tag":124,"props":2875,"children":2877},{"className":2876},[],[2878],{"type":45,"value":2879},"*-list",{"type":45,"value":2881}," commands to see archived items.",{"type":40,"tag":149,"props":2883,"children":2885},{"id":2884},"gotchas",[2886],{"type":45,"value":2887},"Gotchas",{"type":40,"tag":1327,"props":2889,"children":2890},{},[2891,2901,2911,2941],{"type":40,"tag":1331,"props":2892,"children":2893},{},[2894,2899],{"type":40,"tag":54,"props":2895,"children":2896},{},[2897],{"type":45,"value":2898},"Right profile, right instance",{"type":45,"value":2900}," — the eval runs against whatever model lives on the active profile's instance. Confirm the profile before creating sets or runs.",{"type":40,"tag":1331,"props":2902,"children":2903},{},[2904,2909],{"type":40,"tag":54,"props":2905,"children":2906},{},[2907],{"type":45,"value":2908},"Judge is binary and fixed",{"type":45,"value":2910}," — a pass means the answer avoided the high-impact analysis errors, not that it's the single best possible answer. The judge model isn't configurable per run.",{"type":40,"tag":1331,"props":2912,"children":2913},{},[2914,2919,2921,2926,2928,2933,2935,2940],{"type":40,"tag":54,"props":2915,"children":2916},{},[2917],{"type":45,"value":2918},"Failed job ≠ failed judgment",{"type":45,"value":2920}," — a result with ",{"type":40,"tag":124,"props":2922,"children":2924},{"className":2923},[],[2925],{"type":45,"value":1975},{"type":45,"value":2927}," set means the agentic job itself failed (it never produced an answer to score); treat that separately from a judge ",{"type":40,"tag":124,"props":2929,"children":2931},{"className":2930},[],[2932],{"type":45,"value":1946},{"type":45,"value":2934}," of ",{"type":40,"tag":124,"props":2936,"children":2938},{"className":2937},[],[2939],{"type":45,"value":2290},{"type":45,"value":95},{"type":40,"tag":1331,"props":2942,"children":2943},{},[2944,2949,2951,2957,2958,2964],{"type":40,"tag":54,"props":2945,"children":2946},{},[2947],{"type":45,"value":2948},"Snapshotting",{"type":45,"value":2950}," — to keep runs reproducible, capture the model state alongside results: ",{"type":40,"tag":124,"props":2952,"children":2954},{"className":2953},[],[2955],{"type":45,"value":2956},"omni models yaml-get \u003CmodelId> --compact",{"type":45,"value":67},{"type":40,"tag":124,"props":2959,"children":2961},{"className":2960},[],[2962],{"type":45,"value":2963},"omni models validate \u003CmodelId>",{"type":45,"value":2965},". Branch runs already pin the change to a branch.",{"type":40,"tag":149,"props":2967,"children":2969},{"id":2968},"docs-reference",[2970],{"type":45,"value":2971},"Docs reference",{"type":40,"tag":1327,"props":2973,"children":2974},{},[2975],{"type":40,"tag":1331,"props":2976,"children":2977},{},[2978,2984,2986,2993,2994,3001,3002],{"type":40,"tag":561,"props":2979,"children":2981},{"href":563,"rel":2980},[565],[2982],{"type":45,"value":2983},"Evals (concepts + judge)",{"type":45,"value":2985}," · ",{"type":40,"tag":561,"props":2987,"children":2990},{"href":2988,"rel":2989},"https:\u002F\u002Fdocs.omni.co\u002Fapi\u002Fai-eval\u002Flist-eval-prompt-sets",[565],[2991],{"type":45,"value":2992},"AI Eval API",{"type":45,"value":2985},{"type":40,"tag":561,"props":2995,"children":2998},{"href":2996,"rel":2997},"https:\u002F\u002Fdocs.omni.co\u002Fapi\u002Fai\u002Fcreate-ai-job",[565],[2999],{"type":45,"value":3000},"Create AI job API",{"type":45,"value":2985},{"type":40,"tag":561,"props":3003,"children":3006},{"href":3004,"rel":3005},"https:\u002F\u002Fdocs.omni.co\u002Fmodeling\u002Fdevelop\u002Fai-optimization",[565],[3007],{"type":45,"value":3008},"Optimizing models for AI",{"type":40,"tag":149,"props":3010,"children":3012},{"id":3011},"related-skills",[3013],{"type":45,"value":3014},"Related skills",{"type":40,"tag":1327,"props":3016,"children":3017},{},[3018,3042,3051],{"type":40,"tag":1331,"props":3019,"children":3020},{},[3021,3025,3027,3032,3034,3040],{"type":40,"tag":54,"props":3022,"children":3023},{},[3024],{"type":45,"value":129},{"type":45,"value":3026}," — improve AI accuracy based on eval findings (",{"type":40,"tag":124,"props":3028,"children":3030},{"className":3029},[],[3031],{"type":45,"value":2456},{"type":45,"value":3033},", ",{"type":40,"tag":124,"props":3035,"children":3037},{"className":3036},[],[3038],{"type":45,"value":3039},"sample_queries",{"type":45,"value":3041},", field metadata)",{"type":40,"tag":1331,"props":3043,"children":3044},{},[3045,3049],{"type":40,"tag":54,"props":3046,"children":3047},{},[3048],{"type":45,"value":137},{"type":45,"value":3050}," — apply context changes on a branch before A\u002FB testing",{"type":40,"tag":1331,"props":3052,"children":3053},{},[3054,3058],{"type":40,"tag":54,"props":3055,"children":3056},{},[3057],{"type":45,"value":145},{"type":45,"value":3059}," — discover topics and fields when writing prompts",{"type":40,"tag":3061,"props":3062,"children":3063},"style",{},[3064],{"type":45,"value":3065},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3067,"total":883},[3068,3080,3086,3100,3116,3130,3146],{"slug":3069,"name":3069,"fn":3070,"description":3071,"org":3072,"tags":3073,"stars":23,"repoUrl":24,"updatedAt":3079},"omni-admin","administer Omni Analytics","Administer an Omni Analytics instance — manage connections, users, groups, user attributes, permissions, schedules, and schema refreshes via the Omni CLI. Use this skill whenever someone wants to manage users or groups, set up permissions on a dashboard or folder, configure user attributes, create or modify schedules, manage database connections, refresh a schema, set up access controls, provision users, or any variant of \"add a user\", \"give access to\", \"set up permissions\", \"who has access\", \"configure connection\", \"refresh the schema\", or \"schedule a delivery\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3074,3075,3078],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},"CLI","cli",{"name":14,"slug":15,"type":16},"2026-04-06T18:10:58.235995",{"slug":4,"name":4,"fn":5,"description":6,"org":3081,"tags":3082,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3083,3084,3085],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":129,"name":129,"fn":3087,"description":3088,"org":3089,"tags":3090,"stars":23,"repoUrl":24,"updatedAt":3099},"optimize Omni models for AI","Optimize your Omni Analytics model for Blobby, the Omni Agent — configure ai_context, ai_fields, synonyms, sample_queries, and AI-specific topic extensions. Use this skill whenever someone wants to improve AI accuracy in Omni, make Blobby smarter, add AI context or example questions, curate which fields the AI sees, personalize AI context by user attribute, scope context to a model tier or agent, diagnose context-window pruning or truncation, control which topics AI can reach, troubleshoot why Blobby gives wrong answers, or any variant of \"make the AI better\", \"Blobby isn't answering correctly\", \"optimize for AI\", or \"teach the AI about our data\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3091,3094,3095,3098],{"name":3092,"slug":3093,"type":16},"AI Context","ai-context",{"name":21,"slug":22,"type":16},{"name":3096,"slug":3097,"type":16},"Data Modeling","data-modeling",{"name":14,"slug":15,"type":16},"2026-07-27T06:09:42.358967",{"slug":3101,"name":3101,"fn":3102,"description":3103,"org":3104,"tags":3105,"stars":23,"repoUrl":24,"updatedAt":3115},"omni-content-builder","manage Omni Analytics documents and dashboards","Create, update, and manage Omni Analytics documents and dashboards programmatically — document lifecycle, drafts, tiles, visualizations, filters, controls, and layouts — using the Omni CLI. Use this skill whenever someone wants to build a dashboard, create a workbook, add tiles or charts, configure dashboard filters or controls, update an existing dashboard's model, set up a KPI view, create visualizations, lay out a dashboard, arrange tiles or pages, create a document, edit a dashboard as a draft, publish a draft, change dashboard settings, rename a workbook, delete a dashboard, move a document to a folder, duplicate a dashboard, or any variant of \"build a dashboard for\", \"create a report showing\", \"add a chart to\", \"make a dashboard\", \"update the dashboard layout\", \"rename this document\", \"publish this draft\", \"move to folder\", or \"delete this dashboard\". Also use when modifying dashboard-level model customizations like workbook-specific joins or fields.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3106,3107,3108,3111,3114],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3109,"slug":3110,"type":16},"Dashboards","dashboards",{"name":3112,"slug":3113,"type":16},"Data Visualization","data-visualization",{"name":14,"slug":15,"type":16},"2026-04-06T18:11:02.007785",{"slug":3117,"name":3117,"fn":3118,"description":3119,"org":3120,"tags":3121,"stars":23,"repoUrl":24,"updatedAt":3129},"omni-content-explorer","find and organize Omni Analytics content","Find, browse, and organize content in Omni Analytics — dashboards, workbooks, folders, and labels — using the Omni CLI. Use this skill whenever someone wants to find an existing dashboard, search for content, list workbooks, browse folders, see what dashboards exist, find popular reports, download a dashboard as PDF or PNG, favorite content, manage labels on documents, or any variant of \"find the dashboard about\", \"what reports do we have\", \"show me our dashboards\", \"where is the sales report\", or \"download this dashboard\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3122,3123,3124,3125,3126],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3109,"slug":3110,"type":16},{"name":14,"slug":15,"type":16},{"name":3127,"slug":3128,"type":16},"Search","search","2026-04-06T18:11:04.516766",{"slug":3131,"name":3131,"fn":3132,"description":3133,"org":3134,"tags":3135,"stars":23,"repoUrl":24,"updatedAt":3145},"omni-embed","embed Omni Analytics dashboards","Embed Omni Analytics dashboards in external applications — URL signing, custom themes, iframe events, entity workspaces, and permission-aware content — using the @omni-co\u002Fembed SDK and Omni CLI. Use this skill whenever someone wants to embed a dashboard, sign an embed URL, customize the embedded theme, handle embed events, listen for clicks or drills in the iframe, send filters to an embedded dashboard, set up entity workspaces, look up embed users, build a permission-aware content list, white-label an embedded dashboard, or any variant of \"embed this dashboard\", \"customize the iframe theme\", \"handle click events from the embed\", \"filter the embedded dashboard\", \"set up embedding\", or \"what dashboards can this user see\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3136,3137,3138,3141,3142],{"name":21,"slug":22,"type":16},{"name":3109,"slug":3110,"type":16},{"name":3139,"slug":3140,"type":16},"Frontend","frontend",{"name":14,"slug":15,"type":16},{"name":3143,"slug":3144,"type":16},"SDK","sdk","2026-04-06T18:10:55.739544",{"slug":137,"name":137,"fn":3147,"description":3148,"org":3149,"tags":3150,"stars":23,"repoUrl":24,"updatedAt":3158},"build Omni Analytics semantic models","Create and edit Omni Analytics semantic model definitions — views, topics, dimensions, measures, relationships, and query views — using YAML through the Omni CLI. Use this skill whenever someone wants to add a field, create a new dimension or measure, define a topic, set up joins between tables, modify the data model, build a new view, add a calculated field, create a relationship, edit YAML, work on a branch, promote model changes, or any variant of \"model this data\", \"add this metric\", \"create a view for\", or \"set up a join between\". Also use for migrating modeling patterns since Omni's YAML is conceptually similar to other semantic layer definitions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3151,3152,3153,3154,3155],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3096,"slug":3097,"type":16},{"name":14,"slug":15,"type":16},{"name":3156,"slug":3157,"type":16},"YAML","yaml","2026-04-06T18:11:05.775943",{"items":3160,"total":883},[3161,3167,3173,3180,3188,3196,3204,3212,3227,3241,3256],{"slug":3069,"name":3069,"fn":3070,"description":3071,"org":3162,"tags":3163,"stars":23,"repoUrl":24,"updatedAt":3079},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3164,3165,3166],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":14,"slug":15,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":3168,"tags":3169,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3170,3171,3172],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":129,"name":129,"fn":3087,"description":3088,"org":3174,"tags":3175,"stars":23,"repoUrl":24,"updatedAt":3099},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3176,3177,3178,3179],{"name":3092,"slug":3093,"type":16},{"name":21,"slug":22,"type":16},{"name":3096,"slug":3097,"type":16},{"name":14,"slug":15,"type":16},{"slug":3101,"name":3101,"fn":3102,"description":3103,"org":3181,"tags":3182,"stars":23,"repoUrl":24,"updatedAt":3115},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3183,3184,3185,3186,3187],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3109,"slug":3110,"type":16},{"name":3112,"slug":3113,"type":16},{"name":14,"slug":15,"type":16},{"slug":3117,"name":3117,"fn":3118,"description":3119,"org":3189,"tags":3190,"stars":23,"repoUrl":24,"updatedAt":3129},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3191,3192,3193,3194,3195],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3109,"slug":3110,"type":16},{"name":14,"slug":15,"type":16},{"name":3127,"slug":3128,"type":16},{"slug":3131,"name":3131,"fn":3132,"description":3133,"org":3197,"tags":3198,"stars":23,"repoUrl":24,"updatedAt":3145},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3199,3200,3201,3202,3203],{"name":21,"slug":22,"type":16},{"name":3109,"slug":3110,"type":16},{"name":3139,"slug":3140,"type":16},{"name":14,"slug":15,"type":16},{"name":3143,"slug":3144,"type":16},{"slug":137,"name":137,"fn":3147,"description":3148,"org":3205,"tags":3206,"stars":23,"repoUrl":24,"updatedAt":3158},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3207,3208,3209,3210,3211],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3096,"slug":3097,"type":16},{"name":14,"slug":15,"type":16},{"name":3156,"slug":3157,"type":16},{"slug":145,"name":145,"fn":3213,"description":3214,"org":3215,"tags":3216,"stars":23,"repoUrl":24,"updatedAt":3226},"discover and inspect Omni Analytics models","Discover and inspect Omni Analytics models, topics, views, fields, dimensions, measures, and relationships using the Omni CLI. Use this skill whenever someone wants to understand what data is available in Omni, explore their semantic model, find specific fields or views, check how tables join together, see what topics exist, or asks any variant of \"what can I query\", \"what fields are available\", \"show me the model\", \"what data do we have\", or \"how is this data modeled\". Also use when you need to understand the Omni model structure before building or modifying anything.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3217,3218,3219,3222,3225],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3220,"slug":3221,"type":16},"Data Analysis","data-analysis",{"name":3223,"slug":3224,"type":16},"Knowledge Management","knowledge-management",{"name":14,"slug":15,"type":16},"2026-04-06T18:11:00.739003",{"slug":3228,"name":3228,"fn":3229,"description":3230,"org":3231,"tags":3232,"stars":23,"repoUrl":24,"updatedAt":3240},"omni-query","query Omni Analytics semantic layer","Run queries against Omni Analytics' semantic layer using the Omni CLI, interpret results, and chain queries for multi-step analysis. Use this skill whenever someone wants to query data through Omni, run a report, get metrics, pull numbers, analyze data, ask \"how many\" \u002F \"what's the trend\" \u002F \"show me the data\", retrieve dashboard query results, or extract data from an existing dashboard or workbook. Also use for table calculations and computed columns (running totals, percent-of-total, month-over-month \u002F period-over-period change, moving averages, rankings), open-ended multi-step analysis via agentic AI jobs, and running raw SQL through the semantic layer — even when the user doesn't say \"query\" (e.g. \"add a running total column\", \"what's our MoM growth\", \"analyze revenue trends\"). For building or editing a dashboard or chart use omni-content-builder; for adding a field or measure to the model use omni-model-builder — this skill retrieves and computes over data.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3233,3234,3235,3236,3237],{"name":21,"slug":22,"type":16},{"name":3076,"slug":3077,"type":16},{"name":3220,"slug":3221,"type":16},{"name":14,"slug":15,"type":16},{"name":3238,"slug":3239,"type":16},"SQL","sql","2026-04-06T18:10:56.974412",{"slug":3242,"name":3242,"fn":3243,"description":3244,"org":3245,"tags":3246,"stars":23,"repoUrl":24,"updatedAt":3255},"omni-to-databricks-metric-view","convert Omni topics to Databricks Metric Views","Convert an Omni Analytics topic into a Databricks Metric View definition in Unity Catalog. Use this skill whenever someone wants to export Omni metrics to Databricks, create a Metric View from an Omni topic, harden BI metrics into Unity Catalog, or bridge Omni's semantic layer with Databricks AI\u002FBI dashboards and Genie spaces.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3247,3248,3251,3254],{"name":21,"slug":22,"type":16},{"name":3249,"slug":3250,"type":16},"Data Engineering","data-engineering",{"name":3252,"slug":3253,"type":16},"Databricks","databricks",{"name":14,"slug":15,"type":16},"2026-04-09T04:47:50.695014",{"slug":3257,"name":3257,"fn":3258,"description":3259,"org":3260,"tags":3261,"stars":23,"repoUrl":24,"updatedAt":3269},"omni-to-snowflake-semantic-view","convert Omni topics to Snowflake views","Convert an Omni Analytics topic into a Snowflake Semantic View YAML definition. Use this skill whenever someone wants to export Omni metrics to Snowflake, create a Semantic View from an Omni topic, harden BI metrics into the warehouse, or bridge Omni's semantic layer with Snowflake Cortex Analyst.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3262,3263,3264,3265,3268],{"name":21,"slug":22,"type":16},{"name":3249,"slug":3250,"type":16},{"name":14,"slug":15,"type":16},{"name":3266,"slug":3267,"type":16},"Snowflake","snowflake",{"name":3156,"slug":3157,"type":16},"2026-04-06T18:11:25.009706"]