[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-posthog-creating-online-evaluations":3,"mdc-qf4pp0-key":51,"related-repo-posthog-creating-online-evaluations":2905,"related-org-posthog-creating-online-evaluations":3006},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":46,"sourceUrl":49,"mdContent":50},"creating-online-evaluations","author online evaluations in PostHog","Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure mode you've identified. Use when the user wants an evaluation that automatically scores new generations or whole traces going forward β€” \"create an eval to catch X\", \"continuously check that responses do Y\", \"turn this failure into an eval\". Covers choosing the target and eval type (hog \u002F llm_judge \u002F sentiment), configuring a provider, model, and usable provider key for an llm_judge eval, scoping which generations trigger it via conditions (property filters + rollout sampling), creating it disabled, verifying scope, and enabling. Finding and ranking the failure modes worth evaluating is its own job β€” use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"posthog","PostHog","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fposthog.png",[12,16,17,20,23],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"LLM","llm",{"name":21,"slug":22,"type":15},"Evals","evals",{"name":24,"slug":25,"type":15},"Analytics","analytics",35568,"https:\u002F\u002Fgithub.com\u002FPostHog\u002Fposthog","2026-07-28T05:34:14.131102",null,2977,[32,33,25,34,35,36,37,38,39,40,41,42,43,44,45],"ab-testing","ai-analytics","cdp","data-warehouse","experiments","feature-flags","javascript","product-analytics","python","react","session-replay","surveys","typescript","web-analytics",{"repoUrl":27,"stars":26,"forks":30,"topics":47,"description":48},[32,33,25,34,35,36,37,38,39,40,41,42,43,44,45],"πŸ¦” PostHog is an all-in-one developer platform for building successful products. We offer product analytics, web analytics, session replay, error tracking, feature flags, experimentation, surveys, data warehouse, a CDP, and an AI product assistant to help debug your code, ship features faster, and keep all your usage and customer data in one stack.","https:\u002F\u002Fgithub.com\u002FPostHog\u002Fposthog\u002Ftree\u002FHEAD\u002Fproducts\u002Fai_observability\u002Fskills\u002Fcreating-online-evaluations","---\nname: creating-online-evaluations\ndescription: >\n Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure\n mode you've identified. Use when the user wants an evaluation that automatically scores new generations\n or whole traces going forward β€” \"create an eval to catch X\", \"continuously check that responses do Y\",\n \"turn this failure into an eval\". Covers choosing the target and eval type (hog \u002F llm_judge \u002F sentiment),\n configuring a provider, model, and usable provider key for an llm_judge eval, scoping which generations\n trigger it via conditions (property filters + rollout sampling), creating it disabled, verifying scope,\n and enabling.\n Finding and ranking the failure modes worth evaluating is its own job β€” use exploring-ai-failures first.\n To debug or manage evaluations that already exist, use exploring-llm-evaluations.\n---\n\n# Creating online evaluations\n\nAn **online evaluation** automatically scores either each matching `$ai_generation` or the whole trace\ncontaining it, until disabled. A good eval comes from a real failure mode you've found in production traffic,\nnot from a guess or a generic metric like \"hallucination\" or \"helpfulness\". This skill starts once that\nfailure mode is identified and turns it into a scoped, continuously-running eval.\n\n**First, know what you're evaluating.** Finding and ranking the failure modes worth catching is a\nseparate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague\nabout it and don't refer to a specific failure mode, run `exploring-ai-failures` to scope a use case,\nfind failing traces, and produce a ranked list of failure modes.\n\nFor the mechanics of _writing and iterating_ an evaluator (Hog source vs LLM-judge prompt, dry-running,\ndebugging a live eval), defer to `exploring-llm-evaluations`.\n\n## Tools\n\n| Tool | Purpose |\n| -------------------------------------- | ------------------------------------------------------------- |\n| `posthog:llma-evaluation-config-get` | Check the active provider key used by unpinned judges |\n| `posthog:llma-provider-key-list` | Find a usable (`ok` state) provider key to pin |\n| `posthog:llma-evaluation-judge-models` | List valid provider+model combos |\n| `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations before creating |\n| `posthog:llma-evaluation-create` | Create the evaluation (always `enabled: false` first) |\n| `posthog:llma-evaluation-run` | Spot-run a draft eval against one generation |\n| `posthog:llma-evaluation-update` | Iterate config, then flip `enabled: true` |\n| `posthog:execute-sql` | Verify a condition matches the events and volume you expect |\n| `posthog:generate-app-url` | Build a region- and project-qualified deep link to the eval |\n\nThe full create payload (every field, the config schemas, the exact `conditions` shape) is in\n[references\u002Fevaluation-payload.md](references\u002Fevaluation-payload.md).\n\n## Phase 1 β€” Pick the failure mode to evaluate\n\nStart from a real, observed failure, not a metric you picked in advance. If you don't already have one,\nrun `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure\nmodes β€” then come back. With that list in hand, talk with the user to choose what to turn into an eval:\n\n- **Most frequent, most painful first.** A handful of modes usually cover the majority of failures.\n- **Pair obvious fixes with the eval, don't skip it.** If a prompt tweak would likely fix the failure, set\n up the eval anyway and suggest the fix alongside it β€” a rising pass rate is how you confirm the fix landed.\n- **One mode per eval.** Three failure modes is three evals, not one prompt trying to catch everything.\n\nYou should end with a single, crisp, checkable criterion β€” \"the reply must stay on the user's topic\", \"the\ntool call must include an `order_id`\". Then move to Phase 2.\n\n## Phase 2 β€” Build the online eval\n\n### 2.1 β€” Choose the eval type\n\n| Use… | When the criterion is… |\n| ----------- | ------------------------------------------------------------------------------------------------------------------------------------- |\n| `hog` | Structural \u002F rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, **no provider key needed.** |\n| `llm_judge` | Subjective \u002F fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider, model, and usable provider key. |\n| `sentiment` | You want sentiment labels on user messages, not a pass\u002Ffail (unless very specifically asked for, usually not relevant to this skill). |\n\nReach for `hog` first, escalate to `llm_judge` if there is no deterministic way to check for what we want to check.\n\n### 2.2 β€” Choose the target\n\n| Target | Behavior |\n| ------------ | ------------------------------------------------------------------------------------------------------------------ |\n| `generation` | Runs once for each matching `$ai_generation`, immediately after ingestion. This is the default. |\n| `trace` | Runs once for the whole trace after the first matching generation and a configurable wait for the trace to finish. |\n\nFor a trace target, send `\"target\": \"trace\"` plus a settle config that controls when the trace is\nevaluated, discriminated on `strategy`:\n\n- `{ \"strategy\": \"fixed_window\", \"window_seconds\": 1800 }` β€” evaluate a fixed wait after the first\n matching generation. Between 10 seconds and 2 hours, defaults to 30 minutes. A `target_config`\n without a `strategy` key means this.\n- `{ \"strategy\": \"inactivity\", \"quiet_period_seconds\": 300, \"max_age_seconds\": 7200 }` β€” evaluate once\n the trace has had no new activity for the quiet period (10 seconds to 30 minutes,\n defaults to 5 minutes). `max_age_seconds` caps the total wait from the first matching generation\n (1 minute to 2 hours, defaults to 2 hours, must be at least the quiet period).\n\nConditions still match the generation that triggers the run; the evaluator itself receives\nthe complete trace. Sentiment evaluations support only the generation target.\n\nNew Hog source should use the globals shared by both targets:\n\n| Global | Meaning |\n| -------------------------------------- | ------------------------------------------------------------------------------------------------- |\n| `evaluation_events` | One generation event for a generation target, or every captured event for a trace target. |\n| `target` | The target's `type`, `id`, `total_cost_usd`, and `total_latency_seconds`. |\n| `item.input_text` \u002F `item.output_text` | Best-effort readable projections; use these for length, keyword, and regex checks. |\n| `item.input` \u002F `item.output` | Original serialized values; use these when the evaluator needs to parse the captured JSON itself. |\n\nGeneration evaluations still expose top-level `input`, `output`, `properties`, and `event`. Trace evaluations\nstill expose their original `events` and `trace` globals. Those globals are kept for compatibility with saved\nevaluators. Do not use target-specific globals in new source that needs to work for both targets. The text\nprojections recognize common provider payloads but are not authoritative; use `item.input` \u002F `item.output` when\nexact structure matters.\n\n### 2.3 β€” Configure the LLM judge\n\nAn `llm_judge` evaluation requires a valid `provider` and `model`. It also needs a usable provider key\nwhen it runs. `provider_key_id` controls whether the evaluation pins one specific key:\n\n- Set `provider_key_id` to the UUID of an `ok`-state key for the same provider to pin it.\n- Set `provider_key_id` to `null` to use the team's active provider key. The active key must be in the\n `ok` state and use the same provider as `model_configuration.provider`.\n\nHog and sentiment evaluations skip this step.\n\n```json\nposthog:llma-evaluation-config-get \u002F\u002F check active_provider_key for an unpinned judge\nposthog:llma-provider-key-list \u002F\u002F find an ok-state key to pin\nposthog:llma-evaluation-judge-models \u002F\u002F { \"provider\": \"openai\" } β†’ valid models\n```\n\nConfirm the provider and model with `llma-evaluation-judge-models`. Prefer pinning the chosen key so a later\nteam-wide active-key change does not change how the evaluation runs. Leave `provider_key_id` as `null` only\nafter `llma-evaluation-config-get` confirms the active key is usable and its provider matches.\n\nIf there is no usable key, you may still create a disabled draft for the user to review. Do not spot-run or\nenable it. Ask the user to add or validate a key in the UI before continuing.\n\n### 2.4 β€” Create it disabled\n\nCreate with `enabled: false` so nothing fires until the scope is verified. Minimal `hog` example:\n\n```json\nposthog:llma-evaluation-create\n{\n \"name\": \"Output is not empty\",\n \"description\": \"Fails when a generation has no readable output\",\n \"evaluation_type\": \"hog\",\n \"evaluation_config\": { \"source\": \"let count := 0\\nfor (let i, item in evaluation_events) {\\n if (item.event == '$ai_generation') {\\n count := count + 1\\n if (length(trim(item.output_text)) == 0) { return false }\\n }\\n}\\nreturn count > 0\" },\n \"output_type\": \"boolean\",\n \"output_config\": { \"allows_na\": false },\n \"target\": \"generation\",\n \"target_config\": {},\n \"conditions\": [\n { \"id\": \"default\", \"rollout_percentage\": 100, \"properties\": [{ \"key\": \"$ai_model\", \"type\": \"event\", \"operator\": \"icontains\", \"value\": \"gpt\" }] }\n ],\n \"enabled\": false\n}\n```\n\nFor `llm_judge`, swap `evaluation_config` to `{ \"prompt\": \"…\" }` and add\n`\"model_configuration\": { \"provider\": \"openai\", \"model\": \"gpt-5-mini\", \"provider_key_id\": \"\u003Cuuid of an ok-state key from llma-provider-key-list>\" }`.\nUse `null` only when the active team key is `ok` and uses the same provider. Full field reference:\n[references\u002Fevaluation-payload.md](references\u002Fevaluation-payload.md).\n\n### 2.5 β€” Verify the scope before enabling\n\n`conditions` is where online evals go wrong: too broad and you evaluate (and bill) a firehose; too narrow\nand it never fires. Confirm the filter matches the events you expect, and roughly how many per day:\n\n```sql\nposthog:execute-sql\nSELECT count() AS matched, count() \u002F 7 AS per_day\nFROM events\nWHERE event = '$ai_generation'\n AND properties.$ai_model ILIKE '%gpt%' -- mirror each condition property\n AND timestamp >= now() - INTERVAL 7 DAY\n```\n\nFor generation targets, `count()` is the run volume. For trace targets, count distinct non-empty\n`$ai_trace_id` values because matching generations from the same trace schedule only one run.\n\nIf volume is high, set `rollout_percentage` below 100 to sample. Spot-check the evaluator with\n`llma-evaluation-test-hog` (hog) or `llma-evaluation-run` against one generation (llm_judge).\nBoth tools currently use generation samples; for a trace target they can check shared source or prompt behavior,\nbut they do not reproduce the complete settled trace. Review the first live trace results before increasing rollout.\n\n> **Watch out:** some orgs reuse a single `$ai_trace_id` across 100k+ events. Scoping by trace-ID prefix\n> can match far more than expected β€” verify volume with the SQL above before enabling.\n\n### 2.6 β€” Enable, then close the loop\n\n```json\nposthog:llma-evaluation-update\n{ \"evaluationId\": \"\u003Cuuid>\", \"enabled\": true }\n```\n\nIt now runs on every new matching generation, or once per matching trace for a trace target. This isn't\none-and-done: the user should be aware that they need to keep an eye on results and iterate if the outcome\nis not the expected one. To wire results into a Slack feed, see `feature-usage-feed`.\n\n## Scoping with conditions\n\n`conditions` is a **list** of condition sets β€” **OR between sets, AND within a set's `properties`**. Each\nset is `{ id, rollout_percentage, properties[] }`. There is no time window inside conditions; sampling is\nonly `rollout_percentage` (0–100). Property filters use the standard PostHog shape\n(`key`, `type`, `operator`, `value`). For trace targets, these filters still select the generation that\ntriggers the eventual whole-trace evaluation.\n\n```json\n\"conditions\": [\n { \"id\": \"openai\", \"rollout_percentage\": 100, \"properties\": [{\"key\": \"$ai_provider\", \"type\": \"event\", \"operator\": \"exact\", \"value\": \"openai\"}] },\n { \"id\": \"anthropic\", \"rollout_percentage\": 25, \"properties\": [{\"key\": \"$ai_provider\", \"type\": \"event\", \"operator\": \"exact\", \"value\": \"anthropic\"}] }\n]\n```\n\n## Constructing UI links\n\nBuild links with `posthog:generate-app-url` β€” never hand-write the host or the `\u002Fproject\u002F\u003Cid>\u002F` prefix.\nThe `url` must be a canonical catalog template; pass concrete ids via `params`, never inline them into the path.\n\n- **Evaluations list:** `generate-app-url {url: \"\u002Fai-evals\u002Fevaluations\"}`\n- **Single evaluation:** `generate-app-url {url: \"\u002Fai-evals\u002Fevaluations\u002F{id}\", params: {id: \"\u003Cevaluation_id>\"}}`\n\nThese resolve to the correct region host and project prefix (e.g.\n`https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cid>\u002Fai-evals\u002Fevaluations\u002F\u003Cevaluation_id>`). Surface the link after\ncreating so the user can review and toggle it in the UI.\n\n## Tips\n\n- **Evals come from real failures, not generic metrics.** Start from a failure found in this product's\n traffic (via `exploring-ai-failures`), not from \"let's measure hallucination\". A metric nobody traced\n back to a real bad output is noise.\n- **One eval, one failure mode.** Different failure modes need different evals; don't make one eval try to\n catch everything.\n- **Suggest changes along with the eval if possible.** If it's clear a prompt change would fix the issue, for\n instance, set up the eval but also suggest to the user they change the prompt: they should soon see the eval\n go from low pass rate to a higher pass rate.\n- **`hog` first.** No provider key, no AI approval, deterministic. Reach for `llm_judge` only when the\n criterion genuinely can't be coded.\n- **Always create disabled, verify scope, then enable.** An eval firing on the wrong events is worse than\n none β€” noise, and (for llm_judge) cost.\n- **Configure llm_judge credentials before running.** A judge needs a valid provider and model plus a usable\n provider key. `provider_key_id` may be `null` only when the matching active team key can be used.\n- **`bytecode` is server-written** for hog evals β€” never pass it; send only `evaluation_config.source`.\n- For cluster-scoped evals, identify the cluster with `exploring-llm-clusters`, then translate its event\n filter into `conditions`.\n",{"data":52,"body":53},{"name":4,"description":6},{"type":54,"children":55},"root",[56,64,87,105,126,133,336,356,362,374,409,422,428,435,513,532,538,601,622,670,675,680,815,877,883,917,972,977,1034,1069,1074,1080,1099,1817,1870,1876,1886,1943,1964,1992,2013,2019,2101,2113,2119,2188,2666,2672,2708,2742,2755,2761,2899],{"type":57,"tag":58,"props":59,"children":60},"element","h1",{"id":4},[61],{"type":62,"value":63},"text","Creating online evaluations",{"type":57,"tag":65,"props":66,"children":67},"p",{},[68,70,76,78,85],{"type":62,"value":69},"An ",{"type":57,"tag":71,"props":72,"children":73},"strong",{},[74],{"type":62,"value":75},"online evaluation",{"type":62,"value":77}," automatically scores either each matching ",{"type":57,"tag":79,"props":80,"children":82},"code",{"className":81},[],[83],{"type":62,"value":84},"$ai_generation",{"type":62,"value":86}," or the whole trace\ncontaining it, until disabled. A good eval comes from a real failure mode you've found in production traffic,\nnot from a guess or a generic metric like \"hallucination\" or \"helpfulness\". This skill starts once that\nfailure mode is identified and turns it into a scoped, continuously-running eval.",{"type":57,"tag":65,"props":88,"children":89},{},[90,95,97,103],{"type":57,"tag":71,"props":91,"children":92},{},[93],{"type":62,"value":94},"First, know what you're evaluating.",{"type":62,"value":96}," Finding and ranking the failure modes worth catching is a\nseparate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague\nabout it and don't refer to a specific failure mode, run ",{"type":57,"tag":79,"props":98,"children":100},{"className":99},[],[101],{"type":62,"value":102},"exploring-ai-failures",{"type":62,"value":104}," to scope a use case,\nfind failing traces, and produce a ranked list of failure modes.",{"type":57,"tag":65,"props":106,"children":107},{},[108,110,116,118,124],{"type":62,"value":109},"For the mechanics of ",{"type":57,"tag":111,"props":112,"children":113},"em",{},[114],{"type":62,"value":115},"writing and iterating",{"type":62,"value":117}," an evaluator (Hog source vs LLM-judge prompt, dry-running,\ndebugging a live eval), defer to ",{"type":57,"tag":79,"props":119,"children":121},{"className":120},[],[122],{"type":62,"value":123},"exploring-llm-evaluations",{"type":62,"value":125},".",{"type":57,"tag":127,"props":128,"children":130},"h2",{"id":129},"tools",[131],{"type":62,"value":132},"Tools",{"type":57,"tag":134,"props":135,"children":136},"table",{},[137,156],{"type":57,"tag":138,"props":139,"children":140},"thead",{},[141],{"type":57,"tag":142,"props":143,"children":144},"tr",{},[145,151],{"type":57,"tag":146,"props":147,"children":148},"th",{},[149],{"type":62,"value":150},"Tool",{"type":57,"tag":146,"props":152,"children":153},{},[154],{"type":62,"value":155},"Purpose",{"type":57,"tag":157,"props":158,"children":159},"tbody",{},[160,178,203,220,237,262,279,302,319],{"type":57,"tag":142,"props":161,"children":162},{},[163,173],{"type":57,"tag":164,"props":165,"children":166},"td",{},[167],{"type":57,"tag":79,"props":168,"children":170},{"className":169},[],[171],{"type":62,"value":172},"posthog:llma-evaluation-config-get",{"type":57,"tag":164,"props":174,"children":175},{},[176],{"type":62,"value":177},"Check the active provider key used by unpinned judges",{"type":57,"tag":142,"props":179,"children":180},{},[181,190],{"type":57,"tag":164,"props":182,"children":183},{},[184],{"type":57,"tag":79,"props":185,"children":187},{"className":186},[],[188],{"type":62,"value":189},"posthog:llma-provider-key-list",{"type":57,"tag":164,"props":191,"children":192},{},[193,195,201],{"type":62,"value":194},"Find a usable (",{"type":57,"tag":79,"props":196,"children":198},{"className":197},[],[199],{"type":62,"value":200},"ok",{"type":62,"value":202}," state) provider key to pin",{"type":57,"tag":142,"props":204,"children":205},{},[206,215],{"type":57,"tag":164,"props":207,"children":208},{},[209],{"type":57,"tag":79,"props":210,"children":212},{"className":211},[],[213],{"type":62,"value":214},"posthog:llma-evaluation-judge-models",{"type":57,"tag":164,"props":216,"children":217},{},[218],{"type":62,"value":219},"List valid provider+model combos",{"type":57,"tag":142,"props":221,"children":222},{},[223,232],{"type":57,"tag":164,"props":224,"children":225},{},[226],{"type":57,"tag":79,"props":227,"children":229},{"className":228},[],[230],{"type":62,"value":231},"posthog:llma-evaluation-test-hog",{"type":57,"tag":164,"props":233,"children":234},{},[235],{"type":62,"value":236},"Dry-run Hog source against recent generations before creating",{"type":57,"tag":142,"props":238,"children":239},{},[240,249],{"type":57,"tag":164,"props":241,"children":242},{},[243],{"type":57,"tag":79,"props":244,"children":246},{"className":245},[],[247],{"type":62,"value":248},"posthog:llma-evaluation-create",{"type":57,"tag":164,"props":250,"children":251},{},[252,254,260],{"type":62,"value":253},"Create the evaluation (always ",{"type":57,"tag":79,"props":255,"children":257},{"className":256},[],[258],{"type":62,"value":259},"enabled: false",{"type":62,"value":261}," first)",{"type":57,"tag":142,"props":263,"children":264},{},[265,274],{"type":57,"tag":164,"props":266,"children":267},{},[268],{"type":57,"tag":79,"props":269,"children":271},{"className":270},[],[272],{"type":62,"value":273},"posthog:llma-evaluation-run",{"type":57,"tag":164,"props":275,"children":276},{},[277],{"type":62,"value":278},"Spot-run a draft eval against one generation",{"type":57,"tag":142,"props":280,"children":281},{},[282,291],{"type":57,"tag":164,"props":283,"children":284},{},[285],{"type":57,"tag":79,"props":286,"children":288},{"className":287},[],[289],{"type":62,"value":290},"posthog:llma-evaluation-update",{"type":57,"tag":164,"props":292,"children":293},{},[294,296],{"type":62,"value":295},"Iterate config, then flip ",{"type":57,"tag":79,"props":297,"children":299},{"className":298},[],[300],{"type":62,"value":301},"enabled: true",{"type":57,"tag":142,"props":303,"children":304},{},[305,314],{"type":57,"tag":164,"props":306,"children":307},{},[308],{"type":57,"tag":79,"props":309,"children":311},{"className":310},[],[312],{"type":62,"value":313},"posthog:execute-sql",{"type":57,"tag":164,"props":315,"children":316},{},[317],{"type":62,"value":318},"Verify a condition matches the events and volume you expect",{"type":57,"tag":142,"props":320,"children":321},{},[322,331],{"type":57,"tag":164,"props":323,"children":324},{},[325],{"type":57,"tag":79,"props":326,"children":328},{"className":327},[],[329],{"type":62,"value":330},"posthog:generate-app-url",{"type":57,"tag":164,"props":332,"children":333},{},[334],{"type":62,"value":335},"Build a region- and project-qualified deep link to the eval",{"type":57,"tag":65,"props":337,"children":338},{},[339,341,347,349,355],{"type":62,"value":340},"The full create payload (every field, the config schemas, the exact ",{"type":57,"tag":79,"props":342,"children":344},{"className":343},[],[345],{"type":62,"value":346},"conditions",{"type":62,"value":348}," shape) is in\n",{"type":57,"tag":350,"props":351,"children":353},"a",{"href":352},"references\u002Fevaluation-payload.md",[354],{"type":62,"value":352},{"type":62,"value":125},{"type":57,"tag":127,"props":357,"children":359},{"id":358},"phase-1-pick-the-failure-mode-to-evaluate",[360],{"type":62,"value":361},"Phase 1 β€” Pick the failure mode to evaluate",{"type":57,"tag":65,"props":363,"children":364},{},[365,367,372],{"type":62,"value":366},"Start from a real, observed failure, not a metric you picked in advance. If you don't already have one,\nrun ",{"type":57,"tag":79,"props":368,"children":370},{"className":369},[],[371],{"type":62,"value":102},{"type":62,"value":373}," to scope a use case, find failing traces, and produce a ranked list of failure\nmodes β€” then come back. With that list in hand, talk with the user to choose what to turn into an eval:",{"type":57,"tag":375,"props":376,"children":377},"ul",{},[378,389,399],{"type":57,"tag":379,"props":380,"children":381},"li",{},[382,387],{"type":57,"tag":71,"props":383,"children":384},{},[385],{"type":62,"value":386},"Most frequent, most painful first.",{"type":62,"value":388}," A handful of modes usually cover the majority of failures.",{"type":57,"tag":379,"props":390,"children":391},{},[392,397],{"type":57,"tag":71,"props":393,"children":394},{},[395],{"type":62,"value":396},"Pair obvious fixes with the eval, don't skip it.",{"type":62,"value":398}," If a prompt tweak would likely fix the failure, set\nup the eval anyway and suggest the fix alongside it β€” a rising pass rate is how you confirm the fix landed.",{"type":57,"tag":379,"props":400,"children":401},{},[402,407],{"type":57,"tag":71,"props":403,"children":404},{},[405],{"type":62,"value":406},"One mode per eval.",{"type":62,"value":408}," Three failure modes is three evals, not one prompt trying to catch everything.",{"type":57,"tag":65,"props":410,"children":411},{},[412,414,420],{"type":62,"value":413},"You should end with a single, crisp, checkable criterion β€” \"the reply must stay on the user's topic\", \"the\ntool call must include an ",{"type":57,"tag":79,"props":415,"children":417},{"className":416},[],[418],{"type":62,"value":419},"order_id",{"type":62,"value":421},"\". Then move to Phase 2.",{"type":57,"tag":127,"props":423,"children":425},{"id":424},"phase-2-build-the-online-eval",[426],{"type":62,"value":427},"Phase 2 β€” Build the online eval",{"type":57,"tag":429,"props":430,"children":432},"h3",{"id":431},"_21-choose-the-eval-type",[433],{"type":62,"value":434},"2.1 β€” Choose the eval type",{"type":57,"tag":134,"props":436,"children":437},{},[438,454],{"type":57,"tag":138,"props":439,"children":440},{},[441],{"type":57,"tag":142,"props":442,"children":443},{},[444,449],{"type":57,"tag":146,"props":445,"children":446},{},[447],{"type":62,"value":448},"Use…",{"type":57,"tag":146,"props":450,"children":451},{},[452],{"type":62,"value":453},"When the criterion is…",{"type":57,"tag":157,"props":455,"children":456},{},[457,479,496],{"type":57,"tag":142,"props":458,"children":459},{},[460,469],{"type":57,"tag":164,"props":461,"children":462},{},[463],{"type":57,"tag":79,"props":464,"children":466},{"className":465},[],[467],{"type":62,"value":468},"hog",{"type":57,"tag":164,"props":470,"children":471},{},[472,474],{"type":62,"value":473},"Structural \u002F rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, ",{"type":57,"tag":71,"props":475,"children":476},{},[477],{"type":62,"value":478},"no provider key needed.",{"type":57,"tag":142,"props":480,"children":481},{},[482,491],{"type":57,"tag":164,"props":483,"children":484},{},[485],{"type":57,"tag":79,"props":486,"children":488},{"className":487},[],[489],{"type":62,"value":490},"llm_judge",{"type":57,"tag":164,"props":492,"children":493},{},[494],{"type":62,"value":495},"Subjective \u002F fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider, model, and usable provider key.",{"type":57,"tag":142,"props":497,"children":498},{},[499,508],{"type":57,"tag":164,"props":500,"children":501},{},[502],{"type":57,"tag":79,"props":503,"children":505},{"className":504},[],[506],{"type":62,"value":507},"sentiment",{"type":57,"tag":164,"props":509,"children":510},{},[511],{"type":62,"value":512},"You want sentiment labels on user messages, not a pass\u002Ffail (unless very specifically asked for, usually not relevant to this skill).",{"type":57,"tag":65,"props":514,"children":515},{},[516,518,523,525,530],{"type":62,"value":517},"Reach for ",{"type":57,"tag":79,"props":519,"children":521},{"className":520},[],[522],{"type":62,"value":468},{"type":62,"value":524}," first, escalate to ",{"type":57,"tag":79,"props":526,"children":528},{"className":527},[],[529],{"type":62,"value":490},{"type":62,"value":531}," if there is no deterministic way to check for what we want to check.",{"type":57,"tag":429,"props":533,"children":535},{"id":534},"_22-choose-the-target",[536],{"type":62,"value":537},"2.2 β€” Choose the target",{"type":57,"tag":134,"props":539,"children":540},{},[541,557],{"type":57,"tag":138,"props":542,"children":543},{},[544],{"type":57,"tag":142,"props":545,"children":546},{},[547,552],{"type":57,"tag":146,"props":548,"children":549},{},[550],{"type":62,"value":551},"Target",{"type":57,"tag":146,"props":553,"children":554},{},[555],{"type":62,"value":556},"Behavior",{"type":57,"tag":157,"props":558,"children":559},{},[560,584],{"type":57,"tag":142,"props":561,"children":562},{},[563,572],{"type":57,"tag":164,"props":564,"children":565},{},[566],{"type":57,"tag":79,"props":567,"children":569},{"className":568},[],[570],{"type":62,"value":571},"generation",{"type":57,"tag":164,"props":573,"children":574},{},[575,577,582],{"type":62,"value":576},"Runs once for each matching ",{"type":57,"tag":79,"props":578,"children":580},{"className":579},[],[581],{"type":62,"value":84},{"type":62,"value":583},", immediately after ingestion. This is the default.",{"type":57,"tag":142,"props":585,"children":586},{},[587,596],{"type":57,"tag":164,"props":588,"children":589},{},[590],{"type":57,"tag":79,"props":591,"children":593},{"className":592},[],[594],{"type":62,"value":595},"trace",{"type":57,"tag":164,"props":597,"children":598},{},[599],{"type":62,"value":600},"Runs once for the whole trace after the first matching generation and a configurable wait for the trace to finish.",{"type":57,"tag":65,"props":602,"children":603},{},[604,606,612,614,620],{"type":62,"value":605},"For a trace target, send ",{"type":57,"tag":79,"props":607,"children":609},{"className":608},[],[610],{"type":62,"value":611},"\"target\": \"trace\"",{"type":62,"value":613}," plus a settle config that controls when the trace is\nevaluated, discriminated on ",{"type":57,"tag":79,"props":615,"children":617},{"className":616},[],[618],{"type":62,"value":619},"strategy",{"type":62,"value":621},":",{"type":57,"tag":375,"props":623,"children":624},{},[625,651],{"type":57,"tag":379,"props":626,"children":627},{},[628,634,636,642,644,649],{"type":57,"tag":79,"props":629,"children":631},{"className":630},[],[632],{"type":62,"value":633},"{ \"strategy\": \"fixed_window\", \"window_seconds\": 1800 }",{"type":62,"value":635}," β€” evaluate a fixed wait after the first\nmatching generation. Between 10 seconds and 2 hours, defaults to 30 minutes. A ",{"type":57,"tag":79,"props":637,"children":639},{"className":638},[],[640],{"type":62,"value":641},"target_config",{"type":62,"value":643},"\nwithout a ",{"type":57,"tag":79,"props":645,"children":647},{"className":646},[],[648],{"type":62,"value":619},{"type":62,"value":650}," key means this.",{"type":57,"tag":379,"props":652,"children":653},{},[654,660,662,668],{"type":57,"tag":79,"props":655,"children":657},{"className":656},[],[658],{"type":62,"value":659},"{ \"strategy\": \"inactivity\", \"quiet_period_seconds\": 300, \"max_age_seconds\": 7200 }",{"type":62,"value":661}," β€” evaluate once\nthe trace has had no new activity for the quiet period (10 seconds to 30 minutes,\ndefaults to 5 minutes). ",{"type":57,"tag":79,"props":663,"children":665},{"className":664},[],[666],{"type":62,"value":667},"max_age_seconds",{"type":62,"value":669}," caps the total wait from the first matching generation\n(1 minute to 2 hours, defaults to 2 hours, must be at least the quiet period).",{"type":57,"tag":65,"props":671,"children":672},{},[673],{"type":62,"value":674},"Conditions still match the generation that triggers the run; the evaluator itself receives\nthe complete trace. Sentiment evaluations support only the generation target.",{"type":57,"tag":65,"props":676,"children":677},{},[678],{"type":62,"value":679},"New Hog source should use the globals shared by both targets:",{"type":57,"tag":134,"props":681,"children":682},{},[683,699],{"type":57,"tag":138,"props":684,"children":685},{},[686],{"type":57,"tag":142,"props":687,"children":688},{},[689,694],{"type":57,"tag":146,"props":690,"children":691},{},[692],{"type":62,"value":693},"Global",{"type":57,"tag":146,"props":695,"children":696},{},[697],{"type":62,"value":698},"Meaning",{"type":57,"tag":157,"props":700,"children":701},{},[702,719,766,791],{"type":57,"tag":142,"props":703,"children":704},{},[705,714],{"type":57,"tag":164,"props":706,"children":707},{},[708],{"type":57,"tag":79,"props":709,"children":711},{"className":710},[],[712],{"type":62,"value":713},"evaluation_events",{"type":57,"tag":164,"props":715,"children":716},{},[717],{"type":62,"value":718},"One generation event for a generation target, or every captured event for a trace target.",{"type":57,"tag":142,"props":720,"children":721},{},[722,731],{"type":57,"tag":164,"props":723,"children":724},{},[725],{"type":57,"tag":79,"props":726,"children":728},{"className":727},[],[729],{"type":62,"value":730},"target",{"type":57,"tag":164,"props":732,"children":733},{},[734,736,742,744,750,751,757,759,765],{"type":62,"value":735},"The target's ",{"type":57,"tag":79,"props":737,"children":739},{"className":738},[],[740],{"type":62,"value":741},"type",{"type":62,"value":743},", ",{"type":57,"tag":79,"props":745,"children":747},{"className":746},[],[748],{"type":62,"value":749},"id",{"type":62,"value":743},{"type":57,"tag":79,"props":752,"children":754},{"className":753},[],[755],{"type":62,"value":756},"total_cost_usd",{"type":62,"value":758},", and ",{"type":57,"tag":79,"props":760,"children":762},{"className":761},[],[763],{"type":62,"value":764},"total_latency_seconds",{"type":62,"value":125},{"type":57,"tag":142,"props":767,"children":768},{},[769,786],{"type":57,"tag":164,"props":770,"children":771},{},[772,778,780],{"type":57,"tag":79,"props":773,"children":775},{"className":774},[],[776],{"type":62,"value":777},"item.input_text",{"type":62,"value":779}," \u002F ",{"type":57,"tag":79,"props":781,"children":783},{"className":782},[],[784],{"type":62,"value":785},"item.output_text",{"type":57,"tag":164,"props":787,"children":788},{},[789],{"type":62,"value":790},"Best-effort readable projections; use these for length, keyword, and regex checks.",{"type":57,"tag":142,"props":792,"children":793},{},[794,810],{"type":57,"tag":164,"props":795,"children":796},{},[797,803,804],{"type":57,"tag":79,"props":798,"children":800},{"className":799},[],[801],{"type":62,"value":802},"item.input",{"type":62,"value":779},{"type":57,"tag":79,"props":805,"children":807},{"className":806},[],[808],{"type":62,"value":809},"item.output",{"type":57,"tag":164,"props":811,"children":812},{},[813],{"type":62,"value":814},"Original serialized values; use these when the evaluator needs to parse the captured JSON itself.",{"type":57,"tag":65,"props":816,"children":817},{},[818,820,826,827,833,834,840,841,847,849,855,857,862,864,869,870,875],{"type":62,"value":819},"Generation evaluations still expose top-level ",{"type":57,"tag":79,"props":821,"children":823},{"className":822},[],[824],{"type":62,"value":825},"input",{"type":62,"value":743},{"type":57,"tag":79,"props":828,"children":830},{"className":829},[],[831],{"type":62,"value":832},"output",{"type":62,"value":743},{"type":57,"tag":79,"props":835,"children":837},{"className":836},[],[838],{"type":62,"value":839},"properties",{"type":62,"value":758},{"type":57,"tag":79,"props":842,"children":844},{"className":843},[],[845],{"type":62,"value":846},"event",{"type":62,"value":848},". Trace evaluations\nstill expose their original ",{"type":57,"tag":79,"props":850,"children":852},{"className":851},[],[853],{"type":62,"value":854},"events",{"type":62,"value":856}," and ",{"type":57,"tag":79,"props":858,"children":860},{"className":859},[],[861],{"type":62,"value":595},{"type":62,"value":863}," globals. Those globals are kept for compatibility with saved\nevaluators. Do not use target-specific globals in new source that needs to work for both targets. The text\nprojections recognize common provider payloads but are not authoritative; use ",{"type":57,"tag":79,"props":865,"children":867},{"className":866},[],[868],{"type":62,"value":802},{"type":62,"value":779},{"type":57,"tag":79,"props":871,"children":873},{"className":872},[],[874],{"type":62,"value":809},{"type":62,"value":876}," when\nexact structure matters.",{"type":57,"tag":429,"props":878,"children":880},{"id":879},"_23-configure-the-llm-judge",[881],{"type":62,"value":882},"2.3 β€” Configure the LLM judge",{"type":57,"tag":65,"props":884,"children":885},{},[886,887,892,894,900,901,907,909,915],{"type":62,"value":69},{"type":57,"tag":79,"props":888,"children":890},{"className":889},[],[891],{"type":62,"value":490},{"type":62,"value":893}," evaluation requires a valid ",{"type":57,"tag":79,"props":895,"children":897},{"className":896},[],[898],{"type":62,"value":899},"provider",{"type":62,"value":856},{"type":57,"tag":79,"props":902,"children":904},{"className":903},[],[905],{"type":62,"value":906},"model",{"type":62,"value":908},". It also needs a usable provider key\nwhen it runs. ",{"type":57,"tag":79,"props":910,"children":912},{"className":911},[],[913],{"type":62,"value":914},"provider_key_id",{"type":62,"value":916}," controls whether the evaluation pins one specific key:",{"type":57,"tag":375,"props":918,"children":919},{},[920,939],{"type":57,"tag":379,"props":921,"children":922},{},[923,925,930,932,937],{"type":62,"value":924},"Set ",{"type":57,"tag":79,"props":926,"children":928},{"className":927},[],[929],{"type":62,"value":914},{"type":62,"value":931}," to the UUID of an ",{"type":57,"tag":79,"props":933,"children":935},{"className":934},[],[936],{"type":62,"value":200},{"type":62,"value":938},"-state key for the same provider to pin it.",{"type":57,"tag":379,"props":940,"children":941},{},[942,943,948,950,956,958,963,965,971],{"type":62,"value":924},{"type":57,"tag":79,"props":944,"children":946},{"className":945},[],[947],{"type":62,"value":914},{"type":62,"value":949}," to ",{"type":57,"tag":79,"props":951,"children":953},{"className":952},[],[954],{"type":62,"value":955},"null",{"type":62,"value":957}," to use the team's active provider key. The active key must be in the\n",{"type":57,"tag":79,"props":959,"children":961},{"className":960},[],[962],{"type":62,"value":200},{"type":62,"value":964}," state and use the same provider as ",{"type":57,"tag":79,"props":966,"children":968},{"className":967},[],[969],{"type":62,"value":970},"model_configuration.provider",{"type":62,"value":125},{"type":57,"tag":65,"props":973,"children":974},{},[975],{"type":62,"value":976},"Hog and sentiment evaluations skip this step.",{"type":57,"tag":978,"props":979,"children":984},"pre",{"className":980,"code":981,"language":982,"meta":983,"style":983},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","posthog:llma-evaluation-config-get \u002F\u002F check active_provider_key for an unpinned judge\nposthog:llma-provider-key-list \u002F\u002F find an ok-state key to pin\nposthog:llma-evaluation-judge-models \u002F\u002F { \"provider\": \"openai\" } β†’ valid models\n","json","",[985],{"type":57,"tag":79,"props":986,"children":987},{"__ignoreMap":983},[988,1006,1020],{"type":57,"tag":989,"props":990,"children":993},"span",{"class":991,"line":992},"line",1,[994,1000],{"type":57,"tag":989,"props":995,"children":997},{"style":996},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[998],{"type":62,"value":999},"posthog:llma-evaluation-config-get ",{"type":57,"tag":989,"props":1001,"children":1003},{"style":1002},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1004],{"type":62,"value":1005},"\u002F\u002F check active_provider_key for an unpinned judge\n",{"type":57,"tag":989,"props":1007,"children":1009},{"class":991,"line":1008},2,[1010,1015],{"type":57,"tag":989,"props":1011,"children":1012},{"style":996},[1013],{"type":62,"value":1014},"posthog:llma-provider-key-list ",{"type":57,"tag":989,"props":1016,"children":1017},{"style":1002},[1018],{"type":62,"value":1019},"\u002F\u002F find an ok-state key to pin\n",{"type":57,"tag":989,"props":1021,"children":1023},{"class":991,"line":1022},3,[1024,1029],{"type":57,"tag":989,"props":1025,"children":1026},{"style":996},[1027],{"type":62,"value":1028},"posthog:llma-evaluation-judge-models ",{"type":57,"tag":989,"props":1030,"children":1031},{"style":1002},[1032],{"type":62,"value":1033},"\u002F\u002F { \"provider\": \"openai\" } β†’ valid models\n",{"type":57,"tag":65,"props":1035,"children":1036},{},[1037,1039,1045,1047,1052,1054,1059,1061,1067],{"type":62,"value":1038},"Confirm the provider and model with ",{"type":57,"tag":79,"props":1040,"children":1042},{"className":1041},[],[1043],{"type":62,"value":1044},"llma-evaluation-judge-models",{"type":62,"value":1046},". Prefer pinning the chosen key so a later\nteam-wide active-key change does not change how the evaluation runs. Leave ",{"type":57,"tag":79,"props":1048,"children":1050},{"className":1049},[],[1051],{"type":62,"value":914},{"type":62,"value":1053}," as ",{"type":57,"tag":79,"props":1055,"children":1057},{"className":1056},[],[1058],{"type":62,"value":955},{"type":62,"value":1060}," only\nafter ",{"type":57,"tag":79,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":62,"value":1066},"llma-evaluation-config-get",{"type":62,"value":1068}," confirms the active key is usable and its provider matches.",{"type":57,"tag":65,"props":1070,"children":1071},{},[1072],{"type":62,"value":1073},"If there is no usable key, you may still create a disabled draft for the user to review. Do not spot-run or\nenable it. Ask the user to add or validate a key in the UI before continuing.",{"type":57,"tag":429,"props":1075,"children":1077},{"id":1076},"_24-create-it-disabled",[1078],{"type":62,"value":1079},"2.4 β€” Create it disabled",{"type":57,"tag":65,"props":1081,"children":1082},{},[1083,1085,1090,1092,1097],{"type":62,"value":1084},"Create with ",{"type":57,"tag":79,"props":1086,"children":1088},{"className":1087},[],[1089],{"type":62,"value":259},{"type":62,"value":1091}," so nothing fires until the scope is verified. Minimal ",{"type":57,"tag":79,"props":1093,"children":1095},{"className":1094},[],[1096],{"type":62,"value":468},{"type":62,"value":1098}," example:",{"type":57,"tag":978,"props":1100,"children":1102},{"className":980,"code":1101,"language":982,"meta":983,"style":983},"posthog:llma-evaluation-create\n{\n \"name\": \"Output is not empty\",\n \"description\": \"Fails when a generation has no readable output\",\n \"evaluation_type\": \"hog\",\n \"evaluation_config\": { \"source\": \"let count := 0\\nfor (let i, item in evaluation_events) {\\n if (item.event == '$ai_generation') {\\n count := count + 1\\n if (length(trim(item.output_text)) == 0) { return false }\\n }\\n}\\nreturn count > 0\" },\n \"output_type\": \"boolean\",\n \"output_config\": { \"allows_na\": false },\n \"target\": \"generation\",\n \"target_config\": {},\n \"conditions\": [\n { \"id\": \"default\", \"rollout_percentage\": 100, \"properties\": [{ \"key\": \"$ai_model\", \"type\": \"event\", \"operator\": \"icontains\", \"value\": \"gpt\" }] }\n ],\n \"enabled\": false\n}\n",[1103],{"type":57,"tag":79,"props":1104,"children":1105},{"__ignoreMap":983},[1106,1114,1123,1166,1204,1241,1367,1405,1456,1492,1517,1542,1773,1782,1808],{"type":57,"tag":989,"props":1107,"children":1108},{"class":991,"line":992},[1109],{"type":57,"tag":989,"props":1110,"children":1111},{"style":996},[1112],{"type":62,"value":1113},"posthog:llma-evaluation-create\n",{"type":57,"tag":989,"props":1115,"children":1116},{"class":991,"line":1008},[1117],{"type":57,"tag":989,"props":1118,"children":1120},{"style":1119},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1121],{"type":62,"value":1122},"{\n",{"type":57,"tag":989,"props":1124,"children":1125},{"class":991,"line":1022},[1126,1131,1137,1142,1146,1151,1157,1161],{"type":57,"tag":989,"props":1127,"children":1128},{"style":1119},[1129],{"type":62,"value":1130}," \"",{"type":57,"tag":989,"props":1132,"children":1134},{"style":1133},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1135],{"type":62,"value":1136},"name",{"type":57,"tag":989,"props":1138,"children":1139},{"style":1119},[1140],{"type":62,"value":1141},"\"",{"type":57,"tag":989,"props":1143,"children":1144},{"style":1119},[1145],{"type":62,"value":621},{"type":57,"tag":989,"props":1147,"children":1148},{"style":1119},[1149],{"type":62,"value":1150}," \"",{"type":57,"tag":989,"props":1152,"children":1154},{"style":1153},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1155],{"type":62,"value":1156},"Output is not empty",{"type":57,"tag":989,"props":1158,"children":1159},{"style":1119},[1160],{"type":62,"value":1141},{"type":57,"tag":989,"props":1162,"children":1163},{"style":1119},[1164],{"type":62,"value":1165},",\n",{"type":57,"tag":989,"props":1167,"children":1169},{"class":991,"line":1168},4,[1170,1174,1179,1183,1187,1191,1196,1200],{"type":57,"tag":989,"props":1171,"children":1172},{"style":1119},[1173],{"type":62,"value":1130},{"type":57,"tag":989,"props":1175,"children":1176},{"style":1133},[1177],{"type":62,"value":1178},"description",{"type":57,"tag":989,"props":1180,"children":1181},{"style":1119},[1182],{"type":62,"value":1141},{"type":57,"tag":989,"props":1184,"children":1185},{"style":1119},[1186],{"type":62,"value":621},{"type":57,"tag":989,"props":1188,"children":1189},{"style":1119},[1190],{"type":62,"value":1150},{"type":57,"tag":989,"props":1192,"children":1193},{"style":1153},[1194],{"type":62,"value":1195},"Fails when a generation has no readable output",{"type":57,"tag":989,"props":1197,"children":1198},{"style":1119},[1199],{"type":62,"value":1141},{"type":57,"tag":989,"props":1201,"children":1202},{"style":1119},[1203],{"type":62,"value":1165},{"type":57,"tag":989,"props":1205,"children":1207},{"class":991,"line":1206},5,[1208,1212,1217,1221,1225,1229,1233,1237],{"type":57,"tag":989,"props":1209,"children":1210},{"style":1119},[1211],{"type":62,"value":1130},{"type":57,"tag":989,"props":1213,"children":1214},{"style":1133},[1215],{"type":62,"value":1216},"evaluation_type",{"type":57,"tag":989,"props":1218,"children":1219},{"style":1119},[1220],{"type":62,"value":1141},{"type":57,"tag":989,"props":1222,"children":1223},{"style":1119},[1224],{"type":62,"value":621},{"type":57,"tag":989,"props":1226,"children":1227},{"style":1119},[1228],{"type":62,"value":1150},{"type":57,"tag":989,"props":1230,"children":1231},{"style":1153},[1232],{"type":62,"value":468},{"type":57,"tag":989,"props":1234,"children":1235},{"style":1119},[1236],{"type":62,"value":1141},{"type":57,"tag":989,"props":1238,"children":1239},{"style":1119},[1240],{"type":62,"value":1165},{"type":57,"tag":989,"props":1242,"children":1244},{"class":991,"line":1243},6,[1245,1249,1254,1258,1262,1267,1271,1277,1281,1285,1289,1294,1299,1304,1308,1313,1317,1322,1326,1331,1335,1340,1344,1349,1353,1358,1362],{"type":57,"tag":989,"props":1246,"children":1247},{"style":1119},[1248],{"type":62,"value":1130},{"type":57,"tag":989,"props":1250,"children":1251},{"style":1133},[1252],{"type":62,"value":1253},"evaluation_config",{"type":57,"tag":989,"props":1255,"children":1256},{"style":1119},[1257],{"type":62,"value":1141},{"type":57,"tag":989,"props":1259,"children":1260},{"style":1119},[1261],{"type":62,"value":621},{"type":57,"tag":989,"props":1263,"children":1264},{"style":1119},[1265],{"type":62,"value":1266}," {",{"type":57,"tag":989,"props":1268,"children":1269},{"style":1119},[1270],{"type":62,"value":1150},{"type":57,"tag":989,"props":1272,"children":1274},{"style":1273},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1275],{"type":62,"value":1276},"source",{"type":57,"tag":989,"props":1278,"children":1279},{"style":1119},[1280],{"type":62,"value":1141},{"type":57,"tag":989,"props":1282,"children":1283},{"style":1119},[1284],{"type":62,"value":621},{"type":57,"tag":989,"props":1286,"children":1287},{"style":1119},[1288],{"type":62,"value":1150},{"type":57,"tag":989,"props":1290,"children":1291},{"style":1153},[1292],{"type":62,"value":1293},"let count := 0",{"type":57,"tag":989,"props":1295,"children":1296},{"style":996},[1297],{"type":62,"value":1298},"\\n",{"type":57,"tag":989,"props":1300,"children":1301},{"style":1153},[1302],{"type":62,"value":1303},"for (let i, item in evaluation_events) {",{"type":57,"tag":989,"props":1305,"children":1306},{"style":996},[1307],{"type":62,"value":1298},{"type":57,"tag":989,"props":1309,"children":1310},{"style":1153},[1311],{"type":62,"value":1312}," if (item.event == '$ai_generation') {",{"type":57,"tag":989,"props":1314,"children":1315},{"style":996},[1316],{"type":62,"value":1298},{"type":57,"tag":989,"props":1318,"children":1319},{"style":1153},[1320],{"type":62,"value":1321}," count := count + 1",{"type":57,"tag":989,"props":1323,"children":1324},{"style":996},[1325],{"type":62,"value":1298},{"type":57,"tag":989,"props":1327,"children":1328},{"style":1153},[1329],{"type":62,"value":1330}," if (length(trim(item.output_text)) == 0) { return false }",{"type":57,"tag":989,"props":1332,"children":1333},{"style":996},[1334],{"type":62,"value":1298},{"type":57,"tag":989,"props":1336,"children":1337},{"style":1153},[1338],{"type":62,"value":1339}," }",{"type":57,"tag":989,"props":1341,"children":1342},{"style":996},[1343],{"type":62,"value":1298},{"type":57,"tag":989,"props":1345,"children":1346},{"style":1153},[1347],{"type":62,"value":1348},"}",{"type":57,"tag":989,"props":1350,"children":1351},{"style":996},[1352],{"type":62,"value":1298},{"type":57,"tag":989,"props":1354,"children":1355},{"style":1153},[1356],{"type":62,"value":1357},"return count > 0",{"type":57,"tag":989,"props":1359,"children":1360},{"style":1119},[1361],{"type":62,"value":1141},{"type":57,"tag":989,"props":1363,"children":1364},{"style":1119},[1365],{"type":62,"value":1366}," },\n",{"type":57,"tag":989,"props":1368,"children":1370},{"class":991,"line":1369},7,[1371,1375,1380,1384,1388,1392,1397,1401],{"type":57,"tag":989,"props":1372,"children":1373},{"style":1119},[1374],{"type":62,"value":1130},{"type":57,"tag":989,"props":1376,"children":1377},{"style":1133},[1378],{"type":62,"value":1379},"output_type",{"type":57,"tag":989,"props":1381,"children":1382},{"style":1119},[1383],{"type":62,"value":1141},{"type":57,"tag":989,"props":1385,"children":1386},{"style":1119},[1387],{"type":62,"value":621},{"type":57,"tag":989,"props":1389,"children":1390},{"style":1119},[1391],{"type":62,"value":1150},{"type":57,"tag":989,"props":1393,"children":1394},{"style":1153},[1395],{"type":62,"value":1396},"boolean",{"type":57,"tag":989,"props":1398,"children":1399},{"style":1119},[1400],{"type":62,"value":1141},{"type":57,"tag":989,"props":1402,"children":1403},{"style":1119},[1404],{"type":62,"value":1165},{"type":57,"tag":989,"props":1406,"children":1408},{"class":991,"line":1407},8,[1409,1413,1418,1422,1426,1430,1434,1439,1443,1447,1452],{"type":57,"tag":989,"props":1410,"children":1411},{"style":1119},[1412],{"type":62,"value":1130},{"type":57,"tag":989,"props":1414,"children":1415},{"style":1133},[1416],{"type":62,"value":1417},"output_config",{"type":57,"tag":989,"props":1419,"children":1420},{"style":1119},[1421],{"type":62,"value":1141},{"type":57,"tag":989,"props":1423,"children":1424},{"style":1119},[1425],{"type":62,"value":621},{"type":57,"tag":989,"props":1427,"children":1428},{"style":1119},[1429],{"type":62,"value":1266},{"type":57,"tag":989,"props":1431,"children":1432},{"style":1119},[1433],{"type":62,"value":1150},{"type":57,"tag":989,"props":1435,"children":1436},{"style":1273},[1437],{"type":62,"value":1438},"allows_na",{"type":57,"tag":989,"props":1440,"children":1441},{"style":1119},[1442],{"type":62,"value":1141},{"type":57,"tag":989,"props":1444,"children":1445},{"style":1119},[1446],{"type":62,"value":621},{"type":57,"tag":989,"props":1448,"children":1449},{"style":1119},[1450],{"type":62,"value":1451}," false",{"type":57,"tag":989,"props":1453,"children":1454},{"style":1119},[1455],{"type":62,"value":1366},{"type":57,"tag":989,"props":1457,"children":1459},{"class":991,"line":1458},9,[1460,1464,1468,1472,1476,1480,1484,1488],{"type":57,"tag":989,"props":1461,"children":1462},{"style":1119},[1463],{"type":62,"value":1130},{"type":57,"tag":989,"props":1465,"children":1466},{"style":1133},[1467],{"type":62,"value":730},{"type":57,"tag":989,"props":1469,"children":1470},{"style":1119},[1471],{"type":62,"value":1141},{"type":57,"tag":989,"props":1473,"children":1474},{"style":1119},[1475],{"type":62,"value":621},{"type":57,"tag":989,"props":1477,"children":1478},{"style":1119},[1479],{"type":62,"value":1150},{"type":57,"tag":989,"props":1481,"children":1482},{"style":1153},[1483],{"type":62,"value":571},{"type":57,"tag":989,"props":1485,"children":1486},{"style":1119},[1487],{"type":62,"value":1141},{"type":57,"tag":989,"props":1489,"children":1490},{"style":1119},[1491],{"type":62,"value":1165},{"type":57,"tag":989,"props":1493,"children":1495},{"class":991,"line":1494},10,[1496,1500,1504,1508,1512],{"type":57,"tag":989,"props":1497,"children":1498},{"style":1119},[1499],{"type":62,"value":1130},{"type":57,"tag":989,"props":1501,"children":1502},{"style":1133},[1503],{"type":62,"value":641},{"type":57,"tag":989,"props":1505,"children":1506},{"style":1119},[1507],{"type":62,"value":1141},{"type":57,"tag":989,"props":1509,"children":1510},{"style":1119},[1511],{"type":62,"value":621},{"type":57,"tag":989,"props":1513,"children":1514},{"style":1119},[1515],{"type":62,"value":1516}," {},\n",{"type":57,"tag":989,"props":1518,"children":1520},{"class":991,"line":1519},11,[1521,1525,1529,1533,1537],{"type":57,"tag":989,"props":1522,"children":1523},{"style":1119},[1524],{"type":62,"value":1130},{"type":57,"tag":989,"props":1526,"children":1527},{"style":1133},[1528],{"type":62,"value":346},{"type":57,"tag":989,"props":1530,"children":1531},{"style":1119},[1532],{"type":62,"value":1141},{"type":57,"tag":989,"props":1534,"children":1535},{"style":1119},[1536],{"type":62,"value":621},{"type":57,"tag":989,"props":1538,"children":1539},{"style":1119},[1540],{"type":62,"value":1541}," [\n",{"type":57,"tag":989,"props":1543,"children":1545},{"class":991,"line":1544},12,[1546,1551,1555,1559,1563,1567,1571,1576,1580,1585,1589,1594,1598,1602,1608,1612,1616,1620,1624,1628,1633,1637,1642,1646,1650,1654,1659,1663,1667,1671,1675,1679,1683,1687,1691,1695,1699,1703,1708,1712,1716,1720,1725,1729,1733,1737,1742,1746,1750,1754,1759,1763,1768],{"type":57,"tag":989,"props":1547,"children":1548},{"style":1119},[1549],{"type":62,"value":1550}," {",{"type":57,"tag":989,"props":1552,"children":1553},{"style":1119},[1554],{"type":62,"value":1150},{"type":57,"tag":989,"props":1556,"children":1557},{"style":1273},[1558],{"type":62,"value":749},{"type":57,"tag":989,"props":1560,"children":1561},{"style":1119},[1562],{"type":62,"value":1141},{"type":57,"tag":989,"props":1564,"children":1565},{"style":1119},[1566],{"type":62,"value":621},{"type":57,"tag":989,"props":1568,"children":1569},{"style":1119},[1570],{"type":62,"value":1150},{"type":57,"tag":989,"props":1572,"children":1573},{"style":1153},[1574],{"type":62,"value":1575},"default",{"type":57,"tag":989,"props":1577,"children":1578},{"style":1119},[1579],{"type":62,"value":1141},{"type":57,"tag":989,"props":1581,"children":1582},{"style":1119},[1583],{"type":62,"value":1584},",",{"type":57,"tag":989,"props":1586,"children":1587},{"style":1119},[1588],{"type":62,"value":1150},{"type":57,"tag":989,"props":1590,"children":1591},{"style":1273},[1592],{"type":62,"value":1593},"rollout_percentage",{"type":57,"tag":989,"props":1595,"children":1596},{"style":1119},[1597],{"type":62,"value":1141},{"type":57,"tag":989,"props":1599,"children":1600},{"style":1119},[1601],{"type":62,"value":621},{"type":57,"tag":989,"props":1603,"children":1605},{"style":1604},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1606],{"type":62,"value":1607}," 100",{"type":57,"tag":989,"props":1609,"children":1610},{"style":1119},[1611],{"type":62,"value":1584},{"type":57,"tag":989,"props":1613,"children":1614},{"style":1119},[1615],{"type":62,"value":1150},{"type":57,"tag":989,"props":1617,"children":1618},{"style":1273},[1619],{"type":62,"value":839},{"type":57,"tag":989,"props":1621,"children":1622},{"style":1119},[1623],{"type":62,"value":1141},{"type":57,"tag":989,"props":1625,"children":1626},{"style":1119},[1627],{"type":62,"value":621},{"type":57,"tag":989,"props":1629,"children":1630},{"style":1119},[1631],{"type":62,"value":1632}," [{",{"type":57,"tag":989,"props":1634,"children":1635},{"style":1119},[1636],{"type":62,"value":1150},{"type":57,"tag":989,"props":1638,"children":1639},{"style":1604},[1640],{"type":62,"value":1641},"key",{"type":57,"tag":989,"props":1643,"children":1644},{"style":1119},[1645],{"type":62,"value":1141},{"type":57,"tag":989,"props":1647,"children":1648},{"style":1119},[1649],{"type":62,"value":621},{"type":57,"tag":989,"props":1651,"children":1652},{"style":1119},[1653],{"type":62,"value":1150},{"type":57,"tag":989,"props":1655,"children":1656},{"style":1153},[1657],{"type":62,"value":1658},"$ai_model",{"type":57,"tag":989,"props":1660,"children":1661},{"style":1119},[1662],{"type":62,"value":1141},{"type":57,"tag":989,"props":1664,"children":1665},{"style":1119},[1666],{"type":62,"value":1584},{"type":57,"tag":989,"props":1668,"children":1669},{"style":1119},[1670],{"type":62,"value":1150},{"type":57,"tag":989,"props":1672,"children":1673},{"style":1604},[1674],{"type":62,"value":741},{"type":57,"tag":989,"props":1676,"children":1677},{"style":1119},[1678],{"type":62,"value":1141},{"type":57,"tag":989,"props":1680,"children":1681},{"style":1119},[1682],{"type":62,"value":621},{"type":57,"tag":989,"props":1684,"children":1685},{"style":1119},[1686],{"type":62,"value":1150},{"type":57,"tag":989,"props":1688,"children":1689},{"style":1153},[1690],{"type":62,"value":846},{"type":57,"tag":989,"props":1692,"children":1693},{"style":1119},[1694],{"type":62,"value":1141},{"type":57,"tag":989,"props":1696,"children":1697},{"style":1119},[1698],{"type":62,"value":1584},{"type":57,"tag":989,"props":1700,"children":1701},{"style":1119},[1702],{"type":62,"value":1150},{"type":57,"tag":989,"props":1704,"children":1705},{"style":1604},[1706],{"type":62,"value":1707},"operator",{"type":57,"tag":989,"props":1709,"children":1710},{"style":1119},[1711],{"type":62,"value":1141},{"type":57,"tag":989,"props":1713,"children":1714},{"style":1119},[1715],{"type":62,"value":621},{"type":57,"tag":989,"props":1717,"children":1718},{"style":1119},[1719],{"type":62,"value":1150},{"type":57,"tag":989,"props":1721,"children":1722},{"style":1153},[1723],{"type":62,"value":1724},"icontains",{"type":57,"tag":989,"props":1726,"children":1727},{"style":1119},[1728],{"type":62,"value":1141},{"type":57,"tag":989,"props":1730,"children":1731},{"style":1119},[1732],{"type":62,"value":1584},{"type":57,"tag":989,"props":1734,"children":1735},{"style":1119},[1736],{"type":62,"value":1150},{"type":57,"tag":989,"props":1738,"children":1739},{"style":1604},[1740],{"type":62,"value":1741},"value",{"type":57,"tag":989,"props":1743,"children":1744},{"style":1119},[1745],{"type":62,"value":1141},{"type":57,"tag":989,"props":1747,"children":1748},{"style":1119},[1749],{"type":62,"value":621},{"type":57,"tag":989,"props":1751,"children":1752},{"style":1119},[1753],{"type":62,"value":1150},{"type":57,"tag":989,"props":1755,"children":1756},{"style":1153},[1757],{"type":62,"value":1758},"gpt",{"type":57,"tag":989,"props":1760,"children":1761},{"style":1119},[1762],{"type":62,"value":1141},{"type":57,"tag":989,"props":1764,"children":1765},{"style":1119},[1766],{"type":62,"value":1767}," }]",{"type":57,"tag":989,"props":1769,"children":1770},{"style":1119},[1771],{"type":62,"value":1772}," }\n",{"type":57,"tag":989,"props":1774,"children":1776},{"class":991,"line":1775},13,[1777],{"type":57,"tag":989,"props":1778,"children":1779},{"style":1119},[1780],{"type":62,"value":1781}," ],\n",{"type":57,"tag":989,"props":1783,"children":1785},{"class":991,"line":1784},14,[1786,1790,1795,1799,1803],{"type":57,"tag":989,"props":1787,"children":1788},{"style":1119},[1789],{"type":62,"value":1130},{"type":57,"tag":989,"props":1791,"children":1792},{"style":1133},[1793],{"type":62,"value":1794},"enabled",{"type":57,"tag":989,"props":1796,"children":1797},{"style":1119},[1798],{"type":62,"value":1141},{"type":57,"tag":989,"props":1800,"children":1801},{"style":1119},[1802],{"type":62,"value":621},{"type":57,"tag":989,"props":1804,"children":1805},{"style":1119},[1806],{"type":62,"value":1807}," false\n",{"type":57,"tag":989,"props":1809,"children":1811},{"class":991,"line":1810},15,[1812],{"type":57,"tag":989,"props":1813,"children":1814},{"style":1119},[1815],{"type":62,"value":1816},"}\n",{"type":57,"tag":65,"props":1818,"children":1819},{},[1820,1822,1827,1829,1834,1835,1841,1843,1849,1851,1856,1858,1863,1865,1869],{"type":62,"value":1821},"For ",{"type":57,"tag":79,"props":1823,"children":1825},{"className":1824},[],[1826],{"type":62,"value":490},{"type":62,"value":1828},", swap ",{"type":57,"tag":79,"props":1830,"children":1832},{"className":1831},[],[1833],{"type":62,"value":1253},{"type":62,"value":949},{"type":57,"tag":79,"props":1836,"children":1838},{"className":1837},[],[1839],{"type":62,"value":1840},"{ \"prompt\": \"…\" }",{"type":62,"value":1842}," and add\n",{"type":57,"tag":79,"props":1844,"children":1846},{"className":1845},[],[1847],{"type":62,"value":1848},"\"model_configuration\": { \"provider\": \"openai\", \"model\": \"gpt-5-mini\", \"provider_key_id\": \"\u003Cuuid of an ok-state key from llma-provider-key-list>\" }",{"type":62,"value":1850},".\nUse ",{"type":57,"tag":79,"props":1852,"children":1854},{"className":1853},[],[1855],{"type":62,"value":955},{"type":62,"value":1857}," only when the active team key is ",{"type":57,"tag":79,"props":1859,"children":1861},{"className":1860},[],[1862],{"type":62,"value":200},{"type":62,"value":1864}," and uses the same provider. Full field reference:\n",{"type":57,"tag":350,"props":1866,"children":1867},{"href":352},[1868],{"type":62,"value":352},{"type":62,"value":125},{"type":57,"tag":429,"props":1871,"children":1873},{"id":1872},"_25-verify-the-scope-before-enabling",[1874],{"type":62,"value":1875},"2.5 β€” Verify the scope before enabling",{"type":57,"tag":65,"props":1877,"children":1878},{},[1879,1884],{"type":57,"tag":79,"props":1880,"children":1882},{"className":1881},[],[1883],{"type":62,"value":346},{"type":62,"value":1885}," is where online evals go wrong: too broad and you evaluate (and bill) a firehose; too narrow\nand it never fires. Confirm the filter matches the events you expect, and roughly how many per day:",{"type":57,"tag":978,"props":1887,"children":1891},{"className":1888,"code":1889,"language":1890,"meta":983,"style":983},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","posthog:execute-sql\nSELECT count() AS matched, count() \u002F 7 AS per_day\nFROM events\nWHERE event = '$ai_generation'\n AND properties.$ai_model ILIKE '%gpt%' -- mirror each condition property\n AND timestamp >= now() - INTERVAL 7 DAY\n","sql",[1892],{"type":57,"tag":79,"props":1893,"children":1894},{"__ignoreMap":983},[1895,1903,1911,1919,1927,1935],{"type":57,"tag":989,"props":1896,"children":1897},{"class":991,"line":992},[1898],{"type":57,"tag":989,"props":1899,"children":1900},{},[1901],{"type":62,"value":1902},"posthog:execute-sql\n",{"type":57,"tag":989,"props":1904,"children":1905},{"class":991,"line":1008},[1906],{"type":57,"tag":989,"props":1907,"children":1908},{},[1909],{"type":62,"value":1910},"SELECT count() AS matched, count() \u002F 7 AS per_day\n",{"type":57,"tag":989,"props":1912,"children":1913},{"class":991,"line":1022},[1914],{"type":57,"tag":989,"props":1915,"children":1916},{},[1917],{"type":62,"value":1918},"FROM events\n",{"type":57,"tag":989,"props":1920,"children":1921},{"class":991,"line":1168},[1922],{"type":57,"tag":989,"props":1923,"children":1924},{},[1925],{"type":62,"value":1926},"WHERE event = '$ai_generation'\n",{"type":57,"tag":989,"props":1928,"children":1929},{"class":991,"line":1206},[1930],{"type":57,"tag":989,"props":1931,"children":1932},{},[1933],{"type":62,"value":1934}," AND properties.$ai_model ILIKE '%gpt%' -- mirror each condition property\n",{"type":57,"tag":989,"props":1936,"children":1937},{"class":991,"line":1243},[1938],{"type":57,"tag":989,"props":1939,"children":1940},{},[1941],{"type":62,"value":1942}," AND timestamp >= now() - INTERVAL 7 DAY\n",{"type":57,"tag":65,"props":1944,"children":1945},{},[1946,1948,1954,1956,1962],{"type":62,"value":1947},"For generation targets, ",{"type":57,"tag":79,"props":1949,"children":1951},{"className":1950},[],[1952],{"type":62,"value":1953},"count()",{"type":62,"value":1955}," is the run volume. For trace targets, count distinct non-empty\n",{"type":57,"tag":79,"props":1957,"children":1959},{"className":1958},[],[1960],{"type":62,"value":1961},"$ai_trace_id",{"type":62,"value":1963}," values because matching generations from the same trace schedule only one run.",{"type":57,"tag":65,"props":1965,"children":1966},{},[1967,1969,1974,1976,1982,1984,1990],{"type":62,"value":1968},"If volume is high, set ",{"type":57,"tag":79,"props":1970,"children":1972},{"className":1971},[],[1973],{"type":62,"value":1593},{"type":62,"value":1975}," below 100 to sample. Spot-check the evaluator with\n",{"type":57,"tag":79,"props":1977,"children":1979},{"className":1978},[],[1980],{"type":62,"value":1981},"llma-evaluation-test-hog",{"type":62,"value":1983}," (hog) or ",{"type":57,"tag":79,"props":1985,"children":1987},{"className":1986},[],[1988],{"type":62,"value":1989},"llma-evaluation-run",{"type":62,"value":1991}," against one generation (llm_judge).\nBoth tools currently use generation samples; for a trace target they can check shared source or prompt behavior,\nbut they do not reproduce the complete settled trace. Review the first live trace results before increasing rollout.",{"type":57,"tag":1993,"props":1994,"children":1995},"blockquote",{},[1996],{"type":57,"tag":65,"props":1997,"children":1998},{},[1999,2004,2006,2011],{"type":57,"tag":71,"props":2000,"children":2001},{},[2002],{"type":62,"value":2003},"Watch out:",{"type":62,"value":2005}," some orgs reuse a single ",{"type":57,"tag":79,"props":2007,"children":2009},{"className":2008},[],[2010],{"type":62,"value":1961},{"type":62,"value":2012}," across 100k+ events. Scoping by trace-ID prefix\ncan match far more than expected β€” verify volume with the SQL above before enabling.",{"type":57,"tag":429,"props":2014,"children":2016},{"id":2015},"_26-enable-then-close-the-loop",[2017],{"type":62,"value":2018},"2.6 β€” Enable, then close the loop",{"type":57,"tag":978,"props":2020,"children":2022},{"className":980,"code":2021,"language":982,"meta":983,"style":983},"posthog:llma-evaluation-update\n{ \"evaluationId\": \"\u003Cuuid>\", \"enabled\": true }\n",[2023],{"type":57,"tag":79,"props":2024,"children":2025},{"__ignoreMap":983},[2026,2034],{"type":57,"tag":989,"props":2027,"children":2028},{"class":991,"line":992},[2029],{"type":57,"tag":989,"props":2030,"children":2031},{"style":996},[2032],{"type":62,"value":2033},"posthog:llma-evaluation-update\n",{"type":57,"tag":989,"props":2035,"children":2036},{"class":991,"line":1008},[2037,2042,2046,2051,2055,2059,2063,2068,2072,2076,2080,2084,2088,2092,2097],{"type":57,"tag":989,"props":2038,"children":2039},{"style":1119},[2040],{"type":62,"value":2041},"{",{"type":57,"tag":989,"props":2043,"children":2044},{"style":1119},[2045],{"type":62,"value":1150},{"type":57,"tag":989,"props":2047,"children":2048},{"style":1133},[2049],{"type":62,"value":2050},"evaluationId",{"type":57,"tag":989,"props":2052,"children":2053},{"style":1119},[2054],{"type":62,"value":1141},{"type":57,"tag":989,"props":2056,"children":2057},{"style":1119},[2058],{"type":62,"value":621},{"type":57,"tag":989,"props":2060,"children":2061},{"style":1119},[2062],{"type":62,"value":1150},{"type":57,"tag":989,"props":2064,"children":2065},{"style":1153},[2066],{"type":62,"value":2067},"\u003Cuuid>",{"type":57,"tag":989,"props":2069,"children":2070},{"style":1119},[2071],{"type":62,"value":1141},{"type":57,"tag":989,"props":2073,"children":2074},{"style":1119},[2075],{"type":62,"value":1584},{"type":57,"tag":989,"props":2077,"children":2078},{"style":1119},[2079],{"type":62,"value":1150},{"type":57,"tag":989,"props":2081,"children":2082},{"style":1133},[2083],{"type":62,"value":1794},{"type":57,"tag":989,"props":2085,"children":2086},{"style":1119},[2087],{"type":62,"value":1141},{"type":57,"tag":989,"props":2089,"children":2090},{"style":1119},[2091],{"type":62,"value":621},{"type":57,"tag":989,"props":2093,"children":2094},{"style":1119},[2095],{"type":62,"value":2096}," true",{"type":57,"tag":989,"props":2098,"children":2099},{"style":1119},[2100],{"type":62,"value":1772},{"type":57,"tag":65,"props":2102,"children":2103},{},[2104,2106,2112],{"type":62,"value":2105},"It now runs on every new matching generation, or once per matching trace for a trace target. This isn't\none-and-done: the user should be aware that they need to keep an eye on results and iterate if the outcome\nis not the expected one. To wire results into a Slack feed, see ",{"type":57,"tag":79,"props":2107,"children":2109},{"className":2108},[],[2110],{"type":62,"value":2111},"feature-usage-feed",{"type":62,"value":125},{"type":57,"tag":127,"props":2114,"children":2116},{"id":2115},"scoping-with-conditions",[2117],{"type":62,"value":2118},"Scoping with conditions",{"type":57,"tag":65,"props":2120,"children":2121},{},[2122,2127,2129,2134,2136,2146,2148,2154,2156,2161,2163,2168,2169,2174,2175,2180,2181,2186],{"type":57,"tag":79,"props":2123,"children":2125},{"className":2124},[],[2126],{"type":62,"value":346},{"type":62,"value":2128}," is a ",{"type":57,"tag":71,"props":2130,"children":2131},{},[2132],{"type":62,"value":2133},"list",{"type":62,"value":2135}," of condition sets β€” ",{"type":57,"tag":71,"props":2137,"children":2138},{},[2139,2141],{"type":62,"value":2140},"OR between sets, AND within a set's ",{"type":57,"tag":79,"props":2142,"children":2144},{"className":2143},[],[2145],{"type":62,"value":839},{"type":62,"value":2147},". Each\nset is ",{"type":57,"tag":79,"props":2149,"children":2151},{"className":2150},[],[2152],{"type":62,"value":2153},"{ id, rollout_percentage, properties[] }",{"type":62,"value":2155},". There is no time window inside conditions; sampling is\nonly ",{"type":57,"tag":79,"props":2157,"children":2159},{"className":2158},[],[2160],{"type":62,"value":1593},{"type":62,"value":2162}," (0–100). Property filters use the standard PostHog shape\n(",{"type":57,"tag":79,"props":2164,"children":2166},{"className":2165},[],[2167],{"type":62,"value":1641},{"type":62,"value":743},{"type":57,"tag":79,"props":2170,"children":2172},{"className":2171},[],[2173],{"type":62,"value":741},{"type":62,"value":743},{"type":57,"tag":79,"props":2176,"children":2178},{"className":2177},[],[2179],{"type":62,"value":1707},{"type":62,"value":743},{"type":57,"tag":79,"props":2182,"children":2184},{"className":2183},[],[2185],{"type":62,"value":1741},{"type":62,"value":2187},"). For trace targets, these filters still select the generation that\ntriggers the eventual whole-trace evaluation.",{"type":57,"tag":978,"props":2189,"children":2191},{"className":980,"code":2190,"language":982,"meta":983,"style":983},"\"conditions\": [\n { \"id\": \"openai\", \"rollout_percentage\": 100, \"properties\": [{\"key\": \"$ai_provider\", \"type\": \"event\", \"operator\": \"exact\", \"value\": \"openai\"}] },\n { \"id\": \"anthropic\", \"rollout_percentage\": 25, \"properties\": [{\"key\": \"$ai_provider\", \"type\": \"event\", \"operator\": \"exact\", \"value\": \"anthropic\"}] }\n]\n",[2192],{"type":57,"tag":79,"props":2193,"children":2194},{"__ignoreMap":983},[2195,2220,2441,2658],{"type":57,"tag":989,"props":2196,"children":2197},{"class":991,"line":992},[2198,2202,2206,2210,2215],{"type":57,"tag":989,"props":2199,"children":2200},{"style":1119},[2201],{"type":62,"value":1141},{"type":57,"tag":989,"props":2203,"children":2204},{"style":1153},[2205],{"type":62,"value":346},{"type":57,"tag":989,"props":2207,"children":2208},{"style":1119},[2209],{"type":62,"value":1141},{"type":57,"tag":989,"props":2211,"children":2212},{"style":996},[2213],{"type":62,"value":2214},": ",{"type":57,"tag":989,"props":2216,"children":2217},{"style":1119},[2218],{"type":62,"value":2219},"[\n",{"type":57,"tag":989,"props":2221,"children":2222},{"class":991,"line":1008},[2223,2228,2232,2236,2240,2244,2248,2253,2257,2261,2266,2270,2274,2278,2282,2286,2290,2294,2298,2302,2306,2310,2314,2318,2322,2326,2331,2335,2339,2343,2347,2351,2355,2359,2363,2367,2371,2375,2379,2383,2387,2391,2396,2400,2404,2408,2412,2416,2420,2424,2428,2432,2437],{"type":57,"tag":989,"props":2224,"children":2225},{"style":1119},[2226],{"type":62,"value":2227}," {",{"type":57,"tag":989,"props":2229,"children":2230},{"style":1119},[2231],{"type":62,"value":1150},{"type":57,"tag":989,"props":2233,"children":2234},{"style":1133},[2235],{"type":62,"value":749},{"type":57,"tag":989,"props":2237,"children":2238},{"style":1119},[2239],{"type":62,"value":1141},{"type":57,"tag":989,"props":2241,"children":2242},{"style":1119},[2243],{"type":62,"value":621},{"type":57,"tag":989,"props":2245,"children":2246},{"style":1119},[2247],{"type":62,"value":1150},{"type":57,"tag":989,"props":2249,"children":2250},{"style":1153},[2251],{"type":62,"value":2252},"openai",{"type":57,"tag":989,"props":2254,"children":2255},{"style":1119},[2256],{"type":62,"value":1141},{"type":57,"tag":989,"props":2258,"children":2259},{"style":1119},[2260],{"type":62,"value":1584},{"type":57,"tag":989,"props":2262,"children":2263},{"style":1119},[2264],{"type":62,"value":2265}," \"",{"type":57,"tag":989,"props":2267,"children":2268},{"style":1133},[2269],{"type":62,"value":1593},{"type":57,"tag":989,"props":2271,"children":2272},{"style":1119},[2273],{"type":62,"value":1141},{"type":57,"tag":989,"props":2275,"children":2276},{"style":1119},[2277],{"type":62,"value":621},{"type":57,"tag":989,"props":2279,"children":2280},{"style":1604},[2281],{"type":62,"value":1607},{"type":57,"tag":989,"props":2283,"children":2284},{"style":1119},[2285],{"type":62,"value":1584},{"type":57,"tag":989,"props":2287,"children":2288},{"style":1119},[2289],{"type":62,"value":1150},{"type":57,"tag":989,"props":2291,"children":2292},{"style":1133},[2293],{"type":62,"value":839},{"type":57,"tag":989,"props":2295,"children":2296},{"style":1119},[2297],{"type":62,"value":1141},{"type":57,"tag":989,"props":2299,"children":2300},{"style":1119},[2301],{"type":62,"value":621},{"type":57,"tag":989,"props":2303,"children":2304},{"style":1119},[2305],{"type":62,"value":1632},{"type":57,"tag":989,"props":2307,"children":2308},{"style":1119},[2309],{"type":62,"value":1141},{"type":57,"tag":989,"props":2311,"children":2312},{"style":1273},[2313],{"type":62,"value":1641},{"type":57,"tag":989,"props":2315,"children":2316},{"style":1119},[2317],{"type":62,"value":1141},{"type":57,"tag":989,"props":2319,"children":2320},{"style":1119},[2321],{"type":62,"value":621},{"type":57,"tag":989,"props":2323,"children":2324},{"style":1119},[2325],{"type":62,"value":1150},{"type":57,"tag":989,"props":2327,"children":2328},{"style":1153},[2329],{"type":62,"value":2330},"$ai_provider",{"type":57,"tag":989,"props":2332,"children":2333},{"style":1119},[2334],{"type":62,"value":1141},{"type":57,"tag":989,"props":2336,"children":2337},{"style":1119},[2338],{"type":62,"value":1584},{"type":57,"tag":989,"props":2340,"children":2341},{"style":1119},[2342],{"type":62,"value":1150},{"type":57,"tag":989,"props":2344,"children":2345},{"style":1273},[2346],{"type":62,"value":741},{"type":57,"tag":989,"props":2348,"children":2349},{"style":1119},[2350],{"type":62,"value":1141},{"type":57,"tag":989,"props":2352,"children":2353},{"style":1119},[2354],{"type":62,"value":621},{"type":57,"tag":989,"props":2356,"children":2357},{"style":1119},[2358],{"type":62,"value":1150},{"type":57,"tag":989,"props":2360,"children":2361},{"style":1153},[2362],{"type":62,"value":846},{"type":57,"tag":989,"props":2364,"children":2365},{"style":1119},[2366],{"type":62,"value":1141},{"type":57,"tag":989,"props":2368,"children":2369},{"style":1119},[2370],{"type":62,"value":1584},{"type":57,"tag":989,"props":2372,"children":2373},{"style":1119},[2374],{"type":62,"value":1150},{"type":57,"tag":989,"props":2376,"children":2377},{"style":1273},[2378],{"type":62,"value":1707},{"type":57,"tag":989,"props":2380,"children":2381},{"style":1119},[2382],{"type":62,"value":1141},{"type":57,"tag":989,"props":2384,"children":2385},{"style":1119},[2386],{"type":62,"value":621},{"type":57,"tag":989,"props":2388,"children":2389},{"style":1119},[2390],{"type":62,"value":1150},{"type":57,"tag":989,"props":2392,"children":2393},{"style":1153},[2394],{"type":62,"value":2395},"exact",{"type":57,"tag":989,"props":2397,"children":2398},{"style":1119},[2399],{"type":62,"value":1141},{"type":57,"tag":989,"props":2401,"children":2402},{"style":1119},[2403],{"type":62,"value":1584},{"type":57,"tag":989,"props":2405,"children":2406},{"style":1119},[2407],{"type":62,"value":1150},{"type":57,"tag":989,"props":2409,"children":2410},{"style":1273},[2411],{"type":62,"value":1741},{"type":57,"tag":989,"props":2413,"children":2414},{"style":1119},[2415],{"type":62,"value":1141},{"type":57,"tag":989,"props":2417,"children":2418},{"style":1119},[2419],{"type":62,"value":621},{"type":57,"tag":989,"props":2421,"children":2422},{"style":1119},[2423],{"type":62,"value":1150},{"type":57,"tag":989,"props":2425,"children":2426},{"style":1153},[2427],{"type":62,"value":2252},{"type":57,"tag":989,"props":2429,"children":2430},{"style":1119},[2431],{"type":62,"value":1141},{"type":57,"tag":989,"props":2433,"children":2434},{"style":1119},[2435],{"type":62,"value":2436},"}]",{"type":57,"tag":989,"props":2438,"children":2439},{"style":1119},[2440],{"type":62,"value":1366},{"type":57,"tag":989,"props":2442,"children":2443},{"class":991,"line":1022},[2444,2448,2452,2456,2460,2464,2468,2473,2477,2481,2485,2489,2493,2497,2502,2506,2510,2514,2518,2522,2526,2530,2534,2538,2542,2546,2550,2554,2558,2562,2566,2570,2574,2578,2582,2586,2590,2594,2598,2602,2606,2610,2614,2618,2622,2626,2630,2634,2638,2642,2646,2650,2654],{"type":57,"tag":989,"props":2445,"children":2446},{"style":1119},[2447],{"type":62,"value":2227},{"type":57,"tag":989,"props":2449,"children":2450},{"style":1119},[2451],{"type":62,"value":1150},{"type":57,"tag":989,"props":2453,"children":2454},{"style":1133},[2455],{"type":62,"value":749},{"type":57,"tag":989,"props":2457,"children":2458},{"style":1119},[2459],{"type":62,"value":1141},{"type":57,"tag":989,"props":2461,"children":2462},{"style":1119},[2463],{"type":62,"value":621},{"type":57,"tag":989,"props":2465,"children":2466},{"style":1119},[2467],{"type":62,"value":1150},{"type":57,"tag":989,"props":2469,"children":2470},{"style":1153},[2471],{"type":62,"value":2472},"anthropic",{"type":57,"tag":989,"props":2474,"children":2475},{"style":1119},[2476],{"type":62,"value":1141},{"type":57,"tag":989,"props":2478,"children":2479},{"style":1119},[2480],{"type":62,"value":1584},{"type":57,"tag":989,"props":2482,"children":2483},{"style":1119},[2484],{"type":62,"value":1150},{"type":57,"tag":989,"props":2486,"children":2487},{"style":1133},[2488],{"type":62,"value":1593},{"type":57,"tag":989,"props":2490,"children":2491},{"style":1119},[2492],{"type":62,"value":1141},{"type":57,"tag":989,"props":2494,"children":2495},{"style":1119},[2496],{"type":62,"value":621},{"type":57,"tag":989,"props":2498,"children":2499},{"style":1604},[2500],{"type":62,"value":2501}," 25",{"type":57,"tag":989,"props":2503,"children":2504},{"style":1119},[2505],{"type":62,"value":1584},{"type":57,"tag":989,"props":2507,"children":2508},{"style":1119},[2509],{"type":62,"value":1130},{"type":57,"tag":989,"props":2511,"children":2512},{"style":1133},[2513],{"type":62,"value":839},{"type":57,"tag":989,"props":2515,"children":2516},{"style":1119},[2517],{"type":62,"value":1141},{"type":57,"tag":989,"props":2519,"children":2520},{"style":1119},[2521],{"type":62,"value":621},{"type":57,"tag":989,"props":2523,"children":2524},{"style":1119},[2525],{"type":62,"value":1632},{"type":57,"tag":989,"props":2527,"children":2528},{"style":1119},[2529],{"type":62,"value":1141},{"type":57,"tag":989,"props":2531,"children":2532},{"style":1273},[2533],{"type":62,"value":1641},{"type":57,"tag":989,"props":2535,"children":2536},{"style":1119},[2537],{"type":62,"value":1141},{"type":57,"tag":989,"props":2539,"children":2540},{"style":1119},[2541],{"type":62,"value":621},{"type":57,"tag":989,"props":2543,"children":2544},{"style":1119},[2545],{"type":62,"value":1150},{"type":57,"tag":989,"props":2547,"children":2548},{"style":1153},[2549],{"type":62,"value":2330},{"type":57,"tag":989,"props":2551,"children":2552},{"style":1119},[2553],{"type":62,"value":1141},{"type":57,"tag":989,"props":2555,"children":2556},{"style":1119},[2557],{"type":62,"value":1584},{"type":57,"tag":989,"props":2559,"children":2560},{"style":1119},[2561],{"type":62,"value":1150},{"type":57,"tag":989,"props":2563,"children":2564},{"style":1273},[2565],{"type":62,"value":741},{"type":57,"tag":989,"props":2567,"children":2568},{"style":1119},[2569],{"type":62,"value":1141},{"type":57,"tag":989,"props":2571,"children":2572},{"style":1119},[2573],{"type":62,"value":621},{"type":57,"tag":989,"props":2575,"children":2576},{"style":1119},[2577],{"type":62,"value":1150},{"type":57,"tag":989,"props":2579,"children":2580},{"style":1153},[2581],{"type":62,"value":846},{"type":57,"tag":989,"props":2583,"children":2584},{"style":1119},[2585],{"type":62,"value":1141},{"type":57,"tag":989,"props":2587,"children":2588},{"style":1119},[2589],{"type":62,"value":1584},{"type":57,"tag":989,"props":2591,"children":2592},{"style":1119},[2593],{"type":62,"value":1150},{"type":57,"tag":989,"props":2595,"children":2596},{"style":1273},[2597],{"type":62,"value":1707},{"type":57,"tag":989,"props":2599,"children":2600},{"style":1119},[2601],{"type":62,"value":1141},{"type":57,"tag":989,"props":2603,"children":2604},{"style":1119},[2605],{"type":62,"value":621},{"type":57,"tag":989,"props":2607,"children":2608},{"style":1119},[2609],{"type":62,"value":1150},{"type":57,"tag":989,"props":2611,"children":2612},{"style":1153},[2613],{"type":62,"value":2395},{"type":57,"tag":989,"props":2615,"children":2616},{"style":1119},[2617],{"type":62,"value":1141},{"type":57,"tag":989,"props":2619,"children":2620},{"style":1119},[2621],{"type":62,"value":1584},{"type":57,"tag":989,"props":2623,"children":2624},{"style":1119},[2625],{"type":62,"value":1150},{"type":57,"tag":989,"props":2627,"children":2628},{"style":1273},[2629],{"type":62,"value":1741},{"type":57,"tag":989,"props":2631,"children":2632},{"style":1119},[2633],{"type":62,"value":1141},{"type":57,"tag":989,"props":2635,"children":2636},{"style":1119},[2637],{"type":62,"value":621},{"type":57,"tag":989,"props":2639,"children":2640},{"style":1119},[2641],{"type":62,"value":1150},{"type":57,"tag":989,"props":2643,"children":2644},{"style":1153},[2645],{"type":62,"value":2472},{"type":57,"tag":989,"props":2647,"children":2648},{"style":1119},[2649],{"type":62,"value":1141},{"type":57,"tag":989,"props":2651,"children":2652},{"style":1119},[2653],{"type":62,"value":2436},{"type":57,"tag":989,"props":2655,"children":2656},{"style":1119},[2657],{"type":62,"value":1772},{"type":57,"tag":989,"props":2659,"children":2660},{"class":991,"line":1168},[2661],{"type":57,"tag":989,"props":2662,"children":2663},{"style":1119},[2664],{"type":62,"value":2665},"]\n",{"type":57,"tag":127,"props":2667,"children":2669},{"id":2668},"constructing-ui-links",[2670],{"type":62,"value":2671},"Constructing UI links",{"type":57,"tag":65,"props":2673,"children":2674},{},[2675,2677,2682,2684,2690,2692,2698,2700,2706],{"type":62,"value":2676},"Build links with ",{"type":57,"tag":79,"props":2678,"children":2680},{"className":2679},[],[2681],{"type":62,"value":330},{"type":62,"value":2683}," β€” never hand-write the host or the ",{"type":57,"tag":79,"props":2685,"children":2687},{"className":2686},[],[2688],{"type":62,"value":2689},"\u002Fproject\u002F\u003Cid>\u002F",{"type":62,"value":2691}," prefix.\nThe ",{"type":57,"tag":79,"props":2693,"children":2695},{"className":2694},[],[2696],{"type":62,"value":2697},"url",{"type":62,"value":2699}," must be a canonical catalog template; pass concrete ids via ",{"type":57,"tag":79,"props":2701,"children":2703},{"className":2702},[],[2704],{"type":62,"value":2705},"params",{"type":62,"value":2707},", never inline them into the path.",{"type":57,"tag":375,"props":2709,"children":2710},{},[2711,2727],{"type":57,"tag":379,"props":2712,"children":2713},{},[2714,2719,2721],{"type":57,"tag":71,"props":2715,"children":2716},{},[2717],{"type":62,"value":2718},"Evaluations list:",{"type":62,"value":2720}," ",{"type":57,"tag":79,"props":2722,"children":2724},{"className":2723},[],[2725],{"type":62,"value":2726},"generate-app-url {url: \"\u002Fai-evals\u002Fevaluations\"}",{"type":57,"tag":379,"props":2728,"children":2729},{},[2730,2735,2736],{"type":57,"tag":71,"props":2731,"children":2732},{},[2733],{"type":62,"value":2734},"Single evaluation:",{"type":62,"value":2720},{"type":57,"tag":79,"props":2737,"children":2739},{"className":2738},[],[2740],{"type":62,"value":2741},"generate-app-url {url: \"\u002Fai-evals\u002Fevaluations\u002F{id}\", params: {id: \"\u003Cevaluation_id>\"}}",{"type":57,"tag":65,"props":2743,"children":2744},{},[2745,2747,2753],{"type":62,"value":2746},"These resolve to the correct region host and project prefix (e.g.\n",{"type":57,"tag":79,"props":2748,"children":2750},{"className":2749},[],[2751],{"type":62,"value":2752},"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cid>\u002Fai-evals\u002Fevaluations\u002F\u003Cevaluation_id>",{"type":62,"value":2754},"). Surface the link after\ncreating so the user can review and toggle it in the UI.",{"type":57,"tag":127,"props":2756,"children":2758},{"id":2757},"tips",[2759],{"type":62,"value":2760},"Tips",{"type":57,"tag":375,"props":2762,"children":2763},{},[2764,2781,2791,2801,2823,2833,2857,2880],{"type":57,"tag":379,"props":2765,"children":2766},{},[2767,2772,2774,2779],{"type":57,"tag":71,"props":2768,"children":2769},{},[2770],{"type":62,"value":2771},"Evals come from real failures, not generic metrics.",{"type":62,"value":2773}," Start from a failure found in this product's\ntraffic (via ",{"type":57,"tag":79,"props":2775,"children":2777},{"className":2776},[],[2778],{"type":62,"value":102},{"type":62,"value":2780},"), not from \"let's measure hallucination\". A metric nobody traced\nback to a real bad output is noise.",{"type":57,"tag":379,"props":2782,"children":2783},{},[2784,2789],{"type":57,"tag":71,"props":2785,"children":2786},{},[2787],{"type":62,"value":2788},"One eval, one failure mode.",{"type":62,"value":2790}," Different failure modes need different evals; don't make one eval try to\ncatch everything.",{"type":57,"tag":379,"props":2792,"children":2793},{},[2794,2799],{"type":57,"tag":71,"props":2795,"children":2796},{},[2797],{"type":62,"value":2798},"Suggest changes along with the eval if possible.",{"type":62,"value":2800}," If it's clear a prompt change would fix the issue, for\ninstance, set up the eval but also suggest to the user they change the prompt: they should soon see the eval\ngo from low pass rate to a higher pass rate.",{"type":57,"tag":379,"props":2802,"children":2803},{},[2804,2814,2816,2821],{"type":57,"tag":71,"props":2805,"children":2806},{},[2807,2812],{"type":57,"tag":79,"props":2808,"children":2810},{"className":2809},[],[2811],{"type":62,"value":468},{"type":62,"value":2813}," first.",{"type":62,"value":2815}," No provider key, no AI approval, deterministic. Reach for ",{"type":57,"tag":79,"props":2817,"children":2819},{"className":2818},[],[2820],{"type":62,"value":490},{"type":62,"value":2822}," only when the\ncriterion genuinely can't be coded.",{"type":57,"tag":379,"props":2824,"children":2825},{},[2826,2831],{"type":57,"tag":71,"props":2827,"children":2828},{},[2829],{"type":62,"value":2830},"Always create disabled, verify scope, then enable.",{"type":62,"value":2832}," An eval firing on the wrong events is worse than\nnone β€” noise, and (for llm_judge) cost.",{"type":57,"tag":379,"props":2834,"children":2835},{},[2836,2841,2843,2848,2850,2855],{"type":57,"tag":71,"props":2837,"children":2838},{},[2839],{"type":62,"value":2840},"Configure llm_judge credentials before running.",{"type":62,"value":2842}," A judge needs a valid provider and model plus a usable\nprovider key. ",{"type":57,"tag":79,"props":2844,"children":2846},{"className":2845},[],[2847],{"type":62,"value":914},{"type":62,"value":2849}," may be ",{"type":57,"tag":79,"props":2851,"children":2853},{"className":2852},[],[2854],{"type":62,"value":955},{"type":62,"value":2856}," only when the matching active team key can be used.",{"type":57,"tag":379,"props":2858,"children":2859},{},[2860,2871,2873,2879],{"type":57,"tag":71,"props":2861,"children":2862},{},[2863,2869],{"type":57,"tag":79,"props":2864,"children":2866},{"className":2865},[],[2867],{"type":62,"value":2868},"bytecode",{"type":62,"value":2870}," is server-written",{"type":62,"value":2872}," for hog evals β€” never pass it; send only ",{"type":57,"tag":79,"props":2874,"children":2876},{"className":2875},[],[2877],{"type":62,"value":2878},"evaluation_config.source",{"type":62,"value":125},{"type":57,"tag":379,"props":2881,"children":2882},{},[2883,2885,2891,2893,2898],{"type":62,"value":2884},"For cluster-scoped evals, identify the cluster with ",{"type":57,"tag":79,"props":2886,"children":2888},{"className":2887},[],[2889],{"type":62,"value":2890},"exploring-llm-clusters",{"type":62,"value":2892},", then translate its event\nfilter into ",{"type":57,"tag":79,"props":2894,"children":2896},{"className":2895},[],[2897],{"type":62,"value":346},{"type":62,"value":125},{"type":57,"tag":2900,"props":2901,"children":2902},"style",{},[2903],{"type":62,"value":2904},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2906,"total":3005},[2907,2920,2932,2944,2957,2972,2988],{"slug":2908,"name":2908,"fn":2909,"description":2910,"org":2911,"tags":2912,"stars":26,"repoUrl":27,"updatedAt":2919},"analyzing-expensive-users","analyze expensive users in AI observability","Analyze the most expensive users in AI observability and explain why they cost so much. Use when the user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability spend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2913,2914,2917,2918],{"name":24,"slug":25,"type":15},{"name":2915,"slug":2916,"type":15},"Cost Optimization","cost-optimization",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-28T05:34:11.117757",{"slug":2921,"name":2921,"fn":2922,"description":2923,"org":2924,"tags":2925,"stars":26,"repoUrl":27,"updatedAt":2931},"auditing-endpoints","audit PostHog project endpoints","Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised versions. Use when the user asks \"what endpoints can I clean up?\", \"are any of my endpoints broken?\", \"which materialised versions are still being called?\", or wants a one-shot cleanup pass over the Endpoints product. Produces a prioritised report grouped by issue type, with recommended actions but does not modify anything without explicit confirmation.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2926,2927,2930],{"name":24,"slug":25,"type":15},{"name":2928,"slug":2929,"type":15},"Audit","audit",{"name":9,"slug":8,"type":15},"2026-06-08T08:08:33.693989",{"slug":2933,"name":2933,"fn":2934,"description":2935,"org":2936,"tags":2937,"stars":26,"repoUrl":27,"updatedAt":2943},"auditing-warehouse-source-health","audit PostHog data warehouse source health","Audit the health of a PostHog project's data warehouse sources and syncs β€” find every broken or degraded source connection, sync schema, and webhook channel. Use when the user asks \"why are my imports failing?\", \"what's broken with my sources?\", \"why is my warehouse data stale?\", or wants a one-shot triage of source\u002Fsync health before deciding where to dig in. Produces a prioritized report grouped by severity, with recommended next steps. For materialized-view health use `auditing-warehouse-view-health`; for a single failing sync use `diagnosing-failed-warehouse-syncs`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2938,2939,2941,2942],{"name":2928,"slug":2929,"type":15},{"name":2940,"slug":35,"type":15},"Data Warehouse",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-06-18T08:22:57.67984",{"slug":2945,"name":2945,"fn":2946,"description":2947,"org":2948,"tags":2949,"stars":26,"repoUrl":27,"updatedAt":2956},"auditing-warehouse-view-health","audit PostHog materialized view health","Audit the health of a PostHog project's materialized views (saved queries) β€” find every failed materialization and flag unused or stale materialized views that cost storage and compute. Use when the user asks \"which of my views are broken?\", \"why is this materialized view failing?\", \"are any of my views wasting compute?\", or wants a one-shot triage of view health. For source\u002Fsync health use `auditing-warehouse-source-health`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2950,2951,2952,2955],{"name":2928,"slug":2929,"type":15},{"name":2940,"slug":35,"type":15},{"name":2953,"slug":2954,"type":15},"Performance","performance",{"name":9,"slug":8,"type":15},"2026-06-18T08:25:10.936787",{"slug":2958,"name":2958,"fn":2959,"description":2960,"org":2961,"tags":2962,"stars":26,"repoUrl":27,"updatedAt":2971},"authoring-error-tracking-alerts","author PostHog error tracking alerts","Author error tracking alerts that fire when an issue is created, reopened, or starts spiking. Use when the user asks to set up error notifications, route exceptions to Slack\u002Fwebhook\u002FLinear, or evaluate which error events are worth alerting on. Covers trigger-event selection, integration choice, dedup against existing alerts, and shipping with the canonical message body shape.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2963,2966,2969,2970],{"name":2964,"slug":2965,"type":15},"Alerting","alerting",{"name":2967,"slug":2968,"type":15},"Debugging","debugging",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-06-18T08:24:40.318583",{"slug":2973,"name":2973,"fn":2974,"description":2975,"org":2976,"tags":2977,"stars":26,"repoUrl":27,"updatedAt":2987},"authoring-log-alerts","author log alerts in PostHog","Author useful, low-noise log alerts on services in a PostHog project. Use when the user asks to set up alerts for their logs, suggest alerts they should add, or evaluate whether a service is worth monitoring. Covers service triage, baseline characterisation, threshold drafting, back-testing via simulate, and shipping with a notification destination.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2978,2979,2982,2983,2986],{"name":24,"slug":25,"type":15},{"name":2980,"slug":2981,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},{"name":2984,"slug":2985,"type":15},"Operations","operations",{"name":9,"slug":8,"type":15},"2026-07-18T05:10:54.430898",{"slug":2989,"name":2989,"fn":2990,"description":2991,"org":2992,"tags":2993,"stars":26,"repoUrl":27,"updatedAt":3004},"building-workflows","build and edit PostHog workflows","Build, edit, test, enable, and monitor PostHog workflows over MCP. Author the action\u002Fedge graph so it runs and opens cleanly in the visual editor, then change drafts surgically with patch operations. Use when asked to build, set up, automate, change, fix, or debug a workflow, campaign, broadcast, drip sequence, or event-triggered automation in the workflows product.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2994,2997,3000,3001],{"name":2995,"slug":2996,"type":15},"Automation","automation",{"name":2998,"slug":2999,"type":15},"MCP","mcp",{"name":9,"slug":8,"type":15},{"name":3002,"slug":3003,"type":15},"Workflow Automation","workflow-automation","2026-07-28T05:34:12.167015",61,{"items":3007,"total":3124},[3008,3015,3021,3028,3035,3042,3050,3057,3071,3086,3096,3114],{"slug":2908,"name":2908,"fn":2909,"description":2910,"org":3009,"tags":3010,"stars":26,"repoUrl":27,"updatedAt":2919},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3011,3012,3013,3014],{"name":24,"slug":25,"type":15},{"name":2915,"slug":2916,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"slug":2921,"name":2921,"fn":2922,"description":2923,"org":3016,"tags":3017,"stars":26,"repoUrl":27,"updatedAt":2931},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3018,3019,3020],{"name":24,"slug":25,"type":15},{"name":2928,"slug":2929,"type":15},{"name":9,"slug":8,"type":15},{"slug":2933,"name":2933,"fn":2934,"description":2935,"org":3022,"tags":3023,"stars":26,"repoUrl":27,"updatedAt":2943},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3024,3025,3026,3027],{"name":2928,"slug":2929,"type":15},{"name":2940,"slug":35,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"slug":2945,"name":2945,"fn":2946,"description":2947,"org":3029,"tags":3030,"stars":26,"repoUrl":27,"updatedAt":2956},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3031,3032,3033,3034],{"name":2928,"slug":2929,"type":15},{"name":2940,"slug":35,"type":15},{"name":2953,"slug":2954,"type":15},{"name":9,"slug":8,"type":15},{"slug":2958,"name":2958,"fn":2959,"description":2960,"org":3036,"tags":3037,"stars":26,"repoUrl":27,"updatedAt":2971},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3038,3039,3040,3041],{"name":2964,"slug":2965,"type":15},{"name":2967,"slug":2968,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"slug":2973,"name":2973,"fn":2974,"description":2975,"org":3043,"tags":3044,"stars":26,"repoUrl":27,"updatedAt":2987},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3045,3046,3047,3048,3049],{"name":24,"slug":25,"type":15},{"name":2980,"slug":2981,"type":15},{"name":13,"slug":14,"type":15},{"name":2984,"slug":2985,"type":15},{"name":9,"slug":8,"type":15},{"slug":2989,"name":2989,"fn":2990,"description":2991,"org":3051,"tags":3052,"stars":26,"repoUrl":27,"updatedAt":3004},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3053,3054,3055,3056],{"name":2995,"slug":2996,"type":15},{"name":2998,"slug":2999,"type":15},{"name":9,"slug":8,"type":15},{"name":3002,"slug":3003,"type":15},{"slug":3058,"name":3058,"fn":3059,"description":3060,"org":3061,"tags":3062,"stars":26,"repoUrl":27,"updatedAt":3070},"check-posthog-loading","inspect PostHog SDK loading across URLs","Inspect how the PostHog JavaScript SDK is loaded across a list of URLs. Use to confirm consistent installation across pages, find pages missing the snippet, detect mismatched API keys or hosts between pages, and verify the load method (head snippet vs deferred vs array.js).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3063,3064,3065,3068,3069],{"name":24,"slug":25,"type":15},{"name":2967,"slug":2968,"type":15},{"name":3066,"slug":3067,"type":15},"Frontend","frontend",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-05-07T05:56:19.828048",{"slug":3072,"name":3072,"fn":3073,"description":3074,"org":3075,"tags":3076,"stars":26,"repoUrl":27,"updatedAt":3085},"consuming-endpoints-from-client-code","integrate PostHog endpoints into client applications","Wire a PostHog endpoint into a client app or SDK. Covers fetching the OpenAPI spec, generating a typed client with openapi-generator or @hey-api\u002Fopenapi-ts, sending the right auth header, shaping the variables payload (HogQL code_name vs insight breakdown property), handling rate-limit and materialised-endpoint error responses. Use when the user says \"how do I call my endpoint\", \"generate a client for this\", or \"what auth header do I use\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3077,3080,3081,3082],{"name":3078,"slug":3079,"type":15},"API Development","api-development",{"name":3066,"slug":3067,"type":15},{"name":9,"slug":8,"type":15},{"name":3083,"slug":3084,"type":15},"SDK","sdk","2026-06-08T08:08:34.929454",{"slug":3087,"name":3087,"fn":3088,"description":3089,"org":3090,"tags":3091,"stars":26,"repoUrl":27,"updatedAt":3095},"copying-endpoints-across-projects","copy PostHog endpoints across projects","Copy a PostHog endpoint (a saved HogQL\u002Finsight query exposed as an API route) to another project in the same organization, or duplicate it under a new name in the same project. Use when the user wants to duplicate an endpoint, promote an endpoint from staging to production, replicate an endpoint's query\u002Fvariables\u002Ffreshness config in another workspace, or clone an endpoint to iterate on it. Unlike feature flags and experiments, endpoints have NO native cross-project copy tool β€” this skill covers the read-then-recreate flow (endpoint-get then endpoint-create), the active-project switching it requires, name-collision checks, and the safe defaults (land unmaterialised in the target, verify with endpoint-run). Does not cover editing endpoint versions (see managing-endpoint-versions) or authoring a brand-new endpoint from scratch (see creating-an-endpoint).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3092,3093,3094],{"name":3078,"slug":3079,"type":15},{"name":2984,"slug":2985,"type":15},{"name":9,"slug":8,"type":15},"2026-07-15T05:29:58.442727",{"slug":3097,"name":3097,"fn":3098,"description":3099,"org":3100,"tags":3101,"stars":26,"repoUrl":27,"updatedAt":3113},"creating-ai-subscription","schedule recurring AI-generated PostHog reports","Create a recurring AI-generated PostHog report β€” schedule a free-text prompt to run on a cron, with the LLM-synthesized markdown delivered to email or Slack on each tick. Use when the user wants a recurring AI summary of X on any cadence (daily, weekly, monthly, yearly) rather than a one-off report. (To attach an AI summary to an existing insight\u002Fdashboard subscription instead of a free-text prompt, see `managing-subscriptions` and its `summary_enabled` option.)\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3102,3103,3106,3107,3110],{"name":2995,"slug":2996,"type":15},{"name":3104,"slug":3105,"type":15},"Email","email",{"name":9,"slug":8,"type":15},{"name":3108,"slug":3109,"type":15},"Reporting","reporting",{"name":3111,"slug":3112,"type":15},"Slack","slack","2026-06-09T07:32:27.935712",{"slug":3115,"name":3115,"fn":3116,"description":3117,"org":3118,"tags":3119,"stars":26,"repoUrl":27,"updatedAt":3123},"creating-an-endpoint","create PostHog API endpoints","Create a PostHog endpoint with the right shape on the first try β€” covers query kind choice, name conventions, what to expose as variables (HogQL code_name vs insight breakdown), data_freshness_seconds, and whether to materialise on day one. Use when the user says \"create an endpoint\", \"expose this query as an API\", \"turn this insight into an endpoint\", or asks for help structuring a new endpoint. Steers away from common mistakes: materialising a query with cohort breakdowns or compare mode, inline-only variables on a materialised endpoint, unbounded date ranges, ambiguous names.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3120,3121,3122],{"name":24,"slug":25,"type":15},{"name":3078,"slug":3079,"type":15},{"name":9,"slug":8,"type":15},"2026-06-08T08:08:29.624498",231]