[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-posthog-exploring-llm-evaluations":3,"mdc-g4atln-key":34,"related-repo-posthog-exploring-llm-evaluations":3882,"related-org-posthog-exploring-llm-evaluations":3995},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"exploring-llm-evaluations","analyze and run LLM evaluations in PostHog","Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass\u002Ffail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the evaluation lifecycle (create, update, enable\u002Fdisable, delete).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"posthog","PostHog","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fposthog.png",[12,14,17,20],{"name":9,"slug":8,"type":13},"tag",{"name":15,"slug":16,"type":13},"LLM","llm",{"name":18,"slug":19,"type":13},"Evals","evals",{"name":21,"slug":22,"type":13},"Analytics","analytics",56,"https:\u002F\u002Fgithub.com\u002FPostHog\u002Fskills","2026-04-15T05:06:00.669501",null,4,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"PostHog skills (under construction)","https:\u002F\u002Fgithub.com\u002FPostHog\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fomnibus\u002Fexploring-llm-evaluations","---\nname: exploring-llm-evaluations\ndescription: >\n  Investigate AI observability evaluations of both types — `hog` (deterministic\n  code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations,\n  inspect their configuration, run them against specific generations, query\n  individual pass\u002Ffail results, and generate AI-powered summaries of patterns\n  across many runs. Use when the user asks to debug why an evaluation is\n  failing, surface common failure modes, compare results across filters,\n  dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the\n  evaluation lifecycle (create, update, enable\u002Fdisable, delete).\n---\n\n# Exploring LLM evaluations\n\nPostHog evaluations score `$ai_generation` events. Each evaluation is one of two types,\nboth first-class:\n\n- **`hog`** — deterministic Hog code that returns `true`\u002F`false` (and optionally N\u002FA).\n  Best for objective rule-based checks: format validation (JSON parses, schema matches),\n  length limits, keyword presence\u002Fabsence, regex patterns, structural assertions, latency\n  thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this\n  when the criterion can be expressed as code.\n- **`llm_judge`** — an LLM scores generations against a prompt you write. Best for\n  subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic\n  drift, instruction-following. Costs an LLM call per run and requires AI data\n  processing approval at the org level.\n\nResults from both types land in ClickHouse as `$ai_evaluation` events with the same\nschema, so the read\u002Fquery\u002Fsummary workflows are identical regardless of evaluator type —\nthe only thing that changes is whether `$ai_evaluation_reasoning` was written by Hog\ncode or by an LLM.\n\nThis skill covers the full lifecycle: list\u002Finspect\u002Fmanage evaluation configs (Hog or\nLLM judge), run them on specific generations, query individual results, and get an\nAI-generated summary of pass\u002Ffail\u002FN\u002FA patterns across many runs.\n\n## Tools\n\n| Tool                                     | Purpose                                                        |\n| ---------------------------------------- | -------------------------------------------------------------- |\n| `posthog:llma-evaluation-list`           | List\u002Fsearch evaluation configs (filter by name, enabled flag)  |\n| `posthog:llma-evaluation-get`            | Get a single evaluation config by UUID                         |\n| `posthog:llma-evaluation-create`         | Create a new `llm_judge` or `hog` evaluation                   |\n| `posthog:llma-evaluation-update`         | Update an existing evaluation (name, prompt, enabled, …)       |\n| `posthog:llma-evaluation-delete`         | Soft-delete an evaluation                                      |\n| `posthog:llma-evaluation-run`            | Run an evaluation against a specific `$ai_generation` event    |\n| `posthog:llma-evaluation-test-hog`       | Dry-run Hog source against recent generations (no save)        |\n| `posthog:llma-evaluation-summary-create` | AI-powered summary of pass\u002Ffail\u002FN\u002FA patterns across runs       |\n| `posthog:execute-sql`                    | Ad-hoc HogQL over `$ai_evaluation` events                      |\n| `posthog:query-llm-trace`                | Drill into the underlying generation that an evaluation scored |\n\nAll `llma-evaluation-*` tools are defined in `products\u002Fai_observability\u002Fmcp\u002Ftools.yaml`.\n\n## Event schema\n\nEvery run of an evaluation emits an `$ai_evaluation` event. Key properties:\n\n| Property                    | Meaning                                                  |\n| --------------------------- | -------------------------------------------------------- |\n| `$ai_evaluation_id`         | UUID of the evaluation config                            |\n| `$ai_evaluation_name`       | Human-readable name                                      |\n| `$ai_target_event_id`       | UUID of the `$ai_generation` event being scored          |\n| `$ai_trace_id`              | Parent trace ID (for jumping to the trace UI)            |\n| `$ai_evaluation_result`     | `true` = pass, `false` = fail                            |\n| `$ai_evaluation_reasoning`  | Free-text explanation (set by the LLM judge or Hog code) |\n| `$ai_evaluation_applicable` | `false` when the evaluator decided the generation is N\u002FA |\n\nWhen `$ai_evaluation_applicable = false`, the run counts as N\u002FA regardless of `$ai_evaluation_result`.\nFor evaluations that don't support N\u002FA, this property may be `null` — treat null as \"applicable\".\n\n## Workflow: investigate why an evaluation is failing\n\nWorks the same way for `llm_judge` and `hog` evaluations — the differences only matter\nwhen you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source).\n\n### Step 1 — Find the evaluation\n\n```json\nposthog:llma-evaluation-list\n{ \"search\": \"hallucination\", \"enabled\": true }\n```\n\nLook at the returned `id`, `name`, `evaluation_type`, and either:\n\n- `evaluation_config.prompt` for an `llm_judge`\n- `evaluation_config.source` for a `hog` evaluator\n\nThe Hog source is the ground truth for why a hog evaluator passes or fails — read it\nbefore assuming the failure is in the generation.\n\n### Step 2 — Get the AI-generated summary\n\n```json\nposthog:llma-evaluation-summary-create\n{\n  \"evaluation_id\": \"\u003Cuuid>\",\n  \"filter\": \"fail\"\n}\n```\n\nReturns:\n\n- `overall_assessment` — natural-language summary\n- `fail_patterns` — grouped patterns with `title`, `description`, `frequency`, and `example_generation_ids`\n- `pass_patterns` and `na_patterns` — same shape, populated when `filter` includes them\n- `recommendations` — actionable next steps\n- `statistics` — `total_analyzed`, `pass_count`, `fail_count`, `na_count`\n\nThe endpoint analyses the most recent ~250 runs (`EVALUATION_SUMMARY_MAX_RUNS`).\nResults are cached for one hour per `(evaluation_id, filter, set_of_generation_ids)`.\nPass `force_refresh: true` to recompute.\n\n**Compare filters in two calls** to spot what's distinctive about failures vs passes:\n\n```json\nposthog:llma-evaluation-summary-create\n{ \"evaluation_id\": \"\u003Cuuid>\", \"filter\": \"pass\" }\n```\n\nThen diff the `pass_patterns` against the `fail_patterns` from Step 2.\n\n### Step 3 — Drill into example failing runs\n\nEach pattern surfaces `example_generation_ids`. Pull the underlying trace for the most\nrepresentative example:\n\n```json\nposthog:query-llm-trace\n{ \"traceId\": \"\u003Ctrace_id>\", \"dateRange\": {\"date_from\": \"-30d\"} }\n```\n\n(If you only have a generation ID, query for it via `execute-sql` first to find the\nparent trace ID — see below.)\n\n### Step 4 — Verify the pattern with raw SQL\n\nThe summary is LLM-generated and should be verified. Use `execute-sql` to count and\nspot-check:\n\n```sql\nposthog:execute-sql\nSELECT\n    properties.$ai_target_event_id AS generation_id,\n    properties.$ai_trace_id AS trace_id,\n    properties.$ai_evaluation_reasoning AS reasoning,\n    timestamp\nFROM events\nWHERE event = '$ai_evaluation'\n    AND properties.$ai_evaluation_id = '\u003Cevaluation_uuid>'\n    AND properties.$ai_evaluation_result = false\n    AND (\n        properties.$ai_evaluation_applicable IS NULL\n        OR properties.$ai_evaluation_applicable != false\n    )\n    AND timestamp >= now() - INTERVAL 7 DAY\nORDER BY timestamp DESC\nLIMIT 25\n```\n\nThe N\u002FA guard (`IS NULL OR != false`) is important — it matches the same logic the\nbackend uses to bucket runs.\n\n## Workflow: run an evaluation against a specific generation\n\nUse this when the user pastes a trace\u002Fgeneration URL and asks \"what would evaluation X\nsay about this?\".\n\n```json\nposthog:llma-evaluation-run\n{\n  \"evaluationId\": \"\u003Ceval_uuid>\",\n  \"target_event_id\": \"\u003Cgeneration_event_uuid>\",\n  \"timestamp\": \"2026-04-01T19:39:20Z\",\n  \"event\": \"$ai_generation\"\n}\n```\n\nThe `timestamp` is required for an efficient ClickHouse lookup of the target event.\nPass `distinct_id` if you have it — it speeds up the lookup further.\n\n## Workflow: build and test a new evaluator\n\n### Hog evaluator (deterministic, code-based)\n\nReach for this first when the criterion is rule-based — it's cheaper, faster, and\nreproducible. Prototype with `llma-evaluation-test-hog` (no save):\n\n```json\nposthog:llma-evaluation-test-hog\n{\n  \"source\": \"return event.properties.$ai_output_choices[1].content contains 'sorry';\",\n  \"sample_count\": 5,\n  \"allows_na\": false\n}\n```\n\nThe handler returns the boolean result for each of the most recent N `$ai_generation`\nevents. Iterate on the source until it behaves as expected, then promote it via\n`llma-evaluation-create`:\n\n```json\nposthog:llma-evaluation-create\n{\n  \"name\": \"Output is valid JSON\",\n  \"description\": \"Fails when the assistant message can't be parsed as JSON\",\n  \"evaluation_type\": \"hog\",\n  \"evaluation_config\": {\n    \"source\": \"let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }\"\n  },\n  \"output_type\": \"boolean\",\n  \"enabled\": true\n}\n```\n\nHog evaluators have full access to the event and its properties — common patterns\ninclude schema validation, length\u002Ftoken limits, regex matches, and tool-call shape\nchecks. Because they're deterministic, results are reproducible across reruns and\ntrivially diff-able.\n\n### LLM-judge evaluator (subjective, prompt-based)\n\nUse this when the criterion is fuzzy and a code rule would be brittle (tone, factuality,\nhelpfulness, on-topic-ness). There's no equivalent of `llma-evaluation-test-hog` for LLM\njudges — the typical loop is to create the evaluator with `enabled: false`, run it\nmanually against a handful of representative generations via `llma-evaluation-run`, inspect\nthe results, refine the prompt with `llma-evaluation-update`, and then flip `enabled: true`\nwhen you're satisfied:\n\n```json\nposthog:llma-evaluation-create\n{\n  \"name\": \"Response stays on-topic\",\n  \"description\": \"LLM judge — fails if the assistant changes topic from the user's question\",\n  \"evaluation_type\": \"llm_judge\",\n  \"evaluation_config\": {\n    \"prompt\": \"You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N\u002FA if the user did not actually ask a question.\"\n  },\n  \"output_type\": \"boolean\",\n  \"output_config\": { \"allows_na\": true },\n  \"model_configuration\": {\n    \"provider\": \"openai\",\n    \"model\": \"gpt-5-mini\"\n  },\n  \"enabled\": false\n}\n```\n\nThen dry-run against a known-good and a known-bad generation:\n\n```json\nposthog:llma-evaluation-run\n{\n  \"evaluationId\": \"\u003Cnew_eval_uuid>\",\n  \"target_event_id\": \"\u003Cgeneration_uuid>\",\n  \"timestamp\": \"2026-04-01T19:39:20Z\"\n}\n```\n\nLLM judges require organisation AI data processing approval. Hog evaluators do not.\n\n## Workflow: manage the evaluation lifecycle\n\n| Action                     | Tool                                                                                                                  |\n| -------------------------- | --------------------------------------------------------------------------------------------------------------------- |\n| Add a Hog evaluator        | `llma-evaluation-create` with `evaluation_type: \"hog\"` and `evaluation_config.source`                                 |\n| Add an LLM-judge evaluator | `llma-evaluation-create` with `evaluation_type: \"llm_judge\"`, `evaluation_config.prompt`, and a `model_configuration` |\n| Tweak the source or prompt | `llma-evaluation-update` (edits `evaluation_config.source` for Hog, `evaluation_config.prompt` for LLM judge)         |\n| Toggle N\u002FA handling        | `llma-evaluation-update` with `output_config.allows_na`                                                               |\n| Disable temporarily        | `llma-evaluation-update` with `enabled: false`                                                                        |\n| Remove                     | `llma-evaluation-delete` (soft-delete via PATCH `{deleted: true}`)                                                    |\n\n`llm_judge` evaluations require AI data processing approval at the org level\n(`is_ai_data_processing_approved`). The same gate applies to\n`llma-evaluation-summary-create`. Hog evaluations do **not** require this gate\n— they run as plain code on the ingestion pipeline.\n\n## When to use Hog vs LLM judge\n\nReach for **Hog** by default. Switch to LLM judge only when the criterion can't be\nexpressed as code.\n\n| Use Hog when…                                         | Use LLM judge when…                                     |\n| ----------------------------------------------------- | ------------------------------------------------------- |\n| The check is structural (JSON parses, schema matches) | The check is about meaning (on-topic, helpful, factual) |\n| You need a deterministic, reproducible result         | A small amount of judgement variability is acceptable   |\n| The criterion is cheap to compute                     | The criterion requires reading and understanding text   |\n| You can't get AI data processing approval             | You have approval and the criterion is genuinely fuzzy  |\n| You need to enforce a hard limit (length, cost, etc.) | You need to rate a quality dimension                    |\n| You want sub-millisecond evaluation                   | A few hundred milliseconds + LLM cost are acceptable    |\n\nA common pattern is to **layer them**: a Hog evaluator gates obvious format\u002Flength\nviolations cheaply, and an LLM-judge evaluator only fires on the generations that pass\nthe Hog gate (via `conditions`).\n\n## Investigation patterns\n\nThe summarisation tool works the same way regardless of whether the evaluator is `hog`\nor `llm_judge` — it analyses the resulting `$ai_evaluation` events, not the evaluator\nitself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is\nidentical.\n\n### \"Why is evaluation X suddenly failing more?\"\n\n1. `llma-evaluation-list` — confirm the evaluation is still enabled and unchanged\n   (compare `evaluation_config.source` or `evaluation_config.prompt` to the version you\n   expect)\n2. `llma-evaluation-summary-create` with `filter: \"fail\"` — get the dominant\n   failure patterns and example IDs\n3. SQL count of fails per day to confirm the regression window:\n\n   ```sql\n   SELECT toDate(timestamp) AS day, count() AS fails\n   FROM events\n   WHERE event = '$ai_evaluation'\n       AND properties.$ai_evaluation_id = '\u003Cuuid>'\n       AND properties.$ai_evaluation_result = false\n       AND timestamp >= now() - INTERVAL 30 DAY\n   GROUP BY day\n   ORDER BY day\n   ```\n\n4. Drill into a representative trace per pattern via `query-llm-trace`\n\n### \"Are passes and fails caused by the same root content?\"\n\n1. Generate two summaries: one with `filter: \"pass\"`, one with `filter: \"fail\"`\n2. If `pass_patterns` and `fail_patterns` describe similar content:\n   - For an `llm_judge`: the prompt or rubric is probably ambiguous — reword\n     `evaluation_config.prompt` and use `llma-evaluation-update`\n   - For a `hog` evaluator: the rule is probably under- or over-matching — read the\n     source via `llma-evaluation-get`, narrow the predicate, and retest with\n     `llma-evaluation-test-hog` before pushing the fix via `llma-evaluation-update`\n\n### \"Did a Hog evaluator regression after a code change?\"\n\nHog evaluators are reproducible — if the source hasn't changed, identical inputs should\nyield identical outputs. When fail rates jump for a Hog evaluator:\n\n1. `llma-evaluation-get` — note the current source and `updated_at`\n2. Spot-check the latest failing runs with the SQL query from Step 4 above\n3. Re-run the source against those exact generations using `llma-evaluation-test-hog` with a\n   modified `conditions` filter that targets them\n4. If the test results match the live results, the change is in the _generations_, not\n   the evaluator (a model upgrade, prompt change upstream, etc.) — investigate the\n   producer\n5. If they diverge, the evaluator was edited; check git history of the source field via\n   the activity log\n\n### \"What kinds of generations does this evaluator skip as N\u002FA?\"\n\n```json\nposthog:llma-evaluation-summary-create\n{ \"evaluation_id\": \"\u003Cuuid>\", \"filter\": \"na\" }\n```\n\nInspect `na_patterns` to see whether the N\u002FA logic is doing the right thing. If a\npattern in `na_patterns` looks like something that should have been scored:\n\n- For an `llm_judge`: the applicability instruction in the prompt is too broad — narrow\n  it\n- For a `hog` evaluator with `output_config.allows_na: true`: the source is returning\n  `null` (or whatever the N\u002FA signal is) too eagerly — tighten the precondition\n\n### \"Score this single generation right now\"\n\n`llma-evaluation-run` with the trace's generation ID and timestamp. Useful for spot-checking\nor wiring evaluations into a larger agent loop.\n\n## Constructing UI links\n\n- **Evaluations list**: `https:\u002F\u002Fapp.posthog.com\u002Fai-evals\u002Fevaluations`\n- **Single evaluation**: `https:\u002F\u002Fapp.posthog.com\u002Fai-evals\u002Fevaluations\u002F\u003Cevaluation_id>`\n- **Underlying generation\u002Ftrace**: see the `exploring-llm-traces` skill's URL conventions\n\nAlways surface the relevant link so the user can verify in the UI.\n\n## Tips\n\n- The summary tool is **rate-limited** (burst, sustained, daily) and **caches results\n  for one hour** — repeated calls with the same `(evaluation_id, filter)` are cheap; use\n  `force_refresh: true` only when you genuinely need fresh analysis\n- Pass `generation_ids: [...]` to scope a summary to a specific cohort of runs (max 250)\n- The `statistics` block in the summary response is computed from raw data, not the LLM\n  — trust those counts even if a pattern's `frequency` field is qualitative\n- For rich filtering not supported by `llma-evaluation-list` (e.g. by author or model\n  configuration), fall back to `execute-sql` against the `evaluations` Postgres table or\n  the `$ai_evaluation` ClickHouse events\n- When showing failure patterns to the user, always include 1-2 example trace links so\n  they can validate the pattern visually\n- `llma-evaluation-*` tools use `evaluation:read` for read tools and `evaluation:write` for\n  mutating tools; `llma-evaluation-summary-create` uses `llm_analytics:write`\n- Hog evaluators are reproducible — if you suspect a regression, `llma-evaluation-test-hog`\n  with the suspect source against the failing generations is the fastest way to bisect\n  whether the change is in the evaluator or in the producer of the generations\n- LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with\n  a fixed prompt and model. If you're chasing a small regression in fail rate, prefer\n  Hog or pin a deterministic provider\u002Fseed in the `model_configuration`\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,47,62,112,133,138,145,371,392,398,410,574,602,608,627,634,734,762,799,804,810,916,921,1048,1077,1087,1173,1192,1198,1210,1328,1341,1347,1359,1516,1529,1535,1540,1712,1732,1738,1744,1757,1878,1897,2160,2165,2171,2215,2621,2626,2757,2762,2768,2959,2992,2998,3010,3110,3130,3136,3162,3168,3297,3303,3401,3407,3412,3473,3479,3565,3584,3624,3630,3640,3646,3698,3703,3709,3876],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":46},"text","Exploring LLM evaluations",{"type":40,"tag":48,"props":49,"children":50},"p",{},[51,53,60],{"type":45,"value":52},"PostHog evaluations score ",{"type":40,"tag":54,"props":55,"children":57},"code",{"className":56},[],[58],{"type":45,"value":59},"$ai_generation",{"type":45,"value":61}," events. Each evaluation is one of two types,\nboth first-class:",{"type":40,"tag":63,"props":64,"children":65},"ul",{},[66,98],{"type":40,"tag":67,"props":68,"children":69},"li",{},[70,80,82,88,90,96],{"type":40,"tag":71,"props":72,"children":73},"strong",{},[74],{"type":40,"tag":54,"props":75,"children":77},{"className":76},[],[78],{"type":45,"value":79},"hog",{"type":45,"value":81}," — deterministic Hog code that returns ",{"type":40,"tag":54,"props":83,"children":85},{"className":84},[],[86],{"type":45,"value":87},"true",{"type":45,"value":89},"\u002F",{"type":40,"tag":54,"props":91,"children":93},{"className":92},[],[94],{"type":45,"value":95},"false",{"type":45,"value":97}," (and optionally N\u002FA).\nBest for objective rule-based checks: format validation (JSON parses, schema matches),\nlength limits, keyword presence\u002Fabsence, regex patterns, structural assertions, latency\nthresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this\nwhen the criterion can be expressed as code.",{"type":40,"tag":67,"props":99,"children":100},{},[101,110],{"type":40,"tag":71,"props":102,"children":103},{},[104],{"type":40,"tag":54,"props":105,"children":107},{"className":106},[],[108],{"type":45,"value":109},"llm_judge",{"type":45,"value":111}," — an LLM scores generations against a prompt you write. Best for\nsubjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic\ndrift, instruction-following. Costs an LLM call per run and requires AI data\nprocessing approval at the org level.",{"type":40,"tag":48,"props":113,"children":114},{},[115,117,123,125,131],{"type":45,"value":116},"Results from both types land in ClickHouse as ",{"type":40,"tag":54,"props":118,"children":120},{"className":119},[],[121],{"type":45,"value":122},"$ai_evaluation",{"type":45,"value":124}," events with the same\nschema, so the read\u002Fquery\u002Fsummary workflows are identical regardless of evaluator type —\nthe only thing that changes is whether ",{"type":40,"tag":54,"props":126,"children":128},{"className":127},[],[129],{"type":45,"value":130},"$ai_evaluation_reasoning",{"type":45,"value":132}," was written by Hog\ncode or by an LLM.",{"type":40,"tag":48,"props":134,"children":135},{},[136],{"type":45,"value":137},"This skill covers the full lifecycle: list\u002Finspect\u002Fmanage evaluation configs (Hog or\nLLM judge), run them on specific generations, query individual results, and get an\nAI-generated summary of pass\u002Ffail\u002FN\u002FA patterns across many runs.",{"type":40,"tag":139,"props":140,"children":142},"h2",{"id":141},"tools",[143],{"type":45,"value":144},"Tools",{"type":40,"tag":146,"props":147,"children":148},"table",{},[149,168],{"type":40,"tag":150,"props":151,"children":152},"thead",{},[153],{"type":40,"tag":154,"props":155,"children":156},"tr",{},[157,163],{"type":40,"tag":158,"props":159,"children":160},"th",{},[161],{"type":45,"value":162},"Tool",{"type":40,"tag":158,"props":164,"children":165},{},[166],{"type":45,"value":167},"Purpose",{"type":40,"tag":169,"props":170,"children":171},"tbody",{},[172,190,207,238,255,272,296,313,330,354],{"type":40,"tag":154,"props":173,"children":174},{},[175,185],{"type":40,"tag":176,"props":177,"children":178},"td",{},[179],{"type":40,"tag":54,"props":180,"children":182},{"className":181},[],[183],{"type":45,"value":184},"posthog:llma-evaluation-list",{"type":40,"tag":176,"props":186,"children":187},{},[188],{"type":45,"value":189},"List\u002Fsearch evaluation configs (filter by name, enabled flag)",{"type":40,"tag":154,"props":191,"children":192},{},[193,202],{"type":40,"tag":176,"props":194,"children":195},{},[196],{"type":40,"tag":54,"props":197,"children":199},{"className":198},[],[200],{"type":45,"value":201},"posthog:llma-evaluation-get",{"type":40,"tag":176,"props":203,"children":204},{},[205],{"type":45,"value":206},"Get a single evaluation config by UUID",{"type":40,"tag":154,"props":208,"children":209},{},[210,219],{"type":40,"tag":176,"props":211,"children":212},{},[213],{"type":40,"tag":54,"props":214,"children":216},{"className":215},[],[217],{"type":45,"value":218},"posthog:llma-evaluation-create",{"type":40,"tag":176,"props":220,"children":221},{},[222,224,229,231,236],{"type":45,"value":223},"Create a new ",{"type":40,"tag":54,"props":225,"children":227},{"className":226},[],[228],{"type":45,"value":109},{"type":45,"value":230}," or ",{"type":40,"tag":54,"props":232,"children":234},{"className":233},[],[235],{"type":45,"value":79},{"type":45,"value":237}," evaluation",{"type":40,"tag":154,"props":239,"children":240},{},[241,250],{"type":40,"tag":176,"props":242,"children":243},{},[244],{"type":40,"tag":54,"props":245,"children":247},{"className":246},[],[248],{"type":45,"value":249},"posthog:llma-evaluation-update",{"type":40,"tag":176,"props":251,"children":252},{},[253],{"type":45,"value":254},"Update an existing evaluation (name, prompt, enabled, …)",{"type":40,"tag":154,"props":256,"children":257},{},[258,267],{"type":40,"tag":176,"props":259,"children":260},{},[261],{"type":40,"tag":54,"props":262,"children":264},{"className":263},[],[265],{"type":45,"value":266},"posthog:llma-evaluation-delete",{"type":40,"tag":176,"props":268,"children":269},{},[270],{"type":45,"value":271},"Soft-delete an evaluation",{"type":40,"tag":154,"props":273,"children":274},{},[275,284],{"type":40,"tag":176,"props":276,"children":277},{},[278],{"type":40,"tag":54,"props":279,"children":281},{"className":280},[],[282],{"type":45,"value":283},"posthog:llma-evaluation-run",{"type":40,"tag":176,"props":285,"children":286},{},[287,289,294],{"type":45,"value":288},"Run an evaluation against a specific ",{"type":40,"tag":54,"props":290,"children":292},{"className":291},[],[293],{"type":45,"value":59},{"type":45,"value":295}," event",{"type":40,"tag":154,"props":297,"children":298},{},[299,308],{"type":40,"tag":176,"props":300,"children":301},{},[302],{"type":40,"tag":54,"props":303,"children":305},{"className":304},[],[306],{"type":45,"value":307},"posthog:llma-evaluation-test-hog",{"type":40,"tag":176,"props":309,"children":310},{},[311],{"type":45,"value":312},"Dry-run Hog source against recent generations (no save)",{"type":40,"tag":154,"props":314,"children":315},{},[316,325],{"type":40,"tag":176,"props":317,"children":318},{},[319],{"type":40,"tag":54,"props":320,"children":322},{"className":321},[],[323],{"type":45,"value":324},"posthog:llma-evaluation-summary-create",{"type":40,"tag":176,"props":326,"children":327},{},[328],{"type":45,"value":329},"AI-powered summary of pass\u002Ffail\u002FN\u002FA patterns across runs",{"type":40,"tag":154,"props":331,"children":332},{},[333,342],{"type":40,"tag":176,"props":334,"children":335},{},[336],{"type":40,"tag":54,"props":337,"children":339},{"className":338},[],[340],{"type":45,"value":341},"posthog:execute-sql",{"type":40,"tag":176,"props":343,"children":344},{},[345,347,352],{"type":45,"value":346},"Ad-hoc HogQL over ",{"type":40,"tag":54,"props":348,"children":350},{"className":349},[],[351],{"type":45,"value":122},{"type":45,"value":353}," events",{"type":40,"tag":154,"props":355,"children":356},{},[357,366],{"type":40,"tag":176,"props":358,"children":359},{},[360],{"type":40,"tag":54,"props":361,"children":363},{"className":362},[],[364],{"type":45,"value":365},"posthog:query-llm-trace",{"type":40,"tag":176,"props":367,"children":368},{},[369],{"type":45,"value":370},"Drill into the underlying generation that an evaluation scored",{"type":40,"tag":48,"props":372,"children":373},{},[374,376,382,384,390],{"type":45,"value":375},"All ",{"type":40,"tag":54,"props":377,"children":379},{"className":378},[],[380],{"type":45,"value":381},"llma-evaluation-*",{"type":45,"value":383}," tools are defined in ",{"type":40,"tag":54,"props":385,"children":387},{"className":386},[],[388],{"type":45,"value":389},"products\u002Fai_observability\u002Fmcp\u002Ftools.yaml",{"type":45,"value":391},".",{"type":40,"tag":139,"props":393,"children":395},{"id":394},"event-schema",[396],{"type":45,"value":397},"Event schema",{"type":40,"tag":48,"props":399,"children":400},{},[401,403,408],{"type":45,"value":402},"Every run of an evaluation emits an ",{"type":40,"tag":54,"props":404,"children":406},{"className":405},[],[407],{"type":45,"value":122},{"type":45,"value":409}," event. Key properties:",{"type":40,"tag":146,"props":411,"children":412},{},[413,429],{"type":40,"tag":150,"props":414,"children":415},{},[416],{"type":40,"tag":154,"props":417,"children":418},{},[419,424],{"type":40,"tag":158,"props":420,"children":421},{},[422],{"type":45,"value":423},"Property",{"type":40,"tag":158,"props":425,"children":426},{},[427],{"type":45,"value":428},"Meaning",{"type":40,"tag":169,"props":430,"children":431},{},[432,449,466,490,507,536,552],{"type":40,"tag":154,"props":433,"children":434},{},[435,444],{"type":40,"tag":176,"props":436,"children":437},{},[438],{"type":40,"tag":54,"props":439,"children":441},{"className":440},[],[442],{"type":45,"value":443},"$ai_evaluation_id",{"type":40,"tag":176,"props":445,"children":446},{},[447],{"type":45,"value":448},"UUID of the evaluation config",{"type":40,"tag":154,"props":450,"children":451},{},[452,461],{"type":40,"tag":176,"props":453,"children":454},{},[455],{"type":40,"tag":54,"props":456,"children":458},{"className":457},[],[459],{"type":45,"value":460},"$ai_evaluation_name",{"type":40,"tag":176,"props":462,"children":463},{},[464],{"type":45,"value":465},"Human-readable name",{"type":40,"tag":154,"props":467,"children":468},{},[469,478],{"type":40,"tag":176,"props":470,"children":471},{},[472],{"type":40,"tag":54,"props":473,"children":475},{"className":474},[],[476],{"type":45,"value":477},"$ai_target_event_id",{"type":40,"tag":176,"props":479,"children":480},{},[481,483,488],{"type":45,"value":482},"UUID of the ",{"type":40,"tag":54,"props":484,"children":486},{"className":485},[],[487],{"type":45,"value":59},{"type":45,"value":489}," event being scored",{"type":40,"tag":154,"props":491,"children":492},{},[493,502],{"type":40,"tag":176,"props":494,"children":495},{},[496],{"type":40,"tag":54,"props":497,"children":499},{"className":498},[],[500],{"type":45,"value":501},"$ai_trace_id",{"type":40,"tag":176,"props":503,"children":504},{},[505],{"type":45,"value":506},"Parent trace ID (for jumping to the trace UI)",{"type":40,"tag":154,"props":508,"children":509},{},[510,519],{"type":40,"tag":176,"props":511,"children":512},{},[513],{"type":40,"tag":54,"props":514,"children":516},{"className":515},[],[517],{"type":45,"value":518},"$ai_evaluation_result",{"type":40,"tag":176,"props":520,"children":521},{},[522,527,529,534],{"type":40,"tag":54,"props":523,"children":525},{"className":524},[],[526],{"type":45,"value":87},{"type":45,"value":528}," = pass, ",{"type":40,"tag":54,"props":530,"children":532},{"className":531},[],[533],{"type":45,"value":95},{"type":45,"value":535}," = fail",{"type":40,"tag":154,"props":537,"children":538},{},[539,547],{"type":40,"tag":176,"props":540,"children":541},{},[542],{"type":40,"tag":54,"props":543,"children":545},{"className":544},[],[546],{"type":45,"value":130},{"type":40,"tag":176,"props":548,"children":549},{},[550],{"type":45,"value":551},"Free-text explanation (set by the LLM judge or Hog code)",{"type":40,"tag":154,"props":553,"children":554},{},[555,564],{"type":40,"tag":176,"props":556,"children":557},{},[558],{"type":40,"tag":54,"props":559,"children":561},{"className":560},[],[562],{"type":45,"value":563},"$ai_evaluation_applicable",{"type":40,"tag":176,"props":565,"children":566},{},[567,572],{"type":40,"tag":54,"props":568,"children":570},{"className":569},[],[571],{"type":45,"value":95},{"type":45,"value":573}," when the evaluator decided the generation is N\u002FA",{"type":40,"tag":48,"props":575,"children":576},{},[577,579,585,587,592,594,600],{"type":45,"value":578},"When ",{"type":40,"tag":54,"props":580,"children":582},{"className":581},[],[583],{"type":45,"value":584},"$ai_evaluation_applicable = false",{"type":45,"value":586},", the run counts as N\u002FA regardless of ",{"type":40,"tag":54,"props":588,"children":590},{"className":589},[],[591],{"type":45,"value":518},{"type":45,"value":593},".\nFor evaluations that don't support N\u002FA, this property may be ",{"type":40,"tag":54,"props":595,"children":597},{"className":596},[],[598],{"type":45,"value":599},"null",{"type":45,"value":601}," — treat null as \"applicable\".",{"type":40,"tag":139,"props":603,"children":605},{"id":604},"workflow-investigate-why-an-evaluation-is-failing",[606],{"type":45,"value":607},"Workflow: investigate why an evaluation is failing",{"type":40,"tag":48,"props":609,"children":610},{},[611,613,618,620,625],{"type":45,"value":612},"Works the same way for ",{"type":40,"tag":54,"props":614,"children":616},{"className":615},[],[617],{"type":45,"value":109},{"type":45,"value":619}," and ",{"type":40,"tag":54,"props":621,"children":623},{"className":622},[],[624],{"type":45,"value":79},{"type":45,"value":626}," evaluations — the differences only matter\nwhen you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source).",{"type":40,"tag":628,"props":629,"children":631},"h3",{"id":630},"step-1-find-the-evaluation",[632],{"type":45,"value":633},"Step 1 — Find the evaluation",{"type":40,"tag":635,"props":636,"children":641},"pre",{"className":637,"code":638,"language":639,"meta":640,"style":640},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","posthog:llma-evaluation-list\n{ \"search\": \"hallucination\", \"enabled\": true }\n","json","",[642],{"type":40,"tag":54,"props":643,"children":644},{"__ignoreMap":640},[645,657],{"type":40,"tag":646,"props":647,"children":650},"span",{"class":648,"line":649},"line",1,[651],{"type":40,"tag":646,"props":652,"children":654},{"style":653},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[655],{"type":45,"value":656},"posthog:llma-evaluation-list\n",{"type":40,"tag":646,"props":658,"children":660},{"class":648,"line":659},2,[661,667,672,678,683,688,692,698,702,707,711,716,720,724,729],{"type":40,"tag":646,"props":662,"children":664},{"style":663},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[665],{"type":45,"value":666},"{",{"type":40,"tag":646,"props":668,"children":669},{"style":663},[670],{"type":45,"value":671}," \"",{"type":40,"tag":646,"props":673,"children":675},{"style":674},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[676],{"type":45,"value":677},"search",{"type":40,"tag":646,"props":679,"children":680},{"style":663},[681],{"type":45,"value":682},"\"",{"type":40,"tag":646,"props":684,"children":685},{"style":663},[686],{"type":45,"value":687},":",{"type":40,"tag":646,"props":689,"children":690},{"style":663},[691],{"type":45,"value":671},{"type":40,"tag":646,"props":693,"children":695},{"style":694},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[696],{"type":45,"value":697},"hallucination",{"type":40,"tag":646,"props":699,"children":700},{"style":663},[701],{"type":45,"value":682},{"type":40,"tag":646,"props":703,"children":704},{"style":663},[705],{"type":45,"value":706},",",{"type":40,"tag":646,"props":708,"children":709},{"style":663},[710],{"type":45,"value":671},{"type":40,"tag":646,"props":712,"children":713},{"style":674},[714],{"type":45,"value":715},"enabled",{"type":40,"tag":646,"props":717,"children":718},{"style":663},[719],{"type":45,"value":682},{"type":40,"tag":646,"props":721,"children":722},{"style":663},[723],{"type":45,"value":687},{"type":40,"tag":646,"props":725,"children":726},{"style":663},[727],{"type":45,"value":728}," true",{"type":40,"tag":646,"props":730,"children":731},{"style":663},[732],{"type":45,"value":733}," }\n",{"type":40,"tag":48,"props":735,"children":736},{},[737,739,745,747,753,754,760],{"type":45,"value":738},"Look at the returned ",{"type":40,"tag":54,"props":740,"children":742},{"className":741},[],[743],{"type":45,"value":744},"id",{"type":45,"value":746},", ",{"type":40,"tag":54,"props":748,"children":750},{"className":749},[],[751],{"type":45,"value":752},"name",{"type":45,"value":746},{"type":40,"tag":54,"props":755,"children":757},{"className":756},[],[758],{"type":45,"value":759},"evaluation_type",{"type":45,"value":761},", and either:",{"type":40,"tag":63,"props":763,"children":764},{},[765,781],{"type":40,"tag":67,"props":766,"children":767},{},[768,774,776],{"type":40,"tag":54,"props":769,"children":771},{"className":770},[],[772],{"type":45,"value":773},"evaluation_config.prompt",{"type":45,"value":775}," for an ",{"type":40,"tag":54,"props":777,"children":779},{"className":778},[],[780],{"type":45,"value":109},{"type":40,"tag":67,"props":782,"children":783},{},[784,790,792,797],{"type":40,"tag":54,"props":785,"children":787},{"className":786},[],[788],{"type":45,"value":789},"evaluation_config.source",{"type":45,"value":791}," for a ",{"type":40,"tag":54,"props":793,"children":795},{"className":794},[],[796],{"type":45,"value":79},{"type":45,"value":798}," evaluator",{"type":40,"tag":48,"props":800,"children":801},{},[802],{"type":45,"value":803},"The Hog source is the ground truth for why a hog evaluator passes or fails — read it\nbefore assuming the failure is in the generation.",{"type":40,"tag":628,"props":805,"children":807},{"id":806},"step-2-get-the-ai-generated-summary",[808],{"type":45,"value":809},"Step 2 — Get the AI-generated summary",{"type":40,"tag":635,"props":811,"children":813},{"className":637,"code":812,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-summary-create\n{\n  \"evaluation_id\": \"\u003Cuuid>\",\n  \"filter\": \"fail\"\n}\n",[814],{"type":40,"tag":54,"props":815,"children":816},{"__ignoreMap":640},[817,825,833,873,907],{"type":40,"tag":646,"props":818,"children":819},{"class":648,"line":649},[820],{"type":40,"tag":646,"props":821,"children":822},{"style":653},[823],{"type":45,"value":824},"posthog:llma-evaluation-summary-create\n",{"type":40,"tag":646,"props":826,"children":827},{"class":648,"line":659},[828],{"type":40,"tag":646,"props":829,"children":830},{"style":663},[831],{"type":45,"value":832},"{\n",{"type":40,"tag":646,"props":834,"children":836},{"class":648,"line":835},3,[837,842,847,851,855,859,864,868],{"type":40,"tag":646,"props":838,"children":839},{"style":663},[840],{"type":45,"value":841},"  \"",{"type":40,"tag":646,"props":843,"children":844},{"style":674},[845],{"type":45,"value":846},"evaluation_id",{"type":40,"tag":646,"props":848,"children":849},{"style":663},[850],{"type":45,"value":682},{"type":40,"tag":646,"props":852,"children":853},{"style":663},[854],{"type":45,"value":687},{"type":40,"tag":646,"props":856,"children":857},{"style":663},[858],{"type":45,"value":671},{"type":40,"tag":646,"props":860,"children":861},{"style":694},[862],{"type":45,"value":863},"\u003Cuuid>",{"type":40,"tag":646,"props":865,"children":866},{"style":663},[867],{"type":45,"value":682},{"type":40,"tag":646,"props":869,"children":870},{"style":663},[871],{"type":45,"value":872},",\n",{"type":40,"tag":646,"props":874,"children":875},{"class":648,"line":27},[876,880,885,889,893,897,902],{"type":40,"tag":646,"props":877,"children":878},{"style":663},[879],{"type":45,"value":841},{"type":40,"tag":646,"props":881,"children":882},{"style":674},[883],{"type":45,"value":884},"filter",{"type":40,"tag":646,"props":886,"children":887},{"style":663},[888],{"type":45,"value":682},{"type":40,"tag":646,"props":890,"children":891},{"style":663},[892],{"type":45,"value":687},{"type":40,"tag":646,"props":894,"children":895},{"style":663},[896],{"type":45,"value":671},{"type":40,"tag":646,"props":898,"children":899},{"style":694},[900],{"type":45,"value":901},"fail",{"type":40,"tag":646,"props":903,"children":904},{"style":663},[905],{"type":45,"value":906},"\"\n",{"type":40,"tag":646,"props":908,"children":910},{"class":648,"line":909},5,[911],{"type":40,"tag":646,"props":912,"children":913},{"style":663},[914],{"type":45,"value":915},"}\n",{"type":40,"tag":48,"props":917,"children":918},{},[919],{"type":45,"value":920},"Returns:",{"type":40,"tag":63,"props":922,"children":923},{},[924,935,974,999,1010],{"type":40,"tag":67,"props":925,"children":926},{},[927,933],{"type":40,"tag":54,"props":928,"children":930},{"className":929},[],[931],{"type":45,"value":932},"overall_assessment",{"type":45,"value":934}," — natural-language summary",{"type":40,"tag":67,"props":936,"children":937},{},[938,944,946,952,953,959,960,966,968],{"type":40,"tag":54,"props":939,"children":941},{"className":940},[],[942],{"type":45,"value":943},"fail_patterns",{"type":45,"value":945}," — grouped patterns with ",{"type":40,"tag":54,"props":947,"children":949},{"className":948},[],[950],{"type":45,"value":951},"title",{"type":45,"value":746},{"type":40,"tag":54,"props":954,"children":956},{"className":955},[],[957],{"type":45,"value":958},"description",{"type":45,"value":746},{"type":40,"tag":54,"props":961,"children":963},{"className":962},[],[964],{"type":45,"value":965},"frequency",{"type":45,"value":967},", and ",{"type":40,"tag":54,"props":969,"children":971},{"className":970},[],[972],{"type":45,"value":973},"example_generation_ids",{"type":40,"tag":67,"props":975,"children":976},{},[977,983,984,990,992,997],{"type":40,"tag":54,"props":978,"children":980},{"className":979},[],[981],{"type":45,"value":982},"pass_patterns",{"type":45,"value":619},{"type":40,"tag":54,"props":985,"children":987},{"className":986},[],[988],{"type":45,"value":989},"na_patterns",{"type":45,"value":991}," — same shape, populated when ",{"type":40,"tag":54,"props":993,"children":995},{"className":994},[],[996],{"type":45,"value":884},{"type":45,"value":998}," includes them",{"type":40,"tag":67,"props":1000,"children":1001},{},[1002,1008],{"type":40,"tag":54,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":45,"value":1007},"recommendations",{"type":45,"value":1009}," — actionable next steps",{"type":40,"tag":67,"props":1011,"children":1012},{},[1013,1019,1021,1027,1028,1034,1035,1041,1042],{"type":40,"tag":54,"props":1014,"children":1016},{"className":1015},[],[1017],{"type":45,"value":1018},"statistics",{"type":45,"value":1020}," — ",{"type":40,"tag":54,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":45,"value":1026},"total_analyzed",{"type":45,"value":746},{"type":40,"tag":54,"props":1029,"children":1031},{"className":1030},[],[1032],{"type":45,"value":1033},"pass_count",{"type":45,"value":746},{"type":40,"tag":54,"props":1036,"children":1038},{"className":1037},[],[1039],{"type":45,"value":1040},"fail_count",{"type":45,"value":746},{"type":40,"tag":54,"props":1043,"children":1045},{"className":1044},[],[1046],{"type":45,"value":1047},"na_count",{"type":40,"tag":48,"props":1049,"children":1050},{},[1051,1053,1059,1061,1067,1069,1075],{"type":45,"value":1052},"The endpoint analyses the most recent ~250 runs (",{"type":40,"tag":54,"props":1054,"children":1056},{"className":1055},[],[1057],{"type":45,"value":1058},"EVALUATION_SUMMARY_MAX_RUNS",{"type":45,"value":1060},").\nResults are cached for one hour per ",{"type":40,"tag":54,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":45,"value":1066},"(evaluation_id, filter, set_of_generation_ids)",{"type":45,"value":1068},".\nPass ",{"type":40,"tag":54,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":45,"value":1074},"force_refresh: true",{"type":45,"value":1076}," to recompute.",{"type":40,"tag":48,"props":1078,"children":1079},{},[1080,1085],{"type":40,"tag":71,"props":1081,"children":1082},{},[1083],{"type":45,"value":1084},"Compare filters in two calls",{"type":45,"value":1086}," to spot what's distinctive about failures vs passes:",{"type":40,"tag":635,"props":1088,"children":1090},{"className":637,"code":1089,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-summary-create\n{ \"evaluation_id\": \"\u003Cuuid>\", \"filter\": \"pass\" }\n",[1091],{"type":40,"tag":54,"props":1092,"children":1093},{"__ignoreMap":640},[1094,1101],{"type":40,"tag":646,"props":1095,"children":1096},{"class":648,"line":649},[1097],{"type":40,"tag":646,"props":1098,"children":1099},{"style":653},[1100],{"type":45,"value":824},{"type":40,"tag":646,"props":1102,"children":1103},{"class":648,"line":659},[1104,1108,1112,1116,1120,1124,1128,1132,1136,1140,1144,1148,1152,1156,1160,1165,1169],{"type":40,"tag":646,"props":1105,"children":1106},{"style":663},[1107],{"type":45,"value":666},{"type":40,"tag":646,"props":1109,"children":1110},{"style":663},[1111],{"type":45,"value":671},{"type":40,"tag":646,"props":1113,"children":1114},{"style":674},[1115],{"type":45,"value":846},{"type":40,"tag":646,"props":1117,"children":1118},{"style":663},[1119],{"type":45,"value":682},{"type":40,"tag":646,"props":1121,"children":1122},{"style":663},[1123],{"type":45,"value":687},{"type":40,"tag":646,"props":1125,"children":1126},{"style":663},[1127],{"type":45,"value":671},{"type":40,"tag":646,"props":1129,"children":1130},{"style":694},[1131],{"type":45,"value":863},{"type":40,"tag":646,"props":1133,"children":1134},{"style":663},[1135],{"type":45,"value":682},{"type":40,"tag":646,"props":1137,"children":1138},{"style":663},[1139],{"type":45,"value":706},{"type":40,"tag":646,"props":1141,"children":1142},{"style":663},[1143],{"type":45,"value":671},{"type":40,"tag":646,"props":1145,"children":1146},{"style":674},[1147],{"type":45,"value":884},{"type":40,"tag":646,"props":1149,"children":1150},{"style":663},[1151],{"type":45,"value":682},{"type":40,"tag":646,"props":1153,"children":1154},{"style":663},[1155],{"type":45,"value":687},{"type":40,"tag":646,"props":1157,"children":1158},{"style":663},[1159],{"type":45,"value":671},{"type":40,"tag":646,"props":1161,"children":1162},{"style":694},[1163],{"type":45,"value":1164},"pass",{"type":40,"tag":646,"props":1166,"children":1167},{"style":663},[1168],{"type":45,"value":682},{"type":40,"tag":646,"props":1170,"children":1171},{"style":663},[1172],{"type":45,"value":733},{"type":40,"tag":48,"props":1174,"children":1175},{},[1176,1178,1183,1185,1190],{"type":45,"value":1177},"Then diff the ",{"type":40,"tag":54,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":45,"value":982},{"type":45,"value":1184}," against the ",{"type":40,"tag":54,"props":1186,"children":1188},{"className":1187},[],[1189],{"type":45,"value":943},{"type":45,"value":1191}," from Step 2.",{"type":40,"tag":628,"props":1193,"children":1195},{"id":1194},"step-3-drill-into-example-failing-runs",[1196],{"type":45,"value":1197},"Step 3 — Drill into example failing runs",{"type":40,"tag":48,"props":1199,"children":1200},{},[1201,1203,1208],{"type":45,"value":1202},"Each pattern surfaces ",{"type":40,"tag":54,"props":1204,"children":1206},{"className":1205},[],[1207],{"type":45,"value":973},{"type":45,"value":1209},". Pull the underlying trace for the most\nrepresentative example:",{"type":40,"tag":635,"props":1211,"children":1213},{"className":637,"code":1212,"language":639,"meta":640,"style":640},"posthog:query-llm-trace\n{ \"traceId\": \"\u003Ctrace_id>\", \"dateRange\": {\"date_from\": \"-30d\"} }\n",[1214],{"type":40,"tag":54,"props":1215,"children":1216},{"__ignoreMap":640},[1217,1225],{"type":40,"tag":646,"props":1218,"children":1219},{"class":648,"line":649},[1220],{"type":40,"tag":646,"props":1221,"children":1222},{"style":653},[1223],{"type":45,"value":1224},"posthog:query-llm-trace\n",{"type":40,"tag":646,"props":1226,"children":1227},{"class":648,"line":659},[1228,1232,1236,1241,1245,1249,1253,1258,1262,1266,1270,1275,1279,1283,1288,1292,1298,1302,1306,1310,1315,1319,1324],{"type":40,"tag":646,"props":1229,"children":1230},{"style":663},[1231],{"type":45,"value":666},{"type":40,"tag":646,"props":1233,"children":1234},{"style":663},[1235],{"type":45,"value":671},{"type":40,"tag":646,"props":1237,"children":1238},{"style":674},[1239],{"type":45,"value":1240},"traceId",{"type":40,"tag":646,"props":1242,"children":1243},{"style":663},[1244],{"type":45,"value":682},{"type":40,"tag":646,"props":1246,"children":1247},{"style":663},[1248],{"type":45,"value":687},{"type":40,"tag":646,"props":1250,"children":1251},{"style":663},[1252],{"type":45,"value":671},{"type":40,"tag":646,"props":1254,"children":1255},{"style":694},[1256],{"type":45,"value":1257},"\u003Ctrace_id>",{"type":40,"tag":646,"props":1259,"children":1260},{"style":663},[1261],{"type":45,"value":682},{"type":40,"tag":646,"props":1263,"children":1264},{"style":663},[1265],{"type":45,"value":706},{"type":40,"tag":646,"props":1267,"children":1268},{"style":663},[1269],{"type":45,"value":671},{"type":40,"tag":646,"props":1271,"children":1272},{"style":674},[1273],{"type":45,"value":1274},"dateRange",{"type":40,"tag":646,"props":1276,"children":1277},{"style":663},[1278],{"type":45,"value":682},{"type":40,"tag":646,"props":1280,"children":1281},{"style":663},[1282],{"type":45,"value":687},{"type":40,"tag":646,"props":1284,"children":1285},{"style":663},[1286],{"type":45,"value":1287}," {",{"type":40,"tag":646,"props":1289,"children":1290},{"style":663},[1291],{"type":45,"value":682},{"type":40,"tag":646,"props":1293,"children":1295},{"style":1294},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1296],{"type":45,"value":1297},"date_from",{"type":40,"tag":646,"props":1299,"children":1300},{"style":663},[1301],{"type":45,"value":682},{"type":40,"tag":646,"props":1303,"children":1304},{"style":663},[1305],{"type":45,"value":687},{"type":40,"tag":646,"props":1307,"children":1308},{"style":663},[1309],{"type":45,"value":671},{"type":40,"tag":646,"props":1311,"children":1312},{"style":694},[1313],{"type":45,"value":1314},"-30d",{"type":40,"tag":646,"props":1316,"children":1317},{"style":663},[1318],{"type":45,"value":682},{"type":40,"tag":646,"props":1320,"children":1321},{"style":663},[1322],{"type":45,"value":1323},"}",{"type":40,"tag":646,"props":1325,"children":1326},{"style":663},[1327],{"type":45,"value":733},{"type":40,"tag":48,"props":1329,"children":1330},{},[1331,1333,1339],{"type":45,"value":1332},"(If you only have a generation ID, query for it via ",{"type":40,"tag":54,"props":1334,"children":1336},{"className":1335},[],[1337],{"type":45,"value":1338},"execute-sql",{"type":45,"value":1340}," first to find the\nparent trace ID — see below.)",{"type":40,"tag":628,"props":1342,"children":1344},{"id":1343},"step-4-verify-the-pattern-with-raw-sql",[1345],{"type":45,"value":1346},"Step 4 — Verify the pattern with raw SQL",{"type":40,"tag":48,"props":1348,"children":1349},{},[1350,1352,1357],{"type":45,"value":1351},"The summary is LLM-generated and should be verified. Use ",{"type":40,"tag":54,"props":1353,"children":1355},{"className":1354},[],[1356],{"type":45,"value":1338},{"type":45,"value":1358}," to count and\nspot-check:",{"type":40,"tag":635,"props":1360,"children":1364},{"className":1361,"code":1362,"language":1363,"meta":640,"style":640},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","posthog:execute-sql\nSELECT\n    properties.$ai_target_event_id AS generation_id,\n    properties.$ai_trace_id AS trace_id,\n    properties.$ai_evaluation_reasoning AS reasoning,\n    timestamp\nFROM events\nWHERE event = '$ai_evaluation'\n    AND properties.$ai_evaluation_id = '\u003Cevaluation_uuid>'\n    AND properties.$ai_evaluation_result = false\n    AND (\n        properties.$ai_evaluation_applicable IS NULL\n        OR properties.$ai_evaluation_applicable != false\n    )\n    AND timestamp >= now() - INTERVAL 7 DAY\nORDER BY timestamp DESC\nLIMIT 25\n","sql",[1365],{"type":40,"tag":54,"props":1366,"children":1367},{"__ignoreMap":640},[1368,1376,1384,1392,1400,1408,1417,1426,1435,1444,1453,1462,1471,1480,1489,1498,1507],{"type":40,"tag":646,"props":1369,"children":1370},{"class":648,"line":649},[1371],{"type":40,"tag":646,"props":1372,"children":1373},{},[1374],{"type":45,"value":1375},"posthog:execute-sql\n",{"type":40,"tag":646,"props":1377,"children":1378},{"class":648,"line":659},[1379],{"type":40,"tag":646,"props":1380,"children":1381},{},[1382],{"type":45,"value":1383},"SELECT\n",{"type":40,"tag":646,"props":1385,"children":1386},{"class":648,"line":835},[1387],{"type":40,"tag":646,"props":1388,"children":1389},{},[1390],{"type":45,"value":1391},"    properties.$ai_target_event_id AS generation_id,\n",{"type":40,"tag":646,"props":1393,"children":1394},{"class":648,"line":27},[1395],{"type":40,"tag":646,"props":1396,"children":1397},{},[1398],{"type":45,"value":1399},"    properties.$ai_trace_id AS trace_id,\n",{"type":40,"tag":646,"props":1401,"children":1402},{"class":648,"line":909},[1403],{"type":40,"tag":646,"props":1404,"children":1405},{},[1406],{"type":45,"value":1407},"    properties.$ai_evaluation_reasoning AS reasoning,\n",{"type":40,"tag":646,"props":1409,"children":1411},{"class":648,"line":1410},6,[1412],{"type":40,"tag":646,"props":1413,"children":1414},{},[1415],{"type":45,"value":1416},"    timestamp\n",{"type":40,"tag":646,"props":1418,"children":1420},{"class":648,"line":1419},7,[1421],{"type":40,"tag":646,"props":1422,"children":1423},{},[1424],{"type":45,"value":1425},"FROM events\n",{"type":40,"tag":646,"props":1427,"children":1429},{"class":648,"line":1428},8,[1430],{"type":40,"tag":646,"props":1431,"children":1432},{},[1433],{"type":45,"value":1434},"WHERE event = '$ai_evaluation'\n",{"type":40,"tag":646,"props":1436,"children":1438},{"class":648,"line":1437},9,[1439],{"type":40,"tag":646,"props":1440,"children":1441},{},[1442],{"type":45,"value":1443},"    AND properties.$ai_evaluation_id = '\u003Cevaluation_uuid>'\n",{"type":40,"tag":646,"props":1445,"children":1447},{"class":648,"line":1446},10,[1448],{"type":40,"tag":646,"props":1449,"children":1450},{},[1451],{"type":45,"value":1452},"    AND properties.$ai_evaluation_result = false\n",{"type":40,"tag":646,"props":1454,"children":1456},{"class":648,"line":1455},11,[1457],{"type":40,"tag":646,"props":1458,"children":1459},{},[1460],{"type":45,"value":1461},"    AND (\n",{"type":40,"tag":646,"props":1463,"children":1465},{"class":648,"line":1464},12,[1466],{"type":40,"tag":646,"props":1467,"children":1468},{},[1469],{"type":45,"value":1470},"        properties.$ai_evaluation_applicable IS NULL\n",{"type":40,"tag":646,"props":1472,"children":1474},{"class":648,"line":1473},13,[1475],{"type":40,"tag":646,"props":1476,"children":1477},{},[1478],{"type":45,"value":1479},"        OR properties.$ai_evaluation_applicable != false\n",{"type":40,"tag":646,"props":1481,"children":1483},{"class":648,"line":1482},14,[1484],{"type":40,"tag":646,"props":1485,"children":1486},{},[1487],{"type":45,"value":1488},"    )\n",{"type":40,"tag":646,"props":1490,"children":1492},{"class":648,"line":1491},15,[1493],{"type":40,"tag":646,"props":1494,"children":1495},{},[1496],{"type":45,"value":1497},"    AND timestamp >= now() - INTERVAL 7 DAY\n",{"type":40,"tag":646,"props":1499,"children":1501},{"class":648,"line":1500},16,[1502],{"type":40,"tag":646,"props":1503,"children":1504},{},[1505],{"type":45,"value":1506},"ORDER BY timestamp DESC\n",{"type":40,"tag":646,"props":1508,"children":1510},{"class":648,"line":1509},17,[1511],{"type":40,"tag":646,"props":1512,"children":1513},{},[1514],{"type":45,"value":1515},"LIMIT 25\n",{"type":40,"tag":48,"props":1517,"children":1518},{},[1519,1521,1527],{"type":45,"value":1520},"The N\u002FA guard (",{"type":40,"tag":54,"props":1522,"children":1524},{"className":1523},[],[1525],{"type":45,"value":1526},"IS NULL OR != false",{"type":45,"value":1528},") is important — it matches the same logic the\nbackend uses to bucket runs.",{"type":40,"tag":139,"props":1530,"children":1532},{"id":1531},"workflow-run-an-evaluation-against-a-specific-generation",[1533],{"type":45,"value":1534},"Workflow: run an evaluation against a specific generation",{"type":40,"tag":48,"props":1536,"children":1537},{},[1538],{"type":45,"value":1539},"Use this when the user pastes a trace\u002Fgeneration URL and asks \"what would evaluation X\nsay about this?\".",{"type":40,"tag":635,"props":1541,"children":1543},{"className":637,"code":1542,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-run\n{\n  \"evaluationId\": \"\u003Ceval_uuid>\",\n  \"target_event_id\": \"\u003Cgeneration_event_uuid>\",\n  \"timestamp\": \"2026-04-01T19:39:20Z\",\n  \"event\": \"$ai_generation\"\n}\n",[1544],{"type":40,"tag":54,"props":1545,"children":1546},{"__ignoreMap":640},[1547,1555,1562,1599,1636,1673,1705],{"type":40,"tag":646,"props":1548,"children":1549},{"class":648,"line":649},[1550],{"type":40,"tag":646,"props":1551,"children":1552},{"style":653},[1553],{"type":45,"value":1554},"posthog:llma-evaluation-run\n",{"type":40,"tag":646,"props":1556,"children":1557},{"class":648,"line":659},[1558],{"type":40,"tag":646,"props":1559,"children":1560},{"style":663},[1561],{"type":45,"value":832},{"type":40,"tag":646,"props":1563,"children":1564},{"class":648,"line":835},[1565,1569,1574,1578,1582,1586,1591,1595],{"type":40,"tag":646,"props":1566,"children":1567},{"style":663},[1568],{"type":45,"value":841},{"type":40,"tag":646,"props":1570,"children":1571},{"style":674},[1572],{"type":45,"value":1573},"evaluationId",{"type":40,"tag":646,"props":1575,"children":1576},{"style":663},[1577],{"type":45,"value":682},{"type":40,"tag":646,"props":1579,"children":1580},{"style":663},[1581],{"type":45,"value":687},{"type":40,"tag":646,"props":1583,"children":1584},{"style":663},[1585],{"type":45,"value":671},{"type":40,"tag":646,"props":1587,"children":1588},{"style":694},[1589],{"type":45,"value":1590},"\u003Ceval_uuid>",{"type":40,"tag":646,"props":1592,"children":1593},{"style":663},[1594],{"type":45,"value":682},{"type":40,"tag":646,"props":1596,"children":1597},{"style":663},[1598],{"type":45,"value":872},{"type":40,"tag":646,"props":1600,"children":1601},{"class":648,"line":27},[1602,1606,1611,1615,1619,1623,1628,1632],{"type":40,"tag":646,"props":1603,"children":1604},{"style":663},[1605],{"type":45,"value":841},{"type":40,"tag":646,"props":1607,"children":1608},{"style":674},[1609],{"type":45,"value":1610},"target_event_id",{"type":40,"tag":646,"props":1612,"children":1613},{"style":663},[1614],{"type":45,"value":682},{"type":40,"tag":646,"props":1616,"children":1617},{"style":663},[1618],{"type":45,"value":687},{"type":40,"tag":646,"props":1620,"children":1621},{"style":663},[1622],{"type":45,"value":671},{"type":40,"tag":646,"props":1624,"children":1625},{"style":694},[1626],{"type":45,"value":1627},"\u003Cgeneration_event_uuid>",{"type":40,"tag":646,"props":1629,"children":1630},{"style":663},[1631],{"type":45,"value":682},{"type":40,"tag":646,"props":1633,"children":1634},{"style":663},[1635],{"type":45,"value":872},{"type":40,"tag":646,"props":1637,"children":1638},{"class":648,"line":909},[1639,1643,1648,1652,1656,1660,1665,1669],{"type":40,"tag":646,"props":1640,"children":1641},{"style":663},[1642],{"type":45,"value":841},{"type":40,"tag":646,"props":1644,"children":1645},{"style":674},[1646],{"type":45,"value":1647},"timestamp",{"type":40,"tag":646,"props":1649,"children":1650},{"style":663},[1651],{"type":45,"value":682},{"type":40,"tag":646,"props":1653,"children":1654},{"style":663},[1655],{"type":45,"value":687},{"type":40,"tag":646,"props":1657,"children":1658},{"style":663},[1659],{"type":45,"value":671},{"type":40,"tag":646,"props":1661,"children":1662},{"style":694},[1663],{"type":45,"value":1664},"2026-04-01T19:39:20Z",{"type":40,"tag":646,"props":1666,"children":1667},{"style":663},[1668],{"type":45,"value":682},{"type":40,"tag":646,"props":1670,"children":1671},{"style":663},[1672],{"type":45,"value":872},{"type":40,"tag":646,"props":1674,"children":1675},{"class":648,"line":1410},[1676,1680,1685,1689,1693,1697,1701],{"type":40,"tag":646,"props":1677,"children":1678},{"style":663},[1679],{"type":45,"value":841},{"type":40,"tag":646,"props":1681,"children":1682},{"style":674},[1683],{"type":45,"value":1684},"event",{"type":40,"tag":646,"props":1686,"children":1687},{"style":663},[1688],{"type":45,"value":682},{"type":40,"tag":646,"props":1690,"children":1691},{"style":663},[1692],{"type":45,"value":687},{"type":40,"tag":646,"props":1694,"children":1695},{"style":663},[1696],{"type":45,"value":671},{"type":40,"tag":646,"props":1698,"children":1699},{"style":694},[1700],{"type":45,"value":59},{"type":40,"tag":646,"props":1702,"children":1703},{"style":663},[1704],{"type":45,"value":906},{"type":40,"tag":646,"props":1706,"children":1707},{"class":648,"line":1419},[1708],{"type":40,"tag":646,"props":1709,"children":1710},{"style":663},[1711],{"type":45,"value":915},{"type":40,"tag":48,"props":1713,"children":1714},{},[1715,1717,1722,1724,1730],{"type":45,"value":1716},"The ",{"type":40,"tag":54,"props":1718,"children":1720},{"className":1719},[],[1721],{"type":45,"value":1647},{"type":45,"value":1723}," is required for an efficient ClickHouse lookup of the target event.\nPass ",{"type":40,"tag":54,"props":1725,"children":1727},{"className":1726},[],[1728],{"type":45,"value":1729},"distinct_id",{"type":45,"value":1731}," if you have it — it speeds up the lookup further.",{"type":40,"tag":139,"props":1733,"children":1735},{"id":1734},"workflow-build-and-test-a-new-evaluator",[1736],{"type":45,"value":1737},"Workflow: build and test a new evaluator",{"type":40,"tag":628,"props":1739,"children":1741},{"id":1740},"hog-evaluator-deterministic-code-based",[1742],{"type":45,"value":1743},"Hog evaluator (deterministic, code-based)",{"type":40,"tag":48,"props":1745,"children":1746},{},[1747,1749,1755],{"type":45,"value":1748},"Reach for this first when the criterion is rule-based — it's cheaper, faster, and\nreproducible. Prototype with ",{"type":40,"tag":54,"props":1750,"children":1752},{"className":1751},[],[1753],{"type":45,"value":1754},"llma-evaluation-test-hog",{"type":45,"value":1756}," (no save):",{"type":40,"tag":635,"props":1758,"children":1760},{"className":637,"code":1759,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-test-hog\n{\n  \"source\": \"return event.properties.$ai_output_choices[1].content contains 'sorry';\",\n  \"sample_count\": 5,\n  \"allows_na\": false\n}\n",[1761],{"type":40,"tag":54,"props":1762,"children":1763},{"__ignoreMap":640},[1764,1772,1779,1816,1846,1871],{"type":40,"tag":646,"props":1765,"children":1766},{"class":648,"line":649},[1767],{"type":40,"tag":646,"props":1768,"children":1769},{"style":653},[1770],{"type":45,"value":1771},"posthog:llma-evaluation-test-hog\n",{"type":40,"tag":646,"props":1773,"children":1774},{"class":648,"line":659},[1775],{"type":40,"tag":646,"props":1776,"children":1777},{"style":663},[1778],{"type":45,"value":832},{"type":40,"tag":646,"props":1780,"children":1781},{"class":648,"line":835},[1782,1786,1791,1795,1799,1803,1808,1812],{"type":40,"tag":646,"props":1783,"children":1784},{"style":663},[1785],{"type":45,"value":841},{"type":40,"tag":646,"props":1787,"children":1788},{"style":674},[1789],{"type":45,"value":1790},"source",{"type":40,"tag":646,"props":1792,"children":1793},{"style":663},[1794],{"type":45,"value":682},{"type":40,"tag":646,"props":1796,"children":1797},{"style":663},[1798],{"type":45,"value":687},{"type":40,"tag":646,"props":1800,"children":1801},{"style":663},[1802],{"type":45,"value":671},{"type":40,"tag":646,"props":1804,"children":1805},{"style":694},[1806],{"type":45,"value":1807},"return event.properties.$ai_output_choices[1].content contains 'sorry';",{"type":40,"tag":646,"props":1809,"children":1810},{"style":663},[1811],{"type":45,"value":682},{"type":40,"tag":646,"props":1813,"children":1814},{"style":663},[1815],{"type":45,"value":872},{"type":40,"tag":646,"props":1817,"children":1818},{"class":648,"line":27},[1819,1823,1828,1832,1836,1842],{"type":40,"tag":646,"props":1820,"children":1821},{"style":663},[1822],{"type":45,"value":841},{"type":40,"tag":646,"props":1824,"children":1825},{"style":674},[1826],{"type":45,"value":1827},"sample_count",{"type":40,"tag":646,"props":1829,"children":1830},{"style":663},[1831],{"type":45,"value":682},{"type":40,"tag":646,"props":1833,"children":1834},{"style":663},[1835],{"type":45,"value":687},{"type":40,"tag":646,"props":1837,"children":1839},{"style":1838},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1840],{"type":45,"value":1841}," 5",{"type":40,"tag":646,"props":1843,"children":1844},{"style":663},[1845],{"type":45,"value":872},{"type":40,"tag":646,"props":1847,"children":1848},{"class":648,"line":909},[1849,1853,1858,1862,1866],{"type":40,"tag":646,"props":1850,"children":1851},{"style":663},[1852],{"type":45,"value":841},{"type":40,"tag":646,"props":1854,"children":1855},{"style":674},[1856],{"type":45,"value":1857},"allows_na",{"type":40,"tag":646,"props":1859,"children":1860},{"style":663},[1861],{"type":45,"value":682},{"type":40,"tag":646,"props":1863,"children":1864},{"style":663},[1865],{"type":45,"value":687},{"type":40,"tag":646,"props":1867,"children":1868},{"style":663},[1869],{"type":45,"value":1870}," false\n",{"type":40,"tag":646,"props":1872,"children":1873},{"class":648,"line":1410},[1874],{"type":40,"tag":646,"props":1875,"children":1876},{"style":663},[1877],{"type":45,"value":915},{"type":40,"tag":48,"props":1879,"children":1880},{},[1881,1883,1888,1890,1896],{"type":45,"value":1882},"The handler returns the boolean result for each of the most recent N ",{"type":40,"tag":54,"props":1884,"children":1886},{"className":1885},[],[1887],{"type":45,"value":59},{"type":45,"value":1889},"\nevents. Iterate on the source until it behaves as expected, then promote it via\n",{"type":40,"tag":54,"props":1891,"children":1893},{"className":1892},[],[1894],{"type":45,"value":1895},"llma-evaluation-create",{"type":45,"value":687},{"type":40,"tag":635,"props":1898,"children":1900},{"className":637,"code":1899,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-create\n{\n  \"name\": \"Output is valid JSON\",\n  \"description\": \"Fails when the assistant message can't be parsed as JSON\",\n  \"evaluation_type\": \"hog\",\n  \"evaluation_config\": {\n    \"source\": \"let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }\"\n  },\n  \"output_type\": \"boolean\",\n  \"enabled\": true\n}\n",[1901],{"type":40,"tag":54,"props":1902,"children":1903},{"__ignoreMap":640},[1904,1912,1919,1955,1991,2026,2051,2084,2092,2129,2153],{"type":40,"tag":646,"props":1905,"children":1906},{"class":648,"line":649},[1907],{"type":40,"tag":646,"props":1908,"children":1909},{"style":653},[1910],{"type":45,"value":1911},"posthog:llma-evaluation-create\n",{"type":40,"tag":646,"props":1913,"children":1914},{"class":648,"line":659},[1915],{"type":40,"tag":646,"props":1916,"children":1917},{"style":663},[1918],{"type":45,"value":832},{"type":40,"tag":646,"props":1920,"children":1921},{"class":648,"line":835},[1922,1926,1930,1934,1938,1942,1947,1951],{"type":40,"tag":646,"props":1923,"children":1924},{"style":663},[1925],{"type":45,"value":841},{"type":40,"tag":646,"props":1927,"children":1928},{"style":674},[1929],{"type":45,"value":752},{"type":40,"tag":646,"props":1931,"children":1932},{"style":663},[1933],{"type":45,"value":682},{"type":40,"tag":646,"props":1935,"children":1936},{"style":663},[1937],{"type":45,"value":687},{"type":40,"tag":646,"props":1939,"children":1940},{"style":663},[1941],{"type":45,"value":671},{"type":40,"tag":646,"props":1943,"children":1944},{"style":694},[1945],{"type":45,"value":1946},"Output is valid JSON",{"type":40,"tag":646,"props":1948,"children":1949},{"style":663},[1950],{"type":45,"value":682},{"type":40,"tag":646,"props":1952,"children":1953},{"style":663},[1954],{"type":45,"value":872},{"type":40,"tag":646,"props":1956,"children":1957},{"class":648,"line":27},[1958,1962,1966,1970,1974,1978,1983,1987],{"type":40,"tag":646,"props":1959,"children":1960},{"style":663},[1961],{"type":45,"value":841},{"type":40,"tag":646,"props":1963,"children":1964},{"style":674},[1965],{"type":45,"value":958},{"type":40,"tag":646,"props":1967,"children":1968},{"style":663},[1969],{"type":45,"value":682},{"type":40,"tag":646,"props":1971,"children":1972},{"style":663},[1973],{"type":45,"value":687},{"type":40,"tag":646,"props":1975,"children":1976},{"style":663},[1977],{"type":45,"value":671},{"type":40,"tag":646,"props":1979,"children":1980},{"style":694},[1981],{"type":45,"value":1982},"Fails when the assistant message can't be parsed as JSON",{"type":40,"tag":646,"props":1984,"children":1985},{"style":663},[1986],{"type":45,"value":682},{"type":40,"tag":646,"props":1988,"children":1989},{"style":663},[1990],{"type":45,"value":872},{"type":40,"tag":646,"props":1992,"children":1993},{"class":648,"line":909},[1994,1998,2002,2006,2010,2014,2018,2022],{"type":40,"tag":646,"props":1995,"children":1996},{"style":663},[1997],{"type":45,"value":841},{"type":40,"tag":646,"props":1999,"children":2000},{"style":674},[2001],{"type":45,"value":759},{"type":40,"tag":646,"props":2003,"children":2004},{"style":663},[2005],{"type":45,"value":682},{"type":40,"tag":646,"props":2007,"children":2008},{"style":663},[2009],{"type":45,"value":687},{"type":40,"tag":646,"props":2011,"children":2012},{"style":663},[2013],{"type":45,"value":671},{"type":40,"tag":646,"props":2015,"children":2016},{"style":694},[2017],{"type":45,"value":79},{"type":40,"tag":646,"props":2019,"children":2020},{"style":663},[2021],{"type":45,"value":682},{"type":40,"tag":646,"props":2023,"children":2024},{"style":663},[2025],{"type":45,"value":872},{"type":40,"tag":646,"props":2027,"children":2028},{"class":648,"line":1410},[2029,2033,2038,2042,2046],{"type":40,"tag":646,"props":2030,"children":2031},{"style":663},[2032],{"type":45,"value":841},{"type":40,"tag":646,"props":2034,"children":2035},{"style":674},[2036],{"type":45,"value":2037},"evaluation_config",{"type":40,"tag":646,"props":2039,"children":2040},{"style":663},[2041],{"type":45,"value":682},{"type":40,"tag":646,"props":2043,"children":2044},{"style":663},[2045],{"type":45,"value":687},{"type":40,"tag":646,"props":2047,"children":2048},{"style":663},[2049],{"type":45,"value":2050}," {\n",{"type":40,"tag":646,"props":2052,"children":2053},{"class":648,"line":1419},[2054,2059,2063,2067,2071,2075,2080],{"type":40,"tag":646,"props":2055,"children":2056},{"style":663},[2057],{"type":45,"value":2058},"    \"",{"type":40,"tag":646,"props":2060,"children":2061},{"style":1294},[2062],{"type":45,"value":1790},{"type":40,"tag":646,"props":2064,"children":2065},{"style":663},[2066],{"type":45,"value":682},{"type":40,"tag":646,"props":2068,"children":2069},{"style":663},[2070],{"type":45,"value":687},{"type":40,"tag":646,"props":2072,"children":2073},{"style":663},[2074],{"type":45,"value":671},{"type":40,"tag":646,"props":2076,"children":2077},{"style":694},[2078],{"type":45,"value":2079},"let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }",{"type":40,"tag":646,"props":2081,"children":2082},{"style":663},[2083],{"type":45,"value":906},{"type":40,"tag":646,"props":2085,"children":2086},{"class":648,"line":1428},[2087],{"type":40,"tag":646,"props":2088,"children":2089},{"style":663},[2090],{"type":45,"value":2091},"  },\n",{"type":40,"tag":646,"props":2093,"children":2094},{"class":648,"line":1437},[2095,2099,2104,2108,2112,2116,2121,2125],{"type":40,"tag":646,"props":2096,"children":2097},{"style":663},[2098],{"type":45,"value":841},{"type":40,"tag":646,"props":2100,"children":2101},{"style":674},[2102],{"type":45,"value":2103},"output_type",{"type":40,"tag":646,"props":2105,"children":2106},{"style":663},[2107],{"type":45,"value":682},{"type":40,"tag":646,"props":2109,"children":2110},{"style":663},[2111],{"type":45,"value":687},{"type":40,"tag":646,"props":2113,"children":2114},{"style":663},[2115],{"type":45,"value":671},{"type":40,"tag":646,"props":2117,"children":2118},{"style":694},[2119],{"type":45,"value":2120},"boolean",{"type":40,"tag":646,"props":2122,"children":2123},{"style":663},[2124],{"type":45,"value":682},{"type":40,"tag":646,"props":2126,"children":2127},{"style":663},[2128],{"type":45,"value":872},{"type":40,"tag":646,"props":2130,"children":2131},{"class":648,"line":1446},[2132,2136,2140,2144,2148],{"type":40,"tag":646,"props":2133,"children":2134},{"style":663},[2135],{"type":45,"value":841},{"type":40,"tag":646,"props":2137,"children":2138},{"style":674},[2139],{"type":45,"value":715},{"type":40,"tag":646,"props":2141,"children":2142},{"style":663},[2143],{"type":45,"value":682},{"type":40,"tag":646,"props":2145,"children":2146},{"style":663},[2147],{"type":45,"value":687},{"type":40,"tag":646,"props":2149,"children":2150},{"style":663},[2151],{"type":45,"value":2152}," true\n",{"type":40,"tag":646,"props":2154,"children":2155},{"class":648,"line":1455},[2156],{"type":40,"tag":646,"props":2157,"children":2158},{"style":663},[2159],{"type":45,"value":915},{"type":40,"tag":48,"props":2161,"children":2162},{},[2163],{"type":45,"value":2164},"Hog evaluators have full access to the event and its properties — common patterns\ninclude schema validation, length\u002Ftoken limits, regex matches, and tool-call shape\nchecks. Because they're deterministic, results are reproducible across reruns and\ntrivially diff-able.",{"type":40,"tag":628,"props":2166,"children":2168},{"id":2167},"llm-judge-evaluator-subjective-prompt-based",[2169],{"type":45,"value":2170},"LLM-judge evaluator (subjective, prompt-based)",{"type":40,"tag":48,"props":2172,"children":2173},{},[2174,2176,2181,2183,2189,2191,2197,2199,2205,2207,2213],{"type":45,"value":2175},"Use this when the criterion is fuzzy and a code rule would be brittle (tone, factuality,\nhelpfulness, on-topic-ness). There's no equivalent of ",{"type":40,"tag":54,"props":2177,"children":2179},{"className":2178},[],[2180],{"type":45,"value":1754},{"type":45,"value":2182}," for LLM\njudges — the typical loop is to create the evaluator with ",{"type":40,"tag":54,"props":2184,"children":2186},{"className":2185},[],[2187],{"type":45,"value":2188},"enabled: false",{"type":45,"value":2190},", run it\nmanually against a handful of representative generations via ",{"type":40,"tag":54,"props":2192,"children":2194},{"className":2193},[],[2195],{"type":45,"value":2196},"llma-evaluation-run",{"type":45,"value":2198},", inspect\nthe results, refine the prompt with ",{"type":40,"tag":54,"props":2200,"children":2202},{"className":2201},[],[2203],{"type":45,"value":2204},"llma-evaluation-update",{"type":45,"value":2206},", and then flip ",{"type":40,"tag":54,"props":2208,"children":2210},{"className":2209},[],[2211],{"type":45,"value":2212},"enabled: true",{"type":45,"value":2214},"\nwhen you're satisfied:",{"type":40,"tag":635,"props":2216,"children":2218},{"className":637,"code":2217,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-create\n{\n  \"name\": \"Response stays on-topic\",\n  \"description\": \"LLM judge — fails if the assistant changes topic from the user's question\",\n  \"evaluation_type\": \"llm_judge\",\n  \"evaluation_config\": {\n    \"prompt\": \"You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N\u002FA if the user did not actually ask a question.\"\n  },\n  \"output_type\": \"boolean\",\n  \"output_config\": { \"allows_na\": true },\n  \"model_configuration\": {\n    \"provider\": \"openai\",\n    \"model\": \"gpt-5-mini\"\n  },\n  \"enabled\": false\n}\n",[2219],{"type":40,"tag":54,"props":2220,"children":2221},{"__ignoreMap":640},[2222,2229,2236,2272,2308,2343,2366,2399,2406,2441,2490,2514,2551,2584,2591,2614],{"type":40,"tag":646,"props":2223,"children":2224},{"class":648,"line":649},[2225],{"type":40,"tag":646,"props":2226,"children":2227},{"style":653},[2228],{"type":45,"value":1911},{"type":40,"tag":646,"props":2230,"children":2231},{"class":648,"line":659},[2232],{"type":40,"tag":646,"props":2233,"children":2234},{"style":663},[2235],{"type":45,"value":832},{"type":40,"tag":646,"props":2237,"children":2238},{"class":648,"line":835},[2239,2243,2247,2251,2255,2259,2264,2268],{"type":40,"tag":646,"props":2240,"children":2241},{"style":663},[2242],{"type":45,"value":841},{"type":40,"tag":646,"props":2244,"children":2245},{"style":674},[2246],{"type":45,"value":752},{"type":40,"tag":646,"props":2248,"children":2249},{"style":663},[2250],{"type":45,"value":682},{"type":40,"tag":646,"props":2252,"children":2253},{"style":663},[2254],{"type":45,"value":687},{"type":40,"tag":646,"props":2256,"children":2257},{"style":663},[2258],{"type":45,"value":671},{"type":40,"tag":646,"props":2260,"children":2261},{"style":694},[2262],{"type":45,"value":2263},"Response stays on-topic",{"type":40,"tag":646,"props":2265,"children":2266},{"style":663},[2267],{"type":45,"value":682},{"type":40,"tag":646,"props":2269,"children":2270},{"style":663},[2271],{"type":45,"value":872},{"type":40,"tag":646,"props":2273,"children":2274},{"class":648,"line":27},[2275,2279,2283,2287,2291,2295,2300,2304],{"type":40,"tag":646,"props":2276,"children":2277},{"style":663},[2278],{"type":45,"value":841},{"type":40,"tag":646,"props":2280,"children":2281},{"style":674},[2282],{"type":45,"value":958},{"type":40,"tag":646,"props":2284,"children":2285},{"style":663},[2286],{"type":45,"value":682},{"type":40,"tag":646,"props":2288,"children":2289},{"style":663},[2290],{"type":45,"value":687},{"type":40,"tag":646,"props":2292,"children":2293},{"style":663},[2294],{"type":45,"value":671},{"type":40,"tag":646,"props":2296,"children":2297},{"style":694},[2298],{"type":45,"value":2299},"LLM judge — fails if the assistant changes topic from the user's question",{"type":40,"tag":646,"props":2301,"children":2302},{"style":663},[2303],{"type":45,"value":682},{"type":40,"tag":646,"props":2305,"children":2306},{"style":663},[2307],{"type":45,"value":872},{"type":40,"tag":646,"props":2309,"children":2310},{"class":648,"line":909},[2311,2315,2319,2323,2327,2331,2335,2339],{"type":40,"tag":646,"props":2312,"children":2313},{"style":663},[2314],{"type":45,"value":841},{"type":40,"tag":646,"props":2316,"children":2317},{"style":674},[2318],{"type":45,"value":759},{"type":40,"tag":646,"props":2320,"children":2321},{"style":663},[2322],{"type":45,"value":682},{"type":40,"tag":646,"props":2324,"children":2325},{"style":663},[2326],{"type":45,"value":687},{"type":40,"tag":646,"props":2328,"children":2329},{"style":663},[2330],{"type":45,"value":671},{"type":40,"tag":646,"props":2332,"children":2333},{"style":694},[2334],{"type":45,"value":109},{"type":40,"tag":646,"props":2336,"children":2337},{"style":663},[2338],{"type":45,"value":682},{"type":40,"tag":646,"props":2340,"children":2341},{"style":663},[2342],{"type":45,"value":872},{"type":40,"tag":646,"props":2344,"children":2345},{"class":648,"line":1410},[2346,2350,2354,2358,2362],{"type":40,"tag":646,"props":2347,"children":2348},{"style":663},[2349],{"type":45,"value":841},{"type":40,"tag":646,"props":2351,"children":2352},{"style":674},[2353],{"type":45,"value":2037},{"type":40,"tag":646,"props":2355,"children":2356},{"style":663},[2357],{"type":45,"value":682},{"type":40,"tag":646,"props":2359,"children":2360},{"style":663},[2361],{"type":45,"value":687},{"type":40,"tag":646,"props":2363,"children":2364},{"style":663},[2365],{"type":45,"value":2050},{"type":40,"tag":646,"props":2367,"children":2368},{"class":648,"line":1419},[2369,2373,2378,2382,2386,2390,2395],{"type":40,"tag":646,"props":2370,"children":2371},{"style":663},[2372],{"type":45,"value":2058},{"type":40,"tag":646,"props":2374,"children":2375},{"style":1294},[2376],{"type":45,"value":2377},"prompt",{"type":40,"tag":646,"props":2379,"children":2380},{"style":663},[2381],{"type":45,"value":682},{"type":40,"tag":646,"props":2383,"children":2384},{"style":663},[2385],{"type":45,"value":687},{"type":40,"tag":646,"props":2387,"children":2388},{"style":663},[2389],{"type":45,"value":671},{"type":40,"tag":646,"props":2391,"children":2392},{"style":694},[2393],{"type":45,"value":2394},"You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N\u002FA if the user did not actually ask a question.",{"type":40,"tag":646,"props":2396,"children":2397},{"style":663},[2398],{"type":45,"value":906},{"type":40,"tag":646,"props":2400,"children":2401},{"class":648,"line":1428},[2402],{"type":40,"tag":646,"props":2403,"children":2404},{"style":663},[2405],{"type":45,"value":2091},{"type":40,"tag":646,"props":2407,"children":2408},{"class":648,"line":1437},[2409,2413,2417,2421,2425,2429,2433,2437],{"type":40,"tag":646,"props":2410,"children":2411},{"style":663},[2412],{"type":45,"value":841},{"type":40,"tag":646,"props":2414,"children":2415},{"style":674},[2416],{"type":45,"value":2103},{"type":40,"tag":646,"props":2418,"children":2419},{"style":663},[2420],{"type":45,"value":682},{"type":40,"tag":646,"props":2422,"children":2423},{"style":663},[2424],{"type":45,"value":687},{"type":40,"tag":646,"props":2426,"children":2427},{"style":663},[2428],{"type":45,"value":671},{"type":40,"tag":646,"props":2430,"children":2431},{"style":694},[2432],{"type":45,"value":2120},{"type":40,"tag":646,"props":2434,"children":2435},{"style":663},[2436],{"type":45,"value":682},{"type":40,"tag":646,"props":2438,"children":2439},{"style":663},[2440],{"type":45,"value":872},{"type":40,"tag":646,"props":2442,"children":2443},{"class":648,"line":1446},[2444,2448,2453,2457,2461,2465,2469,2473,2477,2481,2485],{"type":40,"tag":646,"props":2445,"children":2446},{"style":663},[2447],{"type":45,"value":841},{"type":40,"tag":646,"props":2449,"children":2450},{"style":674},[2451],{"type":45,"value":2452},"output_config",{"type":40,"tag":646,"props":2454,"children":2455},{"style":663},[2456],{"type":45,"value":682},{"type":40,"tag":646,"props":2458,"children":2459},{"style":663},[2460],{"type":45,"value":687},{"type":40,"tag":646,"props":2462,"children":2463},{"style":663},[2464],{"type":45,"value":1287},{"type":40,"tag":646,"props":2466,"children":2467},{"style":663},[2468],{"type":45,"value":671},{"type":40,"tag":646,"props":2470,"children":2471},{"style":1294},[2472],{"type":45,"value":1857},{"type":40,"tag":646,"props":2474,"children":2475},{"style":663},[2476],{"type":45,"value":682},{"type":40,"tag":646,"props":2478,"children":2479},{"style":663},[2480],{"type":45,"value":687},{"type":40,"tag":646,"props":2482,"children":2483},{"style":663},[2484],{"type":45,"value":728},{"type":40,"tag":646,"props":2486,"children":2487},{"style":663},[2488],{"type":45,"value":2489}," },\n",{"type":40,"tag":646,"props":2491,"children":2492},{"class":648,"line":1455},[2493,2497,2502,2506,2510],{"type":40,"tag":646,"props":2494,"children":2495},{"style":663},[2496],{"type":45,"value":841},{"type":40,"tag":646,"props":2498,"children":2499},{"style":674},[2500],{"type":45,"value":2501},"model_configuration",{"type":40,"tag":646,"props":2503,"children":2504},{"style":663},[2505],{"type":45,"value":682},{"type":40,"tag":646,"props":2507,"children":2508},{"style":663},[2509],{"type":45,"value":687},{"type":40,"tag":646,"props":2511,"children":2512},{"style":663},[2513],{"type":45,"value":2050},{"type":40,"tag":646,"props":2515,"children":2516},{"class":648,"line":1464},[2517,2521,2526,2530,2534,2538,2543,2547],{"type":40,"tag":646,"props":2518,"children":2519},{"style":663},[2520],{"type":45,"value":2058},{"type":40,"tag":646,"props":2522,"children":2523},{"style":1294},[2524],{"type":45,"value":2525},"provider",{"type":40,"tag":646,"props":2527,"children":2528},{"style":663},[2529],{"type":45,"value":682},{"type":40,"tag":646,"props":2531,"children":2532},{"style":663},[2533],{"type":45,"value":687},{"type":40,"tag":646,"props":2535,"children":2536},{"style":663},[2537],{"type":45,"value":671},{"type":40,"tag":646,"props":2539,"children":2540},{"style":694},[2541],{"type":45,"value":2542},"openai",{"type":40,"tag":646,"props":2544,"children":2545},{"style":663},[2546],{"type":45,"value":682},{"type":40,"tag":646,"props":2548,"children":2549},{"style":663},[2550],{"type":45,"value":872},{"type":40,"tag":646,"props":2552,"children":2553},{"class":648,"line":1473},[2554,2558,2563,2567,2571,2575,2580],{"type":40,"tag":646,"props":2555,"children":2556},{"style":663},[2557],{"type":45,"value":2058},{"type":40,"tag":646,"props":2559,"children":2560},{"style":1294},[2561],{"type":45,"value":2562},"model",{"type":40,"tag":646,"props":2564,"children":2565},{"style":663},[2566],{"type":45,"value":682},{"type":40,"tag":646,"props":2568,"children":2569},{"style":663},[2570],{"type":45,"value":687},{"type":40,"tag":646,"props":2572,"children":2573},{"style":663},[2574],{"type":45,"value":671},{"type":40,"tag":646,"props":2576,"children":2577},{"style":694},[2578],{"type":45,"value":2579},"gpt-5-mini",{"type":40,"tag":646,"props":2581,"children":2582},{"style":663},[2583],{"type":45,"value":906},{"type":40,"tag":646,"props":2585,"children":2586},{"class":648,"line":1482},[2587],{"type":40,"tag":646,"props":2588,"children":2589},{"style":663},[2590],{"type":45,"value":2091},{"type":40,"tag":646,"props":2592,"children":2593},{"class":648,"line":1491},[2594,2598,2602,2606,2610],{"type":40,"tag":646,"props":2595,"children":2596},{"style":663},[2597],{"type":45,"value":841},{"type":40,"tag":646,"props":2599,"children":2600},{"style":674},[2601],{"type":45,"value":715},{"type":40,"tag":646,"props":2603,"children":2604},{"style":663},[2605],{"type":45,"value":682},{"type":40,"tag":646,"props":2607,"children":2608},{"style":663},[2609],{"type":45,"value":687},{"type":40,"tag":646,"props":2611,"children":2612},{"style":663},[2613],{"type":45,"value":1870},{"type":40,"tag":646,"props":2615,"children":2616},{"class":648,"line":1500},[2617],{"type":40,"tag":646,"props":2618,"children":2619},{"style":663},[2620],{"type":45,"value":915},{"type":40,"tag":48,"props":2622,"children":2623},{},[2624],{"type":45,"value":2625},"Then dry-run against a known-good and a known-bad generation:",{"type":40,"tag":635,"props":2627,"children":2629},{"className":637,"code":2628,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-run\n{\n  \"evaluationId\": \"\u003Cnew_eval_uuid>\",\n  \"target_event_id\": \"\u003Cgeneration_uuid>\",\n  \"timestamp\": \"2026-04-01T19:39:20Z\"\n}\n",[2630],{"type":40,"tag":54,"props":2631,"children":2632},{"__ignoreMap":640},[2633,2640,2647,2683,2719,2750],{"type":40,"tag":646,"props":2634,"children":2635},{"class":648,"line":649},[2636],{"type":40,"tag":646,"props":2637,"children":2638},{"style":653},[2639],{"type":45,"value":1554},{"type":40,"tag":646,"props":2641,"children":2642},{"class":648,"line":659},[2643],{"type":40,"tag":646,"props":2644,"children":2645},{"style":663},[2646],{"type":45,"value":832},{"type":40,"tag":646,"props":2648,"children":2649},{"class":648,"line":835},[2650,2654,2658,2662,2666,2670,2675,2679],{"type":40,"tag":646,"props":2651,"children":2652},{"style":663},[2653],{"type":45,"value":841},{"type":40,"tag":646,"props":2655,"children":2656},{"style":674},[2657],{"type":45,"value":1573},{"type":40,"tag":646,"props":2659,"children":2660},{"style":663},[2661],{"type":45,"value":682},{"type":40,"tag":646,"props":2663,"children":2664},{"style":663},[2665],{"type":45,"value":687},{"type":40,"tag":646,"props":2667,"children":2668},{"style":663},[2669],{"type":45,"value":671},{"type":40,"tag":646,"props":2671,"children":2672},{"style":694},[2673],{"type":45,"value":2674},"\u003Cnew_eval_uuid>",{"type":40,"tag":646,"props":2676,"children":2677},{"style":663},[2678],{"type":45,"value":682},{"type":40,"tag":646,"props":2680,"children":2681},{"style":663},[2682],{"type":45,"value":872},{"type":40,"tag":646,"props":2684,"children":2685},{"class":648,"line":27},[2686,2690,2694,2698,2702,2706,2711,2715],{"type":40,"tag":646,"props":2687,"children":2688},{"style":663},[2689],{"type":45,"value":841},{"type":40,"tag":646,"props":2691,"children":2692},{"style":674},[2693],{"type":45,"value":1610},{"type":40,"tag":646,"props":2695,"children":2696},{"style":663},[2697],{"type":45,"value":682},{"type":40,"tag":646,"props":2699,"children":2700},{"style":663},[2701],{"type":45,"value":687},{"type":40,"tag":646,"props":2703,"children":2704},{"style":663},[2705],{"type":45,"value":671},{"type":40,"tag":646,"props":2707,"children":2708},{"style":694},[2709],{"type":45,"value":2710},"\u003Cgeneration_uuid>",{"type":40,"tag":646,"props":2712,"children":2713},{"style":663},[2714],{"type":45,"value":682},{"type":40,"tag":646,"props":2716,"children":2717},{"style":663},[2718],{"type":45,"value":872},{"type":40,"tag":646,"props":2720,"children":2721},{"class":648,"line":909},[2722,2726,2730,2734,2738,2742,2746],{"type":40,"tag":646,"props":2723,"children":2724},{"style":663},[2725],{"type":45,"value":841},{"type":40,"tag":646,"props":2727,"children":2728},{"style":674},[2729],{"type":45,"value":1647},{"type":40,"tag":646,"props":2731,"children":2732},{"style":663},[2733],{"type":45,"value":682},{"type":40,"tag":646,"props":2735,"children":2736},{"style":663},[2737],{"type":45,"value":687},{"type":40,"tag":646,"props":2739,"children":2740},{"style":663},[2741],{"type":45,"value":671},{"type":40,"tag":646,"props":2743,"children":2744},{"style":694},[2745],{"type":45,"value":1664},{"type":40,"tag":646,"props":2747,"children":2748},{"style":663},[2749],{"type":45,"value":906},{"type":40,"tag":646,"props":2751,"children":2752},{"class":648,"line":1410},[2753],{"type":40,"tag":646,"props":2754,"children":2755},{"style":663},[2756],{"type":45,"value":915},{"type":40,"tag":48,"props":2758,"children":2759},{},[2760],{"type":45,"value":2761},"LLM judges require organisation AI data processing approval. Hog evaluators do not.",{"type":40,"tag":139,"props":2763,"children":2765},{"id":2764},"workflow-manage-the-evaluation-lifecycle",[2766],{"type":45,"value":2767},"Workflow: manage the evaluation lifecycle",{"type":40,"tag":146,"props":2769,"children":2770},{},[2771,2786],{"type":40,"tag":150,"props":2772,"children":2773},{},[2774],{"type":40,"tag":154,"props":2775,"children":2776},{},[2777,2782],{"type":40,"tag":158,"props":2778,"children":2779},{},[2780],{"type":45,"value":2781},"Action",{"type":40,"tag":158,"props":2783,"children":2784},{},[2785],{"type":45,"value":162},{"type":40,"tag":169,"props":2787,"children":2788},{},[2789,2819,2855,2887,2910,2932],{"type":40,"tag":154,"props":2790,"children":2791},{},[2792,2797],{"type":40,"tag":176,"props":2793,"children":2794},{},[2795],{"type":45,"value":2796},"Add a Hog evaluator",{"type":40,"tag":176,"props":2798,"children":2799},{},[2800,2805,2807,2813,2814],{"type":40,"tag":54,"props":2801,"children":2803},{"className":2802},[],[2804],{"type":45,"value":1895},{"type":45,"value":2806}," with ",{"type":40,"tag":54,"props":2808,"children":2810},{"className":2809},[],[2811],{"type":45,"value":2812},"evaluation_type: \"hog\"",{"type":45,"value":619},{"type":40,"tag":54,"props":2815,"children":2817},{"className":2816},[],[2818],{"type":45,"value":789},{"type":40,"tag":154,"props":2820,"children":2821},{},[2822,2827],{"type":40,"tag":176,"props":2823,"children":2824},{},[2825],{"type":45,"value":2826},"Add an LLM-judge evaluator",{"type":40,"tag":176,"props":2828,"children":2829},{},[2830,2835,2836,2842,2843,2848,2850],{"type":40,"tag":54,"props":2831,"children":2833},{"className":2832},[],[2834],{"type":45,"value":1895},{"type":45,"value":2806},{"type":40,"tag":54,"props":2837,"children":2839},{"className":2838},[],[2840],{"type":45,"value":2841},"evaluation_type: \"llm_judge\"",{"type":45,"value":746},{"type":40,"tag":54,"props":2844,"children":2846},{"className":2845},[],[2847],{"type":45,"value":773},{"type":45,"value":2849},", and a ",{"type":40,"tag":54,"props":2851,"children":2853},{"className":2852},[],[2854],{"type":45,"value":2501},{"type":40,"tag":154,"props":2856,"children":2857},{},[2858,2863],{"type":40,"tag":176,"props":2859,"children":2860},{},[2861],{"type":45,"value":2862},"Tweak the source or prompt",{"type":40,"tag":176,"props":2864,"children":2865},{},[2866,2871,2873,2878,2880,2885],{"type":40,"tag":54,"props":2867,"children":2869},{"className":2868},[],[2870],{"type":45,"value":2204},{"type":45,"value":2872}," (edits ",{"type":40,"tag":54,"props":2874,"children":2876},{"className":2875},[],[2877],{"type":45,"value":789},{"type":45,"value":2879}," for Hog, ",{"type":40,"tag":54,"props":2881,"children":2883},{"className":2882},[],[2884],{"type":45,"value":773},{"type":45,"value":2886}," for LLM judge)",{"type":40,"tag":154,"props":2888,"children":2889},{},[2890,2895],{"type":40,"tag":176,"props":2891,"children":2892},{},[2893],{"type":45,"value":2894},"Toggle N\u002FA handling",{"type":40,"tag":176,"props":2896,"children":2897},{},[2898,2903,2904],{"type":40,"tag":54,"props":2899,"children":2901},{"className":2900},[],[2902],{"type":45,"value":2204},{"type":45,"value":2806},{"type":40,"tag":54,"props":2905,"children":2907},{"className":2906},[],[2908],{"type":45,"value":2909},"output_config.allows_na",{"type":40,"tag":154,"props":2911,"children":2912},{},[2913,2918],{"type":40,"tag":176,"props":2914,"children":2915},{},[2916],{"type":45,"value":2917},"Disable temporarily",{"type":40,"tag":176,"props":2919,"children":2920},{},[2921,2926,2927],{"type":40,"tag":54,"props":2922,"children":2924},{"className":2923},[],[2925],{"type":45,"value":2204},{"type":45,"value":2806},{"type":40,"tag":54,"props":2928,"children":2930},{"className":2929},[],[2931],{"type":45,"value":2188},{"type":40,"tag":154,"props":2933,"children":2934},{},[2935,2940],{"type":40,"tag":176,"props":2936,"children":2937},{},[2938],{"type":45,"value":2939},"Remove",{"type":40,"tag":176,"props":2941,"children":2942},{},[2943,2949,2951,2957],{"type":40,"tag":54,"props":2944,"children":2946},{"className":2945},[],[2947],{"type":45,"value":2948},"llma-evaluation-delete",{"type":45,"value":2950}," (soft-delete via PATCH ",{"type":40,"tag":54,"props":2952,"children":2954},{"className":2953},[],[2955],{"type":45,"value":2956},"{deleted: true}",{"type":45,"value":2958},")",{"type":40,"tag":48,"props":2960,"children":2961},{},[2962,2967,2969,2975,2977,2983,2985,2990],{"type":40,"tag":54,"props":2963,"children":2965},{"className":2964},[],[2966],{"type":45,"value":109},{"type":45,"value":2968}," evaluations require AI data processing approval at the org level\n(",{"type":40,"tag":54,"props":2970,"children":2972},{"className":2971},[],[2973],{"type":45,"value":2974},"is_ai_data_processing_approved",{"type":45,"value":2976},"). The same gate applies to\n",{"type":40,"tag":54,"props":2978,"children":2980},{"className":2979},[],[2981],{"type":45,"value":2982},"llma-evaluation-summary-create",{"type":45,"value":2984},". Hog evaluations do ",{"type":40,"tag":71,"props":2986,"children":2987},{},[2988],{"type":45,"value":2989},"not",{"type":45,"value":2991}," require this gate\n— they run as plain code on the ingestion pipeline.",{"type":40,"tag":139,"props":2993,"children":2995},{"id":2994},"when-to-use-hog-vs-llm-judge",[2996],{"type":45,"value":2997},"When to use Hog vs LLM judge",{"type":40,"tag":48,"props":2999,"children":3000},{},[3001,3003,3008],{"type":45,"value":3002},"Reach for ",{"type":40,"tag":71,"props":3004,"children":3005},{},[3006],{"type":45,"value":3007},"Hog",{"type":45,"value":3009}," by default. Switch to LLM judge only when the criterion can't be\nexpressed as code.",{"type":40,"tag":146,"props":3011,"children":3012},{},[3013,3029],{"type":40,"tag":150,"props":3014,"children":3015},{},[3016],{"type":40,"tag":154,"props":3017,"children":3018},{},[3019,3024],{"type":40,"tag":158,"props":3020,"children":3021},{},[3022],{"type":45,"value":3023},"Use Hog when…",{"type":40,"tag":158,"props":3025,"children":3026},{},[3027],{"type":45,"value":3028},"Use LLM judge when…",{"type":40,"tag":169,"props":3030,"children":3031},{},[3032,3045,3058,3071,3084,3097],{"type":40,"tag":154,"props":3033,"children":3034},{},[3035,3040],{"type":40,"tag":176,"props":3036,"children":3037},{},[3038],{"type":45,"value":3039},"The check is structural (JSON parses, schema matches)",{"type":40,"tag":176,"props":3041,"children":3042},{},[3043],{"type":45,"value":3044},"The check is about meaning (on-topic, helpful, factual)",{"type":40,"tag":154,"props":3046,"children":3047},{},[3048,3053],{"type":40,"tag":176,"props":3049,"children":3050},{},[3051],{"type":45,"value":3052},"You need a deterministic, reproducible result",{"type":40,"tag":176,"props":3054,"children":3055},{},[3056],{"type":45,"value":3057},"A small amount of judgement variability is acceptable",{"type":40,"tag":154,"props":3059,"children":3060},{},[3061,3066],{"type":40,"tag":176,"props":3062,"children":3063},{},[3064],{"type":45,"value":3065},"The criterion is cheap to compute",{"type":40,"tag":176,"props":3067,"children":3068},{},[3069],{"type":45,"value":3070},"The criterion requires reading and understanding text",{"type":40,"tag":154,"props":3072,"children":3073},{},[3074,3079],{"type":40,"tag":176,"props":3075,"children":3076},{},[3077],{"type":45,"value":3078},"You can't get AI data processing approval",{"type":40,"tag":176,"props":3080,"children":3081},{},[3082],{"type":45,"value":3083},"You have approval and the criterion is genuinely fuzzy",{"type":40,"tag":154,"props":3085,"children":3086},{},[3087,3092],{"type":40,"tag":176,"props":3088,"children":3089},{},[3090],{"type":45,"value":3091},"You need to enforce a hard limit (length, cost, etc.)",{"type":40,"tag":176,"props":3093,"children":3094},{},[3095],{"type":45,"value":3096},"You need to rate a quality dimension",{"type":40,"tag":154,"props":3098,"children":3099},{},[3100,3105],{"type":40,"tag":176,"props":3101,"children":3102},{},[3103],{"type":45,"value":3104},"You want sub-millisecond evaluation",{"type":40,"tag":176,"props":3106,"children":3107},{},[3108],{"type":45,"value":3109},"A few hundred milliseconds + LLM cost are acceptable",{"type":40,"tag":48,"props":3111,"children":3112},{},[3113,3115,3120,3122,3128],{"type":45,"value":3114},"A common pattern is to ",{"type":40,"tag":71,"props":3116,"children":3117},{},[3118],{"type":45,"value":3119},"layer them",{"type":45,"value":3121},": a Hog evaluator gates obvious format\u002Flength\nviolations cheaply, and an LLM-judge evaluator only fires on the generations that pass\nthe Hog gate (via ",{"type":40,"tag":54,"props":3123,"children":3125},{"className":3124},[],[3126],{"type":45,"value":3127},"conditions",{"type":45,"value":3129},").",{"type":40,"tag":139,"props":3131,"children":3133},{"id":3132},"investigation-patterns",[3134],{"type":45,"value":3135},"Investigation patterns",{"type":40,"tag":48,"props":3137,"children":3138},{},[3139,3141,3146,3148,3153,3155,3160],{"type":45,"value":3140},"The summarisation tool works the same way regardless of whether the evaluator is ",{"type":40,"tag":54,"props":3142,"children":3144},{"className":3143},[],[3145],{"type":45,"value":79},{"type":45,"value":3147},"\nor ",{"type":40,"tag":54,"props":3149,"children":3151},{"className":3150},[],[3152],{"type":45,"value":109},{"type":45,"value":3154}," — it analyses the resulting ",{"type":40,"tag":54,"props":3156,"children":3158},{"className":3157},[],[3159],{"type":45,"value":122},{"type":45,"value":3161}," events, not the evaluator\nitself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is\nidentical.",{"type":40,"tag":628,"props":3163,"children":3165},{"id":3164},"why-is-evaluation-x-suddenly-failing-more",[3166],{"type":45,"value":3167},"\"Why is evaluation X suddenly failing more?\"",{"type":40,"tag":3169,"props":3170,"children":3171},"ol",{},[3172,3196,3213,3286],{"type":40,"tag":67,"props":3173,"children":3174},{},[3175,3181,3183,3188,3189,3194],{"type":40,"tag":54,"props":3176,"children":3178},{"className":3177},[],[3179],{"type":45,"value":3180},"llma-evaluation-list",{"type":45,"value":3182}," — confirm the evaluation is still enabled and unchanged\n(compare ",{"type":40,"tag":54,"props":3184,"children":3186},{"className":3185},[],[3187],{"type":45,"value":789},{"type":45,"value":230},{"type":40,"tag":54,"props":3190,"children":3192},{"className":3191},[],[3193],{"type":45,"value":773},{"type":45,"value":3195}," to the version you\nexpect)",{"type":40,"tag":67,"props":3197,"children":3198},{},[3199,3204,3205,3211],{"type":40,"tag":54,"props":3200,"children":3202},{"className":3201},[],[3203],{"type":45,"value":2982},{"type":45,"value":2806},{"type":40,"tag":54,"props":3206,"children":3208},{"className":3207},[],[3209],{"type":45,"value":3210},"filter: \"fail\"",{"type":45,"value":3212}," — get the dominant\nfailure patterns and example IDs",{"type":40,"tag":67,"props":3214,"children":3215},{},[3216,3218],{"type":45,"value":3217},"SQL count of fails per day to confirm the regression window:",{"type":40,"tag":635,"props":3219,"children":3221},{"className":1361,"code":3220,"language":1363,"meta":640,"style":640},"SELECT toDate(timestamp) AS day, count() AS fails\nFROM events\nWHERE event = '$ai_evaluation'\n    AND properties.$ai_evaluation_id = '\u003Cuuid>'\n    AND properties.$ai_evaluation_result = false\n    AND timestamp >= now() - INTERVAL 30 DAY\nGROUP BY day\nORDER BY day\n",[3222],{"type":40,"tag":54,"props":3223,"children":3224},{"__ignoreMap":640},[3225,3233,3240,3247,3255,3262,3270,3278],{"type":40,"tag":646,"props":3226,"children":3227},{"class":648,"line":649},[3228],{"type":40,"tag":646,"props":3229,"children":3230},{},[3231],{"type":45,"value":3232},"SELECT toDate(timestamp) AS day, count() AS fails\n",{"type":40,"tag":646,"props":3234,"children":3235},{"class":648,"line":659},[3236],{"type":40,"tag":646,"props":3237,"children":3238},{},[3239],{"type":45,"value":1425},{"type":40,"tag":646,"props":3241,"children":3242},{"class":648,"line":835},[3243],{"type":40,"tag":646,"props":3244,"children":3245},{},[3246],{"type":45,"value":1434},{"type":40,"tag":646,"props":3248,"children":3249},{"class":648,"line":27},[3250],{"type":40,"tag":646,"props":3251,"children":3252},{},[3253],{"type":45,"value":3254},"    AND properties.$ai_evaluation_id = '\u003Cuuid>'\n",{"type":40,"tag":646,"props":3256,"children":3257},{"class":648,"line":909},[3258],{"type":40,"tag":646,"props":3259,"children":3260},{},[3261],{"type":45,"value":1452},{"type":40,"tag":646,"props":3263,"children":3264},{"class":648,"line":1410},[3265],{"type":40,"tag":646,"props":3266,"children":3267},{},[3268],{"type":45,"value":3269},"    AND timestamp >= now() - INTERVAL 30 DAY\n",{"type":40,"tag":646,"props":3271,"children":3272},{"class":648,"line":1419},[3273],{"type":40,"tag":646,"props":3274,"children":3275},{},[3276],{"type":45,"value":3277},"GROUP BY day\n",{"type":40,"tag":646,"props":3279,"children":3280},{"class":648,"line":1428},[3281],{"type":40,"tag":646,"props":3282,"children":3283},{},[3284],{"type":45,"value":3285},"ORDER BY day\n",{"type":40,"tag":67,"props":3287,"children":3288},{},[3289,3291],{"type":45,"value":3290},"Drill into a representative trace per pattern via ",{"type":40,"tag":54,"props":3292,"children":3294},{"className":3293},[],[3295],{"type":45,"value":3296},"query-llm-trace",{"type":40,"tag":628,"props":3298,"children":3300},{"id":3299},"are-passes-and-fails-caused-by-the-same-root-content",[3301],{"type":45,"value":3302},"\"Are passes and fails caused by the same root content?\"",{"type":40,"tag":3169,"props":3304,"children":3305},{},[3306,3324],{"type":40,"tag":67,"props":3307,"children":3308},{},[3309,3311,3317,3319],{"type":45,"value":3310},"Generate two summaries: one with ",{"type":40,"tag":54,"props":3312,"children":3314},{"className":3313},[],[3315],{"type":45,"value":3316},"filter: \"pass\"",{"type":45,"value":3318},", one with ",{"type":40,"tag":54,"props":3320,"children":3322},{"className":3321},[],[3323],{"type":45,"value":3210},{"type":40,"tag":67,"props":3325,"children":3326},{},[3327,3329,3334,3335,3340,3342],{"type":45,"value":3328},"If ",{"type":40,"tag":54,"props":3330,"children":3332},{"className":3331},[],[3333],{"type":45,"value":982},{"type":45,"value":619},{"type":40,"tag":54,"props":3336,"children":3338},{"className":3337},[],[3339],{"type":45,"value":943},{"type":45,"value":3341}," describe similar content:\n",{"type":40,"tag":63,"props":3343,"children":3344},{},[3345,3369],{"type":40,"tag":67,"props":3346,"children":3347},{},[3348,3350,3355,3357,3362,3364],{"type":45,"value":3349},"For an ",{"type":40,"tag":54,"props":3351,"children":3353},{"className":3352},[],[3354],{"type":45,"value":109},{"type":45,"value":3356},": the prompt or rubric is probably ambiguous — reword\n",{"type":40,"tag":54,"props":3358,"children":3360},{"className":3359},[],[3361],{"type":45,"value":773},{"type":45,"value":3363}," and use ",{"type":40,"tag":54,"props":3365,"children":3367},{"className":3366},[],[3368],{"type":45,"value":2204},{"type":40,"tag":67,"props":3370,"children":3371},{},[3372,3374,3379,3381,3387,3389,3394,3396],{"type":45,"value":3373},"For a ",{"type":40,"tag":54,"props":3375,"children":3377},{"className":3376},[],[3378],{"type":45,"value":79},{"type":45,"value":3380}," evaluator: the rule is probably under- or over-matching — read the\nsource via ",{"type":40,"tag":54,"props":3382,"children":3384},{"className":3383},[],[3385],{"type":45,"value":3386},"llma-evaluation-get",{"type":45,"value":3388},", narrow the predicate, and retest with\n",{"type":40,"tag":54,"props":3390,"children":3392},{"className":3391},[],[3393],{"type":45,"value":1754},{"type":45,"value":3395}," before pushing the fix via ",{"type":40,"tag":54,"props":3397,"children":3399},{"className":3398},[],[3400],{"type":45,"value":2204},{"type":40,"tag":628,"props":3402,"children":3404},{"id":3403},"did-a-hog-evaluator-regression-after-a-code-change",[3405],{"type":45,"value":3406},"\"Did a Hog evaluator regression after a code change?\"",{"type":40,"tag":48,"props":3408,"children":3409},{},[3410],{"type":45,"value":3411},"Hog evaluators are reproducible — if the source hasn't changed, identical inputs should\nyield identical outputs. When fail rates jump for a Hog evaluator:",{"type":40,"tag":3169,"props":3413,"children":3414},{},[3415,3431,3436,3455,3468],{"type":40,"tag":67,"props":3416,"children":3417},{},[3418,3423,3425],{"type":40,"tag":54,"props":3419,"children":3421},{"className":3420},[],[3422],{"type":45,"value":3386},{"type":45,"value":3424}," — note the current source and ",{"type":40,"tag":54,"props":3426,"children":3428},{"className":3427},[],[3429],{"type":45,"value":3430},"updated_at",{"type":40,"tag":67,"props":3432,"children":3433},{},[3434],{"type":45,"value":3435},"Spot-check the latest failing runs with the SQL query from Step 4 above",{"type":40,"tag":67,"props":3437,"children":3438},{},[3439,3441,3446,3448,3453],{"type":45,"value":3440},"Re-run the source against those exact generations using ",{"type":40,"tag":54,"props":3442,"children":3444},{"className":3443},[],[3445],{"type":45,"value":1754},{"type":45,"value":3447}," with a\nmodified ",{"type":40,"tag":54,"props":3449,"children":3451},{"className":3450},[],[3452],{"type":45,"value":3127},{"type":45,"value":3454}," filter that targets them",{"type":40,"tag":67,"props":3456,"children":3457},{},[3458,3460,3466],{"type":45,"value":3459},"If the test results match the live results, the change is in the ",{"type":40,"tag":3461,"props":3462,"children":3463},"em",{},[3464],{"type":45,"value":3465},"generations",{"type":45,"value":3467},", not\nthe evaluator (a model upgrade, prompt change upstream, etc.) — investigate the\nproducer",{"type":40,"tag":67,"props":3469,"children":3470},{},[3471],{"type":45,"value":3472},"If they diverge, the evaluator was edited; check git history of the source field via\nthe activity log",{"type":40,"tag":628,"props":3474,"children":3476},{"id":3475},"what-kinds-of-generations-does-this-evaluator-skip-as-na",[3477],{"type":45,"value":3478},"\"What kinds of generations does this evaluator skip as N\u002FA?\"",{"type":40,"tag":635,"props":3480,"children":3482},{"className":637,"code":3481,"language":639,"meta":640,"style":640},"posthog:llma-evaluation-summary-create\n{ \"evaluation_id\": \"\u003Cuuid>\", \"filter\": \"na\" }\n",[3483],{"type":40,"tag":54,"props":3484,"children":3485},{"__ignoreMap":640},[3486,3493],{"type":40,"tag":646,"props":3487,"children":3488},{"class":648,"line":649},[3489],{"type":40,"tag":646,"props":3490,"children":3491},{"style":653},[3492],{"type":45,"value":824},{"type":40,"tag":646,"props":3494,"children":3495},{"class":648,"line":659},[3496,3500,3504,3508,3512,3516,3520,3524,3528,3532,3536,3540,3544,3548,3552,3557,3561],{"type":40,"tag":646,"props":3497,"children":3498},{"style":663},[3499],{"type":45,"value":666},{"type":40,"tag":646,"props":3501,"children":3502},{"style":663},[3503],{"type":45,"value":671},{"type":40,"tag":646,"props":3505,"children":3506},{"style":674},[3507],{"type":45,"value":846},{"type":40,"tag":646,"props":3509,"children":3510},{"style":663},[3511],{"type":45,"value":682},{"type":40,"tag":646,"props":3513,"children":3514},{"style":663},[3515],{"type":45,"value":687},{"type":40,"tag":646,"props":3517,"children":3518},{"style":663},[3519],{"type":45,"value":671},{"type":40,"tag":646,"props":3521,"children":3522},{"style":694},[3523],{"type":45,"value":863},{"type":40,"tag":646,"props":3525,"children":3526},{"style":663},[3527],{"type":45,"value":682},{"type":40,"tag":646,"props":3529,"children":3530},{"style":663},[3531],{"type":45,"value":706},{"type":40,"tag":646,"props":3533,"children":3534},{"style":663},[3535],{"type":45,"value":671},{"type":40,"tag":646,"props":3537,"children":3538},{"style":674},[3539],{"type":45,"value":884},{"type":40,"tag":646,"props":3541,"children":3542},{"style":663},[3543],{"type":45,"value":682},{"type":40,"tag":646,"props":3545,"children":3546},{"style":663},[3547],{"type":45,"value":687},{"type":40,"tag":646,"props":3549,"children":3550},{"style":663},[3551],{"type":45,"value":671},{"type":40,"tag":646,"props":3553,"children":3554},{"style":694},[3555],{"type":45,"value":3556},"na",{"type":40,"tag":646,"props":3558,"children":3559},{"style":663},[3560],{"type":45,"value":682},{"type":40,"tag":646,"props":3562,"children":3563},{"style":663},[3564],{"type":45,"value":733},{"type":40,"tag":48,"props":3566,"children":3567},{},[3568,3570,3575,3577,3582],{"type":45,"value":3569},"Inspect ",{"type":40,"tag":54,"props":3571,"children":3573},{"className":3572},[],[3574],{"type":45,"value":989},{"type":45,"value":3576}," to see whether the N\u002FA logic is doing the right thing. If a\npattern in ",{"type":40,"tag":54,"props":3578,"children":3580},{"className":3579},[],[3581],{"type":45,"value":989},{"type":45,"value":3583}," looks like something that should have been scored:",{"type":40,"tag":63,"props":3585,"children":3586},{},[3587,3598],{"type":40,"tag":67,"props":3588,"children":3589},{},[3590,3591,3596],{"type":45,"value":3349},{"type":40,"tag":54,"props":3592,"children":3594},{"className":3593},[],[3595],{"type":45,"value":109},{"type":45,"value":3597},": the applicability instruction in the prompt is too broad — narrow\nit",{"type":40,"tag":67,"props":3599,"children":3600},{},[3601,3602,3607,3609,3615,3617,3622],{"type":45,"value":3373},{"type":40,"tag":54,"props":3603,"children":3605},{"className":3604},[],[3606],{"type":45,"value":79},{"type":45,"value":3608}," evaluator with ",{"type":40,"tag":54,"props":3610,"children":3612},{"className":3611},[],[3613],{"type":45,"value":3614},"output_config.allows_na: true",{"type":45,"value":3616},": the source is returning\n",{"type":40,"tag":54,"props":3618,"children":3620},{"className":3619},[],[3621],{"type":45,"value":599},{"type":45,"value":3623}," (or whatever the N\u002FA signal is) too eagerly — tighten the precondition",{"type":40,"tag":628,"props":3625,"children":3627},{"id":3626},"score-this-single-generation-right-now",[3628],{"type":45,"value":3629},"\"Score this single generation right now\"",{"type":40,"tag":48,"props":3631,"children":3632},{},[3633,3638],{"type":40,"tag":54,"props":3634,"children":3636},{"className":3635},[],[3637],{"type":45,"value":2196},{"type":45,"value":3639}," with the trace's generation ID and timestamp. Useful for spot-checking\nor wiring evaluations into a larger agent loop.",{"type":40,"tag":139,"props":3641,"children":3643},{"id":3642},"constructing-ui-links",[3644],{"type":45,"value":3645},"Constructing UI links",{"type":40,"tag":63,"props":3647,"children":3648},{},[3649,3665,3680],{"type":40,"tag":67,"props":3650,"children":3651},{},[3652,3657,3659],{"type":40,"tag":71,"props":3653,"children":3654},{},[3655],{"type":45,"value":3656},"Evaluations list",{"type":45,"value":3658},": ",{"type":40,"tag":54,"props":3660,"children":3662},{"className":3661},[],[3663],{"type":45,"value":3664},"https:\u002F\u002Fapp.posthog.com\u002Fai-evals\u002Fevaluations",{"type":40,"tag":67,"props":3666,"children":3667},{},[3668,3673,3674],{"type":40,"tag":71,"props":3669,"children":3670},{},[3671],{"type":45,"value":3672},"Single evaluation",{"type":45,"value":3658},{"type":40,"tag":54,"props":3675,"children":3677},{"className":3676},[],[3678],{"type":45,"value":3679},"https:\u002F\u002Fapp.posthog.com\u002Fai-evals\u002Fevaluations\u002F\u003Cevaluation_id>",{"type":40,"tag":67,"props":3681,"children":3682},{},[3683,3688,3690,3696],{"type":40,"tag":71,"props":3684,"children":3685},{},[3686],{"type":45,"value":3687},"Underlying generation\u002Ftrace",{"type":45,"value":3689},": see the ",{"type":40,"tag":54,"props":3691,"children":3693},{"className":3692},[],[3694],{"type":45,"value":3695},"exploring-llm-traces",{"type":45,"value":3697}," skill's URL conventions",{"type":40,"tag":48,"props":3699,"children":3700},{},[3701],{"type":45,"value":3702},"Always surface the relevant link so the user can verify in the UI.",{"type":40,"tag":139,"props":3704,"children":3706},{"id":3705},"tips",[3707],{"type":45,"value":3708},"Tips",{"type":40,"tag":63,"props":3710,"children":3711},{},[3712,3746,3759,3777,3810,3815,3854,3866],{"type":40,"tag":67,"props":3713,"children":3714},{},[3715,3717,3722,3724,3729,3731,3737,3739,3744],{"type":45,"value":3716},"The summary tool is ",{"type":40,"tag":71,"props":3718,"children":3719},{},[3720],{"type":45,"value":3721},"rate-limited",{"type":45,"value":3723}," (burst, sustained, daily) and ",{"type":40,"tag":71,"props":3725,"children":3726},{},[3727],{"type":45,"value":3728},"caches results\nfor one hour",{"type":45,"value":3730}," — repeated calls with the same ",{"type":40,"tag":54,"props":3732,"children":3734},{"className":3733},[],[3735],{"type":45,"value":3736},"(evaluation_id, filter)",{"type":45,"value":3738}," are cheap; use\n",{"type":40,"tag":54,"props":3740,"children":3742},{"className":3741},[],[3743],{"type":45,"value":1074},{"type":45,"value":3745}," only when you genuinely need fresh analysis",{"type":40,"tag":67,"props":3747,"children":3748},{},[3749,3751,3757],{"type":45,"value":3750},"Pass ",{"type":40,"tag":54,"props":3752,"children":3754},{"className":3753},[],[3755],{"type":45,"value":3756},"generation_ids: [...]",{"type":45,"value":3758}," to scope a summary to a specific cohort of runs (max 250)",{"type":40,"tag":67,"props":3760,"children":3761},{},[3762,3763,3768,3770,3775],{"type":45,"value":1716},{"type":40,"tag":54,"props":3764,"children":3766},{"className":3765},[],[3767],{"type":45,"value":1018},{"type":45,"value":3769}," block in the summary response is computed from raw data, not the LLM\n— trust those counts even if a pattern's ",{"type":40,"tag":54,"props":3771,"children":3773},{"className":3772},[],[3774],{"type":45,"value":965},{"type":45,"value":3776}," field is qualitative",{"type":40,"tag":67,"props":3778,"children":3779},{},[3780,3782,3787,3789,3794,3795,3801,3803,3808],{"type":45,"value":3781},"For rich filtering not supported by ",{"type":40,"tag":54,"props":3783,"children":3785},{"className":3784},[],[3786],{"type":45,"value":3180},{"type":45,"value":3788}," (e.g. by author or model\nconfiguration), fall back to ",{"type":40,"tag":54,"props":3790,"children":3792},{"className":3791},[],[3793],{"type":45,"value":1338},{"type":45,"value":1184},{"type":40,"tag":54,"props":3796,"children":3798},{"className":3797},[],[3799],{"type":45,"value":3800},"evaluations",{"type":45,"value":3802}," Postgres table or\nthe ",{"type":40,"tag":54,"props":3804,"children":3806},{"className":3805},[],[3807],{"type":45,"value":122},{"type":45,"value":3809}," ClickHouse events",{"type":40,"tag":67,"props":3811,"children":3812},{},[3813],{"type":45,"value":3814},"When showing failure patterns to the user, always include 1-2 example trace links so\nthey can validate the pattern visually",{"type":40,"tag":67,"props":3816,"children":3817},{},[3818,3823,3825,3831,3833,3839,3841,3846,3848],{"type":40,"tag":54,"props":3819,"children":3821},{"className":3820},[],[3822],{"type":45,"value":381},{"type":45,"value":3824}," tools use ",{"type":40,"tag":54,"props":3826,"children":3828},{"className":3827},[],[3829],{"type":45,"value":3830},"evaluation:read",{"type":45,"value":3832}," for read tools and ",{"type":40,"tag":54,"props":3834,"children":3836},{"className":3835},[],[3837],{"type":45,"value":3838},"evaluation:write",{"type":45,"value":3840}," for\nmutating tools; ",{"type":40,"tag":54,"props":3842,"children":3844},{"className":3843},[],[3845],{"type":45,"value":2982},{"type":45,"value":3847}," uses ",{"type":40,"tag":54,"props":3849,"children":3851},{"className":3850},[],[3852],{"type":45,"value":3853},"llm_analytics:write",{"type":40,"tag":67,"props":3855,"children":3856},{},[3857,3859,3864],{"type":45,"value":3858},"Hog evaluators are reproducible — if you suspect a regression, ",{"type":40,"tag":54,"props":3860,"children":3862},{"className":3861},[],[3863],{"type":45,"value":1754},{"type":45,"value":3865},"\nwith the suspect source against the failing generations is the fastest way to bisect\nwhether the change is in the evaluator or in the producer of the generations",{"type":40,"tag":67,"props":3867,"children":3868},{},[3869,3871],{"type":45,"value":3870},"LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with\na fixed prompt and model. If you're chasing a small regression in fail rate, prefer\nHog or pin a deterministic provider\u002Fseed in the ",{"type":40,"tag":54,"props":3872,"children":3874},{"className":3873},[],[3875],{"type":45,"value":2501},{"type":40,"tag":3877,"props":3878,"children":3879},"style",{},[3880],{"type":45,"value":3881},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3883,"total":3994},[3884,3901,3921,3935,3953,3967,3978],{"slug":3885,"name":3885,"fn":3886,"description":3887,"org":3888,"tags":3889,"stars":23,"repoUrl":24,"updatedAt":3900},"account-handover","draft sales account handover notes","Draft structured handover notes for transitioning a PostHog account from one TAM or CSM to another. Use this skill when a TAM needs to hand over an account, prepare a transition briefing, write handover notes, create an account summary for a new owner, or any request involving account transitions between TAMs or CSMs. Triggers on \"hand over this account\", \"transition account to\", \"draft handover notes\", \"account briefing for new TAM\", \"prepare account transition\", or when a TAM names an account and says they're leaving or reassigning it.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3890,3893,3896,3897],{"name":3891,"slug":3892,"type":13},"Communications","communications",{"name":3894,"slug":3895,"type":13},"CRM","crm",{"name":9,"slug":8,"type":13},{"name":3898,"slug":3899,"type":13},"Sales","sales","2026-04-16T05:13:00.172732",{"slug":3902,"name":3902,"fn":3903,"description":3904,"org":3905,"tags":3906,"stars":23,"repoUrl":24,"updatedAt":3920},"auditing-warehouse-data-health","audit PostHog data warehouse health","Audit the health of a PostHog project's data warehouse — find every broken or degraded pipeline item across sources, sync schemas, materialized views, batch exports, and transformations. Use when the user asks \"what's broken in my warehouse?\", \"give me a health check\", \"audit my data pipeline\", \"why are some dashboards stale?\", or wants a one-shot triage summary before deciding where to spend time. Produces a prioritized report of issues grouped by severity and type, with recommended next steps.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3907,3910,3913,3916,3919],{"name":3908,"slug":3909,"type":13},"Audit","audit",{"name":3911,"slug":3912,"type":13},"Data Engineering","data-engineering",{"name":3914,"slug":3915,"type":13},"Data Quality","data-quality",{"name":3917,"slug":3918,"type":13},"Observability","observability",{"name":9,"slug":8,"type":13},"2026-06-21T08:19:05.85849",{"slug":3922,"name":3922,"fn":3923,"description":3924,"org":3925,"tags":3926,"stars":23,"repoUrl":24,"updatedAt":3934},"copying-flags-across-projects","copy feature flags across PostHog projects","Copy a feature flag from one PostHog project to one or more target projects in the same organization. Use when the user wants to duplicate a flag, promote a flag from staging to production, sync flags across projects, or replicate a flag configuration in a different workspace. Covers cohort remapping, scheduled-change handling, encrypted payloads, and the safe defaults (disabled in target, no scheduled changes).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3927,3930,3933],{"name":3928,"slug":3929,"type":13},"Deployment","deployment",{"name":3931,"slug":3932,"type":13},"Feature Flags","feature-flags",{"name":9,"slug":8,"type":13},"2026-05-04T05:56:44.484909",{"slug":3936,"name":3936,"fn":3937,"description":3938,"org":3939,"tags":3940,"stars":23,"repoUrl":24,"updatedAt":3952},"diagnosing-experiment-results","diagnose PostHog experiment results and anomalies","Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers empty \u002F 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A\u002FA, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, and surprises after mid-run edits. Symptom-driven dispatch to the right diagnostic.\nTRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange \u002F wrong \u002F off, sees significance flipping, notices PostHog numbers disagreeing with their SQL, sees an A\u002FA test showing significance, or reports surprises after mid-run edits.\nDO NOT TRIGGER when: creating a new experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3941,3944,3945,3948,3951],{"name":3942,"slug":3943,"type":13},"A\u002FB Testing","a-b-testing",{"name":21,"slug":22,"type":13},{"name":3946,"slug":3947,"type":13},"Data Analysis","data-analysis",{"name":3949,"slug":3950,"type":13},"Debugging","debugging",{"name":9,"slug":8,"type":13},"2026-05-22T06:59:58.103867",{"slug":3954,"name":3954,"fn":3955,"description":3956,"org":3957,"tags":3958,"stars":23,"repoUrl":24,"updatedAt":3966},"diagnosing-missing-recordings","diagnose missing PostHog session recordings","Diagnoses why a session recording is missing or was not captured. Use when a user asks why a session has no replay, why recordings aren't appearing, or wants to troubleshoot session replay capture issues for a specific session ID or across their project. Covers SDK diagnostic signals, project settings, sampling, triggers, ad blockers, and quota\u002Fbilling scenarios.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3959,3960,3961,3964,3965],{"name":21,"slug":22,"type":13},{"name":3949,"slug":3950,"type":13},{"name":3962,"slug":3963,"type":13},"Frontend","frontend",{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},"2026-04-22T05:06:51.989772",{"slug":3968,"name":3968,"fn":3969,"description":3970,"org":3971,"tags":3972,"stars":23,"repoUrl":24,"updatedAt":3977},"diagnosing-sdk-health","diagnose PostHog SDK health","Diagnoses the health of a project's PostHog SDK integrations — which SDKs are out of date and how to fix them. Use when a user asks about PostHog SDK versions, outdated SDKs, upgrade recommendations, \"SDK health\", \"SDK doctor\" (the former name), or when events or features seem off and it might be due to an old SDK.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3973,3974,3975,3976],{"name":21,"slug":22,"type":13},{"name":3949,"slug":3950,"type":13},{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},"2026-04-27T05:46:14.554016",{"slug":3979,"name":3979,"fn":3980,"description":3981,"org":3982,"tags":3983,"stars":23,"repoUrl":24,"updatedAt":3993},"error-tracking-android","track Android errors with PostHog","PostHog error tracking for Android",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3984,3987,3988,3991,3992],{"name":3985,"slug":3986,"type":13},"Android","android",{"name":3949,"slug":3950,"type":13},{"name":3989,"slug":3990,"type":13},"Mobile","mobile",{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},"2026-04-06T18:46:26.982494",110,{"items":3996,"total":4159},[3997,4012,4022,4035,4048,4061,4077,4094,4106,4121,4131,4149],{"slug":3998,"name":3998,"fn":3999,"description":4000,"org":4001,"tags":4002,"stars":4009,"repoUrl":4010,"updatedAt":4011},"analyzing-expensive-users","analyze expensive users in AI observability","Analyze the most expensive users in AI observability and explain why they cost so much. Use when the user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability spend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4003,4004,4007,4008],{"name":21,"slug":22,"type":13},{"name":4005,"slug":4006,"type":13},"Cost Optimization","cost-optimization",{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},35568,"https:\u002F\u002Fgithub.com\u002FPostHog\u002Fposthog","2026-07-28T05:34:11.117757",{"slug":4013,"name":4013,"fn":4014,"description":4015,"org":4016,"tags":4017,"stars":4009,"repoUrl":4010,"updatedAt":4021},"auditing-endpoints","audit PostHog project endpoints","Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised versions. Use when the user asks \"what endpoints can I clean up?\", \"are any of my endpoints broken?\", \"which materialised versions are still being called?\", or wants a one-shot cleanup pass over the Endpoints product. Produces a prioritised report grouped by issue type, with recommended actions but does not modify anything without explicit confirmation.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4018,4019,4020],{"name":21,"slug":22,"type":13},{"name":3908,"slug":3909,"type":13},{"name":9,"slug":8,"type":13},"2026-06-08T08:08:33.693989",{"slug":4023,"name":4023,"fn":4024,"description":4025,"org":4026,"tags":4027,"stars":4009,"repoUrl":4010,"updatedAt":4034},"auditing-warehouse-source-health","audit PostHog data warehouse source health","Audit the health of a PostHog project's data warehouse sources and syncs — find every broken or degraded source connection, sync schema, and webhook channel. Use when the user asks \"why are my imports failing?\", \"what's broken with my sources?\", \"why is my warehouse data stale?\", or wants a one-shot triage of source\u002Fsync health before deciding where to dig in. Produces a prioritized report grouped by severity, with recommended next steps. For materialized-view health use `auditing-warehouse-view-health`; for a single failing sync use `diagnosing-failed-warehouse-syncs`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4028,4029,4032,4033],{"name":3908,"slug":3909,"type":13},{"name":4030,"slug":4031,"type":13},"Data Warehouse","data-warehouse",{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},"2026-06-18T08:22:57.67984",{"slug":4036,"name":4036,"fn":4037,"description":4038,"org":4039,"tags":4040,"stars":4009,"repoUrl":4010,"updatedAt":4047},"auditing-warehouse-view-health","audit PostHog materialized view health","Audit the health of a PostHog project's materialized views (saved queries) — find every failed materialization and flag unused or stale materialized views that cost storage and compute. Use when the user asks \"which of my views are broken?\", \"why is this materialized view failing?\", \"are any of my views wasting compute?\", or wants a one-shot triage of view health. For source\u002Fsync health use `auditing-warehouse-source-health`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4041,4042,4043,4046],{"name":3908,"slug":3909,"type":13},{"name":4030,"slug":4031,"type":13},{"name":4044,"slug":4045,"type":13},"Performance","performance",{"name":9,"slug":8,"type":13},"2026-06-18T08:25:10.936787",{"slug":4049,"name":4049,"fn":4050,"description":4051,"org":4052,"tags":4053,"stars":4009,"repoUrl":4010,"updatedAt":4060},"authoring-error-tracking-alerts","author PostHog error tracking alerts","Author error tracking alerts that fire when an issue is created, reopened, or starts spiking. Use when the user asks to set up error notifications, route exceptions to Slack\u002Fwebhook\u002FLinear, or evaluate which error events are worth alerting on. Covers trigger-event selection, integration choice, dedup against existing alerts, and shipping with the canonical message body shape.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4054,4057,4058,4059],{"name":4055,"slug":4056,"type":13},"Alerting","alerting",{"name":3949,"slug":3950,"type":13},{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},"2026-06-18T08:24:40.318583",{"slug":4062,"name":4062,"fn":4063,"description":4064,"org":4065,"tags":4066,"stars":4009,"repoUrl":4010,"updatedAt":4076},"authoring-log-alerts","author log alerts in PostHog","Author useful, low-noise log alerts on services in a PostHog project. Use when the user asks to set up alerts for their logs, suggest alerts they should add, or evaluate whether a service is worth monitoring. Covers service triage, baseline characterisation, threshold drafting, back-testing via simulate, and shipping with a notification destination.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4067,4068,4071,4072,4075],{"name":21,"slug":22,"type":13},{"name":4069,"slug":4070,"type":13},"Monitoring","monitoring",{"name":3917,"slug":3918,"type":13},{"name":4073,"slug":4074,"type":13},"Operations","operations",{"name":9,"slug":8,"type":13},"2026-07-18T05:10:54.430898",{"slug":4078,"name":4078,"fn":4079,"description":4080,"org":4081,"tags":4082,"stars":4009,"repoUrl":4010,"updatedAt":4093},"building-workflows","build and edit PostHog workflows","Build, edit, test, enable, and monitor PostHog workflows over MCP. Author the action\u002Fedge graph so it runs and opens cleanly in the visual editor, then change drafts surgically with patch operations. Use when asked to build, set up, automate, change, fix, or debug a workflow, campaign, broadcast, drip sequence, or event-triggered automation in the workflows product.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4083,4086,4089,4090],{"name":4084,"slug":4085,"type":13},"Automation","automation",{"name":4087,"slug":4088,"type":13},"MCP","mcp",{"name":9,"slug":8,"type":13},{"name":4091,"slug":4092,"type":13},"Workflow Automation","workflow-automation","2026-07-28T05:34:12.167015",{"slug":4095,"name":4095,"fn":4096,"description":4097,"org":4098,"tags":4099,"stars":4009,"repoUrl":4010,"updatedAt":4105},"check-posthog-loading","inspect PostHog SDK loading across URLs","Inspect how the PostHog JavaScript SDK is loaded across a list of URLs. Use to confirm consistent installation across pages, find pages missing the snippet, detect mismatched API keys or hosts between pages, and verify the load method (head snippet vs deferred vs array.js).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4100,4101,4102,4103,4104],{"name":21,"slug":22,"type":13},{"name":3949,"slug":3950,"type":13},{"name":3962,"slug":3963,"type":13},{"name":3917,"slug":3918,"type":13},{"name":9,"slug":8,"type":13},"2026-05-07T05:56:19.828048",{"slug":4107,"name":4107,"fn":4108,"description":4109,"org":4110,"tags":4111,"stars":4009,"repoUrl":4010,"updatedAt":4120},"consuming-endpoints-from-client-code","integrate PostHog endpoints into client applications","Wire a PostHog endpoint into a client app or SDK. Covers fetching the OpenAPI spec, generating a typed client with openapi-generator or @hey-api\u002Fopenapi-ts, sending the right auth header, shaping the variables payload (HogQL code_name vs insight breakdown property), handling rate-limit and materialised-endpoint error responses. Use when the user says \"how do I call my endpoint\", \"generate a client for this\", or \"what auth header do I use\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4112,4115,4116,4117],{"name":4113,"slug":4114,"type":13},"API Development","api-development",{"name":3962,"slug":3963,"type":13},{"name":9,"slug":8,"type":13},{"name":4118,"slug":4119,"type":13},"SDK","sdk","2026-06-08T08:08:34.929454",{"slug":4122,"name":4122,"fn":4123,"description":4124,"org":4125,"tags":4126,"stars":4009,"repoUrl":4010,"updatedAt":4130},"copying-endpoints-across-projects","copy PostHog endpoints across projects","Copy a PostHog endpoint (a saved HogQL\u002Finsight query exposed as an API route) to another project in the same organization, or duplicate it under a new name in the same project. Use when the user wants to duplicate an endpoint, promote an endpoint from staging to production, replicate an endpoint's query\u002Fvariables\u002Ffreshness config in another workspace, or clone an endpoint to iterate on it. Unlike feature flags and experiments, endpoints have NO native cross-project copy tool — this skill covers the read-then-recreate flow (endpoint-get then endpoint-create), the active-project switching it requires, name-collision checks, and the safe defaults (land unmaterialised in the target, verify with endpoint-run). Does not cover editing endpoint versions (see managing-endpoint-versions) or authoring a brand-new endpoint from scratch (see creating-an-endpoint).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4127,4128,4129],{"name":4113,"slug":4114,"type":13},{"name":4073,"slug":4074,"type":13},{"name":9,"slug":8,"type":13},"2026-07-15T05:29:58.442727",{"slug":4132,"name":4132,"fn":4133,"description":4134,"org":4135,"tags":4136,"stars":4009,"repoUrl":4010,"updatedAt":4148},"creating-ai-subscription","schedule recurring AI-generated PostHog reports","Create a recurring AI-generated PostHog report — schedule a free-text prompt to run on a cron, with the LLM-synthesized markdown delivered to email or Slack on each tick. Use when the user wants a recurring AI summary of X on any cadence (daily, weekly, monthly, yearly) rather than a one-off report. (To attach an AI summary to an existing insight\u002Fdashboard subscription instead of a free-text prompt, see `managing-subscriptions` and its `summary_enabled` option.)\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4137,4138,4141,4142,4145],{"name":4084,"slug":4085,"type":13},{"name":4139,"slug":4140,"type":13},"Email","email",{"name":9,"slug":8,"type":13},{"name":4143,"slug":4144,"type":13},"Reporting","reporting",{"name":4146,"slug":4147,"type":13},"Slack","slack","2026-06-09T07:32:27.935712",{"slug":4150,"name":4150,"fn":4151,"description":4152,"org":4153,"tags":4154,"stars":4009,"repoUrl":4010,"updatedAt":4158},"creating-an-endpoint","create PostHog API endpoints","Create a PostHog endpoint with the right shape on the first try — covers query kind choice, name conventions, what to expose as variables (HogQL code_name vs insight breakdown), data_freshness_seconds, and whether to materialise on day one. Use when the user says \"create an endpoint\", \"expose this query as an API\", \"turn this insight into an endpoint\", or asks for help structuring a new endpoint. Steers away from common mistakes: materialising a query with cohort breakdowns or compare mode, inline-only variables on a materialised endpoint, unbounded date ranges, ambiguous names.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4155,4156,4157],{"name":21,"slug":22,"type":13},{"name":4113,"slug":4114,"type":13},{"name":9,"slug":8,"type":13},"2026-06-08T08:08:29.624498",231]