[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-posthog-feature-usage-feed":3,"mdc-tdumyg-key":37,"related-org-posthog-feature-usage-feed":5372,"related-repo-posthog-feature-usage-feed":5543},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"feature-usage-feed","extract feature use cases with LLM judges","Set up an LLM-judge evaluation that extracts canonical use cases for a PostHog feature at scale and streams the results to a Slack channel as a live feed. Use when someone wants to understand how users are actually using a specific AI\u002FLLM-powered feature in production — what they're investigating, what questions they're trying to answer, and what patterns surface — without manually reading hundreds of traces. Assumes the feature emits `$ai_generation` and `$ai_evaluation` events with `$session_id` linkage to the trigger user's recording (the standard setup post the session-summary linkage PRs).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"posthog","PostHog","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fposthog.png",[12,14,17,20,23],{"name":9,"slug":8,"type":13},"tag",{"name":15,"slug":16,"type":13},"LLM","llm",{"name":18,"slug":19,"type":13},"Evals","evals",{"name":21,"slug":22,"type":13},"Slack","slack",{"name":24,"slug":25,"type":13},"Analytics","analytics",56,"https:\u002F\u002Fgithub.com\u002FPostHog\u002Fskills","2026-04-27T05:46:18.151877",null,4,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"PostHog skills (under construction)","https:\u002F\u002Fgithub.com\u002FPostHog\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fomnibus\u002Ffeature-usage-feed","---\nname: feature-usage-feed\ndescription: >\n  Set up an LLM-judge evaluation that extracts canonical use cases for a\n  PostHog feature at scale and streams the results to a Slack channel as a\n  live feed. Use when someone wants to understand how users are actually\n  using a specific AI\u002FLLM-powered feature in production — what they're\n  investigating, what questions they're trying to answer, and what\n  patterns surface — without manually reading hundreds of traces. Assumes\n  the feature emits `$ai_generation` and `$ai_evaluation` events with\n  `$session_id` linkage to the trigger user's recording (the standard\n  setup post the session-summary linkage PRs).\n---\n\n# Building a feature usage feed via LLM evals\n\nSome PostHog features (group session summaries, single session summaries, replay AI search, error tracking AI debug, etc.) generate hundreds or thousands of LLM traces per week. Reading them by hand is not feasible. This skill covers the end-to-end pattern for turning that trace volume into a live Slack feed of canonical use cases — what users are actually doing with the feature.\n\nThe workflow is **mixed, and leans UI**. Trace inspection and filter discovery (steps 1-2) are MCP-driven. Eval creation, dry-running, and enabling (steps 4-5) are MCP-driven _when_ `posthog:llma-evaluation-*` tools are exposed to your agent — but they often aren't, in which case fall back to the UI (Data pipeline → destinations for the alert is always UI). Each step flags its UI fallback. Expect to finish in the UI even when you start from chat.\n\n## When to use\n\n- \"How are people actually using [feature X] in production?\"\n- \"Can we identify the canonical use cases for [feature X] so we can write better docs \u002F prioritize improvements?\"\n- \"I want a Slack feed of representative usage examples without manually skimming traces.\"\n- \"Set up a feed of use cases for [feature X] in #team-[area]-usage.\"\n\nIf the user just wants to debug a single trace or tune an existing eval, redirect to `exploring-llm-traces` or `exploring-llm-evaluations` instead.\n\n## Two filter patterns\n\nThis skill supports two different ways to scope an eval to \"the feature you care about\":\n\n**Pattern A — Feature-native trace_id prefix.** For standalone features that emit their own `$ai_trace_id` pattern (e.g. `session-summary:group:`, `replay-search:`, error-tracking-specific flows). Filter on the prefix.\n\n**Pattern B — PostHog AI agent mode.** For features the user interacts with _via_ PostHog AI in a specific agent mode (error tracking, product analytics, session replay, SQL, flags, surveys, AI observability). Filter on `ai_product = 'posthog_ai' AND agent_mode = '\u003Cmode>'`. This requires PR #55160 (merged April 2026) to be deployed, which threads `agent_mode` and `supermode` onto every `$ai_generation` emitted by the chat agent loop. A useful ergonomic side-effect: `agent_mode IS NOT NULL` is a reliable \"user-facing chat turn\" filter — batch jobs and tool-internal LLM calls go through different code paths and have `agent_mode=null`, so they're excluded for free.\n\nIf the user asks \"what are users trying to DO in [ET \u002F replay \u002F SQL \u002F flags \u002F surveys] mode of PostHog AI\", that's Pattern B. If they ask \"what use cases does [standalone feature] cover\", that's Pattern A. Pick the pattern first — the prompt, filter, and Slack channel naming all follow from it.\n\n## Prerequisites\n\n| Requirement                                                                                              | How to verify                                                                                                                                                                                                             |\n| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| (Pattern A) Feature emits `$ai_generation` events with a stable `$ai_trace_id` pattern                   | `posthog:execute-sql` for distinct `$ai_trace_id` prefixes                                                                                                                                                                |\n| (Pattern B) `agent_mode` property is present on recent `$ai_generation` events                           | `posthog:execute-sql` group-by `properties.agent_mode` on recent `ai_product='posthog_ai'` events. Null bucket is normal (batch jobs + tool-internal calls) — you want non-null coverage across the modes you care about. |\n| `$session_id` is attached to the `$ai_generation` events (links trace to trigger session)                | `posthog:execute-sql` for `countIf($session_id IS NOT NULL) \u002F count()`                                                                                                                                                    |\n| `$session_id` is also attached to the `$ai_evaluation` events (lets the Slack alert link to the session) | Same query but on `$ai_evaluation` events after the eval has run once                                                                                                                                                     |\n| User has organisation-level AI data processing approval                                                  | Required for `llm_judge` evaluations and the eval summary tool                                                                                                                                                            |\n\nIf `$session_id` is missing on either event type, file a backend fix before continuing — there is no UI workaround. The session-summary feature has a worked example of the threading pattern in PR #54952. For Pattern B, the agent-mode threading pattern is in PR #55160.\n\n## Tools\n\n| Tool                                               | Purpose                                                                                                                                                                                                                                                           |\n| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `posthog:query-llm-traces-list`                    | Find sample traces matching the feature's `$ai_trace_id` pattern                                                                                                                                                                                                  |\n| `posthog:query-llm-trace`                          | Inspect a specific trace's contents end-to-end                                                                                                                                                                                                                    |\n| `posthog:execute-sql`                              | Verify trace volume, session_id coverage, eval result distributions                                                                                                                                                                                               |\n| `posthog:llma-evaluation-create`                   | (**often unexposed** — UI fallback: AI observability → Evaluations → New) Create the LLM-judge eval (disabled at first)                                                                                                                                           |\n| `posthog:llma-evaluation-run`                      | (**often unexposed** — UI fallback: the eval's detail page has a \"Run on event\" button) Dry-run the eval against specific generations during prompt iteration                                                                                                     |\n| `posthog:llma-evaluation-update`                   | (**often unexposed** — UI fallback: edit the eval in AI observability → Evaluations) Tweak the prompt \u002F enable when ready                                                                                                                                         |\n| `posthog:llma-evaluation-summary-create`           | (**often unexposed** — UI fallback: the eval detail page has a \"Summarize results\" button) After the feed is running, get an AI summary of pass\u002FN\u002FA patterns to validate signal quality                                                                           |\n| `posthog:workflows-list` \u002F `posthog:workflows-get` | (**often unexposed** — UI: Data pipeline → Workflows) Browse existing workflow configs — useful for cloning an existing feed's structure when setting up a new one. Read-only; no create\u002Fupdate tool is exposed yet, so step 6's Slack workflow setup is UI-only. |\n\nBefore starting, **check which of the `posthog:llma-evaluation-*` tools are actually exposed in your agent's MCP tool set.** If they aren't loaded, treat steps 4-5 as UI walkthroughs rather than tool calls.\n\n## Workflow\n\n### Step 1 — Identify the filter\n\n**Pattern A (feature-native trace_id prefix):** find the prefix that maps to your feature.\n\n```sql\nSELECT\n    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[1] AS root,\n    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[2] AS subtype,\n    count() AS events\nFROM events\nWHERE timestamp > now() - INTERVAL 3 DAY\n    AND event = '$ai_generation'\n    AND properties.$ai_trace_id IS NOT NULL\nGROUP BY root, subtype\nORDER BY events DESC\nLIMIT 25\n```\n\nNote: `coalesce(..., '')` is load-bearing — `splitByChar` on a nullable column errors out in HogQL otherwise.\n\n**Pattern B (PostHog AI agent mode):** verify coverage and volume for the mode you're targeting.\n\n```sql\nSELECT\n    properties.agent_mode AS agent_mode,\n    properties.supermode AS supermode,\n    count() AS events,\n    count(DISTINCT properties.$ai_trace_id) AS traces\nFROM events\nWHERE timestamp > now() - INTERVAL 3 DAY\n    AND event = '$ai_generation'\n    AND properties.ai_product = 'posthog_ai'\nGROUP BY agent_mode, supermode\nORDER BY events DESC\nLIMIT 20\n```\n\nExpected values for `agent_mode`: `error_tracking`, `product_analytics`, `sql`, `session_replay`, `flags`, `survey`, `llm_analytics`, `null`. Null ≈ batch jobs + tool-internal calls (not user chat). `supermode='plan'` splits planning turns from execution turns — worth calling out separately if your feed is about plan-mode specifically.\n\nRecord the mode + rough volume. Low-volume modes (\u003C100 events\u002Fday) will produce a trickle-feed that's hard to validate early; high-volume modes (>1k\u002Fday) may need sampling to avoid Slack flooding. See the \"Tips\" section on sampling.\n\n### Step 2 — Pull a handful of sample traces\n\nUse these for prompt iteration in step 4.\n\n**Pattern A:**\n\n```json\nposthog:query-llm-traces-list\n{\n  \"properties\": [\n    { \"type\": \"event\", \"key\": \"$ai_trace_id\", \"operator\": \"icontains\", \"value\": \"\u003Cyour-prefix-here>\" }\n  ],\n  \"limit\": 10,\n  \"dateRange\": { \"date_from\": \"-2d\" },\n  \"randomOrder\": true\n}\n```\n\n**Pattern B:**\n\n```json\nposthog:query-llm-traces-list\n{\n  \"properties\": [\n    { \"type\": \"event\", \"key\": \"ai_product\", \"operator\": \"exact\", \"value\": \"posthog_ai\" },\n    { \"type\": \"event\", \"key\": \"agent_mode\", \"operator\": \"exact\", \"value\": \"\u003Cmode-here>\" }\n  ],\n  \"limit\": 10,\n  \"dateRange\": { \"date_from\": \"-2d\" },\n  \"randomOrder\": true\n}\n```\n\n`randomOrder: true` matters — recency bias produces a non-representative sample. Pick 5-10 traces to test against.\n\n**Output size warning:** `query-llm-traces-list` with `limit: 10` routinely returns 3-6MB of JSON (full input\u002Foutput per generation). This will blow your context window. **Immediately delegate the summarization to a subagent** the moment you see the \"result exceeds maximum allowed tokens\" error — ask the subagent to extract, per trace: the trace id, the first user message (truncated to ~300 chars), the sampled `$current_url`, and a one-sentence description of what the conversation was about. Don't try to read the raw file in-line.\n\n**Watch for topic drift in Pattern B samples.** The `agent_mode` tag reflects the user's mode selection at the time of the turn — but chat state retains the mode even if the user drifts off-topic within the same conversation (e.g. user selected \"error tracking\" mode, then asked an unrelated pricing question three turns later). Your eval prompt's classification step needs to be permissive about topic-drift: PASS should mean \"user is doing something recognizably in-scope for this mode\", FAIL should catch the off-topic drift. If you don't, your feed will include irrelevant PASS entries that happen to carry the mode tag.\n\n### Step 3 — Draft the LLM-judge prompt\n\nThe prompt has two responsibilities: (a) classify the trace as relevant or not, (b) produce reasoning text that is **directly postable to Slack** (no preamble, no meta-description). The reasoning field becomes the Slack message body.\n\nTemplate:\n\n```text\nYou are analyzing a PostHog [FEATURE NAME] trace to extract its real use case.\nYour reasoning text will be posted directly to a Slack channel as a notification.\nWrite it as a short, ready-to-post message — no preamble, no meta-description.\n\nStep 1 — Classification:\n- PASS = this trace is the [feature kind] you care about\n- FAIL = a different LLM call or a false match\n- N\u002FA = ambiguous from the trace alone\n\nStep 2 — Reasoning (only matters if PASS). Write 2-3 sentences in this exact format:\n\n\"[OPENER] [what they targeted\u002Ffiltered for]. They were\ntrying to [understand X \u002F debug Y \u002F find Z]. The result surfaced [key pattern\nor finding].\"\n\nYour output MUST start with the exact phrase \"[OPENER]\". No other opening is allowed.\n\nRules:\n- No \"This is a [feature]...\" or \"The input contains...\" preamble\n- No JSON, field names, system-prompt references, or meta-description\n- Concrete > generic. \"users hitting error tracking for the first time\" beats \"user behavior\"\n- If you cannot infer one of the three pieces from the trace, write \"(unclear from trace)\" in that slot — do not guess\n```\n\n**Pick an `[OPENER]` that matches how users actually interact with the feature.** The forced opener is load-bearing (it prevents the model from drifting into \"this trace is a...\" meta-description), but the exact verb has to fit the interaction:\n\n| Feature \u002F mode                    | OPENER                                     |\n| --------------------------------- | ------------------------------------------ |\n| Session summary (group \u002F single)  | `A user ran a summary on`                  |\n| Replay AI search                  | `A user searched replays for`              |\n| PostHog AI in error tracking mode | `A user asked PostHog AI about`            |\n| PostHog AI in session replay mode | `A user asked PostHog AI about`            |\n| PostHog AI in SQL mode            | `A user asked PostHog AI to write SQL for` |\n\nNote: `supermode='plan'` is a sub-filter that layers _on top of_ an `agent_mode` row — it's not its own row. If you want plan-mode-only, filter `agent_mode='\u003Cmode>' AND supermode='plan'` and pick an opener like `\"A user asked PostHog AI to plan\"`.\n\nIf you force `\"A user ran\"` on a chat-based feature, the model will produce awkward contortions (\"A user ran a question about...\") that read wrong in Slack. The forced-opener pattern is the mechanism — the specific phrase is per-feature.\n\nThe negative example list (\"No 'This is a...' preamble\", etc.) is load-bearing regardless of opener. Don't remove it.\n\n### Step 4 — Create the eval (disabled), test, iterate\n\nCreate with `enabled: false` so it doesn't immediately fan out to all traces.\n\n**If `posthog:llma-evaluation-create` is exposed**, use this payload:\n\n```json\nposthog:llma-evaluation-create\n{\n  \"name\": \"[feature] use case feed\",\n  \"description\": \"Extracts canonical use cases for [feature] for the #team-[area]-usage Slack feed\",\n  \"evaluation_type\": \"llm_judge\",\n  \"evaluation_config\": {\n    \"prompt\": \"\u003Cfull prompt from step 3>\"\n  },\n  \"output_type\": \"boolean\",\n  \"output_config\": { \"allows_na\": true },\n  \"model_configuration\": {\n    \"provider\": \"\u003Cprovider>\",\n    \"model\": \"\u003Cmodel>\"\n  },\n  \"enabled\": false,\n  \"conditions\": {\n    \"filters\": [\n      \u002F\u002F Pattern A — feature-native trace_id prefix:\n      { \"key\": \"$ai_trace_id\", \"operator\": \"icontains\", \"value\": \"\u003Cyour-prefix>\" }\n\n      \u002F\u002F Pattern B — PostHog AI agent mode (use these INSTEAD of the trace_id filter):\n      \u002F\u002F { \"key\": \"ai_product\", \"operator\": \"exact\", \"value\": \"posthog_ai\" },\n      \u002F\u002F { \"key\": \"agent_mode\", \"operator\": \"exact\", \"value\": \"\u003Cmode>\" }\n    ]\n  }\n}\n```\n\nLeave model choice to the user — LLM-judge cost scales linearly with event volume, and cheap-vs-capable is a real tradeoff they should make based on their own spend tolerance and signal-quality requirements. Don't pick for them.\n\n**UI fallback** (when `llma-evaluation-create` isn't exposed): AI observability → Evaluations → New evaluation. Type = `LLM judge`, output = boolean + allow N\u002FA, filters as above, enabled = off. Paste the prompt from step 3.\n\nThen dry-run against your sample traces.\n\n**If `posthog:llma-evaluation-run` is exposed:**\n\n```json\nposthog:llma-evaluation-run\n{\n  \"evaluationId\": \"\u003Cuuid from create>\",\n  \"target_event_id\": \"\u003Ca $ai_generation event id from step 2>\",\n  \"timestamp\": \"\u003CISO timestamp of that event>\"\n}\n```\n\n**UI fallback:** on the eval detail page, use the \"Run on event\" button with the trace sample's event id.\n\nLook at the returned `$ai_evaluation_reasoning`. If it preambles, drifts, or describes the input, fix the prompt (via `llma-evaluation-update` or by editing in the UI) and re-run. Iterate on 3-5 traces before enabling.\n\nCommon failure modes during iteration:\n\n| Symptom                                                    | Fix                                                                        |\n| ---------------------------------------------------------- | -------------------------------------------------------------------------- |\n| Reasoning starts with \"This is a...\"                       | Strengthen the forced opener instruction; add a counter-example            |\n| Reasoning is generic (\"user behavior\", \"various patterns\") | Add positive examples of concrete phrasing in the prompt                   |\n| Model classifies everything as PASS                        | Tighten the FAIL definition; add an example of what a non-match looks like |\n| Reasoning is too long for Slack                            | Add a hard sentence cap (\"MAX 3 sentences, hard limit\")                    |\n\n### Step 5 — Enable the eval\n\nOnce 3-5 sample runs produce clean Slack-ready output.\n\n**If `posthog:llma-evaluation-update` is exposed:**\n\n```json\nposthog:llma-evaluation-update\n{\n  \"evaluationId\": \"\u003Cuuid>\",\n  \"enabled\": true\n}\n```\n\n**UI fallback:** AI observability → Evaluations → open the eval → toggle enabled.\n\nThe eval will now run on every new matching `$ai_generation` event.\n\n### Step 6 — Build the workflow (UI only)\n\nWorkflow setup is not MCP-accessible for writes (`posthog:workflows-list` \u002F `posthog:workflows-get` are read-only). The steps below are a UI walkthrough.\n\n**Prereq:** before you start, invite the PostHog Slack bot to your target channel (`\u002Finvite @PostHog` in the Slack channel). Without this, the Slack dispatch step will fail with an opaque permission error at send time, not at save time — easy to miss.\n\n#### 6.1 Create the workflow\n\nData pipeline → Workflows → New workflow. Name it `\u003Cfeature> use case feed` to match the eval name from step 4.\n\n#### 6.2 Trigger step\n\n- **Event:** `AI evaluation (LLM)` — i.e. `$ai_evaluation`. This is the event emitted when an eval runs, and it's the only event that carries `$ai_evaluation_*` properties. The original `$ai_generation` event is **not** enriched with eval results, so filtering on `$ai_generation` here matches nothing.\n- **Property filters (both required):**\n  - `AI Evaluation Name (LLM)` equals `\u003Cyour eval name from step 4>`\n  - `AI Evaluation Result (LLM)` equals `true`\n\n**⚠️ LOAD-BEARING:** the stored values for `$ai_evaluation_result` are the strings `'True'` \u002F `'False'` \u002F `'None'` — NOT `'PASS'` \u002F `'FAIL'` \u002F `'N\u002FA'` (despite what the prompt template calls them internally). The Workflows UI property filter normalizes `true` → `'True'`, so selecting `equals true` from the dropdown works. But if you were wiring this in raw SQL somewhere else (say a hog function), you'd need the string literal. Verify the stored distribution before saving:\n\n```sql\nSELECT DISTINCT toString(properties.$ai_evaluation_result) AS result, count() AS n\nFROM events\nWHERE event = '$ai_evaluation'\n  AND properties.$ai_evaluation_name = '\u003Cyour eval name>'\n  AND timestamp > now() - INTERVAL 1 HOUR\nGROUP BY result\n```\n\nIf the only values are `True`\u002F`False`\u002F`None` and `True` dominates, the UI `equals true` filter will match. If you see anything else, adjust accordingly.\n\n#### 6.3 Slack dispatch step\n\n- **Add step → Slack dispatch**\n- **Channel:** `#\u003Cyour-team>-usage-feed`\n- **Sender \u002F bot display name:** something that reads well in the channel (e.g. `PostHog Usage Feed`)\n- **Blocks (Slack block-kit JSON)** — paste this and replace `\u003Cproject_id>` with your actual numeric project ID (e.g. `2`):\n\n```json\n[\n  {\n    \"text\": {\n      \"text\": \"\u003Cemoji> *{event.properties.$ai_evaluation_name}* triggered by *{person.name}*\",\n      \"type\": \"mrkdwn\"\n    },\n    \"type\": \"section\"\n  },\n  {\n    \"text\": {\n      \"text\": \"{event.properties.$ai_evaluation_reasoning}\",\n      \"type\": \"mrkdwn\"\n    },\n    \"type\": \"section\"\n  },\n  {\n    \"type\": \"actions\",\n    \"elements\": [\n      {\n        \"url\": \"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cproject_id>\u002Fai-observability\u002Ftraces\u002F{event.properties.$ai_trace_id}?event={event.properties.$ai_target_event_id}\",\n        \"text\": { \"text\": \"View Trace\", \"type\": \"plain_text\" },\n        \"type\": \"button\"\n      },\n      {\n        \"url\": \"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cproject_id>\u002Freplay\u002F{event.properties.$session_id}\",\n        \"text\": { \"text\": \"View Trigger Session\", \"type\": \"plain_text\" },\n        \"type\": \"button\"\n      },\n      {\n        \"url\": \"{person.url}\",\n        \"text\": { \"text\": \"View Person\", \"type\": \"plain_text\" },\n        \"type\": \"button\"\n      }\n    ]\n  }\n]\n```\n\nPick an `\u003Cemoji>` that matches the feature's shape: 📊 product analytics, 🐛 error tracking, 🎬 session replay, 🔎 search\u002FAI search, 🧪 experiments, 🚩 flags, 📋 surveys, 🧠 generic AI.\n\nThe `{event.properties.X}` and `{person.X}` placeholders are valid PostHog template syntax and resolve at send time.\n\n#### 6.4 Test before enabling\n\nThe Workflows Test panel has two modes — this matters because naively hitting \"Test\" can look like a broken integration when it isn't:\n\n- **Synthetic event** (default) — the Test panel fabricates an `$ai_evaluation` payload and runs the flow without hitting Slack's real API. Useful as a dry-run of the block template, but `{event.properties.$ai_*}` placeholders may resolve to `null` and Slack's block validator will reject the payload with `invalid_blocks`. That's a test-harness artifact, not a real bug — don't chase it.\n- **\"Make real HTTPS requests\"** — flip this toggle on. Workflows then pulls a recent real `$ai_evaluation` event matching your filters and runs the flow end-to-end, including the actual Slack post. This is the test that tells you \"it works\" for real. If no matching real event exists yet (common if the eval was just enabled), trigger the feature yourself, wait ~1 minute, and retry.\n\nRecommended flow: synthetic → sanity-check the block template renders → flip real-requests on → confirm an actual post lands in the channel → save + enable the workflow.\n\n### Step 7 — End-to-end verify in production\n\nOnce the workflow is enabled, trigger the feature yourself. Within a minute or two:\n\n1. The `$ai_generation` event should appear in AI observability\n2. The eval should auto-run and emit an `$ai_evaluation` event\n3. The workflow should fire and the Slack post should land in the configured channel\n4. Click \"View Trigger Session\" — should land on the recording of you using the feature, not the replay homepage\n\nIf \"View Trigger Session\" lands on the replay homepage, `$session_id` is missing on the `$ai_evaluation` event (which is separate from the `$ai_generation` event — threading is independent for the two). Backend fix needed — see prerequisites.\n\n## Worked example A (Pattern A): group session summary use cases\n\nPattern: a `group_summary_use_case_feed` eval streaming to a `#\u003Cteam>-usage-feed` channel. Trace prefix: `session-summary:group:`. Opener: `\"A user ran a group summary on\"`. Slack channel showed e.g.:\n\n> 📊 _group_summary_use_case_feed_ triggered by _some user_\n> \"A user ran a group summary on a company's onboarding sessions from the last 7 days. They were trying to understand why account activation rates are low. The summary surfaced that most users abandon at the company onboarding wizard after creating accounts.\"\n> [View Trace] [View Trigger Session] [View Person]\n\nThe PRs that made this work (linked here as worked examples of the session_id threading pattern, not as steps in the skill itself):\n\n- PostHog\u002Fposthog#54952 — threads `trigger_session_id` through to `$ai_generation` events on the session summary backend\n- (Followup PR — threads `$session_id` onto `$ai_evaluation` events specifically)\n\n## Worked example B (Pattern B): PostHog AI in error tracking mode\n\nPattern: an `agent_mode = 'error_tracking'` scoped feed streaming to a `#\u003Cteam>-usage-feed` channel, answering \"what are users actually trying to DO when they chat with PostHog AI in error tracking mode?\" Mode sizing varies by an order of magnitude or more across agent modes — spot-check volume per §Step 1 before wiring, because a high-volume mode can flood a channel. Opener: `\"A user asked PostHog AI about\"`.\n\nEnabling PR: PostHog\u002Fposthog#55160 — threads `agent_mode` and `supermode` onto every `$ai_generation` emitted by the chat agent loop. Wiring lives in `ee\u002Fhogai\u002Fcore\u002Fagent_modes\u002Fexecutables.py` (`AgentExecutable._get_model`) and passes the dict through the existing `posthog_properties` field on `MaxChatMixin` in `ee\u002Fhogai\u002Fllm.py`. Before this PR, scoping a PostHog AI eval to a specific mode wasn't possible — you'd end up evaluating every PostHog AI generation, which produced noisy feeds with low single-digit PASS rates.\n\nKey observation from setup: the `agent_mode` tag reflects the mode at turn-time, but chat state retains mode selection even when users drift off-topic mid-conversation. Spot-check: a random `agent_mode=error_tracking` sample included a conversation that ended up being about session replay pricing. The eval prompt's classification must be permissive about topic drift — PASS only when the turn is recognizably in-scope for the mode, FAIL when the conversation has drifted to something else entirely.\n\n## Validating signal quality after launch\n\nOnce the feed has been running for a day or two, sanity-check the eval output at scale.\n\n**If `posthog:llma-evaluation-summary-create` is exposed:**\n\n```json\nposthog:llma-evaluation-summary-create\n{\n  \"evaluation_id\": \"\u003Cuuid>\",\n  \"filter\": \"fail\"\n}\n```\n\n**UI fallback:** open the eval in AI observability → Evaluations → \"Summarize results\" button, filter = fail.\n\nIf the FAIL bucket is large, the classification step is too strict — relax it. If the PASS bucket has lots of generic reasonings, iterate on the prompt to enforce concreteness. The summary tool gives a quick read on this without you having to scroll through individual events.\n\nSpot-check raw events when needed (note: the stored result value is `'True'`, not `'PASS'` — see step 6):\n\n```sql\nSELECT\n    properties.$ai_evaluation_reasoning AS reasoning,\n    properties.$ai_trace_id AS trace_id,\n    timestamp\nFROM events\nWHERE event = '$ai_evaluation'\n    AND properties.$ai_evaluation_name = '\u003Cyour eval name>'\n    AND properties.$ai_evaluation_result = 'True'\n    AND timestamp > now() - INTERVAL 1 DAY\nORDER BY timestamp DESC\nLIMIT 25\n```\n\n## Tips\n\n- The reasoning field IS the Slack message — design the prompt for that, not for \"chain of thought before classification.\" Models can produce structured Slack-ready text in one pass.\n- LLM judges are non-deterministic across reruns. Expect 1-5% noise even with a fixed prompt and model. If you need reproducibility, pin a deterministic provider\u002Fseed in `model_configuration`.\n- Keep the eval scoped tightly via `conditions.filters` on `$ai_trace_id` prefix. Otherwise it fans out to every `$ai_generation` event in the project and burns LLM cost.\n- For high-volume features (>10k traces\u002Fweek), consider sampling — set the eval to run on a percentage of matching events rather than all of them. Slack flooding is a real failure mode.\n- The \"View Trigger Session\" button is the highest-value link in the alert. Without it, the feed is just text — you can't watch what the user was actually doing. Verify it works in step 7 before considering the feed shipped.\n- Once the feed is live, periodically re-run the eval summary tool with `filter: \"pass\"` to surface the dominant use case clusters. That's how you turn the feed into actual product insights instead of just a notification stream.\n",{"data":38,"body":39},{"name":4,"description":6},{"type":40,"children":41},"root",[42,51,57,87,94,146,167,173,178,212,277,296,302,509,521,527,725,744,750,757,767,877,898,908,1007,1083,1088,1094,1099,1107,1443,1451,1888,1899,1939,1956,1962,1974,1979,1989,2007,2113,2154,2167,2172,2178,2191,2207,2852,2857,2883,2888,2902,3038,3048,3069,3074,3148,3154,3159,3172,3260,3269,3281,3287,3305,3323,3330,3343,3349,3449,3533,3587,3628,3634,3704,4621,4633,4653,4659,4664,4724,4729,4735,4740,4777,4803,4809,4845,4881,4886,4928,4934,4961,5025,5045,5051,5056,5069,5167,5176,5181,5200,5291,5297,5366],{"type":43,"tag":44,"props":45,"children":47},"element","h1",{"id":46},"building-a-feature-usage-feed-via-llm-evals",[48],{"type":49,"value":50},"text","Building a feature usage feed via LLM evals",{"type":43,"tag":52,"props":53,"children":54},"p",{},[55],{"type":49,"value":56},"Some PostHog features (group session summaries, single session summaries, replay AI search, error tracking AI debug, etc.) generate hundreds or thousands of LLM traces per week. Reading them by hand is not feasible. This skill covers the end-to-end pattern for turning that trace volume into a live Slack feed of canonical use cases — what users are actually doing with the feature.",{"type":43,"tag":52,"props":58,"children":59},{},[60,62,68,70,76,78,85],{"type":49,"value":61},"The workflow is ",{"type":43,"tag":63,"props":64,"children":65},"strong",{},[66],{"type":49,"value":67},"mixed, and leans UI",{"type":49,"value":69},". Trace inspection and filter discovery (steps 1-2) are MCP-driven. Eval creation, dry-running, and enabling (steps 4-5) are MCP-driven ",{"type":43,"tag":71,"props":72,"children":73},"em",{},[74],{"type":49,"value":75},"when",{"type":49,"value":77}," ",{"type":43,"tag":79,"props":80,"children":82},"code",{"className":81},[],[83],{"type":49,"value":84},"posthog:llma-evaluation-*",{"type":49,"value":86}," tools are exposed to your agent — but they often aren't, in which case fall back to the UI (Data pipeline → destinations for the alert is always UI). Each step flags its UI fallback. Expect to finish in the UI even when you start from chat.",{"type":43,"tag":88,"props":89,"children":91},"h2",{"id":90},"when-to-use",[92],{"type":49,"value":93},"When to use",{"type":43,"tag":95,"props":96,"children":97},"ul",{},[98,112,123,128],{"type":43,"tag":99,"props":100,"children":101},"li",{},[102,104,110],{"type":49,"value":103},"\"How are people actually using ",{"type":43,"tag":105,"props":106,"children":107},"span",{},[108],{"type":49,"value":109},"feature X",{"type":49,"value":111}," in production?\"",{"type":43,"tag":99,"props":113,"children":114},{},[115,117,121],{"type":49,"value":116},"\"Can we identify the canonical use cases for ",{"type":43,"tag":105,"props":118,"children":119},{},[120],{"type":49,"value":109},{"type":49,"value":122}," so we can write better docs \u002F prioritize improvements?\"",{"type":43,"tag":99,"props":124,"children":125},{},[126],{"type":49,"value":127},"\"I want a Slack feed of representative usage examples without manually skimming traces.\"",{"type":43,"tag":99,"props":129,"children":130},{},[131,133,137,139,144],{"type":49,"value":132},"\"Set up a feed of use cases for ",{"type":43,"tag":105,"props":134,"children":135},{},[136],{"type":49,"value":109},{"type":49,"value":138}," in #team-",{"type":43,"tag":105,"props":140,"children":141},{},[142],{"type":49,"value":143},"area",{"type":49,"value":145},"-usage.\"",{"type":43,"tag":52,"props":147,"children":148},{},[149,151,157,159,165],{"type":49,"value":150},"If the user just wants to debug a single trace or tune an existing eval, redirect to ",{"type":43,"tag":79,"props":152,"children":154},{"className":153},[],[155],{"type":49,"value":156},"exploring-llm-traces",{"type":49,"value":158}," or ",{"type":43,"tag":79,"props":160,"children":162},{"className":161},[],[163],{"type":49,"value":164},"exploring-llm-evaluations",{"type":49,"value":166}," instead.",{"type":43,"tag":88,"props":168,"children":170},{"id":169},"two-filter-patterns",[171],{"type":49,"value":172},"Two filter patterns",{"type":43,"tag":52,"props":174,"children":175},{},[176],{"type":49,"value":177},"This skill supports two different ways to scope an eval to \"the feature you care about\":",{"type":43,"tag":52,"props":179,"children":180},{},[181,186,188,194,196,202,204,210],{"type":43,"tag":63,"props":182,"children":183},{},[184],{"type":49,"value":185},"Pattern A — Feature-native trace_id prefix.",{"type":49,"value":187}," For standalone features that emit their own ",{"type":43,"tag":79,"props":189,"children":191},{"className":190},[],[192],{"type":49,"value":193},"$ai_trace_id",{"type":49,"value":195}," pattern (e.g. ",{"type":43,"tag":79,"props":197,"children":199},{"className":198},[],[200],{"type":49,"value":201},"session-summary:group:",{"type":49,"value":203},", ",{"type":43,"tag":79,"props":205,"children":207},{"className":206},[],[208],{"type":49,"value":209},"replay-search:",{"type":49,"value":211},", error-tracking-specific flows). Filter on the prefix.",{"type":43,"tag":52,"props":213,"children":214},{},[215,220,222,227,229,235,237,243,245,251,253,259,261,267,269,275],{"type":43,"tag":63,"props":216,"children":217},{},[218],{"type":49,"value":219},"Pattern B — PostHog AI agent mode.",{"type":49,"value":221}," For features the user interacts with ",{"type":43,"tag":71,"props":223,"children":224},{},[225],{"type":49,"value":226},"via",{"type":49,"value":228}," PostHog AI in a specific agent mode (error tracking, product analytics, session replay, SQL, flags, surveys, AI observability). Filter on ",{"type":43,"tag":79,"props":230,"children":232},{"className":231},[],[233],{"type":49,"value":234},"ai_product = 'posthog_ai' AND agent_mode = '\u003Cmode>'",{"type":49,"value":236},". This requires PR #55160 (merged April 2026) to be deployed, which threads ",{"type":43,"tag":79,"props":238,"children":240},{"className":239},[],[241],{"type":49,"value":242},"agent_mode",{"type":49,"value":244}," and ",{"type":43,"tag":79,"props":246,"children":248},{"className":247},[],[249],{"type":49,"value":250},"supermode",{"type":49,"value":252}," onto every ",{"type":43,"tag":79,"props":254,"children":256},{"className":255},[],[257],{"type":49,"value":258},"$ai_generation",{"type":49,"value":260}," emitted by the chat agent loop. A useful ergonomic side-effect: ",{"type":43,"tag":79,"props":262,"children":264},{"className":263},[],[265],{"type":49,"value":266},"agent_mode IS NOT NULL",{"type":49,"value":268}," is a reliable \"user-facing chat turn\" filter — batch jobs and tool-internal LLM calls go through different code paths and have ",{"type":43,"tag":79,"props":270,"children":272},{"className":271},[],[273],{"type":49,"value":274},"agent_mode=null",{"type":49,"value":276},", so they're excluded for free.",{"type":43,"tag":52,"props":278,"children":279},{},[280,282,287,289,294],{"type":49,"value":281},"If the user asks \"what are users trying to DO in ",{"type":43,"tag":105,"props":283,"children":284},{},[285],{"type":49,"value":286},"ET \u002F replay \u002F SQL \u002F flags \u002F surveys",{"type":49,"value":288}," mode of PostHog AI\", that's Pattern B. If they ask \"what use cases does ",{"type":43,"tag":105,"props":290,"children":291},{},[292],{"type":49,"value":293},"standalone feature",{"type":49,"value":295}," cover\", that's Pattern A. Pick the pattern first — the prompt, filter, and Slack channel naming all follow from it.",{"type":43,"tag":88,"props":297,"children":299},{"id":298},"prerequisites",[300],{"type":49,"value":301},"Prerequisites",{"type":43,"tag":303,"props":304,"children":305},"table",{},[306,325],{"type":43,"tag":307,"props":308,"children":309},"thead",{},[310],{"type":43,"tag":311,"props":312,"children":313},"tr",{},[314,320],{"type":43,"tag":315,"props":316,"children":317},"th",{},[318],{"type":49,"value":319},"Requirement",{"type":43,"tag":315,"props":321,"children":322},{},[323],{"type":49,"value":324},"How to verify",{"type":43,"tag":326,"props":327,"children":328},"tbody",{},[329,370,418,455,488],{"type":43,"tag":311,"props":330,"children":331},{},[332,352],{"type":43,"tag":333,"props":334,"children":335},"td",{},[336,338,343,345,350],{"type":49,"value":337},"(Pattern A) Feature emits ",{"type":43,"tag":79,"props":339,"children":341},{"className":340},[],[342],{"type":49,"value":258},{"type":49,"value":344}," events with a stable ",{"type":43,"tag":79,"props":346,"children":348},{"className":347},[],[349],{"type":49,"value":193},{"type":49,"value":351}," pattern",{"type":43,"tag":333,"props":353,"children":354},{},[355,361,363,368],{"type":43,"tag":79,"props":356,"children":358},{"className":357},[],[359],{"type":49,"value":360},"posthog:execute-sql",{"type":49,"value":362}," for distinct ",{"type":43,"tag":79,"props":364,"children":366},{"className":365},[],[367],{"type":49,"value":193},{"type":49,"value":369}," prefixes",{"type":43,"tag":311,"props":371,"children":372},{},[373,392],{"type":43,"tag":333,"props":374,"children":375},{},[376,378,383,385,390],{"type":49,"value":377},"(Pattern B) ",{"type":43,"tag":79,"props":379,"children":381},{"className":380},[],[382],{"type":49,"value":242},{"type":49,"value":384}," property is present on recent ",{"type":43,"tag":79,"props":386,"children":388},{"className":387},[],[389],{"type":49,"value":258},{"type":49,"value":391}," events",{"type":43,"tag":333,"props":393,"children":394},{},[395,400,402,408,410,416],{"type":43,"tag":79,"props":396,"children":398},{"className":397},[],[399],{"type":49,"value":360},{"type":49,"value":401}," group-by ",{"type":43,"tag":79,"props":403,"children":405},{"className":404},[],[406],{"type":49,"value":407},"properties.agent_mode",{"type":49,"value":409}," on recent ",{"type":43,"tag":79,"props":411,"children":413},{"className":412},[],[414],{"type":49,"value":415},"ai_product='posthog_ai'",{"type":49,"value":417}," events. Null bucket is normal (batch jobs + tool-internal calls) — you want non-null coverage across the modes you care about.",{"type":43,"tag":311,"props":419,"children":420},{},[421,439],{"type":43,"tag":333,"props":422,"children":423},{},[424,430,432,437],{"type":43,"tag":79,"props":425,"children":427},{"className":426},[],[428],{"type":49,"value":429},"$session_id",{"type":49,"value":431}," is attached to the ",{"type":43,"tag":79,"props":433,"children":435},{"className":434},[],[436],{"type":49,"value":258},{"type":49,"value":438}," events (links trace to trigger session)",{"type":43,"tag":333,"props":440,"children":441},{},[442,447,449],{"type":43,"tag":79,"props":443,"children":445},{"className":444},[],[446],{"type":49,"value":360},{"type":49,"value":448}," for ",{"type":43,"tag":79,"props":450,"children":452},{"className":451},[],[453],{"type":49,"value":454},"countIf($session_id IS NOT NULL) \u002F count()",{"type":43,"tag":311,"props":456,"children":457},{},[458,476],{"type":43,"tag":333,"props":459,"children":460},{},[461,466,468,474],{"type":43,"tag":79,"props":462,"children":464},{"className":463},[],[465],{"type":49,"value":429},{"type":49,"value":467}," is also attached to the ",{"type":43,"tag":79,"props":469,"children":471},{"className":470},[],[472],{"type":49,"value":473},"$ai_evaluation",{"type":49,"value":475}," events (lets the Slack alert link to the session)",{"type":43,"tag":333,"props":477,"children":478},{},[479,481,486],{"type":49,"value":480},"Same query but on ",{"type":43,"tag":79,"props":482,"children":484},{"className":483},[],[485],{"type":49,"value":473},{"type":49,"value":487}," events after the eval has run once",{"type":43,"tag":311,"props":489,"children":490},{},[491,496],{"type":43,"tag":333,"props":492,"children":493},{},[494],{"type":49,"value":495},"User has organisation-level AI data processing approval",{"type":43,"tag":333,"props":497,"children":498},{},[499,501,507],{"type":49,"value":500},"Required for ",{"type":43,"tag":79,"props":502,"children":504},{"className":503},[],[505],{"type":49,"value":506},"llm_judge",{"type":49,"value":508}," evaluations and the eval summary tool",{"type":43,"tag":52,"props":510,"children":511},{},[512,514,519],{"type":49,"value":513},"If ",{"type":43,"tag":79,"props":515,"children":517},{"className":516},[],[518],{"type":49,"value":429},{"type":49,"value":520}," is missing on either event type, file a backend fix before continuing — there is no UI workaround. The session-summary feature has a worked example of the threading pattern in PR #54952. For Pattern B, the agent-mode threading pattern is in PR #55160.",{"type":43,"tag":88,"props":522,"children":524},{"id":523},"tools",[525],{"type":49,"value":526},"Tools",{"type":43,"tag":303,"props":528,"children":529},{},[530,546],{"type":43,"tag":307,"props":531,"children":532},{},[533],{"type":43,"tag":311,"props":534,"children":535},{},[536,541],{"type":43,"tag":315,"props":537,"children":538},{},[539],{"type":49,"value":540},"Tool",{"type":43,"tag":315,"props":542,"children":543},{},[544],{"type":49,"value":545},"Purpose",{"type":43,"tag":326,"props":547,"children":548},{},[549,572,589,605,629,651,673,695],{"type":43,"tag":311,"props":550,"children":551},{},[552,561],{"type":43,"tag":333,"props":553,"children":554},{},[555],{"type":43,"tag":79,"props":556,"children":558},{"className":557},[],[559],{"type":49,"value":560},"posthog:query-llm-traces-list",{"type":43,"tag":333,"props":562,"children":563},{},[564,566,571],{"type":49,"value":565},"Find sample traces matching the feature's ",{"type":43,"tag":79,"props":567,"children":569},{"className":568},[],[570],{"type":49,"value":193},{"type":49,"value":351},{"type":43,"tag":311,"props":573,"children":574},{},[575,584],{"type":43,"tag":333,"props":576,"children":577},{},[578],{"type":43,"tag":79,"props":579,"children":581},{"className":580},[],[582],{"type":49,"value":583},"posthog:query-llm-trace",{"type":43,"tag":333,"props":585,"children":586},{},[587],{"type":49,"value":588},"Inspect a specific trace's contents end-to-end",{"type":43,"tag":311,"props":590,"children":591},{},[592,600],{"type":43,"tag":333,"props":593,"children":594},{},[595],{"type":43,"tag":79,"props":596,"children":598},{"className":597},[],[599],{"type":49,"value":360},{"type":43,"tag":333,"props":601,"children":602},{},[603],{"type":49,"value":604},"Verify trace volume, session_id coverage, eval result distributions",{"type":43,"tag":311,"props":606,"children":607},{},[608,617],{"type":43,"tag":333,"props":609,"children":610},{},[611],{"type":43,"tag":79,"props":612,"children":614},{"className":613},[],[615],{"type":49,"value":616},"posthog:llma-evaluation-create",{"type":43,"tag":333,"props":618,"children":619},{},[620,622,627],{"type":49,"value":621},"(",{"type":43,"tag":63,"props":623,"children":624},{},[625],{"type":49,"value":626},"often unexposed",{"type":49,"value":628}," — UI fallback: AI observability → Evaluations → New) Create the LLM-judge eval (disabled at first)",{"type":43,"tag":311,"props":630,"children":631},{},[632,641],{"type":43,"tag":333,"props":633,"children":634},{},[635],{"type":43,"tag":79,"props":636,"children":638},{"className":637},[],[639],{"type":49,"value":640},"posthog:llma-evaluation-run",{"type":43,"tag":333,"props":642,"children":643},{},[644,645,649],{"type":49,"value":621},{"type":43,"tag":63,"props":646,"children":647},{},[648],{"type":49,"value":626},{"type":49,"value":650}," — UI fallback: the eval's detail page has a \"Run on event\" button) Dry-run the eval against specific generations during prompt iteration",{"type":43,"tag":311,"props":652,"children":653},{},[654,663],{"type":43,"tag":333,"props":655,"children":656},{},[657],{"type":43,"tag":79,"props":658,"children":660},{"className":659},[],[661],{"type":49,"value":662},"posthog:llma-evaluation-update",{"type":43,"tag":333,"props":664,"children":665},{},[666,667,671],{"type":49,"value":621},{"type":43,"tag":63,"props":668,"children":669},{},[670],{"type":49,"value":626},{"type":49,"value":672}," — UI fallback: edit the eval in AI observability → Evaluations) Tweak the prompt \u002F enable when ready",{"type":43,"tag":311,"props":674,"children":675},{},[676,685],{"type":43,"tag":333,"props":677,"children":678},{},[679],{"type":43,"tag":79,"props":680,"children":682},{"className":681},[],[683],{"type":49,"value":684},"posthog:llma-evaluation-summary-create",{"type":43,"tag":333,"props":686,"children":687},{},[688,689,693],{"type":49,"value":621},{"type":43,"tag":63,"props":690,"children":691},{},[692],{"type":49,"value":626},{"type":49,"value":694}," — UI fallback: the eval detail page has a \"Summarize results\" button) After the feed is running, get an AI summary of pass\u002FN\u002FA patterns to validate signal quality",{"type":43,"tag":311,"props":696,"children":697},{},[698,715],{"type":43,"tag":333,"props":699,"children":700},{},[701,707,709],{"type":43,"tag":79,"props":702,"children":704},{"className":703},[],[705],{"type":49,"value":706},"posthog:workflows-list",{"type":49,"value":708}," \u002F ",{"type":43,"tag":79,"props":710,"children":712},{"className":711},[],[713],{"type":49,"value":714},"posthog:workflows-get",{"type":43,"tag":333,"props":716,"children":717},{},[718,719,723],{"type":49,"value":621},{"type":43,"tag":63,"props":720,"children":721},{},[722],{"type":49,"value":626},{"type":49,"value":724}," — UI: Data pipeline → Workflows) Browse existing workflow configs — useful for cloning an existing feed's structure when setting up a new one. Read-only; no create\u002Fupdate tool is exposed yet, so step 6's Slack workflow setup is UI-only.",{"type":43,"tag":52,"props":726,"children":727},{},[728,730,742],{"type":49,"value":729},"Before starting, ",{"type":43,"tag":63,"props":731,"children":732},{},[733,735,740],{"type":49,"value":734},"check which of the ",{"type":43,"tag":79,"props":736,"children":738},{"className":737},[],[739],{"type":49,"value":84},{"type":49,"value":741}," tools are actually exposed in your agent's MCP tool set.",{"type":49,"value":743}," If they aren't loaded, treat steps 4-5 as UI walkthroughs rather than tool calls.",{"type":43,"tag":88,"props":745,"children":747},{"id":746},"workflow",[748],{"type":49,"value":749},"Workflow",{"type":43,"tag":751,"props":752,"children":754},"h3",{"id":753},"step-1-identify-the-filter",[755],{"type":49,"value":756},"Step 1 — Identify the filter",{"type":43,"tag":52,"props":758,"children":759},{},[760,765],{"type":43,"tag":63,"props":761,"children":762},{},[763],{"type":49,"value":764},"Pattern A (feature-native trace_id prefix):",{"type":49,"value":766}," find the prefix that maps to your feature.",{"type":43,"tag":768,"props":769,"children":774},"pre",{"className":770,"code":771,"language":772,"meta":773,"style":773},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","SELECT\n    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[1] AS root,\n    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[2] AS subtype,\n    count() AS events\nFROM events\nWHERE timestamp > now() - INTERVAL 3 DAY\n    AND event = '$ai_generation'\n    AND properties.$ai_trace_id IS NOT NULL\nGROUP BY root, subtype\nORDER BY events DESC\nLIMIT 25\n","sql","",[775],{"type":43,"tag":79,"props":776,"children":777},{"__ignoreMap":773},[778,788,797,806,814,823,832,841,850,859,868],{"type":43,"tag":105,"props":779,"children":782},{"class":780,"line":781},"line",1,[783],{"type":43,"tag":105,"props":784,"children":785},{},[786],{"type":49,"value":787},"SELECT\n",{"type":43,"tag":105,"props":789,"children":791},{"class":780,"line":790},2,[792],{"type":43,"tag":105,"props":793,"children":794},{},[795],{"type":49,"value":796},"    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[1] AS root,\n",{"type":43,"tag":105,"props":798,"children":800},{"class":780,"line":799},3,[801],{"type":43,"tag":105,"props":802,"children":803},{},[804],{"type":49,"value":805},"    splitByChar(':', coalesce(properties.$ai_trace_id, ''))[2] AS subtype,\n",{"type":43,"tag":105,"props":807,"children":808},{"class":780,"line":30},[809],{"type":43,"tag":105,"props":810,"children":811},{},[812],{"type":49,"value":813},"    count() AS events\n",{"type":43,"tag":105,"props":815,"children":817},{"class":780,"line":816},5,[818],{"type":43,"tag":105,"props":819,"children":820},{},[821],{"type":49,"value":822},"FROM events\n",{"type":43,"tag":105,"props":824,"children":826},{"class":780,"line":825},6,[827],{"type":43,"tag":105,"props":828,"children":829},{},[830],{"type":49,"value":831},"WHERE timestamp > now() - INTERVAL 3 DAY\n",{"type":43,"tag":105,"props":833,"children":835},{"class":780,"line":834},7,[836],{"type":43,"tag":105,"props":837,"children":838},{},[839],{"type":49,"value":840},"    AND event = '$ai_generation'\n",{"type":43,"tag":105,"props":842,"children":844},{"class":780,"line":843},8,[845],{"type":43,"tag":105,"props":846,"children":847},{},[848],{"type":49,"value":849},"    AND properties.$ai_trace_id IS NOT NULL\n",{"type":43,"tag":105,"props":851,"children":853},{"class":780,"line":852},9,[854],{"type":43,"tag":105,"props":855,"children":856},{},[857],{"type":49,"value":858},"GROUP BY root, subtype\n",{"type":43,"tag":105,"props":860,"children":862},{"class":780,"line":861},10,[863],{"type":43,"tag":105,"props":864,"children":865},{},[866],{"type":49,"value":867},"ORDER BY events DESC\n",{"type":43,"tag":105,"props":869,"children":871},{"class":780,"line":870},11,[872],{"type":43,"tag":105,"props":873,"children":874},{},[875],{"type":49,"value":876},"LIMIT 25\n",{"type":43,"tag":52,"props":878,"children":879},{},[880,882,888,890,896],{"type":49,"value":881},"Note: ",{"type":43,"tag":79,"props":883,"children":885},{"className":884},[],[886],{"type":49,"value":887},"coalesce(..., '')",{"type":49,"value":889}," is load-bearing — ",{"type":43,"tag":79,"props":891,"children":893},{"className":892},[],[894],{"type":49,"value":895},"splitByChar",{"type":49,"value":897}," on a nullable column errors out in HogQL otherwise.",{"type":43,"tag":52,"props":899,"children":900},{},[901,906],{"type":43,"tag":63,"props":902,"children":903},{},[904],{"type":49,"value":905},"Pattern B (PostHog AI agent mode):",{"type":49,"value":907}," verify coverage and volume for the mode you're targeting.",{"type":43,"tag":768,"props":909,"children":911},{"className":770,"code":910,"language":772,"meta":773,"style":773},"SELECT\n    properties.agent_mode AS agent_mode,\n    properties.supermode AS supermode,\n    count() AS events,\n    count(DISTINCT properties.$ai_trace_id) AS traces\nFROM events\nWHERE timestamp > now() - INTERVAL 3 DAY\n    AND event = '$ai_generation'\n    AND properties.ai_product = 'posthog_ai'\nGROUP BY agent_mode, supermode\nORDER BY events DESC\nLIMIT 20\n",[912],{"type":43,"tag":79,"props":913,"children":914},{"__ignoreMap":773},[915,922,930,938,946,954,961,968,975,983,991,998],{"type":43,"tag":105,"props":916,"children":917},{"class":780,"line":781},[918],{"type":43,"tag":105,"props":919,"children":920},{},[921],{"type":49,"value":787},{"type":43,"tag":105,"props":923,"children":924},{"class":780,"line":790},[925],{"type":43,"tag":105,"props":926,"children":927},{},[928],{"type":49,"value":929},"    properties.agent_mode AS agent_mode,\n",{"type":43,"tag":105,"props":931,"children":932},{"class":780,"line":799},[933],{"type":43,"tag":105,"props":934,"children":935},{},[936],{"type":49,"value":937},"    properties.supermode AS supermode,\n",{"type":43,"tag":105,"props":939,"children":940},{"class":780,"line":30},[941],{"type":43,"tag":105,"props":942,"children":943},{},[944],{"type":49,"value":945},"    count() AS events,\n",{"type":43,"tag":105,"props":947,"children":948},{"class":780,"line":816},[949],{"type":43,"tag":105,"props":950,"children":951},{},[952],{"type":49,"value":953},"    count(DISTINCT properties.$ai_trace_id) AS traces\n",{"type":43,"tag":105,"props":955,"children":956},{"class":780,"line":825},[957],{"type":43,"tag":105,"props":958,"children":959},{},[960],{"type":49,"value":822},{"type":43,"tag":105,"props":962,"children":963},{"class":780,"line":834},[964],{"type":43,"tag":105,"props":965,"children":966},{},[967],{"type":49,"value":831},{"type":43,"tag":105,"props":969,"children":970},{"class":780,"line":843},[971],{"type":43,"tag":105,"props":972,"children":973},{},[974],{"type":49,"value":840},{"type":43,"tag":105,"props":976,"children":977},{"class":780,"line":852},[978],{"type":43,"tag":105,"props":979,"children":980},{},[981],{"type":49,"value":982},"    AND properties.ai_product = 'posthog_ai'\n",{"type":43,"tag":105,"props":984,"children":985},{"class":780,"line":861},[986],{"type":43,"tag":105,"props":987,"children":988},{},[989],{"type":49,"value":990},"GROUP BY agent_mode, supermode\n",{"type":43,"tag":105,"props":992,"children":993},{"class":780,"line":870},[994],{"type":43,"tag":105,"props":995,"children":996},{},[997],{"type":49,"value":867},{"type":43,"tag":105,"props":999,"children":1001},{"class":780,"line":1000},12,[1002],{"type":43,"tag":105,"props":1003,"children":1004},{},[1005],{"type":49,"value":1006},"LIMIT 20\n",{"type":43,"tag":52,"props":1008,"children":1009},{},[1010,1012,1017,1019,1025,1026,1032,1033,1038,1039,1045,1046,1052,1053,1059,1060,1066,1067,1073,1075,1081],{"type":49,"value":1011},"Expected values for ",{"type":43,"tag":79,"props":1013,"children":1015},{"className":1014},[],[1016],{"type":49,"value":242},{"type":49,"value":1018},": ",{"type":43,"tag":79,"props":1020,"children":1022},{"className":1021},[],[1023],{"type":49,"value":1024},"error_tracking",{"type":49,"value":203},{"type":43,"tag":79,"props":1027,"children":1029},{"className":1028},[],[1030],{"type":49,"value":1031},"product_analytics",{"type":49,"value":203},{"type":43,"tag":79,"props":1034,"children":1036},{"className":1035},[],[1037],{"type":49,"value":772},{"type":49,"value":203},{"type":43,"tag":79,"props":1040,"children":1042},{"className":1041},[],[1043],{"type":49,"value":1044},"session_replay",{"type":49,"value":203},{"type":43,"tag":79,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":49,"value":1051},"flags",{"type":49,"value":203},{"type":43,"tag":79,"props":1054,"children":1056},{"className":1055},[],[1057],{"type":49,"value":1058},"survey",{"type":49,"value":203},{"type":43,"tag":79,"props":1061,"children":1063},{"className":1062},[],[1064],{"type":49,"value":1065},"llm_analytics",{"type":49,"value":203},{"type":43,"tag":79,"props":1068,"children":1070},{"className":1069},[],[1071],{"type":49,"value":1072},"null",{"type":49,"value":1074},". Null ≈ batch jobs + tool-internal calls (not user chat). ",{"type":43,"tag":79,"props":1076,"children":1078},{"className":1077},[],[1079],{"type":49,"value":1080},"supermode='plan'",{"type":49,"value":1082}," splits planning turns from execution turns — worth calling out separately if your feed is about plan-mode specifically.",{"type":43,"tag":52,"props":1084,"children":1085},{},[1086],{"type":49,"value":1087},"Record the mode + rough volume. Low-volume modes (\u003C100 events\u002Fday) will produce a trickle-feed that's hard to validate early; high-volume modes (>1k\u002Fday) may need sampling to avoid Slack flooding. See the \"Tips\" section on sampling.",{"type":43,"tag":751,"props":1089,"children":1091},{"id":1090},"step-2-pull-a-handful-of-sample-traces",[1092],{"type":49,"value":1093},"Step 2 — Pull a handful of sample traces",{"type":43,"tag":52,"props":1095,"children":1096},{},[1097],{"type":49,"value":1098},"Use these for prompt iteration in step 4.",{"type":43,"tag":52,"props":1100,"children":1101},{},[1102],{"type":43,"tag":63,"props":1103,"children":1104},{},[1105],{"type":49,"value":1106},"Pattern A:",{"type":43,"tag":768,"props":1108,"children":1112},{"className":1109,"code":1110,"language":1111,"meta":773,"style":773},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","posthog:query-llm-traces-list\n{\n  \"properties\": [\n    { \"type\": \"event\", \"key\": \"$ai_trace_id\", \"operator\": \"icontains\", \"value\": \"\u003Cyour-prefix-here>\" }\n  ],\n  \"limit\": 10,\n  \"dateRange\": { \"date_from\": \"-2d\" },\n  \"randomOrder\": true\n}\n","json",[1113],{"type":43,"tag":79,"props":1114,"children":1115},{"__ignoreMap":773},[1116,1125,1134,1163,1311,1319,1350,1410,1435],{"type":43,"tag":105,"props":1117,"children":1118},{"class":780,"line":781},[1119],{"type":43,"tag":105,"props":1120,"children":1122},{"style":1121},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1123],{"type":49,"value":1124},"posthog:query-llm-traces-list\n",{"type":43,"tag":105,"props":1126,"children":1127},{"class":780,"line":790},[1128],{"type":43,"tag":105,"props":1129,"children":1131},{"style":1130},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1132],{"type":49,"value":1133},"{\n",{"type":43,"tag":105,"props":1135,"children":1136},{"class":780,"line":799},[1137,1142,1148,1153,1158],{"type":43,"tag":105,"props":1138,"children":1139},{"style":1130},[1140],{"type":49,"value":1141},"  \"",{"type":43,"tag":105,"props":1143,"children":1145},{"style":1144},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1146],{"type":49,"value":1147},"properties",{"type":43,"tag":105,"props":1149,"children":1150},{"style":1130},[1151],{"type":49,"value":1152},"\"",{"type":43,"tag":105,"props":1154,"children":1155},{"style":1130},[1156],{"type":49,"value":1157},":",{"type":43,"tag":105,"props":1159,"children":1160},{"style":1130},[1161],{"type":49,"value":1162}," [\n",{"type":43,"tag":105,"props":1164,"children":1165},{"class":780,"line":30},[1166,1171,1176,1182,1186,1190,1194,1200,1204,1209,1213,1218,1222,1226,1230,1234,1238,1242,1246,1251,1255,1259,1263,1268,1272,1276,1280,1285,1289,1293,1297,1302,1306],{"type":43,"tag":105,"props":1167,"children":1168},{"style":1130},[1169],{"type":49,"value":1170},"    {",{"type":43,"tag":105,"props":1172,"children":1173},{"style":1130},[1174],{"type":49,"value":1175}," \"",{"type":43,"tag":105,"props":1177,"children":1179},{"style":1178},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1180],{"type":49,"value":1181},"type",{"type":43,"tag":105,"props":1183,"children":1184},{"style":1130},[1185],{"type":49,"value":1152},{"type":43,"tag":105,"props":1187,"children":1188},{"style":1130},[1189],{"type":49,"value":1157},{"type":43,"tag":105,"props":1191,"children":1192},{"style":1130},[1193],{"type":49,"value":1175},{"type":43,"tag":105,"props":1195,"children":1197},{"style":1196},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1198],{"type":49,"value":1199},"event",{"type":43,"tag":105,"props":1201,"children":1202},{"style":1130},[1203],{"type":49,"value":1152},{"type":43,"tag":105,"props":1205,"children":1206},{"style":1130},[1207],{"type":49,"value":1208},",",{"type":43,"tag":105,"props":1210,"children":1211},{"style":1130},[1212],{"type":49,"value":1175},{"type":43,"tag":105,"props":1214,"children":1215},{"style":1178},[1216],{"type":49,"value":1217},"key",{"type":43,"tag":105,"props":1219,"children":1220},{"style":1130},[1221],{"type":49,"value":1152},{"type":43,"tag":105,"props":1223,"children":1224},{"style":1130},[1225],{"type":49,"value":1157},{"type":43,"tag":105,"props":1227,"children":1228},{"style":1130},[1229],{"type":49,"value":1175},{"type":43,"tag":105,"props":1231,"children":1232},{"style":1196},[1233],{"type":49,"value":193},{"type":43,"tag":105,"props":1235,"children":1236},{"style":1130},[1237],{"type":49,"value":1152},{"type":43,"tag":105,"props":1239,"children":1240},{"style":1130},[1241],{"type":49,"value":1208},{"type":43,"tag":105,"props":1243,"children":1244},{"style":1130},[1245],{"type":49,"value":1175},{"type":43,"tag":105,"props":1247,"children":1248},{"style":1178},[1249],{"type":49,"value":1250},"operator",{"type":43,"tag":105,"props":1252,"children":1253},{"style":1130},[1254],{"type":49,"value":1152},{"type":43,"tag":105,"props":1256,"children":1257},{"style":1130},[1258],{"type":49,"value":1157},{"type":43,"tag":105,"props":1260,"children":1261},{"style":1130},[1262],{"type":49,"value":1175},{"type":43,"tag":105,"props":1264,"children":1265},{"style":1196},[1266],{"type":49,"value":1267},"icontains",{"type":43,"tag":105,"props":1269,"children":1270},{"style":1130},[1271],{"type":49,"value":1152},{"type":43,"tag":105,"props":1273,"children":1274},{"style":1130},[1275],{"type":49,"value":1208},{"type":43,"tag":105,"props":1277,"children":1278},{"style":1130},[1279],{"type":49,"value":1175},{"type":43,"tag":105,"props":1281,"children":1282},{"style":1178},[1283],{"type":49,"value":1284},"value",{"type":43,"tag":105,"props":1286,"children":1287},{"style":1130},[1288],{"type":49,"value":1152},{"type":43,"tag":105,"props":1290,"children":1291},{"style":1130},[1292],{"type":49,"value":1157},{"type":43,"tag":105,"props":1294,"children":1295},{"style":1130},[1296],{"type":49,"value":1175},{"type":43,"tag":105,"props":1298,"children":1299},{"style":1196},[1300],{"type":49,"value":1301},"\u003Cyour-prefix-here>",{"type":43,"tag":105,"props":1303,"children":1304},{"style":1130},[1305],{"type":49,"value":1152},{"type":43,"tag":105,"props":1307,"children":1308},{"style":1130},[1309],{"type":49,"value":1310}," }\n",{"type":43,"tag":105,"props":1312,"children":1313},{"class":780,"line":816},[1314],{"type":43,"tag":105,"props":1315,"children":1316},{"style":1130},[1317],{"type":49,"value":1318},"  ],\n",{"type":43,"tag":105,"props":1320,"children":1321},{"class":780,"line":825},[1322,1326,1331,1335,1339,1345],{"type":43,"tag":105,"props":1323,"children":1324},{"style":1130},[1325],{"type":49,"value":1141},{"type":43,"tag":105,"props":1327,"children":1328},{"style":1144},[1329],{"type":49,"value":1330},"limit",{"type":43,"tag":105,"props":1332,"children":1333},{"style":1130},[1334],{"type":49,"value":1152},{"type":43,"tag":105,"props":1336,"children":1337},{"style":1130},[1338],{"type":49,"value":1157},{"type":43,"tag":105,"props":1340,"children":1342},{"style":1341},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1343],{"type":49,"value":1344}," 10",{"type":43,"tag":105,"props":1346,"children":1347},{"style":1130},[1348],{"type":49,"value":1349},",\n",{"type":43,"tag":105,"props":1351,"children":1352},{"class":780,"line":834},[1353,1357,1362,1366,1370,1375,1379,1384,1388,1392,1396,1401,1405],{"type":43,"tag":105,"props":1354,"children":1355},{"style":1130},[1356],{"type":49,"value":1141},{"type":43,"tag":105,"props":1358,"children":1359},{"style":1144},[1360],{"type":49,"value":1361},"dateRange",{"type":43,"tag":105,"props":1363,"children":1364},{"style":1130},[1365],{"type":49,"value":1152},{"type":43,"tag":105,"props":1367,"children":1368},{"style":1130},[1369],{"type":49,"value":1157},{"type":43,"tag":105,"props":1371,"children":1372},{"style":1130},[1373],{"type":49,"value":1374}," {",{"type":43,"tag":105,"props":1376,"children":1377},{"style":1130},[1378],{"type":49,"value":1175},{"type":43,"tag":105,"props":1380,"children":1381},{"style":1178},[1382],{"type":49,"value":1383},"date_from",{"type":43,"tag":105,"props":1385,"children":1386},{"style":1130},[1387],{"type":49,"value":1152},{"type":43,"tag":105,"props":1389,"children":1390},{"style":1130},[1391],{"type":49,"value":1157},{"type":43,"tag":105,"props":1393,"children":1394},{"style":1130},[1395],{"type":49,"value":1175},{"type":43,"tag":105,"props":1397,"children":1398},{"style":1196},[1399],{"type":49,"value":1400},"-2d",{"type":43,"tag":105,"props":1402,"children":1403},{"style":1130},[1404],{"type":49,"value":1152},{"type":43,"tag":105,"props":1406,"children":1407},{"style":1130},[1408],{"type":49,"value":1409}," },\n",{"type":43,"tag":105,"props":1411,"children":1412},{"class":780,"line":843},[1413,1417,1422,1426,1430],{"type":43,"tag":105,"props":1414,"children":1415},{"style":1130},[1416],{"type":49,"value":1141},{"type":43,"tag":105,"props":1418,"children":1419},{"style":1144},[1420],{"type":49,"value":1421},"randomOrder",{"type":43,"tag":105,"props":1423,"children":1424},{"style":1130},[1425],{"type":49,"value":1152},{"type":43,"tag":105,"props":1427,"children":1428},{"style":1130},[1429],{"type":49,"value":1157},{"type":43,"tag":105,"props":1431,"children":1432},{"style":1130},[1433],{"type":49,"value":1434}," true\n",{"type":43,"tag":105,"props":1436,"children":1437},{"class":780,"line":852},[1438],{"type":43,"tag":105,"props":1439,"children":1440},{"style":1130},[1441],{"type":49,"value":1442},"}\n",{"type":43,"tag":52,"props":1444,"children":1445},{},[1446],{"type":43,"tag":63,"props":1447,"children":1448},{},[1449],{"type":49,"value":1450},"Pattern B:",{"type":43,"tag":768,"props":1452,"children":1454},{"className":1109,"code":1453,"language":1111,"meta":773,"style":773},"posthog:query-llm-traces-list\n{\n  \"properties\": [\n    { \"type\": \"event\", \"key\": \"ai_product\", \"operator\": \"exact\", \"value\": \"posthog_ai\" },\n    { \"type\": \"event\", \"key\": \"agent_mode\", \"operator\": \"exact\", \"value\": \"\u003Cmode-here>\" }\n  ],\n  \"limit\": 10,\n  \"dateRange\": { \"date_from\": \"-2d\" },\n  \"randomOrder\": true\n}\n",[1455],{"type":43,"tag":79,"props":1456,"children":1457},{"__ignoreMap":773},[1458,1465,1472,1495,1633,1769,1776,1803,1858,1881],{"type":43,"tag":105,"props":1459,"children":1460},{"class":780,"line":781},[1461],{"type":43,"tag":105,"props":1462,"children":1463},{"style":1121},[1464],{"type":49,"value":1124},{"type":43,"tag":105,"props":1466,"children":1467},{"class":780,"line":790},[1468],{"type":43,"tag":105,"props":1469,"children":1470},{"style":1130},[1471],{"type":49,"value":1133},{"type":43,"tag":105,"props":1473,"children":1474},{"class":780,"line":799},[1475,1479,1483,1487,1491],{"type":43,"tag":105,"props":1476,"children":1477},{"style":1130},[1478],{"type":49,"value":1141},{"type":43,"tag":105,"props":1480,"children":1481},{"style":1144},[1482],{"type":49,"value":1147},{"type":43,"tag":105,"props":1484,"children":1485},{"style":1130},[1486],{"type":49,"value":1152},{"type":43,"tag":105,"props":1488,"children":1489},{"style":1130},[1490],{"type":49,"value":1157},{"type":43,"tag":105,"props":1492,"children":1493},{"style":1130},[1494],{"type":49,"value":1162},{"type":43,"tag":105,"props":1496,"children":1497},{"class":780,"line":30},[1498,1502,1506,1510,1514,1518,1522,1526,1530,1534,1538,1542,1546,1550,1554,1559,1563,1567,1571,1575,1579,1583,1587,1592,1596,1600,1604,1608,1612,1616,1620,1625,1629],{"type":43,"tag":105,"props":1499,"children":1500},{"style":1130},[1501],{"type":49,"value":1170},{"type":43,"tag":105,"props":1503,"children":1504},{"style":1130},[1505],{"type":49,"value":1175},{"type":43,"tag":105,"props":1507,"children":1508},{"style":1178},[1509],{"type":49,"value":1181},{"type":43,"tag":105,"props":1511,"children":1512},{"style":1130},[1513],{"type":49,"value":1152},{"type":43,"tag":105,"props":1515,"children":1516},{"style":1130},[1517],{"type":49,"value":1157},{"type":43,"tag":105,"props":1519,"children":1520},{"style":1130},[1521],{"type":49,"value":1175},{"type":43,"tag":105,"props":1523,"children":1524},{"style":1196},[1525],{"type":49,"value":1199},{"type":43,"tag":105,"props":1527,"children":1528},{"style":1130},[1529],{"type":49,"value":1152},{"type":43,"tag":105,"props":1531,"children":1532},{"style":1130},[1533],{"type":49,"value":1208},{"type":43,"tag":105,"props":1535,"children":1536},{"style":1130},[1537],{"type":49,"value":1175},{"type":43,"tag":105,"props":1539,"children":1540},{"style":1178},[1541],{"type":49,"value":1217},{"type":43,"tag":105,"props":1543,"children":1544},{"style":1130},[1545],{"type":49,"value":1152},{"type":43,"tag":105,"props":1547,"children":1548},{"style":1130},[1549],{"type":49,"value":1157},{"type":43,"tag":105,"props":1551,"children":1552},{"style":1130},[1553],{"type":49,"value":1175},{"type":43,"tag":105,"props":1555,"children":1556},{"style":1196},[1557],{"type":49,"value":1558},"ai_product",{"type":43,"tag":105,"props":1560,"children":1561},{"style":1130},[1562],{"type":49,"value":1152},{"type":43,"tag":105,"props":1564,"children":1565},{"style":1130},[1566],{"type":49,"value":1208},{"type":43,"tag":105,"props":1568,"children":1569},{"style":1130},[1570],{"type":49,"value":1175},{"type":43,"tag":105,"props":1572,"children":1573},{"style":1178},[1574],{"type":49,"value":1250},{"type":43,"tag":105,"props":1576,"children":1577},{"style":1130},[1578],{"type":49,"value":1152},{"type":43,"tag":105,"props":1580,"children":1581},{"style":1130},[1582],{"type":49,"value":1157},{"type":43,"tag":105,"props":1584,"children":1585},{"style":1130},[1586],{"type":49,"value":1175},{"type":43,"tag":105,"props":1588,"children":1589},{"style":1196},[1590],{"type":49,"value":1591},"exact",{"type":43,"tag":105,"props":1593,"children":1594},{"style":1130},[1595],{"type":49,"value":1152},{"type":43,"tag":105,"props":1597,"children":1598},{"style":1130},[1599],{"type":49,"value":1208},{"type":43,"tag":105,"props":1601,"children":1602},{"style":1130},[1603],{"type":49,"value":1175},{"type":43,"tag":105,"props":1605,"children":1606},{"style":1178},[1607],{"type":49,"value":1284},{"type":43,"tag":105,"props":1609,"children":1610},{"style":1130},[1611],{"type":49,"value":1152},{"type":43,"tag":105,"props":1613,"children":1614},{"style":1130},[1615],{"type":49,"value":1157},{"type":43,"tag":105,"props":1617,"children":1618},{"style":1130},[1619],{"type":49,"value":1175},{"type":43,"tag":105,"props":1621,"children":1622},{"style":1196},[1623],{"type":49,"value":1624},"posthog_ai",{"type":43,"tag":105,"props":1626,"children":1627},{"style":1130},[1628],{"type":49,"value":1152},{"type":43,"tag":105,"props":1630,"children":1631},{"style":1130},[1632],{"type":49,"value":1409},{"type":43,"tag":105,"props":1634,"children":1635},{"class":780,"line":816},[1636,1640,1644,1648,1652,1656,1660,1664,1668,1672,1676,1680,1684,1688,1692,1696,1700,1704,1708,1712,1716,1720,1724,1728,1732,1736,1740,1744,1748,1752,1756,1761,1765],{"type":43,"tag":105,"props":1637,"children":1638},{"style":1130},[1639],{"type":49,"value":1170},{"type":43,"tag":105,"props":1641,"children":1642},{"style":1130},[1643],{"type":49,"value":1175},{"type":43,"tag":105,"props":1645,"children":1646},{"style":1178},[1647],{"type":49,"value":1181},{"type":43,"tag":105,"props":1649,"children":1650},{"style":1130},[1651],{"type":49,"value":1152},{"type":43,"tag":105,"props":1653,"children":1654},{"style":1130},[1655],{"type":49,"value":1157},{"type":43,"tag":105,"props":1657,"children":1658},{"style":1130},[1659],{"type":49,"value":1175},{"type":43,"tag":105,"props":1661,"children":1662},{"style":1196},[1663],{"type":49,"value":1199},{"type":43,"tag":105,"props":1665,"children":1666},{"style":1130},[1667],{"type":49,"value":1152},{"type":43,"tag":105,"props":1669,"children":1670},{"style":1130},[1671],{"type":49,"value":1208},{"type":43,"tag":105,"props":1673,"children":1674},{"style":1130},[1675],{"type":49,"value":1175},{"type":43,"tag":105,"props":1677,"children":1678},{"style":1178},[1679],{"type":49,"value":1217},{"type":43,"tag":105,"props":1681,"children":1682},{"style":1130},[1683],{"type":49,"value":1152},{"type":43,"tag":105,"props":1685,"children":1686},{"style":1130},[1687],{"type":49,"value":1157},{"type":43,"tag":105,"props":1689,"children":1690},{"style":1130},[1691],{"type":49,"value":1175},{"type":43,"tag":105,"props":1693,"children":1694},{"style":1196},[1695],{"type":49,"value":242},{"type":43,"tag":105,"props":1697,"children":1698},{"style":1130},[1699],{"type":49,"value":1152},{"type":43,"tag":105,"props":1701,"children":1702},{"style":1130},[1703],{"type":49,"value":1208},{"type":43,"tag":105,"props":1705,"children":1706},{"style":1130},[1707],{"type":49,"value":1175},{"type":43,"tag":105,"props":1709,"children":1710},{"style":1178},[1711],{"type":49,"value":1250},{"type":43,"tag":105,"props":1713,"children":1714},{"style":1130},[1715],{"type":49,"value":1152},{"type":43,"tag":105,"props":1717,"children":1718},{"style":1130},[1719],{"type":49,"value":1157},{"type":43,"tag":105,"props":1721,"children":1722},{"style":1130},[1723],{"type":49,"value":1175},{"type":43,"tag":105,"props":1725,"children":1726},{"style":1196},[1727],{"type":49,"value":1591},{"type":43,"tag":105,"props":1729,"children":1730},{"style":1130},[1731],{"type":49,"value":1152},{"type":43,"tag":105,"props":1733,"children":1734},{"style":1130},[1735],{"type":49,"value":1208},{"type":43,"tag":105,"props":1737,"children":1738},{"style":1130},[1739],{"type":49,"value":1175},{"type":43,"tag":105,"props":1741,"children":1742},{"style":1178},[1743],{"type":49,"value":1284},{"type":43,"tag":105,"props":1745,"children":1746},{"style":1130},[1747],{"type":49,"value":1152},{"type":43,"tag":105,"props":1749,"children":1750},{"style":1130},[1751],{"type":49,"value":1157},{"type":43,"tag":105,"props":1753,"children":1754},{"style":1130},[1755],{"type":49,"value":1175},{"type":43,"tag":105,"props":1757,"children":1758},{"style":1196},[1759],{"type":49,"value":1760},"\u003Cmode-here>",{"type":43,"tag":105,"props":1762,"children":1763},{"style":1130},[1764],{"type":49,"value":1152},{"type":43,"tag":105,"props":1766,"children":1767},{"style":1130},[1768],{"type":49,"value":1310},{"type":43,"tag":105,"props":1770,"children":1771},{"class":780,"line":825},[1772],{"type":43,"tag":105,"props":1773,"children":1774},{"style":1130},[1775],{"type":49,"value":1318},{"type":43,"tag":105,"props":1777,"children":1778},{"class":780,"line":834},[1779,1783,1787,1791,1795,1799],{"type":43,"tag":105,"props":1780,"children":1781},{"style":1130},[1782],{"type":49,"value":1141},{"type":43,"tag":105,"props":1784,"children":1785},{"style":1144},[1786],{"type":49,"value":1330},{"type":43,"tag":105,"props":1788,"children":1789},{"style":1130},[1790],{"type":49,"value":1152},{"type":43,"tag":105,"props":1792,"children":1793},{"style":1130},[1794],{"type":49,"value":1157},{"type":43,"tag":105,"props":1796,"children":1797},{"style":1341},[1798],{"type":49,"value":1344},{"type":43,"tag":105,"props":1800,"children":1801},{"style":1130},[1802],{"type":49,"value":1349},{"type":43,"tag":105,"props":1804,"children":1805},{"class":780,"line":843},[1806,1810,1814,1818,1822,1826,1830,1834,1838,1842,1846,1850,1854],{"type":43,"tag":105,"props":1807,"children":1808},{"style":1130},[1809],{"type":49,"value":1141},{"type":43,"tag":105,"props":1811,"children":1812},{"style":1144},[1813],{"type":49,"value":1361},{"type":43,"tag":105,"props":1815,"children":1816},{"style":1130},[1817],{"type":49,"value":1152},{"type":43,"tag":105,"props":1819,"children":1820},{"style":1130},[1821],{"type":49,"value":1157},{"type":43,"tag":105,"props":1823,"children":1824},{"style":1130},[1825],{"type":49,"value":1374},{"type":43,"tag":105,"props":1827,"children":1828},{"style":1130},[1829],{"type":49,"value":1175},{"type":43,"tag":105,"props":1831,"children":1832},{"style":1178},[1833],{"type":49,"value":1383},{"type":43,"tag":105,"props":1835,"children":1836},{"style":1130},[1837],{"type":49,"value":1152},{"type":43,"tag":105,"props":1839,"children":1840},{"style":1130},[1841],{"type":49,"value":1157},{"type":43,"tag":105,"props":1843,"children":1844},{"style":1130},[1845],{"type":49,"value":1175},{"type":43,"tag":105,"props":1847,"children":1848},{"style":1196},[1849],{"type":49,"value":1400},{"type":43,"tag":105,"props":1851,"children":1852},{"style":1130},[1853],{"type":49,"value":1152},{"type":43,"tag":105,"props":1855,"children":1856},{"style":1130},[1857],{"type":49,"value":1409},{"type":43,"tag":105,"props":1859,"children":1860},{"class":780,"line":852},[1861,1865,1869,1873,1877],{"type":43,"tag":105,"props":1862,"children":1863},{"style":1130},[1864],{"type":49,"value":1141},{"type":43,"tag":105,"props":1866,"children":1867},{"style":1144},[1868],{"type":49,"value":1421},{"type":43,"tag":105,"props":1870,"children":1871},{"style":1130},[1872],{"type":49,"value":1152},{"type":43,"tag":105,"props":1874,"children":1875},{"style":1130},[1876],{"type":49,"value":1157},{"type":43,"tag":105,"props":1878,"children":1879},{"style":1130},[1880],{"type":49,"value":1434},{"type":43,"tag":105,"props":1882,"children":1883},{"class":780,"line":861},[1884],{"type":43,"tag":105,"props":1885,"children":1886},{"style":1130},[1887],{"type":49,"value":1442},{"type":43,"tag":52,"props":1889,"children":1890},{},[1891,1897],{"type":43,"tag":79,"props":1892,"children":1894},{"className":1893},[],[1895],{"type":49,"value":1896},"randomOrder: true",{"type":49,"value":1898}," matters — recency bias produces a non-representative sample. Pick 5-10 traces to test against.",{"type":43,"tag":52,"props":1900,"children":1901},{},[1902,1907,1908,1914,1916,1922,1924,1929,1931,1937],{"type":43,"tag":63,"props":1903,"children":1904},{},[1905],{"type":49,"value":1906},"Output size warning:",{"type":49,"value":77},{"type":43,"tag":79,"props":1909,"children":1911},{"className":1910},[],[1912],{"type":49,"value":1913},"query-llm-traces-list",{"type":49,"value":1915}," with ",{"type":43,"tag":79,"props":1917,"children":1919},{"className":1918},[],[1920],{"type":49,"value":1921},"limit: 10",{"type":49,"value":1923}," routinely returns 3-6MB of JSON (full input\u002Foutput per generation). This will blow your context window. ",{"type":43,"tag":63,"props":1925,"children":1926},{},[1927],{"type":49,"value":1928},"Immediately delegate the summarization to a subagent",{"type":49,"value":1930}," the moment you see the \"result exceeds maximum allowed tokens\" error — ask the subagent to extract, per trace: the trace id, the first user message (truncated to ~300 chars), the sampled ",{"type":43,"tag":79,"props":1932,"children":1934},{"className":1933},[],[1935],{"type":49,"value":1936},"$current_url",{"type":49,"value":1938},", and a one-sentence description of what the conversation was about. Don't try to read the raw file in-line.",{"type":43,"tag":52,"props":1940,"children":1941},{},[1942,1947,1949,1954],{"type":43,"tag":63,"props":1943,"children":1944},{},[1945],{"type":49,"value":1946},"Watch for topic drift in Pattern B samples.",{"type":49,"value":1948}," The ",{"type":43,"tag":79,"props":1950,"children":1952},{"className":1951},[],[1953],{"type":49,"value":242},{"type":49,"value":1955}," tag reflects the user's mode selection at the time of the turn — but chat state retains the mode even if the user drifts off-topic within the same conversation (e.g. user selected \"error tracking\" mode, then asked an unrelated pricing question three turns later). Your eval prompt's classification step needs to be permissive about topic-drift: PASS should mean \"user is doing something recognizably in-scope for this mode\", FAIL should catch the off-topic drift. If you don't, your feed will include irrelevant PASS entries that happen to carry the mode tag.",{"type":43,"tag":751,"props":1957,"children":1959},{"id":1958},"step-3-draft-the-llm-judge-prompt",[1960],{"type":49,"value":1961},"Step 3 — Draft the LLM-judge prompt",{"type":43,"tag":52,"props":1963,"children":1964},{},[1965,1967,1972],{"type":49,"value":1966},"The prompt has two responsibilities: (a) classify the trace as relevant or not, (b) produce reasoning text that is ",{"type":43,"tag":63,"props":1968,"children":1969},{},[1970],{"type":49,"value":1971},"directly postable to Slack",{"type":49,"value":1973}," (no preamble, no meta-description). The reasoning field becomes the Slack message body.",{"type":43,"tag":52,"props":1975,"children":1976},{},[1977],{"type":49,"value":1978},"Template:",{"type":43,"tag":768,"props":1980,"children":1984},{"className":1981,"code":1983,"language":49,"meta":773},[1982],"language-text","You are analyzing a PostHog [FEATURE NAME] trace to extract its real use case.\nYour reasoning text will be posted directly to a Slack channel as a notification.\nWrite it as a short, ready-to-post message — no preamble, no meta-description.\n\nStep 1 — Classification:\n- PASS = this trace is the [feature kind] you care about\n- FAIL = a different LLM call or a false match\n- N\u002FA = ambiguous from the trace alone\n\nStep 2 — Reasoning (only matters if PASS). Write 2-3 sentences in this exact format:\n\n\"[OPENER] [what they targeted\u002Ffiltered for]. They were\ntrying to [understand X \u002F debug Y \u002F find Z]. The result surfaced [key pattern\nor finding].\"\n\nYour output MUST start with the exact phrase \"[OPENER]\". No other opening is allowed.\n\nRules:\n- No \"This is a [feature]...\" or \"The input contains...\" preamble\n- No JSON, field names, system-prompt references, or meta-description\n- Concrete > generic. \"users hitting error tracking for the first time\" beats \"user behavior\"\n- If you cannot infer one of the three pieces from the trace, write \"(unclear from trace)\" in that slot — do not guess\n",[1985],{"type":43,"tag":79,"props":1986,"children":1987},{"__ignoreMap":773},[1988],{"type":49,"value":1983},{"type":43,"tag":52,"props":1990,"children":1991},{},[1992,2005],{"type":43,"tag":63,"props":1993,"children":1994},{},[1995,1997,2003],{"type":49,"value":1996},"Pick an ",{"type":43,"tag":79,"props":1998,"children":2000},{"className":1999},[],[2001],{"type":49,"value":2002},"[OPENER]",{"type":49,"value":2004}," that matches how users actually interact with the feature.",{"type":49,"value":2006}," The forced opener is load-bearing (it prevents the model from drifting into \"this trace is a...\" meta-description), but the exact verb has to fit the interaction:",{"type":43,"tag":303,"props":2008,"children":2009},{},[2010,2026],{"type":43,"tag":307,"props":2011,"children":2012},{},[2013],{"type":43,"tag":311,"props":2014,"children":2015},{},[2016,2021],{"type":43,"tag":315,"props":2017,"children":2018},{},[2019],{"type":49,"value":2020},"Feature \u002F mode",{"type":43,"tag":315,"props":2022,"children":2023},{},[2024],{"type":49,"value":2025},"OPENER",{"type":43,"tag":326,"props":2027,"children":2028},{},[2029,2046,2063,2080,2096],{"type":43,"tag":311,"props":2030,"children":2031},{},[2032,2037],{"type":43,"tag":333,"props":2033,"children":2034},{},[2035],{"type":49,"value":2036},"Session summary (group \u002F single)",{"type":43,"tag":333,"props":2038,"children":2039},{},[2040],{"type":43,"tag":79,"props":2041,"children":2043},{"className":2042},[],[2044],{"type":49,"value":2045},"A user ran a summary on",{"type":43,"tag":311,"props":2047,"children":2048},{},[2049,2054],{"type":43,"tag":333,"props":2050,"children":2051},{},[2052],{"type":49,"value":2053},"Replay AI search",{"type":43,"tag":333,"props":2055,"children":2056},{},[2057],{"type":43,"tag":79,"props":2058,"children":2060},{"className":2059},[],[2061],{"type":49,"value":2062},"A user searched replays for",{"type":43,"tag":311,"props":2064,"children":2065},{},[2066,2071],{"type":43,"tag":333,"props":2067,"children":2068},{},[2069],{"type":49,"value":2070},"PostHog AI in error tracking mode",{"type":43,"tag":333,"props":2072,"children":2073},{},[2074],{"type":43,"tag":79,"props":2075,"children":2077},{"className":2076},[],[2078],{"type":49,"value":2079},"A user asked PostHog AI about",{"type":43,"tag":311,"props":2081,"children":2082},{},[2083,2088],{"type":43,"tag":333,"props":2084,"children":2085},{},[2086],{"type":49,"value":2087},"PostHog AI in session replay mode",{"type":43,"tag":333,"props":2089,"children":2090},{},[2091],{"type":43,"tag":79,"props":2092,"children":2094},{"className":2093},[],[2095],{"type":49,"value":2079},{"type":43,"tag":311,"props":2097,"children":2098},{},[2099,2104],{"type":43,"tag":333,"props":2100,"children":2101},{},[2102],{"type":49,"value":2103},"PostHog AI in SQL mode",{"type":43,"tag":333,"props":2105,"children":2106},{},[2107],{"type":43,"tag":79,"props":2108,"children":2110},{"className":2109},[],[2111],{"type":49,"value":2112},"A user asked PostHog AI to write SQL for",{"type":43,"tag":52,"props":2114,"children":2115},{},[2116,2117,2122,2124,2129,2131,2136,2138,2144,2146,2152],{"type":49,"value":881},{"type":43,"tag":79,"props":2118,"children":2120},{"className":2119},[],[2121],{"type":49,"value":1080},{"type":49,"value":2123}," is a sub-filter that layers ",{"type":43,"tag":71,"props":2125,"children":2126},{},[2127],{"type":49,"value":2128},"on top of",{"type":49,"value":2130}," an ",{"type":43,"tag":79,"props":2132,"children":2134},{"className":2133},[],[2135],{"type":49,"value":242},{"type":49,"value":2137}," row — it's not its own row. If you want plan-mode-only, filter ",{"type":43,"tag":79,"props":2139,"children":2141},{"className":2140},[],[2142],{"type":49,"value":2143},"agent_mode='\u003Cmode>' AND supermode='plan'",{"type":49,"value":2145}," and pick an opener like ",{"type":43,"tag":79,"props":2147,"children":2149},{"className":2148},[],[2150],{"type":49,"value":2151},"\"A user asked PostHog AI to plan\"",{"type":49,"value":2153},".",{"type":43,"tag":52,"props":2155,"children":2156},{},[2157,2159,2165],{"type":49,"value":2158},"If you force ",{"type":43,"tag":79,"props":2160,"children":2162},{"className":2161},[],[2163],{"type":49,"value":2164},"\"A user ran\"",{"type":49,"value":2166}," on a chat-based feature, the model will produce awkward contortions (\"A user ran a question about...\") that read wrong in Slack. The forced-opener pattern is the mechanism — the specific phrase is per-feature.",{"type":43,"tag":52,"props":2168,"children":2169},{},[2170],{"type":49,"value":2171},"The negative example list (\"No 'This is a...' preamble\", etc.) is load-bearing regardless of opener. Don't remove it.",{"type":43,"tag":751,"props":2173,"children":2175},{"id":2174},"step-4-create-the-eval-disabled-test-iterate",[2176],{"type":49,"value":2177},"Step 4 — Create the eval (disabled), test, iterate",{"type":43,"tag":52,"props":2179,"children":2180},{},[2181,2183,2189],{"type":49,"value":2182},"Create with ",{"type":43,"tag":79,"props":2184,"children":2186},{"className":2185},[],[2187],{"type":49,"value":2188},"enabled: false",{"type":49,"value":2190}," so it doesn't immediately fan out to all traces.",{"type":43,"tag":52,"props":2192,"children":2193},{},[2194,2205],{"type":43,"tag":63,"props":2195,"children":2196},{},[2197,2198,2203],{"type":49,"value":513},{"type":43,"tag":79,"props":2199,"children":2201},{"className":2200},[],[2202],{"type":49,"value":616},{"type":49,"value":2204}," is exposed",{"type":49,"value":2206},", use this payload:",{"type":43,"tag":768,"props":2208,"children":2210},{"className":1109,"code":2209,"language":1111,"meta":773,"style":773},"posthog:llma-evaluation-create\n{\n  \"name\": \"[feature] use case feed\",\n  \"description\": \"Extracts canonical use cases for [feature] for the #team-[area]-usage Slack feed\",\n  \"evaluation_type\": \"llm_judge\",\n  \"evaluation_config\": {\n    \"prompt\": \"\u003Cfull prompt from step 3>\"\n  },\n  \"output_type\": \"boolean\",\n  \"output_config\": { \"allows_na\": true },\n  \"model_configuration\": {\n    \"provider\": \"\u003Cprovider>\",\n    \"model\": \"\u003Cmodel>\"\n  },\n  \"enabled\": false,\n  \"conditions\": {\n    \"filters\": [\n      \u002F\u002F Pattern A — feature-native trace_id prefix:\n      { \"key\": \"$ai_trace_id\", \"operator\": \"icontains\", \"value\": \"\u003Cyour-prefix>\" }\n\n      \u002F\u002F Pattern B — PostHog AI agent mode (use these INSTEAD of the trace_id filter):\n      \u002F\u002F { \"key\": \"ai_product\", \"operator\": \"exact\", \"value\": \"posthog_ai\" },\n      \u002F\u002F { \"key\": \"agent_mode\", \"operator\": \"exact\", \"value\": \"\u003Cmode>\" }\n    ]\n  }\n}\n",[2211],{"type":43,"tag":79,"props":2212,"children":2213},{"__ignoreMap":773},[2214,2222,2229,2266,2303,2339,2364,2399,2407,2444,2494,2518,2555,2589,2597,2623,2648,2673,2683,2789,2799,2808,2817,2826,2835,2844],{"type":43,"tag":105,"props":2215,"children":2216},{"class":780,"line":781},[2217],{"type":43,"tag":105,"props":2218,"children":2219},{"style":1121},[2220],{"type":49,"value":2221},"posthog:llma-evaluation-create\n",{"type":43,"tag":105,"props":2223,"children":2224},{"class":780,"line":790},[2225],{"type":43,"tag":105,"props":2226,"children":2227},{"style":1130},[2228],{"type":49,"value":1133},{"type":43,"tag":105,"props":2230,"children":2231},{"class":780,"line":799},[2232,2236,2241,2245,2249,2253,2258,2262],{"type":43,"tag":105,"props":2233,"children":2234},{"style":1130},[2235],{"type":49,"value":1141},{"type":43,"tag":105,"props":2237,"children":2238},{"style":1144},[2239],{"type":49,"value":2240},"name",{"type":43,"tag":105,"props":2242,"children":2243},{"style":1130},[2244],{"type":49,"value":1152},{"type":43,"tag":105,"props":2246,"children":2247},{"style":1130},[2248],{"type":49,"value":1157},{"type":43,"tag":105,"props":2250,"children":2251},{"style":1130},[2252],{"type":49,"value":1175},{"type":43,"tag":105,"props":2254,"children":2255},{"style":1196},[2256],{"type":49,"value":2257},"[feature] use case feed",{"type":43,"tag":105,"props":2259,"children":2260},{"style":1130},[2261],{"type":49,"value":1152},{"type":43,"tag":105,"props":2263,"children":2264},{"style":1130},[2265],{"type":49,"value":1349},{"type":43,"tag":105,"props":2267,"children":2268},{"class":780,"line":30},[2269,2273,2278,2282,2286,2290,2295,2299],{"type":43,"tag":105,"props":2270,"children":2271},{"style":1130},[2272],{"type":49,"value":1141},{"type":43,"tag":105,"props":2274,"children":2275},{"style":1144},[2276],{"type":49,"value":2277},"description",{"type":43,"tag":105,"props":2279,"children":2280},{"style":1130},[2281],{"type":49,"value":1152},{"type":43,"tag":105,"props":2283,"children":2284},{"style":1130},[2285],{"type":49,"value":1157},{"type":43,"tag":105,"props":2287,"children":2288},{"style":1130},[2289],{"type":49,"value":1175},{"type":43,"tag":105,"props":2291,"children":2292},{"style":1196},[2293],{"type":49,"value":2294},"Extracts canonical use cases for [feature] for the #team-[area]-usage Slack feed",{"type":43,"tag":105,"props":2296,"children":2297},{"style":1130},[2298],{"type":49,"value":1152},{"type":43,"tag":105,"props":2300,"children":2301},{"style":1130},[2302],{"type":49,"value":1349},{"type":43,"tag":105,"props":2304,"children":2305},{"class":780,"line":816},[2306,2310,2315,2319,2323,2327,2331,2335],{"type":43,"tag":105,"props":2307,"children":2308},{"style":1130},[2309],{"type":49,"value":1141},{"type":43,"tag":105,"props":2311,"children":2312},{"style":1144},[2313],{"type":49,"value":2314},"evaluation_type",{"type":43,"tag":105,"props":2316,"children":2317},{"style":1130},[2318],{"type":49,"value":1152},{"type":43,"tag":105,"props":2320,"children":2321},{"style":1130},[2322],{"type":49,"value":1157},{"type":43,"tag":105,"props":2324,"children":2325},{"style":1130},[2326],{"type":49,"value":1175},{"type":43,"tag":105,"props":2328,"children":2329},{"style":1196},[2330],{"type":49,"value":506},{"type":43,"tag":105,"props":2332,"children":2333},{"style":1130},[2334],{"type":49,"value":1152},{"type":43,"tag":105,"props":2336,"children":2337},{"style":1130},[2338],{"type":49,"value":1349},{"type":43,"tag":105,"props":2340,"children":2341},{"class":780,"line":825},[2342,2346,2351,2355,2359],{"type":43,"tag":105,"props":2343,"children":2344},{"style":1130},[2345],{"type":49,"value":1141},{"type":43,"tag":105,"props":2347,"children":2348},{"style":1144},[2349],{"type":49,"value":2350},"evaluation_config",{"type":43,"tag":105,"props":2352,"children":2353},{"style":1130},[2354],{"type":49,"value":1152},{"type":43,"tag":105,"props":2356,"children":2357},{"style":1130},[2358],{"type":49,"value":1157},{"type":43,"tag":105,"props":2360,"children":2361},{"style":1130},[2362],{"type":49,"value":2363}," {\n",{"type":43,"tag":105,"props":2365,"children":2366},{"class":780,"line":834},[2367,2372,2377,2381,2385,2389,2394],{"type":43,"tag":105,"props":2368,"children":2369},{"style":1130},[2370],{"type":49,"value":2371},"    \"",{"type":43,"tag":105,"props":2373,"children":2374},{"style":1178},[2375],{"type":49,"value":2376},"prompt",{"type":43,"tag":105,"props":2378,"children":2379},{"style":1130},[2380],{"type":49,"value":1152},{"type":43,"tag":105,"props":2382,"children":2383},{"style":1130},[2384],{"type":49,"value":1157},{"type":43,"tag":105,"props":2386,"children":2387},{"style":1130},[2388],{"type":49,"value":1175},{"type":43,"tag":105,"props":2390,"children":2391},{"style":1196},[2392],{"type":49,"value":2393},"\u003Cfull prompt from step 3>",{"type":43,"tag":105,"props":2395,"children":2396},{"style":1130},[2397],{"type":49,"value":2398},"\"\n",{"type":43,"tag":105,"props":2400,"children":2401},{"class":780,"line":843},[2402],{"type":43,"tag":105,"props":2403,"children":2404},{"style":1130},[2405],{"type":49,"value":2406},"  },\n",{"type":43,"tag":105,"props":2408,"children":2409},{"class":780,"line":852},[2410,2414,2419,2423,2427,2431,2436,2440],{"type":43,"tag":105,"props":2411,"children":2412},{"style":1130},[2413],{"type":49,"value":1141},{"type":43,"tag":105,"props":2415,"children":2416},{"style":1144},[2417],{"type":49,"value":2418},"output_type",{"type":43,"tag":105,"props":2420,"children":2421},{"style":1130},[2422],{"type":49,"value":1152},{"type":43,"tag":105,"props":2424,"children":2425},{"style":1130},[2426],{"type":49,"value":1157},{"type":43,"tag":105,"props":2428,"children":2429},{"style":1130},[2430],{"type":49,"value":1175},{"type":43,"tag":105,"props":2432,"children":2433},{"style":1196},[2434],{"type":49,"value":2435},"boolean",{"type":43,"tag":105,"props":2437,"children":2438},{"style":1130},[2439],{"type":49,"value":1152},{"type":43,"tag":105,"props":2441,"children":2442},{"style":1130},[2443],{"type":49,"value":1349},{"type":43,"tag":105,"props":2445,"children":2446},{"class":780,"line":861},[2447,2451,2456,2460,2464,2468,2472,2477,2481,2485,2490],{"type":43,"tag":105,"props":2448,"children":2449},{"style":1130},[2450],{"type":49,"value":1141},{"type":43,"tag":105,"props":2452,"children":2453},{"style":1144},[2454],{"type":49,"value":2455},"output_config",{"type":43,"tag":105,"props":2457,"children":2458},{"style":1130},[2459],{"type":49,"value":1152},{"type":43,"tag":105,"props":2461,"children":2462},{"style":1130},[2463],{"type":49,"value":1157},{"type":43,"tag":105,"props":2465,"children":2466},{"style":1130},[2467],{"type":49,"value":1374},{"type":43,"tag":105,"props":2469,"children":2470},{"style":1130},[2471],{"type":49,"value":1175},{"type":43,"tag":105,"props":2473,"children":2474},{"style":1178},[2475],{"type":49,"value":2476},"allows_na",{"type":43,"tag":105,"props":2478,"children":2479},{"style":1130},[2480],{"type":49,"value":1152},{"type":43,"tag":105,"props":2482,"children":2483},{"style":1130},[2484],{"type":49,"value":1157},{"type":43,"tag":105,"props":2486,"children":2487},{"style":1130},[2488],{"type":49,"value":2489}," true",{"type":43,"tag":105,"props":2491,"children":2492},{"style":1130},[2493],{"type":49,"value":1409},{"type":43,"tag":105,"props":2495,"children":2496},{"class":780,"line":870},[2497,2501,2506,2510,2514],{"type":43,"tag":105,"props":2498,"children":2499},{"style":1130},[2500],{"type":49,"value":1141},{"type":43,"tag":105,"props":2502,"children":2503},{"style":1144},[2504],{"type":49,"value":2505},"model_configuration",{"type":43,"tag":105,"props":2507,"children":2508},{"style":1130},[2509],{"type":49,"value":1152},{"type":43,"tag":105,"props":2511,"children":2512},{"style":1130},[2513],{"type":49,"value":1157},{"type":43,"tag":105,"props":2515,"children":2516},{"style":1130},[2517],{"type":49,"value":2363},{"type":43,"tag":105,"props":2519,"children":2520},{"class":780,"line":1000},[2521,2525,2530,2534,2538,2542,2547,2551],{"type":43,"tag":105,"props":2522,"children":2523},{"style":1130},[2524],{"type":49,"value":2371},{"type":43,"tag":105,"props":2526,"children":2527},{"style":1178},[2528],{"type":49,"value":2529},"provider",{"type":43,"tag":105,"props":2531,"children":2532},{"style":1130},[2533],{"type":49,"value":1152},{"type":43,"tag":105,"props":2535,"children":2536},{"style":1130},[2537],{"type":49,"value":1157},{"type":43,"tag":105,"props":2539,"children":2540},{"style":1130},[2541],{"type":49,"value":1175},{"type":43,"tag":105,"props":2543,"children":2544},{"style":1196},[2545],{"type":49,"value":2546},"\u003Cprovider>",{"type":43,"tag":105,"props":2548,"children":2549},{"style":1130},[2550],{"type":49,"value":1152},{"type":43,"tag":105,"props":2552,"children":2553},{"style":1130},[2554],{"type":49,"value":1349},{"type":43,"tag":105,"props":2556,"children":2558},{"class":780,"line":2557},13,[2559,2563,2568,2572,2576,2580,2585],{"type":43,"tag":105,"props":2560,"children":2561},{"style":1130},[2562],{"type":49,"value":2371},{"type":43,"tag":105,"props":2564,"children":2565},{"style":1178},[2566],{"type":49,"value":2567},"model",{"type":43,"tag":105,"props":2569,"children":2570},{"style":1130},[2571],{"type":49,"value":1152},{"type":43,"tag":105,"props":2573,"children":2574},{"style":1130},[2575],{"type":49,"value":1157},{"type":43,"tag":105,"props":2577,"children":2578},{"style":1130},[2579],{"type":49,"value":1175},{"type":43,"tag":105,"props":2581,"children":2582},{"style":1196},[2583],{"type":49,"value":2584},"\u003Cmodel>",{"type":43,"tag":105,"props":2586,"children":2587},{"style":1130},[2588],{"type":49,"value":2398},{"type":43,"tag":105,"props":2590,"children":2592},{"class":780,"line":2591},14,[2593],{"type":43,"tag":105,"props":2594,"children":2595},{"style":1130},[2596],{"type":49,"value":2406},{"type":43,"tag":105,"props":2598,"children":2600},{"class":780,"line":2599},15,[2601,2605,2610,2614,2618],{"type":43,"tag":105,"props":2602,"children":2603},{"style":1130},[2604],{"type":49,"value":1141},{"type":43,"tag":105,"props":2606,"children":2607},{"style":1144},[2608],{"type":49,"value":2609},"enabled",{"type":43,"tag":105,"props":2611,"children":2612},{"style":1130},[2613],{"type":49,"value":1152},{"type":43,"tag":105,"props":2615,"children":2616},{"style":1130},[2617],{"type":49,"value":1157},{"type":43,"tag":105,"props":2619,"children":2620},{"style":1130},[2621],{"type":49,"value":2622}," false,\n",{"type":43,"tag":105,"props":2624,"children":2626},{"class":780,"line":2625},16,[2627,2631,2636,2640,2644],{"type":43,"tag":105,"props":2628,"children":2629},{"style":1130},[2630],{"type":49,"value":1141},{"type":43,"tag":105,"props":2632,"children":2633},{"style":1144},[2634],{"type":49,"value":2635},"conditions",{"type":43,"tag":105,"props":2637,"children":2638},{"style":1130},[2639],{"type":49,"value":1152},{"type":43,"tag":105,"props":2641,"children":2642},{"style":1130},[2643],{"type":49,"value":1157},{"type":43,"tag":105,"props":2645,"children":2646},{"style":1130},[2647],{"type":49,"value":2363},{"type":43,"tag":105,"props":2649,"children":2651},{"class":780,"line":2650},17,[2652,2656,2661,2665,2669],{"type":43,"tag":105,"props":2653,"children":2654},{"style":1130},[2655],{"type":49,"value":2371},{"type":43,"tag":105,"props":2657,"children":2658},{"style":1178},[2659],{"type":49,"value":2660},"filters",{"type":43,"tag":105,"props":2662,"children":2663},{"style":1130},[2664],{"type":49,"value":1152},{"type":43,"tag":105,"props":2666,"children":2667},{"style":1130},[2668],{"type":49,"value":1157},{"type":43,"tag":105,"props":2670,"children":2671},{"style":1130},[2672],{"type":49,"value":1162},{"type":43,"tag":105,"props":2674,"children":2676},{"class":780,"line":2675},18,[2677],{"type":43,"tag":105,"props":2678,"children":2680},{"style":2679},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[2681],{"type":49,"value":2682},"      \u002F\u002F Pattern A — feature-native trace_id prefix:\n",{"type":43,"tag":105,"props":2684,"children":2686},{"class":780,"line":2685},19,[2687,2692,2696,2700,2704,2708,2712,2716,2720,2724,2728,2732,2736,2740,2744,2748,2752,2756,2760,2764,2768,2772,2776,2781,2785],{"type":43,"tag":105,"props":2688,"children":2689},{"style":1130},[2690],{"type":49,"value":2691},"      {",{"type":43,"tag":105,"props":2693,"children":2694},{"style":1130},[2695],{"type":49,"value":1175},{"type":43,"tag":105,"props":2697,"children":2698},{"style":1341},[2699],{"type":49,"value":1217},{"type":43,"tag":105,"props":2701,"children":2702},{"style":1130},[2703],{"type":49,"value":1152},{"type":43,"tag":105,"props":2705,"children":2706},{"style":1130},[2707],{"type":49,"value":1157},{"type":43,"tag":105,"props":2709,"children":2710},{"style":1130},[2711],{"type":49,"value":1175},{"type":43,"tag":105,"props":2713,"children":2714},{"style":1196},[2715],{"type":49,"value":193},{"type":43,"tag":105,"props":2717,"children":2718},{"style":1130},[2719],{"type":49,"value":1152},{"type":43,"tag":105,"props":2721,"children":2722},{"style":1130},[2723],{"type":49,"value":1208},{"type":43,"tag":105,"props":2725,"children":2726},{"style":1130},[2727],{"type":49,"value":1175},{"type":43,"tag":105,"props":2729,"children":2730},{"style":1341},[2731],{"type":49,"value":1250},{"type":43,"tag":105,"props":2733,"children":2734},{"style":1130},[2735],{"type":49,"value":1152},{"type":43,"tag":105,"props":2737,"children":2738},{"style":1130},[2739],{"type":49,"value":1157},{"type":43,"tag":105,"props":2741,"children":2742},{"style":1130},[2743],{"type":49,"value":1175},{"type":43,"tag":105,"props":2745,"children":2746},{"style":1196},[2747],{"type":49,"value":1267},{"type":43,"tag":105,"props":2749,"children":2750},{"style":1130},[2751],{"type":49,"value":1152},{"type":43,"tag":105,"props":2753,"children":2754},{"style":1130},[2755],{"type":49,"value":1208},{"type":43,"tag":105,"props":2757,"children":2758},{"style":1130},[2759],{"type":49,"value":1175},{"type":43,"tag":105,"props":2761,"children":2762},{"style":1341},[2763],{"type":49,"value":1284},{"type":43,"tag":105,"props":2765,"children":2766},{"style":1130},[2767],{"type":49,"value":1152},{"type":43,"tag":105,"props":2769,"children":2770},{"style":1130},[2771],{"type":49,"value":1157},{"type":43,"tag":105,"props":2773,"children":2774},{"style":1130},[2775],{"type":49,"value":1175},{"type":43,"tag":105,"props":2777,"children":2778},{"style":1196},[2779],{"type":49,"value":2780},"\u003Cyour-prefix>",{"type":43,"tag":105,"props":2782,"children":2783},{"style":1130},[2784],{"type":49,"value":1152},{"type":43,"tag":105,"props":2786,"children":2787},{"style":1130},[2788],{"type":49,"value":1310},{"type":43,"tag":105,"props":2790,"children":2792},{"class":780,"line":2791},20,[2793],{"type":43,"tag":105,"props":2794,"children":2796},{"emptyLinePlaceholder":2795},true,[2797],{"type":49,"value":2798},"\n",{"type":43,"tag":105,"props":2800,"children":2802},{"class":780,"line":2801},21,[2803],{"type":43,"tag":105,"props":2804,"children":2805},{"style":2679},[2806],{"type":49,"value":2807},"      \u002F\u002F Pattern B — PostHog AI agent mode (use these INSTEAD of the trace_id filter):\n",{"type":43,"tag":105,"props":2809,"children":2811},{"class":780,"line":2810},22,[2812],{"type":43,"tag":105,"props":2813,"children":2814},{"style":2679},[2815],{"type":49,"value":2816},"      \u002F\u002F { \"key\": \"ai_product\", \"operator\": \"exact\", \"value\": \"posthog_ai\" },\n",{"type":43,"tag":105,"props":2818,"children":2820},{"class":780,"line":2819},23,[2821],{"type":43,"tag":105,"props":2822,"children":2823},{"style":2679},[2824],{"type":49,"value":2825},"      \u002F\u002F { \"key\": \"agent_mode\", \"operator\": \"exact\", \"value\": \"\u003Cmode>\" }\n",{"type":43,"tag":105,"props":2827,"children":2829},{"class":780,"line":2828},24,[2830],{"type":43,"tag":105,"props":2831,"children":2832},{"style":1130},[2833],{"type":49,"value":2834},"    ]\n",{"type":43,"tag":105,"props":2836,"children":2838},{"class":780,"line":2837},25,[2839],{"type":43,"tag":105,"props":2840,"children":2841},{"style":1130},[2842],{"type":49,"value":2843},"  }\n",{"type":43,"tag":105,"props":2845,"children":2847},{"class":780,"line":2846},26,[2848],{"type":43,"tag":105,"props":2849,"children":2850},{"style":1130},[2851],{"type":49,"value":1442},{"type":43,"tag":52,"props":2853,"children":2854},{},[2855],{"type":49,"value":2856},"Leave model choice to the user — LLM-judge cost scales linearly with event volume, and cheap-vs-capable is a real tradeoff they should make based on their own spend tolerance and signal-quality requirements. Don't pick for them.",{"type":43,"tag":52,"props":2858,"children":2859},{},[2860,2865,2867,2873,2875,2881],{"type":43,"tag":63,"props":2861,"children":2862},{},[2863],{"type":49,"value":2864},"UI fallback",{"type":49,"value":2866}," (when ",{"type":43,"tag":79,"props":2868,"children":2870},{"className":2869},[],[2871],{"type":49,"value":2872},"llma-evaluation-create",{"type":49,"value":2874}," isn't exposed): AI observability → Evaluations → New evaluation. Type = ",{"type":43,"tag":79,"props":2876,"children":2878},{"className":2877},[],[2879],{"type":49,"value":2880},"LLM judge",{"type":49,"value":2882},", output = boolean + allow N\u002FA, filters as above, enabled = off. Paste the prompt from step 3.",{"type":43,"tag":52,"props":2884,"children":2885},{},[2886],{"type":49,"value":2887},"Then dry-run against your sample traces.",{"type":43,"tag":52,"props":2889,"children":2890},{},[2891],{"type":43,"tag":63,"props":2892,"children":2893},{},[2894,2895,2900],{"type":49,"value":513},{"type":43,"tag":79,"props":2896,"children":2898},{"className":2897},[],[2899],{"type":49,"value":640},{"type":49,"value":2901}," is exposed:",{"type":43,"tag":768,"props":2903,"children":2905},{"className":1109,"code":2904,"language":1111,"meta":773,"style":773},"posthog:llma-evaluation-run\n{\n  \"evaluationId\": \"\u003Cuuid from create>\",\n  \"target_event_id\": \"\u003Ca $ai_generation event id from step 2>\",\n  \"timestamp\": \"\u003CISO timestamp of that event>\"\n}\n",[2906],{"type":43,"tag":79,"props":2907,"children":2908},{"__ignoreMap":773},[2909,2917,2924,2961,2998,3031],{"type":43,"tag":105,"props":2910,"children":2911},{"class":780,"line":781},[2912],{"type":43,"tag":105,"props":2913,"children":2914},{"style":1121},[2915],{"type":49,"value":2916},"posthog:llma-evaluation-run\n",{"type":43,"tag":105,"props":2918,"children":2919},{"class":780,"line":790},[2920],{"type":43,"tag":105,"props":2921,"children":2922},{"style":1130},[2923],{"type":49,"value":1133},{"type":43,"tag":105,"props":2925,"children":2926},{"class":780,"line":799},[2927,2931,2936,2940,2944,2948,2953,2957],{"type":43,"tag":105,"props":2928,"children":2929},{"style":1130},[2930],{"type":49,"value":1141},{"type":43,"tag":105,"props":2932,"children":2933},{"style":1144},[2934],{"type":49,"value":2935},"evaluationId",{"type":43,"tag":105,"props":2937,"children":2938},{"style":1130},[2939],{"type":49,"value":1152},{"type":43,"tag":105,"props":2941,"children":2942},{"style":1130},[2943],{"type":49,"value":1157},{"type":43,"tag":105,"props":2945,"children":2946},{"style":1130},[2947],{"type":49,"value":1175},{"type":43,"tag":105,"props":2949,"children":2950},{"style":1196},[2951],{"type":49,"value":2952},"\u003Cuuid from create>",{"type":43,"tag":105,"props":2954,"children":2955},{"style":1130},[2956],{"type":49,"value":1152},{"type":43,"tag":105,"props":2958,"children":2959},{"style":1130},[2960],{"type":49,"value":1349},{"type":43,"tag":105,"props":2962,"children":2963},{"class":780,"line":30},[2964,2968,2973,2977,2981,2985,2990,2994],{"type":43,"tag":105,"props":2965,"children":2966},{"style":1130},[2967],{"type":49,"value":1141},{"type":43,"tag":105,"props":2969,"children":2970},{"style":1144},[2971],{"type":49,"value":2972},"target_event_id",{"type":43,"tag":105,"props":2974,"children":2975},{"style":1130},[2976],{"type":49,"value":1152},{"type":43,"tag":105,"props":2978,"children":2979},{"style":1130},[2980],{"type":49,"value":1157},{"type":43,"tag":105,"props":2982,"children":2983},{"style":1130},[2984],{"type":49,"value":1175},{"type":43,"tag":105,"props":2986,"children":2987},{"style":1196},[2988],{"type":49,"value":2989},"\u003Ca $ai_generation event id from step 2>",{"type":43,"tag":105,"props":2991,"children":2992},{"style":1130},[2993],{"type":49,"value":1152},{"type":43,"tag":105,"props":2995,"children":2996},{"style":1130},[2997],{"type":49,"value":1349},{"type":43,"tag":105,"props":2999,"children":3000},{"class":780,"line":816},[3001,3005,3010,3014,3018,3022,3027],{"type":43,"tag":105,"props":3002,"children":3003},{"style":1130},[3004],{"type":49,"value":1141},{"type":43,"tag":105,"props":3006,"children":3007},{"style":1144},[3008],{"type":49,"value":3009},"timestamp",{"type":43,"tag":105,"props":3011,"children":3012},{"style":1130},[3013],{"type":49,"value":1152},{"type":43,"tag":105,"props":3015,"children":3016},{"style":1130},[3017],{"type":49,"value":1157},{"type":43,"tag":105,"props":3019,"children":3020},{"style":1130},[3021],{"type":49,"value":1175},{"type":43,"tag":105,"props":3023,"children":3024},{"style":1196},[3025],{"type":49,"value":3026},"\u003CISO timestamp of that event>",{"type":43,"tag":105,"props":3028,"children":3029},{"style":1130},[3030],{"type":49,"value":2398},{"type":43,"tag":105,"props":3032,"children":3033},{"class":780,"line":825},[3034],{"type":43,"tag":105,"props":3035,"children":3036},{"style":1130},[3037],{"type":49,"value":1442},{"type":43,"tag":52,"props":3039,"children":3040},{},[3041,3046],{"type":43,"tag":63,"props":3042,"children":3043},{},[3044],{"type":49,"value":3045},"UI fallback:",{"type":49,"value":3047}," on the eval detail page, use the \"Run on event\" button with the trace sample's event id.",{"type":43,"tag":52,"props":3049,"children":3050},{},[3051,3053,3059,3061,3067],{"type":49,"value":3052},"Look at the returned ",{"type":43,"tag":79,"props":3054,"children":3056},{"className":3055},[],[3057],{"type":49,"value":3058},"$ai_evaluation_reasoning",{"type":49,"value":3060},". If it preambles, drifts, or describes the input, fix the prompt (via ",{"type":43,"tag":79,"props":3062,"children":3064},{"className":3063},[],[3065],{"type":49,"value":3066},"llma-evaluation-update",{"type":49,"value":3068}," or by editing in the UI) and re-run. Iterate on 3-5 traces before enabling.",{"type":43,"tag":52,"props":3070,"children":3071},{},[3072],{"type":49,"value":3073},"Common failure modes during iteration:",{"type":43,"tag":303,"props":3075,"children":3076},{},[3077,3093],{"type":43,"tag":307,"props":3078,"children":3079},{},[3080],{"type":43,"tag":311,"props":3081,"children":3082},{},[3083,3088],{"type":43,"tag":315,"props":3084,"children":3085},{},[3086],{"type":49,"value":3087},"Symptom",{"type":43,"tag":315,"props":3089,"children":3090},{},[3091],{"type":49,"value":3092},"Fix",{"type":43,"tag":326,"props":3094,"children":3095},{},[3096,3109,3122,3135],{"type":43,"tag":311,"props":3097,"children":3098},{},[3099,3104],{"type":43,"tag":333,"props":3100,"children":3101},{},[3102],{"type":49,"value":3103},"Reasoning starts with \"This is a...\"",{"type":43,"tag":333,"props":3105,"children":3106},{},[3107],{"type":49,"value":3108},"Strengthen the forced opener instruction; add a counter-example",{"type":43,"tag":311,"props":3110,"children":3111},{},[3112,3117],{"type":43,"tag":333,"props":3113,"children":3114},{},[3115],{"type":49,"value":3116},"Reasoning is generic (\"user behavior\", \"various patterns\")",{"type":43,"tag":333,"props":3118,"children":3119},{},[3120],{"type":49,"value":3121},"Add positive examples of concrete phrasing in the prompt",{"type":43,"tag":311,"props":3123,"children":3124},{},[3125,3130],{"type":43,"tag":333,"props":3126,"children":3127},{},[3128],{"type":49,"value":3129},"Model classifies everything as PASS",{"type":43,"tag":333,"props":3131,"children":3132},{},[3133],{"type":49,"value":3134},"Tighten the FAIL definition; add an example of what a non-match looks like",{"type":43,"tag":311,"props":3136,"children":3137},{},[3138,3143],{"type":43,"tag":333,"props":3139,"children":3140},{},[3141],{"type":49,"value":3142},"Reasoning is too long for Slack",{"type":43,"tag":333,"props":3144,"children":3145},{},[3146],{"type":49,"value":3147},"Add a hard sentence cap (\"MAX 3 sentences, hard limit\")",{"type":43,"tag":751,"props":3149,"children":3151},{"id":3150},"step-5-enable-the-eval",[3152],{"type":49,"value":3153},"Step 5 — Enable the eval",{"type":43,"tag":52,"props":3155,"children":3156},{},[3157],{"type":49,"value":3158},"Once 3-5 sample runs produce clean Slack-ready output.",{"type":43,"tag":52,"props":3160,"children":3161},{},[3162],{"type":43,"tag":63,"props":3163,"children":3164},{},[3165,3166,3171],{"type":49,"value":513},{"type":43,"tag":79,"props":3167,"children":3169},{"className":3168},[],[3170],{"type":49,"value":662},{"type":49,"value":2901},{"type":43,"tag":768,"props":3173,"children":3175},{"className":1109,"code":3174,"language":1111,"meta":773,"style":773},"posthog:llma-evaluation-update\n{\n  \"evaluationId\": \"\u003Cuuid>\",\n  \"enabled\": true\n}\n",[3176],{"type":43,"tag":79,"props":3177,"children":3178},{"__ignoreMap":773},[3179,3187,3194,3230,3253],{"type":43,"tag":105,"props":3180,"children":3181},{"class":780,"line":781},[3182],{"type":43,"tag":105,"props":3183,"children":3184},{"style":1121},[3185],{"type":49,"value":3186},"posthog:llma-evaluation-update\n",{"type":43,"tag":105,"props":3188,"children":3189},{"class":780,"line":790},[3190],{"type":43,"tag":105,"props":3191,"children":3192},{"style":1130},[3193],{"type":49,"value":1133},{"type":43,"tag":105,"props":3195,"children":3196},{"class":780,"line":799},[3197,3201,3205,3209,3213,3217,3222,3226],{"type":43,"tag":105,"props":3198,"children":3199},{"style":1130},[3200],{"type":49,"value":1141},{"type":43,"tag":105,"props":3202,"children":3203},{"style":1144},[3204],{"type":49,"value":2935},{"type":43,"tag":105,"props":3206,"children":3207},{"style":1130},[3208],{"type":49,"value":1152},{"type":43,"tag":105,"props":3210,"children":3211},{"style":1130},[3212],{"type":49,"value":1157},{"type":43,"tag":105,"props":3214,"children":3215},{"style":1130},[3216],{"type":49,"value":1175},{"type":43,"tag":105,"props":3218,"children":3219},{"style":1196},[3220],{"type":49,"value":3221},"\u003Cuuid>",{"type":43,"tag":105,"props":3223,"children":3224},{"style":1130},[3225],{"type":49,"value":1152},{"type":43,"tag":105,"props":3227,"children":3228},{"style":1130},[3229],{"type":49,"value":1349},{"type":43,"tag":105,"props":3231,"children":3232},{"class":780,"line":30},[3233,3237,3241,3245,3249],{"type":43,"tag":105,"props":3234,"children":3235},{"style":1130},[3236],{"type":49,"value":1141},{"type":43,"tag":105,"props":3238,"children":3239},{"style":1144},[3240],{"type":49,"value":2609},{"type":43,"tag":105,"props":3242,"children":3243},{"style":1130},[3244],{"type":49,"value":1152},{"type":43,"tag":105,"props":3246,"children":3247},{"style":1130},[3248],{"type":49,"value":1157},{"type":43,"tag":105,"props":3250,"children":3251},{"style":1130},[3252],{"type":49,"value":1434},{"type":43,"tag":105,"props":3254,"children":3255},{"class":780,"line":816},[3256],{"type":43,"tag":105,"props":3257,"children":3258},{"style":1130},[3259],{"type":49,"value":1442},{"type":43,"tag":52,"props":3261,"children":3262},{},[3263,3267],{"type":43,"tag":63,"props":3264,"children":3265},{},[3266],{"type":49,"value":3045},{"type":49,"value":3268}," AI observability → Evaluations → open the eval → toggle enabled.",{"type":43,"tag":52,"props":3270,"children":3271},{},[3272,3274,3279],{"type":49,"value":3273},"The eval will now run on every new matching ",{"type":43,"tag":79,"props":3275,"children":3277},{"className":3276},[],[3278],{"type":49,"value":258},{"type":49,"value":3280}," event.",{"type":43,"tag":751,"props":3282,"children":3284},{"id":3283},"step-6-build-the-workflow-ui-only",[3285],{"type":49,"value":3286},"Step 6 — Build the workflow (UI only)",{"type":43,"tag":52,"props":3288,"children":3289},{},[3290,3292,3297,3298,3303],{"type":49,"value":3291},"Workflow setup is not MCP-accessible for writes (",{"type":43,"tag":79,"props":3293,"children":3295},{"className":3294},[],[3296],{"type":49,"value":706},{"type":49,"value":708},{"type":43,"tag":79,"props":3299,"children":3301},{"className":3300},[],[3302],{"type":49,"value":714},{"type":49,"value":3304}," are read-only). The steps below are a UI walkthrough.",{"type":43,"tag":52,"props":3306,"children":3307},{},[3308,3313,3315,3321],{"type":43,"tag":63,"props":3309,"children":3310},{},[3311],{"type":49,"value":3312},"Prereq:",{"type":49,"value":3314}," before you start, invite the PostHog Slack bot to your target channel (",{"type":43,"tag":79,"props":3316,"children":3318},{"className":3317},[],[3319],{"type":49,"value":3320},"\u002Finvite @PostHog",{"type":49,"value":3322}," in the Slack channel). Without this, the Slack dispatch step will fail with an opaque permission error at send time, not at save time — easy to miss.",{"type":43,"tag":3324,"props":3325,"children":3327},"h4",{"id":3326},"_61-create-the-workflow",[3328],{"type":49,"value":3329},"6.1 Create the workflow",{"type":43,"tag":52,"props":3331,"children":3332},{},[3333,3335,3341],{"type":49,"value":3334},"Data pipeline → Workflows → New workflow. Name it ",{"type":43,"tag":79,"props":3336,"children":3338},{"className":3337},[],[3339],{"type":49,"value":3340},"\u003Cfeature> use case feed",{"type":49,"value":3342}," to match the eval name from step 4.",{"type":43,"tag":3324,"props":3344,"children":3346},{"id":3345},"_62-trigger-step",[3347],{"type":49,"value":3348},"6.2 Trigger step",{"type":43,"tag":95,"props":3350,"children":3351},{},[3352,3405],{"type":43,"tag":99,"props":3353,"children":3354},{},[3355,3360,3361,3367,3369,3374,3376,3382,3384,3389,3391,3396,3398,3403],{"type":43,"tag":63,"props":3356,"children":3357},{},[3358],{"type":49,"value":3359},"Event:",{"type":49,"value":77},{"type":43,"tag":79,"props":3362,"children":3364},{"className":3363},[],[3365],{"type":49,"value":3366},"AI evaluation (LLM)",{"type":49,"value":3368}," — i.e. ",{"type":43,"tag":79,"props":3370,"children":3372},{"className":3371},[],[3373],{"type":49,"value":473},{"type":49,"value":3375},". This is the event emitted when an eval runs, and it's the only event that carries ",{"type":43,"tag":79,"props":3377,"children":3379},{"className":3378},[],[3380],{"type":49,"value":3381},"$ai_evaluation_*",{"type":49,"value":3383}," properties. The original ",{"type":43,"tag":79,"props":3385,"children":3387},{"className":3386},[],[3388],{"type":49,"value":258},{"type":49,"value":3390}," event is ",{"type":43,"tag":63,"props":3392,"children":3393},{},[3394],{"type":49,"value":3395},"not",{"type":49,"value":3397}," enriched with eval results, so filtering on ",{"type":43,"tag":79,"props":3399,"children":3401},{"className":3400},[],[3402],{"type":49,"value":258},{"type":49,"value":3404}," here matches nothing.",{"type":43,"tag":99,"props":3406,"children":3407},{},[3408,3413],{"type":43,"tag":63,"props":3409,"children":3410},{},[3411],{"type":49,"value":3412},"Property filters (both required):",{"type":43,"tag":95,"props":3414,"children":3415},{},[3416,3433],{"type":43,"tag":99,"props":3417,"children":3418},{},[3419,3425,3427],{"type":43,"tag":79,"props":3420,"children":3422},{"className":3421},[],[3423],{"type":49,"value":3424},"AI Evaluation Name (LLM)",{"type":49,"value":3426}," equals ",{"type":43,"tag":79,"props":3428,"children":3430},{"className":3429},[],[3431],{"type":49,"value":3432},"\u003Cyour eval name from step 4>",{"type":43,"tag":99,"props":3434,"children":3435},{},[3436,3442,3443],{"type":43,"tag":79,"props":3437,"children":3439},{"className":3438},[],[3440],{"type":49,"value":3441},"AI Evaluation Result (LLM)",{"type":49,"value":3426},{"type":43,"tag":79,"props":3444,"children":3446},{"className":3445},[],[3447],{"type":49,"value":3448},"true",{"type":43,"tag":52,"props":3450,"children":3451},{},[3452,3457,3459,3465,3467,3473,3474,3480,3481,3487,3489,3495,3496,3502,3503,3509,3511,3516,3518,3523,3525,3531],{"type":43,"tag":63,"props":3453,"children":3454},{},[3455],{"type":49,"value":3456},"⚠️ LOAD-BEARING:",{"type":49,"value":3458}," the stored values for ",{"type":43,"tag":79,"props":3460,"children":3462},{"className":3461},[],[3463],{"type":49,"value":3464},"$ai_evaluation_result",{"type":49,"value":3466}," are the strings ",{"type":43,"tag":79,"props":3468,"children":3470},{"className":3469},[],[3471],{"type":49,"value":3472},"'True'",{"type":49,"value":708},{"type":43,"tag":79,"props":3475,"children":3477},{"className":3476},[],[3478],{"type":49,"value":3479},"'False'",{"type":49,"value":708},{"type":43,"tag":79,"props":3482,"children":3484},{"className":3483},[],[3485],{"type":49,"value":3486},"'None'",{"type":49,"value":3488}," — NOT ",{"type":43,"tag":79,"props":3490,"children":3492},{"className":3491},[],[3493],{"type":49,"value":3494},"'PASS'",{"type":49,"value":708},{"type":43,"tag":79,"props":3497,"children":3499},{"className":3498},[],[3500],{"type":49,"value":3501},"'FAIL'",{"type":49,"value":708},{"type":43,"tag":79,"props":3504,"children":3506},{"className":3505},[],[3507],{"type":49,"value":3508},"'N\u002FA'",{"type":49,"value":3510}," (despite what the prompt template calls them internally). The Workflows UI property filter normalizes ",{"type":43,"tag":79,"props":3512,"children":3514},{"className":3513},[],[3515],{"type":49,"value":3448},{"type":49,"value":3517}," → ",{"type":43,"tag":79,"props":3519,"children":3521},{"className":3520},[],[3522],{"type":49,"value":3472},{"type":49,"value":3524},", so selecting ",{"type":43,"tag":79,"props":3526,"children":3528},{"className":3527},[],[3529],{"type":49,"value":3530},"equals true",{"type":49,"value":3532}," from the dropdown works. But if you were wiring this in raw SQL somewhere else (say a hog function), you'd need the string literal. Verify the stored distribution before saving:",{"type":43,"tag":768,"props":3534,"children":3536},{"className":770,"code":3535,"language":772,"meta":773,"style":773},"SELECT DISTINCT toString(properties.$ai_evaluation_result) AS result, count() AS n\nFROM events\nWHERE event = '$ai_evaluation'\n  AND properties.$ai_evaluation_name = '\u003Cyour eval name>'\n  AND timestamp > now() - INTERVAL 1 HOUR\nGROUP BY result\n",[3537],{"type":43,"tag":79,"props":3538,"children":3539},{"__ignoreMap":773},[3540,3548,3555,3563,3571,3579],{"type":43,"tag":105,"props":3541,"children":3542},{"class":780,"line":781},[3543],{"type":43,"tag":105,"props":3544,"children":3545},{},[3546],{"type":49,"value":3547},"SELECT DISTINCT toString(properties.$ai_evaluation_result) AS result, count() AS n\n",{"type":43,"tag":105,"props":3549,"children":3550},{"class":780,"line":790},[3551],{"type":43,"tag":105,"props":3552,"children":3553},{},[3554],{"type":49,"value":822},{"type":43,"tag":105,"props":3556,"children":3557},{"class":780,"line":799},[3558],{"type":43,"tag":105,"props":3559,"children":3560},{},[3561],{"type":49,"value":3562},"WHERE event = '$ai_evaluation'\n",{"type":43,"tag":105,"props":3564,"children":3565},{"class":780,"line":30},[3566],{"type":43,"tag":105,"props":3567,"children":3568},{},[3569],{"type":49,"value":3570},"  AND properties.$ai_evaluation_name = '\u003Cyour eval name>'\n",{"type":43,"tag":105,"props":3572,"children":3573},{"class":780,"line":816},[3574],{"type":43,"tag":105,"props":3575,"children":3576},{},[3577],{"type":49,"value":3578},"  AND timestamp > now() - INTERVAL 1 HOUR\n",{"type":43,"tag":105,"props":3580,"children":3581},{"class":780,"line":825},[3582],{"type":43,"tag":105,"props":3583,"children":3584},{},[3585],{"type":49,"value":3586},"GROUP BY result\n",{"type":43,"tag":52,"props":3588,"children":3589},{},[3590,3592,3598,3600,3606,3607,3613,3614,3619,3621,3626],{"type":49,"value":3591},"If the only values are ",{"type":43,"tag":79,"props":3593,"children":3595},{"className":3594},[],[3596],{"type":49,"value":3597},"True",{"type":49,"value":3599},"\u002F",{"type":43,"tag":79,"props":3601,"children":3603},{"className":3602},[],[3604],{"type":49,"value":3605},"False",{"type":49,"value":3599},{"type":43,"tag":79,"props":3608,"children":3610},{"className":3609},[],[3611],{"type":49,"value":3612},"None",{"type":49,"value":244},{"type":43,"tag":79,"props":3615,"children":3617},{"className":3616},[],[3618],{"type":49,"value":3597},{"type":49,"value":3620}," dominates, the UI ",{"type":43,"tag":79,"props":3622,"children":3624},{"className":3623},[],[3625],{"type":49,"value":3530},{"type":49,"value":3627}," filter will match. If you see anything else, adjust accordingly.",{"type":43,"tag":3324,"props":3629,"children":3631},{"id":3630},"_63-slack-dispatch-step",[3632],{"type":49,"value":3633},"6.3 Slack dispatch step",{"type":43,"tag":95,"props":3635,"children":3636},{},[3637,3645,3660,3678],{"type":43,"tag":99,"props":3638,"children":3639},{},[3640],{"type":43,"tag":63,"props":3641,"children":3642},{},[3643],{"type":49,"value":3644},"Add step → Slack dispatch",{"type":43,"tag":99,"props":3646,"children":3647},{},[3648,3653,3654],{"type":43,"tag":63,"props":3649,"children":3650},{},[3651],{"type":49,"value":3652},"Channel:",{"type":49,"value":77},{"type":43,"tag":79,"props":3655,"children":3657},{"className":3656},[],[3658],{"type":49,"value":3659},"#\u003Cyour-team>-usage-feed",{"type":43,"tag":99,"props":3661,"children":3662},{},[3663,3668,3670,3676],{"type":43,"tag":63,"props":3664,"children":3665},{},[3666],{"type":49,"value":3667},"Sender \u002F bot display name:",{"type":49,"value":3669}," something that reads well in the channel (e.g. ",{"type":43,"tag":79,"props":3671,"children":3673},{"className":3672},[],[3674],{"type":49,"value":3675},"PostHog Usage Feed",{"type":49,"value":3677},")",{"type":43,"tag":99,"props":3679,"children":3680},{},[3681,3686,3688,3694,3696,3702],{"type":43,"tag":63,"props":3682,"children":3683},{},[3684],{"type":49,"value":3685},"Blocks (Slack block-kit JSON)",{"type":49,"value":3687}," — paste this and replace ",{"type":43,"tag":79,"props":3689,"children":3691},{"className":3690},[],[3692],{"type":49,"value":3693},"\u003Cproject_id>",{"type":49,"value":3695}," with your actual numeric project ID (e.g. ",{"type":43,"tag":79,"props":3697,"children":3699},{"className":3698},[],[3700],{"type":49,"value":3701},"2",{"type":49,"value":3703},"):",{"type":43,"tag":768,"props":3705,"children":3707},{"className":1109,"code":3706,"language":1111,"meta":773,"style":773},"[\n  {\n    \"text\": {\n      \"text\": \"\u003Cemoji> *{event.properties.$ai_evaluation_name}* triggered by *{person.name}*\",\n      \"type\": \"mrkdwn\"\n    },\n    \"type\": \"section\"\n  },\n  {\n    \"text\": {\n      \"text\": \"{event.properties.$ai_evaluation_reasoning}\",\n      \"type\": \"mrkdwn\"\n    },\n    \"type\": \"section\"\n  },\n  {\n    \"type\": \"actions\",\n    \"elements\": [\n      {\n        \"url\": \"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cproject_id>\u002Fai-observability\u002Ftraces\u002F{event.properties.$ai_trace_id}?event={event.properties.$ai_target_event_id}\",\n        \"text\": { \"text\": \"View Trace\", \"type\": \"plain_text\" },\n        \"type\": \"button\"\n      },\n      {\n        \"url\": \"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cproject_id>\u002Freplay\u002F{event.properties.$session_id}\",\n        \"text\": { \"text\": \"View Trigger Session\", \"type\": \"plain_text\" },\n        \"type\": \"button\"\n      },\n      {\n        \"url\": \"{person.url}\",\n        \"text\": { \"text\": \"View Person\", \"type\": \"plain_text\" },\n        \"type\": \"button\"\n      }\n    ]\n  }\n]\n",[3708],{"type":43,"tag":79,"props":3709,"children":3710},{"__ignoreMap":773},[3711,3719,3727,3750,3787,3819,3827,3859,3866,3873,3896,3932,3963,3970,4001,4008,4015,4051,4075,4083,4121,4210,4242,4250,4257,4293,4381,4413,4421,4429,4466,4555,4587,4596,4604,4612],{"type":43,"tag":105,"props":3712,"children":3713},{"class":780,"line":781},[3714],{"type":43,"tag":105,"props":3715,"children":3716},{"style":1130},[3717],{"type":49,"value":3718},"[\n",{"type":43,"tag":105,"props":3720,"children":3721},{"class":780,"line":790},[3722],{"type":43,"tag":105,"props":3723,"children":3724},{"style":1130},[3725],{"type":49,"value":3726},"  {\n",{"type":43,"tag":105,"props":3728,"children":3729},{"class":780,"line":799},[3730,3734,3738,3742,3746],{"type":43,"tag":105,"props":3731,"children":3732},{"style":1130},[3733],{"type":49,"value":2371},{"type":43,"tag":105,"props":3735,"children":3736},{"style":1144},[3737],{"type":49,"value":49},{"type":43,"tag":105,"props":3739,"children":3740},{"style":1130},[3741],{"type":49,"value":1152},{"type":43,"tag":105,"props":3743,"children":3744},{"style":1130},[3745],{"type":49,"value":1157},{"type":43,"tag":105,"props":3747,"children":3748},{"style":1130},[3749],{"type":49,"value":2363},{"type":43,"tag":105,"props":3751,"children":3752},{"class":780,"line":30},[3753,3758,3762,3766,3770,3774,3779,3783],{"type":43,"tag":105,"props":3754,"children":3755},{"style":1130},[3756],{"type":49,"value":3757},"      \"",{"type":43,"tag":105,"props":3759,"children":3760},{"style":1178},[3761],{"type":49,"value":49},{"type":43,"tag":105,"props":3763,"children":3764},{"style":1130},[3765],{"type":49,"value":1152},{"type":43,"tag":105,"props":3767,"children":3768},{"style":1130},[3769],{"type":49,"value":1157},{"type":43,"tag":105,"props":3771,"children":3772},{"style":1130},[3773],{"type":49,"value":1175},{"type":43,"tag":105,"props":3775,"children":3776},{"style":1196},[3777],{"type":49,"value":3778},"\u003Cemoji> *{event.properties.$ai_evaluation_name}* triggered by *{person.name}*",{"type":43,"tag":105,"props":3780,"children":3781},{"style":1130},[3782],{"type":49,"value":1152},{"type":43,"tag":105,"props":3784,"children":3785},{"style":1130},[3786],{"type":49,"value":1349},{"type":43,"tag":105,"props":3788,"children":3789},{"class":780,"line":816},[3790,3794,3798,3802,3806,3810,3815],{"type":43,"tag":105,"props":3791,"children":3792},{"style":1130},[3793],{"type":49,"value":3757},{"type":43,"tag":105,"props":3795,"children":3796},{"style":1178},[3797],{"type":49,"value":1181},{"type":43,"tag":105,"props":3799,"children":3800},{"style":1130},[3801],{"type":49,"value":1152},{"type":43,"tag":105,"props":3803,"children":3804},{"style":1130},[3805],{"type":49,"value":1157},{"type":43,"tag":105,"props":3807,"children":3808},{"style":1130},[3809],{"type":49,"value":1175},{"type":43,"tag":105,"props":3811,"children":3812},{"style":1196},[3813],{"type":49,"value":3814},"mrkdwn",{"type":43,"tag":105,"props":3816,"children":3817},{"style":1130},[3818],{"type":49,"value":2398},{"type":43,"tag":105,"props":3820,"children":3821},{"class":780,"line":825},[3822],{"type":43,"tag":105,"props":3823,"children":3824},{"style":1130},[3825],{"type":49,"value":3826},"    },\n",{"type":43,"tag":105,"props":3828,"children":3829},{"class":780,"line":834},[3830,3834,3838,3842,3846,3850,3855],{"type":43,"tag":105,"props":3831,"children":3832},{"style":1130},[3833],{"type":49,"value":2371},{"type":43,"tag":105,"props":3835,"children":3836},{"style":1144},[3837],{"type":49,"value":1181},{"type":43,"tag":105,"props":3839,"children":3840},{"style":1130},[3841],{"type":49,"value":1152},{"type":43,"tag":105,"props":3843,"children":3844},{"style":1130},[3845],{"type":49,"value":1157},{"type":43,"tag":105,"props":3847,"children":3848},{"style":1130},[3849],{"type":49,"value":1175},{"type":43,"tag":105,"props":3851,"children":3852},{"style":1196},[3853],{"type":49,"value":3854},"section",{"type":43,"tag":105,"props":3856,"children":3857},{"style":1130},[3858],{"type":49,"value":2398},{"type":43,"tag":105,"props":3860,"children":3861},{"class":780,"line":843},[3862],{"type":43,"tag":105,"props":3863,"children":3864},{"style":1130},[3865],{"type":49,"value":2406},{"type":43,"tag":105,"props":3867,"children":3868},{"class":780,"line":852},[3869],{"type":43,"tag":105,"props":3870,"children":3871},{"style":1130},[3872],{"type":49,"value":3726},{"type":43,"tag":105,"props":3874,"children":3875},{"class":780,"line":861},[3876,3880,3884,3888,3892],{"type":43,"tag":105,"props":3877,"children":3878},{"style":1130},[3879],{"type":49,"value":2371},{"type":43,"tag":105,"props":3881,"children":3882},{"style":1144},[3883],{"type":49,"value":49},{"type":43,"tag":105,"props":3885,"children":3886},{"style":1130},[3887],{"type":49,"value":1152},{"type":43,"tag":105,"props":3889,"children":3890},{"style":1130},[3891],{"type":49,"value":1157},{"type":43,"tag":105,"props":3893,"children":3894},{"style":1130},[3895],{"type":49,"value":2363},{"type":43,"tag":105,"props":3897,"children":3898},{"class":780,"line":870},[3899,3903,3907,3911,3915,3919,3924,3928],{"type":43,"tag":105,"props":3900,"children":3901},{"style":1130},[3902],{"type":49,"value":3757},{"type":43,"tag":105,"props":3904,"children":3905},{"style":1178},[3906],{"type":49,"value":49},{"type":43,"tag":105,"props":3908,"children":3909},{"style":1130},[3910],{"type":49,"value":1152},{"type":43,"tag":105,"props":3912,"children":3913},{"style":1130},[3914],{"type":49,"value":1157},{"type":43,"tag":105,"props":3916,"children":3917},{"style":1130},[3918],{"type":49,"value":1175},{"type":43,"tag":105,"props":3920,"children":3921},{"style":1196},[3922],{"type":49,"value":3923},"{event.properties.$ai_evaluation_reasoning}",{"type":43,"tag":105,"props":3925,"children":3926},{"style":1130},[3927],{"type":49,"value":1152},{"type":43,"tag":105,"props":3929,"children":3930},{"style":1130},[3931],{"type":49,"value":1349},{"type":43,"tag":105,"props":3933,"children":3934},{"class":780,"line":1000},[3935,3939,3943,3947,3951,3955,3959],{"type":43,"tag":105,"props":3936,"children":3937},{"style":1130},[3938],{"type":49,"value":3757},{"type":43,"tag":105,"props":3940,"children":3941},{"style":1178},[3942],{"type":49,"value":1181},{"type":43,"tag":105,"props":3944,"children":3945},{"style":1130},[3946],{"type":49,"value":1152},{"type":43,"tag":105,"props":3948,"children":3949},{"style":1130},[3950],{"type":49,"value":1157},{"type":43,"tag":105,"props":3952,"children":3953},{"style":1130},[3954],{"type":49,"value":1175},{"type":43,"tag":105,"props":3956,"children":3957},{"style":1196},[3958],{"type":49,"value":3814},{"type":43,"tag":105,"props":3960,"children":3961},{"style":1130},[3962],{"type":49,"value":2398},{"type":43,"tag":105,"props":3964,"children":3965},{"class":780,"line":2557},[3966],{"type":43,"tag":105,"props":3967,"children":3968},{"style":1130},[3969],{"type":49,"value":3826},{"type":43,"tag":105,"props":3971,"children":3972},{"class":780,"line":2591},[3973,3977,3981,3985,3989,3993,3997],{"type":43,"tag":105,"props":3974,"children":3975},{"style":1130},[3976],{"type":49,"value":2371},{"type":43,"tag":105,"props":3978,"children":3979},{"style":1144},[3980],{"type":49,"value":1181},{"type":43,"tag":105,"props":3982,"children":3983},{"style":1130},[3984],{"type":49,"value":1152},{"type":43,"tag":105,"props":3986,"children":3987},{"style":1130},[3988],{"type":49,"value":1157},{"type":43,"tag":105,"props":3990,"children":3991},{"style":1130},[3992],{"type":49,"value":1175},{"type":43,"tag":105,"props":3994,"children":3995},{"style":1196},[3996],{"type":49,"value":3854},{"type":43,"tag":105,"props":3998,"children":3999},{"style":1130},[4000],{"type":49,"value":2398},{"type":43,"tag":105,"props":4002,"children":4003},{"class":780,"line":2599},[4004],{"type":43,"tag":105,"props":4005,"children":4006},{"style":1130},[4007],{"type":49,"value":2406},{"type":43,"tag":105,"props":4009,"children":4010},{"class":780,"line":2625},[4011],{"type":43,"tag":105,"props":4012,"children":4013},{"style":1130},[4014],{"type":49,"value":3726},{"type":43,"tag":105,"props":4016,"children":4017},{"class":780,"line":2650},[4018,4022,4026,4030,4034,4038,4043,4047],{"type":43,"tag":105,"props":4019,"children":4020},{"style":1130},[4021],{"type":49,"value":2371},{"type":43,"tag":105,"props":4023,"children":4024},{"style":1144},[4025],{"type":49,"value":1181},{"type":43,"tag":105,"props":4027,"children":4028},{"style":1130},[4029],{"type":49,"value":1152},{"type":43,"tag":105,"props":4031,"children":4032},{"style":1130},[4033],{"type":49,"value":1157},{"type":43,"tag":105,"props":4035,"children":4036},{"style":1130},[4037],{"type":49,"value":1175},{"type":43,"tag":105,"props":4039,"children":4040},{"style":1196},[4041],{"type":49,"value":4042},"actions",{"type":43,"tag":105,"props":4044,"children":4045},{"style":1130},[4046],{"type":49,"value":1152},{"type":43,"tag":105,"props":4048,"children":4049},{"style":1130},[4050],{"type":49,"value":1349},{"type":43,"tag":105,"props":4052,"children":4053},{"class":780,"line":2675},[4054,4058,4063,4067,4071],{"type":43,"tag":105,"props":4055,"children":4056},{"style":1130},[4057],{"type":49,"value":2371},{"type":43,"tag":105,"props":4059,"children":4060},{"style":1144},[4061],{"type":49,"value":4062},"elements",{"type":43,"tag":105,"props":4064,"children":4065},{"style":1130},[4066],{"type":49,"value":1152},{"type":43,"tag":105,"props":4068,"children":4069},{"style":1130},[4070],{"type":49,"value":1157},{"type":43,"tag":105,"props":4072,"children":4073},{"style":1130},[4074],{"type":49,"value":1162},{"type":43,"tag":105,"props":4076,"children":4077},{"class":780,"line":2685},[4078],{"type":43,"tag":105,"props":4079,"children":4080},{"style":1130},[4081],{"type":49,"value":4082},"      {\n",{"type":43,"tag":105,"props":4084,"children":4085},{"class":780,"line":2791},[4086,4091,4096,4100,4104,4108,4113,4117],{"type":43,"tag":105,"props":4087,"children":4088},{"style":1130},[4089],{"type":49,"value":4090},"        \"",{"type":43,"tag":105,"props":4092,"children":4093},{"style":1178},[4094],{"type":49,"value":4095},"url",{"type":43,"tag":105,"props":4097,"children":4098},{"style":1130},[4099],{"type":49,"value":1152},{"type":43,"tag":105,"props":4101,"children":4102},{"style":1130},[4103],{"type":49,"value":1157},{"type":43,"tag":105,"props":4105,"children":4106},{"style":1130},[4107],{"type":49,"value":1175},{"type":43,"tag":105,"props":4109,"children":4110},{"style":1196},[4111],{"type":49,"value":4112},"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cproject_id>\u002Fai-observability\u002Ftraces\u002F{event.properties.$ai_trace_id}?event={event.properties.$ai_target_event_id}",{"type":43,"tag":105,"props":4114,"children":4115},{"style":1130},[4116],{"type":49,"value":1152},{"type":43,"tag":105,"props":4118,"children":4119},{"style":1130},[4120],{"type":49,"value":1349},{"type":43,"tag":105,"props":4122,"children":4123},{"class":780,"line":2801},[4124,4128,4132,4136,4140,4144,4148,4152,4156,4160,4164,4169,4173,4177,4181,4185,4189,4193,4197,4202,4206],{"type":43,"tag":105,"props":4125,"children":4126},{"style":1130},[4127],{"type":49,"value":4090},{"type":43,"tag":105,"props":4129,"children":4130},{"style":1178},[4131],{"type":49,"value":49},{"type":43,"tag":105,"props":4133,"children":4134},{"style":1130},[4135],{"type":49,"value":1152},{"type":43,"tag":105,"props":4137,"children":4138},{"style":1130},[4139],{"type":49,"value":1157},{"type":43,"tag":105,"props":4141,"children":4142},{"style":1130},[4143],{"type":49,"value":1374},{"type":43,"tag":105,"props":4145,"children":4146},{"style":1130},[4147],{"type":49,"value":1175},{"type":43,"tag":105,"props":4149,"children":4150},{"style":1341},[4151],{"type":49,"value":49},{"type":43,"tag":105,"props":4153,"children":4154},{"style":1130},[4155],{"type":49,"value":1152},{"type":43,"tag":105,"props":4157,"children":4158},{"style":1130},[4159],{"type":49,"value":1157},{"type":43,"tag":105,"props":4161,"children":4162},{"style":1130},[4163],{"type":49,"value":1175},{"type":43,"tag":105,"props":4165,"children":4166},{"style":1196},[4167],{"type":49,"value":4168},"View Trace",{"type":43,"tag":105,"props":4170,"children":4171},{"style":1130},[4172],{"type":49,"value":1152},{"type":43,"tag":105,"props":4174,"children":4175},{"style":1130},[4176],{"type":49,"value":1208},{"type":43,"tag":105,"props":4178,"children":4179},{"style":1130},[4180],{"type":49,"value":1175},{"type":43,"tag":105,"props":4182,"children":4183},{"style":1341},[4184],{"type":49,"value":1181},{"type":43,"tag":105,"props":4186,"children":4187},{"style":1130},[4188],{"type":49,"value":1152},{"type":43,"tag":105,"props":4190,"children":4191},{"style":1130},[4192],{"type":49,"value":1157},{"type":43,"tag":105,"props":4194,"children":4195},{"style":1130},[4196],{"type":49,"value":1175},{"type":43,"tag":105,"props":4198,"children":4199},{"style":1196},[4200],{"type":49,"value":4201},"plain_text",{"type":43,"tag":105,"props":4203,"children":4204},{"style":1130},[4205],{"type":49,"value":1152},{"type":43,"tag":105,"props":4207,"children":4208},{"style":1130},[4209],{"type":49,"value":1409},{"type":43,"tag":105,"props":4211,"children":4212},{"class":780,"line":2810},[4213,4217,4221,4225,4229,4233,4238],{"type":43,"tag":105,"props":4214,"children":4215},{"style":1130},[4216],{"type":49,"value":4090},{"type":43,"tag":105,"props":4218,"children":4219},{"style":1178},[4220],{"type":49,"value":1181},{"type":43,"tag":105,"props":4222,"children":4223},{"style":1130},[4224],{"type":49,"value":1152},{"type":43,"tag":105,"props":4226,"children":4227},{"style":1130},[4228],{"type":49,"value":1157},{"type":43,"tag":105,"props":4230,"children":4231},{"style":1130},[4232],{"type":49,"value":1175},{"type":43,"tag":105,"props":4234,"children":4235},{"style":1196},[4236],{"type":49,"value":4237},"button",{"type":43,"tag":105,"props":4239,"children":4240},{"style":1130},[4241],{"type":49,"value":2398},{"type":43,"tag":105,"props":4243,"children":4244},{"class":780,"line":2819},[4245],{"type":43,"tag":105,"props":4246,"children":4247},{"style":1130},[4248],{"type":49,"value":4249},"      },\n",{"type":43,"tag":105,"props":4251,"children":4252},{"class":780,"line":2828},[4253],{"type":43,"tag":105,"props":4254,"children":4255},{"style":1130},[4256],{"type":49,"value":4082},{"type":43,"tag":105,"props":4258,"children":4259},{"class":780,"line":2837},[4260,4264,4268,4272,4276,4280,4285,4289],{"type":43,"tag":105,"props":4261,"children":4262},{"style":1130},[4263],{"type":49,"value":4090},{"type":43,"tag":105,"props":4265,"children":4266},{"style":1178},[4267],{"type":49,"value":4095},{"type":43,"tag":105,"props":4269,"children":4270},{"style":1130},[4271],{"type":49,"value":1152},{"type":43,"tag":105,"props":4273,"children":4274},{"style":1130},[4275],{"type":49,"value":1157},{"type":43,"tag":105,"props":4277,"children":4278},{"style":1130},[4279],{"type":49,"value":1175},{"type":43,"tag":105,"props":4281,"children":4282},{"style":1196},[4283],{"type":49,"value":4284},"https:\u002F\u002Fus.posthog.com\u002Fproject\u002F\u003Cproject_id>\u002Freplay\u002F{event.properties.$session_id}",{"type":43,"tag":105,"props":4286,"children":4287},{"style":1130},[4288],{"type":49,"value":1152},{"type":43,"tag":105,"props":4290,"children":4291},{"style":1130},[4292],{"type":49,"value":1349},{"type":43,"tag":105,"props":4294,"children":4295},{"class":780,"line":2846},[4296,4300,4304,4308,4312,4316,4320,4324,4328,4332,4336,4341,4345,4349,4353,4357,4361,4365,4369,4373,4377],{"type":43,"tag":105,"props":4297,"children":4298},{"style":1130},[4299],{"type":49,"value":4090},{"type":43,"tag":105,"props":4301,"children":4302},{"style":1178},[4303],{"type":49,"value":49},{"type":43,"tag":105,"props":4305,"children":4306},{"style":1130},[4307],{"type":49,"value":1152},{"type":43,"tag":105,"props":4309,"children":4310},{"style":1130},[4311],{"type":49,"value":1157},{"type":43,"tag":105,"props":4313,"children":4314},{"style":1130},[4315],{"type":49,"value":1374},{"type":43,"tag":105,"props":4317,"children":4318},{"style":1130},[4319],{"type":49,"value":1175},{"type":43,"tag":105,"props":4321,"children":4322},{"style":1341},[4323],{"type":49,"value":49},{"type":43,"tag":105,"props":4325,"children":4326},{"style":1130},[4327],{"type":49,"value":1152},{"type":43,"tag":105,"props":4329,"children":4330},{"style":1130},[4331],{"type":49,"value":1157},{"type":43,"tag":105,"props":4333,"children":4334},{"style":1130},[4335],{"type":49,"value":1175},{"type":43,"tag":105,"props":4337,"children":4338},{"style":1196},[4339],{"type":49,"value":4340},"View Trigger Session",{"type":43,"tag":105,"props":4342,"children":4343},{"style":1130},[4344],{"type":49,"value":1152},{"type":43,"tag":105,"props":4346,"children":4347},{"style":1130},[4348],{"type":49,"value":1208},{"type":43,"tag":105,"props":4350,"children":4351},{"style":1130},[4352],{"type":49,"value":1175},{"type":43,"tag":105,"props":4354,"children":4355},{"style":1341},[4356],{"type":49,"value":1181},{"type":43,"tag":105,"props":4358,"children":4359},{"style":1130},[4360],{"type":49,"value":1152},{"type":43,"tag":105,"props":4362,"children":4363},{"style":1130},[4364],{"type":49,"value":1157},{"type":43,"tag":105,"props":4366,"children":4367},{"style":1130},[4368],{"type":49,"value":1175},{"type":43,"tag":105,"props":4370,"children":4371},{"style":1196},[4372],{"type":49,"value":4201},{"type":43,"tag":105,"props":4374,"children":4375},{"style":1130},[4376],{"type":49,"value":1152},{"type":43,"tag":105,"props":4378,"children":4379},{"style":1130},[4380],{"type":49,"value":1409},{"type":43,"tag":105,"props":4382,"children":4384},{"class":780,"line":4383},27,[4385,4389,4393,4397,4401,4405,4409],{"type":43,"tag":105,"props":4386,"children":4387},{"style":1130},[4388],{"type":49,"value":4090},{"type":43,"tag":105,"props":4390,"children":4391},{"style":1178},[4392],{"type":49,"value":1181},{"type":43,"tag":105,"props":4394,"children":4395},{"style":1130},[4396],{"type":49,"value":1152},{"type":43,"tag":105,"props":4398,"children":4399},{"style":1130},[4400],{"type":49,"value":1157},{"type":43,"tag":105,"props":4402,"children":4403},{"style":1130},[4404],{"type":49,"value":1175},{"type":43,"tag":105,"props":4406,"children":4407},{"style":1196},[4408],{"type":49,"value":4237},{"type":43,"tag":105,"props":4410,"children":4411},{"style":1130},[4412],{"type":49,"value":2398},{"type":43,"tag":105,"props":4414,"children":4416},{"class":780,"line":4415},28,[4417],{"type":43,"tag":105,"props":4418,"children":4419},{"style":1130},[4420],{"type":49,"value":4249},{"type":43,"tag":105,"props":4422,"children":4424},{"class":780,"line":4423},29,[4425],{"type":43,"tag":105,"props":4426,"children":4427},{"style":1130},[4428],{"type":49,"value":4082},{"type":43,"tag":105,"props":4430,"children":4432},{"class":780,"line":4431},30,[4433,4437,4441,4445,4449,4453,4458,4462],{"type":43,"tag":105,"props":4434,"children":4435},{"style":1130},[4436],{"type":49,"value":4090},{"type":43,"tag":105,"props":4438,"children":4439},{"style":1178},[4440],{"type":49,"value":4095},{"type":43,"tag":105,"props":4442,"children":4443},{"style":1130},[4444],{"type":49,"value":1152},{"type":43,"tag":105,"props":4446,"children":4447},{"style":1130},[4448],{"type":49,"value":1157},{"type":43,"tag":105,"props":4450,"children":4451},{"style":1130},[4452],{"type":49,"value":1175},{"type":43,"tag":105,"props":4454,"children":4455},{"style":1196},[4456],{"type":49,"value":4457},"{person.url}",{"type":43,"tag":105,"props":4459,"children":4460},{"style":1130},[4461],{"type":49,"value":1152},{"type":43,"tag":105,"props":4463,"children":4464},{"style":1130},[4465],{"type":49,"value":1349},{"type":43,"tag":105,"props":4467,"children":4469},{"class":780,"line":4468},31,[4470,4474,4478,4482,4486,4490,4494,4498,4502,4506,4510,4515,4519,4523,4527,4531,4535,4539,4543,4547,4551],{"type":43,"tag":105,"props":4471,"children":4472},{"style":1130},[4473],{"type":49,"value":4090},{"type":43,"tag":105,"props":4475,"children":4476},{"style":1178},[4477],{"type":49,"value":49},{"type":43,"tag":105,"props":4479,"children":4480},{"style":1130},[4481],{"type":49,"value":1152},{"type":43,"tag":105,"props":4483,"children":4484},{"style":1130},[4485],{"type":49,"value":1157},{"type":43,"tag":105,"props":4487,"children":4488},{"style":1130},[4489],{"type":49,"value":1374},{"type":43,"tag":105,"props":4491,"children":4492},{"style":1130},[4493],{"type":49,"value":1175},{"type":43,"tag":105,"props":4495,"children":4496},{"style":1341},[4497],{"type":49,"value":49},{"type":43,"tag":105,"props":4499,"children":4500},{"style":1130},[4501],{"type":49,"value":1152},{"type":43,"tag":105,"props":4503,"children":4504},{"style":1130},[4505],{"type":49,"value":1157},{"type":43,"tag":105,"props":4507,"children":4508},{"style":1130},[4509],{"type":49,"value":1175},{"type":43,"tag":105,"props":4511,"children":4512},{"style":1196},[4513],{"type":49,"value":4514},"View Person",{"type":43,"tag":105,"props":4516,"children":4517},{"style":1130},[4518],{"type":49,"value":1152},{"type":43,"tag":105,"props":4520,"children":4521},{"style":1130},[4522],{"type":49,"value":1208},{"type":43,"tag":105,"props":4524,"children":4525},{"style":1130},[4526],{"type":49,"value":1175},{"type":43,"tag":105,"props":4528,"children":4529},{"style":1341},[4530],{"type":49,"value":1181},{"type":43,"tag":105,"props":4532,"children":4533},{"style":1130},[4534],{"type":49,"value":1152},{"type":43,"tag":105,"props":4536,"children":4537},{"style":1130},[4538],{"type":49,"value":1157},{"type":43,"tag":105,"props":4540,"children":4541},{"style":1130},[4542],{"type":49,"value":1175},{"type":43,"tag":105,"props":4544,"children":4545},{"style":1196},[4546],{"type":49,"value":4201},{"type":43,"tag":105,"props":4548,"children":4549},{"style":1130},[4550],{"type":49,"value":1152},{"type":43,"tag":105,"props":4552,"children":4553},{"style":1130},[4554],{"type":49,"value":1409},{"type":43,"tag":105,"props":4556,"children":4558},{"class":780,"line":4557},32,[4559,4563,4567,4571,4575,4579,4583],{"type":43,"tag":105,"props":4560,"children":4561},{"style":1130},[4562],{"type":49,"value":4090},{"type":43,"tag":105,"props":4564,"children":4565},{"style":1178},[4566],{"type":49,"value":1181},{"type":43,"tag":105,"props":4568,"children":4569},{"style":1130},[4570],{"type":49,"value":1152},{"type":43,"tag":105,"props":4572,"children":4573},{"style":1130},[4574],{"type":49,"value":1157},{"type":43,"tag":105,"props":4576,"children":4577},{"style":1130},[4578],{"type":49,"value":1175},{"type":43,"tag":105,"props":4580,"children":4581},{"style":1196},[4582],{"type":49,"value":4237},{"type":43,"tag":105,"props":4584,"children":4585},{"style":1130},[4586],{"type":49,"value":2398},{"type":43,"tag":105,"props":4588,"children":4590},{"class":780,"line":4589},33,[4591],{"type":43,"tag":105,"props":4592,"children":4593},{"style":1130},[4594],{"type":49,"value":4595},"      }\n",{"type":43,"tag":105,"props":4597,"children":4599},{"class":780,"line":4598},34,[4600],{"type":43,"tag":105,"props":4601,"children":4602},{"style":1130},[4603],{"type":49,"value":2834},{"type":43,"tag":105,"props":4605,"children":4607},{"class":780,"line":4606},35,[4608],{"type":43,"tag":105,"props":4609,"children":4610},{"style":1130},[4611],{"type":49,"value":2843},{"type":43,"tag":105,"props":4613,"children":4615},{"class":780,"line":4614},36,[4616],{"type":43,"tag":105,"props":4617,"children":4618},{"style":1130},[4619],{"type":49,"value":4620},"]\n",{"type":43,"tag":52,"props":4622,"children":4623},{},[4624,4625,4631],{"type":49,"value":1996},{"type":43,"tag":79,"props":4626,"children":4628},{"className":4627},[],[4629],{"type":49,"value":4630},"\u003Cemoji>",{"type":49,"value":4632}," that matches the feature's shape: 📊 product analytics, 🐛 error tracking, 🎬 session replay, 🔎 search\u002FAI search, 🧪 experiments, 🚩 flags, 📋 surveys, 🧠 generic AI.",{"type":43,"tag":52,"props":4634,"children":4635},{},[4636,4638,4644,4645,4651],{"type":49,"value":4637},"The ",{"type":43,"tag":79,"props":4639,"children":4641},{"className":4640},[],[4642],{"type":49,"value":4643},"{event.properties.X}",{"type":49,"value":244},{"type":43,"tag":79,"props":4646,"children":4648},{"className":4647},[],[4649],{"type":49,"value":4650},"{person.X}",{"type":49,"value":4652}," placeholders are valid PostHog template syntax and resolve at send time.",{"type":43,"tag":3324,"props":4654,"children":4656},{"id":4655},"_64-test-before-enabling",[4657],{"type":49,"value":4658},"6.4 Test before enabling",{"type":43,"tag":52,"props":4660,"children":4661},{},[4662],{"type":49,"value":4663},"The Workflows Test panel has two modes — this matters because naively hitting \"Test\" can look like a broken integration when it isn't:",{"type":43,"tag":95,"props":4665,"children":4666},{},[4667,4707],{"type":43,"tag":99,"props":4668,"children":4669},{},[4670,4675,4677,4682,4684,4690,4692,4697,4699,4705],{"type":43,"tag":63,"props":4671,"children":4672},{},[4673],{"type":49,"value":4674},"Synthetic event",{"type":49,"value":4676}," (default) — the Test panel fabricates an ",{"type":43,"tag":79,"props":4678,"children":4680},{"className":4679},[],[4681],{"type":49,"value":473},{"type":49,"value":4683}," payload and runs the flow without hitting Slack's real API. Useful as a dry-run of the block template, but ",{"type":43,"tag":79,"props":4685,"children":4687},{"className":4686},[],[4688],{"type":49,"value":4689},"{event.properties.$ai_*}",{"type":49,"value":4691}," placeholders may resolve to ",{"type":43,"tag":79,"props":4693,"children":4695},{"className":4694},[],[4696],{"type":49,"value":1072},{"type":49,"value":4698}," and Slack's block validator will reject the payload with ",{"type":43,"tag":79,"props":4700,"children":4702},{"className":4701},[],[4703],{"type":49,"value":4704},"invalid_blocks",{"type":49,"value":4706},". That's a test-harness artifact, not a real bug — don't chase it.",{"type":43,"tag":99,"props":4708,"children":4709},{},[4710,4715,4717,4722],{"type":43,"tag":63,"props":4711,"children":4712},{},[4713],{"type":49,"value":4714},"\"Make real HTTPS requests\"",{"type":49,"value":4716}," — flip this toggle on. Workflows then pulls a recent real ",{"type":43,"tag":79,"props":4718,"children":4720},{"className":4719},[],[4721],{"type":49,"value":473},{"type":49,"value":4723}," event matching your filters and runs the flow end-to-end, including the actual Slack post. This is the test that tells you \"it works\" for real. If no matching real event exists yet (common if the eval was just enabled), trigger the feature yourself, wait ~1 minute, and retry.",{"type":43,"tag":52,"props":4725,"children":4726},{},[4727],{"type":49,"value":4728},"Recommended flow: synthetic → sanity-check the block template renders → flip real-requests on → confirm an actual post lands in the channel → save + enable the workflow.",{"type":43,"tag":751,"props":4730,"children":4732},{"id":4731},"step-7-end-to-end-verify-in-production",[4733],{"type":49,"value":4734},"Step 7 — End-to-end verify in production",{"type":43,"tag":52,"props":4736,"children":4737},{},[4738],{"type":49,"value":4739},"Once the workflow is enabled, trigger the feature yourself. Within a minute or two:",{"type":43,"tag":4741,"props":4742,"children":4743},"ol",{},[4744,4755,4767,4772],{"type":43,"tag":99,"props":4745,"children":4746},{},[4747,4748,4753],{"type":49,"value":4637},{"type":43,"tag":79,"props":4749,"children":4751},{"className":4750},[],[4752],{"type":49,"value":258},{"type":49,"value":4754}," event should appear in AI observability",{"type":43,"tag":99,"props":4756,"children":4757},{},[4758,4760,4765],{"type":49,"value":4759},"The eval should auto-run and emit an ",{"type":43,"tag":79,"props":4761,"children":4763},{"className":4762},[],[4764],{"type":49,"value":473},{"type":49,"value":4766}," event",{"type":43,"tag":99,"props":4768,"children":4769},{},[4770],{"type":49,"value":4771},"The workflow should fire and the Slack post should land in the configured channel",{"type":43,"tag":99,"props":4773,"children":4774},{},[4775],{"type":49,"value":4776},"Click \"View Trigger Session\" — should land on the recording of you using the feature, not the replay homepage",{"type":43,"tag":52,"props":4778,"children":4779},{},[4780,4782,4787,4789,4794,4796,4801],{"type":49,"value":4781},"If \"View Trigger Session\" lands on the replay homepage, ",{"type":43,"tag":79,"props":4783,"children":4785},{"className":4784},[],[4786],{"type":49,"value":429},{"type":49,"value":4788}," is missing on the ",{"type":43,"tag":79,"props":4790,"children":4792},{"className":4791},[],[4793],{"type":49,"value":473},{"type":49,"value":4795}," event (which is separate from the ",{"type":43,"tag":79,"props":4797,"children":4799},{"className":4798},[],[4800],{"type":49,"value":258},{"type":49,"value":4802}," event — threading is independent for the two). Backend fix needed — see prerequisites.",{"type":43,"tag":88,"props":4804,"children":4806},{"id":4805},"worked-example-a-pattern-a-group-session-summary-use-cases",[4807],{"type":49,"value":4808},"Worked example A (Pattern A): group session summary use cases",{"type":43,"tag":52,"props":4810,"children":4811},{},[4812,4814,4820,4822,4828,4830,4835,4837,4843],{"type":49,"value":4813},"Pattern: a ",{"type":43,"tag":79,"props":4815,"children":4817},{"className":4816},[],[4818],{"type":49,"value":4819},"group_summary_use_case_feed",{"type":49,"value":4821}," eval streaming to a ",{"type":43,"tag":79,"props":4823,"children":4825},{"className":4824},[],[4826],{"type":49,"value":4827},"#\u003Cteam>-usage-feed",{"type":49,"value":4829}," channel. Trace prefix: ",{"type":43,"tag":79,"props":4831,"children":4833},{"className":4832},[],[4834],{"type":49,"value":201},{"type":49,"value":4836},". Opener: ",{"type":43,"tag":79,"props":4838,"children":4840},{"className":4839},[],[4841],{"type":49,"value":4842},"\"A user ran a group summary on\"",{"type":49,"value":4844},". Slack channel showed e.g.:",{"type":43,"tag":4846,"props":4847,"children":4848},"blockquote",{},[4849],{"type":43,"tag":52,"props":4850,"children":4851},{},[4852,4854,4858,4860,4865,4867,4871,4872,4876,4877],{"type":49,"value":4853},"📊 ",{"type":43,"tag":71,"props":4855,"children":4856},{},[4857],{"type":49,"value":4819},{"type":49,"value":4859}," triggered by ",{"type":43,"tag":71,"props":4861,"children":4862},{},[4863],{"type":49,"value":4864},"some user",{"type":49,"value":4866},"\n\"A user ran a group summary on a company's onboarding sessions from the last 7 days. They were trying to understand why account activation rates are low. The summary surfaced that most users abandon at the company onboarding wizard after creating accounts.\"\n",{"type":43,"tag":105,"props":4868,"children":4869},{},[4870],{"type":49,"value":4168},{"type":49,"value":77},{"type":43,"tag":105,"props":4873,"children":4874},{},[4875],{"type":49,"value":4340},{"type":49,"value":77},{"type":43,"tag":105,"props":4878,"children":4879},{},[4880],{"type":49,"value":4514},{"type":43,"tag":52,"props":4882,"children":4883},{},[4884],{"type":49,"value":4885},"The PRs that made this work (linked here as worked examples of the session_id threading pattern, not as steps in the skill itself):",{"type":43,"tag":95,"props":4887,"children":4888},{},[4889,4909],{"type":43,"tag":99,"props":4890,"children":4891},{},[4892,4894,4900,4902,4907],{"type":49,"value":4893},"PostHog\u002Fposthog#54952 — threads ",{"type":43,"tag":79,"props":4895,"children":4897},{"className":4896},[],[4898],{"type":49,"value":4899},"trigger_session_id",{"type":49,"value":4901}," through to ",{"type":43,"tag":79,"props":4903,"children":4905},{"className":4904},[],[4906],{"type":49,"value":258},{"type":49,"value":4908}," events on the session summary backend",{"type":43,"tag":99,"props":4910,"children":4911},{},[4912,4914,4919,4921,4926],{"type":49,"value":4913},"(Followup PR — threads ",{"type":43,"tag":79,"props":4915,"children":4917},{"className":4916},[],[4918],{"type":49,"value":429},{"type":49,"value":4920}," onto ",{"type":43,"tag":79,"props":4922,"children":4924},{"className":4923},[],[4925],{"type":49,"value":473},{"type":49,"value":4927}," events specifically)",{"type":43,"tag":88,"props":4929,"children":4931},{"id":4930},"worked-example-b-pattern-b-posthog-ai-in-error-tracking-mode",[4932],{"type":49,"value":4933},"Worked example B (Pattern B): PostHog AI in error tracking mode",{"type":43,"tag":52,"props":4935,"children":4936},{},[4937,4939,4945,4947,4952,4954,4960],{"type":49,"value":4938},"Pattern: an ",{"type":43,"tag":79,"props":4940,"children":4942},{"className":4941},[],[4943],{"type":49,"value":4944},"agent_mode = 'error_tracking'",{"type":49,"value":4946}," scoped feed streaming to a ",{"type":43,"tag":79,"props":4948,"children":4950},{"className":4949},[],[4951],{"type":49,"value":4827},{"type":49,"value":4953}," channel, answering \"what are users actually trying to DO when they chat with PostHog AI in error tracking mode?\" Mode sizing varies by an order of magnitude or more across agent modes — spot-check volume per §Step 1 before wiring, because a high-volume mode can flood a channel. Opener: ",{"type":43,"tag":79,"props":4955,"children":4957},{"className":4956},[],[4958],{"type":49,"value":4959},"\"A user asked PostHog AI about\"",{"type":49,"value":2153},{"type":43,"tag":52,"props":4962,"children":4963},{},[4964,4966,4971,4972,4977,4978,4983,4985,4991,4993,4999,5001,5007,5009,5015,5017,5023],{"type":49,"value":4965},"Enabling PR: PostHog\u002Fposthog#55160 — threads ",{"type":43,"tag":79,"props":4967,"children":4969},{"className":4968},[],[4970],{"type":49,"value":242},{"type":49,"value":244},{"type":43,"tag":79,"props":4973,"children":4975},{"className":4974},[],[4976],{"type":49,"value":250},{"type":49,"value":252},{"type":43,"tag":79,"props":4979,"children":4981},{"className":4980},[],[4982],{"type":49,"value":258},{"type":49,"value":4984}," emitted by the chat agent loop. Wiring lives in ",{"type":43,"tag":79,"props":4986,"children":4988},{"className":4987},[],[4989],{"type":49,"value":4990},"ee\u002Fhogai\u002Fcore\u002Fagent_modes\u002Fexecutables.py",{"type":49,"value":4992}," (",{"type":43,"tag":79,"props":4994,"children":4996},{"className":4995},[],[4997],{"type":49,"value":4998},"AgentExecutable._get_model",{"type":49,"value":5000},") and passes the dict through the existing ",{"type":43,"tag":79,"props":5002,"children":5004},{"className":5003},[],[5005],{"type":49,"value":5006},"posthog_properties",{"type":49,"value":5008}," field on ",{"type":43,"tag":79,"props":5010,"children":5012},{"className":5011},[],[5013],{"type":49,"value":5014},"MaxChatMixin",{"type":49,"value":5016}," in ",{"type":43,"tag":79,"props":5018,"children":5020},{"className":5019},[],[5021],{"type":49,"value":5022},"ee\u002Fhogai\u002Fllm.py",{"type":49,"value":5024},". Before this PR, scoping a PostHog AI eval to a specific mode wasn't possible — you'd end up evaluating every PostHog AI generation, which produced noisy feeds with low single-digit PASS rates.",{"type":43,"tag":52,"props":5026,"children":5027},{},[5028,5030,5035,5037,5043],{"type":49,"value":5029},"Key observation from setup: the ",{"type":43,"tag":79,"props":5031,"children":5033},{"className":5032},[],[5034],{"type":49,"value":242},{"type":49,"value":5036}," tag reflects the mode at turn-time, but chat state retains mode selection even when users drift off-topic mid-conversation. Spot-check: a random ",{"type":43,"tag":79,"props":5038,"children":5040},{"className":5039},[],[5041],{"type":49,"value":5042},"agent_mode=error_tracking",{"type":49,"value":5044}," sample included a conversation that ended up being about session replay pricing. The eval prompt's classification must be permissive about topic drift — PASS only when the turn is recognizably in-scope for the mode, FAIL when the conversation has drifted to something else entirely.",{"type":43,"tag":88,"props":5046,"children":5048},{"id":5047},"validating-signal-quality-after-launch",[5049],{"type":49,"value":5050},"Validating signal quality after launch",{"type":43,"tag":52,"props":5052,"children":5053},{},[5054],{"type":49,"value":5055},"Once the feed has been running for a day or two, sanity-check the eval output at scale.",{"type":43,"tag":52,"props":5057,"children":5058},{},[5059],{"type":43,"tag":63,"props":5060,"children":5061},{},[5062,5063,5068],{"type":49,"value":513},{"type":43,"tag":79,"props":5064,"children":5066},{"className":5065},[],[5067],{"type":49,"value":684},{"type":49,"value":2901},{"type":43,"tag":768,"props":5070,"children":5072},{"className":1109,"code":5071,"language":1111,"meta":773,"style":773},"posthog:llma-evaluation-summary-create\n{\n  \"evaluation_id\": \"\u003Cuuid>\",\n  \"filter\": \"fail\"\n}\n",[5073],{"type":43,"tag":79,"props":5074,"children":5075},{"__ignoreMap":773},[5076,5084,5091,5127,5160],{"type":43,"tag":105,"props":5077,"children":5078},{"class":780,"line":781},[5079],{"type":43,"tag":105,"props":5080,"children":5081},{"style":1121},[5082],{"type":49,"value":5083},"posthog:llma-evaluation-summary-create\n",{"type":43,"tag":105,"props":5085,"children":5086},{"class":780,"line":790},[5087],{"type":43,"tag":105,"props":5088,"children":5089},{"style":1130},[5090],{"type":49,"value":1133},{"type":43,"tag":105,"props":5092,"children":5093},{"class":780,"line":799},[5094,5098,5103,5107,5111,5115,5119,5123],{"type":43,"tag":105,"props":5095,"children":5096},{"style":1130},[5097],{"type":49,"value":1141},{"type":43,"tag":105,"props":5099,"children":5100},{"style":1144},[5101],{"type":49,"value":5102},"evaluation_id",{"type":43,"tag":105,"props":5104,"children":5105},{"style":1130},[5106],{"type":49,"value":1152},{"type":43,"tag":105,"props":5108,"children":5109},{"style":1130},[5110],{"type":49,"value":1157},{"type":43,"tag":105,"props":5112,"children":5113},{"style":1130},[5114],{"type":49,"value":1175},{"type":43,"tag":105,"props":5116,"children":5117},{"style":1196},[5118],{"type":49,"value":3221},{"type":43,"tag":105,"props":5120,"children":5121},{"style":1130},[5122],{"type":49,"value":1152},{"type":43,"tag":105,"props":5124,"children":5125},{"style":1130},[5126],{"type":49,"value":1349},{"type":43,"tag":105,"props":5128,"children":5129},{"class":780,"line":30},[5130,5134,5139,5143,5147,5151,5156],{"type":43,"tag":105,"props":5131,"children":5132},{"style":1130},[5133],{"type":49,"value":1141},{"type":43,"tag":105,"props":5135,"children":5136},{"style":1144},[5137],{"type":49,"value":5138},"filter",{"type":43,"tag":105,"props":5140,"children":5141},{"style":1130},[5142],{"type":49,"value":1152},{"type":43,"tag":105,"props":5144,"children":5145},{"style":1130},[5146],{"type":49,"value":1157},{"type":43,"tag":105,"props":5148,"children":5149},{"style":1130},[5150],{"type":49,"value":1175},{"type":43,"tag":105,"props":5152,"children":5153},{"style":1196},[5154],{"type":49,"value":5155},"fail",{"type":43,"tag":105,"props":5157,"children":5158},{"style":1130},[5159],{"type":49,"value":2398},{"type":43,"tag":105,"props":5161,"children":5162},{"class":780,"line":816},[5163],{"type":43,"tag":105,"props":5164,"children":5165},{"style":1130},[5166],{"type":49,"value":1442},{"type":43,"tag":52,"props":5168,"children":5169},{},[5170,5174],{"type":43,"tag":63,"props":5171,"children":5172},{},[5173],{"type":49,"value":3045},{"type":49,"value":5175}," open the eval in AI observability → Evaluations → \"Summarize results\" button, filter = fail.",{"type":43,"tag":52,"props":5177,"children":5178},{},[5179],{"type":49,"value":5180},"If the FAIL bucket is large, the classification step is too strict — relax it. If the PASS bucket has lots of generic reasonings, iterate on the prompt to enforce concreteness. The summary tool gives a quick read on this without you having to scroll through individual events.",{"type":43,"tag":52,"props":5182,"children":5183},{},[5184,5186,5191,5193,5198],{"type":49,"value":5185},"Spot-check raw events when needed (note: the stored result value is ",{"type":43,"tag":79,"props":5187,"children":5189},{"className":5188},[],[5190],{"type":49,"value":3472},{"type":49,"value":5192},", not ",{"type":43,"tag":79,"props":5194,"children":5196},{"className":5195},[],[5197],{"type":49,"value":3494},{"type":49,"value":5199}," — see step 6):",{"type":43,"tag":768,"props":5201,"children":5203},{"className":770,"code":5202,"language":772,"meta":773,"style":773},"SELECT\n    properties.$ai_evaluation_reasoning AS reasoning,\n    properties.$ai_trace_id AS trace_id,\n    timestamp\nFROM events\nWHERE event = '$ai_evaluation'\n    AND properties.$ai_evaluation_name = '\u003Cyour eval name>'\n    AND properties.$ai_evaluation_result = 'True'\n    AND timestamp > now() - INTERVAL 1 DAY\nORDER BY timestamp DESC\nLIMIT 25\n",[5204],{"type":43,"tag":79,"props":5205,"children":5206},{"__ignoreMap":773},[5207,5214,5222,5230,5238,5245,5252,5260,5268,5276,5284],{"type":43,"tag":105,"props":5208,"children":5209},{"class":780,"line":781},[5210],{"type":43,"tag":105,"props":5211,"children":5212},{},[5213],{"type":49,"value":787},{"type":43,"tag":105,"props":5215,"children":5216},{"class":780,"line":790},[5217],{"type":43,"tag":105,"props":5218,"children":5219},{},[5220],{"type":49,"value":5221},"    properties.$ai_evaluation_reasoning AS reasoning,\n",{"type":43,"tag":105,"props":5223,"children":5224},{"class":780,"line":799},[5225],{"type":43,"tag":105,"props":5226,"children":5227},{},[5228],{"type":49,"value":5229},"    properties.$ai_trace_id AS trace_id,\n",{"type":43,"tag":105,"props":5231,"children":5232},{"class":780,"line":30},[5233],{"type":43,"tag":105,"props":5234,"children":5235},{},[5236],{"type":49,"value":5237},"    timestamp\n",{"type":43,"tag":105,"props":5239,"children":5240},{"class":780,"line":816},[5241],{"type":43,"tag":105,"props":5242,"children":5243},{},[5244],{"type":49,"value":822},{"type":43,"tag":105,"props":5246,"children":5247},{"class":780,"line":825},[5248],{"type":43,"tag":105,"props":5249,"children":5250},{},[5251],{"type":49,"value":3562},{"type":43,"tag":105,"props":5253,"children":5254},{"class":780,"line":834},[5255],{"type":43,"tag":105,"props":5256,"children":5257},{},[5258],{"type":49,"value":5259},"    AND properties.$ai_evaluation_name = '\u003Cyour eval name>'\n",{"type":43,"tag":105,"props":5261,"children":5262},{"class":780,"line":843},[5263],{"type":43,"tag":105,"props":5264,"children":5265},{},[5266],{"type":49,"value":5267},"    AND properties.$ai_evaluation_result = 'True'\n",{"type":43,"tag":105,"props":5269,"children":5270},{"class":780,"line":852},[5271],{"type":43,"tag":105,"props":5272,"children":5273},{},[5274],{"type":49,"value":5275},"    AND timestamp > now() - INTERVAL 1 DAY\n",{"type":43,"tag":105,"props":5277,"children":5278},{"class":780,"line":861},[5279],{"type":43,"tag":105,"props":5280,"children":5281},{},[5282],{"type":49,"value":5283},"ORDER BY timestamp DESC\n",{"type":43,"tag":105,"props":5285,"children":5286},{"class":780,"line":870},[5287],{"type":43,"tag":105,"props":5288,"children":5289},{},[5290],{"type":49,"value":876},{"type":43,"tag":88,"props":5292,"children":5294},{"id":5293},"tips",[5295],{"type":49,"value":5296},"Tips",{"type":43,"tag":95,"props":5298,"children":5299},{},[5300,5305,5316,5343,5348,5353],{"type":43,"tag":99,"props":5301,"children":5302},{},[5303],{"type":49,"value":5304},"The reasoning field IS the Slack message — design the prompt for that, not for \"chain of thought before classification.\" Models can produce structured Slack-ready text in one pass.",{"type":43,"tag":99,"props":5306,"children":5307},{},[5308,5310,5315],{"type":49,"value":5309},"LLM judges are non-deterministic across reruns. Expect 1-5% noise even with a fixed prompt and model. If you need reproducibility, pin a deterministic provider\u002Fseed in ",{"type":43,"tag":79,"props":5311,"children":5313},{"className":5312},[],[5314],{"type":49,"value":2505},{"type":49,"value":2153},{"type":43,"tag":99,"props":5317,"children":5318},{},[5319,5321,5327,5329,5334,5336,5341],{"type":49,"value":5320},"Keep the eval scoped tightly via ",{"type":43,"tag":79,"props":5322,"children":5324},{"className":5323},[],[5325],{"type":49,"value":5326},"conditions.filters",{"type":49,"value":5328}," on ",{"type":43,"tag":79,"props":5330,"children":5332},{"className":5331},[],[5333],{"type":49,"value":193},{"type":49,"value":5335}," prefix. Otherwise it fans out to every ",{"type":43,"tag":79,"props":5337,"children":5339},{"className":5338},[],[5340],{"type":49,"value":258},{"type":49,"value":5342}," event in the project and burns LLM cost.",{"type":43,"tag":99,"props":5344,"children":5345},{},[5346],{"type":49,"value":5347},"For high-volume features (>10k traces\u002Fweek), consider sampling — set the eval to run on a percentage of matching events rather than all of them. Slack flooding is a real failure mode.",{"type":43,"tag":99,"props":5349,"children":5350},{},[5351],{"type":49,"value":5352},"The \"View Trigger Session\" button is the highest-value link in the alert. Without it, the feed is just text — you can't watch what the user was actually doing. Verify it works in step 7 before considering the feed shipped.",{"type":43,"tag":99,"props":5354,"children":5355},{},[5356,5358,5364],{"type":49,"value":5357},"Once the feed is live, periodically re-run the eval summary tool with ",{"type":43,"tag":79,"props":5359,"children":5361},{"className":5360},[],[5362],{"type":49,"value":5363},"filter: \"pass\"",{"type":49,"value":5365}," to surface the dominant use case clusters. That's how you turn the feed into actual product insights instead of just a notification stream.",{"type":43,"tag":5367,"props":5368,"children":5369},"style",{},[5370],{"type":49,"value":5371},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":5373,"total":5542},[5374,5391,5403,5416,5429,5444,5460,5477,5491,5506,5516,5532],{"slug":5375,"name":5375,"fn":5376,"description":5377,"org":5378,"tags":5379,"stars":5388,"repoUrl":5389,"updatedAt":5390},"analyzing-expensive-users","analyze expensive users in AI observability","Analyze the most expensive users in AI observability and explain why they cost so much. Use when the user asks about top spenders, expensive users, per-user LLM cost, user-level cost drivers, or patterns behind high AI observability spend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5380,5381,5384,5387],{"name":24,"slug":25,"type":13},{"name":5382,"slug":5383,"type":13},"Cost Optimization","cost-optimization",{"name":5385,"slug":5386,"type":13},"Observability","observability",{"name":9,"slug":8,"type":13},35568,"https:\u002F\u002Fgithub.com\u002FPostHog\u002Fposthog","2026-07-28T05:34:11.117757",{"slug":5392,"name":5392,"fn":5393,"description":5394,"org":5395,"tags":5396,"stars":5388,"repoUrl":5389,"updatedAt":5402},"auditing-endpoints","audit PostHog project endpoints","Audit every endpoint in a PostHog project for staleness, failed materialisations, and unused materialised versions. Use when the user asks \"what endpoints can I clean up?\", \"are any of my endpoints broken?\", \"which materialised versions are still being called?\", or wants a one-shot cleanup pass over the Endpoints product. Produces a prioritised report grouped by issue type, with recommended actions but does not modify anything without explicit confirmation.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5397,5398,5401],{"name":24,"slug":25,"type":13},{"name":5399,"slug":5400,"type":13},"Audit","audit",{"name":9,"slug":8,"type":13},"2026-06-08T08:08:33.693989",{"slug":5404,"name":5404,"fn":5405,"description":5406,"org":5407,"tags":5408,"stars":5388,"repoUrl":5389,"updatedAt":5415},"auditing-warehouse-source-health","audit PostHog data warehouse source health","Audit the health of a PostHog project's data warehouse sources and syncs — find every broken or degraded source connection, sync schema, and webhook channel. Use when the user asks \"why are my imports failing?\", \"what's broken with my sources?\", \"why is my warehouse data stale?\", or wants a one-shot triage of source\u002Fsync health before deciding where to dig in. Produces a prioritized report grouped by severity, with recommended next steps. For materialized-view health use `auditing-warehouse-view-health`; for a single failing sync use `diagnosing-failed-warehouse-syncs`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5409,5410,5413,5414],{"name":5399,"slug":5400,"type":13},{"name":5411,"slug":5412,"type":13},"Data Warehouse","data-warehouse",{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-06-18T08:22:57.67984",{"slug":5417,"name":5417,"fn":5418,"description":5419,"org":5420,"tags":5421,"stars":5388,"repoUrl":5389,"updatedAt":5428},"auditing-warehouse-view-health","audit PostHog materialized view health","Audit the health of a PostHog project's materialized views (saved queries) — find every failed materialization and flag unused or stale materialized views that cost storage and compute. Use when the user asks \"which of my views are broken?\", \"why is this materialized view failing?\", \"are any of my views wasting compute?\", or wants a one-shot triage of view health. For source\u002Fsync health use `auditing-warehouse-source-health`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5422,5423,5424,5427],{"name":5399,"slug":5400,"type":13},{"name":5411,"slug":5412,"type":13},{"name":5425,"slug":5426,"type":13},"Performance","performance",{"name":9,"slug":8,"type":13},"2026-06-18T08:25:10.936787",{"slug":5430,"name":5430,"fn":5431,"description":5432,"org":5433,"tags":5434,"stars":5388,"repoUrl":5389,"updatedAt":5443},"authoring-error-tracking-alerts","author PostHog error tracking alerts","Author error tracking alerts that fire when an issue is created, reopened, or starts spiking. Use when the user asks to set up error notifications, route exceptions to Slack\u002Fwebhook\u002FLinear, or evaluate which error events are worth alerting on. Covers trigger-event selection, integration choice, dedup against existing alerts, and shipping with the canonical message body shape.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5435,5438,5441,5442],{"name":5436,"slug":5437,"type":13},"Alerting","alerting",{"name":5439,"slug":5440,"type":13},"Debugging","debugging",{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-06-18T08:24:40.318583",{"slug":5445,"name":5445,"fn":5446,"description":5447,"org":5448,"tags":5449,"stars":5388,"repoUrl":5389,"updatedAt":5459},"authoring-log-alerts","author log alerts in PostHog","Author useful, low-noise log alerts on services in a PostHog project. Use when the user asks to set up alerts for their logs, suggest alerts they should add, or evaluate whether a service is worth monitoring. Covers service triage, baseline characterisation, threshold drafting, back-testing via simulate, and shipping with a notification destination.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5450,5451,5454,5455,5458],{"name":24,"slug":25,"type":13},{"name":5452,"slug":5453,"type":13},"Monitoring","monitoring",{"name":5385,"slug":5386,"type":13},{"name":5456,"slug":5457,"type":13},"Operations","operations",{"name":9,"slug":8,"type":13},"2026-07-18T05:10:54.430898",{"slug":5461,"name":5461,"fn":5462,"description":5463,"org":5464,"tags":5465,"stars":5388,"repoUrl":5389,"updatedAt":5476},"building-workflows","build and edit PostHog workflows","Build, edit, test, enable, and monitor PostHog workflows over MCP. Author the action\u002Fedge graph so it runs and opens cleanly in the visual editor, then change drafts surgically with patch operations. Use when asked to build, set up, automate, change, fix, or debug a workflow, campaign, broadcast, drip sequence, or event-triggered automation in the workflows product.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5466,5469,5472,5473],{"name":5467,"slug":5468,"type":13},"Automation","automation",{"name":5470,"slug":5471,"type":13},"MCP","mcp",{"name":9,"slug":8,"type":13},{"name":5474,"slug":5475,"type":13},"Workflow Automation","workflow-automation","2026-07-28T05:34:12.167015",{"slug":5478,"name":5478,"fn":5479,"description":5480,"org":5481,"tags":5482,"stars":5388,"repoUrl":5389,"updatedAt":5490},"check-posthog-loading","inspect PostHog SDK loading across URLs","Inspect how the PostHog JavaScript SDK is loaded across a list of URLs. Use to confirm consistent installation across pages, find pages missing the snippet, detect mismatched API keys or hosts between pages, and verify the load method (head snippet vs deferred vs array.js).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5483,5484,5485,5488,5489],{"name":24,"slug":25,"type":13},{"name":5439,"slug":5440,"type":13},{"name":5486,"slug":5487,"type":13},"Frontend","frontend",{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-05-07T05:56:19.828048",{"slug":5492,"name":5492,"fn":5493,"description":5494,"org":5495,"tags":5496,"stars":5388,"repoUrl":5389,"updatedAt":5505},"consuming-endpoints-from-client-code","integrate PostHog endpoints into client applications","Wire a PostHog endpoint into a client app or SDK. Covers fetching the OpenAPI spec, generating a typed client with openapi-generator or @hey-api\u002Fopenapi-ts, sending the right auth header, shaping the variables payload (HogQL code_name vs insight breakdown property), handling rate-limit and materialised-endpoint error responses. Use when the user says \"how do I call my endpoint\", \"generate a client for this\", or \"what auth header do I use\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5497,5500,5501,5502],{"name":5498,"slug":5499,"type":13},"API Development","api-development",{"name":5486,"slug":5487,"type":13},{"name":9,"slug":8,"type":13},{"name":5503,"slug":5504,"type":13},"SDK","sdk","2026-06-08T08:08:34.929454",{"slug":5507,"name":5507,"fn":5508,"description":5509,"org":5510,"tags":5511,"stars":5388,"repoUrl":5389,"updatedAt":5515},"copying-endpoints-across-projects","copy PostHog endpoints across projects","Copy a PostHog endpoint (a saved HogQL\u002Finsight query exposed as an API route) to another project in the same organization, or duplicate it under a new name in the same project. Use when the user wants to duplicate an endpoint, promote an endpoint from staging to production, replicate an endpoint's query\u002Fvariables\u002Ffreshness config in another workspace, or clone an endpoint to iterate on it. Unlike feature flags and experiments, endpoints have NO native cross-project copy tool — this skill covers the read-then-recreate flow (endpoint-get then endpoint-create), the active-project switching it requires, name-collision checks, and the safe defaults (land unmaterialised in the target, verify with endpoint-run). Does not cover editing endpoint versions (see managing-endpoint-versions) or authoring a brand-new endpoint from scratch (see creating-an-endpoint).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5512,5513,5514],{"name":5498,"slug":5499,"type":13},{"name":5456,"slug":5457,"type":13},{"name":9,"slug":8,"type":13},"2026-07-15T05:29:58.442727",{"slug":5517,"name":5517,"fn":5518,"description":5519,"org":5520,"tags":5521,"stars":5388,"repoUrl":5389,"updatedAt":5531},"creating-ai-subscription","schedule recurring AI-generated PostHog reports","Create a recurring AI-generated PostHog report — schedule a free-text prompt to run on a cron, with the LLM-synthesized markdown delivered to email or Slack on each tick. Use when the user wants a recurring AI summary of X on any cadence (daily, weekly, monthly, yearly) rather than a one-off report. (To attach an AI summary to an existing insight\u002Fdashboard subscription instead of a free-text prompt, see `managing-subscriptions` and its `summary_enabled` option.)\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5522,5523,5526,5527,5530],{"name":5467,"slug":5468,"type":13},{"name":5524,"slug":5525,"type":13},"Email","email",{"name":9,"slug":8,"type":13},{"name":5528,"slug":5529,"type":13},"Reporting","reporting",{"name":21,"slug":22,"type":13},"2026-06-09T07:32:27.935712",{"slug":5533,"name":5533,"fn":5534,"description":5535,"org":5536,"tags":5537,"stars":5388,"repoUrl":5389,"updatedAt":5541},"creating-an-endpoint","create PostHog API endpoints","Create a PostHog endpoint with the right shape on the first try — covers query kind choice, name conventions, what to expose as variables (HogQL code_name vs insight breakdown), data_freshness_seconds, and whether to materialise on day one. Use when the user says \"create an endpoint\", \"expose this query as an API\", \"turn this insight into an endpoint\", or asks for help structuring a new endpoint. Steers away from common mistakes: materialising a query with cohort breakdowns or compare mode, inline-only variables on a materialised endpoint, unbounded date ranges, ambiguous names.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5538,5539,5540],{"name":24,"slug":25,"type":13},{"name":5498,"slug":5499,"type":13},{"name":9,"slug":8,"type":13},"2026-06-08T08:08:29.624498",231,{"items":5544,"total":5647},[5545,5562,5578,5592,5608,5620,5631],{"slug":5546,"name":5546,"fn":5547,"description":5548,"org":5549,"tags":5550,"stars":26,"repoUrl":27,"updatedAt":5561},"account-handover","draft sales account handover notes","Draft structured handover notes for transitioning a PostHog account from one TAM or CSM to another. Use this skill when a TAM needs to hand over an account, prepare a transition briefing, write handover notes, create an account summary for a new owner, or any request involving account transitions between TAMs or CSMs. Triggers on \"hand over this account\", \"transition account to\", \"draft handover notes\", \"account briefing for new TAM\", \"prepare account transition\", or when a TAM names an account and says they're leaving or reassigning it.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5551,5554,5557,5558],{"name":5552,"slug":5553,"type":13},"Communications","communications",{"name":5555,"slug":5556,"type":13},"CRM","crm",{"name":9,"slug":8,"type":13},{"name":5559,"slug":5560,"type":13},"Sales","sales","2026-04-16T05:13:00.172732",{"slug":5563,"name":5563,"fn":5564,"description":5565,"org":5566,"tags":5567,"stars":26,"repoUrl":27,"updatedAt":5577},"auditing-warehouse-data-health","audit PostHog data warehouse health","Audit the health of a PostHog project's data warehouse — find every broken or degraded pipeline item across sources, sync schemas, materialized views, batch exports, and transformations. Use when the user asks \"what's broken in my warehouse?\", \"give me a health check\", \"audit my data pipeline\", \"why are some dashboards stale?\", or wants a one-shot triage summary before deciding where to spend time. Produces a prioritized report of issues grouped by severity and type, with recommended next steps.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5568,5569,5572,5575,5576],{"name":5399,"slug":5400,"type":13},{"name":5570,"slug":5571,"type":13},"Data Engineering","data-engineering",{"name":5573,"slug":5574,"type":13},"Data Quality","data-quality",{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-06-21T08:19:05.85849",{"slug":5579,"name":5579,"fn":5580,"description":5581,"org":5582,"tags":5583,"stars":26,"repoUrl":27,"updatedAt":5591},"copying-flags-across-projects","copy feature flags across PostHog projects","Copy a feature flag from one PostHog project to one or more target projects in the same organization. Use when the user wants to duplicate a flag, promote a flag from staging to production, sync flags across projects, or replicate a flag configuration in a different workspace. Covers cohort remapping, scheduled-change handling, encrypted payloads, and the safe defaults (disabled in target, no scheduled changes).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5584,5587,5590],{"name":5585,"slug":5586,"type":13},"Deployment","deployment",{"name":5588,"slug":5589,"type":13},"Feature Flags","feature-flags",{"name":9,"slug":8,"type":13},"2026-05-04T05:56:44.484909",{"slug":5593,"name":5593,"fn":5594,"description":5595,"org":5596,"tags":5597,"stars":26,"repoUrl":27,"updatedAt":5607},"diagnosing-experiment-results","diagnose PostHog experiment results and anomalies","Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers empty \u002F 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A\u002FA, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, and surprises after mid-run edits. Symptom-driven dispatch to the right diagnostic.\nTRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange \u002F wrong \u002F off, sees significance flipping, notices PostHog numbers disagreeing with their SQL, sees an A\u002FA test showing significance, or reports surprises after mid-run edits.\nDO NOT TRIGGER when: creating a new experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5598,5601,5602,5605,5606],{"name":5599,"slug":5600,"type":13},"A\u002FB Testing","a-b-testing",{"name":24,"slug":25,"type":13},{"name":5603,"slug":5604,"type":13},"Data Analysis","data-analysis",{"name":5439,"slug":5440,"type":13},{"name":9,"slug":8,"type":13},"2026-05-22T06:59:58.103867",{"slug":5609,"name":5609,"fn":5610,"description":5611,"org":5612,"tags":5613,"stars":26,"repoUrl":27,"updatedAt":5619},"diagnosing-missing-recordings","diagnose missing PostHog session recordings","Diagnoses why a session recording is missing or was not captured. Use when a user asks why a session has no replay, why recordings aren't appearing, or wants to troubleshoot session replay capture issues for a specific session ID or across their project. Covers SDK diagnostic signals, project settings, sampling, triggers, ad blockers, and quota\u002Fbilling scenarios.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5614,5615,5616,5617,5618],{"name":24,"slug":25,"type":13},{"name":5439,"slug":5440,"type":13},{"name":5486,"slug":5487,"type":13},{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-04-22T05:06:51.989772",{"slug":5621,"name":5621,"fn":5622,"description":5623,"org":5624,"tags":5625,"stars":26,"repoUrl":27,"updatedAt":5630},"diagnosing-sdk-health","diagnose PostHog SDK health","Diagnoses the health of a project's PostHog SDK integrations — which SDKs are out of date and how to fix them. Use when a user asks about PostHog SDK versions, outdated SDKs, upgrade recommendations, \"SDK health\", \"SDK doctor\" (the former name), or when events or features seem off and it might be due to an old SDK.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5626,5627,5628,5629],{"name":24,"slug":25,"type":13},{"name":5439,"slug":5440,"type":13},{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-04-27T05:46:14.554016",{"slug":5632,"name":5632,"fn":5633,"description":5634,"org":5635,"tags":5636,"stars":26,"repoUrl":27,"updatedAt":5646},"error-tracking-android","track Android errors with PostHog","PostHog error tracking for Android",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5637,5640,5641,5644,5645],{"name":5638,"slug":5639,"type":13},"Android","android",{"name":5439,"slug":5440,"type":13},{"name":5642,"slug":5643,"type":13},"Mobile","mobile",{"name":5385,"slug":5386,"type":13},{"name":9,"slug":8,"type":13},"2026-04-06T18:46:26.982494",110]