[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-k6-trend-analysis":3,"mdc-6vxex9-key":33,"related-repo-grafana-k6-trend-analysis":3260,"related-org-grafana-k6-trend-analysis":3377},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"k6-trend-analysis","analyze k6 performance test trends","Analyze Grafana Cloud k6 test run trends over time. Detects slow metric drift (e.g., P95 latency creeping up while still passing thresholds), computes headroom to thresholds, flags anomalies, and recommends threshold tightening. Use when the user asks about test performance trends, wants to know if metrics are degrading, asks whether thresholds should be tightened, or wants a health check across recent runs for a specific test. Trigger on phrases like \"how is my test trending\", \"is P95 getting worse\", \"check for performance regression\", \"should I tighten thresholds\", \"are my tests degrading\", \"show me trends for test X\", \"analyze my k6 test runs\", or \"is my test getting slower\". Also trigger when a user asks to check all tests in a project -- run this skill once per test and synthesize.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Monitoring","monitoring",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Analytics","analytics",189,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills","2026-07-18T05:48:29.35871",null,16,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fgrafana-k6\u002Fk6-trend-analysis","---\nname: k6-trend-analysis\ndescription: >\n  Analyze Grafana Cloud k6 test run trends over time. Detects slow metric drift\n  (e.g., P95 latency creeping up while still passing thresholds), computes\n  headroom to thresholds, flags anomalies, and recommends threshold tightening.\n  Use when the user asks about test performance trends, wants to know if metrics\n  are degrading, asks whether thresholds should be tightened, or wants a health\n  check across recent runs for a specific test. Trigger on phrases like \"how is\n  my test trending\", \"is P95 getting worse\", \"check for performance regression\",\n  \"should I tighten thresholds\", \"are my tests degrading\", \"show me trends for\n  test X\", \"analyze my k6 test runs\", or \"is my test getting slower\". Also\n  trigger when a user asks to check all tests in a project -- run this skill\n  once per test and synthesize.\n---\n\n# k6 Trend Analysis\n\nAnalyze metric trends across multiple runs of a Grafana Cloud k6 test to catch\ndegradation early -- before thresholds breach and alerts fire. A P95 at 380ms\nagainst a 500ms threshold is \"green\" today, but if it was 250ms a month ago,\nsomething is quietly degrading; this skill surfaces that drift and recommends action.\n\n## What this skill does NOT do\n\n- **Deep-dive into a single run's failure**: load `k6-cloud-investigate-test`\n- **Edit scripts or apply threshold changes**: load `k6-test-maintenance`\n- **Create new test scripts**: use the appropriate test creation workflow\n- **Query service-side metrics directly**: this skill hands off to\n  `debug-with-grafana` when observability correlation is needed\n\n## Dependencies\n\nThis skill delegates all GCk6 API mechanics to **`k6-manage`**. Read it before\nexecuting any API call -- it covers auth, path construction (the doubled\n`cloud\u002Fcloud\u002F` prefix), pagination with `@nextLink`, the spill envelope, and\nmetric query syntax. Do not duplicate that knowledge here.\n\nTools used: **`gcx`** (via k6-manage patterns).\n\n---\n\n## Workflow\n\nFollow these steps in order. Present findings at the end -- do not apply changes.\n\n### Step 1: Identify the test\n\nThe user provides one of:\n- A GCk6 URL (extract the load test ID from the path)\n- A test ID directly\n- A test name (search via `gcx k6 tests list` or the v6 API)\n\nConfirm the test exists by fetching its metadata. Record the `id`, `name`,\n`project_id`, and `created` timestamp. You will need the load test ID (not a\nrun ID) for the multi-run metric endpoints.\n\n### Step 2: Determine the analysis window\n\nDefault to **last 30 days**. Adjust if:\n- The user requests a specific window\n- The test has very few runs (\u003C5 in 30 days) -- widen the window and note this\n- The test runs very frequently (hundreds of runs in 30 days) -- consider\n  sampling or narrowing. Ask the user if the volume is extreme (>200 runs)\n\nState the window explicitly: \"Analyzing runs from {start_date} to {end_date}.\"\n\n### Step 3: Fetch runs in the window\n\nList all runs for the test using the v6 API with `$orderby=created desc`\npagination (see k6-manage Section 3 for the `@nextLink` loop pattern). Filter\nto runs within the analysis window.\n\nFor each run, record:\n- `id`, `created`, `ended`\n- `result` (passed\u002Ffailed\u002Ftimed_out)\n- `status` (created\u002Fqueued\u002Finitializing\u002Frunning\u002Ffinished\u002Faborted)\n- `note` (if present -- users sometimes annotate runs)\n\nDiscard runs that did not reach `finished` status (aborted, timed_out, etc.)\nunless the user specifically asks about them -- incomplete runs produce\nunreliable metric aggregates.\n\nCount the runs. If fewer than 3 usable runs exist, inform the user that trend\nanalysis is not meaningful with this sample size and suggest widening the window\nor waiting for more runs.\n\n### Step 4: Fetch all metrics and their types\n\nBefore querying values, discover what metrics the test emits. Use the multi-run\nmetric listing endpoint (k6-manage references\u002Fmetrics.md Section 2):\n\n```\nGET \u002Fcloud\u002Fv5\u002Fload_tests\u002F{loadTestId}\u002Fmetrics(test_run_ids=[{id1},{id2},...])\n```\n\nPass a representative subset of run IDs (the first and last few) to catch\nmetrics that may have been added or removed over the window. Record each\nmetric's `name` and `type` (counter, gauge, trend, rate).\n\nGroup metrics by type -- the query method must match the metric type (see\nmetrics.md \"Query methods\"). Using the wrong method returns empty results.\n\n### Step 5: Fetch per-run aggregate values\n\nFor each metric, query its aggregate value across all runs in the window using\nthe multi-run aggregate endpoint (metrics.md Section 8):\n\n```\nGET \u002Fcloud\u002Fv5\u002Fload_tests\u002F{loadTestId}\u002Fquery_aggregate_k6(\n  query='\u003Cmethod>',\n  metric='\u003Cmetric_name>',\n  test_run_ids=[{id1},{id2},...]\n)\n```\n\nConcrete `gcx` form (proxy prefix per k6-manage §2; `-o json` avoids the spill envelope):\n\n```bash\nLT=\u003Cload_test_id>; IDS=\"123,124,125\"\ngcx --context \u003Cctx> api \"\u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv5\u002Fload_tests\u002F$LT\u002Fquery_aggregate_k6(query='histogram_quantile(0.95)',metric='http_req_duration',test_run_ids=[$IDS])\" -o json\n```\n\nChoose the aggregate method based on metric type:\n\n| Metric type | Primary method | What it captures |\n|-------------|---------------|------------------|\n| **trend** | `histogram_quantile(0.95)` | P95 latency -- the most common SLO target |\n| **trend** | `histogram_quantile(0.5)` | Median -- shows typical behavior |\n| **trend** | `histogram_avg` | Mean -- sensitive to outliers |\n| **counter** | `increase` | Total count per run |\n| **rate** | `ratio` | Success\u002Ffailure ratio |\n| **gauge** | `max` | Peak value per run |\n\nFor trend-type metrics (latencies), query multiple quantiles (P50, P90, P95,\nP99) to see if degradation is uniform or concentrated in the tail.\n\n**Always break down by request grouping for multi-target tests.** A single\naggregate p95 across a whole test obscures regressions confined to one\nendpoint or one page -- a 3x slowdown on one URL can be invisible at the\ntest-level p95 if the test hits many URLs. Default to grouped queries:\n\n| Metric pattern | Default grouping | Why |\n|----------------|------------------|-----|\n| `browser_web_vital_*` (LCP, FCP, CLS, TTFB, INP, FID) | `by (url)` | Each navigation emits its own web vital; per-URL trends pin regressions to a specific page. |\n| `http_req_duration`, `http_req_failed`, `http_reqs` | `by (name, status)` if requests are tagged with `name`; otherwise `by (url, status)` | Different endpoints have different baselines; mixing them hides per-endpoint drift. |\n| `iteration_duration` (multi-scenario tests) | `by (scenario)` | Browser and protocol scenarios have very different durations; mixing them is meaningless. |\n| Custom trends with tags | `by (\u003Cthe tag>)` | Whatever the user tagged on is presumably what they care about. |\n\nDiscover available labels first via the labels endpoint (metrics.md §4) and\nlabel-values endpoint (metrics.md §5) before constructing the grouped query.\nDon't assume labels from reading the script -- a tag rename or refactor can\nsilently shift the label space. Example:\n\n```\n# List labels for a specific metric on a representative run\nGET \u002Fcloud\u002Fv5\u002Ftest_runs\u002F{id}\u002Flabels?match[]=browser_web_vital_lcp\n\n# Then enumerate values for a useful label\nGET \u002Fcloud\u002Fv5\u002Ftest_runs\u002F{id}\u002Flabel\u002Furl\u002Fvalues\n```\n\nOnly fall back to the bare aggregate (no `by`) for metrics where grouping\nadds no information -- e.g., `vus`, `load_generator_cpu_percent`,\nsingle-target tests where every request hits the same URL.\n\nThe response includes `test_run_id` as a label -- use this to map each value\nback to its run timestamp from Step 3. Grouped queries return one series per\n(group, run) combination; flatten into a tidy \"rid x group\" table for the\ntrend computation in Step 7.\n\nIf there are too many run IDs to fit in a single URL (hundreds), batch the\nqueries into groups of 50 run IDs and merge the results.\n\n### Step 6: Extract thresholds\n\nThresholds define what \"passing\" means. Fetch the test's current script (via\nk6-manage Section 5) and parse the `export const options = { thresholds: {...} }`\nblock. Record each threshold's:\n- Metric name and selector (e.g., `http_req_duration{name:homepage}`)\n- Condition (e.g., `p(95)\u003C500`)\n- Whether `abortOnFail` is set\n\nAlso check the most recent run's threshold results from the run data to see\nwhich thresholds are currently passing vs. failing.\n\nNot all metrics will have thresholds -- that's fine. Metrics without thresholds\nstill get trend analysis; they just won't have headroom calculations.\n\n### Step 7: Compute trends\n\n**Detect inflection points and rule out script changes deterministically.**\nBefore drawing conclusions about a regression, look for discontinuities in\nthe metric values -- sudden jumps or drops that align across multiple\nmetrics on the same date. When you spot one, do not guess whether the\nscript changed. The run-bundled script endpoint gives a deterministic\nanswer in seconds:\n\n1. **sha256-diff the bundled scripts at the boundary.** Fetch the snapshot\n   from the last \"before\" run and the first \"after\" run via\n   `GET \u002Fcloud\u002Fv6\u002Ftest_runs\u002F{id}\u002Fscript` (k6-manage §5, \"Two distinct\n   script endpoints\" -- use the run-scoped endpoint, not the load-test\n   one, since the latter only shows the current version). Compare with\n   `shasum -a 256`.\n\n   ```bash\n   gcx --context \u003Cctx> api \u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv6\u002Ftest_runs\u002F\u003Cbefore_id>\u002Fscript > \u002Ftmp\u002Fbefore.bin\n   gcx --context \u003Cctx> api \u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv6\u002Ftest_runs\u002F\u003Cafter_id>\u002Fscript > \u002Ftmp\u002Fafter.bin\n   shasum -a 256 \u002Ftmp\u002Fbefore.bin \u002Ftmp\u002Fafter.bin\n   ```\n\n   For thoroughness on multi-run inflections, hash every run across the\n   transition -- if N consecutive runs share one hash and N more share\n   another, you have a clean before\u002Fafter boundary. If hashes change\n   mid-stream, the test was edited multiple times.\n\n2. **If the sha256 differs**, the test script changed -- the inflection\n   may be a test-side artifact, not a service regression. Split the\n   analysis into distinct eras at the boundary and compute trends within\n   each era separately. Comparing metrics across script changes produces\n   misleading trends -- a P95 drop from 3,000ms to 150ms is not an\n   \"improvement\" if the script simply stopped hitting a slow endpoint.\n   State the eras explicitly in the report and focus recommendations on\n   the most recent era.\n\n3. **If the sha256 matches**, the script is byte-identical and the\n   regression is external to the test (service-side, infrastructure, or\n   load-zone). This is high-confidence information -- carry it into Step 9\n   (service-side correlation) instead of leaving \"did the test change?\"\n   as an open question.\n\nThis sha256 diff is a 5-second deterministic check that rules out a huge\nclass of causes. Run it at every detected inflection, not just when the\nuser asks.\n\nFor each metric, build a time-ordered series of (run_timestamp, value) pairs.\nThen compute:\n\n**Basic statistics:**\n- Mean, standard deviation, min, max across all runs\n- Current value (most recent run)\n- Baseline value (oldest run in window, or mean of first 3 runs for stability)\n\n**Trend direction:**\nSplit the runs into two halves (first half and second half of the time window).\nCompare the mean of each half:\n- **Degrading**: second-half mean is worse by >10% (higher for latency\u002Ferrors,\n  lower for success rates)\n- **Improving**: second-half mean is better by >10%\n- **Stable**: change is within 10%\n- **Volatile**: standard deviation exceeds 25% of the mean, regardless of\n  direction\n\nThe 10% and 25% thresholds are starting points. If the user's test has very\ntight tolerances or very noisy metrics, adjust and explain the reasoning.\n\n**Rate of change:**\nExpress as percentage change per week: `((recent_mean - baseline_mean) \u002F baseline_mean) * 100 \u002F weeks_in_window`. This normalizes across different window sizes.\n\n**Anomaly detection:**\nFlag any run where the metric value is more than 2 standard deviations from the\noverall mean. These are potential inflection points worth investigating\nindividually.\n\n### Step 8: Headroom analysis\n\nFor metrics that have thresholds defined (from Step 6), compute headroom:\n\n```\nheadroom_pct = ((threshold_value - current_value) \u002F threshold_value) * 100\n```\n\nClassify headroom:\n- **Comfortable** (>50%): well within limits\n- **Adequate** (20-50%): healthy but worth monitoring\n- **Thin** (\u003C20%): at risk of breaching if trends continue\n- **Breached** (\u003C0%): already failing\n\nFor degrading metrics with thin headroom, estimate when the threshold will be\nbreached if the current rate of change continues:\n\n```\nweeks_until_breach = headroom_absolute \u002F rate_of_change_per_week\n```\n\nThis is a rough projection, not a prediction -- present it as \"at the current\nrate of degradation, this metric could breach in approximately N weeks.\"\n\n### Step 9: Service-side correlation (when warranted)\n\nIf Step 7 reveals degradation, offer (don't auto-run) to correlate with\nservice-side data to separate **service degradation** (fix the service),\n**test-environment changes** (load-zone latency, LG exhaustion), and\n**script changes** (a slower edit). Present findings and ask first.\n\nHand off to `debug-with-grafana` (via `gcx`) to query the service's Prometheus\nmetrics and Loki logs for the same window; look for service error-rate changes,\nupstream latency, resource pressure (CPU\u002Fmemory\u002Fpools), and deploys coinciding\nwith inflection points.\n\n**When the service has no observability** (third-party or another team's\nservice -- no Prometheus job, Loki stream, or probe), fall back to client-side\nsignal, which still localises the regression:\n\n- **Browser tests:** compare Tempo iteration traces (k6-manage §7,\n  `\u002Fapi\u002Fv1\u002Ftempo\u002Fapi\u002Fsearch` + `\u002Ftraces\u002F{id}`) for a before\u002Fafter run. The\n  span-name rollup, slowest spans, and per-URL navigation durations pin the\n  regression to a URL, locator action, or asset; web vitals are `web_vital.*`\n  span attributes -- read them directly.\n- **Protocol tests:** compare `http_req_duration by (name, status)` and\n  `http_req_failed by (name, status)`; a regression on one named request\n  points at that endpoint. Split `http_req_waiting` (server time) vs\n  `http_req_receiving` (transfer) to separate slow processing from slow download.\n- **Payload corroboration** (both): query `data_received` \u002F\n  `browser_data_received`. Latency up + payload up -> server content changes;\n  latency up + payload stable -> server processing changes; intermittent\n  payload -> flaky cache or A\u002FB test.\n\nPrefer server-side correlation when available -- it answers \"why\" directly; the\nfallbacks answer \"where\" and \"what kind of change\", and make a good ticket for\nthe owning team.\n\n### Step 10: Present the report\n\nAlways present findings as a structured report. Never apply changes directly.\n\n---\n\n## Report template\n\nUse this structure. Omit sections that don't apply (e.g., skip \"Anomalies\" if\nnone were detected).\n\n```markdown\n# Trend Analysis: {test_name}\n\n**Test ID**: {test_id}\n**Analysis window**: {start_date} to {end_date} ({N} runs analyzed)\n**Overall health**: {Healthy | Watch | Degrading | Critical}\n\n## Run Summary\n\n| Period | Runs | Passed | Failed | Pass Rate |\n|--------|------|--------|--------|-----------|\n| First half | N | N | N | N% |\n| Second half | N | N | N | N% |\n\n## Metric Trends\n\n| Metric | Type | Current | Baseline | Change | Trend | Threshold | Headroom |\n|--------|------|---------|----------|--------|-------|-----------|----------|\n| http_req_duration (P95) | trend | 380ms | 250ms | +52% | Degrading | 500ms | 24% |\n| http_req_failed | rate | 0.8% | 0.3% | +167% | Degrading | 1% | 20% |\n| http_reqs | counter | 15,230 | 15,100 | +0.9% | Stable | - | - |\n\n## Flagged Issues\n\n### 1. {metric_name}: {classification}\n- **Current**: {value} | **Baseline**: {value} | **Change**: {pct}%\n- **Threshold**: {threshold} | **Headroom**: {pct}%\n- **Rate of change**: {pct}% per week\n- **Projected breach**: ~{N} weeks at current rate\n- **Anomalous runs**: {run_ids with dates, if any}\n\n## Threshold Recommendations\n\n| Metric | Current Threshold | Recommended | Rationale |\n|--------|-------------------|-------------|-----------|\n| http_req_duration | p(95)\u003C500 | p(95)\u003C420 | Current P95 is 380ms; tightening to 420ms gives 10% headroom from current performance while surfacing further degradation early |\n\n## Suggested Next Steps\n\n- [ ] **Investigate service side**: P95 latency has increased 52% -- consider\n      loading `debug-with-grafana` to check service health\n- [ ] **Deep-dive run {run_id}**: anomalous P95 spike on {date} -- consider\n      loading `k6-cloud-investigate-test` for this run\n- [ ] **Tighten thresholds**: 2 metrics have >30% headroom that could be\n      tightened -- consider loading `k6-test-maintenance` to apply changes\n```\n\n## Overall health classification\n\nDerive the overall health from the worst-case metric:\n- **Healthy**: all metrics stable or improving, headroom comfortable or adequate\n- **Watch**: at least one metric degrading but headroom still adequate\n- **Degrading**: at least one metric degrading with thin headroom\n- **Critical**: at least one metric has breached its threshold, or multiple\n  metrics are degrading with thin headroom\n\n---\n\n## Threshold recommendations\n\nWhen recommending threshold changes, follow these principles:\n\n- **Only tighten, never loosen** unless the user asks. Loosening thresholds\n  masks problems.\n- **Target 10-20% headroom** above the recent P95 of the metric. Enough room\n  for normal variance but tight enough to catch real degradation.\n- **Use the second-half mean as the baseline**, not the single most recent run\n  (which could be an outlier).\n- **Respect the user's intent**: if thresholds are currently very loose\n  (>100% headroom), they may be intentionally permissive. Mention the\n  opportunity to tighten but don't push hard -- the user knows their context.\n- **Consider grouped thresholds**: if the test uses tag-based thresholds\n  (e.g., `http_req_duration{name:homepage}`), recommend per-endpoint\n  thresholds where the trends differ between endpoints.\n\n---\n\n## Gotchas\n\n| Issue | Detail |\n|-------|--------|\n| **Zero-observation thresholds** | A threshold with zero observations passes by default in k6. If a metric appears to pass but has no data, flag it -- the threshold is not actually being evaluated. |\n| **Metric type changes across runs** | If a metric's type changed between runs (e.g., script refactor), the multi-run aggregate endpoint uses the latest type. Earlier runs queried with the wrong method return empty. Flag this if detected. |\n| **Incomplete runs skew trends** | Aborted or timed-out runs typically have shorter durations and fewer iterations, producing unrepresentative metric values. Exclude them by default. |\n| **LG resource metrics** | `load_generator_cpu_percent` and `load_generator_file_handles` trending up may indicate the test is outgrowing its load generator allocation, not that the service is degrading. Call this out separately. |\n| **Rate metric direction** | For `ratio`-type rate metrics (like check pass rates), \"degrading\" means the value is *decreasing* (fewer passes), which is the opposite direction from latency metrics. |\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,46,52,59,125,131,163,179,183,189,194,201,206,232,269,275,287,305,310,316,336,341,398,411,416,422,427,439,459,464,470,475,484,504,650,655,836,841,851,1011,1016,1025,1053,1066,1071,1077,1090,1131,1136,1141,1147,1157,1384,1389,1394,1402,1420,1430,1473,1478,1496,1506,1512,1517,1526,1531,1574,1579,1588,1593,1599,1625,1644,1654,1759,1764,1770,1775,1778,1784,1789,2989,2995,3000,3042,3045,3051,3056,3116,3119,3125,3254],{"type":39,"tag":40,"props":41,"children":42},"element","h1",{"id":4},[43],{"type":44,"value":45},"text","k6 Trend Analysis",{"type":39,"tag":47,"props":48,"children":49},"p",{},[50],{"type":44,"value":51},"Analyze metric trends across multiple runs of a Grafana Cloud k6 test to catch\ndegradation early -- before thresholds breach and alerts fire. A P95 at 380ms\nagainst a 500ms threshold is \"green\" today, but if it was 250ms a month ago,\nsomething is quietly degrading; this skill surfaces that drift and recommends action.",{"type":39,"tag":53,"props":54,"children":56},"h2",{"id":55},"what-this-skill-does-not-do",[57],{"type":44,"value":58},"What this skill does NOT do",{"type":39,"tag":60,"props":61,"children":62},"ul",{},[63,82,97,107],{"type":39,"tag":64,"props":65,"children":66},"li",{},[67,73,75],{"type":39,"tag":68,"props":69,"children":70},"strong",{},[71],{"type":44,"value":72},"Deep-dive into a single run's failure",{"type":44,"value":74},": load ",{"type":39,"tag":76,"props":77,"children":79},"code",{"className":78},[],[80],{"type":44,"value":81},"k6-cloud-investigate-test",{"type":39,"tag":64,"props":83,"children":84},{},[85,90,91],{"type":39,"tag":68,"props":86,"children":87},{},[88],{"type":44,"value":89},"Edit scripts or apply threshold changes",{"type":44,"value":74},{"type":39,"tag":76,"props":92,"children":94},{"className":93},[],[95],{"type":44,"value":96},"k6-test-maintenance",{"type":39,"tag":64,"props":98,"children":99},{},[100,105],{"type":39,"tag":68,"props":101,"children":102},{},[103],{"type":44,"value":104},"Create new test scripts",{"type":44,"value":106},": use the appropriate test creation workflow",{"type":39,"tag":64,"props":108,"children":109},{},[110,115,117,123],{"type":39,"tag":68,"props":111,"children":112},{},[113],{"type":44,"value":114},"Query service-side metrics directly",{"type":44,"value":116},": this skill hands off to\n",{"type":39,"tag":76,"props":118,"children":120},{"className":119},[],[121],{"type":44,"value":122},"debug-with-grafana",{"type":44,"value":124}," when observability correlation is needed",{"type":39,"tag":53,"props":126,"children":128},{"id":127},"dependencies",[129],{"type":44,"value":130},"Dependencies",{"type":39,"tag":47,"props":132,"children":133},{},[134,136,145,147,153,155,161],{"type":44,"value":135},"This skill delegates all GCk6 API mechanics to ",{"type":39,"tag":68,"props":137,"children":138},{},[139],{"type":39,"tag":76,"props":140,"children":142},{"className":141},[],[143],{"type":44,"value":144},"k6-manage",{"type":44,"value":146},". Read it before\nexecuting any API call -- it covers auth, path construction (the doubled\n",{"type":39,"tag":76,"props":148,"children":150},{"className":149},[],[151],{"type":44,"value":152},"cloud\u002Fcloud\u002F",{"type":44,"value":154}," prefix), pagination with ",{"type":39,"tag":76,"props":156,"children":158},{"className":157},[],[159],{"type":44,"value":160},"@nextLink",{"type":44,"value":162},", the spill envelope, and\nmetric query syntax. Do not duplicate that knowledge here.",{"type":39,"tag":47,"props":164,"children":165},{},[166,168,177],{"type":44,"value":167},"Tools used: ",{"type":39,"tag":68,"props":169,"children":170},{},[171],{"type":39,"tag":76,"props":172,"children":174},{"className":173},[],[175],{"type":44,"value":176},"gcx",{"type":44,"value":178}," (via k6-manage patterns).",{"type":39,"tag":180,"props":181,"children":182},"hr",{},[],{"type":39,"tag":53,"props":184,"children":186},{"id":185},"workflow",[187],{"type":44,"value":188},"Workflow",{"type":39,"tag":47,"props":190,"children":191},{},[192],{"type":44,"value":193},"Follow these steps in order. Present findings at the end -- do not apply changes.",{"type":39,"tag":195,"props":196,"children":198},"h3",{"id":197},"step-1-identify-the-test",[199],{"type":44,"value":200},"Step 1: Identify the test",{"type":39,"tag":47,"props":202,"children":203},{},[204],{"type":44,"value":205},"The user provides one of:",{"type":39,"tag":60,"props":207,"children":208},{},[209,214,219],{"type":39,"tag":64,"props":210,"children":211},{},[212],{"type":44,"value":213},"A GCk6 URL (extract the load test ID from the path)",{"type":39,"tag":64,"props":215,"children":216},{},[217],{"type":44,"value":218},"A test ID directly",{"type":39,"tag":64,"props":220,"children":221},{},[222,224,230],{"type":44,"value":223},"A test name (search via ",{"type":39,"tag":76,"props":225,"children":227},{"className":226},[],[228],{"type":44,"value":229},"gcx k6 tests list",{"type":44,"value":231}," or the v6 API)",{"type":39,"tag":47,"props":233,"children":234},{},[235,237,243,245,251,253,259,261,267],{"type":44,"value":236},"Confirm the test exists by fetching its metadata. Record the ",{"type":39,"tag":76,"props":238,"children":240},{"className":239},[],[241],{"type":44,"value":242},"id",{"type":44,"value":244},", ",{"type":39,"tag":76,"props":246,"children":248},{"className":247},[],[249],{"type":44,"value":250},"name",{"type":44,"value":252},",\n",{"type":39,"tag":76,"props":254,"children":256},{"className":255},[],[257],{"type":44,"value":258},"project_id",{"type":44,"value":260},", and ",{"type":39,"tag":76,"props":262,"children":264},{"className":263},[],[265],{"type":44,"value":266},"created",{"type":44,"value":268}," timestamp. You will need the load test ID (not a\nrun ID) for the multi-run metric endpoints.",{"type":39,"tag":195,"props":270,"children":272},{"id":271},"step-2-determine-the-analysis-window",[273],{"type":44,"value":274},"Step 2: Determine the analysis window",{"type":39,"tag":47,"props":276,"children":277},{},[278,280,285],{"type":44,"value":279},"Default to ",{"type":39,"tag":68,"props":281,"children":282},{},[283],{"type":44,"value":284},"last 30 days",{"type":44,"value":286},". Adjust if:",{"type":39,"tag":60,"props":288,"children":289},{},[290,295,300],{"type":39,"tag":64,"props":291,"children":292},{},[293],{"type":44,"value":294},"The user requests a specific window",{"type":39,"tag":64,"props":296,"children":297},{},[298],{"type":44,"value":299},"The test has very few runs (\u003C5 in 30 days) -- widen the window and note this",{"type":39,"tag":64,"props":301,"children":302},{},[303],{"type":44,"value":304},"The test runs very frequently (hundreds of runs in 30 days) -- consider\nsampling or narrowing. Ask the user if the volume is extreme (>200 runs)",{"type":39,"tag":47,"props":306,"children":307},{},[308],{"type":44,"value":309},"State the window explicitly: \"Analyzing runs from {start_date} to {end_date}.\"",{"type":39,"tag":195,"props":311,"children":313},{"id":312},"step-3-fetch-runs-in-the-window",[314],{"type":44,"value":315},"Step 3: Fetch runs in the window",{"type":39,"tag":47,"props":317,"children":318},{},[319,321,327,329,334],{"type":44,"value":320},"List all runs for the test using the v6 API with ",{"type":39,"tag":76,"props":322,"children":324},{"className":323},[],[325],{"type":44,"value":326},"$orderby=created desc",{"type":44,"value":328},"\npagination (see k6-manage Section 3 for the ",{"type":39,"tag":76,"props":330,"children":332},{"className":331},[],[333],{"type":44,"value":160},{"type":44,"value":335}," loop pattern). Filter\nto runs within the analysis window.",{"type":39,"tag":47,"props":337,"children":338},{},[339],{"type":44,"value":340},"For each run, record:",{"type":39,"tag":60,"props":342,"children":343},{},[344,365,376,387],{"type":39,"tag":64,"props":345,"children":346},{},[347,352,353,358,359],{"type":39,"tag":76,"props":348,"children":350},{"className":349},[],[351],{"type":44,"value":242},{"type":44,"value":244},{"type":39,"tag":76,"props":354,"children":356},{"className":355},[],[357],{"type":44,"value":266},{"type":44,"value":244},{"type":39,"tag":76,"props":360,"children":362},{"className":361},[],[363],{"type":44,"value":364},"ended",{"type":39,"tag":64,"props":366,"children":367},{},[368,374],{"type":39,"tag":76,"props":369,"children":371},{"className":370},[],[372],{"type":44,"value":373},"result",{"type":44,"value":375}," (passed\u002Ffailed\u002Ftimed_out)",{"type":39,"tag":64,"props":377,"children":378},{},[379,385],{"type":39,"tag":76,"props":380,"children":382},{"className":381},[],[383],{"type":44,"value":384},"status",{"type":44,"value":386}," (created\u002Fqueued\u002Finitializing\u002Frunning\u002Ffinished\u002Faborted)",{"type":39,"tag":64,"props":388,"children":389},{},[390,396],{"type":39,"tag":76,"props":391,"children":393},{"className":392},[],[394],{"type":44,"value":395},"note",{"type":44,"value":397}," (if present -- users sometimes annotate runs)",{"type":39,"tag":47,"props":399,"children":400},{},[401,403,409],{"type":44,"value":402},"Discard runs that did not reach ",{"type":39,"tag":76,"props":404,"children":406},{"className":405},[],[407],{"type":44,"value":408},"finished",{"type":44,"value":410}," status (aborted, timed_out, etc.)\nunless the user specifically asks about them -- incomplete runs produce\nunreliable metric aggregates.",{"type":39,"tag":47,"props":412,"children":413},{},[414],{"type":44,"value":415},"Count the runs. If fewer than 3 usable runs exist, inform the user that trend\nanalysis is not meaningful with this sample size and suggest widening the window\nor waiting for more runs.",{"type":39,"tag":195,"props":417,"children":419},{"id":418},"step-4-fetch-all-metrics-and-their-types",[420],{"type":44,"value":421},"Step 4: Fetch all metrics and their types",{"type":39,"tag":47,"props":423,"children":424},{},[425],{"type":44,"value":426},"Before querying values, discover what metrics the test emits. Use the multi-run\nmetric listing endpoint (k6-manage references\u002Fmetrics.md Section 2):",{"type":39,"tag":428,"props":429,"children":433},"pre",{"className":430,"code":432,"language":44},[431],"language-text","GET \u002Fcloud\u002Fv5\u002Fload_tests\u002F{loadTestId}\u002Fmetrics(test_run_ids=[{id1},{id2},...])\n",[434],{"type":39,"tag":76,"props":435,"children":437},{"__ignoreMap":436},"",[438],{"type":44,"value":432},{"type":39,"tag":47,"props":440,"children":441},{},[442,444,449,451,457],{"type":44,"value":443},"Pass a representative subset of run IDs (the first and last few) to catch\nmetrics that may have been added or removed over the window. Record each\nmetric's ",{"type":39,"tag":76,"props":445,"children":447},{"className":446},[],[448],{"type":44,"value":250},{"type":44,"value":450}," and ",{"type":39,"tag":76,"props":452,"children":454},{"className":453},[],[455],{"type":44,"value":456},"type",{"type":44,"value":458}," (counter, gauge, trend, rate).",{"type":39,"tag":47,"props":460,"children":461},{},[462],{"type":44,"value":463},"Group metrics by type -- the query method must match the metric type (see\nmetrics.md \"Query methods\"). Using the wrong method returns empty results.",{"type":39,"tag":195,"props":465,"children":467},{"id":466},"step-5-fetch-per-run-aggregate-values",[468],{"type":44,"value":469},"Step 5: Fetch per-run aggregate values",{"type":39,"tag":47,"props":471,"children":472},{},[473],{"type":44,"value":474},"For each metric, query its aggregate value across all runs in the window using\nthe multi-run aggregate endpoint (metrics.md Section 8):",{"type":39,"tag":428,"props":476,"children":479},{"className":477,"code":478,"language":44},[431],"GET \u002Fcloud\u002Fv5\u002Fload_tests\u002F{loadTestId}\u002Fquery_aggregate_k6(\n  query='\u003Cmethod>',\n  metric='\u003Cmetric_name>',\n  test_run_ids=[{id1},{id2},...]\n)\n",[480],{"type":39,"tag":76,"props":481,"children":482},{"__ignoreMap":436},[483],{"type":44,"value":478},{"type":39,"tag":47,"props":485,"children":486},{},[487,489,494,496,502],{"type":44,"value":488},"Concrete ",{"type":39,"tag":76,"props":490,"children":492},{"className":491},[],[493],{"type":44,"value":176},{"type":44,"value":495}," form (proxy prefix per k6-manage §2; ",{"type":39,"tag":76,"props":497,"children":499},{"className":498},[],[500],{"type":44,"value":501},"-o json",{"type":44,"value":503}," avoids the spill envelope):",{"type":39,"tag":428,"props":505,"children":509},{"className":506,"code":507,"language":508,"meta":436,"style":436},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","LT=\u003Cload_test_id>; IDS=\"123,124,125\"\ngcx --context \u003Cctx> api \"\u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv5\u002Fload_tests\u002F$LT\u002Fquery_aggregate_k6(query='histogram_quantile(0.95)',metric='http_req_duration',test_run_ids=[$IDS])\" -o json\n","bash",[510],{"type":39,"tag":76,"props":511,"children":512},{"__ignoreMap":436},[513,567],{"type":39,"tag":514,"props":515,"children":518},"span",{"class":516,"line":517},"line",1,[519,525,531,537,542,547,552,557,562],{"type":39,"tag":514,"props":520,"children":522},{"style":521},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[523],{"type":44,"value":524},"LT",{"type":39,"tag":514,"props":526,"children":528},{"style":527},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[529],{"type":44,"value":530},"=\u003C",{"type":39,"tag":514,"props":532,"children":534},{"style":533},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[535],{"type":44,"value":536},"load_test_id",{"type":39,"tag":514,"props":538,"children":539},{"style":527},[540],{"type":44,"value":541},">;",{"type":39,"tag":514,"props":543,"children":544},{"style":521},[545],{"type":44,"value":546}," IDS",{"type":39,"tag":514,"props":548,"children":549},{"style":527},[550],{"type":44,"value":551},"=",{"type":39,"tag":514,"props":553,"children":554},{"style":527},[555],{"type":44,"value":556},"\"",{"type":39,"tag":514,"props":558,"children":559},{"style":533},[560],{"type":44,"value":561},"123,124,125",{"type":39,"tag":514,"props":563,"children":564},{"style":527},[565],{"type":44,"value":566},"\"\n",{"type":39,"tag":514,"props":568,"children":570},{"class":516,"line":569},2,[571,576,581,586,591,596,601,606,611,616,621,626,631,636,640,645],{"type":39,"tag":514,"props":572,"children":574},{"style":573},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[575],{"type":44,"value":176},{"type":39,"tag":514,"props":577,"children":578},{"style":533},[579],{"type":44,"value":580}," --context",{"type":39,"tag":514,"props":582,"children":583},{"style":527},[584],{"type":44,"value":585}," \u003C",{"type":39,"tag":514,"props":587,"children":588},{"style":533},[589],{"type":44,"value":590},"ct",{"type":39,"tag":514,"props":592,"children":593},{"style":521},[594],{"type":44,"value":595},"x",{"type":39,"tag":514,"props":597,"children":598},{"style":527},[599],{"type":44,"value":600},">",{"type":39,"tag":514,"props":602,"children":603},{"style":533},[604],{"type":44,"value":605}," api",{"type":39,"tag":514,"props":607,"children":608},{"style":527},[609],{"type":44,"value":610}," \"",{"type":39,"tag":514,"props":612,"children":613},{"style":533},[614],{"type":44,"value":615},"\u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv5\u002Fload_tests\u002F",{"type":39,"tag":514,"props":617,"children":618},{"style":521},[619],{"type":44,"value":620},"$LT",{"type":39,"tag":514,"props":622,"children":623},{"style":533},[624],{"type":44,"value":625},"\u002Fquery_aggregate_k6(query='histogram_quantile(0.95)',metric='http_req_duration',test_run_ids=[",{"type":39,"tag":514,"props":627,"children":628},{"style":521},[629],{"type":44,"value":630},"$IDS",{"type":39,"tag":514,"props":632,"children":633},{"style":533},[634],{"type":44,"value":635},"])",{"type":39,"tag":514,"props":637,"children":638},{"style":527},[639],{"type":44,"value":556},{"type":39,"tag":514,"props":641,"children":642},{"style":533},[643],{"type":44,"value":644}," -o",{"type":39,"tag":514,"props":646,"children":647},{"style":533},[648],{"type":44,"value":649}," json\n",{"type":39,"tag":47,"props":651,"children":652},{},[653],{"type":44,"value":654},"Choose the aggregate method based on metric type:",{"type":39,"tag":656,"props":657,"children":658},"table",{},[659,683],{"type":39,"tag":660,"props":661,"children":662},"thead",{},[663],{"type":39,"tag":664,"props":665,"children":666},"tr",{},[667,673,678],{"type":39,"tag":668,"props":669,"children":670},"th",{},[671],{"type":44,"value":672},"Metric type",{"type":39,"tag":668,"props":674,"children":675},{},[676],{"type":44,"value":677},"Primary method",{"type":39,"tag":668,"props":679,"children":680},{},[681],{"type":44,"value":682},"What it captures",{"type":39,"tag":684,"props":685,"children":686},"tbody",{},[687,713,737,761,786,811],{"type":39,"tag":664,"props":688,"children":689},{},[690,699,708],{"type":39,"tag":691,"props":692,"children":693},"td",{},[694],{"type":39,"tag":68,"props":695,"children":696},{},[697],{"type":44,"value":698},"trend",{"type":39,"tag":691,"props":700,"children":701},{},[702],{"type":39,"tag":76,"props":703,"children":705},{"className":704},[],[706],{"type":44,"value":707},"histogram_quantile(0.95)",{"type":39,"tag":691,"props":709,"children":710},{},[711],{"type":44,"value":712},"P95 latency -- the most common SLO target",{"type":39,"tag":664,"props":714,"children":715},{},[716,723,732],{"type":39,"tag":691,"props":717,"children":718},{},[719],{"type":39,"tag":68,"props":720,"children":721},{},[722],{"type":44,"value":698},{"type":39,"tag":691,"props":724,"children":725},{},[726],{"type":39,"tag":76,"props":727,"children":729},{"className":728},[],[730],{"type":44,"value":731},"histogram_quantile(0.5)",{"type":39,"tag":691,"props":733,"children":734},{},[735],{"type":44,"value":736},"Median -- shows typical behavior",{"type":39,"tag":664,"props":738,"children":739},{},[740,747,756],{"type":39,"tag":691,"props":741,"children":742},{},[743],{"type":39,"tag":68,"props":744,"children":745},{},[746],{"type":44,"value":698},{"type":39,"tag":691,"props":748,"children":749},{},[750],{"type":39,"tag":76,"props":751,"children":753},{"className":752},[],[754],{"type":44,"value":755},"histogram_avg",{"type":39,"tag":691,"props":757,"children":758},{},[759],{"type":44,"value":760},"Mean -- sensitive to outliers",{"type":39,"tag":664,"props":762,"children":763},{},[764,772,781],{"type":39,"tag":691,"props":765,"children":766},{},[767],{"type":39,"tag":68,"props":768,"children":769},{},[770],{"type":44,"value":771},"counter",{"type":39,"tag":691,"props":773,"children":774},{},[775],{"type":39,"tag":76,"props":776,"children":778},{"className":777},[],[779],{"type":44,"value":780},"increase",{"type":39,"tag":691,"props":782,"children":783},{},[784],{"type":44,"value":785},"Total count per run",{"type":39,"tag":664,"props":787,"children":788},{},[789,797,806],{"type":39,"tag":691,"props":790,"children":791},{},[792],{"type":39,"tag":68,"props":793,"children":794},{},[795],{"type":44,"value":796},"rate",{"type":39,"tag":691,"props":798,"children":799},{},[800],{"type":39,"tag":76,"props":801,"children":803},{"className":802},[],[804],{"type":44,"value":805},"ratio",{"type":39,"tag":691,"props":807,"children":808},{},[809],{"type":44,"value":810},"Success\u002Ffailure ratio",{"type":39,"tag":664,"props":812,"children":813},{},[814,822,831],{"type":39,"tag":691,"props":815,"children":816},{},[817],{"type":39,"tag":68,"props":818,"children":819},{},[820],{"type":44,"value":821},"gauge",{"type":39,"tag":691,"props":823,"children":824},{},[825],{"type":39,"tag":76,"props":826,"children":828},{"className":827},[],[829],{"type":44,"value":830},"max",{"type":39,"tag":691,"props":832,"children":833},{},[834],{"type":44,"value":835},"Peak value per run",{"type":39,"tag":47,"props":837,"children":838},{},[839],{"type":44,"value":840},"For trend-type metrics (latencies), query multiple quantiles (P50, P90, P95,\nP99) to see if degradation is uniform or concentrated in the tail.",{"type":39,"tag":47,"props":842,"children":843},{},[844,849],{"type":39,"tag":68,"props":845,"children":846},{},[847],{"type":44,"value":848},"Always break down by request grouping for multi-target tests.",{"type":44,"value":850}," A single\naggregate p95 across a whole test obscures regressions confined to one\nendpoint or one page -- a 3x slowdown on one URL can be invisible at the\ntest-level p95 if the test hits many URLs. Default to grouped queries:",{"type":39,"tag":656,"props":852,"children":853},{},[854,875],{"type":39,"tag":660,"props":855,"children":856},{},[857],{"type":39,"tag":664,"props":858,"children":859},{},[860,865,870],{"type":39,"tag":668,"props":861,"children":862},{},[863],{"type":44,"value":864},"Metric pattern",{"type":39,"tag":668,"props":866,"children":867},{},[868],{"type":44,"value":869},"Default grouping",{"type":39,"tag":668,"props":871,"children":872},{},[873],{"type":44,"value":874},"Why",{"type":39,"tag":684,"props":876,"children":877},{},[878,906,961,989],{"type":39,"tag":664,"props":879,"children":880},{},[881,892,901],{"type":39,"tag":691,"props":882,"children":883},{},[884,890],{"type":39,"tag":76,"props":885,"children":887},{"className":886},[],[888],{"type":44,"value":889},"browser_web_vital_*",{"type":44,"value":891}," (LCP, FCP, CLS, TTFB, INP, FID)",{"type":39,"tag":691,"props":893,"children":894},{},[895],{"type":39,"tag":76,"props":896,"children":898},{"className":897},[],[899],{"type":44,"value":900},"by (url)",{"type":39,"tag":691,"props":902,"children":903},{},[904],{"type":44,"value":905},"Each navigation emits its own web vital; per-URL trends pin regressions to a specific page.",{"type":39,"tag":664,"props":907,"children":908},{},[909,932,956],{"type":39,"tag":691,"props":910,"children":911},{},[912,918,919,925,926],{"type":39,"tag":76,"props":913,"children":915},{"className":914},[],[916],{"type":44,"value":917},"http_req_duration",{"type":44,"value":244},{"type":39,"tag":76,"props":920,"children":922},{"className":921},[],[923],{"type":44,"value":924},"http_req_failed",{"type":44,"value":244},{"type":39,"tag":76,"props":927,"children":929},{"className":928},[],[930],{"type":44,"value":931},"http_reqs",{"type":39,"tag":691,"props":933,"children":934},{},[935,941,943,948,950],{"type":39,"tag":76,"props":936,"children":938},{"className":937},[],[939],{"type":44,"value":940},"by (name, status)",{"type":44,"value":942}," if requests are tagged with ",{"type":39,"tag":76,"props":944,"children":946},{"className":945},[],[947],{"type":44,"value":250},{"type":44,"value":949},"; otherwise ",{"type":39,"tag":76,"props":951,"children":953},{"className":952},[],[954],{"type":44,"value":955},"by (url, status)",{"type":39,"tag":691,"props":957,"children":958},{},[959],{"type":44,"value":960},"Different endpoints have different baselines; mixing them hides per-endpoint drift.",{"type":39,"tag":664,"props":962,"children":963},{},[964,975,984],{"type":39,"tag":691,"props":965,"children":966},{},[967,973],{"type":39,"tag":76,"props":968,"children":970},{"className":969},[],[971],{"type":44,"value":972},"iteration_duration",{"type":44,"value":974}," (multi-scenario tests)",{"type":39,"tag":691,"props":976,"children":977},{},[978],{"type":39,"tag":76,"props":979,"children":981},{"className":980},[],[982],{"type":44,"value":983},"by (scenario)",{"type":39,"tag":691,"props":985,"children":986},{},[987],{"type":44,"value":988},"Browser and protocol scenarios have very different durations; mixing them is meaningless.",{"type":39,"tag":664,"props":990,"children":991},{},[992,997,1006],{"type":39,"tag":691,"props":993,"children":994},{},[995],{"type":44,"value":996},"Custom trends with tags",{"type":39,"tag":691,"props":998,"children":999},{},[1000],{"type":39,"tag":76,"props":1001,"children":1003},{"className":1002},[],[1004],{"type":44,"value":1005},"by (\u003Cthe tag>)",{"type":39,"tag":691,"props":1007,"children":1008},{},[1009],{"type":44,"value":1010},"Whatever the user tagged on is presumably what they care about.",{"type":39,"tag":47,"props":1012,"children":1013},{},[1014],{"type":44,"value":1015},"Discover available labels first via the labels endpoint (metrics.md §4) and\nlabel-values endpoint (metrics.md §5) before constructing the grouped query.\nDon't assume labels from reading the script -- a tag rename or refactor can\nsilently shift the label space. Example:",{"type":39,"tag":428,"props":1017,"children":1020},{"className":1018,"code":1019,"language":44},[431],"# List labels for a specific metric on a representative run\nGET \u002Fcloud\u002Fv5\u002Ftest_runs\u002F{id}\u002Flabels?match[]=browser_web_vital_lcp\n\n# Then enumerate values for a useful label\nGET \u002Fcloud\u002Fv5\u002Ftest_runs\u002F{id}\u002Flabel\u002Furl\u002Fvalues\n",[1021],{"type":39,"tag":76,"props":1022,"children":1023},{"__ignoreMap":436},[1024],{"type":44,"value":1019},{"type":39,"tag":47,"props":1026,"children":1027},{},[1028,1030,1036,1038,1044,1045,1051],{"type":44,"value":1029},"Only fall back to the bare aggregate (no ",{"type":39,"tag":76,"props":1031,"children":1033},{"className":1032},[],[1034],{"type":44,"value":1035},"by",{"type":44,"value":1037},") for metrics where grouping\nadds no information -- e.g., ",{"type":39,"tag":76,"props":1039,"children":1041},{"className":1040},[],[1042],{"type":44,"value":1043},"vus",{"type":44,"value":244},{"type":39,"tag":76,"props":1046,"children":1048},{"className":1047},[],[1049],{"type":44,"value":1050},"load_generator_cpu_percent",{"type":44,"value":1052},",\nsingle-target tests where every request hits the same URL.",{"type":39,"tag":47,"props":1054,"children":1055},{},[1056,1058,1064],{"type":44,"value":1057},"The response includes ",{"type":39,"tag":76,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":44,"value":1063},"test_run_id",{"type":44,"value":1065}," as a label -- use this to map each value\nback to its run timestamp from Step 3. Grouped queries return one series per\n(group, run) combination; flatten into a tidy \"rid x group\" table for the\ntrend computation in Step 7.",{"type":39,"tag":47,"props":1067,"children":1068},{},[1069],{"type":44,"value":1070},"If there are too many run IDs to fit in a single URL (hundreds), batch the\nqueries into groups of 50 run IDs and merge the results.",{"type":39,"tag":195,"props":1072,"children":1074},{"id":1073},"step-6-extract-thresholds",[1075],{"type":44,"value":1076},"Step 6: Extract thresholds",{"type":39,"tag":47,"props":1078,"children":1079},{},[1080,1082,1088],{"type":44,"value":1081},"Thresholds define what \"passing\" means. Fetch the test's current script (via\nk6-manage Section 5) and parse the ",{"type":39,"tag":76,"props":1083,"children":1085},{"className":1084},[],[1086],{"type":44,"value":1087},"export const options = { thresholds: {...} }",{"type":44,"value":1089},"\nblock. Record each threshold's:",{"type":39,"tag":60,"props":1091,"children":1092},{},[1093,1106,1118],{"type":39,"tag":64,"props":1094,"children":1095},{},[1096,1098,1104],{"type":44,"value":1097},"Metric name and selector (e.g., ",{"type":39,"tag":76,"props":1099,"children":1101},{"className":1100},[],[1102],{"type":44,"value":1103},"http_req_duration{name:homepage}",{"type":44,"value":1105},")",{"type":39,"tag":64,"props":1107,"children":1108},{},[1109,1111,1117],{"type":44,"value":1110},"Condition (e.g., ",{"type":39,"tag":76,"props":1112,"children":1114},{"className":1113},[],[1115],{"type":44,"value":1116},"p(95)\u003C500",{"type":44,"value":1105},{"type":39,"tag":64,"props":1119,"children":1120},{},[1121,1123,1129],{"type":44,"value":1122},"Whether ",{"type":39,"tag":76,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":44,"value":1128},"abortOnFail",{"type":44,"value":1130}," is set",{"type":39,"tag":47,"props":1132,"children":1133},{},[1134],{"type":44,"value":1135},"Also check the most recent run's threshold results from the run data to see\nwhich thresholds are currently passing vs. failing.",{"type":39,"tag":47,"props":1137,"children":1138},{},[1139],{"type":44,"value":1140},"Not all metrics will have thresholds -- that's fine. Metrics without thresholds\nstill get trend analysis; they just won't have headroom calculations.",{"type":39,"tag":195,"props":1142,"children":1144},{"id":1143},"step-7-compute-trends",[1145],{"type":44,"value":1146},"Step 7: Compute trends",{"type":39,"tag":47,"props":1148,"children":1149},{},[1150,1155],{"type":39,"tag":68,"props":1151,"children":1152},{},[1153],{"type":44,"value":1154},"Detect inflection points and rule out script changes deterministically.",{"type":44,"value":1156},"\nBefore drawing conclusions about a regression, look for discontinuities in\nthe metric values -- sudden jumps or drops that align across multiple\nmetrics on the same date. When you spot one, do not guess whether the\nscript changed. The run-bundled script endpoint gives a deterministic\nanswer in seconds:",{"type":39,"tag":1158,"props":1159,"children":1160},"ol",{},[1161,1364,1374],{"type":39,"tag":64,"props":1162,"children":1163},{},[1164,1169,1171,1177,1179,1185,1187,1358,1362],{"type":39,"tag":68,"props":1165,"children":1166},{},[1167],{"type":44,"value":1168},"sha256-diff the bundled scripts at the boundary.",{"type":44,"value":1170}," Fetch the snapshot\nfrom the last \"before\" run and the first \"after\" run via\n",{"type":39,"tag":76,"props":1172,"children":1174},{"className":1173},[],[1175],{"type":44,"value":1176},"GET \u002Fcloud\u002Fv6\u002Ftest_runs\u002F{id}\u002Fscript",{"type":44,"value":1178}," (k6-manage §5, \"Two distinct\nscript endpoints\" -- use the run-scoped endpoint, not the load-test\none, since the latter only shows the current version). Compare with\n",{"type":39,"tag":76,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":44,"value":1184},"shasum -a 256",{"type":44,"value":1186},".",{"type":39,"tag":428,"props":1188,"children":1190},{"className":506,"code":1189,"language":508,"meta":436,"style":436},"gcx --context \u003Cctx> api \u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv6\u002Ftest_runs\u002F\u003Cbefore_id>\u002Fscript > \u002Ftmp\u002Fbefore.bin\ngcx --context \u003Cctx> api \u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv6\u002Ftest_runs\u002F\u003Cafter_id>\u002Fscript > \u002Ftmp\u002Fafter.bin\nshasum -a 256 \u002Ftmp\u002Fbefore.bin \u002Ftmp\u002Fafter.bin\n",[1191],{"type":39,"tag":76,"props":1192,"children":1193},{"__ignoreMap":436},[1194,1264,1329],{"type":39,"tag":514,"props":1195,"children":1196},{"class":516,"line":517},[1197,1201,1205,1209,1213,1217,1221,1225,1230,1235,1240,1245,1249,1254,1259],{"type":39,"tag":514,"props":1198,"children":1199},{"style":573},[1200],{"type":44,"value":176},{"type":39,"tag":514,"props":1202,"children":1203},{"style":533},[1204],{"type":44,"value":580},{"type":39,"tag":514,"props":1206,"children":1207},{"style":527},[1208],{"type":44,"value":585},{"type":39,"tag":514,"props":1210,"children":1211},{"style":533},[1212],{"type":44,"value":590},{"type":39,"tag":514,"props":1214,"children":1215},{"style":521},[1216],{"type":44,"value":595},{"type":39,"tag":514,"props":1218,"children":1219},{"style":527},[1220],{"type":44,"value":600},{"type":39,"tag":514,"props":1222,"children":1223},{"style":533},[1224],{"type":44,"value":605},{"type":39,"tag":514,"props":1226,"children":1227},{"style":533},[1228],{"type":44,"value":1229}," \u002Fapi\u002Fplugins\u002Fk6-app\u002Fresources\u002Fcloud\u002Fcloud\u002Fv6\u002Ftest_runs\u002F",{"type":39,"tag":514,"props":1231,"children":1232},{"style":527},[1233],{"type":44,"value":1234},"\u003C",{"type":39,"tag":514,"props":1236,"children":1237},{"style":533},[1238],{"type":44,"value":1239},"before_i",{"type":39,"tag":514,"props":1241,"children":1242},{"style":521},[1243],{"type":44,"value":1244},"d",{"type":39,"tag":514,"props":1246,"children":1247},{"style":527},[1248],{"type":44,"value":600},{"type":39,"tag":514,"props":1250,"children":1251},{"style":533},[1252],{"type":44,"value":1253},"\u002Fscript",{"type":39,"tag":514,"props":1255,"children":1256},{"style":527},[1257],{"type":44,"value":1258}," >",{"type":39,"tag":514,"props":1260,"children":1261},{"style":533},[1262],{"type":44,"value":1263}," \u002Ftmp\u002Fbefore.bin\n",{"type":39,"tag":514,"props":1265,"children":1266},{"class":516,"line":569},[1267,1271,1275,1279,1283,1287,1291,1295,1299,1303,1308,1312,1316,1320,1324],{"type":39,"tag":514,"props":1268,"children":1269},{"style":573},[1270],{"type":44,"value":176},{"type":39,"tag":514,"props":1272,"children":1273},{"style":533},[1274],{"type":44,"value":580},{"type":39,"tag":514,"props":1276,"children":1277},{"style":527},[1278],{"type":44,"value":585},{"type":39,"tag":514,"props":1280,"children":1281},{"style":533},[1282],{"type":44,"value":590},{"type":39,"tag":514,"props":1284,"children":1285},{"style":521},[1286],{"type":44,"value":595},{"type":39,"tag":514,"props":1288,"children":1289},{"style":527},[1290],{"type":44,"value":600},{"type":39,"tag":514,"props":1292,"children":1293},{"style":533},[1294],{"type":44,"value":605},{"type":39,"tag":514,"props":1296,"children":1297},{"style":533},[1298],{"type":44,"value":1229},{"type":39,"tag":514,"props":1300,"children":1301},{"style":527},[1302],{"type":44,"value":1234},{"type":39,"tag":514,"props":1304,"children":1305},{"style":533},[1306],{"type":44,"value":1307},"after_i",{"type":39,"tag":514,"props":1309,"children":1310},{"style":521},[1311],{"type":44,"value":1244},{"type":39,"tag":514,"props":1313,"children":1314},{"style":527},[1315],{"type":44,"value":600},{"type":39,"tag":514,"props":1317,"children":1318},{"style":533},[1319],{"type":44,"value":1253},{"type":39,"tag":514,"props":1321,"children":1322},{"style":527},[1323],{"type":44,"value":1258},{"type":39,"tag":514,"props":1325,"children":1326},{"style":533},[1327],{"type":44,"value":1328}," \u002Ftmp\u002Fafter.bin\n",{"type":39,"tag":514,"props":1330,"children":1332},{"class":516,"line":1331},3,[1333,1338,1343,1349,1354],{"type":39,"tag":514,"props":1334,"children":1335},{"style":573},[1336],{"type":44,"value":1337},"shasum",{"type":39,"tag":514,"props":1339,"children":1340},{"style":533},[1341],{"type":44,"value":1342}," -a",{"type":39,"tag":514,"props":1344,"children":1346},{"style":1345},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1347],{"type":44,"value":1348}," 256",{"type":39,"tag":514,"props":1350,"children":1351},{"style":533},[1352],{"type":44,"value":1353}," \u002Ftmp\u002Fbefore.bin",{"type":39,"tag":514,"props":1355,"children":1356},{"style":533},[1357],{"type":44,"value":1328},{"type":39,"tag":1359,"props":1360,"children":1361},"br",{},[],{"type":44,"value":1363},"For thoroughness on multi-run inflections, hash every run across the\ntransition -- if N consecutive runs share one hash and N more share\nanother, you have a clean before\u002Fafter boundary. If hashes change\nmid-stream, the test was edited multiple times.",{"type":39,"tag":64,"props":1365,"children":1366},{},[1367,1372],{"type":39,"tag":68,"props":1368,"children":1369},{},[1370],{"type":44,"value":1371},"If the sha256 differs",{"type":44,"value":1373},", the test script changed -- the inflection\nmay be a test-side artifact, not a service regression. Split the\nanalysis into distinct eras at the boundary and compute trends within\neach era separately. Comparing metrics across script changes produces\nmisleading trends -- a P95 drop from 3,000ms to 150ms is not an\n\"improvement\" if the script simply stopped hitting a slow endpoint.\nState the eras explicitly in the report and focus recommendations on\nthe most recent era.",{"type":39,"tag":64,"props":1375,"children":1376},{},[1377,1382],{"type":39,"tag":68,"props":1378,"children":1379},{},[1380],{"type":44,"value":1381},"If the sha256 matches",{"type":44,"value":1383},", the script is byte-identical and the\nregression is external to the test (service-side, infrastructure, or\nload-zone). This is high-confidence information -- carry it into Step 9\n(service-side correlation) instead of leaving \"did the test change?\"\nas an open question.",{"type":39,"tag":47,"props":1385,"children":1386},{},[1387],{"type":44,"value":1388},"This sha256 diff is a 5-second deterministic check that rules out a huge\nclass of causes. Run it at every detected inflection, not just when the\nuser asks.",{"type":39,"tag":47,"props":1390,"children":1391},{},[1392],{"type":44,"value":1393},"For each metric, build a time-ordered series of (run_timestamp, value) pairs.\nThen compute:",{"type":39,"tag":47,"props":1395,"children":1396},{},[1397],{"type":39,"tag":68,"props":1398,"children":1399},{},[1400],{"type":44,"value":1401},"Basic statistics:",{"type":39,"tag":60,"props":1403,"children":1404},{},[1405,1410,1415],{"type":39,"tag":64,"props":1406,"children":1407},{},[1408],{"type":44,"value":1409},"Mean, standard deviation, min, max across all runs",{"type":39,"tag":64,"props":1411,"children":1412},{},[1413],{"type":44,"value":1414},"Current value (most recent run)",{"type":39,"tag":64,"props":1416,"children":1417},{},[1418],{"type":44,"value":1419},"Baseline value (oldest run in window, or mean of first 3 runs for stability)",{"type":39,"tag":47,"props":1421,"children":1422},{},[1423,1428],{"type":39,"tag":68,"props":1424,"children":1425},{},[1426],{"type":44,"value":1427},"Trend direction:",{"type":44,"value":1429},"\nSplit the runs into two halves (first half and second half of the time window).\nCompare the mean of each half:",{"type":39,"tag":60,"props":1431,"children":1432},{},[1433,1443,1453,1463],{"type":39,"tag":64,"props":1434,"children":1435},{},[1436,1441],{"type":39,"tag":68,"props":1437,"children":1438},{},[1439],{"type":44,"value":1440},"Degrading",{"type":44,"value":1442},": second-half mean is worse by >10% (higher for latency\u002Ferrors,\nlower for success rates)",{"type":39,"tag":64,"props":1444,"children":1445},{},[1446,1451],{"type":39,"tag":68,"props":1447,"children":1448},{},[1449],{"type":44,"value":1450},"Improving",{"type":44,"value":1452},": second-half mean is better by >10%",{"type":39,"tag":64,"props":1454,"children":1455},{},[1456,1461],{"type":39,"tag":68,"props":1457,"children":1458},{},[1459],{"type":44,"value":1460},"Stable",{"type":44,"value":1462},": change is within 10%",{"type":39,"tag":64,"props":1464,"children":1465},{},[1466,1471],{"type":39,"tag":68,"props":1467,"children":1468},{},[1469],{"type":44,"value":1470},"Volatile",{"type":44,"value":1472},": standard deviation exceeds 25% of the mean, regardless of\ndirection",{"type":39,"tag":47,"props":1474,"children":1475},{},[1476],{"type":44,"value":1477},"The 10% and 25% thresholds are starting points. If the user's test has very\ntight tolerances or very noisy metrics, adjust and explain the reasoning.",{"type":39,"tag":47,"props":1479,"children":1480},{},[1481,1486,1488,1494],{"type":39,"tag":68,"props":1482,"children":1483},{},[1484],{"type":44,"value":1485},"Rate of change:",{"type":44,"value":1487},"\nExpress as percentage change per week: ",{"type":39,"tag":76,"props":1489,"children":1491},{"className":1490},[],[1492],{"type":44,"value":1493},"((recent_mean - baseline_mean) \u002F baseline_mean) * 100 \u002F weeks_in_window",{"type":44,"value":1495},". This normalizes across different window sizes.",{"type":39,"tag":47,"props":1497,"children":1498},{},[1499,1504],{"type":39,"tag":68,"props":1500,"children":1501},{},[1502],{"type":44,"value":1503},"Anomaly detection:",{"type":44,"value":1505},"\nFlag any run where the metric value is more than 2 standard deviations from the\noverall mean. These are potential inflection points worth investigating\nindividually.",{"type":39,"tag":195,"props":1507,"children":1509},{"id":1508},"step-8-headroom-analysis",[1510],{"type":44,"value":1511},"Step 8: Headroom analysis",{"type":39,"tag":47,"props":1513,"children":1514},{},[1515],{"type":44,"value":1516},"For metrics that have thresholds defined (from Step 6), compute headroom:",{"type":39,"tag":428,"props":1518,"children":1521},{"className":1519,"code":1520,"language":44},[431],"headroom_pct = ((threshold_value - current_value) \u002F threshold_value) * 100\n",[1522],{"type":39,"tag":76,"props":1523,"children":1524},{"__ignoreMap":436},[1525],{"type":44,"value":1520},{"type":39,"tag":47,"props":1527,"children":1528},{},[1529],{"type":44,"value":1530},"Classify headroom:",{"type":39,"tag":60,"props":1532,"children":1533},{},[1534,1544,1554,1564],{"type":39,"tag":64,"props":1535,"children":1536},{},[1537,1542],{"type":39,"tag":68,"props":1538,"children":1539},{},[1540],{"type":44,"value":1541},"Comfortable",{"type":44,"value":1543}," (>50%): well within limits",{"type":39,"tag":64,"props":1545,"children":1546},{},[1547,1552],{"type":39,"tag":68,"props":1548,"children":1549},{},[1550],{"type":44,"value":1551},"Adequate",{"type":44,"value":1553}," (20-50%): healthy but worth monitoring",{"type":39,"tag":64,"props":1555,"children":1556},{},[1557,1562],{"type":39,"tag":68,"props":1558,"children":1559},{},[1560],{"type":44,"value":1561},"Thin",{"type":44,"value":1563}," (\u003C20%): at risk of breaching if trends continue",{"type":39,"tag":64,"props":1565,"children":1566},{},[1567,1572],{"type":39,"tag":68,"props":1568,"children":1569},{},[1570],{"type":44,"value":1571},"Breached",{"type":44,"value":1573}," (\u003C0%): already failing",{"type":39,"tag":47,"props":1575,"children":1576},{},[1577],{"type":44,"value":1578},"For degrading metrics with thin headroom, estimate when the threshold will be\nbreached if the current rate of change continues:",{"type":39,"tag":428,"props":1580,"children":1583},{"className":1581,"code":1582,"language":44},[431],"weeks_until_breach = headroom_absolute \u002F rate_of_change_per_week\n",[1584],{"type":39,"tag":76,"props":1585,"children":1586},{"__ignoreMap":436},[1587],{"type":44,"value":1582},{"type":39,"tag":47,"props":1589,"children":1590},{},[1591],{"type":44,"value":1592},"This is a rough projection, not a prediction -- present it as \"at the current\nrate of degradation, this metric could breach in approximately N weeks.\"",{"type":39,"tag":195,"props":1594,"children":1596},{"id":1595},"step-9-service-side-correlation-when-warranted",[1597],{"type":44,"value":1598},"Step 9: Service-side correlation (when warranted)",{"type":39,"tag":47,"props":1600,"children":1601},{},[1602,1604,1609,1611,1616,1618,1623],{"type":44,"value":1603},"If Step 7 reveals degradation, offer (don't auto-run) to correlate with\nservice-side data to separate ",{"type":39,"tag":68,"props":1605,"children":1606},{},[1607],{"type":44,"value":1608},"service degradation",{"type":44,"value":1610}," (fix the service),\n",{"type":39,"tag":68,"props":1612,"children":1613},{},[1614],{"type":44,"value":1615},"test-environment changes",{"type":44,"value":1617}," (load-zone latency, LG exhaustion), and\n",{"type":39,"tag":68,"props":1619,"children":1620},{},[1621],{"type":44,"value":1622},"script changes",{"type":44,"value":1624}," (a slower edit). Present findings and ask first.",{"type":39,"tag":47,"props":1626,"children":1627},{},[1628,1630,1635,1637,1642],{"type":44,"value":1629},"Hand off to ",{"type":39,"tag":76,"props":1631,"children":1633},{"className":1632},[],[1634],{"type":44,"value":122},{"type":44,"value":1636}," (via ",{"type":39,"tag":76,"props":1638,"children":1640},{"className":1639},[],[1641],{"type":44,"value":176},{"type":44,"value":1643},") to query the service's Prometheus\nmetrics and Loki logs for the same window; look for service error-rate changes,\nupstream latency, resource pressure (CPU\u002Fmemory\u002Fpools), and deploys coinciding\nwith inflection points.",{"type":39,"tag":47,"props":1645,"children":1646},{},[1647,1652],{"type":39,"tag":68,"props":1648,"children":1649},{},[1650],{"type":44,"value":1651},"When the service has no observability",{"type":44,"value":1653}," (third-party or another team's\nservice -- no Prometheus job, Loki stream, or probe), fall back to client-side\nsignal, which still localises the regression:",{"type":39,"tag":60,"props":1655,"children":1656},{},[1657,1691,1733],{"type":39,"tag":64,"props":1658,"children":1659},{},[1660,1665,1667,1673,1675,1681,1683,1689],{"type":39,"tag":68,"props":1661,"children":1662},{},[1663],{"type":44,"value":1664},"Browser tests:",{"type":44,"value":1666}," compare Tempo iteration traces (k6-manage §7,\n",{"type":39,"tag":76,"props":1668,"children":1670},{"className":1669},[],[1671],{"type":44,"value":1672},"\u002Fapi\u002Fv1\u002Ftempo\u002Fapi\u002Fsearch",{"type":44,"value":1674}," + ",{"type":39,"tag":76,"props":1676,"children":1678},{"className":1677},[],[1679],{"type":44,"value":1680},"\u002Ftraces\u002F{id}",{"type":44,"value":1682},") for a before\u002Fafter run. The\nspan-name rollup, slowest spans, and per-URL navigation durations pin the\nregression to a URL, locator action, or asset; web vitals are ",{"type":39,"tag":76,"props":1684,"children":1686},{"className":1685},[],[1687],{"type":44,"value":1688},"web_vital.*",{"type":44,"value":1690},"\nspan attributes -- read them directly.",{"type":39,"tag":64,"props":1692,"children":1693},{},[1694,1699,1701,1707,1709,1715,1717,1723,1725,1731],{"type":39,"tag":68,"props":1695,"children":1696},{},[1697],{"type":44,"value":1698},"Protocol tests:",{"type":44,"value":1700}," compare ",{"type":39,"tag":76,"props":1702,"children":1704},{"className":1703},[],[1705],{"type":44,"value":1706},"http_req_duration by (name, status)",{"type":44,"value":1708}," and\n",{"type":39,"tag":76,"props":1710,"children":1712},{"className":1711},[],[1713],{"type":44,"value":1714},"http_req_failed by (name, status)",{"type":44,"value":1716},"; a regression on one named request\npoints at that endpoint. Split ",{"type":39,"tag":76,"props":1718,"children":1720},{"className":1719},[],[1721],{"type":44,"value":1722},"http_req_waiting",{"type":44,"value":1724}," (server time) vs\n",{"type":39,"tag":76,"props":1726,"children":1728},{"className":1727},[],[1729],{"type":44,"value":1730},"http_req_receiving",{"type":44,"value":1732}," (transfer) to separate slow processing from slow download.",{"type":39,"tag":64,"props":1734,"children":1735},{},[1736,1741,1743,1749,1751,1757],{"type":39,"tag":68,"props":1737,"children":1738},{},[1739],{"type":44,"value":1740},"Payload corroboration",{"type":44,"value":1742}," (both): query ",{"type":39,"tag":76,"props":1744,"children":1746},{"className":1745},[],[1747],{"type":44,"value":1748},"data_received",{"type":44,"value":1750}," \u002F\n",{"type":39,"tag":76,"props":1752,"children":1754},{"className":1753},[],[1755],{"type":44,"value":1756},"browser_data_received",{"type":44,"value":1758},". Latency up + payload up -> server content changes;\nlatency up + payload stable -> server processing changes; intermittent\npayload -> flaky cache or A\u002FB test.",{"type":39,"tag":47,"props":1760,"children":1761},{},[1762],{"type":44,"value":1763},"Prefer server-side correlation when available -- it answers \"why\" directly; the\nfallbacks answer \"where\" and \"what kind of change\", and make a good ticket for\nthe owning team.",{"type":39,"tag":195,"props":1765,"children":1767},{"id":1766},"step-10-present-the-report",[1768],{"type":44,"value":1769},"Step 10: Present the report",{"type":39,"tag":47,"props":1771,"children":1772},{},[1773],{"type":44,"value":1774},"Always present findings as a structured report. Never apply changes directly.",{"type":39,"tag":180,"props":1776,"children":1777},{},[],{"type":39,"tag":53,"props":1779,"children":1781},{"id":1780},"report-template",[1782],{"type":44,"value":1783},"Report template",{"type":39,"tag":47,"props":1785,"children":1786},{},[1787],{"type":44,"value":1788},"Use this structure. Omit sections that don't apply (e.g., skip \"Anomalies\" if\nnone were detected).",{"type":39,"tag":428,"props":1790,"children":1794},{"className":1791,"code":1792,"language":1793,"meta":436,"style":436},"language-markdown shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Trend Analysis: {test_name}\n\n**Test ID**: {test_id}\n**Analysis window**: {start_date} to {end_date} ({N} runs analyzed)\n**Overall health**: {Healthy | Watch | Degrading | Critical}\n\n## Run Summary\n\n| Period | Runs | Passed | Failed | Pass Rate |\n|--------|------|--------|--------|-----------|\n| First half | N | N | N | N% |\n| Second half | N | N | N | N% |\n\n## Metric Trends\n\n| Metric | Type | Current | Baseline | Change | Trend | Threshold | Headroom |\n|--------|------|---------|----------|--------|-------|-----------|----------|\n| http_req_duration (P95) | trend | 380ms | 250ms | +52% | Degrading | 500ms | 24% |\n| http_req_failed | rate | 0.8% | 0.3% | +167% | Degrading | 1% | 20% |\n| http_reqs | counter | 15,230 | 15,100 | +0.9% | Stable | - | - |\n\n## Flagged Issues\n\n### 1. {metric_name}: {classification}\n- **Current**: {value} | **Baseline**: {value} | **Change**: {pct}%\n- **Threshold**: {threshold} | **Headroom**: {pct}%\n- **Rate of change**: {pct}% per week\n- **Projected breach**: ~{N} weeks at current rate\n- **Anomalous runs**: {run_ids with dates, if any}\n\n## Threshold Recommendations\n\n| Metric | Current Threshold | Recommended | Rationale |\n|--------|-------------------|-------------|-----------|\n| http_req_duration | p(95)\u003C500 | p(95)\u003C420 | Current P95 is 380ms; tightening to 420ms gives 10% headroom from current performance while surfacing further degradation early |\n\n## Suggested Next Steps\n\n- [ ] **Investigate service side**: P95 latency has increased 52% -- consider\n      loading `debug-with-grafana` to check service health\n- [ ] **Deep-dive run {run_id}**: anomalous P95 spike on {date} -- consider\n      loading `k6-cloud-investigate-test` for this run\n- [ ] **Tighten thresholds**: 2 metrics have >30% headroom that could be\n      tightened -- consider loading `k6-test-maintenance` to apply changes\n","markdown",[1795],{"type":39,"tag":76,"props":1796,"children":1797},{"__ignoreMap":436},[1798,1811,1820,1844,1866,1888,1896,1910,1918,1973,1982,2033,2082,2090,2103,2111,2190,2199,2279,2358,2439,2447,2460,2468,2482,2545,2588,2614,2640,2666,2674,2687,2695,2738,2747,2791,2799,2812,2820,2851,2878,2908,2933,2963],{"type":39,"tag":514,"props":1799,"children":1800},{"class":516,"line":517},[1801,1806],{"type":39,"tag":514,"props":1802,"children":1803},{"style":527},[1804],{"type":44,"value":1805},"# ",{"type":39,"tag":514,"props":1807,"children":1808},{"style":573},[1809],{"type":44,"value":1810},"Trend Analysis: {test_name}\n",{"type":39,"tag":514,"props":1812,"children":1813},{"class":516,"line":569},[1814],{"type":39,"tag":514,"props":1815,"children":1817},{"emptyLinePlaceholder":1816},true,[1818],{"type":44,"value":1819},"\n",{"type":39,"tag":514,"props":1821,"children":1822},{"class":516,"line":1331},[1823,1829,1835,1839],{"type":39,"tag":514,"props":1824,"children":1826},{"style":1825},"--shiki-light:#39ADB5;--shiki-light-font-weight:bold;--shiki-default:#89DDFF;--shiki-default-font-weight:bold;--shiki-dark:#89DDFF;--shiki-dark-font-weight:bold",[1827],{"type":44,"value":1828},"**",{"type":39,"tag":514,"props":1830,"children":1832},{"style":1831},"--shiki-light:#E53935;--shiki-light-font-weight:bold;--shiki-default:#F07178;--shiki-default-font-weight:bold;--shiki-dark:#F07178;--shiki-dark-font-weight:bold",[1833],{"type":44,"value":1834},"Test ID",{"type":39,"tag":514,"props":1836,"children":1837},{"style":1825},[1838],{"type":44,"value":1828},{"type":39,"tag":514,"props":1840,"children":1841},{"style":521},[1842],{"type":44,"value":1843},": {test_id}\n",{"type":39,"tag":514,"props":1845,"children":1847},{"class":516,"line":1846},4,[1848,1852,1857,1861],{"type":39,"tag":514,"props":1849,"children":1850},{"style":1825},[1851],{"type":44,"value":1828},{"type":39,"tag":514,"props":1853,"children":1854},{"style":1831},[1855],{"type":44,"value":1856},"Analysis window",{"type":39,"tag":514,"props":1858,"children":1859},{"style":1825},[1860],{"type":44,"value":1828},{"type":39,"tag":514,"props":1862,"children":1863},{"style":521},[1864],{"type":44,"value":1865},": {start_date} to {end_date} ({N} runs analyzed)\n",{"type":39,"tag":514,"props":1867,"children":1869},{"class":516,"line":1868},5,[1870,1874,1879,1883],{"type":39,"tag":514,"props":1871,"children":1872},{"style":1825},[1873],{"type":44,"value":1828},{"type":39,"tag":514,"props":1875,"children":1876},{"style":1831},[1877],{"type":44,"value":1878},"Overall health",{"type":39,"tag":514,"props":1880,"children":1881},{"style":1825},[1882],{"type":44,"value":1828},{"type":39,"tag":514,"props":1884,"children":1885},{"style":521},[1886],{"type":44,"value":1887},": {Healthy | Watch | Degrading | Critical}\n",{"type":39,"tag":514,"props":1889,"children":1891},{"class":516,"line":1890},6,[1892],{"type":39,"tag":514,"props":1893,"children":1894},{"emptyLinePlaceholder":1816},[1895],{"type":44,"value":1819},{"type":39,"tag":514,"props":1897,"children":1899},{"class":516,"line":1898},7,[1900,1905],{"type":39,"tag":514,"props":1901,"children":1902},{"style":527},[1903],{"type":44,"value":1904},"## ",{"type":39,"tag":514,"props":1906,"children":1907},{"style":573},[1908],{"type":44,"value":1909},"Run Summary\n",{"type":39,"tag":514,"props":1911,"children":1913},{"class":516,"line":1912},8,[1914],{"type":39,"tag":514,"props":1915,"children":1916},{"emptyLinePlaceholder":1816},[1917],{"type":44,"value":1819},{"type":39,"tag":514,"props":1919,"children":1921},{"class":516,"line":1920},9,[1922,1927,1932,1936,1941,1945,1950,1954,1959,1963,1968],{"type":39,"tag":514,"props":1923,"children":1924},{"style":527},[1925],{"type":44,"value":1926},"|",{"type":39,"tag":514,"props":1928,"children":1929},{"style":521},[1930],{"type":44,"value":1931}," Period ",{"type":39,"tag":514,"props":1933,"children":1934},{"style":527},[1935],{"type":44,"value":1926},{"type":39,"tag":514,"props":1937,"children":1938},{"style":521},[1939],{"type":44,"value":1940}," Runs ",{"type":39,"tag":514,"props":1942,"children":1943},{"style":527},[1944],{"type":44,"value":1926},{"type":39,"tag":514,"props":1946,"children":1947},{"style":521},[1948],{"type":44,"value":1949}," Passed ",{"type":39,"tag":514,"props":1951,"children":1952},{"style":527},[1953],{"type":44,"value":1926},{"type":39,"tag":514,"props":1955,"children":1956},{"style":521},[1957],{"type":44,"value":1958}," Failed ",{"type":39,"tag":514,"props":1960,"children":1961},{"style":527},[1962],{"type":44,"value":1926},{"type":39,"tag":514,"props":1964,"children":1965},{"style":521},[1966],{"type":44,"value":1967}," Pass Rate ",{"type":39,"tag":514,"props":1969,"children":1970},{"style":527},[1971],{"type":44,"value":1972},"|\n",{"type":39,"tag":514,"props":1974,"children":1976},{"class":516,"line":1975},10,[1977],{"type":39,"tag":514,"props":1978,"children":1979},{"style":527},[1980],{"type":44,"value":1981},"|--------|------|--------|--------|-----------|\n",{"type":39,"tag":514,"props":1983,"children":1985},{"class":516,"line":1984},11,[1986,1990,1995,1999,2004,2008,2012,2016,2020,2024,2029],{"type":39,"tag":514,"props":1987,"children":1988},{"style":527},[1989],{"type":44,"value":1926},{"type":39,"tag":514,"props":1991,"children":1992},{"style":521},[1993],{"type":44,"value":1994}," First half ",{"type":39,"tag":514,"props":1996,"children":1997},{"style":527},[1998],{"type":44,"value":1926},{"type":39,"tag":514,"props":2000,"children":2001},{"style":521},[2002],{"type":44,"value":2003}," N ",{"type":39,"tag":514,"props":2005,"children":2006},{"style":527},[2007],{"type":44,"value":1926},{"type":39,"tag":514,"props":2009,"children":2010},{"style":521},[2011],{"type":44,"value":2003},{"type":39,"tag":514,"props":2013,"children":2014},{"style":527},[2015],{"type":44,"value":1926},{"type":39,"tag":514,"props":2017,"children":2018},{"style":521},[2019],{"type":44,"value":2003},{"type":39,"tag":514,"props":2021,"children":2022},{"style":527},[2023],{"type":44,"value":1926},{"type":39,"tag":514,"props":2025,"children":2026},{"style":521},[2027],{"type":44,"value":2028}," N% ",{"type":39,"tag":514,"props":2030,"children":2031},{"style":527},[2032],{"type":44,"value":1972},{"type":39,"tag":514,"props":2034,"children":2036},{"class":516,"line":2035},12,[2037,2041,2046,2050,2054,2058,2062,2066,2070,2074,2078],{"type":39,"tag":514,"props":2038,"children":2039},{"style":527},[2040],{"type":44,"value":1926},{"type":39,"tag":514,"props":2042,"children":2043},{"style":521},[2044],{"type":44,"value":2045}," Second half ",{"type":39,"tag":514,"props":2047,"children":2048},{"style":527},[2049],{"type":44,"value":1926},{"type":39,"tag":514,"props":2051,"children":2052},{"style":521},[2053],{"type":44,"value":2003},{"type":39,"tag":514,"props":2055,"children":2056},{"style":527},[2057],{"type":44,"value":1926},{"type":39,"tag":514,"props":2059,"children":2060},{"style":521},[2061],{"type":44,"value":2003},{"type":39,"tag":514,"props":2063,"children":2064},{"style":527},[2065],{"type":44,"value":1926},{"type":39,"tag":514,"props":2067,"children":2068},{"style":521},[2069],{"type":44,"value":2003},{"type":39,"tag":514,"props":2071,"children":2072},{"style":527},[2073],{"type":44,"value":1926},{"type":39,"tag":514,"props":2075,"children":2076},{"style":521},[2077],{"type":44,"value":2028},{"type":39,"tag":514,"props":2079,"children":2080},{"style":527},[2081],{"type":44,"value":1972},{"type":39,"tag":514,"props":2083,"children":2085},{"class":516,"line":2084},13,[2086],{"type":39,"tag":514,"props":2087,"children":2088},{"emptyLinePlaceholder":1816},[2089],{"type":44,"value":1819},{"type":39,"tag":514,"props":2091,"children":2093},{"class":516,"line":2092},14,[2094,2098],{"type":39,"tag":514,"props":2095,"children":2096},{"style":527},[2097],{"type":44,"value":1904},{"type":39,"tag":514,"props":2099,"children":2100},{"style":573},[2101],{"type":44,"value":2102},"Metric Trends\n",{"type":39,"tag":514,"props":2104,"children":2106},{"class":516,"line":2105},15,[2107],{"type":39,"tag":514,"props":2108,"children":2109},{"emptyLinePlaceholder":1816},[2110],{"type":44,"value":1819},{"type":39,"tag":514,"props":2112,"children":2113},{"class":516,"line":27},[2114,2118,2123,2127,2132,2136,2141,2145,2150,2154,2159,2163,2168,2172,2177,2181,2186],{"type":39,"tag":514,"props":2115,"children":2116},{"style":527},[2117],{"type":44,"value":1926},{"type":39,"tag":514,"props":2119,"children":2120},{"style":521},[2121],{"type":44,"value":2122}," Metric ",{"type":39,"tag":514,"props":2124,"children":2125},{"style":527},[2126],{"type":44,"value":1926},{"type":39,"tag":514,"props":2128,"children":2129},{"style":521},[2130],{"type":44,"value":2131}," Type ",{"type":39,"tag":514,"props":2133,"children":2134},{"style":527},[2135],{"type":44,"value":1926},{"type":39,"tag":514,"props":2137,"children":2138},{"style":521},[2139],{"type":44,"value":2140}," Current ",{"type":39,"tag":514,"props":2142,"children":2143},{"style":527},[2144],{"type":44,"value":1926},{"type":39,"tag":514,"props":2146,"children":2147},{"style":521},[2148],{"type":44,"value":2149}," Baseline ",{"type":39,"tag":514,"props":2151,"children":2152},{"style":527},[2153],{"type":44,"value":1926},{"type":39,"tag":514,"props":2155,"children":2156},{"style":521},[2157],{"type":44,"value":2158}," Change ",{"type":39,"tag":514,"props":2160,"children":2161},{"style":527},[2162],{"type":44,"value":1926},{"type":39,"tag":514,"props":2164,"children":2165},{"style":521},[2166],{"type":44,"value":2167}," Trend ",{"type":39,"tag":514,"props":2169,"children":2170},{"style":527},[2171],{"type":44,"value":1926},{"type":39,"tag":514,"props":2173,"children":2174},{"style":521},[2175],{"type":44,"value":2176}," Threshold ",{"type":39,"tag":514,"props":2178,"children":2179},{"style":527},[2180],{"type":44,"value":1926},{"type":39,"tag":514,"props":2182,"children":2183},{"style":521},[2184],{"type":44,"value":2185}," Headroom ",{"type":39,"tag":514,"props":2187,"children":2188},{"style":527},[2189],{"type":44,"value":1972},{"type":39,"tag":514,"props":2191,"children":2193},{"class":516,"line":2192},17,[2194],{"type":39,"tag":514,"props":2195,"children":2196},{"style":527},[2197],{"type":44,"value":2198},"|--------|------|---------|----------|--------|-------|-----------|----------|\n",{"type":39,"tag":514,"props":2200,"children":2202},{"class":516,"line":2201},18,[2203,2207,2212,2216,2221,2225,2230,2234,2239,2243,2248,2252,2257,2261,2266,2270,2275],{"type":39,"tag":514,"props":2204,"children":2205},{"style":527},[2206],{"type":44,"value":1926},{"type":39,"tag":514,"props":2208,"children":2209},{"style":521},[2210],{"type":44,"value":2211}," http_req_duration (P95) ",{"type":39,"tag":514,"props":2213,"children":2214},{"style":527},[2215],{"type":44,"value":1926},{"type":39,"tag":514,"props":2217,"children":2218},{"style":521},[2219],{"type":44,"value":2220}," trend ",{"type":39,"tag":514,"props":2222,"children":2223},{"style":527},[2224],{"type":44,"value":1926},{"type":39,"tag":514,"props":2226,"children":2227},{"style":521},[2228],{"type":44,"value":2229}," 380ms ",{"type":39,"tag":514,"props":2231,"children":2232},{"style":527},[2233],{"type":44,"value":1926},{"type":39,"tag":514,"props":2235,"children":2236},{"style":521},[2237],{"type":44,"value":2238}," 250ms ",{"type":39,"tag":514,"props":2240,"children":2241},{"style":527},[2242],{"type":44,"value":1926},{"type":39,"tag":514,"props":2244,"children":2245},{"style":521},[2246],{"type":44,"value":2247}," +52% ",{"type":39,"tag":514,"props":2249,"children":2250},{"style":527},[2251],{"type":44,"value":1926},{"type":39,"tag":514,"props":2253,"children":2254},{"style":521},[2255],{"type":44,"value":2256}," Degrading ",{"type":39,"tag":514,"props":2258,"children":2259},{"style":527},[2260],{"type":44,"value":1926},{"type":39,"tag":514,"props":2262,"children":2263},{"style":521},[2264],{"type":44,"value":2265}," 500ms ",{"type":39,"tag":514,"props":2267,"children":2268},{"style":527},[2269],{"type":44,"value":1926},{"type":39,"tag":514,"props":2271,"children":2272},{"style":521},[2273],{"type":44,"value":2274}," 24% ",{"type":39,"tag":514,"props":2276,"children":2277},{"style":527},[2278],{"type":44,"value":1972},{"type":39,"tag":514,"props":2280,"children":2282},{"class":516,"line":2281},19,[2283,2287,2292,2296,2301,2305,2310,2314,2319,2323,2328,2332,2336,2340,2345,2349,2354],{"type":39,"tag":514,"props":2284,"children":2285},{"style":527},[2286],{"type":44,"value":1926},{"type":39,"tag":514,"props":2288,"children":2289},{"style":521},[2290],{"type":44,"value":2291}," http_req_failed ",{"type":39,"tag":514,"props":2293,"children":2294},{"style":527},[2295],{"type":44,"value":1926},{"type":39,"tag":514,"props":2297,"children":2298},{"style":521},[2299],{"type":44,"value":2300}," rate ",{"type":39,"tag":514,"props":2302,"children":2303},{"style":527},[2304],{"type":44,"value":1926},{"type":39,"tag":514,"props":2306,"children":2307},{"style":521},[2308],{"type":44,"value":2309}," 0.8% ",{"type":39,"tag":514,"props":2311,"children":2312},{"style":527},[2313],{"type":44,"value":1926},{"type":39,"tag":514,"props":2315,"children":2316},{"style":521},[2317],{"type":44,"value":2318}," 0.3% ",{"type":39,"tag":514,"props":2320,"children":2321},{"style":527},[2322],{"type":44,"value":1926},{"type":39,"tag":514,"props":2324,"children":2325},{"style":521},[2326],{"type":44,"value":2327}," +167% ",{"type":39,"tag":514,"props":2329,"children":2330},{"style":527},[2331],{"type":44,"value":1926},{"type":39,"tag":514,"props":2333,"children":2334},{"style":521},[2335],{"type":44,"value":2256},{"type":39,"tag":514,"props":2337,"children":2338},{"style":527},[2339],{"type":44,"value":1926},{"type":39,"tag":514,"props":2341,"children":2342},{"style":521},[2343],{"type":44,"value":2344}," 1% ",{"type":39,"tag":514,"props":2346,"children":2347},{"style":527},[2348],{"type":44,"value":1926},{"type":39,"tag":514,"props":2350,"children":2351},{"style":521},[2352],{"type":44,"value":2353}," 20% ",{"type":39,"tag":514,"props":2355,"children":2356},{"style":527},[2357],{"type":44,"value":1972},{"type":39,"tag":514,"props":2359,"children":2361},{"class":516,"line":2360},20,[2362,2366,2371,2375,2380,2384,2389,2393,2398,2402,2407,2411,2416,2420,2425,2430,2434],{"type":39,"tag":514,"props":2363,"children":2364},{"style":527},[2365],{"type":44,"value":1926},{"type":39,"tag":514,"props":2367,"children":2368},{"style":521},[2369],{"type":44,"value":2370}," http_reqs ",{"type":39,"tag":514,"props":2372,"children":2373},{"style":527},[2374],{"type":44,"value":1926},{"type":39,"tag":514,"props":2376,"children":2377},{"style":521},[2378],{"type":44,"value":2379}," counter ",{"type":39,"tag":514,"props":2381,"children":2382},{"style":527},[2383],{"type":44,"value":1926},{"type":39,"tag":514,"props":2385,"children":2386},{"style":521},[2387],{"type":44,"value":2388}," 15,230 ",{"type":39,"tag":514,"props":2390,"children":2391},{"style":527},[2392],{"type":44,"value":1926},{"type":39,"tag":514,"props":2394,"children":2395},{"style":521},[2396],{"type":44,"value":2397}," 15,100 ",{"type":39,"tag":514,"props":2399,"children":2400},{"style":527},[2401],{"type":44,"value":1926},{"type":39,"tag":514,"props":2403,"children":2404},{"style":521},[2405],{"type":44,"value":2406}," +0.9% ",{"type":39,"tag":514,"props":2408,"children":2409},{"style":527},[2410],{"type":44,"value":1926},{"type":39,"tag":514,"props":2412,"children":2413},{"style":521},[2414],{"type":44,"value":2415}," Stable ",{"type":39,"tag":514,"props":2417,"children":2418},{"style":527},[2419],{"type":44,"value":1926},{"type":39,"tag":514,"props":2421,"children":2422},{"style":527},[2423],{"type":44,"value":2424}," -",{"type":39,"tag":514,"props":2426,"children":2427},{"style":527},[2428],{"type":44,"value":2429}," |",{"type":39,"tag":514,"props":2431,"children":2432},{"style":527},[2433],{"type":44,"value":2424},{"type":39,"tag":514,"props":2435,"children":2436},{"style":527},[2437],{"type":44,"value":2438}," |\n",{"type":39,"tag":514,"props":2440,"children":2442},{"class":516,"line":2441},21,[2443],{"type":39,"tag":514,"props":2444,"children":2445},{"emptyLinePlaceholder":1816},[2446],{"type":44,"value":1819},{"type":39,"tag":514,"props":2448,"children":2450},{"class":516,"line":2449},22,[2451,2455],{"type":39,"tag":514,"props":2452,"children":2453},{"style":527},[2454],{"type":44,"value":1904},{"type":39,"tag":514,"props":2456,"children":2457},{"style":573},[2458],{"type":44,"value":2459},"Flagged Issues\n",{"type":39,"tag":514,"props":2461,"children":2463},{"class":516,"line":2462},23,[2464],{"type":39,"tag":514,"props":2465,"children":2466},{"emptyLinePlaceholder":1816},[2467],{"type":44,"value":1819},{"type":39,"tag":514,"props":2469,"children":2471},{"class":516,"line":2470},24,[2472,2477],{"type":39,"tag":514,"props":2473,"children":2474},{"style":527},[2475],{"type":44,"value":2476},"### ",{"type":39,"tag":514,"props":2478,"children":2479},{"style":573},[2480],{"type":44,"value":2481},"1. {metric_name}: {classification}\n",{"type":39,"tag":514,"props":2483,"children":2485},{"class":516,"line":2484},25,[2486,2491,2496,2501,2505,2510,2514,2519,2523,2527,2531,2536,2540],{"type":39,"tag":514,"props":2487,"children":2488},{"style":527},[2489],{"type":44,"value":2490},"-",{"type":39,"tag":514,"props":2492,"children":2493},{"style":1825},[2494],{"type":44,"value":2495}," **",{"type":39,"tag":514,"props":2497,"children":2498},{"style":1831},[2499],{"type":44,"value":2500},"Current",{"type":39,"tag":514,"props":2502,"children":2503},{"style":1825},[2504],{"type":44,"value":1828},{"type":39,"tag":514,"props":2506,"children":2507},{"style":521},[2508],{"type":44,"value":2509},": {value} | ",{"type":39,"tag":514,"props":2511,"children":2512},{"style":1825},[2513],{"type":44,"value":1828},{"type":39,"tag":514,"props":2515,"children":2516},{"style":1831},[2517],{"type":44,"value":2518},"Baseline",{"type":39,"tag":514,"props":2520,"children":2521},{"style":1825},[2522],{"type":44,"value":1828},{"type":39,"tag":514,"props":2524,"children":2525},{"style":521},[2526],{"type":44,"value":2509},{"type":39,"tag":514,"props":2528,"children":2529},{"style":1825},[2530],{"type":44,"value":1828},{"type":39,"tag":514,"props":2532,"children":2533},{"style":1831},[2534],{"type":44,"value":2535},"Change",{"type":39,"tag":514,"props":2537,"children":2538},{"style":1825},[2539],{"type":44,"value":1828},{"type":39,"tag":514,"props":2541,"children":2542},{"style":521},[2543],{"type":44,"value":2544},": {pct}%\n",{"type":39,"tag":514,"props":2546,"children":2548},{"class":516,"line":2547},26,[2549,2553,2557,2562,2566,2571,2575,2580,2584],{"type":39,"tag":514,"props":2550,"children":2551},{"style":527},[2552],{"type":44,"value":2490},{"type":39,"tag":514,"props":2554,"children":2555},{"style":1825},[2556],{"type":44,"value":2495},{"type":39,"tag":514,"props":2558,"children":2559},{"style":1831},[2560],{"type":44,"value":2561},"Threshold",{"type":39,"tag":514,"props":2563,"children":2564},{"style":1825},[2565],{"type":44,"value":1828},{"type":39,"tag":514,"props":2567,"children":2568},{"style":521},[2569],{"type":44,"value":2570},": {threshold} | ",{"type":39,"tag":514,"props":2572,"children":2573},{"style":1825},[2574],{"type":44,"value":1828},{"type":39,"tag":514,"props":2576,"children":2577},{"style":1831},[2578],{"type":44,"value":2579},"Headroom",{"type":39,"tag":514,"props":2581,"children":2582},{"style":1825},[2583],{"type":44,"value":1828},{"type":39,"tag":514,"props":2585,"children":2586},{"style":521},[2587],{"type":44,"value":2544},{"type":39,"tag":514,"props":2589,"children":2591},{"class":516,"line":2590},27,[2592,2596,2600,2605,2609],{"type":39,"tag":514,"props":2593,"children":2594},{"style":527},[2595],{"type":44,"value":2490},{"type":39,"tag":514,"props":2597,"children":2598},{"style":1825},[2599],{"type":44,"value":2495},{"type":39,"tag":514,"props":2601,"children":2602},{"style":1831},[2603],{"type":44,"value":2604},"Rate of change",{"type":39,"tag":514,"props":2606,"children":2607},{"style":1825},[2608],{"type":44,"value":1828},{"type":39,"tag":514,"props":2610,"children":2611},{"style":521},[2612],{"type":44,"value":2613},": {pct}% per week\n",{"type":39,"tag":514,"props":2615,"children":2617},{"class":516,"line":2616},28,[2618,2622,2626,2631,2635],{"type":39,"tag":514,"props":2619,"children":2620},{"style":527},[2621],{"type":44,"value":2490},{"type":39,"tag":514,"props":2623,"children":2624},{"style":1825},[2625],{"type":44,"value":2495},{"type":39,"tag":514,"props":2627,"children":2628},{"style":1831},[2629],{"type":44,"value":2630},"Projected breach",{"type":39,"tag":514,"props":2632,"children":2633},{"style":1825},[2634],{"type":44,"value":1828},{"type":39,"tag":514,"props":2636,"children":2637},{"style":521},[2638],{"type":44,"value":2639},": ~{N} weeks at current rate\n",{"type":39,"tag":514,"props":2641,"children":2643},{"class":516,"line":2642},29,[2644,2648,2652,2657,2661],{"type":39,"tag":514,"props":2645,"children":2646},{"style":527},[2647],{"type":44,"value":2490},{"type":39,"tag":514,"props":2649,"children":2650},{"style":1825},[2651],{"type":44,"value":2495},{"type":39,"tag":514,"props":2653,"children":2654},{"style":1831},[2655],{"type":44,"value":2656},"Anomalous runs",{"type":39,"tag":514,"props":2658,"children":2659},{"style":1825},[2660],{"type":44,"value":1828},{"type":39,"tag":514,"props":2662,"children":2663},{"style":521},[2664],{"type":44,"value":2665},": {run_ids with dates, if any}\n",{"type":39,"tag":514,"props":2667,"children":2669},{"class":516,"line":2668},30,[2670],{"type":39,"tag":514,"props":2671,"children":2672},{"emptyLinePlaceholder":1816},[2673],{"type":44,"value":1819},{"type":39,"tag":514,"props":2675,"children":2677},{"class":516,"line":2676},31,[2678,2682],{"type":39,"tag":514,"props":2679,"children":2680},{"style":527},[2681],{"type":44,"value":1904},{"type":39,"tag":514,"props":2683,"children":2684},{"style":573},[2685],{"type":44,"value":2686},"Threshold Recommendations\n",{"type":39,"tag":514,"props":2688,"children":2690},{"class":516,"line":2689},32,[2691],{"type":39,"tag":514,"props":2692,"children":2693},{"emptyLinePlaceholder":1816},[2694],{"type":44,"value":1819},{"type":39,"tag":514,"props":2696,"children":2698},{"class":516,"line":2697},33,[2699,2703,2707,2711,2716,2720,2725,2729,2734],{"type":39,"tag":514,"props":2700,"children":2701},{"style":527},[2702],{"type":44,"value":1926},{"type":39,"tag":514,"props":2704,"children":2705},{"style":521},[2706],{"type":44,"value":2122},{"type":39,"tag":514,"props":2708,"children":2709},{"style":527},[2710],{"type":44,"value":1926},{"type":39,"tag":514,"props":2712,"children":2713},{"style":521},[2714],{"type":44,"value":2715}," Current Threshold ",{"type":39,"tag":514,"props":2717,"children":2718},{"style":527},[2719],{"type":44,"value":1926},{"type":39,"tag":514,"props":2721,"children":2722},{"style":521},[2723],{"type":44,"value":2724}," Recommended ",{"type":39,"tag":514,"props":2726,"children":2727},{"style":527},[2728],{"type":44,"value":1926},{"type":39,"tag":514,"props":2730,"children":2731},{"style":521},[2732],{"type":44,"value":2733}," Rationale ",{"type":39,"tag":514,"props":2735,"children":2736},{"style":527},[2737],{"type":44,"value":1972},{"type":39,"tag":514,"props":2739,"children":2741},{"class":516,"line":2740},34,[2742],{"type":39,"tag":514,"props":2743,"children":2744},{"style":527},[2745],{"type":44,"value":2746},"|--------|-------------------|-------------|-----------|\n",{"type":39,"tag":514,"props":2748,"children":2750},{"class":516,"line":2749},35,[2751,2755,2760,2764,2769,2773,2778,2782,2787],{"type":39,"tag":514,"props":2752,"children":2753},{"style":527},[2754],{"type":44,"value":1926},{"type":39,"tag":514,"props":2756,"children":2757},{"style":521},[2758],{"type":44,"value":2759}," http_req_duration ",{"type":39,"tag":514,"props":2761,"children":2762},{"style":527},[2763],{"type":44,"value":1926},{"type":39,"tag":514,"props":2765,"children":2766},{"style":521},[2767],{"type":44,"value":2768}," p(95)\u003C500 ",{"type":39,"tag":514,"props":2770,"children":2771},{"style":527},[2772],{"type":44,"value":1926},{"type":39,"tag":514,"props":2774,"children":2775},{"style":521},[2776],{"type":44,"value":2777}," p(95)\u003C420 ",{"type":39,"tag":514,"props":2779,"children":2780},{"style":527},[2781],{"type":44,"value":1926},{"type":39,"tag":514,"props":2783,"children":2784},{"style":521},[2785],{"type":44,"value":2786}," Current P95 is 380ms; tightening to 420ms gives 10% headroom from current performance while surfacing further degradation early ",{"type":39,"tag":514,"props":2788,"children":2789},{"style":527},[2790],{"type":44,"value":1972},{"type":39,"tag":514,"props":2792,"children":2794},{"class":516,"line":2793},36,[2795],{"type":39,"tag":514,"props":2796,"children":2797},{"emptyLinePlaceholder":1816},[2798],{"type":44,"value":1819},{"type":39,"tag":514,"props":2800,"children":2802},{"class":516,"line":2801},37,[2803,2807],{"type":39,"tag":514,"props":2804,"children":2805},{"style":527},[2806],{"type":44,"value":1904},{"type":39,"tag":514,"props":2808,"children":2809},{"style":573},[2810],{"type":44,"value":2811},"Suggested Next Steps\n",{"type":39,"tag":514,"props":2813,"children":2815},{"class":516,"line":2814},38,[2816],{"type":39,"tag":514,"props":2817,"children":2818},{"emptyLinePlaceholder":1816},[2819],{"type":44,"value":1819},{"type":39,"tag":514,"props":2821,"children":2823},{"class":516,"line":2822},39,[2824,2828,2833,2837,2842,2846],{"type":39,"tag":514,"props":2825,"children":2826},{"style":527},[2827],{"type":44,"value":2490},{"type":39,"tag":514,"props":2829,"children":2830},{"style":521},[2831],{"type":44,"value":2832}," [ ] ",{"type":39,"tag":514,"props":2834,"children":2835},{"style":1825},[2836],{"type":44,"value":1828},{"type":39,"tag":514,"props":2838,"children":2839},{"style":1831},[2840],{"type":44,"value":2841},"Investigate service side",{"type":39,"tag":514,"props":2843,"children":2844},{"style":1825},[2845],{"type":44,"value":1828},{"type":39,"tag":514,"props":2847,"children":2848},{"style":521},[2849],{"type":44,"value":2850},": P95 latency has increased 52% -- consider\n",{"type":39,"tag":514,"props":2852,"children":2854},{"class":516,"line":2853},40,[2855,2860,2865,2869,2873],{"type":39,"tag":514,"props":2856,"children":2857},{"style":521},[2858],{"type":44,"value":2859},"      loading ",{"type":39,"tag":514,"props":2861,"children":2862},{"style":527},[2863],{"type":44,"value":2864},"`",{"type":39,"tag":514,"props":2866,"children":2867},{"style":533},[2868],{"type":44,"value":122},{"type":39,"tag":514,"props":2870,"children":2871},{"style":527},[2872],{"type":44,"value":2864},{"type":39,"tag":514,"props":2874,"children":2875},{"style":521},[2876],{"type":44,"value":2877}," to check service health\n",{"type":39,"tag":514,"props":2879,"children":2881},{"class":516,"line":2880},41,[2882,2886,2890,2894,2899,2903],{"type":39,"tag":514,"props":2883,"children":2884},{"style":527},[2885],{"type":44,"value":2490},{"type":39,"tag":514,"props":2887,"children":2888},{"style":521},[2889],{"type":44,"value":2832},{"type":39,"tag":514,"props":2891,"children":2892},{"style":1825},[2893],{"type":44,"value":1828},{"type":39,"tag":514,"props":2895,"children":2896},{"style":1831},[2897],{"type":44,"value":2898},"Deep-dive run {run_id}",{"type":39,"tag":514,"props":2900,"children":2901},{"style":1825},[2902],{"type":44,"value":1828},{"type":39,"tag":514,"props":2904,"children":2905},{"style":521},[2906],{"type":44,"value":2907},": anomalous P95 spike on {date} -- consider\n",{"type":39,"tag":514,"props":2909,"children":2911},{"class":516,"line":2910},42,[2912,2916,2920,2924,2928],{"type":39,"tag":514,"props":2913,"children":2914},{"style":521},[2915],{"type":44,"value":2859},{"type":39,"tag":514,"props":2917,"children":2918},{"style":527},[2919],{"type":44,"value":2864},{"type":39,"tag":514,"props":2921,"children":2922},{"style":533},[2923],{"type":44,"value":81},{"type":39,"tag":514,"props":2925,"children":2926},{"style":527},[2927],{"type":44,"value":2864},{"type":39,"tag":514,"props":2929,"children":2930},{"style":521},[2931],{"type":44,"value":2932}," for this run\n",{"type":39,"tag":514,"props":2934,"children":2936},{"class":516,"line":2935},43,[2937,2941,2945,2949,2954,2958],{"type":39,"tag":514,"props":2938,"children":2939},{"style":527},[2940],{"type":44,"value":2490},{"type":39,"tag":514,"props":2942,"children":2943},{"style":521},[2944],{"type":44,"value":2832},{"type":39,"tag":514,"props":2946,"children":2947},{"style":1825},[2948],{"type":44,"value":1828},{"type":39,"tag":514,"props":2950,"children":2951},{"style":1831},[2952],{"type":44,"value":2953},"Tighten thresholds",{"type":39,"tag":514,"props":2955,"children":2956},{"style":1825},[2957],{"type":44,"value":1828},{"type":39,"tag":514,"props":2959,"children":2960},{"style":521},[2961],{"type":44,"value":2962},": 2 metrics have >30% headroom that could be\n",{"type":39,"tag":514,"props":2964,"children":2966},{"class":516,"line":2965},44,[2967,2972,2976,2980,2984],{"type":39,"tag":514,"props":2968,"children":2969},{"style":521},[2970],{"type":44,"value":2971},"      tightened -- consider loading ",{"type":39,"tag":514,"props":2973,"children":2974},{"style":527},[2975],{"type":44,"value":2864},{"type":39,"tag":514,"props":2977,"children":2978},{"style":533},[2979],{"type":44,"value":96},{"type":39,"tag":514,"props":2981,"children":2982},{"style":527},[2983],{"type":44,"value":2864},{"type":39,"tag":514,"props":2985,"children":2986},{"style":521},[2987],{"type":44,"value":2988}," to apply changes\n",{"type":39,"tag":53,"props":2990,"children":2992},{"id":2991},"overall-health-classification",[2993],{"type":44,"value":2994},"Overall health classification",{"type":39,"tag":47,"props":2996,"children":2997},{},[2998],{"type":44,"value":2999},"Derive the overall health from the worst-case metric:",{"type":39,"tag":60,"props":3001,"children":3002},{},[3003,3013,3023,3032],{"type":39,"tag":64,"props":3004,"children":3005},{},[3006,3011],{"type":39,"tag":68,"props":3007,"children":3008},{},[3009],{"type":44,"value":3010},"Healthy",{"type":44,"value":3012},": all metrics stable or improving, headroom comfortable or adequate",{"type":39,"tag":64,"props":3014,"children":3015},{},[3016,3021],{"type":39,"tag":68,"props":3017,"children":3018},{},[3019],{"type":44,"value":3020},"Watch",{"type":44,"value":3022},": at least one metric degrading but headroom still adequate",{"type":39,"tag":64,"props":3024,"children":3025},{},[3026,3030],{"type":39,"tag":68,"props":3027,"children":3028},{},[3029],{"type":44,"value":1440},{"type":44,"value":3031},": at least one metric degrading with thin headroom",{"type":39,"tag":64,"props":3033,"children":3034},{},[3035,3040],{"type":39,"tag":68,"props":3036,"children":3037},{},[3038],{"type":44,"value":3039},"Critical",{"type":44,"value":3041},": at least one metric has breached its threshold, or multiple\nmetrics are degrading with thin headroom",{"type":39,"tag":180,"props":3043,"children":3044},{},[],{"type":39,"tag":53,"props":3046,"children":3048},{"id":3047},"threshold-recommendations",[3049],{"type":44,"value":3050},"Threshold recommendations",{"type":39,"tag":47,"props":3052,"children":3053},{},[3054],{"type":44,"value":3055},"When recommending threshold changes, follow these principles:",{"type":39,"tag":60,"props":3057,"children":3058},{},[3059,3069,3079,3089,3099],{"type":39,"tag":64,"props":3060,"children":3061},{},[3062,3067],{"type":39,"tag":68,"props":3063,"children":3064},{},[3065],{"type":44,"value":3066},"Only tighten, never loosen",{"type":44,"value":3068}," unless the user asks. Loosening thresholds\nmasks problems.",{"type":39,"tag":64,"props":3070,"children":3071},{},[3072,3077],{"type":39,"tag":68,"props":3073,"children":3074},{},[3075],{"type":44,"value":3076},"Target 10-20% headroom",{"type":44,"value":3078}," above the recent P95 of the metric. Enough room\nfor normal variance but tight enough to catch real degradation.",{"type":39,"tag":64,"props":3080,"children":3081},{},[3082,3087],{"type":39,"tag":68,"props":3083,"children":3084},{},[3085],{"type":44,"value":3086},"Use the second-half mean as the baseline",{"type":44,"value":3088},", not the single most recent run\n(which could be an outlier).",{"type":39,"tag":64,"props":3090,"children":3091},{},[3092,3097],{"type":39,"tag":68,"props":3093,"children":3094},{},[3095],{"type":44,"value":3096},"Respect the user's intent",{"type":44,"value":3098},": if thresholds are currently very loose\n(>100% headroom), they may be intentionally permissive. Mention the\nopportunity to tighten but don't push hard -- the user knows their context.",{"type":39,"tag":64,"props":3100,"children":3101},{},[3102,3107,3109,3114],{"type":39,"tag":68,"props":3103,"children":3104},{},[3105],{"type":44,"value":3106},"Consider grouped thresholds",{"type":44,"value":3108},": if the test uses tag-based thresholds\n(e.g., ",{"type":39,"tag":76,"props":3110,"children":3112},{"className":3111},[],[3113],{"type":44,"value":1103},{"type":44,"value":3115},"), recommend per-endpoint\nthresholds where the trends differ between endpoints.",{"type":39,"tag":180,"props":3117,"children":3118},{},[],{"type":39,"tag":53,"props":3120,"children":3122},{"id":3121},"gotchas",[3123],{"type":44,"value":3124},"Gotchas",{"type":39,"tag":656,"props":3126,"children":3127},{},[3128,3144],{"type":39,"tag":660,"props":3129,"children":3130},{},[3131],{"type":39,"tag":664,"props":3132,"children":3133},{},[3134,3139],{"type":39,"tag":668,"props":3135,"children":3136},{},[3137],{"type":44,"value":3138},"Issue",{"type":39,"tag":668,"props":3140,"children":3141},{},[3142],{"type":44,"value":3143},"Detail",{"type":39,"tag":684,"props":3145,"children":3146},{},[3147,3163,3179,3195,3223],{"type":39,"tag":664,"props":3148,"children":3149},{},[3150,3158],{"type":39,"tag":691,"props":3151,"children":3152},{},[3153],{"type":39,"tag":68,"props":3154,"children":3155},{},[3156],{"type":44,"value":3157},"Zero-observation thresholds",{"type":39,"tag":691,"props":3159,"children":3160},{},[3161],{"type":44,"value":3162},"A threshold with zero observations passes by default in k6. If a metric appears to pass but has no data, flag it -- the threshold is not actually being evaluated.",{"type":39,"tag":664,"props":3164,"children":3165},{},[3166,3174],{"type":39,"tag":691,"props":3167,"children":3168},{},[3169],{"type":39,"tag":68,"props":3170,"children":3171},{},[3172],{"type":44,"value":3173},"Metric type changes across runs",{"type":39,"tag":691,"props":3175,"children":3176},{},[3177],{"type":44,"value":3178},"If a metric's type changed between runs (e.g., script refactor), the multi-run aggregate endpoint uses the latest type. Earlier runs queried with the wrong method return empty. Flag this if detected.",{"type":39,"tag":664,"props":3180,"children":3181},{},[3182,3190],{"type":39,"tag":691,"props":3183,"children":3184},{},[3185],{"type":39,"tag":68,"props":3186,"children":3187},{},[3188],{"type":44,"value":3189},"Incomplete runs skew trends",{"type":39,"tag":691,"props":3191,"children":3192},{},[3193],{"type":44,"value":3194},"Aborted or timed-out runs typically have shorter durations and fewer iterations, producing unrepresentative metric values. Exclude them by default.",{"type":39,"tag":664,"props":3196,"children":3197},{},[3198,3206],{"type":39,"tag":691,"props":3199,"children":3200},{},[3201],{"type":39,"tag":68,"props":3202,"children":3203},{},[3204],{"type":44,"value":3205},"LG resource metrics",{"type":39,"tag":691,"props":3207,"children":3208},{},[3209,3214,3215,3221],{"type":39,"tag":76,"props":3210,"children":3212},{"className":3211},[],[3213],{"type":44,"value":1050},{"type":44,"value":450},{"type":39,"tag":76,"props":3216,"children":3218},{"className":3217},[],[3219],{"type":44,"value":3220},"load_generator_file_handles",{"type":44,"value":3222}," trending up may indicate the test is outgrowing its load generator allocation, not that the service is degrading. Call this out separately.",{"type":39,"tag":664,"props":3224,"children":3225},{},[3226,3234],{"type":39,"tag":691,"props":3227,"children":3228},{},[3229],{"type":39,"tag":68,"props":3230,"children":3231},{},[3232],{"type":44,"value":3233},"Rate metric direction",{"type":39,"tag":691,"props":3235,"children":3236},{},[3237,3239,3244,3246,3252],{"type":44,"value":3238},"For ",{"type":39,"tag":76,"props":3240,"children":3242},{"className":3241},[],[3243],{"type":44,"value":805},{"type":44,"value":3245},"-type rate metrics (like check pass rates), \"degrading\" means the value is ",{"type":39,"tag":3247,"props":3248,"children":3249},"em",{},[3250],{"type":44,"value":3251},"decreasing",{"type":44,"value":3253}," (fewer passes), which is the opposite direction from latency metrics.",{"type":39,"tag":3255,"props":3256,"children":3257},"style",{},[3258],{"type":44,"value":3259},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3261,"total":3376},[3262,3279,3296,3312,3327,3343,3361],{"slug":3263,"name":3263,"fn":3264,"description":3265,"org":3266,"tags":3267,"stars":23,"repoUrl":24,"updatedAt":3278},"adaptive-metrics","optimize Grafana Cloud metrics costs","Cut Grafana Cloud Metrics cost by shrinking active-series count with Adaptive Metrics aggregation rules — auto-recommendations from query history, custom exact\u002Fregex rules, label-drop config, unused-metric detection, and Alloy remote_write fallback. Use when investigating a high Mimir\u002FGrafana Cloud bill, hunting high-cardinality labels (`pod_uid`, `service_instance_id`, `version`), pre-aggregating counters\u002Fgauges, dropping unused metrics, or measuring `grafanacloud_instance_active_series` before\u002Fafter — even when the user says \"reduce cardinality\", \"too many series\", \"metrics spend\", \"active series count is exploding\", or \"drop the version label\" without naming Adaptive Metrics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3268,3271,3272,3275],{"name":3269,"slug":3270,"type":15},"Cost Optimization","cost-optimization",{"name":9,"slug":8,"type":15},{"name":3273,"slug":3274,"type":15},"Metrics","metrics",{"name":3276,"slug":3277,"type":15},"Observability","observability","2026-07-12T07:44:27.451068",{"slug":3280,"name":3280,"fn":3281,"description":3282,"org":3283,"tags":3284,"stars":23,"repoUrl":24,"updatedAt":3295},"admin","manage Grafana Cloud accounts and RBAC","Manage Grafana Cloud accounts — organizations, stacks, RBAC roles and assignments, SSO\u002FSAML\u002FOAuth\u002FGitHub auth, service accounts for CI\u002FCD, user invites, team membership, and API-driven provisioning. Creates stacks via the Cloud API, mints service-account tokens, applies role assignments, configures SSO providers, and provisions teams\u002Ffolders\u002Fdashboards via Terraform. Use when managing Grafana Cloud access, configuring SSO\u002FSAML\u002FOAuth, setting up service accounts for Terraform\u002FCI\u002FCD, assigning RBAC roles, inviting users, managing multiple stacks or organizations, provisioning cloud resources via API or Terraform, or auditing admin actions — even when the user says \"set up SSO\", \"create a stack\", \"make a service account\", or \"onboard a team\" without explicitly saying \"admin\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3285,3288,3291,3292],{"name":3286,"slug":3287,"type":15},"Access Control","access-control",{"name":3289,"slug":3290,"type":15},"Auth","auth",{"name":9,"slug":8,"type":15},{"name":3293,"slug":3294,"type":15},"Operations","operations","2026-07-12T07:44:12.078436",{"slug":3297,"name":3297,"fn":3298,"description":3299,"org":3300,"tags":3301,"stars":23,"repoUrl":24,"updatedAt":3311},"admission-control","implement admission control webhooks","Use when the user asks to \"write a validator\", \"add validation\", \"implement admission control\", \"write a mutating webhook\", \"add a mutation handler\", \"validate incoming resources\", \"implement admission logic\", \"add admission webhooks\", \"write ingress validation\", or asks how to validate or mutate resources before they are persisted in a grafana-app-sdk app. Provides guidance on implementing validation and mutation admission handlers for grafana-app-sdk apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3302,3305,3308],{"name":3303,"slug":3304,"type":15},"Architecture","architecture",{"name":3306,"slug":3307,"type":15},"Security","security",{"name":3309,"slug":3310,"type":15},"Validation","validation","2026-07-12T07:45:06.148973",{"slug":3313,"name":3313,"fn":3314,"description":3315,"org":3316,"tags":3317,"stars":23,"repoUrl":24,"updatedAt":3326},"alerting-irm","configure Grafana Alerting and Incident Management","Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack\u002FPagerDuty\u002Femail\u002Fwebhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty\u002FSlack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says \"page me on errors\", \"alert me when X happens\", \"route this to the platform team\", or \"set up an SLO\" without naming Alerting or IRM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3318,3321,3322,3325],{"name":3319,"slug":3320,"type":15},"Alerting","alerting",{"name":9,"slug":8,"type":15},{"name":3323,"slug":3324,"type":15},"Incident Response","incident-response",{"name":17,"slug":18,"type":15},"2026-07-12T07:44:02.393397",{"slug":3328,"name":3328,"fn":3329,"description":3330,"org":3331,"tags":3332,"stars":23,"repoUrl":24,"updatedAt":3342},"alloy","build unified telemetry pipelines with Grafana Alloy","Build a unified telemetry pipeline with Grafana Alloy — one OpenTelemetry-compatible binary that collects metrics, logs, traces, and profiles and ships to Grafana Cloud \u002F Prometheus \u002F Loki \u002F Tempo \u002F Pyroscope. Covers the Alloy config language (blocks, `sys.env`, component refs), `prometheus.scrape` → `remote_write`, `loki.source.file` + `loki.process` → `loki.write`, `otelcol.receiver.otlp` → `otelcol.exporter.otlp`, `pyroscope.scrape`, K8s \u002F Docker \u002F EC2 discovery, relabeling, modules (`import.file\u002Fgit\u002Fhttp`), clustering, Fleet Management `remotecfg`, the Alloy UI at `:12345`, and `alloy fmt` \u002F `alloy validate`. Use when writing a `config.alloy`, replacing Grafana Agent \u002F OTel Collector, scraping K8s pods, parsing logs, ingesting OTLP, or debugging \"Alloy isn't sending anything\" — even when the user says \"set up the agent\", \"write me a scrape config\", \"drop these logs before sending\", or \"OTel collector config\" without naming Alloy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3333,3334,3337,3338,3339],{"name":9,"slug":8,"type":15},{"name":3335,"slug":3336,"type":15},"Logs","logs",{"name":3273,"slug":3274,"type":15},{"name":3276,"slug":3277,"type":15},{"name":3340,"slug":3341,"type":15},"OpenTelemetry","opentelemetry","2026-07-12T07:43:54.817139",{"slug":3344,"name":3344,"fn":3345,"description":3346,"org":3347,"tags":3348,"stars":23,"repoUrl":24,"updatedAt":3360},"app-observability","monitor application performance in Grafana Cloud","Get RED metrics + service maps + frontend RUM + AI\u002FLLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50\u002Fp95\u002Fp99 latency, exemplar-to-trace, traces-to-logs \u002F profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token \u002F cost \u002F latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says \"set up APM\", \"show service map\", \"monitor browser perf\", \"session replay\", \"RUM SDK\", or \"watch our OpenAI bill\" without naming App \u002F Frontend \u002F AI Observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3349,3352,3355,3356,3359],{"name":3350,"slug":3351,"type":15},"APM","apm",{"name":3353,"slug":3354,"type":15},"Distributed Tracing","distributed-tracing",{"name":9,"slug":8,"type":15},{"name":3357,"slug":3358,"type":15},"LLM","llm",{"name":3276,"slug":3277,"type":15},"2026-07-12T07:44:34.500406",{"slug":3362,"name":3362,"fn":3363,"description":3364,"org":3365,"tags":3366,"stars":23,"repoUrl":24,"updatedAt":3375},"app-sdk-concepts","scaffold and configure Grafana apps","Use when starting any grafana-app-sdk work — scaffolding a Grafana app, initializing a Grafana App Platform app, picking a deployment mode (standalone operator \u002F grafana\u002Fapps \u002F frontend-only), wiring app-specific config, or onboarding to the SDK. Covers `grafana-app-sdk` CLI install, `project init` per deployment mode, project layout, the schema-centric workflow (CUE kinds → generated code → reconciler\u002Fadmission logic), and `SpecificConfig` for env-driven app configuration. Use even if the user just says \"make a new app\", \"Grafana app platform\", \"kinds and watchers\", \"operator scaffold\", or asks about `apps\u002F` inside the Grafana repo, without naming the SDK explicitly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3367,3368,3371,3374],{"name":3303,"slug":3304,"type":15},{"name":3369,"slug":3370,"type":15},"Deployment","deployment",{"name":3372,"slug":3373,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},"2026-07-12T07:45:08.595757",48,{"items":3378,"total":3556},[3379,3394,3413,3433,3448,3462,3475,3490,3507,3519,3532,3544],{"slug":3380,"name":3380,"fn":3381,"description":3382,"org":3383,"tags":3384,"stars":3391,"repoUrl":3392,"updatedAt":3393},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3385,3386,3389,3390],{"name":3353,"slug":3354,"type":15},{"name":3387,"slug":3388,"type":15},"Frontend","frontend",{"name":17,"slug":18,"type":15},{"name":3276,"slug":3277,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":3395,"name":3395,"fn":3396,"description":3397,"org":3398,"tags":3399,"stars":3410,"repoUrl":3411,"updatedAt":3412},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3400,3403,3406,3409],{"name":3401,"slug":3402,"type":15},"API Development","api-development",{"name":3404,"slug":3405,"type":15},"Authentication","authentication",{"name":3407,"slug":3408,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":3414,"name":3414,"fn":3415,"description":3416,"org":3417,"tags":3418,"stars":3410,"repoUrl":3411,"updatedAt":3432},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3419,3422,3425,3426,3429],{"name":3420,"slug":3421,"type":15},"CSV","csv",{"name":3423,"slug":3424,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":3427,"slug":3428,"type":15},"GraphQL","graphql",{"name":3430,"slug":3431,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":3434,"name":3434,"fn":3435,"description":3436,"org":3437,"tags":3438,"stars":3445,"repoUrl":3446,"updatedAt":3447},"agento11y","manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3439,3442,3443,3444],{"name":3440,"slug":3441,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":3276,"slug":3277,"type":15},430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-25T05:30:40.29622",{"slug":3449,"name":3449,"fn":3450,"description":3451,"org":3452,"tags":3453,"stars":3445,"repoUrl":3446,"updatedAt":3461},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3454,3455,3456,3459,3460],{"name":3440,"slug":3441,"type":15},{"name":9,"slug":8,"type":15},{"name":3457,"slug":3458,"type":15},"Instrumentation","instrumentation",{"name":3357,"slug":3358,"type":15},{"name":3276,"slug":3277,"type":15},"2026-07-31T05:53:52.580237",{"slug":3463,"name":3463,"fn":3464,"description":3465,"org":3466,"tags":3467,"stars":3445,"repoUrl":3446,"updatedAt":3474},"agento11y-prod-setup","setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3468,3469,3472,3473],{"name":3440,"slug":3441,"type":15},{"name":3470,"slug":3471,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":3276,"slug":3277,"type":15},"2026-07-31T05:53:53.576347",{"slug":3476,"name":3476,"fn":3477,"description":3478,"org":3479,"tags":3480,"stars":3445,"repoUrl":3446,"updatedAt":3489},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3481,3482,3483,3486],{"name":3440,"slug":3441,"type":15},{"name":9,"slug":8,"type":15},{"name":3484,"slug":3485,"type":15},"QA","qa",{"name":3487,"slug":3488,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":3491,"name":3491,"fn":3492,"description":3493,"org":3494,"tags":3495,"stars":3445,"repoUrl":3446,"updatedAt":3506},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3496,3499,3502,3505],{"name":3497,"slug":3498,"type":15},"Dashboards","dashboards",{"name":3500,"slug":3501,"type":15},"Data Visualization","data-visualization",{"name":3503,"slug":3504,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":122,"name":122,"fn":3508,"description":3509,"org":3510,"tags":3511,"stars":3445,"repoUrl":3446,"updatedAt":3518},"investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3512,3515,3516,3517],{"name":3513,"slug":3514,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":3323,"slug":3324,"type":15},{"name":3276,"slug":3277,"type":15},"2026-07-18T05:11:10.445428",{"slug":3520,"name":3520,"fn":3521,"description":3522,"org":3523,"tags":3524,"stars":3445,"repoUrl":3446,"updatedAt":3531},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3525,3526,3527,3530],{"name":3513,"slug":3514,"type":15},{"name":9,"slug":8,"type":15},{"name":3528,"slug":3529,"type":15},"Graph Analysis","graph-analysis",{"name":3276,"slug":3277,"type":15},"2026-07-25T05:30:39.380934",{"slug":176,"name":176,"fn":3533,"description":3534,"org":3535,"tags":3536,"stars":3445,"repoUrl":3446,"updatedAt":3543},"manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3537,3540,3541,3542],{"name":3538,"slug":3539,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":3293,"slug":3294,"type":15},"2026-07-31T05:53:50.587304",{"slug":3545,"name":3545,"fn":3546,"description":3547,"org":3548,"tags":3549,"stars":3445,"repoUrl":3446,"updatedAt":3555},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3550,3551,3552],{"name":3538,"slug":3539,"type":15},{"name":9,"slug":8,"type":15},{"name":3553,"slug":3554,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80]