[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-synth-investigate-check":3,"mdc--xyo2hd-key":34,"related-repo-grafana-synth-investigate-check":1328,"related-org-grafana-synth-investigate-check":1431},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"synth-investigate-check","diagnose Synthetic Monitoring check failures","Diagnoses why a Synthetic Monitoring check is failing - triages probe failures, classifies failure scope, runs per-probe breakdown, and identifies root cause. Use when the user wants to investigate a failing check. Trigger on phrases like \"why is my check failing\", \"investigate synthetic check\", \"probe failures\", \"check is down\". For check status overview use synth-check-status. For creating or managing checks use synth-manage-checks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,20],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Monitoring","monitoring",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Debugging","debugging",430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-12T07:43:47.1811",null,29,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A CLI for managing Grafana Cloud resources. Optimized for agentic usage.","https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx\u002Ftree\u002FHEAD\u002Fclaude-plugin\u002Fskills\u002Fsynth-investigate-check","---\nname: synth-investigate-check\ndescription: Diagnoses why a Synthetic Monitoring check is failing - triages probe failures, classifies failure scope, runs per-probe breakdown, and identifies root cause. Use when the user wants to investigate a failing check. Trigger on phrases like \"why is my check failing\", \"investigate synthetic check\", \"probe failures\", \"check is down\". For check status overview use synth-check-status. For creating or managing checks use synth-manage-checks.\nallowed-tools: Bash\n---\n\n# Synthetic Check Investigator\n\nInvestigate Synthetic Monitoring check failures by triaging probe data, classifying failure scope, and identifying root cause.\n\n## Core Principles\n\n1. Use gcx commands — do not call Grafana APIs directly\n2. Trust the user's expertise — skip background explanations\n3. Use `-o json` for agent processing, default format for user display\n4. Show timeline graphs for time-series data — they communicate trends faster than text\n5. Collect errors; report them at the end, not interleaved in workflow steps\n\n## Investigation Workflow\n\n### Step 1: Get Check Status (with early exit)\n\n```bash\ngcx synthetic-monitoring checks status \u003CID>\n```\n\nIf the user provided a name instead of ID, list first with a job glob (the numeric ID is the NAME suffix, e.g. `web-check-1001` is ID `1001`):\n```bash\ngcx synthetic-monitoring checks list --job '*\u003Cname>*'\n```\n\n**Early exit — OK:** Status command reports `OK` (success rate at or above the check's alertSensitivity threshold: high = 95%, medium\u002Fdefault = 90%, low = 75%).\nReport: \"Check `\u003Cjob>` is healthy. Success rate: \u003Crate>%. \u003Cprobe_count> probes up.\"\nStop unless the user asks for more.\n\n**Early exit — NODATA:** No Prometheus metrics available.\n1. Get check config to verify `enabled: true` (`gcx synthetic-monitoring checks get \u003CID> -o json | jq .spec.enabled`)\n2. If disabled: report \"Check is disabled — no metrics will appear until it is re-enabled.\"\n3. If enabled: report \"No metrics found. Check datasource config or whether the SM stack is healthy.\"\nStop after reporting.\n\n### Step 2: Get Check Configuration\n\n```bash\ngcx synthetic-monitoring checks get \u003CID> -o json\n```\n\nExtract: job name, target, check type (http\u002Fping\u002Fdns\u002Ftcp\u002Ftraceroute), probe list, frequency, timeout, alertSensitivity, enabled flag.\n\nFor HTTP checks also note: any assertion settings, TLS config, expected status codes.\n\n### Step 3: Timeline Triage\n\n```bash\ngcx synthetic-monitoring checks timeline \u003CID> --from now-1h --to now\n```\n\nShow the graph output to the user. Then analyze the pattern:\n\n| Pattern | Classification |\n|---------|---------------|\n| All probes at 0 (or near 0) | Target down |\n| Subset of probes at 0, others healthy | Regional \u002F network |\n| Intermittent drops across multiple probes | Flapping \u002F timeout |\n| All probes drop at a specific point in time | Sudden onset — possible deployment or config change |\n| Gradual decline | Degradation — timeout drift or resource exhaustion |\n\nUse a longer window if the failure started more than 1h ago:\n```bash\ngcx synthetic-monitoring checks timeline \u003CID> --from now-6h --to now\n```\n\n### Step 4: Classify Failure Scope and Map Probes\n\nGet the probe list for geographic mapping:\n```bash\ngcx synthetic-monitoring probes list -o json\n```\n\nCross-reference the probe names from the check config against each probe's `region` field. Map failing probes to their regions.\n\n**All probes failing:** Target\u002Fservice issue — likely target down, SSL error, or DNS failure.\n\n**Subset of probes failing:** Regional or network issue. Note which regions are affected:\n- Single region → ISP\u002FCDN routing issue or regional outage\n- Multiple contiguous regions → CDN edge or routing policy issue\n- Probe-specific → private probe infra issue (if using private probes)\n\n**Intermittent failures:** Flapping. Consider: rate limiting, timeout too tight, flaky connectivity.\n\n### Step 5: Per-Probe Breakdown via PromQL (when datasource is available)\n\nResolve datasource UID if not already known:\n```bash\ngcx datasources list --type prometheus\n```\n\nIf the filtered list comes back empty, the stack may leave the `type` field\nblank in list payloads (known issue) — rerun without `--type` and pick the\nPrometheus datasource by name.\n\nRun per-probe success rate to pinpoint failing probes (use `-o json` for parsing, `-o graph` to show the user):\n```bash\ngcx metrics query -d \u003Cdatasource-uid> \\\n  'avg by (probe) (probe_success{job=\"\u003Cjob>\",instance=\"\u003Ctarget>\"})' \\\n  --from now-1h --to now --step 1m -o graph\n```\n\nFor HTTP checks, also run HTTP phase latency to locate where time is spent:\n```bash\ngcx metrics query -d \u003Cdatasource-uid> \\\n  'avg by (phase) (probe_http_duration_seconds{job=\"\u003Cjob>\",instance=\"\u003Ctarget>\"})' \\\n  --from now-1h --to now --step 1m -o graph\n```\n\nFor SSL\u002FTLS cert expiry, DNS latency, per-probe error rates, and other patterns, see [sm-promql-patterns.md](references\u002Fsm-promql-patterns.md).\n\n### Step 6: Classify Failure Mode\n\nCross-reference signals against [failure-modes.md](references\u002Ffailure-modes.md) (full signal\u002Fcause\u002Fnext-action table and decision tree) to select the most likely failure mode:\n\n1. All probes failing + HTTP non-2xx or connection refused → **Target down**\n2. Subset of probes failing → **Regional\u002FCDN**\n3. TLS handshake error or cert expiry \u003C 14 days → **SSL\u002FTLS**\n4. DNS resolution errors across probes → **DNS resolution**\n5. All probes timing out, phase latency high in `connect` or `tls` → **Timeout**\n6. Probes reaching target but assertion fails (status code, body match) → **Content\u002Fassertion**\n7. Single private probe failing, public probes healthy → **Private probe infra**\n8. HTTP 429 responses, intermittent failures with backoff pattern → **Rate limiting**\n\n### Step 7: Diagnosis and Next Actions\n\nSynthesize findings into an actionable report (see Output Format below). Take next actions from the \"Next Action\" column of the failure mode's row in [failure-modes.md](references\u002Ffailure-modes.md).\n\nIf deeper investigation is needed (e.g., logs, infra repos), ask the user if they want to proceed.\n\nIf check config needs changes (probe selection, frequency, assertions), route to **synth-manage-checks**.\n\n## Output Format\n\n**Early exit (OK):**\n```\nCheck: \u003Cjob> (\u003Ctarget>)\nStatus: OK\nSuccess rate: \u003Crate>%\nProbes up: \u003Ccount>\u002F\u003Ctotal>\n```\n\n**Early exit (NODATA):**\n```\nCheck: \u003Cjob> (\u003Ctarget>)\nStatus: NODATA\nEnabled: \u003Cyes\u002Fno>\nNext: \u003Cdatasource check \u002F re-enable instruction>\n```\n\n**Full investigation:**\n```\nCheck: \u003Cjob> (\u003Ctarget>)\nType: \u003Chttp|ping|dns|tcp|traceroute>\nStatus: FAILING\nSuccess rate: \u003Crate>% (window: \u003Cfrom> – \u003Cto>)\n\n[Timeline graph]\n\nFailure classification: \u003CTarget down | Regional\u002FCDN | SSL\u002FTLS | DNS | Timeout | Content\u002Fassertion | Private probe infra | Rate limiting>\n\nAffected probes: \u003Ccount>\u002F\u003Ctotal>\n  - \u003Cprobe-name> (\u003Cregion>): failing since \u003Ctime>\n  - \u003Cprobe-name> (\u003Cregion>): intermittent\n\nOnset: \u003Ctime\u002Fduration or \"unknown\">\n\nDiagnosis:\n\u003C2-4 sentences describing what the data shows and the most likely cause>\n\nNext actions:\n1. \u003Caction>\n2. \u003Caction>\n3. \u003Caction>\n```\n\nUse minimal formatting. Avoid excessive bold text. Trust the user to prioritize.\n\n## Error Handling\n\n- `gcx synthetic-monitoring checks status` returns no rows: check ID may be wrong — list all checks and confirm\n- `gcx synthetic-monitoring probes list` fails: skip geographic mapping; classify probes by name where possible\n- `gcx metrics query` fails with datasource error: note it, skip PromQL steps, classify using timeline data only\n- Multiple checks match the search name: list all with IDs and targets, ask which to investigate\n- Timeline returns no data for the window: widen to `--from now-6h --to now` before concluding NODATA\n",{"data":35,"body":37},{"name":4,"description":6,"allowed-tools":36},"Bash",{"type":38,"children":39},"root",[40,49,55,62,101,107,114,175,196,243,276,286,320,326,380,385,390,396,459,464,557,562,621,627,632,667,680,690,700,719,729,735,740,773,794,814,930,935,1036,1050,1056,1069,1167,1173,1183,1188,1199,1205,1213,1223,1231,1240,1248,1257,1262,1268,1322],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"synthetic-check-investigator",[46],{"type":47,"value":48},"text","Synthetic Check Investigator",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53],{"type":47,"value":54},"Investigate Synthetic Monitoring check failures by triaging probe data, classifying failure scope, and identifying root cause.",{"type":41,"tag":56,"props":57,"children":59},"h2",{"id":58},"core-principles",[60],{"type":47,"value":61},"Core Principles",{"type":41,"tag":63,"props":64,"children":65},"ol",{},[66,72,77,91,96],{"type":41,"tag":67,"props":68,"children":69},"li",{},[70],{"type":47,"value":71},"Use gcx commands — do not call Grafana APIs directly",{"type":41,"tag":67,"props":73,"children":74},{},[75],{"type":47,"value":76},"Trust the user's expertise — skip background explanations",{"type":41,"tag":67,"props":78,"children":79},{},[80,82,89],{"type":47,"value":81},"Use ",{"type":41,"tag":83,"props":84,"children":86},"code",{"className":85},[],[87],{"type":47,"value":88},"-o json",{"type":47,"value":90}," for agent processing, default format for user display",{"type":41,"tag":67,"props":92,"children":93},{},[94],{"type":47,"value":95},"Show timeline graphs for time-series data — they communicate trends faster than text",{"type":41,"tag":67,"props":97,"children":98},{},[99],{"type":47,"value":100},"Collect errors; report them at the end, not interleaved in workflow steps",{"type":41,"tag":56,"props":102,"children":104},{"id":103},"investigation-workflow",[105],{"type":47,"value":106},"Investigation Workflow",{"type":41,"tag":108,"props":109,"children":111},"h3",{"id":110},"step-1-get-check-status-with-early-exit",[112],{"type":47,"value":113},"Step 1: Get Check Status (with early exit)",{"type":41,"tag":115,"props":116,"children":121},"pre",{"className":117,"code":118,"language":119,"meta":120,"style":120},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","gcx synthetic-monitoring checks status \u003CID>\n","bash","",[122],{"type":41,"tag":83,"props":123,"children":124},{"__ignoreMap":120},[125],{"type":41,"tag":126,"props":127,"children":130},"span",{"class":128,"line":129},"line",1,[131,137,143,148,153,159,164,170],{"type":41,"tag":126,"props":132,"children":134},{"style":133},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[135],{"type":47,"value":136},"gcx",{"type":41,"tag":126,"props":138,"children":140},{"style":139},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[141],{"type":47,"value":142}," synthetic-monitoring",{"type":41,"tag":126,"props":144,"children":145},{"style":139},[146],{"type":47,"value":147}," checks",{"type":41,"tag":126,"props":149,"children":150},{"style":139},[151],{"type":47,"value":152}," status",{"type":41,"tag":126,"props":154,"children":156},{"style":155},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[157],{"type":47,"value":158}," \u003C",{"type":41,"tag":126,"props":160,"children":161},{"style":139},[162],{"type":47,"value":163},"I",{"type":41,"tag":126,"props":165,"children":167},{"style":166},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[168],{"type":47,"value":169},"D",{"type":41,"tag":126,"props":171,"children":172},{"style":155},[173],{"type":47,"value":174},">\n",{"type":41,"tag":50,"props":176,"children":177},{},[178,180,186,188,194],{"type":47,"value":179},"If the user provided a name instead of ID, list first with a job glob (the numeric ID is the NAME suffix, e.g. ",{"type":41,"tag":83,"props":181,"children":183},{"className":182},[],[184],{"type":47,"value":185},"web-check-1001",{"type":47,"value":187}," is ID ",{"type":41,"tag":83,"props":189,"children":191},{"className":190},[],[192],{"type":47,"value":193},"1001",{"type":47,"value":195},"):",{"type":41,"tag":115,"props":197,"children":199},{"className":117,"code":198,"language":119,"meta":120,"style":120},"gcx synthetic-monitoring checks list --job '*\u003Cname>*'\n",[200],{"type":41,"tag":83,"props":201,"children":202},{"__ignoreMap":120},[203],{"type":41,"tag":126,"props":204,"children":205},{"class":128,"line":129},[206,210,214,218,223,228,233,238],{"type":41,"tag":126,"props":207,"children":208},{"style":133},[209],{"type":47,"value":136},{"type":41,"tag":126,"props":211,"children":212},{"style":139},[213],{"type":47,"value":142},{"type":41,"tag":126,"props":215,"children":216},{"style":139},[217],{"type":47,"value":147},{"type":41,"tag":126,"props":219,"children":220},{"style":139},[221],{"type":47,"value":222}," list",{"type":41,"tag":126,"props":224,"children":225},{"style":139},[226],{"type":47,"value":227}," --job",{"type":41,"tag":126,"props":229,"children":230},{"style":155},[231],{"type":47,"value":232}," '",{"type":41,"tag":126,"props":234,"children":235},{"style":139},[236],{"type":47,"value":237},"*\u003Cname>*",{"type":41,"tag":126,"props":239,"children":240},{"style":155},[241],{"type":47,"value":242},"'\n",{"type":41,"tag":50,"props":244,"children":245},{},[246,252,254,260,262,268,270],{"type":41,"tag":247,"props":248,"children":249},"strong",{},[250],{"type":47,"value":251},"Early exit — OK:",{"type":47,"value":253}," Status command reports ",{"type":41,"tag":83,"props":255,"children":257},{"className":256},[],[258],{"type":47,"value":259},"OK",{"type":47,"value":261}," (success rate at or above the check's alertSensitivity threshold: high = 95%, medium\u002Fdefault = 90%, low = 75%).\nReport: \"Check ",{"type":41,"tag":83,"props":263,"children":265},{"className":264},[],[266],{"type":47,"value":267},"\u003Cjob>",{"type":47,"value":269}," is healthy. Success rate: ",{"type":41,"tag":271,"props":272,"children":273},"rate",{},[274],{"type":47,"value":275},"%. \u003Cprobe_count> probes up.\"\nStop unless the user asks for more.",{"type":41,"tag":50,"props":277,"children":278},{},[279,284],{"type":41,"tag":247,"props":280,"children":281},{},[282],{"type":47,"value":283},"Early exit — NODATA:",{"type":47,"value":285}," No Prometheus metrics available.",{"type":41,"tag":63,"props":287,"children":288},{},[289,310,315],{"type":41,"tag":67,"props":290,"children":291},{},[292,294,300,302,308],{"type":47,"value":293},"Get check config to verify ",{"type":41,"tag":83,"props":295,"children":297},{"className":296},[],[298],{"type":47,"value":299},"enabled: true",{"type":47,"value":301}," (",{"type":41,"tag":83,"props":303,"children":305},{"className":304},[],[306],{"type":47,"value":307},"gcx synthetic-monitoring checks get \u003CID> -o json | jq .spec.enabled",{"type":47,"value":309},")",{"type":41,"tag":67,"props":311,"children":312},{},[313],{"type":47,"value":314},"If disabled: report \"Check is disabled — no metrics will appear until it is re-enabled.\"",{"type":41,"tag":67,"props":316,"children":317},{},[318],{"type":47,"value":319},"If enabled: report \"No metrics found. Check datasource config or whether the SM stack is healthy.\"\nStop after reporting.",{"type":41,"tag":108,"props":321,"children":323},{"id":322},"step-2-get-check-configuration",[324],{"type":47,"value":325},"Step 2: Get Check Configuration",{"type":41,"tag":115,"props":327,"children":329},{"className":117,"code":328,"language":119,"meta":120,"style":120},"gcx synthetic-monitoring checks get \u003CID> -o json\n",[330],{"type":41,"tag":83,"props":331,"children":332},{"__ignoreMap":120},[333],{"type":41,"tag":126,"props":334,"children":335},{"class":128,"line":129},[336,340,344,348,353,357,361,365,370,375],{"type":41,"tag":126,"props":337,"children":338},{"style":133},[339],{"type":47,"value":136},{"type":41,"tag":126,"props":341,"children":342},{"style":139},[343],{"type":47,"value":142},{"type":41,"tag":126,"props":345,"children":346},{"style":139},[347],{"type":47,"value":147},{"type":41,"tag":126,"props":349,"children":350},{"style":139},[351],{"type":47,"value":352}," get",{"type":41,"tag":126,"props":354,"children":355},{"style":155},[356],{"type":47,"value":158},{"type":41,"tag":126,"props":358,"children":359},{"style":139},[360],{"type":47,"value":163},{"type":41,"tag":126,"props":362,"children":363},{"style":166},[364],{"type":47,"value":169},{"type":41,"tag":126,"props":366,"children":367},{"style":155},[368],{"type":47,"value":369},">",{"type":41,"tag":126,"props":371,"children":372},{"style":139},[373],{"type":47,"value":374}," -o",{"type":41,"tag":126,"props":376,"children":377},{"style":139},[378],{"type":47,"value":379}," json\n",{"type":41,"tag":50,"props":381,"children":382},{},[383],{"type":47,"value":384},"Extract: job name, target, check type (http\u002Fping\u002Fdns\u002Ftcp\u002Ftraceroute), probe list, frequency, timeout, alertSensitivity, enabled flag.",{"type":41,"tag":50,"props":386,"children":387},{},[388],{"type":47,"value":389},"For HTTP checks also note: any assertion settings, TLS config, expected status codes.",{"type":41,"tag":108,"props":391,"children":393},{"id":392},"step-3-timeline-triage",[394],{"type":47,"value":395},"Step 3: Timeline Triage",{"type":41,"tag":115,"props":397,"children":399},{"className":117,"code":398,"language":119,"meta":120,"style":120},"gcx synthetic-monitoring checks timeline \u003CID> --from now-1h --to now\n",[400],{"type":41,"tag":83,"props":401,"children":402},{"__ignoreMap":120},[403],{"type":41,"tag":126,"props":404,"children":405},{"class":128,"line":129},[406,410,414,418,423,427,431,435,439,444,449,454],{"type":41,"tag":126,"props":407,"children":408},{"style":133},[409],{"type":47,"value":136},{"type":41,"tag":126,"props":411,"children":412},{"style":139},[413],{"type":47,"value":142},{"type":41,"tag":126,"props":415,"children":416},{"style":139},[417],{"type":47,"value":147},{"type":41,"tag":126,"props":419,"children":420},{"style":139},[421],{"type":47,"value":422}," timeline",{"type":41,"tag":126,"props":424,"children":425},{"style":155},[426],{"type":47,"value":158},{"type":41,"tag":126,"props":428,"children":429},{"style":139},[430],{"type":47,"value":163},{"type":41,"tag":126,"props":432,"children":433},{"style":166},[434],{"type":47,"value":169},{"type":41,"tag":126,"props":436,"children":437},{"style":155},[438],{"type":47,"value":369},{"type":41,"tag":126,"props":440,"children":441},{"style":139},[442],{"type":47,"value":443}," --from",{"type":41,"tag":126,"props":445,"children":446},{"style":139},[447],{"type":47,"value":448}," now-1h",{"type":41,"tag":126,"props":450,"children":451},{"style":139},[452],{"type":47,"value":453}," --to",{"type":41,"tag":126,"props":455,"children":456},{"style":139},[457],{"type":47,"value":458}," now\n",{"type":41,"tag":50,"props":460,"children":461},{},[462],{"type":47,"value":463},"Show the graph output to the user. Then analyze the pattern:",{"type":41,"tag":465,"props":466,"children":467},"table",{},[468,487],{"type":41,"tag":469,"props":470,"children":471},"thead",{},[472],{"type":41,"tag":473,"props":474,"children":475},"tr",{},[476,482],{"type":41,"tag":477,"props":478,"children":479},"th",{},[480],{"type":47,"value":481},"Pattern",{"type":41,"tag":477,"props":483,"children":484},{},[485],{"type":47,"value":486},"Classification",{"type":41,"tag":488,"props":489,"children":490},"tbody",{},[491,505,518,531,544],{"type":41,"tag":473,"props":492,"children":493},{},[494,500],{"type":41,"tag":495,"props":496,"children":497},"td",{},[498],{"type":47,"value":499},"All probes at 0 (or near 0)",{"type":41,"tag":495,"props":501,"children":502},{},[503],{"type":47,"value":504},"Target down",{"type":41,"tag":473,"props":506,"children":507},{},[508,513],{"type":41,"tag":495,"props":509,"children":510},{},[511],{"type":47,"value":512},"Subset of probes at 0, others healthy",{"type":41,"tag":495,"props":514,"children":515},{},[516],{"type":47,"value":517},"Regional \u002F network",{"type":41,"tag":473,"props":519,"children":520},{},[521,526],{"type":41,"tag":495,"props":522,"children":523},{},[524],{"type":47,"value":525},"Intermittent drops across multiple probes",{"type":41,"tag":495,"props":527,"children":528},{},[529],{"type":47,"value":530},"Flapping \u002F timeout",{"type":41,"tag":473,"props":532,"children":533},{},[534,539],{"type":41,"tag":495,"props":535,"children":536},{},[537],{"type":47,"value":538},"All probes drop at a specific point in time",{"type":41,"tag":495,"props":540,"children":541},{},[542],{"type":47,"value":543},"Sudden onset — possible deployment or config change",{"type":41,"tag":473,"props":545,"children":546},{},[547,552],{"type":41,"tag":495,"props":548,"children":549},{},[550],{"type":47,"value":551},"Gradual decline",{"type":41,"tag":495,"props":553,"children":554},{},[555],{"type":47,"value":556},"Degradation — timeout drift or resource exhaustion",{"type":41,"tag":50,"props":558,"children":559},{},[560],{"type":47,"value":561},"Use a longer window if the failure started more than 1h ago:",{"type":41,"tag":115,"props":563,"children":565},{"className":117,"code":564,"language":119,"meta":120,"style":120},"gcx synthetic-monitoring checks timeline \u003CID> --from now-6h --to now\n",[566],{"type":41,"tag":83,"props":567,"children":568},{"__ignoreMap":120},[569],{"type":41,"tag":126,"props":570,"children":571},{"class":128,"line":129},[572,576,580,584,588,592,596,600,604,608,613,617],{"type":41,"tag":126,"props":573,"children":574},{"style":133},[575],{"type":47,"value":136},{"type":41,"tag":126,"props":577,"children":578},{"style":139},[579],{"type":47,"value":142},{"type":41,"tag":126,"props":581,"children":582},{"style":139},[583],{"type":47,"value":147},{"type":41,"tag":126,"props":585,"children":586},{"style":139},[587],{"type":47,"value":422},{"type":41,"tag":126,"props":589,"children":590},{"style":155},[591],{"type":47,"value":158},{"type":41,"tag":126,"props":593,"children":594},{"style":139},[595],{"type":47,"value":163},{"type":41,"tag":126,"props":597,"children":598},{"style":166},[599],{"type":47,"value":169},{"type":41,"tag":126,"props":601,"children":602},{"style":155},[603],{"type":47,"value":369},{"type":41,"tag":126,"props":605,"children":606},{"style":139},[607],{"type":47,"value":443},{"type":41,"tag":126,"props":609,"children":610},{"style":139},[611],{"type":47,"value":612}," now-6h",{"type":41,"tag":126,"props":614,"children":615},{"style":139},[616],{"type":47,"value":453},{"type":41,"tag":126,"props":618,"children":619},{"style":139},[620],{"type":47,"value":458},{"type":41,"tag":108,"props":622,"children":624},{"id":623},"step-4-classify-failure-scope-and-map-probes",[625],{"type":47,"value":626},"Step 4: Classify Failure Scope and Map Probes",{"type":41,"tag":50,"props":628,"children":629},{},[630],{"type":47,"value":631},"Get the probe list for geographic mapping:",{"type":41,"tag":115,"props":633,"children":635},{"className":117,"code":634,"language":119,"meta":120,"style":120},"gcx synthetic-monitoring probes list -o json\n",[636],{"type":41,"tag":83,"props":637,"children":638},{"__ignoreMap":120},[639],{"type":41,"tag":126,"props":640,"children":641},{"class":128,"line":129},[642,646,650,655,659,663],{"type":41,"tag":126,"props":643,"children":644},{"style":133},[645],{"type":47,"value":136},{"type":41,"tag":126,"props":647,"children":648},{"style":139},[649],{"type":47,"value":142},{"type":41,"tag":126,"props":651,"children":652},{"style":139},[653],{"type":47,"value":654}," probes",{"type":41,"tag":126,"props":656,"children":657},{"style":139},[658],{"type":47,"value":222},{"type":41,"tag":126,"props":660,"children":661},{"style":139},[662],{"type":47,"value":374},{"type":41,"tag":126,"props":664,"children":665},{"style":139},[666],{"type":47,"value":379},{"type":41,"tag":50,"props":668,"children":669},{},[670,672,678],{"type":47,"value":671},"Cross-reference the probe names from the check config against each probe's ",{"type":41,"tag":83,"props":673,"children":675},{"className":674},[],[676],{"type":47,"value":677},"region",{"type":47,"value":679}," field. Map failing probes to their regions.",{"type":41,"tag":50,"props":681,"children":682},{},[683,688],{"type":41,"tag":247,"props":684,"children":685},{},[686],{"type":47,"value":687},"All probes failing:",{"type":47,"value":689}," Target\u002Fservice issue — likely target down, SSL error, or DNS failure.",{"type":41,"tag":50,"props":691,"children":692},{},[693,698],{"type":41,"tag":247,"props":694,"children":695},{},[696],{"type":47,"value":697},"Subset of probes failing:",{"type":47,"value":699}," Regional or network issue. Note which regions are affected:",{"type":41,"tag":701,"props":702,"children":703},"ul",{},[704,709,714],{"type":41,"tag":67,"props":705,"children":706},{},[707],{"type":47,"value":708},"Single region → ISP\u002FCDN routing issue or regional outage",{"type":41,"tag":67,"props":710,"children":711},{},[712],{"type":47,"value":713},"Multiple contiguous regions → CDN edge or routing policy issue",{"type":41,"tag":67,"props":715,"children":716},{},[717],{"type":47,"value":718},"Probe-specific → private probe infra issue (if using private probes)",{"type":41,"tag":50,"props":720,"children":721},{},[722,727],{"type":41,"tag":247,"props":723,"children":724},{},[725],{"type":47,"value":726},"Intermittent failures:",{"type":47,"value":728}," Flapping. Consider: rate limiting, timeout too tight, flaky connectivity.",{"type":41,"tag":108,"props":730,"children":732},{"id":731},"step-5-per-probe-breakdown-via-promql-when-datasource-is-available",[733],{"type":47,"value":734},"Step 5: Per-Probe Breakdown via PromQL (when datasource is available)",{"type":41,"tag":50,"props":736,"children":737},{},[738],{"type":47,"value":739},"Resolve datasource UID if not already known:",{"type":41,"tag":115,"props":741,"children":743},{"className":117,"code":742,"language":119,"meta":120,"style":120},"gcx datasources list --type prometheus\n",[744],{"type":41,"tag":83,"props":745,"children":746},{"__ignoreMap":120},[747],{"type":41,"tag":126,"props":748,"children":749},{"class":128,"line":129},[750,754,759,763,768],{"type":41,"tag":126,"props":751,"children":752},{"style":133},[753],{"type":47,"value":136},{"type":41,"tag":126,"props":755,"children":756},{"style":139},[757],{"type":47,"value":758}," datasources",{"type":41,"tag":126,"props":760,"children":761},{"style":139},[762],{"type":47,"value":222},{"type":41,"tag":126,"props":764,"children":765},{"style":139},[766],{"type":47,"value":767}," --type",{"type":41,"tag":126,"props":769,"children":770},{"style":139},[771],{"type":47,"value":772}," prometheus\n",{"type":41,"tag":50,"props":774,"children":775},{},[776,778,784,786,792],{"type":47,"value":777},"If the filtered list comes back empty, the stack may leave the ",{"type":41,"tag":83,"props":779,"children":781},{"className":780},[],[782],{"type":47,"value":783},"type",{"type":47,"value":785}," field\nblank in list payloads (known issue) — rerun without ",{"type":41,"tag":83,"props":787,"children":789},{"className":788},[],[790],{"type":47,"value":791},"--type",{"type":47,"value":793}," and pick the\nPrometheus datasource by name.",{"type":41,"tag":50,"props":795,"children":796},{},[797,799,804,806,812],{"type":47,"value":798},"Run per-probe success rate to pinpoint failing probes (use ",{"type":41,"tag":83,"props":800,"children":802},{"className":801},[],[803],{"type":47,"value":88},{"type":47,"value":805}," for parsing, ",{"type":41,"tag":83,"props":807,"children":809},{"className":808},[],[810],{"type":47,"value":811},"-o graph",{"type":47,"value":813}," to show the user):",{"type":41,"tag":115,"props":815,"children":817},{"className":117,"code":816,"language":119,"meta":120,"style":120},"gcx metrics query -d \u003Cdatasource-uid> \\\n  'avg by (probe) (probe_success{job=\"\u003Cjob>\",instance=\"\u003Ctarget>\"})' \\\n  --from now-1h --to now --step 1m -o graph\n",[818],{"type":41,"tag":83,"props":819,"children":820},{"__ignoreMap":120},[821,866,889],{"type":41,"tag":126,"props":822,"children":823},{"class":128,"line":129},[824,828,833,838,843,847,852,857,861],{"type":41,"tag":126,"props":825,"children":826},{"style":133},[827],{"type":47,"value":136},{"type":41,"tag":126,"props":829,"children":830},{"style":139},[831],{"type":47,"value":832}," metrics",{"type":41,"tag":126,"props":834,"children":835},{"style":139},[836],{"type":47,"value":837}," query",{"type":41,"tag":126,"props":839,"children":840},{"style":139},[841],{"type":47,"value":842}," -d",{"type":41,"tag":126,"props":844,"children":845},{"style":155},[846],{"type":47,"value":158},{"type":41,"tag":126,"props":848,"children":849},{"style":139},[850],{"type":47,"value":851},"datasource-ui",{"type":41,"tag":126,"props":853,"children":854},{"style":166},[855],{"type":47,"value":856},"d",{"type":41,"tag":126,"props":858,"children":859},{"style":155},[860],{"type":47,"value":369},{"type":41,"tag":126,"props":862,"children":863},{"style":166},[864],{"type":47,"value":865}," \\\n",{"type":41,"tag":126,"props":867,"children":869},{"class":128,"line":868},2,[870,875,880,885],{"type":41,"tag":126,"props":871,"children":872},{"style":155},[873],{"type":47,"value":874},"  '",{"type":41,"tag":126,"props":876,"children":877},{"style":139},[878],{"type":47,"value":879},"avg by (probe) (probe_success{job=\"\u003Cjob>\",instance=\"\u003Ctarget>\"})",{"type":41,"tag":126,"props":881,"children":882},{"style":155},[883],{"type":47,"value":884},"'",{"type":41,"tag":126,"props":886,"children":887},{"style":166},[888],{"type":47,"value":865},{"type":41,"tag":126,"props":890,"children":892},{"class":128,"line":891},3,[893,898,902,906,911,916,921,925],{"type":41,"tag":126,"props":894,"children":895},{"style":139},[896],{"type":47,"value":897},"  --from",{"type":41,"tag":126,"props":899,"children":900},{"style":139},[901],{"type":47,"value":448},{"type":41,"tag":126,"props":903,"children":904},{"style":139},[905],{"type":47,"value":453},{"type":41,"tag":126,"props":907,"children":908},{"style":139},[909],{"type":47,"value":910}," now",{"type":41,"tag":126,"props":912,"children":913},{"style":139},[914],{"type":47,"value":915}," --step",{"type":41,"tag":126,"props":917,"children":918},{"style":139},[919],{"type":47,"value":920}," 1m",{"type":41,"tag":126,"props":922,"children":923},{"style":139},[924],{"type":47,"value":374},{"type":41,"tag":126,"props":926,"children":927},{"style":139},[928],{"type":47,"value":929}," graph\n",{"type":41,"tag":50,"props":931,"children":932},{},[933],{"type":47,"value":934},"For HTTP checks, also run HTTP phase latency to locate where time is spent:",{"type":41,"tag":115,"props":936,"children":938},{"className":117,"code":937,"language":119,"meta":120,"style":120},"gcx metrics query -d \u003Cdatasource-uid> \\\n  'avg by (phase) (probe_http_duration_seconds{job=\"\u003Cjob>\",instance=\"\u003Ctarget>\"})' \\\n  --from now-1h --to now --step 1m -o graph\n",[939],{"type":41,"tag":83,"props":940,"children":941},{"__ignoreMap":120},[942,981,1001],{"type":41,"tag":126,"props":943,"children":944},{"class":128,"line":129},[945,949,953,957,961,965,969,973,977],{"type":41,"tag":126,"props":946,"children":947},{"style":133},[948],{"type":47,"value":136},{"type":41,"tag":126,"props":950,"children":951},{"style":139},[952],{"type":47,"value":832},{"type":41,"tag":126,"props":954,"children":955},{"style":139},[956],{"type":47,"value":837},{"type":41,"tag":126,"props":958,"children":959},{"style":139},[960],{"type":47,"value":842},{"type":41,"tag":126,"props":962,"children":963},{"style":155},[964],{"type":47,"value":158},{"type":41,"tag":126,"props":966,"children":967},{"style":139},[968],{"type":47,"value":851},{"type":41,"tag":126,"props":970,"children":971},{"style":166},[972],{"type":47,"value":856},{"type":41,"tag":126,"props":974,"children":975},{"style":155},[976],{"type":47,"value":369},{"type":41,"tag":126,"props":978,"children":979},{"style":166},[980],{"type":47,"value":865},{"type":41,"tag":126,"props":982,"children":983},{"class":128,"line":868},[984,988,993,997],{"type":41,"tag":126,"props":985,"children":986},{"style":155},[987],{"type":47,"value":874},{"type":41,"tag":126,"props":989,"children":990},{"style":139},[991],{"type":47,"value":992},"avg by (phase) (probe_http_duration_seconds{job=\"\u003Cjob>\",instance=\"\u003Ctarget>\"})",{"type":41,"tag":126,"props":994,"children":995},{"style":155},[996],{"type":47,"value":884},{"type":41,"tag":126,"props":998,"children":999},{"style":166},[1000],{"type":47,"value":865},{"type":41,"tag":126,"props":1002,"children":1003},{"class":128,"line":891},[1004,1008,1012,1016,1020,1024,1028,1032],{"type":41,"tag":126,"props":1005,"children":1006},{"style":139},[1007],{"type":47,"value":897},{"type":41,"tag":126,"props":1009,"children":1010},{"style":139},[1011],{"type":47,"value":448},{"type":41,"tag":126,"props":1013,"children":1014},{"style":139},[1015],{"type":47,"value":453},{"type":41,"tag":126,"props":1017,"children":1018},{"style":139},[1019],{"type":47,"value":910},{"type":41,"tag":126,"props":1021,"children":1022},{"style":139},[1023],{"type":47,"value":915},{"type":41,"tag":126,"props":1025,"children":1026},{"style":139},[1027],{"type":47,"value":920},{"type":41,"tag":126,"props":1029,"children":1030},{"style":139},[1031],{"type":47,"value":374},{"type":41,"tag":126,"props":1033,"children":1034},{"style":139},[1035],{"type":47,"value":929},{"type":41,"tag":50,"props":1037,"children":1038},{},[1039,1041,1048],{"type":47,"value":1040},"For SSL\u002FTLS cert expiry, DNS latency, per-probe error rates, and other patterns, see ",{"type":41,"tag":1042,"props":1043,"children":1045},"a",{"href":1044},"references\u002Fsm-promql-patterns.md",[1046],{"type":47,"value":1047},"sm-promql-patterns.md",{"type":47,"value":1049},".",{"type":41,"tag":108,"props":1051,"children":1053},{"id":1052},"step-6-classify-failure-mode",[1054],{"type":47,"value":1055},"Step 6: Classify Failure Mode",{"type":41,"tag":50,"props":1057,"children":1058},{},[1059,1061,1067],{"type":47,"value":1060},"Cross-reference signals against ",{"type":41,"tag":1042,"props":1062,"children":1064},{"href":1063},"references\u002Ffailure-modes.md",[1065],{"type":47,"value":1066},"failure-modes.md",{"type":47,"value":1068}," (full signal\u002Fcause\u002Fnext-action table and decision tree) to select the most likely failure mode:",{"type":41,"tag":63,"props":1070,"children":1071},{},[1072,1081,1091,1101,1111,1137,1147,1157],{"type":41,"tag":67,"props":1073,"children":1074},{},[1075,1077],{"type":47,"value":1076},"All probes failing + HTTP non-2xx or connection refused → ",{"type":41,"tag":247,"props":1078,"children":1079},{},[1080],{"type":47,"value":504},{"type":41,"tag":67,"props":1082,"children":1083},{},[1084,1086],{"type":47,"value":1085},"Subset of probes failing → ",{"type":41,"tag":247,"props":1087,"children":1088},{},[1089],{"type":47,"value":1090},"Regional\u002FCDN",{"type":41,"tag":67,"props":1092,"children":1093},{},[1094,1096],{"type":47,"value":1095},"TLS handshake error or cert expiry \u003C 14 days → ",{"type":41,"tag":247,"props":1097,"children":1098},{},[1099],{"type":47,"value":1100},"SSL\u002FTLS",{"type":41,"tag":67,"props":1102,"children":1103},{},[1104,1106],{"type":47,"value":1105},"DNS resolution errors across probes → ",{"type":41,"tag":247,"props":1107,"children":1108},{},[1109],{"type":47,"value":1110},"DNS resolution",{"type":41,"tag":67,"props":1112,"children":1113},{},[1114,1116,1122,1124,1130,1132],{"type":47,"value":1115},"All probes timing out, phase latency high in ",{"type":41,"tag":83,"props":1117,"children":1119},{"className":1118},[],[1120],{"type":47,"value":1121},"connect",{"type":47,"value":1123}," or ",{"type":41,"tag":83,"props":1125,"children":1127},{"className":1126},[],[1128],{"type":47,"value":1129},"tls",{"type":47,"value":1131}," → ",{"type":41,"tag":247,"props":1133,"children":1134},{},[1135],{"type":47,"value":1136},"Timeout",{"type":41,"tag":67,"props":1138,"children":1139},{},[1140,1142],{"type":47,"value":1141},"Probes reaching target but assertion fails (status code, body match) → ",{"type":41,"tag":247,"props":1143,"children":1144},{},[1145],{"type":47,"value":1146},"Content\u002Fassertion",{"type":41,"tag":67,"props":1148,"children":1149},{},[1150,1152],{"type":47,"value":1151},"Single private probe failing, public probes healthy → ",{"type":41,"tag":247,"props":1153,"children":1154},{},[1155],{"type":47,"value":1156},"Private probe infra",{"type":41,"tag":67,"props":1158,"children":1159},{},[1160,1162],{"type":47,"value":1161},"HTTP 429 responses, intermittent failures with backoff pattern → ",{"type":41,"tag":247,"props":1163,"children":1164},{},[1165],{"type":47,"value":1166},"Rate limiting",{"type":41,"tag":108,"props":1168,"children":1170},{"id":1169},"step-7-diagnosis-and-next-actions",[1171],{"type":47,"value":1172},"Step 7: Diagnosis and Next Actions",{"type":41,"tag":50,"props":1174,"children":1175},{},[1176,1178,1182],{"type":47,"value":1177},"Synthesize findings into an actionable report (see Output Format below). Take next actions from the \"Next Action\" column of the failure mode's row in ",{"type":41,"tag":1042,"props":1179,"children":1180},{"href":1063},[1181],{"type":47,"value":1066},{"type":47,"value":1049},{"type":41,"tag":50,"props":1184,"children":1185},{},[1186],{"type":47,"value":1187},"If deeper investigation is needed (e.g., logs, infra repos), ask the user if they want to proceed.",{"type":41,"tag":50,"props":1189,"children":1190},{},[1191,1193,1198],{"type":47,"value":1192},"If check config needs changes (probe selection, frequency, assertions), route to ",{"type":41,"tag":247,"props":1194,"children":1195},{},[1196],{"type":47,"value":1197},"synth-manage-checks",{"type":47,"value":1049},{"type":41,"tag":56,"props":1200,"children":1202},{"id":1201},"output-format",[1203],{"type":47,"value":1204},"Output Format",{"type":41,"tag":50,"props":1206,"children":1207},{},[1208],{"type":41,"tag":247,"props":1209,"children":1210},{},[1211],{"type":47,"value":1212},"Early exit (OK):",{"type":41,"tag":115,"props":1214,"children":1218},{"className":1215,"code":1217,"language":47},[1216],"language-text","Check: \u003Cjob> (\u003Ctarget>)\nStatus: OK\nSuccess rate: \u003Crate>%\nProbes up: \u003Ccount>\u002F\u003Ctotal>\n",[1219],{"type":41,"tag":83,"props":1220,"children":1221},{"__ignoreMap":120},[1222],{"type":47,"value":1217},{"type":41,"tag":50,"props":1224,"children":1225},{},[1226],{"type":41,"tag":247,"props":1227,"children":1228},{},[1229],{"type":47,"value":1230},"Early exit (NODATA):",{"type":41,"tag":115,"props":1232,"children":1235},{"className":1233,"code":1234,"language":47},[1216],"Check: \u003Cjob> (\u003Ctarget>)\nStatus: NODATA\nEnabled: \u003Cyes\u002Fno>\nNext: \u003Cdatasource check \u002F re-enable instruction>\n",[1236],{"type":41,"tag":83,"props":1237,"children":1238},{"__ignoreMap":120},[1239],{"type":47,"value":1234},{"type":41,"tag":50,"props":1241,"children":1242},{},[1243],{"type":41,"tag":247,"props":1244,"children":1245},{},[1246],{"type":47,"value":1247},"Full investigation:",{"type":41,"tag":115,"props":1249,"children":1252},{"className":1250,"code":1251,"language":47},[1216],"Check: \u003Cjob> (\u003Ctarget>)\nType: \u003Chttp|ping|dns|tcp|traceroute>\nStatus: FAILING\nSuccess rate: \u003Crate>% (window: \u003Cfrom> – \u003Cto>)\n\n[Timeline graph]\n\nFailure classification: \u003CTarget down | Regional\u002FCDN | SSL\u002FTLS | DNS | Timeout | Content\u002Fassertion | Private probe infra | Rate limiting>\n\nAffected probes: \u003Ccount>\u002F\u003Ctotal>\n  - \u003Cprobe-name> (\u003Cregion>): failing since \u003Ctime>\n  - \u003Cprobe-name> (\u003Cregion>): intermittent\n\nOnset: \u003Ctime\u002Fduration or \"unknown\">\n\nDiagnosis:\n\u003C2-4 sentences describing what the data shows and the most likely cause>\n\nNext actions:\n1. \u003Caction>\n2. \u003Caction>\n3. \u003Caction>\n",[1253],{"type":41,"tag":83,"props":1254,"children":1255},{"__ignoreMap":120},[1256],{"type":47,"value":1251},{"type":41,"tag":50,"props":1258,"children":1259},{},[1260],{"type":47,"value":1261},"Use minimal formatting. Avoid excessive bold text. Trust the user to prioritize.",{"type":41,"tag":56,"props":1263,"children":1265},{"id":1264},"error-handling",[1266],{"type":47,"value":1267},"Error Handling",{"type":41,"tag":701,"props":1269,"children":1270},{},[1271,1282,1293,1304,1309],{"type":41,"tag":67,"props":1272,"children":1273},{},[1274,1280],{"type":41,"tag":83,"props":1275,"children":1277},{"className":1276},[],[1278],{"type":47,"value":1279},"gcx synthetic-monitoring checks status",{"type":47,"value":1281}," returns no rows: check ID may be wrong — list all checks and confirm",{"type":41,"tag":67,"props":1283,"children":1284},{},[1285,1291],{"type":41,"tag":83,"props":1286,"children":1288},{"className":1287},[],[1289],{"type":47,"value":1290},"gcx synthetic-monitoring probes list",{"type":47,"value":1292}," fails: skip geographic mapping; classify probes by name where possible",{"type":41,"tag":67,"props":1294,"children":1295},{},[1296,1302],{"type":41,"tag":83,"props":1297,"children":1299},{"className":1298},[],[1300],{"type":47,"value":1301},"gcx metrics query",{"type":47,"value":1303}," fails with datasource error: note it, skip PromQL steps, classify using timeline data only",{"type":41,"tag":67,"props":1305,"children":1306},{},[1307],{"type":47,"value":1308},"Multiple checks match the search name: list all with IDs and targets, ask which to investigate",{"type":41,"tag":67,"props":1310,"children":1311},{},[1312,1314,1320],{"type":47,"value":1313},"Timeline returns no data for the window: widen to ",{"type":41,"tag":83,"props":1315,"children":1317},{"className":1316},[],[1318],{"type":47,"value":1319},"--from now-6h --to now",{"type":47,"value":1321}," before concluding NODATA",{"type":41,"tag":1323,"props":1324,"children":1325},"style",{},[1326],{"type":47,"value":1327},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1329,"total":1430},[1330,1343,1359,1372,1387,1404,1417],{"slug":1331,"name":1331,"fn":1332,"description":1333,"org":1334,"tags":1335,"stars":23,"repoUrl":24,"updatedAt":1342},"agento11y","manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1336,1339,1340,1341],{"name":1337,"slug":1338,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-07-25T05:30:40.29622",{"slug":1344,"name":1344,"fn":1345,"description":1346,"org":1347,"tags":1348,"stars":23,"repoUrl":24,"updatedAt":1358},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1349,1350,1351,1354,1357],{"name":1337,"slug":1338,"type":15},{"name":9,"slug":8,"type":15},{"name":1352,"slug":1353,"type":15},"Instrumentation","instrumentation",{"name":1355,"slug":1356,"type":15},"LLM","llm",{"name":13,"slug":14,"type":15},"2026-07-31T05:53:52.580237",{"slug":1360,"name":1360,"fn":1361,"description":1362,"org":1363,"tags":1364,"stars":23,"repoUrl":24,"updatedAt":1371},"agento11y-prod-setup","setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1365,1366,1369,1370],{"name":1337,"slug":1338,"type":15},{"name":1367,"slug":1368,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:53.576347",{"slug":1373,"name":1373,"fn":1374,"description":1375,"org":1376,"tags":1377,"stars":23,"repoUrl":24,"updatedAt":1386},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1378,1379,1380,1383],{"name":1337,"slug":1338,"type":15},{"name":9,"slug":8,"type":15},{"name":1381,"slug":1382,"type":15},"QA","qa",{"name":1384,"slug":1385,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":1388,"name":1388,"fn":1389,"description":1390,"org":1391,"tags":1392,"stars":23,"repoUrl":24,"updatedAt":1403},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1393,1396,1399,1402],{"name":1394,"slug":1395,"type":15},"Dashboards","dashboards",{"name":1397,"slug":1398,"type":15},"Data Visualization","data-visualization",{"name":1400,"slug":1401,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":1405,"name":1405,"fn":1406,"description":1407,"org":1408,"tags":1409,"stars":23,"repoUrl":24,"updatedAt":1416},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1410,1411,1412,1415],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":1413,"slug":1414,"type":15},"Incident Response","incident-response",{"name":13,"slug":14,"type":15},"2026-07-18T05:11:10.445428",{"slug":1418,"name":1418,"fn":1419,"description":1420,"org":1421,"tags":1422,"stars":23,"repoUrl":24,"updatedAt":1429},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1423,1424,1425,1428],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":1426,"slug":1427,"type":15},"Graph Analysis","graph-analysis",{"name":13,"slug":14,"type":15},"2026-07-25T05:30:39.380934",24,{"items":1432,"total":1565},[1433,1450,1469,1489,1496,1504,1511,1518,1525,1532,1539,1553],{"slug":1434,"name":1434,"fn":1435,"description":1436,"org":1437,"tags":1438,"stars":1447,"repoUrl":1448,"updatedAt":1449},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1439,1442,1445,1446],{"name":1440,"slug":1441,"type":15},"Distributed Tracing","distributed-tracing",{"name":1443,"slug":1444,"type":15},"Frontend","frontend",{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":1451,"name":1451,"fn":1452,"description":1453,"org":1454,"tags":1455,"stars":1466,"repoUrl":1467,"updatedAt":1468},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1456,1459,1462,1465],{"name":1457,"slug":1458,"type":15},"API Development","api-development",{"name":1460,"slug":1461,"type":15},"Authentication","authentication",{"name":1463,"slug":1464,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":1470,"name":1470,"fn":1471,"description":1472,"org":1473,"tags":1474,"stars":1466,"repoUrl":1467,"updatedAt":1488},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1475,1478,1481,1482,1485],{"name":1476,"slug":1477,"type":15},"CSV","csv",{"name":1479,"slug":1480,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":1483,"slug":1484,"type":15},"GraphQL","graphql",{"name":1486,"slug":1487,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":1331,"name":1331,"fn":1332,"description":1333,"org":1490,"tags":1491,"stars":23,"repoUrl":24,"updatedAt":1342},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1492,1493,1494,1495],{"name":1337,"slug":1338,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":1344,"name":1344,"fn":1345,"description":1346,"org":1497,"tags":1498,"stars":23,"repoUrl":24,"updatedAt":1358},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1499,1500,1501,1502,1503],{"name":1337,"slug":1338,"type":15},{"name":9,"slug":8,"type":15},{"name":1352,"slug":1353,"type":15},{"name":1355,"slug":1356,"type":15},{"name":13,"slug":14,"type":15},{"slug":1360,"name":1360,"fn":1361,"description":1362,"org":1505,"tags":1506,"stars":23,"repoUrl":24,"updatedAt":1371},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1507,1508,1509,1510],{"name":1337,"slug":1338,"type":15},{"name":1367,"slug":1368,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":1373,"name":1373,"fn":1374,"description":1375,"org":1512,"tags":1513,"stars":23,"repoUrl":24,"updatedAt":1386},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1514,1515,1516,1517],{"name":1337,"slug":1338,"type":15},{"name":9,"slug":8,"type":15},{"name":1381,"slug":1382,"type":15},{"name":1384,"slug":1385,"type":15},{"slug":1388,"name":1388,"fn":1389,"description":1390,"org":1519,"tags":1520,"stars":23,"repoUrl":24,"updatedAt":1403},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1521,1522,1523,1524],{"name":1394,"slug":1395,"type":15},{"name":1397,"slug":1398,"type":15},{"name":1400,"slug":1401,"type":15},{"name":9,"slug":8,"type":15},{"slug":1405,"name":1405,"fn":1406,"description":1407,"org":1526,"tags":1527,"stars":23,"repoUrl":24,"updatedAt":1416},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1528,1529,1530,1531],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":1413,"slug":1414,"type":15},{"name":13,"slug":14,"type":15},{"slug":1418,"name":1418,"fn":1419,"description":1420,"org":1533,"tags":1534,"stars":23,"repoUrl":24,"updatedAt":1429},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1535,1536,1537,1538],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":1426,"slug":1427,"type":15},{"name":13,"slug":14,"type":15},{"slug":136,"name":136,"fn":1540,"description":1541,"org":1542,"tags":1543,"stars":23,"repoUrl":24,"updatedAt":1552},"manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1544,1547,1548,1549],{"name":1545,"slug":1546,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":1550,"slug":1551,"type":15},"Operations","operations","2026-07-31T05:53:50.587304",{"slug":1554,"name":1554,"fn":1555,"description":1556,"org":1557,"tags":1558,"stars":23,"repoUrl":24,"updatedAt":1564},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1559,1560,1561],{"name":1545,"slug":1546,"type":15},{"name":9,"slug":8,"type":15},{"name":1562,"slug":1563,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80]