[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-diagnose-entity-graph":3,"mdc--kmru85-key":34,"related-org-grafana-diagnose-entity-graph":2689,"related-repo-grafana-diagnose-entity-graph":2870},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,20],{"name":13,"slug":14,"type":15},"Graph Analysis","graph-analysis","tag",{"name":17,"slug":18,"type":15},"Observability","observability",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Debugging","debugging",430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-25T05:30:39.380934",null,29,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A CLI for managing Grafana Cloud resources. Optimized for agentic usage.","https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx\u002Ftree\u002FHEAD\u002Fclaude-plugin\u002Fskills\u002Fdiagnose-entity-graph","---\nname: diagnose-entity-graph\ndescription: >\n  Diagnose Entity Graph problems: missing entities, missing edges, disconnected\n  clusters, or filtering issues. Use when the user reports that Entity Graph\n  doesn't look right, services are missing, edges aren't appearing, or\n  environments can't be filtered. Triggers for: \"entity graph is empty\",\n  \"services missing from entity graph\", \"no edges in entity graph\",\n  \"disconnected services\", \"can't filter entity graph\", \"entity graph not\n  working\", \"diagnose entity graph\", \"debug knowledge graph\".\n---\n\n# Diagnose Entity Graph\n\nSystematic diagnosis of Entity Graph problems using gcx commands. Follow the\nsteps in order — each step narrows the cause. Be direct and report findings\nconcisely.\n\n## Start here: read the Orientation block\n\n`gcx kg diagnose` prints an Orientation block above its check table. Read it\nfirst. It tells you which (if any) of the five common Entity Graph scenarios\nthe run matches:\n\n- \"I see no entities at all\"\n- \"Some expected entities are missing\"\n- \"I see entities with no edges\"\n- \"I see disconnected clusters of entities\"\n- \"I can't filter to the entities I want\"\n\nWhen the Orientation block names a matched scenario, follow its `Next:`\ncommands. The per-step playbook below is the fallback for runs where no\nscenario matched or the matched scenario's hints didn't resolve the issue.\n\n## Reading Diagnose Output\n\nTreat `gcx kg diagnose`'s verdicts as authoritative for the queries it\nran. Two non-obvious classifications:\n\n- **`WARN — metric exists … but no series match the requested\n  env\u002Fnamespace scope`** means the metric is flowing on the stack but\n  doesn't carry the scoped label value. This is a label-mapping issue\n  (asserts_env vs. deployment_environment, etc.), not a missing-data\n  issue. Investigate the label pipeline (Step 6) before suggesting the\n  user enable new telemetry.\n- **`FAIL — no data` (without the WARN above)** means the metric was\n  not found, even unscoped — the integration \u002F recording rule is\n  genuinely absent.\n\nFor ad-hoc PromQL outside `kg diagnose`, apply the same discipline:\nre-run the query without the env \u002F namespace filter before concluding\nthe data is missing.\n\nIf a user reports an entity by name, see Step 7's entity-existence\nworkflow before assuming it exists on this stack.\n\n## Prerequisites\n\ngcx must be installed and configured with a valid context. `gcx kg diagnose`\nbundles most of the per-step checks below; the individual commands produce\nequivalent results when you need to drill into one check.\n\n```bash\ngcx config view\ngcx kg status\n```\n\nIf `kg status` fails with a config or auth error (no context, connection\nrefused, 401), use the `setup-gcx` skill first. Do not route every error\nthere: a 404 means the Asserts plugin isn't installed on this stack, and 403s\nfrom entity endpoints while `kg status` itself succeeds usually mean the\nKnowledge Graph isn't onboarded (or the token lacks plugin access) — those are\nAsserts onboarding\u002Fpermission issues `setup-gcx` cannot fix; handle them via\nStep 1's onboarding stop instead.\n\n## Step 1: Stack Health\n\n```bash\ngcx kg status\n```\n\n**Check:** `status` must be `\"complete\"` and `enabled` must be `true`. If not,\nthe Knowledge Graph hasn't been onboarded — stop here and direct the user to\nthe Asserts app onboarding flow.\n\n**Shortcut:** `gcx kg diagnose` runs this plus all subsequent checks in parallel.\n\n## Step 2: Entity Counts and Scopes\n\n```bash\ngcx kg stats --since 1h\ngcx kg meta scopes\n```\n\n**Check:** entity counts should be > 0. The `meta scopes` output shows\navailable `env`, `site`, and `namespace` values.\n\nIf scoping to a specific environment, note the exact `env` value — you'll\nuse it in all subsequent queries.\n\n## Step 3: Source Metrics in Mimir\n\nCheck whether the raw telemetry that feeds Entity Graph exists. Raw Tempo\nmetrics use `deployment_environment`, not `asserts_env`.\n\nNote the label shape difference between the two metrics: `traces_target_info`\ndescribes a single service so it has one `deployment_environment` label;\n`traces_service_graph_request_total` describes an edge between two services\nand exposes the env on both sides as `client_deployment_environment` and\n`server_deployment_environment` — there is no unified `deployment_environment`\nlabel.\n\n```bash\n# Service identity (OTel traces)\ngcx metrics query 'count(traces_target_info)' --since 1h\ngcx metrics query 'count(traces_target_info{deployment_environment=\"ENV\"})' --since 1h\n\n# Call data (inter-service HTTP\u002FgRPC)\ngcx metrics query 'count(traces_service_graph_request_total)' --since 1h\n# Filter on server side (use client_deployment_environment for outbound view):\ngcx metrics query 'count(traces_service_graph_request_total{server_deployment_environment=\"ENV\"})' --since 1h\n```\n\n**Interpret:**\n- Both have data → traces are flowing. Continue to Step 4.\n- Both empty → no OTel traces for this environment. Entities may still exist\n  via Prometheus scraping. Continue to Step 4.\n\nFor more specific verdicts on this metric pair (Tempo metrics generation\ndisabled, broken trace context propagation), run `gcx kg diagnose --env ENV`\nand read the check results — the command encodes the detection logic for\nthose two cases and emits a targeted recommendation. Service-name collisions\nare **not** detected by the command; spot them manually via self-loop edges\n(Step 7).\n\n## Step 4: Recording Rules\n\nRecording rules convert raw metrics into the `asserts:*` metrics that Entity\nGraph consumes. These use `asserts_env`, not `deployment_environment`.\n\n```bash\n# Entity discovery (central to how services appear)\ngcx metrics query 'count(asserts:mixin_workload_job{asserts_env=\"ENV\"})' --since 1h\n\n# CALLS edges\ngcx metrics query 'count(asserts:relation:calls{asserts_env=\"ENV\"})' --since 1h\n\n# Request rate KPI\ngcx metrics query 'count(asserts:request:rate5m{asserts_env=\"ENV\"})' --since 1h\n```\n\n**Interpret:**\n- `asserts:mixin_workload_job` has data but `asserts:relation:calls` doesn't →\n  entities are discovered but no edges exist. Continue to Step 5.\n- All empty → recording rules aren't producing output. Check Step 6 (labels).\n- All have data → pipeline is healthy. For a specific missing service, go to Step 7.\n\n## Step 5: Edge Source Analysis\n\nCALLS edges can come from many sources, not just OTel traces:\n\n| Source | Input Metric | Requires Traces? |\n|--------|-------------|-----------------|\n| `app_o11y_servicegraph` | `traces_service_graph_request_total` | Yes |\n| `springboot` | `http_server_requests_seconds_count` | No |\n| `nginx_ingress` | `nginx_ingress_controller_requests` | No |\n| `istio` | `istio_requests_total` | No |\n| `aws_rds` | CloudWatch RDS metrics | No |\n| `aws_dynamodb` | CloudWatch DynamoDB metrics | No |\n| `aws_s3` | CloudWatch S3 metrics | No |\n| `aws_applicationelb` | CloudWatch ALB metrics | No |\n| `azure_flexible_server` | Azure DB metrics | No |\n| `kafka_exporter` | Kafka exporter metrics | No |\n| `dbo11y_*` | Database observability metrics | No |\n\n```bash\n# What edge sources are active on this stack?\ngcx metrics labels --label asserts_source\n\n# Check common Prometheus-based sources for a namespace:\ngcx metrics query 'count(http_server_requests_seconds_count{namespace=\"NS\"})' --since 1h\ngcx metrics query 'count(nginx_ingress_controller_requests{namespace=\"NS\"})' --since 1h\ngcx metrics query 'count(istio_requests_total{namespace=\"NS\"})' --since 1h\n```\n\n**Critical: Check for the asserts_env gap.** If a source metric exists but has\nno `asserts_env` label, the recording rules silently drop it. This is the most\ncommon reason for \"metrics present but no edges\":\n\n```bash\n# For each source that returned data above, check if it has asserts_env:\ngcx metrics query 'count(istio_requests_total{asserts_env!=\"\"})' --since 1h\ngcx metrics query 'count(http_server_requests_seconds_count{asserts_env!=\"\"})' --since 1h\ngcx metrics query 'count(nginx_ingress_controller_requests{asserts_env!=\"\"})' --since 1h\n```\n\nIf the metric exists but the `asserts_env!=\"\"` query returns \"No data\", the\nMimir relabeling rules don't cover this source. The fix is to add a relabeling\nrule that maps `namespace` or another label to `asserts_env` for this metric.\n\n**Interpret:**\n- No edge sources for this environment → edges are expected to be missing.\n  Services need tracing or one of the Prometheus-based sources above.\n- Edge source exists but missing `asserts_env` → relabeling gap. Recording\n  rules require `asserts_env!=\"\"` and will silently ignore this data.\n- If services are discovered via JMX (`job` contains `jmx`) → JMX alone\n  cannot produce edges. Spring Boot Actuator or OTel tracing is needed.\n\n**Shortcut:** `gcx kg diagnose` detects this gap automatically and warns\nwhen edge source metrics exist but lack `asserts_env`.\n\n**Most common fix:** If metrics have `deployment_environment` but not\n`asserts_env`, the Asserts environment mapping is misconfigured. Go to\n**Asserts app → Configuration → Connect Environment → Prometheus** and set\nthe environment label to `deployment_environment`. This tells the Mimir\nrelabeling pipeline to derive `asserts_env` from `deployment_environment`\non **all** incoming metrics — not just `target_info`.\n\n**If metrics lack both `deployment_environment` AND `asserts_env`:** The\nscrape pipeline needs to add `deployment_environment` first. In Alloy, use\n`prometheus.relabel` to copy `namespace` (or another label) to\n`deployment_environment` before `remote_write`. Then configure the Connect\nEnvironment page as above.\n\n**Alternative path:** Enable OTel tracing to get edges via\n`traces_service_graph_request_total` instead. Tempo generates this metric\nserver-side with `asserts_env` already populated, bypassing the Mimir\nrelabeling pipeline entirely.\n\n## Step 6: Label Pipeline\n\nThe most common issue: `deployment_environment` isn't mapped to `asserts_env`.\n\n```bash\ngcx metrics labels --label deployment_environment\ngcx metrics labels --label asserts_env\n```\n\n**Check:** Every `deployment_environment` value should have a corresponding\n`asserts_env` value. If one is missing, the Mimir relabeling rules aren't\nconfigured for that environment.\n\nExtra `asserts_env` values (like AWS account IDs) that don't match any\n`deployment_environment` are normal — they come from non-OTel sources.\n\n**Shortcut:** `gcx kg diagnose labels` automates this cross-reference.\n\n## Step 7: Per-Service Investigation\n\nFor a specific missing or edge-less service:\n\n```bash\n# Find in graph\ngcx kg entities query \"MATCH (s:Service {name: \\\"SERVICE\\\"}) RETURN s\" --since 1h\n\n# Check relationships\ngcx kg entities query \"MATCH (s:Service {name: \\\"SERVICE\\\"})-[r]-(other) RETURN s, r, other\" --since 1h\n\n# Source metrics\ngcx metrics query 'count(traces_service_graph_request_total{client=\"SERVICE\"})' --since 1h\ngcx metrics query 'count(traces_service_graph_request_total{server=\"SERVICE\"})' --since 1h\n\n# Recording rule output\ngcx metrics query 'count(asserts:relation:calls{service=\"SERVICE\"})' --since 1h\ngcx metrics query 'count(asserts:mixin_workload_job{service=\"SERVICE\"})' --since 1h\n```\n\n**Interpret:**\n- Found via Cypher but no relationships → check source metrics above.\n- `server` series exist but `asserts:relation:calls` doesn't → recording rule\n  label mismatch (check `asserts_env` and `namespace`).\n- Not found via Cypher → check `traces_target_info{service_name=\"SERVICE\"}`.\n- Leaf services (queue consumers, processors) correctly have no outgoing edges.\n\n**Shortcut:** `gcx kg diagnose service SERVICE --env ENV` runs the checks\nabove and produces an interpreted diagnosis with suggested next steps.\n\nTwo common patterns that present as \"missing entities\" are **not** detected\nby the command — check for them manually:\n\n- **Service-name collision** (multiple workloads share one `service.name`,\n  collapsing into one entity): look for self-loop edges — series where both\n  `client` and `server` equal SERVICE in `traces_service_graph_request_total`\n  — and for a single entity where you expected several workloads.\n- **Env-scope split** (workloads in the same namespace disagree on\n  `deployment.environment`, so cross-env calls don't render as edges):\n  compare `client_deployment_environment` vs `server_deployment_environment`\n  on the service's edge series from Step 3.\n\n## Producing a Report\n\nSummarize findings as:\n\n1. **Stack health** — KG enabled and complete?\n2. **Entity count** — how many for the scoped environment?\n3. **Discovery path** — OTel traces, Prometheus scrape, or cloud integration?\n4. **Trace data** — do `traces_target_info` and `traces_service_graph_request_total` exist?\n5. **Edge data** — does `asserts:relation:calls` exist? Which `asserts_source` values?\n6. **Alternative edge sources** — Spring Boot, nginx, Istio, cloud integrations available?\n7. **Label mapping** — `deployment_environment` correctly mapped to `asserts_env`?\n8. **Conclusion** — expected state or configuration issue?\n9. **Recommendations** — what would fix it?\n\nWhen recommending a fix, set expectations on convergence time. The metrics\nthe Knowledge Graph reads from (`asserts:*` recording rules, and the\n`traces_*` series Tempo generates) are time-series with a query lookback\nwindow — old data with the broken state will keep appearing in queries\nfor at least 5–15 minutes after the fix is applied. The Entity Graph UI\nshould fully stabilize on the corrected state within that window.\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,47,53,60,72,102,115,121,133,167,180,185,191,203,255,290,296,318,359,375,381,435,476,488,494,515,566,762,770,783,803,809,834,987,994,1026,1032,1037,1318,1482,1499,1622,1649,1656,1704,1725,1791,1852,1876,1882,1900,1955,1978,1997,2013,2019,2024,2352,2359,2415,2431,2442,2516,2522,2527,2663,2683],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":46},"text","Diagnose Entity Graph",{"type":40,"tag":48,"props":49,"children":50},"p",{},[51],{"type":45,"value":52},"Systematic diagnosis of Entity Graph problems using gcx commands. Follow the\nsteps in order — each step narrows the cause. Be direct and report findings\nconcisely.",{"type":40,"tag":54,"props":55,"children":57},"h2",{"id":56},"start-here-read-the-orientation-block",[58],{"type":45,"value":59},"Start here: read the Orientation block",{"type":40,"tag":48,"props":61,"children":62},{},[63,70],{"type":40,"tag":64,"props":65,"children":67},"code",{"className":66},[],[68],{"type":45,"value":69},"gcx kg diagnose",{"type":45,"value":71}," prints an Orientation block above its check table. Read it\nfirst. It tells you which (if any) of the five common Entity Graph scenarios\nthe run matches:",{"type":40,"tag":73,"props":74,"children":75},"ul",{},[76,82,87,92,97],{"type":40,"tag":77,"props":78,"children":79},"li",{},[80],{"type":45,"value":81},"\"I see no entities at all\"",{"type":40,"tag":77,"props":83,"children":84},{},[85],{"type":45,"value":86},"\"Some expected entities are missing\"",{"type":40,"tag":77,"props":88,"children":89},{},[90],{"type":45,"value":91},"\"I see entities with no edges\"",{"type":40,"tag":77,"props":93,"children":94},{},[95],{"type":45,"value":96},"\"I see disconnected clusters of entities\"",{"type":40,"tag":77,"props":98,"children":99},{},[100],{"type":45,"value":101},"\"I can't filter to the entities I want\"",{"type":40,"tag":48,"props":103,"children":104},{},[105,107,113],{"type":45,"value":106},"When the Orientation block names a matched scenario, follow its ",{"type":40,"tag":64,"props":108,"children":110},{"className":109},[],[111],{"type":45,"value":112},"Next:",{"type":45,"value":114},"\ncommands. The per-step playbook below is the fallback for runs where no\nscenario matched or the matched scenario's hints didn't resolve the issue.",{"type":40,"tag":54,"props":116,"children":118},{"id":117},"reading-diagnose-output",[119],{"type":45,"value":120},"Reading Diagnose Output",{"type":40,"tag":48,"props":122,"children":123},{},[124,126,131],{"type":45,"value":125},"Treat ",{"type":40,"tag":64,"props":127,"children":129},{"className":128},[],[130],{"type":45,"value":69},{"type":45,"value":132},"'s verdicts as authoritative for the queries it\nran. Two non-obvious classifications:",{"type":40,"tag":73,"props":134,"children":135},{},[136,151],{"type":40,"tag":77,"props":137,"children":138},{},[139,149],{"type":40,"tag":140,"props":141,"children":142},"strong",{},[143],{"type":40,"tag":64,"props":144,"children":146},{"className":145},[],[147],{"type":45,"value":148},"WARN — metric exists … but no series match the requested env\u002Fnamespace scope",{"type":45,"value":150}," means the metric is flowing on the stack but\ndoesn't carry the scoped label value. This is a label-mapping issue\n(asserts_env vs. deployment_environment, etc.), not a missing-data\nissue. Investigate the label pipeline (Step 6) before suggesting the\nuser enable new telemetry.",{"type":40,"tag":77,"props":152,"children":153},{},[154,165],{"type":40,"tag":140,"props":155,"children":156},{},[157,163],{"type":40,"tag":64,"props":158,"children":160},{"className":159},[],[161],{"type":45,"value":162},"FAIL — no data",{"type":45,"value":164}," (without the WARN above)",{"type":45,"value":166}," means the metric was\nnot found, even unscoped — the integration \u002F recording rule is\ngenuinely absent.",{"type":40,"tag":48,"props":168,"children":169},{},[170,172,178],{"type":45,"value":171},"For ad-hoc PromQL outside ",{"type":40,"tag":64,"props":173,"children":175},{"className":174},[],[176],{"type":45,"value":177},"kg diagnose",{"type":45,"value":179},", apply the same discipline:\nre-run the query without the env \u002F namespace filter before concluding\nthe data is missing.",{"type":40,"tag":48,"props":181,"children":182},{},[183],{"type":45,"value":184},"If a user reports an entity by name, see Step 7's entity-existence\nworkflow before assuming it exists on this stack.",{"type":40,"tag":54,"props":186,"children":188},{"id":187},"prerequisites",[189],{"type":45,"value":190},"Prerequisites",{"type":40,"tag":48,"props":192,"children":193},{},[194,196,201],{"type":45,"value":195},"gcx must be installed and configured with a valid context. ",{"type":40,"tag":64,"props":197,"children":199},{"className":198},[],[200],{"type":45,"value":69},{"type":45,"value":202},"\nbundles most of the per-step checks below; the individual commands produce\nequivalent results when you need to drill into one check.",{"type":40,"tag":204,"props":205,"children":210},"pre",{"className":206,"code":207,"language":208,"meta":209,"style":209},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","gcx config view\ngcx kg status\n","bash","",[211],{"type":40,"tag":64,"props":212,"children":213},{"__ignoreMap":209},[214,237],{"type":40,"tag":215,"props":216,"children":219},"span",{"class":217,"line":218},"line",1,[220,226,232],{"type":40,"tag":215,"props":221,"children":223},{"style":222},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[224],{"type":45,"value":225},"gcx",{"type":40,"tag":215,"props":227,"children":229},{"style":228},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[230],{"type":45,"value":231}," config",{"type":40,"tag":215,"props":233,"children":234},{"style":228},[235],{"type":45,"value":236}," view\n",{"type":40,"tag":215,"props":238,"children":240},{"class":217,"line":239},2,[241,245,250],{"type":40,"tag":215,"props":242,"children":243},{"style":222},[244],{"type":45,"value":225},{"type":40,"tag":215,"props":246,"children":247},{"style":228},[248],{"type":45,"value":249}," kg",{"type":40,"tag":215,"props":251,"children":252},{"style":228},[253],{"type":45,"value":254}," status\n",{"type":40,"tag":48,"props":256,"children":257},{},[258,260,266,268,274,276,281,283,288],{"type":45,"value":259},"If ",{"type":40,"tag":64,"props":261,"children":263},{"className":262},[],[264],{"type":45,"value":265},"kg status",{"type":45,"value":267}," fails with a config or auth error (no context, connection\nrefused, 401), use the ",{"type":40,"tag":64,"props":269,"children":271},{"className":270},[],[272],{"type":45,"value":273},"setup-gcx",{"type":45,"value":275}," skill first. Do not route every error\nthere: a 404 means the Asserts plugin isn't installed on this stack, and 403s\nfrom entity endpoints while ",{"type":40,"tag":64,"props":277,"children":279},{"className":278},[],[280],{"type":45,"value":265},{"type":45,"value":282}," itself succeeds usually mean the\nKnowledge Graph isn't onboarded (or the token lacks plugin access) — those are\nAsserts onboarding\u002Fpermission issues ",{"type":40,"tag":64,"props":284,"children":286},{"className":285},[],[287],{"type":45,"value":273},{"type":45,"value":289}," cannot fix; handle them via\nStep 1's onboarding stop instead.",{"type":40,"tag":54,"props":291,"children":293},{"id":292},"step-1-stack-health",[294],{"type":45,"value":295},"Step 1: Stack Health",{"type":40,"tag":204,"props":297,"children":299},{"className":206,"code":298,"language":208,"meta":209,"style":209},"gcx kg status\n",[300],{"type":40,"tag":64,"props":301,"children":302},{"__ignoreMap":209},[303],{"type":40,"tag":215,"props":304,"children":305},{"class":217,"line":218},[306,310,314],{"type":40,"tag":215,"props":307,"children":308},{"style":222},[309],{"type":45,"value":225},{"type":40,"tag":215,"props":311,"children":312},{"style":228},[313],{"type":45,"value":249},{"type":40,"tag":215,"props":315,"children":316},{"style":228},[317],{"type":45,"value":254},{"type":40,"tag":48,"props":319,"children":320},{},[321,326,328,334,336,342,344,350,351,357],{"type":40,"tag":140,"props":322,"children":323},{},[324],{"type":45,"value":325},"Check:",{"type":45,"value":327}," ",{"type":40,"tag":64,"props":329,"children":331},{"className":330},[],[332],{"type":45,"value":333},"status",{"type":45,"value":335}," must be ",{"type":40,"tag":64,"props":337,"children":339},{"className":338},[],[340],{"type":45,"value":341},"\"complete\"",{"type":45,"value":343}," and ",{"type":40,"tag":64,"props":345,"children":347},{"className":346},[],[348],{"type":45,"value":349},"enabled",{"type":45,"value":335},{"type":40,"tag":64,"props":352,"children":354},{"className":353},[],[355],{"type":45,"value":356},"true",{"type":45,"value":358},". If not,\nthe Knowledge Graph hasn't been onboarded — stop here and direct the user to\nthe Asserts app onboarding flow.",{"type":40,"tag":48,"props":360,"children":361},{},[362,367,368,373],{"type":40,"tag":140,"props":363,"children":364},{},[365],{"type":45,"value":366},"Shortcut:",{"type":45,"value":327},{"type":40,"tag":64,"props":369,"children":371},{"className":370},[],[372],{"type":45,"value":69},{"type":45,"value":374}," runs this plus all subsequent checks in parallel.",{"type":40,"tag":54,"props":376,"children":378},{"id":377},"step-2-entity-counts-and-scopes",[379],{"type":45,"value":380},"Step 2: Entity Counts and Scopes",{"type":40,"tag":204,"props":382,"children":384},{"className":206,"code":383,"language":208,"meta":209,"style":209},"gcx kg stats --since 1h\ngcx kg meta scopes\n",[385],{"type":40,"tag":64,"props":386,"children":387},{"__ignoreMap":209},[388,414],{"type":40,"tag":215,"props":389,"children":390},{"class":217,"line":218},[391,395,399,404,409],{"type":40,"tag":215,"props":392,"children":393},{"style":222},[394],{"type":45,"value":225},{"type":40,"tag":215,"props":396,"children":397},{"style":228},[398],{"type":45,"value":249},{"type":40,"tag":215,"props":400,"children":401},{"style":228},[402],{"type":45,"value":403}," stats",{"type":40,"tag":215,"props":405,"children":406},{"style":228},[407],{"type":45,"value":408}," --since",{"type":40,"tag":215,"props":410,"children":411},{"style":228},[412],{"type":45,"value":413}," 1h\n",{"type":40,"tag":215,"props":415,"children":416},{"class":217,"line":239},[417,421,425,430],{"type":40,"tag":215,"props":418,"children":419},{"style":222},[420],{"type":45,"value":225},{"type":40,"tag":215,"props":422,"children":423},{"style":228},[424],{"type":45,"value":249},{"type":40,"tag":215,"props":426,"children":427},{"style":228},[428],{"type":45,"value":429}," meta",{"type":40,"tag":215,"props":431,"children":432},{"style":228},[433],{"type":45,"value":434}," scopes\n",{"type":40,"tag":48,"props":436,"children":437},{},[438,442,444,450,452,458,460,466,468,474],{"type":40,"tag":140,"props":439,"children":440},{},[441],{"type":45,"value":325},{"type":45,"value":443}," entity counts should be > 0. The ",{"type":40,"tag":64,"props":445,"children":447},{"className":446},[],[448],{"type":45,"value":449},"meta scopes",{"type":45,"value":451}," output shows\navailable ",{"type":40,"tag":64,"props":453,"children":455},{"className":454},[],[456],{"type":45,"value":457},"env",{"type":45,"value":459},", ",{"type":40,"tag":64,"props":461,"children":463},{"className":462},[],[464],{"type":45,"value":465},"site",{"type":45,"value":467},", and ",{"type":40,"tag":64,"props":469,"children":471},{"className":470},[],[472],{"type":45,"value":473},"namespace",{"type":45,"value":475}," values.",{"type":40,"tag":48,"props":477,"children":478},{},[479,481,486],{"type":45,"value":480},"If scoping to a specific environment, note the exact ",{"type":40,"tag":64,"props":482,"children":484},{"className":483},[],[485],{"type":45,"value":457},{"type":45,"value":487}," value — you'll\nuse it in all subsequent queries.",{"type":40,"tag":54,"props":489,"children":491},{"id":490},"step-3-source-metrics-in-mimir",[492],{"type":45,"value":493},"Step 3: Source Metrics in Mimir",{"type":40,"tag":48,"props":495,"children":496},{},[497,499,505,507,513],{"type":45,"value":498},"Check whether the raw telemetry that feeds Entity Graph exists. Raw Tempo\nmetrics use ",{"type":40,"tag":64,"props":500,"children":502},{"className":501},[],[503],{"type":45,"value":504},"deployment_environment",{"type":45,"value":506},", not ",{"type":40,"tag":64,"props":508,"children":510},{"className":509},[],[511],{"type":45,"value":512},"asserts_env",{"type":45,"value":514},".",{"type":40,"tag":48,"props":516,"children":517},{},[518,520,526,528,533,535,541,543,549,551,557,559,564],{"type":45,"value":519},"Note the label shape difference between the two metrics: ",{"type":40,"tag":64,"props":521,"children":523},{"className":522},[],[524],{"type":45,"value":525},"traces_target_info",{"type":45,"value":527},"\ndescribes a single service so it has one ",{"type":40,"tag":64,"props":529,"children":531},{"className":530},[],[532],{"type":45,"value":504},{"type":45,"value":534}," label;\n",{"type":40,"tag":64,"props":536,"children":538},{"className":537},[],[539],{"type":45,"value":540},"traces_service_graph_request_total",{"type":45,"value":542}," describes an edge between two services\nand exposes the env on both sides as ",{"type":40,"tag":64,"props":544,"children":546},{"className":545},[],[547],{"type":45,"value":548},"client_deployment_environment",{"type":45,"value":550}," and\n",{"type":40,"tag":64,"props":552,"children":554},{"className":553},[],[555],{"type":45,"value":556},"server_deployment_environment",{"type":45,"value":558}," — there is no unified ",{"type":40,"tag":64,"props":560,"children":562},{"className":561},[],[563],{"type":45,"value":504},{"type":45,"value":565},"\nlabel.",{"type":40,"tag":204,"props":567,"children":569},{"className":206,"code":568,"language":208,"meta":209,"style":209},"# Service identity (OTel traces)\ngcx metrics query 'count(traces_target_info)' --since 1h\ngcx metrics query 'count(traces_target_info{deployment_environment=\"ENV\"})' --since 1h\n\n# Call data (inter-service HTTP\u002FgRPC)\ngcx metrics query 'count(traces_service_graph_request_total)' --since 1h\n# Filter on server side (use client_deployment_environment for outbound view):\ngcx metrics query 'count(traces_service_graph_request_total{server_deployment_environment=\"ENV\"})' --since 1h\n",[570],{"type":40,"tag":64,"props":571,"children":572},{"__ignoreMap":209},[573,582,623,660,670,679,716,725],{"type":40,"tag":215,"props":574,"children":575},{"class":217,"line":218},[576],{"type":40,"tag":215,"props":577,"children":579},{"style":578},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[580],{"type":45,"value":581},"# Service identity (OTel traces)\n",{"type":40,"tag":215,"props":583,"children":584},{"class":217,"line":239},[585,589,594,599,605,610,615,619],{"type":40,"tag":215,"props":586,"children":587},{"style":222},[588],{"type":45,"value":225},{"type":40,"tag":215,"props":590,"children":591},{"style":228},[592],{"type":45,"value":593}," metrics",{"type":40,"tag":215,"props":595,"children":596},{"style":228},[597],{"type":45,"value":598}," query",{"type":40,"tag":215,"props":600,"children":602},{"style":601},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[603],{"type":45,"value":604}," '",{"type":40,"tag":215,"props":606,"children":607},{"style":228},[608],{"type":45,"value":609},"count(traces_target_info)",{"type":40,"tag":215,"props":611,"children":612},{"style":601},[613],{"type":45,"value":614},"'",{"type":40,"tag":215,"props":616,"children":617},{"style":228},[618],{"type":45,"value":408},{"type":40,"tag":215,"props":620,"children":621},{"style":228},[622],{"type":45,"value":413},{"type":40,"tag":215,"props":624,"children":626},{"class":217,"line":625},3,[627,631,635,639,643,648,652,656],{"type":40,"tag":215,"props":628,"children":629},{"style":222},[630],{"type":45,"value":225},{"type":40,"tag":215,"props":632,"children":633},{"style":228},[634],{"type":45,"value":593},{"type":40,"tag":215,"props":636,"children":637},{"style":228},[638],{"type":45,"value":598},{"type":40,"tag":215,"props":640,"children":641},{"style":601},[642],{"type":45,"value":604},{"type":40,"tag":215,"props":644,"children":645},{"style":228},[646],{"type":45,"value":647},"count(traces_target_info{deployment_environment=\"ENV\"})",{"type":40,"tag":215,"props":649,"children":650},{"style":601},[651],{"type":45,"value":614},{"type":40,"tag":215,"props":653,"children":654},{"style":228},[655],{"type":45,"value":408},{"type":40,"tag":215,"props":657,"children":658},{"style":228},[659],{"type":45,"value":413},{"type":40,"tag":215,"props":661,"children":663},{"class":217,"line":662},4,[664],{"type":40,"tag":215,"props":665,"children":667},{"emptyLinePlaceholder":666},true,[668],{"type":45,"value":669},"\n",{"type":40,"tag":215,"props":671,"children":673},{"class":217,"line":672},5,[674],{"type":40,"tag":215,"props":675,"children":676},{"style":578},[677],{"type":45,"value":678},"# Call data (inter-service HTTP\u002FgRPC)\n",{"type":40,"tag":215,"props":680,"children":682},{"class":217,"line":681},6,[683,687,691,695,699,704,708,712],{"type":40,"tag":215,"props":684,"children":685},{"style":222},[686],{"type":45,"value":225},{"type":40,"tag":215,"props":688,"children":689},{"style":228},[690],{"type":45,"value":593},{"type":40,"tag":215,"props":692,"children":693},{"style":228},[694],{"type":45,"value":598},{"type":40,"tag":215,"props":696,"children":697},{"style":601},[698],{"type":45,"value":604},{"type":40,"tag":215,"props":700,"children":701},{"style":228},[702],{"type":45,"value":703},"count(traces_service_graph_request_total)",{"type":40,"tag":215,"props":705,"children":706},{"style":601},[707],{"type":45,"value":614},{"type":40,"tag":215,"props":709,"children":710},{"style":228},[711],{"type":45,"value":408},{"type":40,"tag":215,"props":713,"children":714},{"style":228},[715],{"type":45,"value":413},{"type":40,"tag":215,"props":717,"children":719},{"class":217,"line":718},7,[720],{"type":40,"tag":215,"props":721,"children":722},{"style":578},[723],{"type":45,"value":724},"# Filter on server side (use client_deployment_environment for outbound view):\n",{"type":40,"tag":215,"props":726,"children":728},{"class":217,"line":727},8,[729,733,737,741,745,750,754,758],{"type":40,"tag":215,"props":730,"children":731},{"style":222},[732],{"type":45,"value":225},{"type":40,"tag":215,"props":734,"children":735},{"style":228},[736],{"type":45,"value":593},{"type":40,"tag":215,"props":738,"children":739},{"style":228},[740],{"type":45,"value":598},{"type":40,"tag":215,"props":742,"children":743},{"style":601},[744],{"type":45,"value":604},{"type":40,"tag":215,"props":746,"children":747},{"style":228},[748],{"type":45,"value":749},"count(traces_service_graph_request_total{server_deployment_environment=\"ENV\"})",{"type":40,"tag":215,"props":751,"children":752},{"style":601},[753],{"type":45,"value":614},{"type":40,"tag":215,"props":755,"children":756},{"style":228},[757],{"type":45,"value":408},{"type":40,"tag":215,"props":759,"children":760},{"style":228},[761],{"type":45,"value":413},{"type":40,"tag":48,"props":763,"children":764},{},[765],{"type":40,"tag":140,"props":766,"children":767},{},[768],{"type":45,"value":769},"Interpret:",{"type":40,"tag":73,"props":771,"children":772},{},[773,778],{"type":40,"tag":77,"props":774,"children":775},{},[776],{"type":45,"value":777},"Both have data → traces are flowing. Continue to Step 4.",{"type":40,"tag":77,"props":779,"children":780},{},[781],{"type":45,"value":782},"Both empty → no OTel traces for this environment. Entities may still exist\nvia Prometheus scraping. Continue to Step 4.",{"type":40,"tag":48,"props":784,"children":785},{},[786,788,794,796,801],{"type":45,"value":787},"For more specific verdicts on this metric pair (Tempo metrics generation\ndisabled, broken trace context propagation), run ",{"type":40,"tag":64,"props":789,"children":791},{"className":790},[],[792],{"type":45,"value":793},"gcx kg diagnose --env ENV",{"type":45,"value":795},"\nand read the check results — the command encodes the detection logic for\nthose two cases and emits a targeted recommendation. Service-name collisions\nare ",{"type":40,"tag":140,"props":797,"children":798},{},[799],{"type":45,"value":800},"not",{"type":45,"value":802}," detected by the command; spot them manually via self-loop edges\n(Step 7).",{"type":40,"tag":54,"props":804,"children":806},{"id":805},"step-4-recording-rules",[807],{"type":45,"value":808},"Step 4: Recording Rules",{"type":40,"tag":48,"props":810,"children":811},{},[812,814,820,822,827,828,833],{"type":45,"value":813},"Recording rules convert raw metrics into the ",{"type":40,"tag":64,"props":815,"children":817},{"className":816},[],[818],{"type":45,"value":819},"asserts:*",{"type":45,"value":821}," metrics that Entity\nGraph consumes. These use ",{"type":40,"tag":64,"props":823,"children":825},{"className":824},[],[826],{"type":45,"value":512},{"type":45,"value":506},{"type":40,"tag":64,"props":829,"children":831},{"className":830},[],[832],{"type":45,"value":504},{"type":45,"value":514},{"type":40,"tag":204,"props":835,"children":837},{"className":206,"code":836,"language":208,"meta":209,"style":209},"# Entity discovery (central to how services appear)\ngcx metrics query 'count(asserts:mixin_workload_job{asserts_env=\"ENV\"})' --since 1h\n\n# CALLS edges\ngcx metrics query 'count(asserts:relation:calls{asserts_env=\"ENV\"})' --since 1h\n\n# Request rate KPI\ngcx metrics query 'count(asserts:request:rate5m{asserts_env=\"ENV\"})' --since 1h\n",[838],{"type":40,"tag":64,"props":839,"children":840},{"__ignoreMap":209},[841,849,885,892,900,936,943,951],{"type":40,"tag":215,"props":842,"children":843},{"class":217,"line":218},[844],{"type":40,"tag":215,"props":845,"children":846},{"style":578},[847],{"type":45,"value":848},"# Entity discovery (central to how services appear)\n",{"type":40,"tag":215,"props":850,"children":851},{"class":217,"line":239},[852,856,860,864,868,873,877,881],{"type":40,"tag":215,"props":853,"children":854},{"style":222},[855],{"type":45,"value":225},{"type":40,"tag":215,"props":857,"children":858},{"style":228},[859],{"type":45,"value":593},{"type":40,"tag":215,"props":861,"children":862},{"style":228},[863],{"type":45,"value":598},{"type":40,"tag":215,"props":865,"children":866},{"style":601},[867],{"type":45,"value":604},{"type":40,"tag":215,"props":869,"children":870},{"style":228},[871],{"type":45,"value":872},"count(asserts:mixin_workload_job{asserts_env=\"ENV\"})",{"type":40,"tag":215,"props":874,"children":875},{"style":601},[876],{"type":45,"value":614},{"type":40,"tag":215,"props":878,"children":879},{"style":228},[880],{"type":45,"value":408},{"type":40,"tag":215,"props":882,"children":883},{"style":228},[884],{"type":45,"value":413},{"type":40,"tag":215,"props":886,"children":887},{"class":217,"line":625},[888],{"type":40,"tag":215,"props":889,"children":890},{"emptyLinePlaceholder":666},[891],{"type":45,"value":669},{"type":40,"tag":215,"props":893,"children":894},{"class":217,"line":662},[895],{"type":40,"tag":215,"props":896,"children":897},{"style":578},[898],{"type":45,"value":899},"# CALLS edges\n",{"type":40,"tag":215,"props":901,"children":902},{"class":217,"line":672},[903,907,911,915,919,924,928,932],{"type":40,"tag":215,"props":904,"children":905},{"style":222},[906],{"type":45,"value":225},{"type":40,"tag":215,"props":908,"children":909},{"style":228},[910],{"type":45,"value":593},{"type":40,"tag":215,"props":912,"children":913},{"style":228},[914],{"type":45,"value":598},{"type":40,"tag":215,"props":916,"children":917},{"style":601},[918],{"type":45,"value":604},{"type":40,"tag":215,"props":920,"children":921},{"style":228},[922],{"type":45,"value":923},"count(asserts:relation:calls{asserts_env=\"ENV\"})",{"type":40,"tag":215,"props":925,"children":926},{"style":601},[927],{"type":45,"value":614},{"type":40,"tag":215,"props":929,"children":930},{"style":228},[931],{"type":45,"value":408},{"type":40,"tag":215,"props":933,"children":934},{"style":228},[935],{"type":45,"value":413},{"type":40,"tag":215,"props":937,"children":938},{"class":217,"line":681},[939],{"type":40,"tag":215,"props":940,"children":941},{"emptyLinePlaceholder":666},[942],{"type":45,"value":669},{"type":40,"tag":215,"props":944,"children":945},{"class":217,"line":718},[946],{"type":40,"tag":215,"props":947,"children":948},{"style":578},[949],{"type":45,"value":950},"# Request rate KPI\n",{"type":40,"tag":215,"props":952,"children":953},{"class":217,"line":727},[954,958,962,966,970,975,979,983],{"type":40,"tag":215,"props":955,"children":956},{"style":222},[957],{"type":45,"value":225},{"type":40,"tag":215,"props":959,"children":960},{"style":228},[961],{"type":45,"value":593},{"type":40,"tag":215,"props":963,"children":964},{"style":228},[965],{"type":45,"value":598},{"type":40,"tag":215,"props":967,"children":968},{"style":601},[969],{"type":45,"value":604},{"type":40,"tag":215,"props":971,"children":972},{"style":228},[973],{"type":45,"value":974},"count(asserts:request:rate5m{asserts_env=\"ENV\"})",{"type":40,"tag":215,"props":976,"children":977},{"style":601},[978],{"type":45,"value":614},{"type":40,"tag":215,"props":980,"children":981},{"style":228},[982],{"type":45,"value":408},{"type":40,"tag":215,"props":984,"children":985},{"style":228},[986],{"type":45,"value":413},{"type":40,"tag":48,"props":988,"children":989},{},[990],{"type":40,"tag":140,"props":991,"children":992},{},[993],{"type":45,"value":769},{"type":40,"tag":73,"props":995,"children":996},{},[997,1016,1021],{"type":40,"tag":77,"props":998,"children":999},{},[1000,1006,1008,1014],{"type":40,"tag":64,"props":1001,"children":1003},{"className":1002},[],[1004],{"type":45,"value":1005},"asserts:mixin_workload_job",{"type":45,"value":1007}," has data but ",{"type":40,"tag":64,"props":1009,"children":1011},{"className":1010},[],[1012],{"type":45,"value":1013},"asserts:relation:calls",{"type":45,"value":1015}," doesn't →\nentities are discovered but no edges exist. Continue to Step 5.",{"type":40,"tag":77,"props":1017,"children":1018},{},[1019],{"type":45,"value":1020},"All empty → recording rules aren't producing output. Check Step 6 (labels).",{"type":40,"tag":77,"props":1022,"children":1023},{},[1024],{"type":45,"value":1025},"All have data → pipeline is healthy. For a specific missing service, go to Step 7.",{"type":40,"tag":54,"props":1027,"children":1029},{"id":1028},"step-5-edge-source-analysis",[1030],{"type":45,"value":1031},"Step 5: Edge Source Analysis",{"type":40,"tag":48,"props":1033,"children":1034},{},[1035],{"type":45,"value":1036},"CALLS edges can come from many sources, not just OTel traces:",{"type":40,"tag":1038,"props":1039,"children":1040},"table",{},[1041,1065],{"type":40,"tag":1042,"props":1043,"children":1044},"thead",{},[1045],{"type":40,"tag":1046,"props":1047,"children":1048},"tr",{},[1049,1055,1060],{"type":40,"tag":1050,"props":1051,"children":1052},"th",{},[1053],{"type":45,"value":1054},"Source",{"type":40,"tag":1050,"props":1056,"children":1057},{},[1058],{"type":45,"value":1059},"Input Metric",{"type":40,"tag":1050,"props":1061,"children":1062},{},[1063],{"type":45,"value":1064},"Requires Traces?",{"type":40,"tag":1066,"props":1067,"children":1068},"tbody",{},[1069,1095,1121,1146,1171,1192,1213,1234,1255,1276,1297],{"type":40,"tag":1046,"props":1070,"children":1071},{},[1072,1082,1090],{"type":40,"tag":1073,"props":1074,"children":1075},"td",{},[1076],{"type":40,"tag":64,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":45,"value":1081},"app_o11y_servicegraph",{"type":40,"tag":1073,"props":1083,"children":1084},{},[1085],{"type":40,"tag":64,"props":1086,"children":1088},{"className":1087},[],[1089],{"type":45,"value":540},{"type":40,"tag":1073,"props":1091,"children":1092},{},[1093],{"type":45,"value":1094},"Yes",{"type":40,"tag":1046,"props":1096,"children":1097},{},[1098,1107,1116],{"type":40,"tag":1073,"props":1099,"children":1100},{},[1101],{"type":40,"tag":64,"props":1102,"children":1104},{"className":1103},[],[1105],{"type":45,"value":1106},"springboot",{"type":40,"tag":1073,"props":1108,"children":1109},{},[1110],{"type":40,"tag":64,"props":1111,"children":1113},{"className":1112},[],[1114],{"type":45,"value":1115},"http_server_requests_seconds_count",{"type":40,"tag":1073,"props":1117,"children":1118},{},[1119],{"type":45,"value":1120},"No",{"type":40,"tag":1046,"props":1122,"children":1123},{},[1124,1133,1142],{"type":40,"tag":1073,"props":1125,"children":1126},{},[1127],{"type":40,"tag":64,"props":1128,"children":1130},{"className":1129},[],[1131],{"type":45,"value":1132},"nginx_ingress",{"type":40,"tag":1073,"props":1134,"children":1135},{},[1136],{"type":40,"tag":64,"props":1137,"children":1139},{"className":1138},[],[1140],{"type":45,"value":1141},"nginx_ingress_controller_requests",{"type":40,"tag":1073,"props":1143,"children":1144},{},[1145],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1147,"children":1148},{},[1149,1158,1167],{"type":40,"tag":1073,"props":1150,"children":1151},{},[1152],{"type":40,"tag":64,"props":1153,"children":1155},{"className":1154},[],[1156],{"type":45,"value":1157},"istio",{"type":40,"tag":1073,"props":1159,"children":1160},{},[1161],{"type":40,"tag":64,"props":1162,"children":1164},{"className":1163},[],[1165],{"type":45,"value":1166},"istio_requests_total",{"type":40,"tag":1073,"props":1168,"children":1169},{},[1170],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1172,"children":1173},{},[1174,1183,1188],{"type":40,"tag":1073,"props":1175,"children":1176},{},[1177],{"type":40,"tag":64,"props":1178,"children":1180},{"className":1179},[],[1181],{"type":45,"value":1182},"aws_rds",{"type":40,"tag":1073,"props":1184,"children":1185},{},[1186],{"type":45,"value":1187},"CloudWatch RDS metrics",{"type":40,"tag":1073,"props":1189,"children":1190},{},[1191],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1193,"children":1194},{},[1195,1204,1209],{"type":40,"tag":1073,"props":1196,"children":1197},{},[1198],{"type":40,"tag":64,"props":1199,"children":1201},{"className":1200},[],[1202],{"type":45,"value":1203},"aws_dynamodb",{"type":40,"tag":1073,"props":1205,"children":1206},{},[1207],{"type":45,"value":1208},"CloudWatch DynamoDB metrics",{"type":40,"tag":1073,"props":1210,"children":1211},{},[1212],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1214,"children":1215},{},[1216,1225,1230],{"type":40,"tag":1073,"props":1217,"children":1218},{},[1219],{"type":40,"tag":64,"props":1220,"children":1222},{"className":1221},[],[1223],{"type":45,"value":1224},"aws_s3",{"type":40,"tag":1073,"props":1226,"children":1227},{},[1228],{"type":45,"value":1229},"CloudWatch S3 metrics",{"type":40,"tag":1073,"props":1231,"children":1232},{},[1233],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1235,"children":1236},{},[1237,1246,1251],{"type":40,"tag":1073,"props":1238,"children":1239},{},[1240],{"type":40,"tag":64,"props":1241,"children":1243},{"className":1242},[],[1244],{"type":45,"value":1245},"aws_applicationelb",{"type":40,"tag":1073,"props":1247,"children":1248},{},[1249],{"type":45,"value":1250},"CloudWatch ALB metrics",{"type":40,"tag":1073,"props":1252,"children":1253},{},[1254],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1256,"children":1257},{},[1258,1267,1272],{"type":40,"tag":1073,"props":1259,"children":1260},{},[1261],{"type":40,"tag":64,"props":1262,"children":1264},{"className":1263},[],[1265],{"type":45,"value":1266},"azure_flexible_server",{"type":40,"tag":1073,"props":1268,"children":1269},{},[1270],{"type":45,"value":1271},"Azure DB metrics",{"type":40,"tag":1073,"props":1273,"children":1274},{},[1275],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1277,"children":1278},{},[1279,1288,1293],{"type":40,"tag":1073,"props":1280,"children":1281},{},[1282],{"type":40,"tag":64,"props":1283,"children":1285},{"className":1284},[],[1286],{"type":45,"value":1287},"kafka_exporter",{"type":40,"tag":1073,"props":1289,"children":1290},{},[1291],{"type":45,"value":1292},"Kafka exporter metrics",{"type":40,"tag":1073,"props":1294,"children":1295},{},[1296],{"type":45,"value":1120},{"type":40,"tag":1046,"props":1298,"children":1299},{},[1300,1309,1314],{"type":40,"tag":1073,"props":1301,"children":1302},{},[1303],{"type":40,"tag":64,"props":1304,"children":1306},{"className":1305},[],[1307],{"type":45,"value":1308},"dbo11y_*",{"type":40,"tag":1073,"props":1310,"children":1311},{},[1312],{"type":45,"value":1313},"Database observability metrics",{"type":40,"tag":1073,"props":1315,"children":1316},{},[1317],{"type":45,"value":1120},{"type":40,"tag":204,"props":1319,"children":1321},{"className":206,"code":1320,"language":208,"meta":209,"style":209},"# What edge sources are active on this stack?\ngcx metrics labels --label asserts_source\n\n# Check common Prometheus-based sources for a namespace:\ngcx metrics query 'count(http_server_requests_seconds_count{namespace=\"NS\"})' --since 1h\ngcx metrics query 'count(nginx_ingress_controller_requests{namespace=\"NS\"})' --since 1h\ngcx metrics query 'count(istio_requests_total{namespace=\"NS\"})' --since 1h\n",[1322],{"type":40,"tag":64,"props":1323,"children":1324},{"__ignoreMap":209},[1325,1333,1359,1366,1374,1410,1446],{"type":40,"tag":215,"props":1326,"children":1327},{"class":217,"line":218},[1328],{"type":40,"tag":215,"props":1329,"children":1330},{"style":578},[1331],{"type":45,"value":1332},"# What edge sources are active on this stack?\n",{"type":40,"tag":215,"props":1334,"children":1335},{"class":217,"line":239},[1336,1340,1344,1349,1354],{"type":40,"tag":215,"props":1337,"children":1338},{"style":222},[1339],{"type":45,"value":225},{"type":40,"tag":215,"props":1341,"children":1342},{"style":228},[1343],{"type":45,"value":593},{"type":40,"tag":215,"props":1345,"children":1346},{"style":228},[1347],{"type":45,"value":1348}," labels",{"type":40,"tag":215,"props":1350,"children":1351},{"style":228},[1352],{"type":45,"value":1353}," --label",{"type":40,"tag":215,"props":1355,"children":1356},{"style":228},[1357],{"type":45,"value":1358}," asserts_source\n",{"type":40,"tag":215,"props":1360,"children":1361},{"class":217,"line":625},[1362],{"type":40,"tag":215,"props":1363,"children":1364},{"emptyLinePlaceholder":666},[1365],{"type":45,"value":669},{"type":40,"tag":215,"props":1367,"children":1368},{"class":217,"line":662},[1369],{"type":40,"tag":215,"props":1370,"children":1371},{"style":578},[1372],{"type":45,"value":1373},"# Check common Prometheus-based sources for a namespace:\n",{"type":40,"tag":215,"props":1375,"children":1376},{"class":217,"line":672},[1377,1381,1385,1389,1393,1398,1402,1406],{"type":40,"tag":215,"props":1378,"children":1379},{"style":222},[1380],{"type":45,"value":225},{"type":40,"tag":215,"props":1382,"children":1383},{"style":228},[1384],{"type":45,"value":593},{"type":40,"tag":215,"props":1386,"children":1387},{"style":228},[1388],{"type":45,"value":598},{"type":40,"tag":215,"props":1390,"children":1391},{"style":601},[1392],{"type":45,"value":604},{"type":40,"tag":215,"props":1394,"children":1395},{"style":228},[1396],{"type":45,"value":1397},"count(http_server_requests_seconds_count{namespace=\"NS\"})",{"type":40,"tag":215,"props":1399,"children":1400},{"style":601},[1401],{"type":45,"value":614},{"type":40,"tag":215,"props":1403,"children":1404},{"style":228},[1405],{"type":45,"value":408},{"type":40,"tag":215,"props":1407,"children":1408},{"style":228},[1409],{"type":45,"value":413},{"type":40,"tag":215,"props":1411,"children":1412},{"class":217,"line":681},[1413,1417,1421,1425,1429,1434,1438,1442],{"type":40,"tag":215,"props":1414,"children":1415},{"style":222},[1416],{"type":45,"value":225},{"type":40,"tag":215,"props":1418,"children":1419},{"style":228},[1420],{"type":45,"value":593},{"type":40,"tag":215,"props":1422,"children":1423},{"style":228},[1424],{"type":45,"value":598},{"type":40,"tag":215,"props":1426,"children":1427},{"style":601},[1428],{"type":45,"value":604},{"type":40,"tag":215,"props":1430,"children":1431},{"style":228},[1432],{"type":45,"value":1433},"count(nginx_ingress_controller_requests{namespace=\"NS\"})",{"type":40,"tag":215,"props":1435,"children":1436},{"style":601},[1437],{"type":45,"value":614},{"type":40,"tag":215,"props":1439,"children":1440},{"style":228},[1441],{"type":45,"value":408},{"type":40,"tag":215,"props":1443,"children":1444},{"style":228},[1445],{"type":45,"value":413},{"type":40,"tag":215,"props":1447,"children":1448},{"class":217,"line":718},[1449,1453,1457,1461,1465,1470,1474,1478],{"type":40,"tag":215,"props":1450,"children":1451},{"style":222},[1452],{"type":45,"value":225},{"type":40,"tag":215,"props":1454,"children":1455},{"style":228},[1456],{"type":45,"value":593},{"type":40,"tag":215,"props":1458,"children":1459},{"style":228},[1460],{"type":45,"value":598},{"type":40,"tag":215,"props":1462,"children":1463},{"style":601},[1464],{"type":45,"value":604},{"type":40,"tag":215,"props":1466,"children":1467},{"style":228},[1468],{"type":45,"value":1469},"count(istio_requests_total{namespace=\"NS\"})",{"type":40,"tag":215,"props":1471,"children":1472},{"style":601},[1473],{"type":45,"value":614},{"type":40,"tag":215,"props":1475,"children":1476},{"style":228},[1477],{"type":45,"value":408},{"type":40,"tag":215,"props":1479,"children":1480},{"style":228},[1481],{"type":45,"value":413},{"type":40,"tag":48,"props":1483,"children":1484},{},[1485,1490,1492,1497],{"type":40,"tag":140,"props":1486,"children":1487},{},[1488],{"type":45,"value":1489},"Critical: Check for the asserts_env gap.",{"type":45,"value":1491}," If a source metric exists but has\nno ",{"type":40,"tag":64,"props":1493,"children":1495},{"className":1494},[],[1496],{"type":45,"value":512},{"type":45,"value":1498}," label, the recording rules silently drop it. This is the most\ncommon reason for \"metrics present but no edges\":",{"type":40,"tag":204,"props":1500,"children":1502},{"className":206,"code":1501,"language":208,"meta":209,"style":209},"# For each source that returned data above, check if it has asserts_env:\ngcx metrics query 'count(istio_requests_total{asserts_env!=\"\"})' --since 1h\ngcx metrics query 'count(http_server_requests_seconds_count{asserts_env!=\"\"})' --since 1h\ngcx metrics query 'count(nginx_ingress_controller_requests{asserts_env!=\"\"})' --since 1h\n",[1503],{"type":40,"tag":64,"props":1504,"children":1505},{"__ignoreMap":209},[1506,1514,1550,1586],{"type":40,"tag":215,"props":1507,"children":1508},{"class":217,"line":218},[1509],{"type":40,"tag":215,"props":1510,"children":1511},{"style":578},[1512],{"type":45,"value":1513},"# For each source that returned data above, check if it has asserts_env:\n",{"type":40,"tag":215,"props":1515,"children":1516},{"class":217,"line":239},[1517,1521,1525,1529,1533,1538,1542,1546],{"type":40,"tag":215,"props":1518,"children":1519},{"style":222},[1520],{"type":45,"value":225},{"type":40,"tag":215,"props":1522,"children":1523},{"style":228},[1524],{"type":45,"value":593},{"type":40,"tag":215,"props":1526,"children":1527},{"style":228},[1528],{"type":45,"value":598},{"type":40,"tag":215,"props":1530,"children":1531},{"style":601},[1532],{"type":45,"value":604},{"type":40,"tag":215,"props":1534,"children":1535},{"style":228},[1536],{"type":45,"value":1537},"count(istio_requests_total{asserts_env!=\"\"})",{"type":40,"tag":215,"props":1539,"children":1540},{"style":601},[1541],{"type":45,"value":614},{"type":40,"tag":215,"props":1543,"children":1544},{"style":228},[1545],{"type":45,"value":408},{"type":40,"tag":215,"props":1547,"children":1548},{"style":228},[1549],{"type":45,"value":413},{"type":40,"tag":215,"props":1551,"children":1552},{"class":217,"line":625},[1553,1557,1561,1565,1569,1574,1578,1582],{"type":40,"tag":215,"props":1554,"children":1555},{"style":222},[1556],{"type":45,"value":225},{"type":40,"tag":215,"props":1558,"children":1559},{"style":228},[1560],{"type":45,"value":593},{"type":40,"tag":215,"props":1562,"children":1563},{"style":228},[1564],{"type":45,"value":598},{"type":40,"tag":215,"props":1566,"children":1567},{"style":601},[1568],{"type":45,"value":604},{"type":40,"tag":215,"props":1570,"children":1571},{"style":228},[1572],{"type":45,"value":1573},"count(http_server_requests_seconds_count{asserts_env!=\"\"})",{"type":40,"tag":215,"props":1575,"children":1576},{"style":601},[1577],{"type":45,"value":614},{"type":40,"tag":215,"props":1579,"children":1580},{"style":228},[1581],{"type":45,"value":408},{"type":40,"tag":215,"props":1583,"children":1584},{"style":228},[1585],{"type":45,"value":413},{"type":40,"tag":215,"props":1587,"children":1588},{"class":217,"line":662},[1589,1593,1597,1601,1605,1610,1614,1618],{"type":40,"tag":215,"props":1590,"children":1591},{"style":222},[1592],{"type":45,"value":225},{"type":40,"tag":215,"props":1594,"children":1595},{"style":228},[1596],{"type":45,"value":593},{"type":40,"tag":215,"props":1598,"children":1599},{"style":228},[1600],{"type":45,"value":598},{"type":40,"tag":215,"props":1602,"children":1603},{"style":601},[1604],{"type":45,"value":604},{"type":40,"tag":215,"props":1606,"children":1607},{"style":228},[1608],{"type":45,"value":1609},"count(nginx_ingress_controller_requests{asserts_env!=\"\"})",{"type":40,"tag":215,"props":1611,"children":1612},{"style":601},[1613],{"type":45,"value":614},{"type":40,"tag":215,"props":1615,"children":1616},{"style":228},[1617],{"type":45,"value":408},{"type":40,"tag":215,"props":1619,"children":1620},{"style":228},[1621],{"type":45,"value":413},{"type":40,"tag":48,"props":1623,"children":1624},{},[1625,1627,1633,1635,1640,1642,1647],{"type":45,"value":1626},"If the metric exists but the ",{"type":40,"tag":64,"props":1628,"children":1630},{"className":1629},[],[1631],{"type":45,"value":1632},"asserts_env!=\"\"",{"type":45,"value":1634}," query returns \"No data\", the\nMimir relabeling rules don't cover this source. The fix is to add a relabeling\nrule that maps ",{"type":40,"tag":64,"props":1636,"children":1638},{"className":1637},[],[1639],{"type":45,"value":473},{"type":45,"value":1641}," or another label to ",{"type":40,"tag":64,"props":1643,"children":1645},{"className":1644},[],[1646],{"type":45,"value":512},{"type":45,"value":1648}," for this metric.",{"type":40,"tag":48,"props":1650,"children":1651},{},[1652],{"type":40,"tag":140,"props":1653,"children":1654},{},[1655],{"type":45,"value":769},{"type":40,"tag":73,"props":1657,"children":1658},{},[1659,1664,1683],{"type":40,"tag":77,"props":1660,"children":1661},{},[1662],{"type":45,"value":1663},"No edge sources for this environment → edges are expected to be missing.\nServices need tracing or one of the Prometheus-based sources above.",{"type":40,"tag":77,"props":1665,"children":1666},{},[1667,1669,1674,1676,1681],{"type":45,"value":1668},"Edge source exists but missing ",{"type":40,"tag":64,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":45,"value":512},{"type":45,"value":1675}," → relabeling gap. Recording\nrules require ",{"type":40,"tag":64,"props":1677,"children":1679},{"className":1678},[],[1680],{"type":45,"value":1632},{"type":45,"value":1682}," and will silently ignore this data.",{"type":40,"tag":77,"props":1684,"children":1685},{},[1686,1688,1694,1696,1702],{"type":45,"value":1687},"If services are discovered via JMX (",{"type":40,"tag":64,"props":1689,"children":1691},{"className":1690},[],[1692],{"type":45,"value":1693},"job",{"type":45,"value":1695}," contains ",{"type":40,"tag":64,"props":1697,"children":1699},{"className":1698},[],[1700],{"type":45,"value":1701},"jmx",{"type":45,"value":1703},") → JMX alone\ncannot produce edges. Spring Boot Actuator or OTel tracing is needed.",{"type":40,"tag":48,"props":1705,"children":1706},{},[1707,1711,1712,1717,1719,1724],{"type":40,"tag":140,"props":1708,"children":1709},{},[1710],{"type":45,"value":366},{"type":45,"value":327},{"type":40,"tag":64,"props":1713,"children":1715},{"className":1714},[],[1716],{"type":45,"value":69},{"type":45,"value":1718}," detects this gap automatically and warns\nwhen edge source metrics exist but lack ",{"type":40,"tag":64,"props":1720,"children":1722},{"className":1721},[],[1723],{"type":45,"value":512},{"type":45,"value":514},{"type":40,"tag":48,"props":1726,"children":1727},{},[1728,1733,1735,1740,1742,1747,1749,1754,1756,1761,1763,1768,1770,1775,1777,1782,1784,1790],{"type":40,"tag":140,"props":1729,"children":1730},{},[1731],{"type":45,"value":1732},"Most common fix:",{"type":45,"value":1734}," If metrics have ",{"type":40,"tag":64,"props":1736,"children":1738},{"className":1737},[],[1739],{"type":45,"value":504},{"type":45,"value":1741}," but not\n",{"type":40,"tag":64,"props":1743,"children":1745},{"className":1744},[],[1746],{"type":45,"value":512},{"type":45,"value":1748},", the Asserts environment mapping is misconfigured. Go to\n",{"type":40,"tag":140,"props":1750,"children":1751},{},[1752],{"type":45,"value":1753},"Asserts app → Configuration → Connect Environment → Prometheus",{"type":45,"value":1755}," and set\nthe environment label to ",{"type":40,"tag":64,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":45,"value":504},{"type":45,"value":1762},". This tells the Mimir\nrelabeling pipeline to derive ",{"type":40,"tag":64,"props":1764,"children":1766},{"className":1765},[],[1767],{"type":45,"value":512},{"type":45,"value":1769}," from ",{"type":40,"tag":64,"props":1771,"children":1773},{"className":1772},[],[1774],{"type":45,"value":504},{"type":45,"value":1776},"\non ",{"type":40,"tag":140,"props":1778,"children":1779},{},[1780],{"type":45,"value":1781},"all",{"type":45,"value":1783}," incoming metrics — not just ",{"type":40,"tag":64,"props":1785,"children":1787},{"className":1786},[],[1788],{"type":45,"value":1789},"target_info",{"type":45,"value":514},{"type":40,"tag":48,"props":1792,"children":1793},{},[1794,1813,1815,1820,1822,1828,1830,1835,1837,1842,1844,1850],{"type":40,"tag":140,"props":1795,"children":1796},{},[1797,1799,1804,1806,1811],{"type":45,"value":1798},"If metrics lack both ",{"type":40,"tag":64,"props":1800,"children":1802},{"className":1801},[],[1803],{"type":45,"value":504},{"type":45,"value":1805}," AND ",{"type":40,"tag":64,"props":1807,"children":1809},{"className":1808},[],[1810],{"type":45,"value":512},{"type":45,"value":1812},":",{"type":45,"value":1814}," The\nscrape pipeline needs to add ",{"type":40,"tag":64,"props":1816,"children":1818},{"className":1817},[],[1819],{"type":45,"value":504},{"type":45,"value":1821}," first. In Alloy, use\n",{"type":40,"tag":64,"props":1823,"children":1825},{"className":1824},[],[1826],{"type":45,"value":1827},"prometheus.relabel",{"type":45,"value":1829}," to copy ",{"type":40,"tag":64,"props":1831,"children":1833},{"className":1832},[],[1834],{"type":45,"value":473},{"type":45,"value":1836}," (or another label) to\n",{"type":40,"tag":64,"props":1838,"children":1840},{"className":1839},[],[1841],{"type":45,"value":504},{"type":45,"value":1843}," before ",{"type":40,"tag":64,"props":1845,"children":1847},{"className":1846},[],[1848],{"type":45,"value":1849},"remote_write",{"type":45,"value":1851},". Then configure the Connect\nEnvironment page as above.",{"type":40,"tag":48,"props":1853,"children":1854},{},[1855,1860,1862,1867,1869,1874],{"type":40,"tag":140,"props":1856,"children":1857},{},[1858],{"type":45,"value":1859},"Alternative path:",{"type":45,"value":1861}," Enable OTel tracing to get edges via\n",{"type":40,"tag":64,"props":1863,"children":1865},{"className":1864},[],[1866],{"type":45,"value":540},{"type":45,"value":1868}," instead. Tempo generates this metric\nserver-side with ",{"type":40,"tag":64,"props":1870,"children":1872},{"className":1871},[],[1873],{"type":45,"value":512},{"type":45,"value":1875}," already populated, bypassing the Mimir\nrelabeling pipeline entirely.",{"type":40,"tag":54,"props":1877,"children":1879},{"id":1878},"step-6-label-pipeline",[1880],{"type":45,"value":1881},"Step 6: Label Pipeline",{"type":40,"tag":48,"props":1883,"children":1884},{},[1885,1887,1892,1894,1899],{"type":45,"value":1886},"The most common issue: ",{"type":40,"tag":64,"props":1888,"children":1890},{"className":1889},[],[1891],{"type":45,"value":504},{"type":45,"value":1893}," isn't mapped to ",{"type":40,"tag":64,"props":1895,"children":1897},{"className":1896},[],[1898],{"type":45,"value":512},{"type":45,"value":514},{"type":40,"tag":204,"props":1901,"children":1903},{"className":206,"code":1902,"language":208,"meta":209,"style":209},"gcx metrics labels --label deployment_environment\ngcx metrics labels --label asserts_env\n",[1904],{"type":40,"tag":64,"props":1905,"children":1906},{"__ignoreMap":209},[1907,1931],{"type":40,"tag":215,"props":1908,"children":1909},{"class":217,"line":218},[1910,1914,1918,1922,1926],{"type":40,"tag":215,"props":1911,"children":1912},{"style":222},[1913],{"type":45,"value":225},{"type":40,"tag":215,"props":1915,"children":1916},{"style":228},[1917],{"type":45,"value":593},{"type":40,"tag":215,"props":1919,"children":1920},{"style":228},[1921],{"type":45,"value":1348},{"type":40,"tag":215,"props":1923,"children":1924},{"style":228},[1925],{"type":45,"value":1353},{"type":40,"tag":215,"props":1927,"children":1928},{"style":228},[1929],{"type":45,"value":1930}," deployment_environment\n",{"type":40,"tag":215,"props":1932,"children":1933},{"class":217,"line":239},[1934,1938,1942,1946,1950],{"type":40,"tag":215,"props":1935,"children":1936},{"style":222},[1937],{"type":45,"value":225},{"type":40,"tag":215,"props":1939,"children":1940},{"style":228},[1941],{"type":45,"value":593},{"type":40,"tag":215,"props":1943,"children":1944},{"style":228},[1945],{"type":45,"value":1348},{"type":40,"tag":215,"props":1947,"children":1948},{"style":228},[1949],{"type":45,"value":1353},{"type":40,"tag":215,"props":1951,"children":1952},{"style":228},[1953],{"type":45,"value":1954}," asserts_env\n",{"type":40,"tag":48,"props":1956,"children":1957},{},[1958,1962,1964,1969,1971,1976],{"type":40,"tag":140,"props":1959,"children":1960},{},[1961],{"type":45,"value":325},{"type":45,"value":1963}," Every ",{"type":40,"tag":64,"props":1965,"children":1967},{"className":1966},[],[1968],{"type":45,"value":504},{"type":45,"value":1970}," value should have a corresponding\n",{"type":40,"tag":64,"props":1972,"children":1974},{"className":1973},[],[1975],{"type":45,"value":512},{"type":45,"value":1977}," value. If one is missing, the Mimir relabeling rules aren't\nconfigured for that environment.",{"type":40,"tag":48,"props":1979,"children":1980},{},[1981,1983,1988,1990,1995],{"type":45,"value":1982},"Extra ",{"type":40,"tag":64,"props":1984,"children":1986},{"className":1985},[],[1987],{"type":45,"value":512},{"type":45,"value":1989}," values (like AWS account IDs) that don't match any\n",{"type":40,"tag":64,"props":1991,"children":1993},{"className":1992},[],[1994],{"type":45,"value":504},{"type":45,"value":1996}," are normal — they come from non-OTel sources.",{"type":40,"tag":48,"props":1998,"children":1999},{},[2000,2004,2005,2011],{"type":40,"tag":140,"props":2001,"children":2002},{},[2003],{"type":45,"value":366},{"type":45,"value":327},{"type":40,"tag":64,"props":2006,"children":2008},{"className":2007},[],[2009],{"type":45,"value":2010},"gcx kg diagnose labels",{"type":45,"value":2012}," automates this cross-reference.",{"type":40,"tag":54,"props":2014,"children":2016},{"id":2015},"step-7-per-service-investigation",[2017],{"type":45,"value":2018},"Step 7: Per-Service Investigation",{"type":40,"tag":48,"props":2020,"children":2021},{},[2022],{"type":45,"value":2023},"For a specific missing or edge-less service:",{"type":40,"tag":204,"props":2025,"children":2027},{"className":206,"code":2026,"language":208,"meta":209,"style":209},"# Find in graph\ngcx kg entities query \"MATCH (s:Service {name: \\\"SERVICE\\\"}) RETURN s\" --since 1h\n\n# Check relationships\ngcx kg entities query \"MATCH (s:Service {name: \\\"SERVICE\\\"})-[r]-(other) RETURN s, r, other\" --since 1h\n\n# Source metrics\ngcx metrics query 'count(traces_service_graph_request_total{client=\"SERVICE\"})' --since 1h\ngcx metrics query 'count(traces_service_graph_request_total{server=\"SERVICE\"})' --since 1h\n\n# Recording rule output\ngcx metrics query 'count(asserts:relation:calls{service=\"SERVICE\"})' --since 1h\ngcx metrics query 'count(asserts:mixin_workload_job{service=\"SERVICE\"})' --since 1h\n",[2028],{"type":40,"tag":64,"props":2029,"children":2030},{"__ignoreMap":209},[2031,2039,2102,2109,2117,2173,2180,2188,2224,2261,2269,2278,2315],{"type":40,"tag":215,"props":2032,"children":2033},{"class":217,"line":218},[2034],{"type":40,"tag":215,"props":2035,"children":2036},{"style":578},[2037],{"type":45,"value":2038},"# Find in graph\n",{"type":40,"tag":215,"props":2040,"children":2041},{"class":217,"line":239},[2042,2046,2050,2055,2059,2064,2069,2075,2080,2084,2089,2094,2098],{"type":40,"tag":215,"props":2043,"children":2044},{"style":222},[2045],{"type":45,"value":225},{"type":40,"tag":215,"props":2047,"children":2048},{"style":228},[2049],{"type":45,"value":249},{"type":40,"tag":215,"props":2051,"children":2052},{"style":228},[2053],{"type":45,"value":2054}," entities",{"type":40,"tag":215,"props":2056,"children":2057},{"style":228},[2058],{"type":45,"value":598},{"type":40,"tag":215,"props":2060,"children":2061},{"style":601},[2062],{"type":45,"value":2063}," \"",{"type":40,"tag":215,"props":2065,"children":2066},{"style":228},[2067],{"type":45,"value":2068},"MATCH (s:Service {name: ",{"type":40,"tag":215,"props":2070,"children":2072},{"style":2071},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[2073],{"type":45,"value":2074},"\\\"",{"type":40,"tag":215,"props":2076,"children":2077},{"style":228},[2078],{"type":45,"value":2079},"SERVICE",{"type":40,"tag":215,"props":2081,"children":2082},{"style":2071},[2083],{"type":45,"value":2074},{"type":40,"tag":215,"props":2085,"children":2086},{"style":228},[2087],{"type":45,"value":2088},"}) RETURN s",{"type":40,"tag":215,"props":2090,"children":2091},{"style":601},[2092],{"type":45,"value":2093},"\"",{"type":40,"tag":215,"props":2095,"children":2096},{"style":228},[2097],{"type":45,"value":408},{"type":40,"tag":215,"props":2099,"children":2100},{"style":228},[2101],{"type":45,"value":413},{"type":40,"tag":215,"props":2103,"children":2104},{"class":217,"line":625},[2105],{"type":40,"tag":215,"props":2106,"children":2107},{"emptyLinePlaceholder":666},[2108],{"type":45,"value":669},{"type":40,"tag":215,"props":2110,"children":2111},{"class":217,"line":662},[2112],{"type":40,"tag":215,"props":2113,"children":2114},{"style":578},[2115],{"type":45,"value":2116},"# Check relationships\n",{"type":40,"tag":215,"props":2118,"children":2119},{"class":217,"line":672},[2120,2124,2128,2132,2136,2140,2144,2148,2152,2156,2161,2165,2169],{"type":40,"tag":215,"props":2121,"children":2122},{"style":222},[2123],{"type":45,"value":225},{"type":40,"tag":215,"props":2125,"children":2126},{"style":228},[2127],{"type":45,"value":249},{"type":40,"tag":215,"props":2129,"children":2130},{"style":228},[2131],{"type":45,"value":2054},{"type":40,"tag":215,"props":2133,"children":2134},{"style":228},[2135],{"type":45,"value":598},{"type":40,"tag":215,"props":2137,"children":2138},{"style":601},[2139],{"type":45,"value":2063},{"type":40,"tag":215,"props":2141,"children":2142},{"style":228},[2143],{"type":45,"value":2068},{"type":40,"tag":215,"props":2145,"children":2146},{"style":2071},[2147],{"type":45,"value":2074},{"type":40,"tag":215,"props":2149,"children":2150},{"style":228},[2151],{"type":45,"value":2079},{"type":40,"tag":215,"props":2153,"children":2154},{"style":2071},[2155],{"type":45,"value":2074},{"type":40,"tag":215,"props":2157,"children":2158},{"style":228},[2159],{"type":45,"value":2160},"})-[r]-(other) RETURN s, r, other",{"type":40,"tag":215,"props":2162,"children":2163},{"style":601},[2164],{"type":45,"value":2093},{"type":40,"tag":215,"props":2166,"children":2167},{"style":228},[2168],{"type":45,"value":408},{"type":40,"tag":215,"props":2170,"children":2171},{"style":228},[2172],{"type":45,"value":413},{"type":40,"tag":215,"props":2174,"children":2175},{"class":217,"line":681},[2176],{"type":40,"tag":215,"props":2177,"children":2178},{"emptyLinePlaceholder":666},[2179],{"type":45,"value":669},{"type":40,"tag":215,"props":2181,"children":2182},{"class":217,"line":718},[2183],{"type":40,"tag":215,"props":2184,"children":2185},{"style":578},[2186],{"type":45,"value":2187},"# Source metrics\n",{"type":40,"tag":215,"props":2189,"children":2190},{"class":217,"line":727},[2191,2195,2199,2203,2207,2212,2216,2220],{"type":40,"tag":215,"props":2192,"children":2193},{"style":222},[2194],{"type":45,"value":225},{"type":40,"tag":215,"props":2196,"children":2197},{"style":228},[2198],{"type":45,"value":593},{"type":40,"tag":215,"props":2200,"children":2201},{"style":228},[2202],{"type":45,"value":598},{"type":40,"tag":215,"props":2204,"children":2205},{"style":601},[2206],{"type":45,"value":604},{"type":40,"tag":215,"props":2208,"children":2209},{"style":228},[2210],{"type":45,"value":2211},"count(traces_service_graph_request_total{client=\"SERVICE\"})",{"type":40,"tag":215,"props":2213,"children":2214},{"style":601},[2215],{"type":45,"value":614},{"type":40,"tag":215,"props":2217,"children":2218},{"style":228},[2219],{"type":45,"value":408},{"type":40,"tag":215,"props":2221,"children":2222},{"style":228},[2223],{"type":45,"value":413},{"type":40,"tag":215,"props":2225,"children":2227},{"class":217,"line":2226},9,[2228,2232,2236,2240,2244,2249,2253,2257],{"type":40,"tag":215,"props":2229,"children":2230},{"style":222},[2231],{"type":45,"value":225},{"type":40,"tag":215,"props":2233,"children":2234},{"style":228},[2235],{"type":45,"value":593},{"type":40,"tag":215,"props":2237,"children":2238},{"style":228},[2239],{"type":45,"value":598},{"type":40,"tag":215,"props":2241,"children":2242},{"style":601},[2243],{"type":45,"value":604},{"type":40,"tag":215,"props":2245,"children":2246},{"style":228},[2247],{"type":45,"value":2248},"count(traces_service_graph_request_total{server=\"SERVICE\"})",{"type":40,"tag":215,"props":2250,"children":2251},{"style":601},[2252],{"type":45,"value":614},{"type":40,"tag":215,"props":2254,"children":2255},{"style":228},[2256],{"type":45,"value":408},{"type":40,"tag":215,"props":2258,"children":2259},{"style":228},[2260],{"type":45,"value":413},{"type":40,"tag":215,"props":2262,"children":2264},{"class":217,"line":2263},10,[2265],{"type":40,"tag":215,"props":2266,"children":2267},{"emptyLinePlaceholder":666},[2268],{"type":45,"value":669},{"type":40,"tag":215,"props":2270,"children":2272},{"class":217,"line":2271},11,[2273],{"type":40,"tag":215,"props":2274,"children":2275},{"style":578},[2276],{"type":45,"value":2277},"# Recording rule output\n",{"type":40,"tag":215,"props":2279,"children":2281},{"class":217,"line":2280},12,[2282,2286,2290,2294,2298,2303,2307,2311],{"type":40,"tag":215,"props":2283,"children":2284},{"style":222},[2285],{"type":45,"value":225},{"type":40,"tag":215,"props":2287,"children":2288},{"style":228},[2289],{"type":45,"value":593},{"type":40,"tag":215,"props":2291,"children":2292},{"style":228},[2293],{"type":45,"value":598},{"type":40,"tag":215,"props":2295,"children":2296},{"style":601},[2297],{"type":45,"value":604},{"type":40,"tag":215,"props":2299,"children":2300},{"style":228},[2301],{"type":45,"value":2302},"count(asserts:relation:calls{service=\"SERVICE\"})",{"type":40,"tag":215,"props":2304,"children":2305},{"style":601},[2306],{"type":45,"value":614},{"type":40,"tag":215,"props":2308,"children":2309},{"style":228},[2310],{"type":45,"value":408},{"type":40,"tag":215,"props":2312,"children":2313},{"style":228},[2314],{"type":45,"value":413},{"type":40,"tag":215,"props":2316,"children":2318},{"class":217,"line":2317},13,[2319,2323,2327,2331,2335,2340,2344,2348],{"type":40,"tag":215,"props":2320,"children":2321},{"style":222},[2322],{"type":45,"value":225},{"type":40,"tag":215,"props":2324,"children":2325},{"style":228},[2326],{"type":45,"value":593},{"type":40,"tag":215,"props":2328,"children":2329},{"style":228},[2330],{"type":45,"value":598},{"type":40,"tag":215,"props":2332,"children":2333},{"style":601},[2334],{"type":45,"value":604},{"type":40,"tag":215,"props":2336,"children":2337},{"style":228},[2338],{"type":45,"value":2339},"count(asserts:mixin_workload_job{service=\"SERVICE\"})",{"type":40,"tag":215,"props":2341,"children":2342},{"style":601},[2343],{"type":45,"value":614},{"type":40,"tag":215,"props":2345,"children":2346},{"style":228},[2347],{"type":45,"value":408},{"type":40,"tag":215,"props":2349,"children":2350},{"style":228},[2351],{"type":45,"value":413},{"type":40,"tag":48,"props":2353,"children":2354},{},[2355],{"type":40,"tag":140,"props":2356,"children":2357},{},[2358],{"type":45,"value":769},{"type":40,"tag":73,"props":2360,"children":2361},{},[2362,2367,2398,2410],{"type":40,"tag":77,"props":2363,"children":2364},{},[2365],{"type":45,"value":2366},"Found via Cypher but no relationships → check source metrics above.",{"type":40,"tag":77,"props":2368,"children":2369},{},[2370,2376,2378,2383,2385,2390,2391,2396],{"type":40,"tag":64,"props":2371,"children":2373},{"className":2372},[],[2374],{"type":45,"value":2375},"server",{"type":45,"value":2377}," series exist but ",{"type":40,"tag":64,"props":2379,"children":2381},{"className":2380},[],[2382],{"type":45,"value":1013},{"type":45,"value":2384}," doesn't → recording rule\nlabel mismatch (check ",{"type":40,"tag":64,"props":2386,"children":2388},{"className":2387},[],[2389],{"type":45,"value":512},{"type":45,"value":343},{"type":40,"tag":64,"props":2392,"children":2394},{"className":2393},[],[2395],{"type":45,"value":473},{"type":45,"value":2397},").",{"type":40,"tag":77,"props":2399,"children":2400},{},[2401,2403,2409],{"type":45,"value":2402},"Not found via Cypher → check ",{"type":40,"tag":64,"props":2404,"children":2406},{"className":2405},[],[2407],{"type":45,"value":2408},"traces_target_info{service_name=\"SERVICE\"}",{"type":45,"value":514},{"type":40,"tag":77,"props":2411,"children":2412},{},[2413],{"type":45,"value":2414},"Leaf services (queue consumers, processors) correctly have no outgoing edges.",{"type":40,"tag":48,"props":2416,"children":2417},{},[2418,2422,2423,2429],{"type":40,"tag":140,"props":2419,"children":2420},{},[2421],{"type":45,"value":366},{"type":45,"value":327},{"type":40,"tag":64,"props":2424,"children":2426},{"className":2425},[],[2427],{"type":45,"value":2428},"gcx kg diagnose service SERVICE --env ENV",{"type":45,"value":2430}," runs the checks\nabove and produces an interpreted diagnosis with suggested next steps.",{"type":40,"tag":48,"props":2432,"children":2433},{},[2434,2436,2440],{"type":45,"value":2435},"Two common patterns that present as \"missing entities\" are ",{"type":40,"tag":140,"props":2437,"children":2438},{},[2439],{"type":45,"value":800},{"type":45,"value":2441}," detected\nby the command — check for them manually:",{"type":40,"tag":73,"props":2443,"children":2444},{},[2445,2484],{"type":40,"tag":77,"props":2446,"children":2447},{},[2448,2453,2455,2461,2463,2469,2470,2475,2477,2482],{"type":40,"tag":140,"props":2449,"children":2450},{},[2451],{"type":45,"value":2452},"Service-name collision",{"type":45,"value":2454}," (multiple workloads share one ",{"type":40,"tag":64,"props":2456,"children":2458},{"className":2457},[],[2459],{"type":45,"value":2460},"service.name",{"type":45,"value":2462},",\ncollapsing into one entity): look for self-loop edges — series where both\n",{"type":40,"tag":64,"props":2464,"children":2466},{"className":2465},[],[2467],{"type":45,"value":2468},"client",{"type":45,"value":343},{"type":40,"tag":64,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":45,"value":2375},{"type":45,"value":2476}," equal SERVICE in ",{"type":40,"tag":64,"props":2478,"children":2480},{"className":2479},[],[2481],{"type":45,"value":540},{"type":45,"value":2483},"\n— and for a single entity where you expected several workloads.",{"type":40,"tag":77,"props":2485,"children":2486},{},[2487,2492,2494,2500,2502,2507,2509,2514],{"type":40,"tag":140,"props":2488,"children":2489},{},[2490],{"type":45,"value":2491},"Env-scope split",{"type":45,"value":2493}," (workloads in the same namespace disagree on\n",{"type":40,"tag":64,"props":2495,"children":2497},{"className":2496},[],[2498],{"type":45,"value":2499},"deployment.environment",{"type":45,"value":2501},", so cross-env calls don't render as edges):\ncompare ",{"type":40,"tag":64,"props":2503,"children":2505},{"className":2504},[],[2506],{"type":45,"value":548},{"type":45,"value":2508}," vs ",{"type":40,"tag":64,"props":2510,"children":2512},{"className":2511},[],[2513],{"type":45,"value":556},{"type":45,"value":2515},"\non the service's edge series from Step 3.",{"type":40,"tag":54,"props":2517,"children":2519},{"id":2518},"producing-a-report",[2520],{"type":45,"value":2521},"Producing a Report",{"type":40,"tag":48,"props":2523,"children":2524},{},[2525],{"type":45,"value":2526},"Summarize findings as:",{"type":40,"tag":2528,"props":2529,"children":2530},"ol",{},[2531,2541,2551,2561,2584,2609,2619,2643,2653],{"type":40,"tag":77,"props":2532,"children":2533},{},[2534,2539],{"type":40,"tag":140,"props":2535,"children":2536},{},[2537],{"type":45,"value":2538},"Stack health",{"type":45,"value":2540}," — KG enabled and complete?",{"type":40,"tag":77,"props":2542,"children":2543},{},[2544,2549],{"type":40,"tag":140,"props":2545,"children":2546},{},[2547],{"type":45,"value":2548},"Entity count",{"type":45,"value":2550}," — how many for the scoped environment?",{"type":40,"tag":77,"props":2552,"children":2553},{},[2554,2559],{"type":40,"tag":140,"props":2555,"children":2556},{},[2557],{"type":45,"value":2558},"Discovery path",{"type":45,"value":2560}," — OTel traces, Prometheus scrape, or cloud integration?",{"type":40,"tag":77,"props":2562,"children":2563},{},[2564,2569,2571,2576,2577,2582],{"type":40,"tag":140,"props":2565,"children":2566},{},[2567],{"type":45,"value":2568},"Trace data",{"type":45,"value":2570}," — do ",{"type":40,"tag":64,"props":2572,"children":2574},{"className":2573},[],[2575],{"type":45,"value":525},{"type":45,"value":343},{"type":40,"tag":64,"props":2578,"children":2580},{"className":2579},[],[2581],{"type":45,"value":540},{"type":45,"value":2583}," exist?",{"type":40,"tag":77,"props":2585,"children":2586},{},[2587,2592,2594,2599,2601,2607],{"type":40,"tag":140,"props":2588,"children":2589},{},[2590],{"type":45,"value":2591},"Edge data",{"type":45,"value":2593}," — does ",{"type":40,"tag":64,"props":2595,"children":2597},{"className":2596},[],[2598],{"type":45,"value":1013},{"type":45,"value":2600}," exist? Which ",{"type":40,"tag":64,"props":2602,"children":2604},{"className":2603},[],[2605],{"type":45,"value":2606},"asserts_source",{"type":45,"value":2608}," values?",{"type":40,"tag":77,"props":2610,"children":2611},{},[2612,2617],{"type":40,"tag":140,"props":2613,"children":2614},{},[2615],{"type":45,"value":2616},"Alternative edge sources",{"type":45,"value":2618}," — Spring Boot, nginx, Istio, cloud integrations available?",{"type":40,"tag":77,"props":2620,"children":2621},{},[2622,2627,2629,2634,2636,2641],{"type":40,"tag":140,"props":2623,"children":2624},{},[2625],{"type":45,"value":2626},"Label mapping",{"type":45,"value":2628}," — ",{"type":40,"tag":64,"props":2630,"children":2632},{"className":2631},[],[2633],{"type":45,"value":504},{"type":45,"value":2635}," correctly mapped to ",{"type":40,"tag":64,"props":2637,"children":2639},{"className":2638},[],[2640],{"type":45,"value":512},{"type":45,"value":2642},"?",{"type":40,"tag":77,"props":2644,"children":2645},{},[2646,2651],{"type":40,"tag":140,"props":2647,"children":2648},{},[2649],{"type":45,"value":2650},"Conclusion",{"type":45,"value":2652}," — expected state or configuration issue?",{"type":40,"tag":77,"props":2654,"children":2655},{},[2656,2661],{"type":40,"tag":140,"props":2657,"children":2658},{},[2659],{"type":45,"value":2660},"Recommendations",{"type":45,"value":2662}," — what would fix it?",{"type":40,"tag":48,"props":2664,"children":2665},{},[2666,2668,2673,2675,2681],{"type":45,"value":2667},"When recommending a fix, set expectations on convergence time. The metrics\nthe Knowledge Graph reads from (",{"type":40,"tag":64,"props":2669,"children":2671},{"className":2670},[],[2672],{"type":45,"value":819},{"type":45,"value":2674}," recording rules, and the\n",{"type":40,"tag":64,"props":2676,"children":2678},{"className":2677},[],[2679],{"type":45,"value":2680},"traces_*",{"type":45,"value":2682}," series Tempo generates) are time-series with a query lookback\nwindow — old data with the broken state will keep appearing in queries\nfor at least 5–15 minutes after the fix is applied. The Entity Graph UI\nshould fully stabilize on the corrected state within that window.",{"type":40,"tag":2684,"props":2685,"children":2686},"style",{},[2687],{"type":45,"value":2688},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2690,"total":2869},[2691,2710,2729,2749,2762,2778,2791,2806,2823,2836,2843,2857],{"slug":2692,"name":2692,"fn":2693,"description":2694,"org":2695,"tags":2696,"stars":2707,"repoUrl":2708,"updatedAt":2709},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2697,2700,2703,2706],{"name":2698,"slug":2699,"type":15},"Distributed Tracing","distributed-tracing",{"name":2701,"slug":2702,"type":15},"Frontend","frontend",{"name":2704,"slug":2705,"type":15},"Monitoring","monitoring",{"name":17,"slug":18,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":2711,"name":2711,"fn":2712,"description":2713,"org":2714,"tags":2715,"stars":2726,"repoUrl":2727,"updatedAt":2728},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2716,2719,2722,2725],{"name":2717,"slug":2718,"type":15},"API Development","api-development",{"name":2720,"slug":2721,"type":15},"Authentication","authentication",{"name":2723,"slug":2724,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":2730,"name":2730,"fn":2731,"description":2732,"org":2733,"tags":2734,"stars":2726,"repoUrl":2727,"updatedAt":2748},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2735,2738,2741,2742,2745],{"name":2736,"slug":2737,"type":15},"CSV","csv",{"name":2739,"slug":2740,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":2743,"slug":2744,"type":15},"GraphQL","graphql",{"name":2746,"slug":2747,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":2750,"name":2750,"fn":2751,"description":2752,"org":2753,"tags":2754,"stars":23,"repoUrl":24,"updatedAt":2761},"agento11y","manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2755,2758,2759,2760],{"name":2756,"slug":2757,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2704,"slug":2705,"type":15},{"name":17,"slug":18,"type":15},"2026-07-25T05:30:40.29622",{"slug":2763,"name":2763,"fn":2764,"description":2765,"org":2766,"tags":2767,"stars":23,"repoUrl":24,"updatedAt":2777},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2768,2769,2770,2773,2776],{"name":2756,"slug":2757,"type":15},{"name":9,"slug":8,"type":15},{"name":2771,"slug":2772,"type":15},"Instrumentation","instrumentation",{"name":2774,"slug":2775,"type":15},"LLM","llm",{"name":17,"slug":18,"type":15},"2026-07-31T05:53:52.580237",{"slug":2779,"name":2779,"fn":2780,"description":2781,"org":2782,"tags":2783,"stars":23,"repoUrl":24,"updatedAt":2790},"agento11y-prod-setup","setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2784,2785,2788,2789],{"name":2756,"slug":2757,"type":15},{"name":2786,"slug":2787,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},"2026-07-31T05:53:53.576347",{"slug":2792,"name":2792,"fn":2793,"description":2794,"org":2795,"tags":2796,"stars":23,"repoUrl":24,"updatedAt":2805},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2797,2798,2799,2802],{"name":2756,"slug":2757,"type":15},{"name":9,"slug":8,"type":15},{"name":2800,"slug":2801,"type":15},"QA","qa",{"name":2803,"slug":2804,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":2807,"name":2807,"fn":2808,"description":2809,"org":2810,"tags":2811,"stars":23,"repoUrl":24,"updatedAt":2822},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2812,2815,2818,2821],{"name":2813,"slug":2814,"type":15},"Dashboards","dashboards",{"name":2816,"slug":2817,"type":15},"Data Visualization","data-visualization",{"name":2819,"slug":2820,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":2824,"name":2824,"fn":2825,"description":2826,"org":2827,"tags":2828,"stars":23,"repoUrl":24,"updatedAt":2835},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2829,2830,2831,2834],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":2832,"slug":2833,"type":15},"Incident Response","incident-response",{"name":17,"slug":18,"type":15},"2026-07-18T05:11:10.445428",{"slug":4,"name":4,"fn":5,"description":6,"org":2837,"tags":2838,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2839,2840,2841,2842],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"slug":225,"name":225,"fn":2844,"description":2845,"org":2846,"tags":2847,"stars":23,"repoUrl":24,"updatedAt":2856},"manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2848,2851,2852,2853],{"name":2849,"slug":2850,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":2704,"slug":2705,"type":15},{"name":2854,"slug":2855,"type":15},"Operations","operations","2026-07-31T05:53:50.587304",{"slug":2858,"name":2858,"fn":2859,"description":2860,"org":2861,"tags":2862,"stars":23,"repoUrl":24,"updatedAt":2868},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2863,2864,2865],{"name":2849,"slug":2850,"type":15},{"name":9,"slug":8,"type":15},{"name":2866,"slug":2867,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80,{"items":2871,"total":2922},[2872,2879,2887,2894,2901,2908,2915],{"slug":2750,"name":2750,"fn":2751,"description":2752,"org":2873,"tags":2874,"stars":23,"repoUrl":24,"updatedAt":2761},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2875,2876,2877,2878],{"name":2756,"slug":2757,"type":15},{"name":9,"slug":8,"type":15},{"name":2704,"slug":2705,"type":15},{"name":17,"slug":18,"type":15},{"slug":2763,"name":2763,"fn":2764,"description":2765,"org":2880,"tags":2881,"stars":23,"repoUrl":24,"updatedAt":2777},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2882,2883,2884,2885,2886],{"name":2756,"slug":2757,"type":15},{"name":9,"slug":8,"type":15},{"name":2771,"slug":2772,"type":15},{"name":2774,"slug":2775,"type":15},{"name":17,"slug":18,"type":15},{"slug":2779,"name":2779,"fn":2780,"description":2781,"org":2888,"tags":2889,"stars":23,"repoUrl":24,"updatedAt":2790},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2890,2891,2892,2893],{"name":2756,"slug":2757,"type":15},{"name":2786,"slug":2787,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"slug":2792,"name":2792,"fn":2793,"description":2794,"org":2895,"tags":2896,"stars":23,"repoUrl":24,"updatedAt":2805},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2897,2898,2899,2900],{"name":2756,"slug":2757,"type":15},{"name":9,"slug":8,"type":15},{"name":2800,"slug":2801,"type":15},{"name":2803,"slug":2804,"type":15},{"slug":2807,"name":2807,"fn":2808,"description":2809,"org":2902,"tags":2903,"stars":23,"repoUrl":24,"updatedAt":2822},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2904,2905,2906,2907],{"name":2813,"slug":2814,"type":15},{"name":2816,"slug":2817,"type":15},{"name":2819,"slug":2820,"type":15},{"name":9,"slug":8,"type":15},{"slug":2824,"name":2824,"fn":2825,"description":2826,"org":2909,"tags":2910,"stars":23,"repoUrl":24,"updatedAt":2835},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2911,2912,2913,2914],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":2832,"slug":2833,"type":15},{"name":17,"slug":18,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":2916,"tags":2917,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2918,2919,2920,2921],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},24]