[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-prometheus-cardinality-troubleshooter":3,"mdc-pcxim8-key":37,"related-repo-grafana-prometheus-cardinality-troubleshooter":3722,"related-org-grafana-prometheus-cardinality-troubleshooter":3835},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"prometheus-cardinality-troubleshooter","troubleshoot Prometheus cardinality issues","Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing Prometheus, high Grafana Cloud Active Series or DPM bills, \"too many samples\" ingest errors, series churn, or rapid memory growth. Walks through tsdb status endpoints, per-metric and per-label drill-downs, common-culprit galleries, and remediation paths. Use when the user is *currently experiencing* a cardinality fire. For preventing cardinality issues at the source, route to prometheus-label-strategy. For post-ingest aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,20,23],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Prometheus","prometheus",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Metrics","metrics",{"name":24,"slug":25,"type":15},"Debugging","debugging",189,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills","2026-07-12T07:44:24.575127","Apache-2.0",16,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],null,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fgrafana-cloud\u002Fprometheus-cardinality-troubleshooter","---\nname: prometheus-cardinality-troubleshooter\nlicense: Apache-2.0\ndescription: >\n  Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing\n  Prometheus, high Grafana Cloud Active Series or DPM bills, \"too many samples\" ingest\n  errors, series churn, or rapid memory growth. Walks through tsdb status endpoints,\n  per-metric and per-label drill-downs, common-culprit galleries, and remediation paths.\n  Use when the user is *currently experiencing* a cardinality fire. For preventing\n  cardinality issues at the source, route to prometheus-label-strategy. For post-ingest\n  aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.\n---\n\n# Prometheus Cardinality Troubleshooter\n\nYou are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.\n\nThis skill is **diagnostic and operational**. For schema design and prevention, route to `prometheus-label-strategy`.\n\n---\n\n## Before You Remediate: The One Rule\n\nUnder pressure, the tempting move is to `labeldrop` the high-cardinality label at scrape time. **Do not.** You cannot remove, at scrape time, any label that makes a series unique — not `pod`, not `instance`, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually **breaks the data**:\n\n- Counter resets from different series get merged → `rate()` and `increase()` return garbage, often *absurdly high* values.\n- Multiple samples land on the same series per scrape → duplicate-sample \u002F out-of-order errors and **inflated** DPM, not reduced.\n- The breakage is silent (no config error) and leaves no evidence in the data of where it went wrong. Weeks later someone asks \"why is my DPM so high \u002F why is `rate()` absurd?\" and there's nothing to point to.\n\nThe only safe remediations are:\n\n1. **Drop an *entire* unwanted metric** (`action: drop` on `__name__`) — you're discarding the whole metric, not merging distinct series.\n2. **Fix the source** — stop the application emitting the bad label (the real fix for unbounded `path`, `user_id`, etc.).\n3. **Adaptive Metrics** — for structural cardinality on series you can't fix at the source. It aggregates *correctly* (counter-reset-aware, audited, reversible). This is the right way to reduce the cost of a label like `pod`. Route to `adaptive-metrics`.\n\nEverywhere below that says \"drop a label,\" read it through this rule: drop whole metrics, fix the source, or use Adaptive Metrics — never `labeldrop` a distinguishing label.\n\n---\n\n## Symptom → Likely Cause\n\n| Symptom | Likely Cause | First Action |\n|---|---|---|\n| Prometheus OOMKilled or memory growing linearly | Active series growth (often from a new bad metric or label) | [Active Series triage](#step-1-active-series-triage) |\n| Single PromQL query slow or OOMs the querier | One or more metrics in the query have high cardinality | [Per-query drill-down](#step-3-per-metric-drill-down) |\n| Remote write lagging, WAL growing | Sample throughput spike — series count OR scrape interval changed | [Active Series triage](#step-1-active-series-triage) + check scrape intervals |\n| `429 Too Many Samples` \u002F `out of bounds` errors | Hitting Mimir\u002FCortex ingester per-tenant series limit | [Per-metric drill-down](#step-3-per-metric-drill-down), find the new offender |\n| Grafana Cloud Active Series bill spiked | New metric, new label, or rollout creating churn | [Per-metric drill-down](#step-3-per-metric-drill-down) + churn check |\n| Grafana Cloud DPM bill spiked but Active Series flat | Scrape interval shortened, OR remote_write sending duplicates | DPM-side issue — route to `dpm-finder` |\n| `series_limit_per_user` errors after a deploy | Application change introduced a new bad label | [Recent change diff](#step-4-recent-change-diff) |\n| Series count grows then resets every restart | Series churn from ephemeral label values | [Churn diagnosis](#step-5-churn-diagnosis) |\n\n---\n\n## Step 1: Active Series Triage\n\n### Get the headline number\n\n```promql\n# Total active series in the local Prometheus\nprometheus_tsdb_head_series\n\n# Or for Mimir \u002F Grafana Cloud Metrics (per tenant)\ncortex_ingester_memory_series{user=\"\u003Ctenant>\"}\n```\n\nCompare to recent history:\n```promql\n# Growth over the last 7 days\nderiv(prometheus_tsdb_head_series[7d]) * 86400\n```\n\nA growth rate > a few % per day on a stable application set is a red flag.\n\n### Use the TSDB status endpoint\n\nPrometheus exposes a built-in cardinality breakdown:\n\n```bash\ncurl -s http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Fstatus\u002Ftsdb | jq\n```\n\nReturns:\n- `seriesCountByMetricName` — top metrics by series count\n- `labelValueCountByLabelName` — top labels by unique value count\n- `memoryInBytesByLabelName` — top labels by memory footprint\n- `seriesCountByLabelValuePair` — top label-value pairs by series count\n\nThis is usually the fastest path to \"which metric \u002F which label is the problem.\"\n\nFor Grafana Cloud:\n```bash\n# Same endpoint, authenticated against the per-tenant Mimir\ncurl -s -u \"\u003Cuser>:\u003Ctoken>\" \\\n  \"https:\u002F\u002Fprometheus-prod-XX.grafana.net\u002Fapi\u002Fprom\u002Fapi\u002Fv1\u002Fstatus\u002Ftsdb\" | jq\n```\n\n---\n\n## Step 2: Read the Output\n\n### Top metrics by series count\n\n```json\n\"seriesCountByMetricName\": [\n  { \"name\": \"http_request_duration_seconds_bucket\", \"value\": 184320 },\n  { \"name\": \"go_gc_duration_seconds\",               \"value\": 80 },\n  ...\n]\n```\n\n**Heuristics**:\n- A histogram (`_bucket`) at the top is almost always the answer — those have a 14× multiplier (bucket count + 3). The fix is usually **reducing the labels on the underlying histogram at the source** (in instrumentation code), not stripping them at scrape and not touching the buckets themselves.\n- A metric in the top 5 you don't recognize → grep the codebase for it; it's likely a new feature flag or a debug metric that shipped to prod\n- The same metric showing up under multiple variants (`_total`, `_count`, `_sum`) — that's a histogram or summary, count all variants together for the true impact\n\n### Top labels by unique value count\n\n```json\n\"labelValueCountByLabelName\": [\n  { \"name\": \"url\",       \"value\": 84210 },\n  { \"name\": \"trace_id\",  \"value\": 41000 },\n  { \"name\": \"pod\",       \"value\": 1820 }\n]\n```\n\n**Red flags**:\n- Any label with >10K unique values is almost certainly a bug. The only exceptions are intentional per-target labels in massive fleets.\n- `trace_id`, `request_id`, `session_id`, `query`, `email`, `path`, `url` — these should *never* be labels. They belong in exemplars, logs, or traces.\n- `pod` with thousands of values — see [Churn diagnosis](#step-5-churn-diagnosis); recent churn often inflates this number\n\n---\n\n## Step 3: Per-Metric Drill-Down\n\nOnce you've identified a suspect metric, find which label is responsible.\n\n### Count distinct label values per label, for one metric\n\n```promql\n# How many unique values does each label have on this metric?\ncount by (__name__) (\n  count by (__name__, label_name_here) (\n    http_request_duration_seconds_bucket\n  )\n)\n```\n\nRepeat per label, or use the helper:\n\n```bash\n# Via the Prometheus HTTP API\ncurl -s \"http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Flabels?match[]=http_request_duration_seconds_bucket\" | jq -r '.data[]' | \\\n  while read label; do\n    count=$(curl -s \"http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Flabel\u002F${label}\u002Fvalues?match[]=http_request_duration_seconds_bucket\" | jq '.data | length')\n    echo \"${count}  ${label}\"\n  done | sort -rn | head -20\n```\n\n### Find the top label values for one label\n\n```promql\n# Top 20 path values for http_requests_total\ntopk(20,\n  count by (path) (http_requests_total)\n)\n```\n\nIf you see UUIDs, hashes, timestamps, or numeric IDs in the top values → that label has unbounded values from the source.\n\n### Per-metric series count, grouped\n\n```promql\n# Series-per-instance breakdown — if uneven, one instance is misbehaving\nsum by (job, instance) ({__name__=~\"my_metric.*\"})\n```\n\n---\n\n## Step 4: Recent Change Diff\n\nIf the cardinality fire started recently, the cause is almost always a recent change. Diff what's there now against what was there before.\n\n### List of metrics, current vs. yesterday\n\nVia Grafana Cloud cardinality dashboard, or:\n```promql\n# Current metrics\ngroup by (__name__) ({__name__!=\"\"})\n\n# Compare to last week (offset)\ngroup by (__name__) ({__name__!=\"\"} offset 7d)\n```\n\nDiff externally. A new metric near the top of `seriesCountByMetricName` that wasn't there a week ago → that's your offender.\n\n### Correlate with deploys\n\n```promql\n# Active series correlated with build_info\nprometheus_tsdb_head_series\n# Overlay with:\nchanges(app_build_info[1d])\n```\n\nA vertical step in series count aligned with a deploy is conclusive.\n\n---\n\n## Step 5: Churn Diagnosis\n\nHigh churn means series are being created and abandoned faster than they age out. Symptoms: series count keeps climbing, then drops sharply on Prometheus restart.\n\n### Churn signal\n\n```promql\n# Series created vs. removed per second\nrate(prometheus_tsdb_head_series_created_total[5m])\nrate(prometheus_tsdb_head_series_removed_total[5m])\n\n# Ratio of churned to live\nprometheus_tsdb_head_series_created_total \u002F prometheus_tsdb_head_series\n```\n\nA creation rate that materially exceeds the removal rate, sustained, means cardinality is on a one-way trip up. Common causes:\n\n| Cause | Tell |\n|---|---|\n| Pod rollouts emitting `pod` label | Churn spike aligns with deploy timing; affects pod-discovered scrapes |\n| `version` \u002F `git_sha` \u002F `image_tag` label on every metric | Churn spike on every deploy across many metrics |\n| Ephemeral hostnames in `instance` | Cloud autoscaling event timing |\n| Bug: dynamic label names | Churn climbs forever, never plateaus |\n| Application bug emitting fresh UUIDs as labels | Linear unbounded growth, no deploy correlation |\n\n### Memory impact of churn\n\n```promql\n# A churn-driven head block carries old series until tsdb compaction\nprometheus_tsdb_head_chunks\ngo_memstats_heap_inuse_bytes{job=\"prometheus\"}\n```\n\nRestarting Prometheus drops churned series but is not a fix. The fix is at the source.\n\n---\n\n## Common-Culprit Gallery\n\n### Histogram blowup\n\n**Tell**: `*_bucket` metric at the top of `seriesCountByMetricName`. Multiplier ≈ 14×.\n\n**Fix**:\n1. First, **reduce labels on the histogram at the source** — every label removed saves 14× series. Trim `path`, `method`, or `status_code` in the instrumentation code (don't `labeldrop` them at scrape — that merges distinct histograms and corrupts the buckets). For series already in Grafana Cloud you can't change, aggregate them with Adaptive Metrics.\n2. Then, reduce bucket count if appropriate (custom buckets vs. defaults).\n3. For high-resolution latency tracking, consider **native histograms** (Prometheus 2.40+) — single sparse series replaces the bucket family.\n\n### kube-state-metrics label explosion\n\n**Tell**: `kube_pod_labels` or `kube_pod_annotations` at the top, with `label_*` or `annotation_*` labels driving cardinality.\n\n**Fix**: configure kube-state-metrics with `--metric-labels-allowlist` and `--metric-annotations-allowlist`. By default it emits *all* labels and annotations as series.\n\n```yaml\n# kube-state-metrics flags\n--metric-labels-allowlist=pods=[app,team,version]\n--metric-annotations-allowlist=pods=[checksum\u002Fconfig]\n```\n\n### Path \u002F route blowup from a new endpoint\n\n**Tell**: `http_requests_total` (or framework equivalent) grew 10×+ overnight. `topk(20, count by (path) (http_requests_total))` shows hundreds of `\u002Fusers\u002F123456`-style values.\n\n**Fix**: the real fix is to **template the path in application code** (`\u002Fusers\u002F:id`) — route the user to `prometheus-label-strategy`. For series already in Grafana Cloud, **Adaptive Metrics** can aggregate `path` away correctly — route to `adaptive-metrics`.\n\nDo **not** \"normalize\" `path` with a relabel `replacement` rule — collapsing `\u002Fusers\u002F123`, `\u002Fusers\u002F456`, … into one `\u002Fusers\u002F:id` value at scrape merges distinct series and produces duplicate-sample errors and broken `rate()`. The merge has to happen at the source (templating) or post-ingest (Adaptive Metrics), never at scrape.\n\nIf you must stop a production fire *right now* and templating isn't deployable yet, the only safe scrape-time action is to drop the **entire** offending metric (you lose it completely until the code fix lands — a deliberate trade, not a silent corruption):\n\n```yaml\n# Emergency: drop the whole metric until the source is templated\nmetric_relabel_configs:\n  - source_labels: [__name__]\n    regex: http_requests_total\n    action: drop\n```\n\n### Application emitting a debug metric in prod\n\n**Tell**: A metric you don't recognize in the top 10. Grep the source — often a `_details` or `_per_request` debug metric the developer forgot to gate.\n\n**Fix**: drop entirely at scrape:\n```yaml\nmetric_relabel_configs:\n  - source_labels: [__name__]\n    regex: my_app_request_details\n    action: drop\n```\n\nOpen a ticket against the team to remove it from the code.\n\n### App-emitted labels colliding with target labels\n\n**Tell**: Series count for one job is several × what it should be. Looking at one series, you see both an app-emitted `instance=...` AND the target `instance=...` collided into something weird (Prometheus renames the conflicting one to `exported_instance`).\n\n**Fix**: the right fix is **in the application** — stop emitting `instance`\u002F`node`\u002F`host` from code; they belong to the scrape target. Confirm `honor_labels` is `false` (the default) so the target labels win.\n\nIf you need a scrape-time stopgap, you may remove a label *only* where it **exactly duplicates** a target label — that's the one safe `labeldrop`, because the target label still provides uniqueness. Scope it tightly to the duplicated names and **never include `pod`** (or any other label that is the source of uniqueness):\n\n```yaml\n# Stopgap ONLY for app-emitted duplicates of target labels.\n# Drops the `exported_*` collisions — NOT pod, which makes K8s series unique.\nmetric_relabel_configs:\n  - regex: exported_(instance|node|host)\n    action: labeldrop\n```\n\nThen the target labels from `relabel_configs` apply cleanly. Prefer fixing the app.\n\n### Federation amplifying cardinality\n\n**Tell**: A federated Prometheus or Mimir global view has way more series than expected. Each source has its own `cluster` \u002F `region` label, multiplying.\n\n**Fix**: this is usually expected — federation by design preserves source labels. If the series count is too high, federate only aggregated recording rules, not raw metrics:\n\n```yaml\n- job_name: federate\n  honor_labels: true\n  metrics_path: \u002Ffederate\n  params:\n    'match[]':\n      - '{__name__=~\".*:.*\"}'  # Recording-rule naming convention only\n```\n\n---\n\n## Remediation Decision Tree\n\n```\nCardinality fire confirmed\n│\n├── Need to stop the bleeding NOW (production OOM, ingest 429s)\n│   └── Drop the ENTIRE offending metric via metric_relabel_configs (action: drop on __name__)\n│       (also applies to Alloy\u002FAgent — same syntax)\n│       Do NOT labeldrop a distinguishing label — it breaks the data, see \"The One Rule\".\n│       Then schedule the proper fix.\n│\n├── It's a Grafana Cloud Active Series bill issue, not a perf issue\n│   ├── Cardinality is structural and you can't fix the app\n│   │   └── Route to `adaptive-metrics` skill (post-ingest aggregation rules — the safe way)\n│   └── You want metric-by-metric DPM breakdown\n│       └── Route to `dpm-finder` skill\n│\n├── It's a fixable application bug (unbounded label, debug metric in prod)\n│   ├── Short-term: drop the whole metric at scrape, OR aggregate via Adaptive Metrics\n│   └── Long-term: fix in code; route to `prometheus-label-strategy` for design guidance\n│\n├── It's histogram cardinality\n│   ├── Reduce labels on the underlying histogram AT THE SOURCE (14× win per label)\n│   ├── Reduce bucket count if appropriate\n│   └── Consider native histograms for high-resolution latency\n│\n└── It's churn (deploy-driven)\n    ├── Stop EMITTING `version`\u002F`git_sha`\u002F`instance` from app code (use info-metric for version)\n    ├── Keep `pod` — never drop it; if pod-level series are too costly, use Adaptive Metrics\n    └── Verify K8s SD relabel rules aren't mapping in `uid` or other ephemeral fields\n```\n\n---\n\n## Emergency Drop Patterns (copy-paste ready)\n\nThese are the **safe** scrape-time emergency actions: dropping an *entire* unwanted metric. They do not merge distinct series, so they don't corrupt the data.\n\n> ⚠️ There is intentionally **no `labeldrop` of a distinguishing label** and **no value-normalizing relabel** here. Both merge distinct series and break `rate()`\u002FDPM (see [The One Rule](#before-you-remediate-the-one-rule)). To reduce cardinality *without* dropping the whole metric, fix the source or use **Adaptive Metrics** (route to `adaptive-metrics`). The only safe `labeldrop` is removing a label that *exactly duplicates* a target label (e.g. `exported_instance`) — see [App-emitted labels colliding with target labels](#app-emitted-labels-colliding-with-target-labels).\n\nFor Prometheus `scrape_configs`:\n\n```yaml\nmetric_relabel_configs:\n  # Drop a specific bad metric entirely\n  - source_labels: [__name__]\n    regex: bad_metric_name\n    action: drop\n\n  # Drop a set of debug\u002Ftemporary metrics by name prefix\n  - source_labels: [__name__]\n    regex: debug_.*\n    action: drop\n```\n\nFor Grafana Alloy (`prometheus.relabel` component):\n\n```alloy\nprometheus.relabel \"drop_bad_metric\" {\n  forward_to = [prometheus.remote_write.default.receiver]\n\n  rule {\n    source_labels = [\"__name__\"]\n    regex = \"bad_metric_name\"\n    action = \"drop\"\n  }\n}\n```\n\n**Always test in staging first**, and prefer fixing the source or using Adaptive Metrics over any scrape-time drop.\n\n---\n\n## When to Hand Off\n\n- **\"Now design a label strategy so this doesn't happen again\"** → `prometheus-label-strategy`\n- **\"We need to keep these metrics but reduce cost\"** → `adaptive-metrics`\n- **\"Which metric is the most expensive in DPM terms?\"** → `dpm-finder`\n- **\"Write the PromQL to find this\"** → `promql`\n- **\"Configure this in Alloy\"** → `alloy`\n- **\"Why is my Loki slow?\"** → `loki-label-analyzer` (different system, same family of problems)\n\nThis skill's lane is **diagnosis under pressure**. Prevention, design, and post-ingest cost optimization live elsewhere.\n",{"data":38,"body":39},{"name":4,"license":29,"description":6},{"type":40,"children":41},"root",[42,50,56,78,82,89,132,190,195,289,301,304,310,543,546,552,559,618,623,646,651,657,662,702,707,754,759,764,842,845,851,857,1045,1054,1109,1115,1348,1357,1438,1441,1447,1452,1458,1514,1519,1776,1782,1820,1825,1831,1854,1857,1863,1868,1874,1879,1925,1937,1943,1981,1986,1989,1995,2000,2006,2060,2065,2183,2189,2220,2225,2228,2234,2240,2263,2272,2333,2339,2378,2409,2442,2448,2480,2529,2585,2603,2696,2702,2726,2735,2811,2816,2822,2854,2908,2946,3017,3030,3036,3060,3069,3192,3195,3201,3211,3214,3220,3238,3328,3340,3501,3514,3594,3604,3607,3613,3704,3716],{"type":43,"tag":44,"props":45,"children":46},"element","h1",{"id":4},[47],{"type":48,"value":49},"text","Prometheus Cardinality Troubleshooter",{"type":43,"tag":51,"props":52,"children":53},"p",{},[54],{"type":48,"value":55},"You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.",{"type":43,"tag":51,"props":57,"children":58},{},[59,61,67,69,76],{"type":48,"value":60},"This skill is ",{"type":43,"tag":62,"props":63,"children":64},"strong",{},[65],{"type":48,"value":66},"diagnostic and operational",{"type":48,"value":68},". For schema design and prevention, route to ",{"type":43,"tag":70,"props":71,"children":73},"code",{"className":72},[],[74],{"type":48,"value":75},"prometheus-label-strategy",{"type":48,"value":77},".",{"type":43,"tag":79,"props":80,"children":81},"hr",{},[],{"type":43,"tag":83,"props":84,"children":86},"h2",{"id":85},"before-you-remediate-the-one-rule",[87],{"type":48,"value":88},"Before You Remediate: The One Rule",{"type":43,"tag":51,"props":90,"children":91},{},[92,94,100,102,107,109,115,117,123,125,130],{"type":48,"value":93},"Under pressure, the tempting move is to ",{"type":43,"tag":70,"props":95,"children":97},{"className":96},[],[98],{"type":48,"value":99},"labeldrop",{"type":48,"value":101}," the high-cardinality label at scrape time. ",{"type":43,"tag":62,"props":103,"children":104},{},[105],{"type":48,"value":106},"Do not.",{"type":48,"value":108}," You cannot remove, at scrape time, any label that makes a series unique — not ",{"type":43,"tag":70,"props":110,"children":112},{"className":111},[],[113],{"type":48,"value":114},"pod",{"type":48,"value":116},", not ",{"type":43,"tag":70,"props":118,"children":120},{"className":119},[],[121],{"type":48,"value":122},"instance",{"type":48,"value":124},", not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually ",{"type":43,"tag":62,"props":126,"children":127},{},[128],{"type":48,"value":129},"breaks the data",{"type":48,"value":131},":",{"type":43,"tag":133,"props":134,"children":135},"ul",{},[136,166,178],{"type":43,"tag":137,"props":138,"children":139},"li",{},[140,142,148,150,156,158,164],{"type":48,"value":141},"Counter resets from different series get merged → ",{"type":43,"tag":70,"props":143,"children":145},{"className":144},[],[146],{"type":48,"value":147},"rate()",{"type":48,"value":149}," and ",{"type":43,"tag":70,"props":151,"children":153},{"className":152},[],[154],{"type":48,"value":155},"increase()",{"type":48,"value":157}," return garbage, often ",{"type":43,"tag":159,"props":160,"children":161},"em",{},[162],{"type":48,"value":163},"absurdly high",{"type":48,"value":165}," values.",{"type":43,"tag":137,"props":167,"children":168},{},[169,171,176],{"type":48,"value":170},"Multiple samples land on the same series per scrape → duplicate-sample \u002F out-of-order errors and ",{"type":43,"tag":62,"props":172,"children":173},{},[174],{"type":48,"value":175},"inflated",{"type":48,"value":177}," DPM, not reduced.",{"type":43,"tag":137,"props":179,"children":180},{},[181,183,188],{"type":48,"value":182},"The breakage is silent (no config error) and leaves no evidence in the data of where it went wrong. Weeks later someone asks \"why is my DPM so high \u002F why is ",{"type":43,"tag":70,"props":184,"children":186},{"className":185},[],[187],{"type":48,"value":147},{"type":48,"value":189}," absurd?\" and there's nothing to point to.",{"type":43,"tag":51,"props":191,"children":192},{},[193],{"type":48,"value":194},"The only safe remediations are:",{"type":43,"tag":196,"props":197,"children":198},"ol",{},[199,232,258],{"type":43,"tag":137,"props":200,"children":201},{},[202,214,216,222,224,230],{"type":43,"tag":62,"props":203,"children":204},{},[205,207,212],{"type":48,"value":206},"Drop an ",{"type":43,"tag":159,"props":208,"children":209},{},[210],{"type":48,"value":211},"entire",{"type":48,"value":213}," unwanted metric",{"type":48,"value":215}," (",{"type":43,"tag":70,"props":217,"children":219},{"className":218},[],[220],{"type":48,"value":221},"action: drop",{"type":48,"value":223}," on ",{"type":43,"tag":70,"props":225,"children":227},{"className":226},[],[228],{"type":48,"value":229},"__name__",{"type":48,"value":231},") — you're discarding the whole metric, not merging distinct series.",{"type":43,"tag":137,"props":233,"children":234},{},[235,240,242,248,250,256],{"type":43,"tag":62,"props":236,"children":237},{},[238],{"type":48,"value":239},"Fix the source",{"type":48,"value":241}," — stop the application emitting the bad label (the real fix for unbounded ",{"type":43,"tag":70,"props":243,"children":245},{"className":244},[],[246],{"type":48,"value":247},"path",{"type":48,"value":249},", ",{"type":43,"tag":70,"props":251,"children":253},{"className":252},[],[254],{"type":48,"value":255},"user_id",{"type":48,"value":257},", etc.).",{"type":43,"tag":137,"props":259,"children":260},{},[261,266,268,273,275,280,282,288],{"type":43,"tag":62,"props":262,"children":263},{},[264],{"type":48,"value":265},"Adaptive Metrics",{"type":48,"value":267}," — for structural cardinality on series you can't fix at the source. It aggregates ",{"type":43,"tag":159,"props":269,"children":270},{},[271],{"type":48,"value":272},"correctly",{"type":48,"value":274}," (counter-reset-aware, audited, reversible). This is the right way to reduce the cost of a label like ",{"type":43,"tag":70,"props":276,"children":278},{"className":277},[],[279],{"type":48,"value":114},{"type":48,"value":281},". Route to ",{"type":43,"tag":70,"props":283,"children":285},{"className":284},[],[286],{"type":48,"value":287},"adaptive-metrics",{"type":48,"value":77},{"type":43,"tag":51,"props":290,"children":291},{},[292,294,299],{"type":48,"value":293},"Everywhere below that says \"drop a label,\" read it through this rule: drop whole metrics, fix the source, or use Adaptive Metrics — never ",{"type":43,"tag":70,"props":295,"children":297},{"className":296},[],[298],{"type":48,"value":99},{"type":48,"value":300}," a distinguishing label.",{"type":43,"tag":79,"props":302,"children":303},{},[],{"type":43,"tag":83,"props":305,"children":307},{"id":306},"symptom-likely-cause",[308],{"type":48,"value":309},"Symptom → Likely Cause",{"type":43,"tag":311,"props":312,"children":313},"table",{},[314,338],{"type":43,"tag":315,"props":316,"children":317},"thead",{},[318],{"type":43,"tag":319,"props":320,"children":321},"tr",{},[322,328,333],{"type":43,"tag":323,"props":324,"children":325},"th",{},[326],{"type":48,"value":327},"Symptom",{"type":43,"tag":323,"props":329,"children":330},{},[331],{"type":48,"value":332},"Likely Cause",{"type":43,"tag":323,"props":334,"children":335},{},[336],{"type":48,"value":337},"First Action",{"type":43,"tag":339,"props":340,"children":341},"tbody",{},[342,366,388,410,447,469,493,521],{"type":43,"tag":319,"props":343,"children":344},{},[345,351,356],{"type":43,"tag":346,"props":347,"children":348},"td",{},[349],{"type":48,"value":350},"Prometheus OOMKilled or memory growing linearly",{"type":43,"tag":346,"props":352,"children":353},{},[354],{"type":48,"value":355},"Active series growth (often from a new bad metric or label)",{"type":43,"tag":346,"props":357,"children":358},{},[359],{"type":43,"tag":360,"props":361,"children":363},"a",{"href":362},"#step-1-active-series-triage",[364],{"type":48,"value":365},"Active Series triage",{"type":43,"tag":319,"props":367,"children":368},{},[369,374,379],{"type":43,"tag":346,"props":370,"children":371},{},[372],{"type":48,"value":373},"Single PromQL query slow or OOMs the querier",{"type":43,"tag":346,"props":375,"children":376},{},[377],{"type":48,"value":378},"One or more metrics in the query have high cardinality",{"type":43,"tag":346,"props":380,"children":381},{},[382],{"type":43,"tag":360,"props":383,"children":385},{"href":384},"#step-3-per-metric-drill-down",[386],{"type":48,"value":387},"Per-query drill-down",{"type":43,"tag":319,"props":389,"children":390},{},[391,396,401],{"type":43,"tag":346,"props":392,"children":393},{},[394],{"type":48,"value":395},"Remote write lagging, WAL growing",{"type":43,"tag":346,"props":397,"children":398},{},[399],{"type":48,"value":400},"Sample throughput spike — series count OR scrape interval changed",{"type":43,"tag":346,"props":402,"children":403},{},[404,408],{"type":43,"tag":360,"props":405,"children":406},{"href":362},[407],{"type":48,"value":365},{"type":48,"value":409}," + check scrape intervals",{"type":43,"tag":319,"props":411,"children":412},{},[413,432,437],{"type":43,"tag":346,"props":414,"children":415},{},[416,422,424,430],{"type":43,"tag":70,"props":417,"children":419},{"className":418},[],[420],{"type":48,"value":421},"429 Too Many Samples",{"type":48,"value":423}," \u002F ",{"type":43,"tag":70,"props":425,"children":427},{"className":426},[],[428],{"type":48,"value":429},"out of bounds",{"type":48,"value":431}," errors",{"type":43,"tag":346,"props":433,"children":434},{},[435],{"type":48,"value":436},"Hitting Mimir\u002FCortex ingester per-tenant series limit",{"type":43,"tag":346,"props":438,"children":439},{},[440,445],{"type":43,"tag":360,"props":441,"children":442},{"href":384},[443],{"type":48,"value":444},"Per-metric drill-down",{"type":48,"value":446},", find the new offender",{"type":43,"tag":319,"props":448,"children":449},{},[450,455,460],{"type":43,"tag":346,"props":451,"children":452},{},[453],{"type":48,"value":454},"Grafana Cloud Active Series bill spiked",{"type":43,"tag":346,"props":456,"children":457},{},[458],{"type":48,"value":459},"New metric, new label, or rollout creating churn",{"type":43,"tag":346,"props":461,"children":462},{},[463,467],{"type":43,"tag":360,"props":464,"children":465},{"href":384},[466],{"type":48,"value":444},{"type":48,"value":468}," + churn check",{"type":43,"tag":319,"props":470,"children":471},{},[472,477,482],{"type":43,"tag":346,"props":473,"children":474},{},[475],{"type":48,"value":476},"Grafana Cloud DPM bill spiked but Active Series flat",{"type":43,"tag":346,"props":478,"children":479},{},[480],{"type":48,"value":481},"Scrape interval shortened, OR remote_write sending duplicates",{"type":43,"tag":346,"props":483,"children":484},{},[485,487],{"type":48,"value":486},"DPM-side issue — route to ",{"type":43,"tag":70,"props":488,"children":490},{"className":489},[],[491],{"type":48,"value":492},"dpm-finder",{"type":43,"tag":319,"props":494,"children":495},{},[496,507,512],{"type":43,"tag":346,"props":497,"children":498},{},[499,505],{"type":43,"tag":70,"props":500,"children":502},{"className":501},[],[503],{"type":48,"value":504},"series_limit_per_user",{"type":48,"value":506}," errors after a deploy",{"type":43,"tag":346,"props":508,"children":509},{},[510],{"type":48,"value":511},"Application change introduced a new bad label",{"type":43,"tag":346,"props":513,"children":514},{},[515],{"type":43,"tag":360,"props":516,"children":518},{"href":517},"#step-4-recent-change-diff",[519],{"type":48,"value":520},"Recent change diff",{"type":43,"tag":319,"props":522,"children":523},{},[524,529,534],{"type":43,"tag":346,"props":525,"children":526},{},[527],{"type":48,"value":528},"Series count grows then resets every restart",{"type":43,"tag":346,"props":530,"children":531},{},[532],{"type":48,"value":533},"Series churn from ephemeral label values",{"type":43,"tag":346,"props":535,"children":536},{},[537],{"type":43,"tag":360,"props":538,"children":540},{"href":539},"#step-5-churn-diagnosis",[541],{"type":48,"value":542},"Churn diagnosis",{"type":43,"tag":79,"props":544,"children":545},{},[],{"type":43,"tag":83,"props":547,"children":549},{"id":548},"step-1-active-series-triage",[550],{"type":48,"value":551},"Step 1: Active Series Triage",{"type":43,"tag":553,"props":554,"children":556},"h3",{"id":555},"get-the-headline-number",[557],{"type":48,"value":558},"Get the headline number",{"type":43,"tag":560,"props":561,"children":566},"pre",{"className":562,"code":563,"language":564,"meta":565,"style":565},"language-promql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Total active series in the local Prometheus\nprometheus_tsdb_head_series\n\n# Or for Mimir \u002F Grafana Cloud Metrics (per tenant)\ncortex_ingester_memory_series{user=\"\u003Ctenant>\"}\n","promql","",[567],{"type":43,"tag":70,"props":568,"children":569},{"__ignoreMap":565},[570,581,590,600,609],{"type":43,"tag":571,"props":572,"children":575},"span",{"class":573,"line":574},"line",1,[576],{"type":43,"tag":571,"props":577,"children":578},{},[579],{"type":48,"value":580},"# Total active series in the local Prometheus\n",{"type":43,"tag":571,"props":582,"children":584},{"class":573,"line":583},2,[585],{"type":43,"tag":571,"props":586,"children":587},{},[588],{"type":48,"value":589},"prometheus_tsdb_head_series\n",{"type":43,"tag":571,"props":591,"children":593},{"class":573,"line":592},3,[594],{"type":43,"tag":571,"props":595,"children":597},{"emptyLinePlaceholder":596},true,[598],{"type":48,"value":599},"\n",{"type":43,"tag":571,"props":601,"children":603},{"class":573,"line":602},4,[604],{"type":43,"tag":571,"props":605,"children":606},{},[607],{"type":48,"value":608},"# Or for Mimir \u002F Grafana Cloud Metrics (per tenant)\n",{"type":43,"tag":571,"props":610,"children":612},{"class":573,"line":611},5,[613],{"type":43,"tag":571,"props":614,"children":615},{},[616],{"type":48,"value":617},"cortex_ingester_memory_series{user=\"\u003Ctenant>\"}\n",{"type":43,"tag":51,"props":619,"children":620},{},[621],{"type":48,"value":622},"Compare to recent history:",{"type":43,"tag":560,"props":624,"children":626},{"className":562,"code":625,"language":564,"meta":565,"style":565},"# Growth over the last 7 days\nderiv(prometheus_tsdb_head_series[7d]) * 86400\n",[627],{"type":43,"tag":70,"props":628,"children":629},{"__ignoreMap":565},[630,638],{"type":43,"tag":571,"props":631,"children":632},{"class":573,"line":574},[633],{"type":43,"tag":571,"props":634,"children":635},{},[636],{"type":48,"value":637},"# Growth over the last 7 days\n",{"type":43,"tag":571,"props":639,"children":640},{"class":573,"line":583},[641],{"type":43,"tag":571,"props":642,"children":643},{},[644],{"type":48,"value":645},"deriv(prometheus_tsdb_head_series[7d]) * 86400\n",{"type":43,"tag":51,"props":647,"children":648},{},[649],{"type":48,"value":650},"A growth rate > a few % per day on a stable application set is a red flag.",{"type":43,"tag":553,"props":652,"children":654},{"id":653},"use-the-tsdb-status-endpoint",[655],{"type":48,"value":656},"Use the TSDB status endpoint",{"type":43,"tag":51,"props":658,"children":659},{},[660],{"type":48,"value":661},"Prometheus exposes a built-in cardinality breakdown:",{"type":43,"tag":560,"props":663,"children":667},{"className":664,"code":665,"language":666,"meta":565,"style":565},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","curl -s http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Fstatus\u002Ftsdb | jq\n","bash",[668],{"type":43,"tag":70,"props":669,"children":670},{"__ignoreMap":565},[671],{"type":43,"tag":571,"props":672,"children":673},{"class":573,"line":574},[674,680,686,691,697],{"type":43,"tag":571,"props":675,"children":677},{"style":676},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[678],{"type":48,"value":679},"curl",{"type":43,"tag":571,"props":681,"children":683},{"style":682},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[684],{"type":48,"value":685}," -s",{"type":43,"tag":571,"props":687,"children":688},{"style":682},[689],{"type":48,"value":690}," http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Fstatus\u002Ftsdb",{"type":43,"tag":571,"props":692,"children":694},{"style":693},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[695],{"type":48,"value":696}," |",{"type":43,"tag":571,"props":698,"children":699},{"style":676},[700],{"type":48,"value":701}," jq\n",{"type":43,"tag":51,"props":703,"children":704},{},[705],{"type":48,"value":706},"Returns:",{"type":43,"tag":133,"props":708,"children":709},{},[710,721,732,743],{"type":43,"tag":137,"props":711,"children":712},{},[713,719],{"type":43,"tag":70,"props":714,"children":716},{"className":715},[],[717],{"type":48,"value":718},"seriesCountByMetricName",{"type":48,"value":720}," — top metrics by series count",{"type":43,"tag":137,"props":722,"children":723},{},[724,730],{"type":43,"tag":70,"props":725,"children":727},{"className":726},[],[728],{"type":48,"value":729},"labelValueCountByLabelName",{"type":48,"value":731}," — top labels by unique value count",{"type":43,"tag":137,"props":733,"children":734},{},[735,741],{"type":43,"tag":70,"props":736,"children":738},{"className":737},[],[739],{"type":48,"value":740},"memoryInBytesByLabelName",{"type":48,"value":742}," — top labels by memory footprint",{"type":43,"tag":137,"props":744,"children":745},{},[746,752],{"type":43,"tag":70,"props":747,"children":749},{"className":748},[],[750],{"type":48,"value":751},"seriesCountByLabelValuePair",{"type":48,"value":753}," — top label-value pairs by series count",{"type":43,"tag":51,"props":755,"children":756},{},[757],{"type":48,"value":758},"This is usually the fastest path to \"which metric \u002F which label is the problem.\"",{"type":43,"tag":51,"props":760,"children":761},{},[762],{"type":48,"value":763},"For Grafana Cloud:",{"type":43,"tag":560,"props":765,"children":767},{"className":664,"code":766,"language":666,"meta":565,"style":565},"# Same endpoint, authenticated against the per-tenant Mimir\ncurl -s -u \"\u003Cuser>:\u003Ctoken>\" \\\n  \"https:\u002F\u002Fprometheus-prod-XX.grafana.net\u002Fapi\u002Fprom\u002Fapi\u002Fv1\u002Fstatus\u002Ftsdb\" | jq\n",[768],{"type":43,"tag":70,"props":769,"children":770},{"__ignoreMap":565},[771,780,817],{"type":43,"tag":571,"props":772,"children":773},{"class":573,"line":574},[774],{"type":43,"tag":571,"props":775,"children":777},{"style":776},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[778],{"type":48,"value":779},"# Same endpoint, authenticated against the per-tenant Mimir\n",{"type":43,"tag":571,"props":781,"children":782},{"class":573,"line":583},[783,787,791,796,801,806,811],{"type":43,"tag":571,"props":784,"children":785},{"style":676},[786],{"type":48,"value":679},{"type":43,"tag":571,"props":788,"children":789},{"style":682},[790],{"type":48,"value":685},{"type":43,"tag":571,"props":792,"children":793},{"style":682},[794],{"type":48,"value":795}," -u",{"type":43,"tag":571,"props":797,"children":798},{"style":693},[799],{"type":48,"value":800}," \"",{"type":43,"tag":571,"props":802,"children":803},{"style":682},[804],{"type":48,"value":805},"\u003Cuser>:\u003Ctoken>",{"type":43,"tag":571,"props":807,"children":808},{"style":693},[809],{"type":48,"value":810},"\"",{"type":43,"tag":571,"props":812,"children":814},{"style":813},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[815],{"type":48,"value":816}," \\\n",{"type":43,"tag":571,"props":818,"children":819},{"class":573,"line":592},[820,825,830,834,838],{"type":43,"tag":571,"props":821,"children":822},{"style":693},[823],{"type":48,"value":824},"  \"",{"type":43,"tag":571,"props":826,"children":827},{"style":682},[828],{"type":48,"value":829},"https:\u002F\u002Fprometheus-prod-XX.grafana.net\u002Fapi\u002Fprom\u002Fapi\u002Fv1\u002Fstatus\u002Ftsdb",{"type":43,"tag":571,"props":831,"children":832},{"style":693},[833],{"type":48,"value":810},{"type":43,"tag":571,"props":835,"children":836},{"style":693},[837],{"type":48,"value":696},{"type":43,"tag":571,"props":839,"children":840},{"style":676},[841],{"type":48,"value":701},{"type":43,"tag":79,"props":843,"children":844},{},[],{"type":43,"tag":83,"props":846,"children":848},{"id":847},"step-2-read-the-output",[849],{"type":48,"value":850},"Step 2: Read the Output",{"type":43,"tag":553,"props":852,"children":854},{"id":853},"top-metrics-by-series-count",[855],{"type":48,"value":856},"Top metrics by series count",{"type":43,"tag":560,"props":858,"children":862},{"className":859,"code":860,"language":861,"meta":565,"style":565},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\"seriesCountByMetricName\": [\n  { \"name\": \"http_request_duration_seconds_bucket\", \"value\": 184320 },\n  { \"name\": \"go_gc_duration_seconds\",               \"value\": 80 },\n  ...\n]\n","json",[863],{"type":43,"tag":70,"props":864,"children":865},{"__ignoreMap":565},[866,891,963,1029,1037],{"type":43,"tag":571,"props":867,"children":868},{"class":573,"line":574},[869,873,877,881,886],{"type":43,"tag":571,"props":870,"children":871},{"style":693},[872],{"type":48,"value":810},{"type":43,"tag":571,"props":874,"children":875},{"style":682},[876],{"type":48,"value":718},{"type":43,"tag":571,"props":878,"children":879},{"style":693},[880],{"type":48,"value":810},{"type":43,"tag":571,"props":882,"children":883},{"style":813},[884],{"type":48,"value":885},": ",{"type":43,"tag":571,"props":887,"children":888},{"style":693},[889],{"type":48,"value":890},"[\n",{"type":43,"tag":571,"props":892,"children":893},{"class":573,"line":583},[894,899,903,909,913,917,921,926,930,935,939,944,948,952,958],{"type":43,"tag":571,"props":895,"children":896},{"style":693},[897],{"type":48,"value":898},"  {",{"type":43,"tag":571,"props":900,"children":901},{"style":693},[902],{"type":48,"value":800},{"type":43,"tag":571,"props":904,"children":906},{"style":905},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[907],{"type":48,"value":908},"name",{"type":43,"tag":571,"props":910,"children":911},{"style":693},[912],{"type":48,"value":810},{"type":43,"tag":571,"props":914,"children":915},{"style":693},[916],{"type":48,"value":131},{"type":43,"tag":571,"props":918,"children":919},{"style":693},[920],{"type":48,"value":800},{"type":43,"tag":571,"props":922,"children":923},{"style":682},[924],{"type":48,"value":925},"http_request_duration_seconds_bucket",{"type":43,"tag":571,"props":927,"children":928},{"style":693},[929],{"type":48,"value":810},{"type":43,"tag":571,"props":931,"children":932},{"style":693},[933],{"type":48,"value":934},",",{"type":43,"tag":571,"props":936,"children":937},{"style":693},[938],{"type":48,"value":800},{"type":43,"tag":571,"props":940,"children":941},{"style":905},[942],{"type":48,"value":943},"value",{"type":43,"tag":571,"props":945,"children":946},{"style":693},[947],{"type":48,"value":810},{"type":43,"tag":571,"props":949,"children":950},{"style":693},[951],{"type":48,"value":131},{"type":43,"tag":571,"props":953,"children":955},{"style":954},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[956],{"type":48,"value":957}," 184320",{"type":43,"tag":571,"props":959,"children":960},{"style":693},[961],{"type":48,"value":962}," },\n",{"type":43,"tag":571,"props":964,"children":965},{"class":573,"line":592},[966,970,974,978,982,986,990,995,999,1003,1008,1012,1016,1020,1025],{"type":43,"tag":571,"props":967,"children":968},{"style":693},[969],{"type":48,"value":898},{"type":43,"tag":571,"props":971,"children":972},{"style":693},[973],{"type":48,"value":800},{"type":43,"tag":571,"props":975,"children":976},{"style":905},[977],{"type":48,"value":908},{"type":43,"tag":571,"props":979,"children":980},{"style":693},[981],{"type":48,"value":810},{"type":43,"tag":571,"props":983,"children":984},{"style":693},[985],{"type":48,"value":131},{"type":43,"tag":571,"props":987,"children":988},{"style":693},[989],{"type":48,"value":800},{"type":43,"tag":571,"props":991,"children":992},{"style":682},[993],{"type":48,"value":994},"go_gc_duration_seconds",{"type":43,"tag":571,"props":996,"children":997},{"style":693},[998],{"type":48,"value":810},{"type":43,"tag":571,"props":1000,"children":1001},{"style":693},[1002],{"type":48,"value":934},{"type":43,"tag":571,"props":1004,"children":1005},{"style":693},[1006],{"type":48,"value":1007},"               \"",{"type":43,"tag":571,"props":1009,"children":1010},{"style":905},[1011],{"type":48,"value":943},{"type":43,"tag":571,"props":1013,"children":1014},{"style":693},[1015],{"type":48,"value":810},{"type":43,"tag":571,"props":1017,"children":1018},{"style":693},[1019],{"type":48,"value":131},{"type":43,"tag":571,"props":1021,"children":1022},{"style":954},[1023],{"type":48,"value":1024}," 80",{"type":43,"tag":571,"props":1026,"children":1027},{"style":693},[1028],{"type":48,"value":962},{"type":43,"tag":571,"props":1030,"children":1031},{"class":573,"line":602},[1032],{"type":43,"tag":571,"props":1033,"children":1034},{"style":813},[1035],{"type":48,"value":1036},"  ...\n",{"type":43,"tag":571,"props":1038,"children":1039},{"class":573,"line":611},[1040],{"type":43,"tag":571,"props":1041,"children":1042},{"style":693},[1043],{"type":48,"value":1044},"]\n",{"type":43,"tag":51,"props":1046,"children":1047},{},[1048,1053],{"type":43,"tag":62,"props":1049,"children":1050},{},[1051],{"type":48,"value":1052},"Heuristics",{"type":48,"value":131},{"type":43,"tag":133,"props":1055,"children":1056},{},[1057,1077,1082],{"type":43,"tag":137,"props":1058,"children":1059},{},[1060,1062,1068,1070,1075],{"type":48,"value":1061},"A histogram (",{"type":43,"tag":70,"props":1063,"children":1065},{"className":1064},[],[1066],{"type":48,"value":1067},"_bucket",{"type":48,"value":1069},") at the top is almost always the answer — those have a 14× multiplier (bucket count + 3). The fix is usually ",{"type":43,"tag":62,"props":1071,"children":1072},{},[1073],{"type":48,"value":1074},"reducing the labels on the underlying histogram at the source",{"type":48,"value":1076}," (in instrumentation code), not stripping them at scrape and not touching the buckets themselves.",{"type":43,"tag":137,"props":1078,"children":1079},{},[1080],{"type":48,"value":1081},"A metric in the top 5 you don't recognize → grep the codebase for it; it's likely a new feature flag or a debug metric that shipped to prod",{"type":43,"tag":137,"props":1083,"children":1084},{},[1085,1087,1093,1094,1100,1101,1107],{"type":48,"value":1086},"The same metric showing up under multiple variants (",{"type":43,"tag":70,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":48,"value":1092},"_total",{"type":48,"value":249},{"type":43,"tag":70,"props":1095,"children":1097},{"className":1096},[],[1098],{"type":48,"value":1099},"_count",{"type":48,"value":249},{"type":43,"tag":70,"props":1102,"children":1104},{"className":1103},[],[1105],{"type":48,"value":1106},"_sum",{"type":48,"value":1108},") — that's a histogram or summary, count all variants together for the true impact",{"type":43,"tag":553,"props":1110,"children":1112},{"id":1111},"top-labels-by-unique-value-count",[1113],{"type":48,"value":1114},"Top labels by unique value count",{"type":43,"tag":560,"props":1116,"children":1118},{"className":859,"code":1117,"language":861,"meta":565,"style":565},"\"labelValueCountByLabelName\": [\n  { \"name\": \"url\",       \"value\": 84210 },\n  { \"name\": \"trace_id\",  \"value\": 41000 },\n  { \"name\": \"pod\",       \"value\": 1820 }\n]\n",[1119],{"type":43,"tag":70,"props":1120,"children":1121},{"__ignoreMap":565},[1122,1145,1211,1276,1341],{"type":43,"tag":571,"props":1123,"children":1124},{"class":573,"line":574},[1125,1129,1133,1137,1141],{"type":43,"tag":571,"props":1126,"children":1127},{"style":693},[1128],{"type":48,"value":810},{"type":43,"tag":571,"props":1130,"children":1131},{"style":682},[1132],{"type":48,"value":729},{"type":43,"tag":571,"props":1134,"children":1135},{"style":693},[1136],{"type":48,"value":810},{"type":43,"tag":571,"props":1138,"children":1139},{"style":813},[1140],{"type":48,"value":885},{"type":43,"tag":571,"props":1142,"children":1143},{"style":693},[1144],{"type":48,"value":890},{"type":43,"tag":571,"props":1146,"children":1147},{"class":573,"line":583},[1148,1152,1156,1160,1164,1168,1172,1177,1181,1185,1190,1194,1198,1202,1207],{"type":43,"tag":571,"props":1149,"children":1150},{"style":693},[1151],{"type":48,"value":898},{"type":43,"tag":571,"props":1153,"children":1154},{"style":693},[1155],{"type":48,"value":800},{"type":43,"tag":571,"props":1157,"children":1158},{"style":905},[1159],{"type":48,"value":908},{"type":43,"tag":571,"props":1161,"children":1162},{"style":693},[1163],{"type":48,"value":810},{"type":43,"tag":571,"props":1165,"children":1166},{"style":693},[1167],{"type":48,"value":131},{"type":43,"tag":571,"props":1169,"children":1170},{"style":693},[1171],{"type":48,"value":800},{"type":43,"tag":571,"props":1173,"children":1174},{"style":682},[1175],{"type":48,"value":1176},"url",{"type":43,"tag":571,"props":1178,"children":1179},{"style":693},[1180],{"type":48,"value":810},{"type":43,"tag":571,"props":1182,"children":1183},{"style":693},[1184],{"type":48,"value":934},{"type":43,"tag":571,"props":1186,"children":1187},{"style":693},[1188],{"type":48,"value":1189},"       \"",{"type":43,"tag":571,"props":1191,"children":1192},{"style":905},[1193],{"type":48,"value":943},{"type":43,"tag":571,"props":1195,"children":1196},{"style":693},[1197],{"type":48,"value":810},{"type":43,"tag":571,"props":1199,"children":1200},{"style":693},[1201],{"type":48,"value":131},{"type":43,"tag":571,"props":1203,"children":1204},{"style":954},[1205],{"type":48,"value":1206}," 84210",{"type":43,"tag":571,"props":1208,"children":1209},{"style":693},[1210],{"type":48,"value":962},{"type":43,"tag":571,"props":1212,"children":1213},{"class":573,"line":592},[1214,1218,1222,1226,1230,1234,1238,1243,1247,1251,1255,1259,1263,1267,1272],{"type":43,"tag":571,"props":1215,"children":1216},{"style":693},[1217],{"type":48,"value":898},{"type":43,"tag":571,"props":1219,"children":1220},{"style":693},[1221],{"type":48,"value":800},{"type":43,"tag":571,"props":1223,"children":1224},{"style":905},[1225],{"type":48,"value":908},{"type":43,"tag":571,"props":1227,"children":1228},{"style":693},[1229],{"type":48,"value":810},{"type":43,"tag":571,"props":1231,"children":1232},{"style":693},[1233],{"type":48,"value":131},{"type":43,"tag":571,"props":1235,"children":1236},{"style":693},[1237],{"type":48,"value":800},{"type":43,"tag":571,"props":1239,"children":1240},{"style":682},[1241],{"type":48,"value":1242},"trace_id",{"type":43,"tag":571,"props":1244,"children":1245},{"style":693},[1246],{"type":48,"value":810},{"type":43,"tag":571,"props":1248,"children":1249},{"style":693},[1250],{"type":48,"value":934},{"type":43,"tag":571,"props":1252,"children":1253},{"style":693},[1254],{"type":48,"value":824},{"type":43,"tag":571,"props":1256,"children":1257},{"style":905},[1258],{"type":48,"value":943},{"type":43,"tag":571,"props":1260,"children":1261},{"style":693},[1262],{"type":48,"value":810},{"type":43,"tag":571,"props":1264,"children":1265},{"style":693},[1266],{"type":48,"value":131},{"type":43,"tag":571,"props":1268,"children":1269},{"style":954},[1270],{"type":48,"value":1271}," 41000",{"type":43,"tag":571,"props":1273,"children":1274},{"style":693},[1275],{"type":48,"value":962},{"type":43,"tag":571,"props":1277,"children":1278},{"class":573,"line":602},[1279,1283,1287,1291,1295,1299,1303,1307,1311,1315,1319,1323,1327,1331,1336],{"type":43,"tag":571,"props":1280,"children":1281},{"style":693},[1282],{"type":48,"value":898},{"type":43,"tag":571,"props":1284,"children":1285},{"style":693},[1286],{"type":48,"value":800},{"type":43,"tag":571,"props":1288,"children":1289},{"style":905},[1290],{"type":48,"value":908},{"type":43,"tag":571,"props":1292,"children":1293},{"style":693},[1294],{"type":48,"value":810},{"type":43,"tag":571,"props":1296,"children":1297},{"style":693},[1298],{"type":48,"value":131},{"type":43,"tag":571,"props":1300,"children":1301},{"style":693},[1302],{"type":48,"value":800},{"type":43,"tag":571,"props":1304,"children":1305},{"style":682},[1306],{"type":48,"value":114},{"type":43,"tag":571,"props":1308,"children":1309},{"style":693},[1310],{"type":48,"value":810},{"type":43,"tag":571,"props":1312,"children":1313},{"style":693},[1314],{"type":48,"value":934},{"type":43,"tag":571,"props":1316,"children":1317},{"style":693},[1318],{"type":48,"value":1189},{"type":43,"tag":571,"props":1320,"children":1321},{"style":905},[1322],{"type":48,"value":943},{"type":43,"tag":571,"props":1324,"children":1325},{"style":693},[1326],{"type":48,"value":810},{"type":43,"tag":571,"props":1328,"children":1329},{"style":693},[1330],{"type":48,"value":131},{"type":43,"tag":571,"props":1332,"children":1333},{"style":954},[1334],{"type":48,"value":1335}," 1820",{"type":43,"tag":571,"props":1337,"children":1338},{"style":693},[1339],{"type":48,"value":1340}," }\n",{"type":43,"tag":571,"props":1342,"children":1343},{"class":573,"line":611},[1344],{"type":43,"tag":571,"props":1345,"children":1346},{"style":693},[1347],{"type":48,"value":1044},{"type":43,"tag":51,"props":1349,"children":1350},{},[1351,1356],{"type":43,"tag":62,"props":1352,"children":1353},{},[1354],{"type":48,"value":1355},"Red flags",{"type":48,"value":131},{"type":43,"tag":133,"props":1358,"children":1359},{},[1360,1365,1422],{"type":43,"tag":137,"props":1361,"children":1362},{},[1363],{"type":48,"value":1364},"Any label with >10K unique values is almost certainly a bug. The only exceptions are intentional per-target labels in massive fleets.",{"type":43,"tag":137,"props":1366,"children":1367},{},[1368,1373,1374,1380,1381,1387,1388,1394,1395,1401,1402,1407,1408,1413,1415,1420],{"type":43,"tag":70,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":48,"value":1242},{"type":48,"value":249},{"type":43,"tag":70,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":48,"value":1379},"request_id",{"type":48,"value":249},{"type":43,"tag":70,"props":1382,"children":1384},{"className":1383},[],[1385],{"type":48,"value":1386},"session_id",{"type":48,"value":249},{"type":43,"tag":70,"props":1389,"children":1391},{"className":1390},[],[1392],{"type":48,"value":1393},"query",{"type":48,"value":249},{"type":43,"tag":70,"props":1396,"children":1398},{"className":1397},[],[1399],{"type":48,"value":1400},"email",{"type":48,"value":249},{"type":43,"tag":70,"props":1403,"children":1405},{"className":1404},[],[1406],{"type":48,"value":247},{"type":48,"value":249},{"type":43,"tag":70,"props":1409,"children":1411},{"className":1410},[],[1412],{"type":48,"value":1176},{"type":48,"value":1414}," — these should ",{"type":43,"tag":159,"props":1416,"children":1417},{},[1418],{"type":48,"value":1419},"never",{"type":48,"value":1421}," be labels. They belong in exemplars, logs, or traces.",{"type":43,"tag":137,"props":1423,"children":1424},{},[1425,1430,1432,1436],{"type":43,"tag":70,"props":1426,"children":1428},{"className":1427},[],[1429],{"type":48,"value":114},{"type":48,"value":1431}," with thousands of values — see ",{"type":43,"tag":360,"props":1433,"children":1434},{"href":539},[1435],{"type":48,"value":542},{"type":48,"value":1437},"; recent churn often inflates this number",{"type":43,"tag":79,"props":1439,"children":1440},{},[],{"type":43,"tag":83,"props":1442,"children":1444},{"id":1443},"step-3-per-metric-drill-down",[1445],{"type":48,"value":1446},"Step 3: Per-Metric Drill-Down",{"type":43,"tag":51,"props":1448,"children":1449},{},[1450],{"type":48,"value":1451},"Once you've identified a suspect metric, find which label is responsible.",{"type":43,"tag":553,"props":1453,"children":1455},{"id":1454},"count-distinct-label-values-per-label-for-one-metric",[1456],{"type":48,"value":1457},"Count distinct label values per label, for one metric",{"type":43,"tag":560,"props":1459,"children":1461},{"className":562,"code":1460,"language":564,"meta":565,"style":565},"# How many unique values does each label have on this metric?\ncount by (__name__) (\n  count by (__name__, label_name_here) (\n    http_request_duration_seconds_bucket\n  )\n)\n",[1462],{"type":43,"tag":70,"props":1463,"children":1464},{"__ignoreMap":565},[1465,1473,1481,1489,1497,1505],{"type":43,"tag":571,"props":1466,"children":1467},{"class":573,"line":574},[1468],{"type":43,"tag":571,"props":1469,"children":1470},{},[1471],{"type":48,"value":1472},"# How many unique values does each label have on this metric?\n",{"type":43,"tag":571,"props":1474,"children":1475},{"class":573,"line":583},[1476],{"type":43,"tag":571,"props":1477,"children":1478},{},[1479],{"type":48,"value":1480},"count by (__name__) (\n",{"type":43,"tag":571,"props":1482,"children":1483},{"class":573,"line":592},[1484],{"type":43,"tag":571,"props":1485,"children":1486},{},[1487],{"type":48,"value":1488},"  count by (__name__, label_name_here) (\n",{"type":43,"tag":571,"props":1490,"children":1491},{"class":573,"line":602},[1492],{"type":43,"tag":571,"props":1493,"children":1494},{},[1495],{"type":48,"value":1496},"    http_request_duration_seconds_bucket\n",{"type":43,"tag":571,"props":1498,"children":1499},{"class":573,"line":611},[1500],{"type":43,"tag":571,"props":1501,"children":1502},{},[1503],{"type":48,"value":1504},"  )\n",{"type":43,"tag":571,"props":1506,"children":1508},{"class":573,"line":1507},6,[1509],{"type":43,"tag":571,"props":1510,"children":1511},{},[1512],{"type":48,"value":1513},")\n",{"type":43,"tag":51,"props":1515,"children":1516},{},[1517],{"type":48,"value":1518},"Repeat per label, or use the helper:",{"type":43,"tag":560,"props":1520,"children":1522},{"className":664,"code":1521,"language":666,"meta":565,"style":565},"# Via the Prometheus HTTP API\ncurl -s \"http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Flabels?match[]=http_request_duration_seconds_bucket\" | jq -r '.data[]' | \\\n  while read label; do\n    count=$(curl -s \"http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Flabel\u002F${label}\u002Fvalues?match[]=http_request_duration_seconds_bucket\" | jq '.data | length')\n    echo \"${count}  ${label}\"\n  done | sort -rn | head -20\n",[1523],{"type":43,"tag":70,"props":1524,"children":1525},{"__ignoreMap":565},[1526,1534,1595,1624,1703,1740],{"type":43,"tag":571,"props":1527,"children":1528},{"class":573,"line":574},[1529],{"type":43,"tag":571,"props":1530,"children":1531},{"style":776},[1532],{"type":48,"value":1533},"# Via the Prometheus HTTP API\n",{"type":43,"tag":571,"props":1535,"children":1536},{"class":573,"line":583},[1537,1541,1545,1549,1554,1558,1562,1567,1572,1577,1582,1587,1591],{"type":43,"tag":571,"props":1538,"children":1539},{"style":676},[1540],{"type":48,"value":679},{"type":43,"tag":571,"props":1542,"children":1543},{"style":682},[1544],{"type":48,"value":685},{"type":43,"tag":571,"props":1546,"children":1547},{"style":693},[1548],{"type":48,"value":800},{"type":43,"tag":571,"props":1550,"children":1551},{"style":682},[1552],{"type":48,"value":1553},"http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Flabels?match[]=http_request_duration_seconds_bucket",{"type":43,"tag":571,"props":1555,"children":1556},{"style":693},[1557],{"type":48,"value":810},{"type":43,"tag":571,"props":1559,"children":1560},{"style":693},[1561],{"type":48,"value":696},{"type":43,"tag":571,"props":1563,"children":1564},{"style":676},[1565],{"type":48,"value":1566}," jq",{"type":43,"tag":571,"props":1568,"children":1569},{"style":682},[1570],{"type":48,"value":1571}," -r",{"type":43,"tag":571,"props":1573,"children":1574},{"style":693},[1575],{"type":48,"value":1576}," '",{"type":43,"tag":571,"props":1578,"children":1579},{"style":682},[1580],{"type":48,"value":1581},".data[]",{"type":43,"tag":571,"props":1583,"children":1584},{"style":693},[1585],{"type":48,"value":1586},"'",{"type":43,"tag":571,"props":1588,"children":1589},{"style":693},[1590],{"type":48,"value":696},{"type":43,"tag":571,"props":1592,"children":1593},{"style":813},[1594],{"type":48,"value":816},{"type":43,"tag":571,"props":1596,"children":1597},{"class":573,"line":592},[1598,1604,1609,1614,1619],{"type":43,"tag":571,"props":1599,"children":1601},{"style":1600},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[1602],{"type":48,"value":1603},"  while",{"type":43,"tag":571,"props":1605,"children":1606},{"style":682},[1607],{"type":48,"value":1608}," read",{"type":43,"tag":571,"props":1610,"children":1611},{"style":682},[1612],{"type":48,"value":1613}," label",{"type":43,"tag":571,"props":1615,"children":1616},{"style":693},[1617],{"type":48,"value":1618},";",{"type":43,"tag":571,"props":1620,"children":1621},{"style":1600},[1622],{"type":48,"value":1623}," do\n",{"type":43,"tag":571,"props":1625,"children":1626},{"class":573,"line":602},[1627,1632,1637,1641,1645,1649,1654,1659,1664,1669,1674,1678,1682,1686,1690,1695,1699],{"type":43,"tag":571,"props":1628,"children":1629},{"style":813},[1630],{"type":48,"value":1631},"    count",{"type":43,"tag":571,"props":1633,"children":1634},{"style":693},[1635],{"type":48,"value":1636},"=$(",{"type":43,"tag":571,"props":1638,"children":1639},{"style":676},[1640],{"type":48,"value":679},{"type":43,"tag":571,"props":1642,"children":1643},{"style":682},[1644],{"type":48,"value":685},{"type":43,"tag":571,"props":1646,"children":1647},{"style":693},[1648],{"type":48,"value":800},{"type":43,"tag":571,"props":1650,"children":1651},{"style":682},[1652],{"type":48,"value":1653},"http:\u002F\u002Fprometheus:9090\u002Fapi\u002Fv1\u002Flabel\u002F",{"type":43,"tag":571,"props":1655,"children":1656},{"style":693},[1657],{"type":48,"value":1658},"${",{"type":43,"tag":571,"props":1660,"children":1661},{"style":813},[1662],{"type":48,"value":1663},"label",{"type":43,"tag":571,"props":1665,"children":1666},{"style":693},[1667],{"type":48,"value":1668},"}",{"type":43,"tag":571,"props":1670,"children":1671},{"style":682},[1672],{"type":48,"value":1673},"\u002Fvalues?match[]=http_request_duration_seconds_bucket",{"type":43,"tag":571,"props":1675,"children":1676},{"style":693},[1677],{"type":48,"value":810},{"type":43,"tag":571,"props":1679,"children":1680},{"style":693},[1681],{"type":48,"value":696},{"type":43,"tag":571,"props":1683,"children":1684},{"style":676},[1685],{"type":48,"value":1566},{"type":43,"tag":571,"props":1687,"children":1688},{"style":693},[1689],{"type":48,"value":1576},{"type":43,"tag":571,"props":1691,"children":1692},{"style":682},[1693],{"type":48,"value":1694},".data | length",{"type":43,"tag":571,"props":1696,"children":1697},{"style":693},[1698],{"type":48,"value":1586},{"type":43,"tag":571,"props":1700,"children":1701},{"style":693},[1702],{"type":48,"value":1513},{"type":43,"tag":571,"props":1704,"children":1705},{"class":573,"line":611},[1706,1712,1717,1722,1726,1731,1735],{"type":43,"tag":571,"props":1707,"children":1709},{"style":1708},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1710],{"type":48,"value":1711},"    echo",{"type":43,"tag":571,"props":1713,"children":1714},{"style":693},[1715],{"type":48,"value":1716}," \"${",{"type":43,"tag":571,"props":1718,"children":1719},{"style":813},[1720],{"type":48,"value":1721},"count",{"type":43,"tag":571,"props":1723,"children":1724},{"style":693},[1725],{"type":48,"value":1668},{"type":43,"tag":571,"props":1727,"children":1728},{"style":693},[1729],{"type":48,"value":1730},"  ${",{"type":43,"tag":571,"props":1732,"children":1733},{"style":813},[1734],{"type":48,"value":1663},{"type":43,"tag":571,"props":1736,"children":1737},{"style":693},[1738],{"type":48,"value":1739},"}\"\n",{"type":43,"tag":571,"props":1741,"children":1742},{"class":573,"line":1507},[1743,1748,1752,1757,1762,1766,1771],{"type":43,"tag":571,"props":1744,"children":1745},{"style":1600},[1746],{"type":48,"value":1747},"  done",{"type":43,"tag":571,"props":1749,"children":1750},{"style":693},[1751],{"type":48,"value":696},{"type":43,"tag":571,"props":1753,"children":1754},{"style":676},[1755],{"type":48,"value":1756}," sort",{"type":43,"tag":571,"props":1758,"children":1759},{"style":682},[1760],{"type":48,"value":1761}," -rn",{"type":43,"tag":571,"props":1763,"children":1764},{"style":693},[1765],{"type":48,"value":696},{"type":43,"tag":571,"props":1767,"children":1768},{"style":676},[1769],{"type":48,"value":1770}," head",{"type":43,"tag":571,"props":1772,"children":1773},{"style":682},[1774],{"type":48,"value":1775}," -20\n",{"type":43,"tag":553,"props":1777,"children":1779},{"id":1778},"find-the-top-label-values-for-one-label",[1780],{"type":48,"value":1781},"Find the top label values for one label",{"type":43,"tag":560,"props":1783,"children":1785},{"className":562,"code":1784,"language":564,"meta":565,"style":565},"# Top 20 path values for http_requests_total\ntopk(20,\n  count by (path) (http_requests_total)\n)\n",[1786],{"type":43,"tag":70,"props":1787,"children":1788},{"__ignoreMap":565},[1789,1797,1805,1813],{"type":43,"tag":571,"props":1790,"children":1791},{"class":573,"line":574},[1792],{"type":43,"tag":571,"props":1793,"children":1794},{},[1795],{"type":48,"value":1796},"# Top 20 path values for http_requests_total\n",{"type":43,"tag":571,"props":1798,"children":1799},{"class":573,"line":583},[1800],{"type":43,"tag":571,"props":1801,"children":1802},{},[1803],{"type":48,"value":1804},"topk(20,\n",{"type":43,"tag":571,"props":1806,"children":1807},{"class":573,"line":592},[1808],{"type":43,"tag":571,"props":1809,"children":1810},{},[1811],{"type":48,"value":1812},"  count by (path) (http_requests_total)\n",{"type":43,"tag":571,"props":1814,"children":1815},{"class":573,"line":602},[1816],{"type":43,"tag":571,"props":1817,"children":1818},{},[1819],{"type":48,"value":1513},{"type":43,"tag":51,"props":1821,"children":1822},{},[1823],{"type":48,"value":1824},"If you see UUIDs, hashes, timestamps, or numeric IDs in the top values → that label has unbounded values from the source.",{"type":43,"tag":553,"props":1826,"children":1828},{"id":1827},"per-metric-series-count-grouped",[1829],{"type":48,"value":1830},"Per-metric series count, grouped",{"type":43,"tag":560,"props":1832,"children":1834},{"className":562,"code":1833,"language":564,"meta":565,"style":565},"# Series-per-instance breakdown — if uneven, one instance is misbehaving\nsum by (job, instance) ({__name__=~\"my_metric.*\"})\n",[1835],{"type":43,"tag":70,"props":1836,"children":1837},{"__ignoreMap":565},[1838,1846],{"type":43,"tag":571,"props":1839,"children":1840},{"class":573,"line":574},[1841],{"type":43,"tag":571,"props":1842,"children":1843},{},[1844],{"type":48,"value":1845},"# Series-per-instance breakdown — if uneven, one instance is misbehaving\n",{"type":43,"tag":571,"props":1847,"children":1848},{"class":573,"line":583},[1849],{"type":43,"tag":571,"props":1850,"children":1851},{},[1852],{"type":48,"value":1853},"sum by (job, instance) ({__name__=~\"my_metric.*\"})\n",{"type":43,"tag":79,"props":1855,"children":1856},{},[],{"type":43,"tag":83,"props":1858,"children":1860},{"id":1859},"step-4-recent-change-diff",[1861],{"type":48,"value":1862},"Step 4: Recent Change Diff",{"type":43,"tag":51,"props":1864,"children":1865},{},[1866],{"type":48,"value":1867},"If the cardinality fire started recently, the cause is almost always a recent change. Diff what's there now against what was there before.",{"type":43,"tag":553,"props":1869,"children":1871},{"id":1870},"list-of-metrics-current-vs-yesterday",[1872],{"type":48,"value":1873},"List of metrics, current vs. yesterday",{"type":43,"tag":51,"props":1875,"children":1876},{},[1877],{"type":48,"value":1878},"Via Grafana Cloud cardinality dashboard, or:",{"type":43,"tag":560,"props":1880,"children":1882},{"className":562,"code":1881,"language":564,"meta":565,"style":565},"# Current metrics\ngroup by (__name__) ({__name__!=\"\"})\n\n# Compare to last week (offset)\ngroup by (__name__) ({__name__!=\"\"} offset 7d)\n",[1883],{"type":43,"tag":70,"props":1884,"children":1885},{"__ignoreMap":565},[1886,1894,1902,1909,1917],{"type":43,"tag":571,"props":1887,"children":1888},{"class":573,"line":574},[1889],{"type":43,"tag":571,"props":1890,"children":1891},{},[1892],{"type":48,"value":1893},"# Current metrics\n",{"type":43,"tag":571,"props":1895,"children":1896},{"class":573,"line":583},[1897],{"type":43,"tag":571,"props":1898,"children":1899},{},[1900],{"type":48,"value":1901},"group by (__name__) ({__name__!=\"\"})\n",{"type":43,"tag":571,"props":1903,"children":1904},{"class":573,"line":592},[1905],{"type":43,"tag":571,"props":1906,"children":1907},{"emptyLinePlaceholder":596},[1908],{"type":48,"value":599},{"type":43,"tag":571,"props":1910,"children":1911},{"class":573,"line":602},[1912],{"type":43,"tag":571,"props":1913,"children":1914},{},[1915],{"type":48,"value":1916},"# Compare to last week (offset)\n",{"type":43,"tag":571,"props":1918,"children":1919},{"class":573,"line":611},[1920],{"type":43,"tag":571,"props":1921,"children":1922},{},[1923],{"type":48,"value":1924},"group by (__name__) ({__name__!=\"\"} offset 7d)\n",{"type":43,"tag":51,"props":1926,"children":1927},{},[1928,1930,1935],{"type":48,"value":1929},"Diff externally. A new metric near the top of ",{"type":43,"tag":70,"props":1931,"children":1933},{"className":1932},[],[1934],{"type":48,"value":718},{"type":48,"value":1936}," that wasn't there a week ago → that's your offender.",{"type":43,"tag":553,"props":1938,"children":1940},{"id":1939},"correlate-with-deploys",[1941],{"type":48,"value":1942},"Correlate with deploys",{"type":43,"tag":560,"props":1944,"children":1946},{"className":562,"code":1945,"language":564,"meta":565,"style":565},"# Active series correlated with build_info\nprometheus_tsdb_head_series\n# Overlay with:\nchanges(app_build_info[1d])\n",[1947],{"type":43,"tag":70,"props":1948,"children":1949},{"__ignoreMap":565},[1950,1958,1965,1973],{"type":43,"tag":571,"props":1951,"children":1952},{"class":573,"line":574},[1953],{"type":43,"tag":571,"props":1954,"children":1955},{},[1956],{"type":48,"value":1957},"# Active series correlated with build_info\n",{"type":43,"tag":571,"props":1959,"children":1960},{"class":573,"line":583},[1961],{"type":43,"tag":571,"props":1962,"children":1963},{},[1964],{"type":48,"value":589},{"type":43,"tag":571,"props":1966,"children":1967},{"class":573,"line":592},[1968],{"type":43,"tag":571,"props":1969,"children":1970},{},[1971],{"type":48,"value":1972},"# Overlay with:\n",{"type":43,"tag":571,"props":1974,"children":1975},{"class":573,"line":602},[1976],{"type":43,"tag":571,"props":1977,"children":1978},{},[1979],{"type":48,"value":1980},"changes(app_build_info[1d])\n",{"type":43,"tag":51,"props":1982,"children":1983},{},[1984],{"type":48,"value":1985},"A vertical step in series count aligned with a deploy is conclusive.",{"type":43,"tag":79,"props":1987,"children":1988},{},[],{"type":43,"tag":83,"props":1990,"children":1992},{"id":1991},"step-5-churn-diagnosis",[1993],{"type":48,"value":1994},"Step 5: Churn Diagnosis",{"type":43,"tag":51,"props":1996,"children":1997},{},[1998],{"type":48,"value":1999},"High churn means series are being created and abandoned faster than they age out. Symptoms: series count keeps climbing, then drops sharply on Prometheus restart.",{"type":43,"tag":553,"props":2001,"children":2003},{"id":2002},"churn-signal",[2004],{"type":48,"value":2005},"Churn signal",{"type":43,"tag":560,"props":2007,"children":2009},{"className":562,"code":2008,"language":564,"meta":565,"style":565},"# Series created vs. removed per second\nrate(prometheus_tsdb_head_series_created_total[5m])\nrate(prometheus_tsdb_head_series_removed_total[5m])\n\n# Ratio of churned to live\nprometheus_tsdb_head_series_created_total \u002F prometheus_tsdb_head_series\n",[2010],{"type":43,"tag":70,"props":2011,"children":2012},{"__ignoreMap":565},[2013,2021,2029,2037,2044,2052],{"type":43,"tag":571,"props":2014,"children":2015},{"class":573,"line":574},[2016],{"type":43,"tag":571,"props":2017,"children":2018},{},[2019],{"type":48,"value":2020},"# Series created vs. removed per second\n",{"type":43,"tag":571,"props":2022,"children":2023},{"class":573,"line":583},[2024],{"type":43,"tag":571,"props":2025,"children":2026},{},[2027],{"type":48,"value":2028},"rate(prometheus_tsdb_head_series_created_total[5m])\n",{"type":43,"tag":571,"props":2030,"children":2031},{"class":573,"line":592},[2032],{"type":43,"tag":571,"props":2033,"children":2034},{},[2035],{"type":48,"value":2036},"rate(prometheus_tsdb_head_series_removed_total[5m])\n",{"type":43,"tag":571,"props":2038,"children":2039},{"class":573,"line":602},[2040],{"type":43,"tag":571,"props":2041,"children":2042},{"emptyLinePlaceholder":596},[2043],{"type":48,"value":599},{"type":43,"tag":571,"props":2045,"children":2046},{"class":573,"line":611},[2047],{"type":43,"tag":571,"props":2048,"children":2049},{},[2050],{"type":48,"value":2051},"# Ratio of churned to live\n",{"type":43,"tag":571,"props":2053,"children":2054},{"class":573,"line":1507},[2055],{"type":43,"tag":571,"props":2056,"children":2057},{},[2058],{"type":48,"value":2059},"prometheus_tsdb_head_series_created_total \u002F prometheus_tsdb_head_series\n",{"type":43,"tag":51,"props":2061,"children":2062},{},[2063],{"type":48,"value":2064},"A creation rate that materially exceeds the removal rate, sustained, means cardinality is on a one-way trip up. Common causes:",{"type":43,"tag":311,"props":2066,"children":2067},{},[2068,2084],{"type":43,"tag":315,"props":2069,"children":2070},{},[2071],{"type":43,"tag":319,"props":2072,"children":2073},{},[2074,2079],{"type":43,"tag":323,"props":2075,"children":2076},{},[2077],{"type":48,"value":2078},"Cause",{"type":43,"tag":323,"props":2080,"children":2081},{},[2082],{"type":48,"value":2083},"Tell",{"type":43,"tag":339,"props":2085,"children":2086},{},[2087,2106,2139,2157,2170],{"type":43,"tag":319,"props":2088,"children":2089},{},[2090,2101],{"type":43,"tag":346,"props":2091,"children":2092},{},[2093,2095,2100],{"type":48,"value":2094},"Pod rollouts emitting ",{"type":43,"tag":70,"props":2096,"children":2098},{"className":2097},[],[2099],{"type":48,"value":114},{"type":48,"value":1613},{"type":43,"tag":346,"props":2102,"children":2103},{},[2104],{"type":48,"value":2105},"Churn spike aligns with deploy timing; affects pod-discovered scrapes",{"type":43,"tag":319,"props":2107,"children":2108},{},[2109,2134],{"type":43,"tag":346,"props":2110,"children":2111},{},[2112,2118,2119,2125,2126,2132],{"type":43,"tag":70,"props":2113,"children":2115},{"className":2114},[],[2116],{"type":48,"value":2117},"version",{"type":48,"value":423},{"type":43,"tag":70,"props":2120,"children":2122},{"className":2121},[],[2123],{"type":48,"value":2124},"git_sha",{"type":48,"value":423},{"type":43,"tag":70,"props":2127,"children":2129},{"className":2128},[],[2130],{"type":48,"value":2131},"image_tag",{"type":48,"value":2133}," label on every metric",{"type":43,"tag":346,"props":2135,"children":2136},{},[2137],{"type":48,"value":2138},"Churn spike on every deploy across many metrics",{"type":43,"tag":319,"props":2140,"children":2141},{},[2142,2152],{"type":43,"tag":346,"props":2143,"children":2144},{},[2145,2147],{"type":48,"value":2146},"Ephemeral hostnames in ",{"type":43,"tag":70,"props":2148,"children":2150},{"className":2149},[],[2151],{"type":48,"value":122},{"type":43,"tag":346,"props":2153,"children":2154},{},[2155],{"type":48,"value":2156},"Cloud autoscaling event timing",{"type":43,"tag":319,"props":2158,"children":2159},{},[2160,2165],{"type":43,"tag":346,"props":2161,"children":2162},{},[2163],{"type":48,"value":2164},"Bug: dynamic label names",{"type":43,"tag":346,"props":2166,"children":2167},{},[2168],{"type":48,"value":2169},"Churn climbs forever, never plateaus",{"type":43,"tag":319,"props":2171,"children":2172},{},[2173,2178],{"type":43,"tag":346,"props":2174,"children":2175},{},[2176],{"type":48,"value":2177},"Application bug emitting fresh UUIDs as labels",{"type":43,"tag":346,"props":2179,"children":2180},{},[2181],{"type":48,"value":2182},"Linear unbounded growth, no deploy correlation",{"type":43,"tag":553,"props":2184,"children":2186},{"id":2185},"memory-impact-of-churn",[2187],{"type":48,"value":2188},"Memory impact of churn",{"type":43,"tag":560,"props":2190,"children":2192},{"className":562,"code":2191,"language":564,"meta":565,"style":565},"# A churn-driven head block carries old series until tsdb compaction\nprometheus_tsdb_head_chunks\ngo_memstats_heap_inuse_bytes{job=\"prometheus\"}\n",[2193],{"type":43,"tag":70,"props":2194,"children":2195},{"__ignoreMap":565},[2196,2204,2212],{"type":43,"tag":571,"props":2197,"children":2198},{"class":573,"line":574},[2199],{"type":43,"tag":571,"props":2200,"children":2201},{},[2202],{"type":48,"value":2203},"# A churn-driven head block carries old series until tsdb compaction\n",{"type":43,"tag":571,"props":2205,"children":2206},{"class":573,"line":583},[2207],{"type":43,"tag":571,"props":2208,"children":2209},{},[2210],{"type":48,"value":2211},"prometheus_tsdb_head_chunks\n",{"type":43,"tag":571,"props":2213,"children":2214},{"class":573,"line":592},[2215],{"type":43,"tag":571,"props":2216,"children":2217},{},[2218],{"type":48,"value":2219},"go_memstats_heap_inuse_bytes{job=\"prometheus\"}\n",{"type":43,"tag":51,"props":2221,"children":2222},{},[2223],{"type":48,"value":2224},"Restarting Prometheus drops churned series but is not a fix. The fix is at the source.",{"type":43,"tag":79,"props":2226,"children":2227},{},[],{"type":43,"tag":83,"props":2229,"children":2231},{"id":2230},"common-culprit-gallery",[2232],{"type":48,"value":2233},"Common-Culprit Gallery",{"type":43,"tag":553,"props":2235,"children":2237},{"id":2236},"histogram-blowup",[2238],{"type":48,"value":2239},"Histogram blowup",{"type":43,"tag":51,"props":2241,"children":2242},{},[2243,2247,2248,2254,2256,2261],{"type":43,"tag":62,"props":2244,"children":2245},{},[2246],{"type":48,"value":2083},{"type":48,"value":885},{"type":43,"tag":70,"props":2249,"children":2251},{"className":2250},[],[2252],{"type":48,"value":2253},"*_bucket",{"type":48,"value":2255}," metric at the top of ",{"type":43,"tag":70,"props":2257,"children":2259},{"className":2258},[],[2260],{"type":48,"value":718},{"type":48,"value":2262},". Multiplier ≈ 14×.",{"type":43,"tag":51,"props":2264,"children":2265},{},[2266,2271],{"type":43,"tag":62,"props":2267,"children":2268},{},[2269],{"type":48,"value":2270},"Fix",{"type":48,"value":131},{"type":43,"tag":196,"props":2273,"children":2274},{},[2275,2316,2321],{"type":43,"tag":137,"props":2276,"children":2277},{},[2278,2280,2285,2287,2292,2293,2299,2301,2307,2309,2314],{"type":48,"value":2279},"First, ",{"type":43,"tag":62,"props":2281,"children":2282},{},[2283],{"type":48,"value":2284},"reduce labels on the histogram at the source",{"type":48,"value":2286}," — every label removed saves 14× series. Trim ",{"type":43,"tag":70,"props":2288,"children":2290},{"className":2289},[],[2291],{"type":48,"value":247},{"type":48,"value":249},{"type":43,"tag":70,"props":2294,"children":2296},{"className":2295},[],[2297],{"type":48,"value":2298},"method",{"type":48,"value":2300},", or ",{"type":43,"tag":70,"props":2302,"children":2304},{"className":2303},[],[2305],{"type":48,"value":2306},"status_code",{"type":48,"value":2308}," in the instrumentation code (don't ",{"type":43,"tag":70,"props":2310,"children":2312},{"className":2311},[],[2313],{"type":48,"value":99},{"type":48,"value":2315}," them at scrape — that merges distinct histograms and corrupts the buckets). For series already in Grafana Cloud you can't change, aggregate them with Adaptive Metrics.",{"type":43,"tag":137,"props":2317,"children":2318},{},[2319],{"type":48,"value":2320},"Then, reduce bucket count if appropriate (custom buckets vs. defaults).",{"type":43,"tag":137,"props":2322,"children":2323},{},[2324,2326,2331],{"type":48,"value":2325},"For high-resolution latency tracking, consider ",{"type":43,"tag":62,"props":2327,"children":2328},{},[2329],{"type":48,"value":2330},"native histograms",{"type":48,"value":2332}," (Prometheus 2.40+) — single sparse series replaces the bucket family.",{"type":43,"tag":553,"props":2334,"children":2336},{"id":2335},"kube-state-metrics-label-explosion",[2337],{"type":48,"value":2338},"kube-state-metrics label explosion",{"type":43,"tag":51,"props":2340,"children":2341},{},[2342,2346,2347,2353,2355,2361,2363,2369,2370,2376],{"type":43,"tag":62,"props":2343,"children":2344},{},[2345],{"type":48,"value":2083},{"type":48,"value":885},{"type":43,"tag":70,"props":2348,"children":2350},{"className":2349},[],[2351],{"type":48,"value":2352},"kube_pod_labels",{"type":48,"value":2354}," or ",{"type":43,"tag":70,"props":2356,"children":2358},{"className":2357},[],[2359],{"type":48,"value":2360},"kube_pod_annotations",{"type":48,"value":2362}," at the top, with ",{"type":43,"tag":70,"props":2364,"children":2366},{"className":2365},[],[2367],{"type":48,"value":2368},"label_*",{"type":48,"value":2354},{"type":43,"tag":70,"props":2371,"children":2373},{"className":2372},[],[2374],{"type":48,"value":2375},"annotation_*",{"type":48,"value":2377}," labels driving cardinality.",{"type":43,"tag":51,"props":2379,"children":2380},{},[2381,2385,2387,2393,2394,2400,2402,2407],{"type":43,"tag":62,"props":2382,"children":2383},{},[2384],{"type":48,"value":2270},{"type":48,"value":2386},": configure kube-state-metrics with ",{"type":43,"tag":70,"props":2388,"children":2390},{"className":2389},[],[2391],{"type":48,"value":2392},"--metric-labels-allowlist",{"type":48,"value":149},{"type":43,"tag":70,"props":2395,"children":2397},{"className":2396},[],[2398],{"type":48,"value":2399},"--metric-annotations-allowlist",{"type":48,"value":2401},". By default it emits ",{"type":43,"tag":159,"props":2403,"children":2404},{},[2405],{"type":48,"value":2406},"all",{"type":48,"value":2408}," labels and annotations as series.",{"type":43,"tag":560,"props":2410,"children":2414},{"className":2411,"code":2412,"language":2413,"meta":565,"style":565},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# kube-state-metrics flags\n--metric-labels-allowlist=pods=[app,team,version]\n--metric-annotations-allowlist=pods=[checksum\u002Fconfig]\n","yaml",[2415],{"type":43,"tag":70,"props":2416,"children":2417},{"__ignoreMap":565},[2418,2426,2434],{"type":43,"tag":571,"props":2419,"children":2420},{"class":573,"line":574},[2421],{"type":43,"tag":571,"props":2422,"children":2423},{"style":776},[2424],{"type":48,"value":2425},"# kube-state-metrics flags\n",{"type":43,"tag":571,"props":2427,"children":2428},{"class":573,"line":583},[2429],{"type":43,"tag":571,"props":2430,"children":2431},{"style":682},[2432],{"type":48,"value":2433},"--metric-labels-allowlist=pods=[app,team,version]\n",{"type":43,"tag":571,"props":2435,"children":2436},{"class":573,"line":592},[2437],{"type":43,"tag":571,"props":2438,"children":2439},{"style":682},[2440],{"type":48,"value":2441},"--metric-annotations-allowlist=pods=[checksum\u002Fconfig]\n",{"type":43,"tag":553,"props":2443,"children":2445},{"id":2444},"path-route-blowup-from-a-new-endpoint",[2446],{"type":48,"value":2447},"Path \u002F route blowup from a new endpoint",{"type":43,"tag":51,"props":2449,"children":2450},{},[2451,2455,2456,2462,2464,2470,2472,2478],{"type":43,"tag":62,"props":2452,"children":2453},{},[2454],{"type":48,"value":2083},{"type":48,"value":885},{"type":43,"tag":70,"props":2457,"children":2459},{"className":2458},[],[2460],{"type":48,"value":2461},"http_requests_total",{"type":48,"value":2463}," (or framework equivalent) grew 10×+ overnight. ",{"type":43,"tag":70,"props":2465,"children":2467},{"className":2466},[],[2468],{"type":48,"value":2469},"topk(20, count by (path) (http_requests_total))",{"type":48,"value":2471}," shows hundreds of ",{"type":43,"tag":70,"props":2473,"children":2475},{"className":2474},[],[2476],{"type":48,"value":2477},"\u002Fusers\u002F123456",{"type":48,"value":2479},"-style values.",{"type":43,"tag":51,"props":2481,"children":2482},{},[2483,2487,2489,2494,2495,2501,2503,2508,2510,2514,2516,2521,2523,2528],{"type":43,"tag":62,"props":2484,"children":2485},{},[2486],{"type":48,"value":2270},{"type":48,"value":2488},": the real fix is to ",{"type":43,"tag":62,"props":2490,"children":2491},{},[2492],{"type":48,"value":2493},"template the path in application code",{"type":48,"value":215},{"type":43,"tag":70,"props":2496,"children":2498},{"className":2497},[],[2499],{"type":48,"value":2500},"\u002Fusers\u002F:id",{"type":48,"value":2502},") — route the user to ",{"type":43,"tag":70,"props":2504,"children":2506},{"className":2505},[],[2507],{"type":48,"value":75},{"type":48,"value":2509},". For series already in Grafana Cloud, ",{"type":43,"tag":62,"props":2511,"children":2512},{},[2513],{"type":48,"value":265},{"type":48,"value":2515}," can aggregate ",{"type":43,"tag":70,"props":2517,"children":2519},{"className":2518},[],[2520],{"type":48,"value":247},{"type":48,"value":2522}," away correctly — route to ",{"type":43,"tag":70,"props":2524,"children":2526},{"className":2525},[],[2527],{"type":48,"value":287},{"type":48,"value":77},{"type":43,"tag":51,"props":2530,"children":2531},{},[2532,2534,2539,2541,2546,2548,2554,2556,2562,2563,2569,2571,2576,2578,2583],{"type":48,"value":2533},"Do ",{"type":43,"tag":62,"props":2535,"children":2536},{},[2537],{"type":48,"value":2538},"not",{"type":48,"value":2540}," \"normalize\" ",{"type":43,"tag":70,"props":2542,"children":2544},{"className":2543},[],[2545],{"type":48,"value":247},{"type":48,"value":2547}," with a relabel ",{"type":43,"tag":70,"props":2549,"children":2551},{"className":2550},[],[2552],{"type":48,"value":2553},"replacement",{"type":48,"value":2555}," rule — collapsing ",{"type":43,"tag":70,"props":2557,"children":2559},{"className":2558},[],[2560],{"type":48,"value":2561},"\u002Fusers\u002F123",{"type":48,"value":249},{"type":43,"tag":70,"props":2564,"children":2566},{"className":2565},[],[2567],{"type":48,"value":2568},"\u002Fusers\u002F456",{"type":48,"value":2570},", … into one ",{"type":43,"tag":70,"props":2572,"children":2574},{"className":2573},[],[2575],{"type":48,"value":2500},{"type":48,"value":2577}," value at scrape merges distinct series and produces duplicate-sample errors and broken ",{"type":43,"tag":70,"props":2579,"children":2581},{"className":2580},[],[2582],{"type":48,"value":147},{"type":48,"value":2584},". The merge has to happen at the source (templating) or post-ingest (Adaptive Metrics), never at scrape.",{"type":43,"tag":51,"props":2586,"children":2587},{},[2588,2590,2595,2597,2601],{"type":48,"value":2589},"If you must stop a production fire ",{"type":43,"tag":159,"props":2591,"children":2592},{},[2593],{"type":48,"value":2594},"right now",{"type":48,"value":2596}," and templating isn't deployable yet, the only safe scrape-time action is to drop the ",{"type":43,"tag":62,"props":2598,"children":2599},{},[2600],{"type":48,"value":211},{"type":48,"value":2602}," offending metric (you lose it completely until the code fix lands — a deliberate trade, not a silent corruption):",{"type":43,"tag":560,"props":2604,"children":2606},{"className":2411,"code":2605,"language":2413,"meta":565,"style":565},"# Emergency: drop the whole metric until the source is templated\nmetric_relabel_configs:\n  - source_labels: [__name__]\n    regex: http_requests_total\n    action: drop\n",[2607],{"type":43,"tag":70,"props":2608,"children":2609},{"__ignoreMap":565},[2610,2618,2632,2662,2679],{"type":43,"tag":571,"props":2611,"children":2612},{"class":573,"line":574},[2613],{"type":43,"tag":571,"props":2614,"children":2615},{"style":776},[2616],{"type":48,"value":2617},"# Emergency: drop the whole metric until the source is templated\n",{"type":43,"tag":571,"props":2619,"children":2620},{"class":573,"line":583},[2621,2627],{"type":43,"tag":571,"props":2622,"children":2624},{"style":2623},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[2625],{"type":48,"value":2626},"metric_relabel_configs",{"type":43,"tag":571,"props":2628,"children":2629},{"style":693},[2630],{"type":48,"value":2631},":\n",{"type":43,"tag":571,"props":2633,"children":2634},{"class":573,"line":592},[2635,2640,2645,2649,2654,2658],{"type":43,"tag":571,"props":2636,"children":2637},{"style":693},[2638],{"type":48,"value":2639},"  -",{"type":43,"tag":571,"props":2641,"children":2642},{"style":2623},[2643],{"type":48,"value":2644}," source_labels",{"type":43,"tag":571,"props":2646,"children":2647},{"style":693},[2648],{"type":48,"value":131},{"type":43,"tag":571,"props":2650,"children":2651},{"style":693},[2652],{"type":48,"value":2653}," [",{"type":43,"tag":571,"props":2655,"children":2656},{"style":682},[2657],{"type":48,"value":229},{"type":43,"tag":571,"props":2659,"children":2660},{"style":693},[2661],{"type":48,"value":1044},{"type":43,"tag":571,"props":2663,"children":2664},{"class":573,"line":602},[2665,2670,2674],{"type":43,"tag":571,"props":2666,"children":2667},{"style":2623},[2668],{"type":48,"value":2669},"    regex",{"type":43,"tag":571,"props":2671,"children":2672},{"style":693},[2673],{"type":48,"value":131},{"type":43,"tag":571,"props":2675,"children":2676},{"style":682},[2677],{"type":48,"value":2678}," http_requests_total\n",{"type":43,"tag":571,"props":2680,"children":2681},{"class":573,"line":611},[2682,2687,2691],{"type":43,"tag":571,"props":2683,"children":2684},{"style":2623},[2685],{"type":48,"value":2686},"    action",{"type":43,"tag":571,"props":2688,"children":2689},{"style":693},[2690],{"type":48,"value":131},{"type":43,"tag":571,"props":2692,"children":2693},{"style":682},[2694],{"type":48,"value":2695}," drop\n",{"type":43,"tag":553,"props":2697,"children":2699},{"id":2698},"application-emitting-a-debug-metric-in-prod",[2700],{"type":48,"value":2701},"Application emitting a debug metric in prod",{"type":43,"tag":51,"props":2703,"children":2704},{},[2705,2709,2711,2717,2718,2724],{"type":43,"tag":62,"props":2706,"children":2707},{},[2708],{"type":48,"value":2083},{"type":48,"value":2710},": A metric you don't recognize in the top 10. Grep the source — often a ",{"type":43,"tag":70,"props":2712,"children":2714},{"className":2713},[],[2715],{"type":48,"value":2716},"_details",{"type":48,"value":2354},{"type":43,"tag":70,"props":2719,"children":2721},{"className":2720},[],[2722],{"type":48,"value":2723},"_per_request",{"type":48,"value":2725}," debug metric the developer forgot to gate.",{"type":43,"tag":51,"props":2727,"children":2728},{},[2729,2733],{"type":43,"tag":62,"props":2730,"children":2731},{},[2732],{"type":48,"value":2270},{"type":48,"value":2734},": drop entirely at scrape:",{"type":43,"tag":560,"props":2736,"children":2738},{"className":2411,"code":2737,"language":2413,"meta":565,"style":565},"metric_relabel_configs:\n  - source_labels: [__name__]\n    regex: my_app_request_details\n    action: drop\n",[2739],{"type":43,"tag":70,"props":2740,"children":2741},{"__ignoreMap":565},[2742,2753,2780,2796],{"type":43,"tag":571,"props":2743,"children":2744},{"class":573,"line":574},[2745,2749],{"type":43,"tag":571,"props":2746,"children":2747},{"style":2623},[2748],{"type":48,"value":2626},{"type":43,"tag":571,"props":2750,"children":2751},{"style":693},[2752],{"type":48,"value":2631},{"type":43,"tag":571,"props":2754,"children":2755},{"class":573,"line":583},[2756,2760,2764,2768,2772,2776],{"type":43,"tag":571,"props":2757,"children":2758},{"style":693},[2759],{"type":48,"value":2639},{"type":43,"tag":571,"props":2761,"children":2762},{"style":2623},[2763],{"type":48,"value":2644},{"type":43,"tag":571,"props":2765,"children":2766},{"style":693},[2767],{"type":48,"value":131},{"type":43,"tag":571,"props":2769,"children":2770},{"style":693},[2771],{"type":48,"value":2653},{"type":43,"tag":571,"props":2773,"children":2774},{"style":682},[2775],{"type":48,"value":229},{"type":43,"tag":571,"props":2777,"children":2778},{"style":693},[2779],{"type":48,"value":1044},{"type":43,"tag":571,"props":2781,"children":2782},{"class":573,"line":592},[2783,2787,2791],{"type":43,"tag":571,"props":2784,"children":2785},{"style":2623},[2786],{"type":48,"value":2669},{"type":43,"tag":571,"props":2788,"children":2789},{"style":693},[2790],{"type":48,"value":131},{"type":43,"tag":571,"props":2792,"children":2793},{"style":682},[2794],{"type":48,"value":2795}," my_app_request_details\n",{"type":43,"tag":571,"props":2797,"children":2798},{"class":573,"line":602},[2799,2803,2807],{"type":43,"tag":571,"props":2800,"children":2801},{"style":2623},[2802],{"type":48,"value":2686},{"type":43,"tag":571,"props":2804,"children":2805},{"style":693},[2806],{"type":48,"value":131},{"type":43,"tag":571,"props":2808,"children":2809},{"style":682},[2810],{"type":48,"value":2695},{"type":43,"tag":51,"props":2812,"children":2813},{},[2814],{"type":48,"value":2815},"Open a ticket against the team to remove it from the code.",{"type":43,"tag":553,"props":2817,"children":2819},{"id":2818},"app-emitted-labels-colliding-with-target-labels",[2820],{"type":48,"value":2821},"App-emitted labels colliding with target labels",{"type":43,"tag":51,"props":2823,"children":2824},{},[2825,2829,2831,2837,2839,2844,2846,2852],{"type":43,"tag":62,"props":2826,"children":2827},{},[2828],{"type":48,"value":2083},{"type":48,"value":2830},": Series count for one job is several × what it should be. Looking at one series, you see both an app-emitted ",{"type":43,"tag":70,"props":2832,"children":2834},{"className":2833},[],[2835],{"type":48,"value":2836},"instance=...",{"type":48,"value":2838}," AND the target ",{"type":43,"tag":70,"props":2840,"children":2842},{"className":2841},[],[2843],{"type":48,"value":2836},{"type":48,"value":2845}," collided into something weird (Prometheus renames the conflicting one to ",{"type":43,"tag":70,"props":2847,"children":2849},{"className":2848},[],[2850],{"type":48,"value":2851},"exported_instance",{"type":48,"value":2853},").",{"type":43,"tag":51,"props":2855,"children":2856},{},[2857,2861,2863,2868,2870,2875,2877,2883,2884,2890,2892,2898,2900,2906],{"type":43,"tag":62,"props":2858,"children":2859},{},[2860],{"type":48,"value":2270},{"type":48,"value":2862},": the right fix is ",{"type":43,"tag":62,"props":2864,"children":2865},{},[2866],{"type":48,"value":2867},"in the application",{"type":48,"value":2869}," — stop emitting ",{"type":43,"tag":70,"props":2871,"children":2873},{"className":2872},[],[2874],{"type":48,"value":122},{"type":48,"value":2876},"\u002F",{"type":43,"tag":70,"props":2878,"children":2880},{"className":2879},[],[2881],{"type":48,"value":2882},"node",{"type":48,"value":2876},{"type":43,"tag":70,"props":2885,"children":2887},{"className":2886},[],[2888],{"type":48,"value":2889},"host",{"type":48,"value":2891}," from code; they belong to the scrape target. Confirm ",{"type":43,"tag":70,"props":2893,"children":2895},{"className":2894},[],[2896],{"type":48,"value":2897},"honor_labels",{"type":48,"value":2899}," is ",{"type":43,"tag":70,"props":2901,"children":2903},{"className":2902},[],[2904],{"type":48,"value":2905},"false",{"type":48,"value":2907}," (the default) so the target labels win.",{"type":43,"tag":51,"props":2909,"children":2910},{},[2911,2913,2918,2920,2925,2927,2932,2934,2944],{"type":48,"value":2912},"If you need a scrape-time stopgap, you may remove a label ",{"type":43,"tag":159,"props":2914,"children":2915},{},[2916],{"type":48,"value":2917},"only",{"type":48,"value":2919}," where it ",{"type":43,"tag":62,"props":2921,"children":2922},{},[2923],{"type":48,"value":2924},"exactly duplicates",{"type":48,"value":2926}," a target label — that's the one safe ",{"type":43,"tag":70,"props":2928,"children":2930},{"className":2929},[],[2931],{"type":48,"value":99},{"type":48,"value":2933},", because the target label still provides uniqueness. Scope it tightly to the duplicated names and ",{"type":43,"tag":62,"props":2935,"children":2936},{},[2937,2939],{"type":48,"value":2938},"never include ",{"type":43,"tag":70,"props":2940,"children":2942},{"className":2941},[],[2943],{"type":48,"value":114},{"type":48,"value":2945}," (or any other label that is the source of uniqueness):",{"type":43,"tag":560,"props":2947,"children":2949},{"className":2411,"code":2948,"language":2413,"meta":565,"style":565},"# Stopgap ONLY for app-emitted duplicates of target labels.\n# Drops the `exported_*` collisions — NOT pod, which makes K8s series unique.\nmetric_relabel_configs:\n  - regex: exported_(instance|node|host)\n    action: labeldrop\n",[2950],{"type":43,"tag":70,"props":2951,"children":2952},{"__ignoreMap":565},[2953,2961,2969,2980,3001],{"type":43,"tag":571,"props":2954,"children":2955},{"class":573,"line":574},[2956],{"type":43,"tag":571,"props":2957,"children":2958},{"style":776},[2959],{"type":48,"value":2960},"# Stopgap ONLY for app-emitted duplicates of target labels.\n",{"type":43,"tag":571,"props":2962,"children":2963},{"class":573,"line":583},[2964],{"type":43,"tag":571,"props":2965,"children":2966},{"style":776},[2967],{"type":48,"value":2968},"# Drops the `exported_*` collisions — NOT pod, which makes K8s series unique.\n",{"type":43,"tag":571,"props":2970,"children":2971},{"class":573,"line":592},[2972,2976],{"type":43,"tag":571,"props":2973,"children":2974},{"style":2623},[2975],{"type":48,"value":2626},{"type":43,"tag":571,"props":2977,"children":2978},{"style":693},[2979],{"type":48,"value":2631},{"type":43,"tag":571,"props":2981,"children":2982},{"class":573,"line":602},[2983,2987,2992,2996],{"type":43,"tag":571,"props":2984,"children":2985},{"style":693},[2986],{"type":48,"value":2639},{"type":43,"tag":571,"props":2988,"children":2989},{"style":2623},[2990],{"type":48,"value":2991}," regex",{"type":43,"tag":571,"props":2993,"children":2994},{"style":693},[2995],{"type":48,"value":131},{"type":43,"tag":571,"props":2997,"children":2998},{"style":682},[2999],{"type":48,"value":3000}," exported_(instance|node|host)\n",{"type":43,"tag":571,"props":3002,"children":3003},{"class":573,"line":611},[3004,3008,3012],{"type":43,"tag":571,"props":3005,"children":3006},{"style":2623},[3007],{"type":48,"value":2686},{"type":43,"tag":571,"props":3009,"children":3010},{"style":693},[3011],{"type":48,"value":131},{"type":43,"tag":571,"props":3013,"children":3014},{"style":682},[3015],{"type":48,"value":3016}," labeldrop\n",{"type":43,"tag":51,"props":3018,"children":3019},{},[3020,3022,3028],{"type":48,"value":3021},"Then the target labels from ",{"type":43,"tag":70,"props":3023,"children":3025},{"className":3024},[],[3026],{"type":48,"value":3027},"relabel_configs",{"type":48,"value":3029}," apply cleanly. Prefer fixing the app.",{"type":43,"tag":553,"props":3031,"children":3033},{"id":3032},"federation-amplifying-cardinality",[3034],{"type":48,"value":3035},"Federation amplifying cardinality",{"type":43,"tag":51,"props":3037,"children":3038},{},[3039,3043,3045,3051,3052,3058],{"type":43,"tag":62,"props":3040,"children":3041},{},[3042],{"type":48,"value":2083},{"type":48,"value":3044},": A federated Prometheus or Mimir global view has way more series than expected. Each source has its own ",{"type":43,"tag":70,"props":3046,"children":3048},{"className":3047},[],[3049],{"type":48,"value":3050},"cluster",{"type":48,"value":423},{"type":43,"tag":70,"props":3053,"children":3055},{"className":3054},[],[3056],{"type":48,"value":3057},"region",{"type":48,"value":3059}," label, multiplying.",{"type":43,"tag":51,"props":3061,"children":3062},{},[3063,3067],{"type":43,"tag":62,"props":3064,"children":3065},{},[3066],{"type":48,"value":2270},{"type":48,"value":3068},": this is usually expected — federation by design preserves source labels. If the series count is too high, federate only aggregated recording rules, not raw metrics:",{"type":43,"tag":560,"props":3070,"children":3072},{"className":2411,"code":3071,"language":2413,"meta":565,"style":565},"- job_name: federate\n  honor_labels: true\n  metrics_path: \u002Ffederate\n  params:\n    'match[]':\n      - '{__name__=~\".*:.*\"}'  # Recording-rule naming convention only\n",[3073],{"type":43,"tag":70,"props":3074,"children":3075},{"__ignoreMap":565},[3076,3098,3116,3133,3145,3166],{"type":43,"tag":571,"props":3077,"children":3078},{"class":573,"line":574},[3079,3084,3089,3093],{"type":43,"tag":571,"props":3080,"children":3081},{"style":693},[3082],{"type":48,"value":3083},"-",{"type":43,"tag":571,"props":3085,"children":3086},{"style":2623},[3087],{"type":48,"value":3088}," job_name",{"type":43,"tag":571,"props":3090,"children":3091},{"style":693},[3092],{"type":48,"value":131},{"type":43,"tag":571,"props":3094,"children":3095},{"style":682},[3096],{"type":48,"value":3097}," federate\n",{"type":43,"tag":571,"props":3099,"children":3100},{"class":573,"line":583},[3101,3106,3110],{"type":43,"tag":571,"props":3102,"children":3103},{"style":2623},[3104],{"type":48,"value":3105},"  honor_labels",{"type":43,"tag":571,"props":3107,"children":3108},{"style":693},[3109],{"type":48,"value":131},{"type":43,"tag":571,"props":3111,"children":3113},{"style":3112},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[3114],{"type":48,"value":3115}," true\n",{"type":43,"tag":571,"props":3117,"children":3118},{"class":573,"line":592},[3119,3124,3128],{"type":43,"tag":571,"props":3120,"children":3121},{"style":2623},[3122],{"type":48,"value":3123},"  metrics_path",{"type":43,"tag":571,"props":3125,"children":3126},{"style":693},[3127],{"type":48,"value":131},{"type":43,"tag":571,"props":3129,"children":3130},{"style":682},[3131],{"type":48,"value":3132}," \u002Ffederate\n",{"type":43,"tag":571,"props":3134,"children":3135},{"class":573,"line":602},[3136,3141],{"type":43,"tag":571,"props":3137,"children":3138},{"style":2623},[3139],{"type":48,"value":3140},"  params",{"type":43,"tag":571,"props":3142,"children":3143},{"style":693},[3144],{"type":48,"value":2631},{"type":43,"tag":571,"props":3146,"children":3147},{"class":573,"line":611},[3148,3153,3158,3162],{"type":43,"tag":571,"props":3149,"children":3150},{"style":693},[3151],{"type":48,"value":3152},"    '",{"type":43,"tag":571,"props":3154,"children":3155},{"style":682},[3156],{"type":48,"value":3157},"match[]",{"type":43,"tag":571,"props":3159,"children":3160},{"style":693},[3161],{"type":48,"value":1586},{"type":43,"tag":571,"props":3163,"children":3164},{"style":693},[3165],{"type":48,"value":2631},{"type":43,"tag":571,"props":3167,"children":3168},{"class":573,"line":1507},[3169,3174,3178,3183,3187],{"type":43,"tag":571,"props":3170,"children":3171},{"style":693},[3172],{"type":48,"value":3173},"      -",{"type":43,"tag":571,"props":3175,"children":3176},{"style":693},[3177],{"type":48,"value":1576},{"type":43,"tag":571,"props":3179,"children":3180},{"style":682},[3181],{"type":48,"value":3182},"{__name__=~\".*:.*\"}",{"type":43,"tag":571,"props":3184,"children":3185},{"style":693},[3186],{"type":48,"value":1586},{"type":43,"tag":571,"props":3188,"children":3189},{"style":776},[3190],{"type":48,"value":3191},"  # Recording-rule naming convention only\n",{"type":43,"tag":79,"props":3193,"children":3194},{},[],{"type":43,"tag":83,"props":3196,"children":3198},{"id":3197},"remediation-decision-tree",[3199],{"type":48,"value":3200},"Remediation Decision Tree",{"type":43,"tag":560,"props":3202,"children":3206},{"className":3203,"code":3205,"language":48},[3204],"language-text","Cardinality fire confirmed\n│\n├── Need to stop the bleeding NOW (production OOM, ingest 429s)\n│   └── Drop the ENTIRE offending metric via metric_relabel_configs (action: drop on __name__)\n│       (also applies to Alloy\u002FAgent — same syntax)\n│       Do NOT labeldrop a distinguishing label — it breaks the data, see \"The One Rule\".\n│       Then schedule the proper fix.\n│\n├── It's a Grafana Cloud Active Series bill issue, not a perf issue\n│   ├── Cardinality is structural and you can't fix the app\n│   │   └── Route to `adaptive-metrics` skill (post-ingest aggregation rules — the safe way)\n│   └── You want metric-by-metric DPM breakdown\n│       └── Route to `dpm-finder` skill\n│\n├── It's a fixable application bug (unbounded label, debug metric in prod)\n│   ├── Short-term: drop the whole metric at scrape, OR aggregate via Adaptive Metrics\n│   └── Long-term: fix in code; route to `prometheus-label-strategy` for design guidance\n│\n├── It's histogram cardinality\n│   ├── Reduce labels on the underlying histogram AT THE SOURCE (14× win per label)\n│   ├── Reduce bucket count if appropriate\n│   └── Consider native histograms for high-resolution latency\n│\n└── It's churn (deploy-driven)\n    ├── Stop EMITTING `version`\u002F`git_sha`\u002F`instance` from app code (use info-metric for version)\n    ├── Keep `pod` — never drop it; if pod-level series are too costly, use Adaptive Metrics\n    └── Verify K8s SD relabel rules aren't mapping in `uid` or other ephemeral fields\n",[3207],{"type":43,"tag":70,"props":3208,"children":3209},{"__ignoreMap":565},[3210],{"type":48,"value":3205},{"type":43,"tag":79,"props":3212,"children":3213},{},[],{"type":43,"tag":83,"props":3215,"children":3217},{"id":3216},"emergency-drop-patterns-copy-paste-ready",[3218],{"type":48,"value":3219},"Emergency Drop Patterns (copy-paste ready)",{"type":43,"tag":51,"props":3221,"children":3222},{},[3223,3225,3230,3232,3236],{"type":48,"value":3224},"These are the ",{"type":43,"tag":62,"props":3226,"children":3227},{},[3228],{"type":48,"value":3229},"safe",{"type":48,"value":3231}," scrape-time emergency actions: dropping an ",{"type":43,"tag":159,"props":3233,"children":3234},{},[3235],{"type":48,"value":211},{"type":48,"value":3237}," unwanted metric. They do not merge distinct series, so they don't corrupt the data.",{"type":43,"tag":3239,"props":3240,"children":3241},"blockquote",{},[3242],{"type":43,"tag":51,"props":3243,"children":3244},{},[3245,3247,3259,3260,3265,3267,3272,3274,3280,3282,3287,3289,3293,3295,3300,3302,3307,3309,3313,3315,3320,3322,3327],{"type":48,"value":3246},"⚠️ There is intentionally ",{"type":43,"tag":62,"props":3248,"children":3249},{},[3250,3252,3257],{"type":48,"value":3251},"no ",{"type":43,"tag":70,"props":3253,"children":3255},{"className":3254},[],[3256],{"type":48,"value":99},{"type":48,"value":3258}," of a distinguishing label",{"type":48,"value":149},{"type":43,"tag":62,"props":3261,"children":3262},{},[3263],{"type":48,"value":3264},"no value-normalizing relabel",{"type":48,"value":3266}," here. Both merge distinct series and break ",{"type":43,"tag":70,"props":3268,"children":3270},{"className":3269},[],[3271],{"type":48,"value":147},{"type":48,"value":3273},"\u002FDPM (see ",{"type":43,"tag":360,"props":3275,"children":3277},{"href":3276},"#before-you-remediate-the-one-rule",[3278],{"type":48,"value":3279},"The One Rule",{"type":48,"value":3281},"). To reduce cardinality ",{"type":43,"tag":159,"props":3283,"children":3284},{},[3285],{"type":48,"value":3286},"without",{"type":48,"value":3288}," dropping the whole metric, fix the source or use ",{"type":43,"tag":62,"props":3290,"children":3291},{},[3292],{"type":48,"value":265},{"type":48,"value":3294}," (route to ",{"type":43,"tag":70,"props":3296,"children":3298},{"className":3297},[],[3299],{"type":48,"value":287},{"type":48,"value":3301},"). The only safe ",{"type":43,"tag":70,"props":3303,"children":3305},{"className":3304},[],[3306],{"type":48,"value":99},{"type":48,"value":3308}," is removing a label that ",{"type":43,"tag":159,"props":3310,"children":3311},{},[3312],{"type":48,"value":2924},{"type":48,"value":3314}," a target label (e.g. ",{"type":43,"tag":70,"props":3316,"children":3318},{"className":3317},[],[3319],{"type":48,"value":2851},{"type":48,"value":3321},") — see ",{"type":43,"tag":360,"props":3323,"children":3325},{"href":3324},"#app-emitted-labels-colliding-with-target-labels",[3326],{"type":48,"value":2821},{"type":48,"value":77},{"type":43,"tag":51,"props":3329,"children":3330},{},[3331,3333,3339],{"type":48,"value":3332},"For Prometheus ",{"type":43,"tag":70,"props":3334,"children":3336},{"className":3335},[],[3337],{"type":48,"value":3338},"scrape_configs",{"type":48,"value":131},{"type":43,"tag":560,"props":3341,"children":3343},{"className":2411,"code":3342,"language":2413,"meta":565,"style":565},"metric_relabel_configs:\n  # Drop a specific bad metric entirely\n  - source_labels: [__name__]\n    regex: bad_metric_name\n    action: drop\n\n  # Drop a set of debug\u002Ftemporary metrics by name prefix\n  - source_labels: [__name__]\n    regex: debug_.*\n    action: drop\n",[3344],{"type":43,"tag":70,"props":3345,"children":3346},{"__ignoreMap":565},[3347,3358,3366,3393,3409,3424,3431,3440,3468,3485],{"type":43,"tag":571,"props":3348,"children":3349},{"class":573,"line":574},[3350,3354],{"type":43,"tag":571,"props":3351,"children":3352},{"style":2623},[3353],{"type":48,"value":2626},{"type":43,"tag":571,"props":3355,"children":3356},{"style":693},[3357],{"type":48,"value":2631},{"type":43,"tag":571,"props":3359,"children":3360},{"class":573,"line":583},[3361],{"type":43,"tag":571,"props":3362,"children":3363},{"style":776},[3364],{"type":48,"value":3365},"  # Drop a specific bad metric entirely\n",{"type":43,"tag":571,"props":3367,"children":3368},{"class":573,"line":592},[3369,3373,3377,3381,3385,3389],{"type":43,"tag":571,"props":3370,"children":3371},{"style":693},[3372],{"type":48,"value":2639},{"type":43,"tag":571,"props":3374,"children":3375},{"style":2623},[3376],{"type":48,"value":2644},{"type":43,"tag":571,"props":3378,"children":3379},{"style":693},[3380],{"type":48,"value":131},{"type":43,"tag":571,"props":3382,"children":3383},{"style":693},[3384],{"type":48,"value":2653},{"type":43,"tag":571,"props":3386,"children":3387},{"style":682},[3388],{"type":48,"value":229},{"type":43,"tag":571,"props":3390,"children":3391},{"style":693},[3392],{"type":48,"value":1044},{"type":43,"tag":571,"props":3394,"children":3395},{"class":573,"line":602},[3396,3400,3404],{"type":43,"tag":571,"props":3397,"children":3398},{"style":2623},[3399],{"type":48,"value":2669},{"type":43,"tag":571,"props":3401,"children":3402},{"style":693},[3403],{"type":48,"value":131},{"type":43,"tag":571,"props":3405,"children":3406},{"style":682},[3407],{"type":48,"value":3408}," bad_metric_name\n",{"type":43,"tag":571,"props":3410,"children":3411},{"class":573,"line":611},[3412,3416,3420],{"type":43,"tag":571,"props":3413,"children":3414},{"style":2623},[3415],{"type":48,"value":2686},{"type":43,"tag":571,"props":3417,"children":3418},{"style":693},[3419],{"type":48,"value":131},{"type":43,"tag":571,"props":3421,"children":3422},{"style":682},[3423],{"type":48,"value":2695},{"type":43,"tag":571,"props":3425,"children":3426},{"class":573,"line":1507},[3427],{"type":43,"tag":571,"props":3428,"children":3429},{"emptyLinePlaceholder":596},[3430],{"type":48,"value":599},{"type":43,"tag":571,"props":3432,"children":3434},{"class":573,"line":3433},7,[3435],{"type":43,"tag":571,"props":3436,"children":3437},{"style":776},[3438],{"type":48,"value":3439},"  # Drop a set of debug\u002Ftemporary metrics by name prefix\n",{"type":43,"tag":571,"props":3441,"children":3443},{"class":573,"line":3442},8,[3444,3448,3452,3456,3460,3464],{"type":43,"tag":571,"props":3445,"children":3446},{"style":693},[3447],{"type":48,"value":2639},{"type":43,"tag":571,"props":3449,"children":3450},{"style":2623},[3451],{"type":48,"value":2644},{"type":43,"tag":571,"props":3453,"children":3454},{"style":693},[3455],{"type":48,"value":131},{"type":43,"tag":571,"props":3457,"children":3458},{"style":693},[3459],{"type":48,"value":2653},{"type":43,"tag":571,"props":3461,"children":3462},{"style":682},[3463],{"type":48,"value":229},{"type":43,"tag":571,"props":3465,"children":3466},{"style":693},[3467],{"type":48,"value":1044},{"type":43,"tag":571,"props":3469,"children":3471},{"class":573,"line":3470},9,[3472,3476,3480],{"type":43,"tag":571,"props":3473,"children":3474},{"style":2623},[3475],{"type":48,"value":2669},{"type":43,"tag":571,"props":3477,"children":3478},{"style":693},[3479],{"type":48,"value":131},{"type":43,"tag":571,"props":3481,"children":3482},{"style":682},[3483],{"type":48,"value":3484}," debug_.*\n",{"type":43,"tag":571,"props":3486,"children":3488},{"class":573,"line":3487},10,[3489,3493,3497],{"type":43,"tag":571,"props":3490,"children":3491},{"style":2623},[3492],{"type":48,"value":2686},{"type":43,"tag":571,"props":3494,"children":3495},{"style":693},[3496],{"type":48,"value":131},{"type":43,"tag":571,"props":3498,"children":3499},{"style":682},[3500],{"type":48,"value":2695},{"type":43,"tag":51,"props":3502,"children":3503},{},[3504,3506,3512],{"type":48,"value":3505},"For Grafana Alloy (",{"type":43,"tag":70,"props":3507,"children":3509},{"className":3508},[],[3510],{"type":48,"value":3511},"prometheus.relabel",{"type":48,"value":3513}," component):",{"type":43,"tag":560,"props":3515,"children":3519},{"className":3516,"code":3517,"language":3518,"meta":565,"style":565},"language-alloy shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","prometheus.relabel \"drop_bad_metric\" {\n  forward_to = [prometheus.remote_write.default.receiver]\n\n  rule {\n    source_labels = [\"__name__\"]\n    regex = \"bad_metric_name\"\n    action = \"drop\"\n  }\n}\n","alloy",[3520],{"type":43,"tag":70,"props":3521,"children":3522},{"__ignoreMap":565},[3523,3531,3539,3546,3554,3562,3570,3578,3586],{"type":43,"tag":571,"props":3524,"children":3525},{"class":573,"line":574},[3526],{"type":43,"tag":571,"props":3527,"children":3528},{},[3529],{"type":48,"value":3530},"prometheus.relabel \"drop_bad_metric\" {\n",{"type":43,"tag":571,"props":3532,"children":3533},{"class":573,"line":583},[3534],{"type":43,"tag":571,"props":3535,"children":3536},{},[3537],{"type":48,"value":3538},"  forward_to = [prometheus.remote_write.default.receiver]\n",{"type":43,"tag":571,"props":3540,"children":3541},{"class":573,"line":592},[3542],{"type":43,"tag":571,"props":3543,"children":3544},{"emptyLinePlaceholder":596},[3545],{"type":48,"value":599},{"type":43,"tag":571,"props":3547,"children":3548},{"class":573,"line":602},[3549],{"type":43,"tag":571,"props":3550,"children":3551},{},[3552],{"type":48,"value":3553},"  rule {\n",{"type":43,"tag":571,"props":3555,"children":3556},{"class":573,"line":611},[3557],{"type":43,"tag":571,"props":3558,"children":3559},{},[3560],{"type":48,"value":3561},"    source_labels = [\"__name__\"]\n",{"type":43,"tag":571,"props":3563,"children":3564},{"class":573,"line":1507},[3565],{"type":43,"tag":571,"props":3566,"children":3567},{},[3568],{"type":48,"value":3569},"    regex = \"bad_metric_name\"\n",{"type":43,"tag":571,"props":3571,"children":3572},{"class":573,"line":3433},[3573],{"type":43,"tag":571,"props":3574,"children":3575},{},[3576],{"type":48,"value":3577},"    action = \"drop\"\n",{"type":43,"tag":571,"props":3579,"children":3580},{"class":573,"line":3442},[3581],{"type":43,"tag":571,"props":3582,"children":3583},{},[3584],{"type":48,"value":3585},"  }\n",{"type":43,"tag":571,"props":3587,"children":3588},{"class":573,"line":3470},[3589],{"type":43,"tag":571,"props":3590,"children":3591},{},[3592],{"type":48,"value":3593},"}\n",{"type":43,"tag":51,"props":3595,"children":3596},{},[3597,3602],{"type":43,"tag":62,"props":3598,"children":3599},{},[3600],{"type":48,"value":3601},"Always test in staging first",{"type":48,"value":3603},", and prefer fixing the source or using Adaptive Metrics over any scrape-time drop.",{"type":43,"tag":79,"props":3605,"children":3606},{},[],{"type":43,"tag":83,"props":3608,"children":3610},{"id":3609},"when-to-hand-off",[3611],{"type":48,"value":3612},"When to Hand Off",{"type":43,"tag":133,"props":3614,"children":3615},{},[3616,3631,3645,3659,3673,3687],{"type":43,"tag":137,"props":3617,"children":3618},{},[3619,3624,3626],{"type":43,"tag":62,"props":3620,"children":3621},{},[3622],{"type":48,"value":3623},"\"Now design a label strategy so this doesn't happen again\"",{"type":48,"value":3625}," → ",{"type":43,"tag":70,"props":3627,"children":3629},{"className":3628},[],[3630],{"type":48,"value":75},{"type":43,"tag":137,"props":3632,"children":3633},{},[3634,3639,3640],{"type":43,"tag":62,"props":3635,"children":3636},{},[3637],{"type":48,"value":3638},"\"We need to keep these metrics but reduce cost\"",{"type":48,"value":3625},{"type":43,"tag":70,"props":3641,"children":3643},{"className":3642},[],[3644],{"type":48,"value":287},{"type":43,"tag":137,"props":3646,"children":3647},{},[3648,3653,3654],{"type":43,"tag":62,"props":3649,"children":3650},{},[3651],{"type":48,"value":3652},"\"Which metric is the most expensive in DPM terms?\"",{"type":48,"value":3625},{"type":43,"tag":70,"props":3655,"children":3657},{"className":3656},[],[3658],{"type":48,"value":492},{"type":43,"tag":137,"props":3660,"children":3661},{},[3662,3667,3668],{"type":43,"tag":62,"props":3663,"children":3664},{},[3665],{"type":48,"value":3666},"\"Write the PromQL to find this\"",{"type":48,"value":3625},{"type":43,"tag":70,"props":3669,"children":3671},{"className":3670},[],[3672],{"type":48,"value":564},{"type":43,"tag":137,"props":3674,"children":3675},{},[3676,3681,3682],{"type":43,"tag":62,"props":3677,"children":3678},{},[3679],{"type":48,"value":3680},"\"Configure this in Alloy\"",{"type":48,"value":3625},{"type":43,"tag":70,"props":3683,"children":3685},{"className":3684},[],[3686],{"type":48,"value":3518},{"type":43,"tag":137,"props":3688,"children":3689},{},[3690,3695,3696,3702],{"type":43,"tag":62,"props":3691,"children":3692},{},[3693],{"type":48,"value":3694},"\"Why is my Loki slow?\"",{"type":48,"value":3625},{"type":43,"tag":70,"props":3697,"children":3699},{"className":3698},[],[3700],{"type":48,"value":3701},"loki-label-analyzer",{"type":48,"value":3703}," (different system, same family of problems)",{"type":43,"tag":51,"props":3705,"children":3706},{},[3707,3709,3714],{"type":48,"value":3708},"This skill's lane is ",{"type":43,"tag":62,"props":3710,"children":3711},{},[3712],{"type":48,"value":3713},"diagnosis under pressure",{"type":48,"value":3715},". Prevention, design, and post-ingest cost optimization live elsewhere.",{"type":43,"tag":3717,"props":3718,"children":3719},"style",{},[3720],{"type":48,"value":3721},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3723,"total":3834},[3724,3736,3753,3769,3786,3801,3819],{"slug":287,"name":287,"fn":3725,"description":3726,"org":3727,"tags":3728,"stars":26,"repoUrl":27,"updatedAt":3735},"optimize Grafana Cloud metrics costs","Cut Grafana Cloud Metrics cost by shrinking active-series count with Adaptive Metrics aggregation rules — auto-recommendations from query history, custom exact\u002Fregex rules, label-drop config, unused-metric detection, and Alloy remote_write fallback. Use when investigating a high Mimir\u002FGrafana Cloud bill, hunting high-cardinality labels (`pod_uid`, `service_instance_id`, `version`), pre-aggregating counters\u002Fgauges, dropping unused metrics, or measuring `grafanacloud_instance_active_series` before\u002Fafter — even when the user says \"reduce cardinality\", \"too many series\", \"metrics spend\", \"active series count is exploding\", or \"drop the version label\" without naming Adaptive Metrics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3729,3732,3733,3734],{"name":3730,"slug":3731,"type":15},"Cost Optimization","cost-optimization",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},{"name":13,"slug":14,"type":15},"2026-07-12T07:44:27.451068",{"slug":3737,"name":3737,"fn":3738,"description":3739,"org":3740,"tags":3741,"stars":26,"repoUrl":27,"updatedAt":3752},"admin","manage Grafana Cloud accounts and RBAC","Manage Grafana Cloud accounts — organizations, stacks, RBAC roles and assignments, SSO\u002FSAML\u002FOAuth\u002FGitHub auth, service accounts for CI\u002FCD, user invites, team membership, and API-driven provisioning. Creates stacks via the Cloud API, mints service-account tokens, applies role assignments, configures SSO providers, and provisions teams\u002Ffolders\u002Fdashboards via Terraform. Use when managing Grafana Cloud access, configuring SSO\u002FSAML\u002FOAuth, setting up service accounts for Terraform\u002FCI\u002FCD, assigning RBAC roles, inviting users, managing multiple stacks or organizations, provisioning cloud resources via API or Terraform, or auditing admin actions — even when the user says \"set up SSO\", \"create a stack\", \"make a service account\", or \"onboard a team\" without explicitly saying \"admin\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3742,3745,3748,3749],{"name":3743,"slug":3744,"type":15},"Access Control","access-control",{"name":3746,"slug":3747,"type":15},"Auth","auth",{"name":9,"slug":8,"type":15},{"name":3750,"slug":3751,"type":15},"Operations","operations","2026-07-12T07:44:12.078436",{"slug":3754,"name":3754,"fn":3755,"description":3756,"org":3757,"tags":3758,"stars":26,"repoUrl":27,"updatedAt":3768},"admission-control","implement admission control webhooks","Use when the user asks to \"write a validator\", \"add validation\", \"implement admission control\", \"write a mutating webhook\", \"add a mutation handler\", \"validate incoming resources\", \"implement admission logic\", \"add admission webhooks\", \"write ingress validation\", or asks how to validate or mutate resources before they are persisted in a grafana-app-sdk app. Provides guidance on implementing validation and mutation admission handlers for grafana-app-sdk apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3759,3762,3765],{"name":3760,"slug":3761,"type":15},"Architecture","architecture",{"name":3763,"slug":3764,"type":15},"Security","security",{"name":3766,"slug":3767,"type":15},"Validation","validation","2026-07-12T07:45:06.148973",{"slug":3770,"name":3770,"fn":3771,"description":3772,"org":3773,"tags":3774,"stars":26,"repoUrl":27,"updatedAt":3785},"alerting-irm","configure Grafana Alerting and Incident Management","Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack\u002FPagerDuty\u002Femail\u002Fwebhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty\u002FSlack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says \"page me on errors\", \"alert me when X happens\", \"route this to the platform team\", or \"set up an SLO\" without naming Alerting or IRM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3775,3778,3779,3782],{"name":3776,"slug":3777,"type":15},"Alerting","alerting",{"name":9,"slug":8,"type":15},{"name":3780,"slug":3781,"type":15},"Incident Response","incident-response",{"name":3783,"slug":3784,"type":15},"Monitoring","monitoring","2026-07-12T07:44:02.393397",{"slug":3518,"name":3518,"fn":3787,"description":3788,"org":3789,"tags":3790,"stars":26,"repoUrl":27,"updatedAt":3800},"build unified telemetry pipelines with Grafana Alloy","Build a unified telemetry pipeline with Grafana Alloy — one OpenTelemetry-compatible binary that collects metrics, logs, traces, and profiles and ships to Grafana Cloud \u002F Prometheus \u002F Loki \u002F Tempo \u002F Pyroscope. Covers the Alloy config language (blocks, `sys.env`, component refs), `prometheus.scrape` → `remote_write`, `loki.source.file` + `loki.process` → `loki.write`, `otelcol.receiver.otlp` → `otelcol.exporter.otlp`, `pyroscope.scrape`, K8s \u002F Docker \u002F EC2 discovery, relabeling, modules (`import.file\u002Fgit\u002Fhttp`), clustering, Fleet Management `remotecfg`, the Alloy UI at `:12345`, and `alloy fmt` \u002F `alloy validate`. Use when writing a `config.alloy`, replacing Grafana Agent \u002F OTel Collector, scraping K8s pods, parsing logs, ingesting OTLP, or debugging \"Alloy isn't sending anything\" — even when the user says \"set up the agent\", \"write me a scrape config\", \"drop these logs before sending\", or \"OTel collector config\" without naming Alloy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3791,3792,3795,3796,3797],{"name":9,"slug":8,"type":15},{"name":3793,"slug":3794,"type":15},"Logs","logs",{"name":21,"slug":22,"type":15},{"name":13,"slug":14,"type":15},{"name":3798,"slug":3799,"type":15},"OpenTelemetry","opentelemetry","2026-07-12T07:43:54.817139",{"slug":3802,"name":3802,"fn":3803,"description":3804,"org":3805,"tags":3806,"stars":26,"repoUrl":27,"updatedAt":3818},"app-observability","monitor application performance in Grafana Cloud","Get RED metrics + service maps + frontend RUM + AI\u002FLLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50\u002Fp95\u002Fp99 latency, exemplar-to-trace, traces-to-logs \u002F profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token \u002F cost \u002F latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says \"set up APM\", \"show service map\", \"monitor browser perf\", \"session replay\", \"RUM SDK\", or \"watch our OpenAI bill\" without naming App \u002F Frontend \u002F AI Observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3807,3810,3813,3814,3817],{"name":3808,"slug":3809,"type":15},"APM","apm",{"name":3811,"slug":3812,"type":15},"Distributed Tracing","distributed-tracing",{"name":9,"slug":8,"type":15},{"name":3815,"slug":3816,"type":15},"LLM","llm",{"name":13,"slug":14,"type":15},"2026-07-12T07:44:34.500406",{"slug":3820,"name":3820,"fn":3821,"description":3822,"org":3823,"tags":3824,"stars":26,"repoUrl":27,"updatedAt":3833},"app-sdk-concepts","scaffold and configure Grafana apps","Use when starting any grafana-app-sdk work — scaffolding a Grafana app, initializing a Grafana App Platform app, picking a deployment mode (standalone operator \u002F grafana\u002Fapps \u002F frontend-only), wiring app-specific config, or onboarding to the SDK. Covers `grafana-app-sdk` CLI install, `project init` per deployment mode, project layout, the schema-centric workflow (CUE kinds → generated code → reconciler\u002Fadmission logic), and `SpecificConfig` for env-driven app configuration. Use even if the user just says \"make a new app\", \"Grafana app platform\", \"kinds and watchers\", \"operator scaffold\", or asks about `apps\u002F` inside the Grafana repo, without naming the SDK explicitly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3825,3826,3829,3832],{"name":3760,"slug":3761,"type":15},{"name":3827,"slug":3828,"type":15},"Deployment","deployment",{"name":3830,"slug":3831,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},"2026-07-12T07:45:08.595757",48,{"items":3836,"total":4013},[3837,3852,3871,3890,3905,3919,3932,3947,3964,3975,3988,4001],{"slug":3838,"name":3838,"fn":3839,"description":3840,"org":3841,"tags":3842,"stars":3849,"repoUrl":3850,"updatedAt":3851},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3843,3844,3847,3848],{"name":3811,"slug":3812,"type":15},{"name":3845,"slug":3846,"type":15},"Frontend","frontend",{"name":3783,"slug":3784,"type":15},{"name":13,"slug":14,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":3853,"name":3853,"fn":3854,"description":3855,"org":3856,"tags":3857,"stars":3868,"repoUrl":3869,"updatedAt":3870},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3858,3861,3864,3867],{"name":3859,"slug":3860,"type":15},"API Development","api-development",{"name":3862,"slug":3863,"type":15},"Authentication","authentication",{"name":3865,"slug":3866,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":3872,"name":3872,"fn":3873,"description":3874,"org":3875,"tags":3876,"stars":3868,"repoUrl":3869,"updatedAt":3889},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3877,3880,3883,3884,3887],{"name":3878,"slug":3879,"type":15},"CSV","csv",{"name":3881,"slug":3882,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":3885,"slug":3886,"type":15},"GraphQL","graphql",{"name":3888,"slug":861,"type":15},"JSON","2026-07-15T05:34:05.773947",{"slug":3891,"name":3891,"fn":3892,"description":3893,"org":3894,"tags":3895,"stars":3902,"repoUrl":3903,"updatedAt":3904},"agento11y","manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3896,3899,3900,3901],{"name":3897,"slug":3898,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":3783,"slug":3784,"type":15},{"name":13,"slug":14,"type":15},430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-25T05:30:40.29622",{"slug":3906,"name":3906,"fn":3907,"description":3908,"org":3909,"tags":3910,"stars":3902,"repoUrl":3903,"updatedAt":3918},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3911,3912,3913,3916,3917],{"name":3897,"slug":3898,"type":15},{"name":9,"slug":8,"type":15},{"name":3914,"slug":3915,"type":15},"Instrumentation","instrumentation",{"name":3815,"slug":3816,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:52.580237",{"slug":3920,"name":3920,"fn":3921,"description":3922,"org":3923,"tags":3924,"stars":3902,"repoUrl":3903,"updatedAt":3931},"agento11y-prod-setup","setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3925,3926,3929,3930],{"name":3897,"slug":3898,"type":15},{"name":3927,"slug":3928,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:53.576347",{"slug":3933,"name":3933,"fn":3934,"description":3935,"org":3936,"tags":3937,"stars":3902,"repoUrl":3903,"updatedAt":3946},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3938,3939,3940,3943],{"name":3897,"slug":3898,"type":15},{"name":9,"slug":8,"type":15},{"name":3941,"slug":3942,"type":15},"QA","qa",{"name":3944,"slug":3945,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":3948,"name":3948,"fn":3949,"description":3950,"org":3951,"tags":3952,"stars":3902,"repoUrl":3903,"updatedAt":3963},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3953,3956,3959,3962],{"name":3954,"slug":3955,"type":15},"Dashboards","dashboards",{"name":3957,"slug":3958,"type":15},"Data Visualization","data-visualization",{"name":3960,"slug":3961,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":3965,"name":3965,"fn":3966,"description":3967,"org":3968,"tags":3969,"stars":3902,"repoUrl":3903,"updatedAt":3974},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3970,3971,3972,3973],{"name":24,"slug":25,"type":15},{"name":9,"slug":8,"type":15},{"name":3780,"slug":3781,"type":15},{"name":13,"slug":14,"type":15},"2026-07-18T05:11:10.445428",{"slug":3976,"name":3976,"fn":3977,"description":3978,"org":3979,"tags":3980,"stars":3902,"repoUrl":3903,"updatedAt":3987},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3981,3982,3983,3986],{"name":24,"slug":25,"type":15},{"name":9,"slug":8,"type":15},{"name":3984,"slug":3985,"type":15},"Graph Analysis","graph-analysis",{"name":13,"slug":14,"type":15},"2026-07-25T05:30:39.380934",{"slug":3989,"name":3989,"fn":3990,"description":3991,"org":3992,"tags":3993,"stars":3902,"repoUrl":3903,"updatedAt":4000},"gcx","manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3994,3997,3998,3999],{"name":3995,"slug":3996,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":3783,"slug":3784,"type":15},{"name":3750,"slug":3751,"type":15},"2026-07-31T05:53:50.587304",{"slug":4002,"name":4002,"fn":4003,"description":4004,"org":4005,"tags":4006,"stars":3902,"repoUrl":3903,"updatedAt":4012},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4007,4008,4009],{"name":3995,"slug":3996,"type":15},{"name":9,"slug":8,"type":15},{"name":4010,"slug":4011,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80]