[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-prometheus-label-strategy":3,"mdc--tiy77j-key":34,"related-org-grafana-prometheus-label-strategy":4543,"related-repo-grafana-prometheus-label-strategy":4735},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"prometheus-label-strategy","design Prometheus label strategies","Expert evaluator for Prometheus label strategy on Grafana Cloud. Audits, designs, and improves label schemas using cardinality scoring, access-pattern alignment, static vs. dynamic label rules, histogram bucket discipline, and instrumentation hygiene. Prevents high cardinality at the source — in application code and scrape target labels — without dropping labels that make series unique (which breaks the data). For reducing the cost of series already in Grafana Cloud, routes to the adaptive-metrics skill. Use when the user asks to evaluate, audit, design, or improve Prometheus labels — or asks how to prevent high cardinality at the source. For \"why is my Prometheus slow \u002F expensive right now\" triage, see prometheus-cardinality-troubleshooter.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,20],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Prometheus","prometheus",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Metrics","metrics",189,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills","2026-07-12T07:44:23.316295","Apache-2.0",16,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],null,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fgrafana-cloud\u002Fprometheus-label-strategy","---\nname: prometheus-label-strategy\nlicense: Apache-2.0\ndescription: >\n Expert evaluator for Prometheus label strategy on Grafana Cloud. Audits, designs, and\n improves label schemas using cardinality scoring, access-pattern alignment, static vs.\n dynamic label rules, histogram bucket discipline, and instrumentation hygiene. Prevents\n high cardinality at the source — in application code and scrape target labels — without\n dropping labels that make series unique (which breaks the data). For reducing the cost of\n series already in Grafana Cloud, routes to the adaptive-metrics skill. Use when the user\n asks to evaluate, audit, design, or improve Prometheus labels — or asks how to prevent\n high cardinality at the source. For \"why is my Prometheus slow \u002F expensive right now\"\n triage, see prometheus-cardinality-troubleshooter.\n---\n\n# Prometheus Label Strategy Evaluator\n\nYou are an expert in Prometheus label strategy. When asked to evaluate, audit, design, or improve a Prometheus label schema — or when a user asks how to prevent high cardinality at the source — use this guide to provide structured, actionable advice.\n\nThis skill is about **preventing bad labels at the source** — in application instrumentation and in scrape *target* labels — so they never enter storage. It is **not** about stripping labels off metrics after they've been emitted: removing a label that makes a series unique at scrape time silently breaks the data (see [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) below). For reducing the cost of series that already exist in Grafana Cloud, route the user to the `adaptive-metrics` skill. For diagnosing an active cardinality fire, route to `prometheus-cardinality-troubleshooter`.\n\n---\n\n## The One Rule: Never Drop a Label That Makes a Series Unique\n\n**You cannot remove, at scrape time, any label that makes a series unique.** Not `pod`, not `instance`, not anything that distinguishes one real series from another. This includes `metric_relabel_configs` with `action: labeldrop` and the equivalent `prometheus.relabel` rules in Alloy.\n\nIt looks like a cardinality win. It is not — it **breaks the data**, silently and permanently:\n\n- **Counter resets get mixed together.** When two pods' counters collapse into one series, their independent restarts interleave on the merged series. `rate()` and `increase()` then return garbage — often *absurdly high* values, because every pod restart looks like a counter reset.\n- **DPM inflates instead of dropping.** Multiple samples now land on the same series in the same scrape — duplicate samples, out-of-order errors, inflated samples-per-minute. People come back weeks later asking \"why is my DPM so high?\" or \"why is `rate()` returning absurd numbers?\" — and there is **no evidence left in the data** of where it broke.\n- **The aggregation is wrong, not just coarse.** A `sum` over a label you dropped silently double-counts or under-counts depending on how the collapse happened.\n\nThe trap is that none of this errors at config time. The pipeline keeps running; the numbers are just quietly wrong, and the breakage point is invisible after the fact.\n\n**The right tools, in order:**\n\n1. **Don't emit the bad label in the first place** — fix the application code. This is the only place a label can be *removed* without consequence, because the series was never unique on it to begin with.\n2. **For series already flowing into Grafana Cloud that you can't fix at the source → Adaptive Metrics.** This is exactly what it is for: it aggregates series *correctly* — counter-reset-aware, with a recorded audit trail, and reversible — instead of blindly stripping labels. Route the user to the `adaptive-metrics` skill.\n\n`metric_relabel_configs` has a couple of narrow, safe uses (dropping an *entire* unwanted metric; removing a label that *exactly duplicates* a target label) — covered in [Source-Side Prevention](#4-metric_relabel_configs-narrow-safe-uses-only) — but **reducing cardinality by dropping a distinguishing label is never one of them.**\n\n---\n\n## Core Concepts\n\n**Series** are the fundamental unit in Prometheus. Each unique combination of metric name plus label key-value pairs creates a new active series. Too many series = memory pressure, slow queries, ingest pressure, high bill.\n\n**Cardinality** = the number of unique values a label can have. Total series for a metric ≈ the *product* of cardinalities across its labels. A metric with `path` (100 values), `status_code` (10 values), `method` (5 values), and `instance` (50 values) = **250,000 series per metric**. Adding one more high-cardinality label often 10–100×s the count.\n\n**The dual impact rule**: High-cardinality labels hurt on both paths:\n- **Ingestion path**: More active series → larger head block, larger WAL, more memory, larger remote_write payloads, higher Grafana Cloud bill (Active Series + DPM)\n- **Query path**: PromQL operators (`sum by`, `rate`, joins) must materialize matching series in memory. High cardinality balloons query memory and latency\n\n**Series churn** is the silent killer. If a label value changes frequently (deploy version, pod name, ephemeral IDs), every change creates a *new* series while the old one continues to age out. Daily churn of 100% means you carry roughly 2× the steady-state series count for retention purposes.\n\n**The key question for any proposed label**: \"Will queries that use this metric reliably specify or aggregate on this label?\" If no → it should NOT be a label.\n\n---\n\n## Label Evaluation Framework\n\nWhen auditing a label set, assess each label against these criteria.\n\n### Cardinality Scoring\n\n| Label Example | Cardinality | Verdict |\n|---|---|---|\n| `env` (prod\u002Fstaging\u002Fdev) | 2–5 values | ✅ Good |\n| `job` (Prometheus scrape job) | 5–50 values | ✅ Good |\n| `cluster`, `region` | Tens | ✅ Good |\n| `namespace` (K8s) | Tens–low hundreds | ✅ Acceptable |\n| `service`, `workload`, `container` | Tens–hundreds | ✅ Acceptable |\n| `instance` (host:port) | Hundreds–low thousands | ⚠️ Evaluate — fine on per-instance metrics, risky on aggregated ones |\n| `pod` (K8s) | Thousands + transient = high churn | ⚠️ Required for K8s monitoring and series uniqueness — keep it. If `pod`-level series are too expensive, reduce them with Adaptive Metrics; **never** drop at scrape |\n| `path` \u002F `route` (HTTP) | Bounded if templated; unbounded if raw URLs | ⚠️ Only with templated values (`\u002Fusers\u002F:id`) |\n| `version`, `image_tag`, `git_sha` | Grows on every deploy → churn | ⚠️ Use sparingly; consider info-metric pattern |\n| `user_id`, `request_id`, `trace_id` | Unbounded | ❌ Never as label — use exemplars |\n| `customer_id`, `tenant_id` | Often unbounded | ❌ Only acceptable for small fixed tenant counts |\n| `error_message`, `query`, `sql` | Unbounded text | ❌ Never |\n\n### Access Pattern Alignment\n\nFor each label, ask:\n- Do queries on this metric reliably aggregate by or filter on this label?\n- Does this label logically segment the metric the way users think about it?\n- Would removing this label force users to use exemplars, logs, or traces instead — and would that be acceptable for the rare lookup case?\n\n### Static vs. Dynamic Label Values\n\n- **Static \u002F target labels** (set once per scrape target via `relabel_configs`, e.g., `env=prod`, `cluster=us-east`, `team=payments`) add cardinality proportional to *targets*, not requests. Cheap and high-value. Use freely.\n- **Dynamic \u002F sample labels** (emitted by the application per measurement, e.g., `status_code`, `method`, `cache_hit`) multiply cardinality by *value count*. Keep possible values in the single digits or low tens. **The application code is the source of truth — fix it there, not in Prometheus.**\n\n### Consistency Check\n\n- Label *names* consistent across services? (`status` vs `status_code` vs `http_status` produces three separate label families — joins break)\n- Label *values* normalized? (`200` vs `\"200\"`, `GET` vs `get`, `Error` vs `error`)\n- Naming convention consistent? Prometheus convention is `snake_case` for both metric and label names\n- Same concept, same name across services? (`service` vs `svc` vs `app_name`)\n\n### Histogram Bucket Discipline (critical, often missed)\n\nEvery histogram metric multiplies its base cardinality by **(bucket count + 3)** — buckets via `_bucket{le=\"...\"}` plus `_sum`, `_count`, and `_created` (Prometheus 2.39+).\n\n- Default `prometheus.DefBuckets` has 11 buckets → **14× multiplier**\n- A histogram with `method`, `path`, `status` already at 1,000 series becomes **14,000 series** after adding histogram cardinality\n- **Always trim histogram label cardinality first** — labels matter 14× more on histograms than on counters\u002Fgauges\n- Consider native histograms (Prometheus 2.40+) which use a single sparse series instead of one-per-bucket — major cardinality reduction for high-resolution latency tracking\n\n### Info-Metric Pattern (for high-churn metadata)\n\nWhen you want to *know* about a label (e.g., `version`, `git_sha`, `image_tag`) without paying for it on every metric, use an info metric:\n\n```\n# A single low-cardinality counter\u002Fgauge of value 1, with the metadata attached\napp_build_info{app=\"payment-api\", version=\"2.4.1\", git_sha=\"a1b2c3\"} 1\n```\n\nThen join at query time. The classic approach is a vector match with `group_left`:\n```promql\nsum by (version) (\n rate(http_requests_total{app=\"payment-api\"}[5m])\n * on (app) group_left (version) app_build_info\n)\n```\n\nThe `version` label lives on exactly one series per build, not on every metric.\n\n#### The `info()` function (simpler join)\n\nPromQL's `info()` function (experimental, Prometheus 3.0+; enable with `--enable-feature=promql-experimental-functions`) automates the info-metric join so you don't have to hand-write the `* on (...) group_left (...)` match:\n\n```promql\ninfo(\n rate(http_requests_total{app=\"payment-api\"}[5m]),\n {version=~\".+\"}\n)\n```\n\n`info(v, [labelselector])` takes a range\u002Finstant vector `v` and, for each series, finds matching info metrics and adds their labels. The optional second argument is a label-matcher restricting which info labels are attached (here, only `version`). By default `info()` joins against the conventional `target_info` metric and matches on identifying labels (e.g. `instance`, `job`), so it's especially ergonomic for OpenTelemetry-style `target_info`. For custom info metrics like `app_build_info` the explicit `group_left` form above is still the most portable.\n\nPrefer `info()` when you're on Prometheus 3.x and joining against `target_info`; fall back to the explicit `group_left` match for older versions, custom info metrics, or when the experimental feature flag isn't enabled.\n\n---\n\n## Evaluation Output Format\n\nWhen auditing a label set, produce a report in this structure:\n\n```\n## Prometheus Label Strategy Audit\n\n### Summary\n[1-2 sentence overall assessment — total estimated active series, biggest risks]\n\n### Per-Label Analysis\n| Metric Family | Label | Cardinality | Used in Queries? | Verdict | Action |\n|---|---|---|---|---|---|\n| http_requests_total | path | Unbounded (raw URLs) | Sometimes | ❌ Remove | Template in code: `\u002Fusers\u002F:id` not `\u002Fusers\u002F12345` |\n| http_requests_total | pod | High + churn | Rarely | ⚠️ Keep — makes the series unique | If too expensive, aggregate away with Adaptive Metrics; query by `workload` for the common case |\n\n### Histogram-Specific Findings\n[Highlight any histograms with high label cardinality — these are 14×+ amplified]\n\n### Estimated Impact\n- Active series reduction: [X series → Y series]\n- DPM reduction: [X DPM → Y DPM] (samples-per-minute = series × ~6 at 10s scrape)\n- Memory impact: [if measurable]\n\n### Recommended Label Set\n[Final recommended labels per metric family]\n\n### Implementation Plan\n1. [Code changes — instrumentation hygiene: stop emitting bad labels at the source]\n2. [Scrape target labels — relabel_configs (additive: env, cluster, team, workload)]\n3. [Post-ingest cost reduction on series you can't fix at the source — Adaptive Metrics]\n4. [Recording rules to materialize useful aggregates]\n```\n\n---\n\n## Recommended Common Target Labels\n\nThese should be set as **target labels** (via `relabel_configs` on the scrape job, NOT emitted by the app) — they're per-target, low cardinality, high query value:\n\n| Label | Purpose | Notes |\n|---|---|---|\n| `job` | Prometheus scrape job name | Set automatically by Prometheus |\n| `instance` | Target endpoint (`host:port`) | Set automatically; rename via `relabel_configs` to a friendlier value if needed |\n| `env` | Environment (`prod`, `staging`, `dev`) | Set via static_configs labels or service discovery |\n| `cluster` | Multi-cluster differentiation | Critical for federation\u002FMimir multi-tenant |\n| `region` | Geographic region | |\n| `team` \u002F `squad` | Ownership — also useful for access control | |\n| `service` | Logical service identity | One service may span multiple jobs |\n\nThese should **NOT** be re-emitted by the application. If the app emits a `cluster` label, it duplicates the target label and creates collisions \u002F `honor_labels` decisions you don't want to make.\n\n---\n\n## Kubernetes Patterns\n\n### Recommended Labels (from kubernetes_sd_configs)\n\n| Label | Source | Notes |\n|---|---|---|\n| `namespace` | Pod metadata | Always keep |\n| `container` | Pod spec | Low cardinality, useful for multi-container pods |\n| `workload` | Derived: `{controller_kind}\u002F{controller_name}` | Add as a stable aggregation key *alongside* `pod` — static, predictable. It's an addition, not a replacement: don't use it as an excuse to drop `pod` |\n| `service` | K8s Service | If scraping via Service |\n\n### Handling the `pod` Label\n\n`pod` is high-cardinality and transient — it rolls on every deploy and restart, so it dominates churn and series count. But it is also a label that **makes K8s series unique**, and Kubernetes monitoring (per-pod resource attribution, kube-state-metrics joins) depends on it. [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) applies: **do not drop `pod` at scrape time.** Collapsing pods into one series mixes their counter resets and breaks `rate()`.\n\nInstead:\n- **Add `workload`** (`{controller_kind}\u002F{controller_name}`) as a *target* label via `relabel_configs`, so dashboards and alerts can aggregate on the stable workload identity (`sum by (workload)`) without touching `pod`. This is additive — it removes nothing.\n- **Don't emit `pod` from application code** — let it come from Kubernetes service discovery, so there is exactly one source of truth (see below).\n- **If `pod`-level series are genuinely too expensive in Grafana Cloud**, reduce them with **Adaptive Metrics**, which aggregates `pod` away *correctly* (post-ingest, counter-reset-aware, reversible) rather than corrupting the raw data at scrape. Route to the `adaptive-metrics` skill.\n\n### Don't Map Ephemeral Fields into Labels in the First Place\n\n**`uid`** regenerates on every pod recreation and has no legitimate query use. The fix is to **never map it into a label** — leave it out of your `relabel_configs`. (It isn't in default `kubernetes_sd_configs` output unless you explicitly target it.) Don't try to `labeldrop` it after the fact — by then it's already distinguishing series, and removing it breaks the data exactly like dropping any other unique label.\n\n### One Source of Truth for Target Identity\n\n`instance`, `pod`, `node`, and `host` should come from **scrape target labels**, not from application code. If the app *also* emits its own `instance`\u002F`node`, you get duplicates and `honor_labels` collisions. The fix is **in the application** — stop emitting them — not a scrape-time `labeldrop`. (Removing a label that *exactly duplicates* a target label is the one narrow exception; see [metric_relabel_configs](#4-metric_relabel_configs-narrow-safe-uses-only).)\n\n### kube-state-metrics label propagation ⚠️\n- `kube_pod_labels{label_app_kubernetes_io_*=...}` can carry dozens of metadata labels\n- Each unique pod label combination is a new series\n- Restrict at the source with kube-state-metrics' `--metric-labels-allowlist` — this controls what is *ever emitted*, so it's prevention, not destructive after-the-fact dropping\n\n---\n\n## Source-Side Prevention: Where to Fix What\n\nThere are five levers, in **order of preference**:\n\n### 1. Fix in the Application (best)\n\nBad labels emitted by the app are the root cause. Examples:\n- HTTP paths: use templated routes (`\u002Fusers\u002F:id`) not raw paths\n- Error metrics: use a small enum (`error_type=\"timeout\"`) not the error message string\n- User-scoped metrics: don't include `user_id` — use exemplars to point to logs\u002Ftraces\n- Free-form input: never emit user-supplied strings as label values\n\nIf you control the code, this is always the right fix. It saves cost on every downstream system (Prometheus, remote_write, Mimir, Grafana Cloud).\n\n### 2. `relabel_configs` (target-time relabeling)\n\nRuns *before* the scrape. Used to:\n- Set target labels (`env`, `cluster`, `team`) on discovered targets\n- Drop entire targets you don't want to scrape\n- Rewrite `instance` to a friendly value\n- Add identity from service discovery metadata\n\n```yaml\nscrape_configs:\n - job_name: my-app\n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n # Set workload from controller metadata\n - source_labels: [__meta_kubernetes_pod_controller_kind, __meta_kubernetes_pod_controller_name]\n target_label: workload\n separator: \u002F\n # Set env from a pod label\n - source_labels: [__meta_kubernetes_pod_label_env]\n target_label: env\n # Only scrape pods explicitly opted in\n - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n regex: \"true\"\n action: keep\n```\n\n### 3. Adaptive Metrics (Grafana Cloud — post-ingest, the safe way to reduce cardinality)\n\nWhen the cardinality is structural and you *can't* fix it at the source — the label legitimately exists and makes series unique, you just don't need every value at full resolution — **Adaptive Metrics is the correct tool, and the only safe way to reduce the cost of series that already exist.**\n\nIt works *after* ingest, as aggregation rules applied in Grafana Cloud. Crucially, it aggregates series **correctly**:\n- It handles counter resets properly, so `rate()` and `increase()` stay accurate.\n- It records what was aggregated, so there's an audit trail — you can answer \"why did this change?\" later.\n- It's reversible: drop a rule and the full-resolution series come back.\n\nThis is the difference between \"the data is now cheaper\" (Adaptive Metrics) and \"the data is now wrong\" (`labeldrop` at scrape). Route the user to the `adaptive-metrics` skill for rule design.\n\n### 4. `metric_relabel_configs` (narrow, safe uses only)\n\nRuns *after* the scrape, *before* storage.\n\n> ⚠️ **Do not use `metric_relabel_configs` (or Alloy `prometheus.relabel`) to drop a label that distinguishes series — `pod`, `instance`, `user_id`, `path`, anything.** See [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique). It looks like a cardinality fix and silently breaks `rate()`, inflates DPM, and corrupts aggregations. Use the application code (lever 1) or Adaptive Metrics (lever 3) instead. The same caution applies to *normalizing* a label value (e.g. collapsing `status_code` to `2xx`) at scrape — it merges distinct series and produces duplicate-sample errors; do that in code or via Adaptive Metrics, never here.\n\nThe genuinely safe uses are:\n\n- **Drop an entire metric you never want stored** — you're discarding the whole metric, not collapsing distinct series into one:\n ```yaml\n metric_relabel_configs:\n - source_labels: [__name__]\n regex: my_app_request_details\n action: drop\n ```\n- **Remove a label that *exactly duplicates* a target label.** If the app emits its own `cluster`\u002F`instance` that already comes from the scrape target, the target label still provides uniqueness, so removing the duplicate breaks nothing. Prefer fixing the app, but this is a safe stopgap.\n\nThat's the whole list. If you're reaching for `metric_relabel_configs` to bring down a series count, you almost certainly want Adaptive Metrics instead.\n\n### 5. Recording Rules (query-time cardinality reduction)\n\nPre-aggregate expensive series into a lower-cardinality recorded series. Stored at the same data point density but with far fewer series.\n\n```yaml\ngroups:\n - name: http-requests-aggregates\n interval: 30s\n rules:\n # Drop pod\u002Finstance dimension; keep only service-level rollup\n - record: service:http_requests:rate5m\n expr: sum by (service, env, cluster, status_code) (rate(http_requests_total[5m]))\n```\n\nQueries that target the rollup are dramatically cheaper. The raw series still exist — recording rules don't reduce ingest cost (use **Adaptive Metrics** for that — *not* a scrape-time `labeldrop`). They reduce query cost.\n\n---\n\n## Instrumentation Hygiene (for app developers)\n\nIf the user is *writing* instrumentation code, these are the rules:\n\n| Rule | Why |\n|---|---|\n| Never use unbounded user input as a label value | `email`, `user_id`, `query string`, `error message` — they're the #1 cardinality bug |\n| Template HTTP paths before recording | `\u002Fusers\u002F{id}` not `\u002Fusers\u002F12345`. Most frameworks do this via routing metadata |\n| Bound error labels via small enums | `error_type=\"timeout\"` not `error=\"connection to db-shard-7 timed out at 14:32:09\"` |\n| Don't put `version` \u002F `git_sha` \u002F `build_id` on every metric | Use an info metric and join at query time |\n| Don't emit `pod` \u002F `node` \u002F `host` from code | Comes from scrape targets — duplicating creates collisions |\n| Avoid dynamically constructed label *names* (keys) | `metric{[user]=1}` cannot be bounded — use a fixed key |\n| Use histograms sparingly and trim labels first | 14× cardinality amplification |\n| Prefer exemplars over labels for trace correlation | Exemplars carry `trace_id` without inflating cardinality |\n\n### Exemplars (the escape hatch)\n\nExemplars attach a `trace_id` (or any key-value pair) to specific samples *without* making it a label dimension. The ideal home for high-cardinality correlation data.\n\nRequires OpenMetrics format, Prometheus 2.26+, scrape config:\n```yaml\nscrape_configs:\n - job_name: my-app\n enable_protobuf_negotiation: true\n # Or for text-format:\n follow_redirects: true\n```\n\nAnd on the Prometheus server:\n```yaml\nstorage:\n exemplars:\n max_exemplars: 100000\n```\n\nUse exemplars for:\n- `trace_id` correlation (Tempo, Jaeger)\n- `request_id` for specific debug lookups\n- Any sparse \"useful when you need it\" key\n\nQuery exemplars via Grafana's exemplars-on-graph feature, not via PromQL aggregation.\n\n---\n\n## The 80\u002F20 Rule\n\nThe most impactful improvements almost always come from these five changes:\n\n1. **Drop unbounded labels at the app layer** — `path` (untemplated), `user_id`, `error_message`. Single biggest win.\n2. **Trim histogram label cardinality before anything else** — 14× amplification on every histogram.\n3. **Don't emit `pod`\u002F`instance`\u002F`node` from application code** — let them come from scrape targets, and add a stable `workload` target label to aggregate on. (Never *drop* the real `pod` at scrape to cut cardinality — if `pod`-level series are too expensive, use Adaptive Metrics.)\n4. **Use info metrics for `version` \u002F `git_sha` \u002F `image_tag`** — eliminates deploy-driven churn.\n5. **Set target labels via `relabel_configs`, not app code** — `env`, `cluster`, `team`, `service` should never be emitted by the application.\n\nFocus on these before anything else.\n\n---\n\n## Labels to Avoid — Quick Reference\n\n| Label | Why | Alternative |\n|---|---|---|\n| `user_id`, `customer_id` (large tenant base) | Unbounded | Exemplars; aggregate by `tenant_tier` |\n| `request_id`, `trace_id` | Unbounded | Exemplars |\n| `path` \u002F `route` (raw URLs) | Unbounded | Template in code: `\u002Fusers\u002F:id` |\n| `error_message`, `query`, `sql` | Unbounded text | Bounded `error_type` enum |\n| `version`, `git_sha`, `image_tag` (on every metric) | Churn on every deploy | Info metric pattern |\n| App-emitted `pod` (duplicating SD) | Should come from K8s service discovery, not code | Stop emitting it in code; keep the discovered `pod`. Never drop the real `pod` to cut cardinality — use Adaptive Metrics |\n| `uid` (K8s) | Unbounded; regenerates on restart | Never map it into a label in the first place (leave it out of `relabel_configs`) |\n| Application-emitted `instance`, `node`, `host` | Should come from scrape target | Stop emitting in code (removing an *exact* target-label duplicate at scrape is the only safe drop) |\n| Dynamically-named label keys | Cannot be bounded | Use fixed keys with bounded values |\n| Raw `status_code` on histograms | 14× amplification | Bucket to `status_class` (`2xx`, `4xx`, `5xx`) |\n\n---\n\n## When to Route Elsewhere\n\n- **\"Reduce my Grafana Cloud bill\"** \u002F **\"reduce cardinality on series already ingested\"** → engage `adaptive-metrics` skill (post-ingest aggregation rules — the safe, counter-reset-aware way; never `labeldrop` distinguishing labels at scrape)\n- **\"Which metrics are driving my DPM?\"** → engage `dpm-finder` skill\n- **\"My Prometheus is OOMing \u002F scraping is failing right now\"** → engage `prometheus-cardinality-troubleshooter` skill\n- **\"How do I write the query to find the bad metric?\"** → engage `promql` skill\n- **\"How do I configure relabel rules in Alloy?\"** → engage `alloy` skill\n\nThis skill's lane is **strategy and design**. Other skills own **diagnosis** and **operational remediation**.\n",{"data":35,"body":36},{"name":4,"license":26,"description":6},{"type":37,"children":38},"root",[39,48,54,108,112,119,169,181,261,266,274,319,356,359,365,375,430,440,479,496,506,509,515,520,527,928,934,939,957,963,1055,1061,1189,1195,1238,1305,1311,1342,1354,1367,1414,1426,1440,1468,1506,1582,1608,1611,1617,1622,1631,1634,1640,1659,1872,1899,1902,1908,1914,2048,2061,2104,2109,2222,2228,2272,2278,2369,2375,2414,2417,2423,2434,2440,2445,2490,2495,2508,2520,2569,2889,2895,2912,2929,2960,2979,2992,3008,3098,3103,3225,3237,3243,3248,3363,3387,3390,3396,3408,3641,3647,3666,3671,3750,3755,3804,3809,3837,3842,3845,3851,3856,4022,4027,4030,4036,4408,4411,4417,4513,4537],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"prometheus-label-strategy-evaluator",[45],{"type":46,"value":47},"text","Prometheus Label Strategy Evaluator",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52],{"type":46,"value":53},"You are an expert in Prometheus label strategy. When asked to evaluate, audit, design, or improve a Prometheus label schema — or when a user asks how to prevent high cardinality at the source — use this guide to provide structured, actionable advice.",{"type":40,"tag":49,"props":55,"children":56},{},[57,59,65,67,73,75,80,82,89,91,98,100,106],{"type":46,"value":58},"This skill is about ",{"type":40,"tag":60,"props":61,"children":62},"strong",{},[63],{"type":46,"value":64},"preventing bad labels at the source",{"type":46,"value":66}," — in application instrumentation and in scrape ",{"type":40,"tag":68,"props":69,"children":70},"em",{},[71],{"type":46,"value":72},"target",{"type":46,"value":74}," labels — so they never enter storage. It is ",{"type":40,"tag":60,"props":76,"children":77},{},[78],{"type":46,"value":79},"not",{"type":46,"value":81}," about stripping labels off metrics after they've been emitted: removing a label that makes a series unique at scrape time silently breaks the data (see ",{"type":40,"tag":83,"props":84,"children":86},"a",{"href":85},"#the-one-rule-never-drop-a-label-that-makes-a-series-unique",[87],{"type":46,"value":88},"The One Rule",{"type":46,"value":90}," below). For reducing the cost of series that already exist in Grafana Cloud, route the user to the ",{"type":40,"tag":92,"props":93,"children":95},"code",{"className":94},[],[96],{"type":46,"value":97},"adaptive-metrics",{"type":46,"value":99}," skill. For diagnosing an active cardinality fire, route to ",{"type":40,"tag":92,"props":101,"children":103},{"className":102},[],[104],{"type":46,"value":105},"prometheus-cardinality-troubleshooter",{"type":46,"value":107},".",{"type":40,"tag":109,"props":110,"children":111},"hr",{},[],{"type":40,"tag":113,"props":114,"children":116},"h2",{"id":115},"the-one-rule-never-drop-a-label-that-makes-a-series-unique",[117],{"type":46,"value":118},"The One Rule: Never Drop a Label That Makes a Series Unique",{"type":40,"tag":49,"props":120,"children":121},{},[122,127,129,135,137,143,145,151,153,159,161,167],{"type":40,"tag":60,"props":123,"children":124},{},[125],{"type":46,"value":126},"You cannot remove, at scrape time, any label that makes a series unique.",{"type":46,"value":128}," Not ",{"type":40,"tag":92,"props":130,"children":132},{"className":131},[],[133],{"type":46,"value":134},"pod",{"type":46,"value":136},", not ",{"type":40,"tag":92,"props":138,"children":140},{"className":139},[],[141],{"type":46,"value":142},"instance",{"type":46,"value":144},", not anything that distinguishes one real series from another. This includes ",{"type":40,"tag":92,"props":146,"children":148},{"className":147},[],[149],{"type":46,"value":150},"metric_relabel_configs",{"type":46,"value":152}," with ",{"type":40,"tag":92,"props":154,"children":156},{"className":155},[],[157],{"type":46,"value":158},"action: labeldrop",{"type":46,"value":160}," and the equivalent ",{"type":40,"tag":92,"props":162,"children":164},{"className":163},[],[165],{"type":46,"value":166},"prometheus.relabel",{"type":46,"value":168}," rules in Alloy.",{"type":40,"tag":49,"props":170,"children":171},{},[172,174,179],{"type":46,"value":173},"It looks like a cardinality win. It is not — it ",{"type":40,"tag":60,"props":175,"children":176},{},[177],{"type":46,"value":178},"breaks the data",{"type":46,"value":180},", silently and permanently:",{"type":40,"tag":182,"props":183,"children":184},"ul",{},[185,219,243],{"type":40,"tag":186,"props":187,"children":188},"li",{},[189,194,196,202,204,210,212,217],{"type":40,"tag":60,"props":190,"children":191},{},[192],{"type":46,"value":193},"Counter resets get mixed together.",{"type":46,"value":195}," When two pods' counters collapse into one series, their independent restarts interleave on the merged series. ",{"type":40,"tag":92,"props":197,"children":199},{"className":198},[],[200],{"type":46,"value":201},"rate()",{"type":46,"value":203}," and ",{"type":40,"tag":92,"props":205,"children":207},{"className":206},[],[208],{"type":46,"value":209},"increase()",{"type":46,"value":211}," then return garbage — often ",{"type":40,"tag":68,"props":213,"children":214},{},[215],{"type":46,"value":216},"absurdly high",{"type":46,"value":218}," values, because every pod restart looks like a counter reset.",{"type":40,"tag":186,"props":220,"children":221},{},[222,227,229,234,236,241],{"type":40,"tag":60,"props":223,"children":224},{},[225],{"type":46,"value":226},"DPM inflates instead of dropping.",{"type":46,"value":228}," Multiple samples now land on the same series in the same scrape — duplicate samples, out-of-order errors, inflated samples-per-minute. People come back weeks later asking \"why is my DPM so high?\" or \"why is ",{"type":40,"tag":92,"props":230,"children":232},{"className":231},[],[233],{"type":46,"value":201},{"type":46,"value":235}," returning absurd numbers?\" — and there is ",{"type":40,"tag":60,"props":237,"children":238},{},[239],{"type":46,"value":240},"no evidence left in the data",{"type":46,"value":242}," of where it broke.",{"type":40,"tag":186,"props":244,"children":245},{},[246,251,253,259],{"type":40,"tag":60,"props":247,"children":248},{},[249],{"type":46,"value":250},"The aggregation is wrong, not just coarse.",{"type":46,"value":252}," A ",{"type":40,"tag":92,"props":254,"children":256},{"className":255},[],[257],{"type":46,"value":258},"sum",{"type":46,"value":260}," over a label you dropped silently double-counts or under-counts depending on how the collapse happened.",{"type":40,"tag":49,"props":262,"children":263},{},[264],{"type":46,"value":265},"The trap is that none of this errors at config time. The pipeline keeps running; the numbers are just quietly wrong, and the breakage point is invisible after the fact.",{"type":40,"tag":49,"props":267,"children":268},{},[269],{"type":40,"tag":60,"props":270,"children":271},{},[272],{"type":46,"value":273},"The right tools, in order:",{"type":40,"tag":275,"props":276,"children":277},"ol",{},[278,295],{"type":40,"tag":186,"props":279,"children":280},{},[281,286,288,293],{"type":40,"tag":60,"props":282,"children":283},{},[284],{"type":46,"value":285},"Don't emit the bad label in the first place",{"type":46,"value":287}," — fix the application code. This is the only place a label can be ",{"type":40,"tag":68,"props":289,"children":290},{},[291],{"type":46,"value":292},"removed",{"type":46,"value":294}," without consequence, because the series was never unique on it to begin with.",{"type":40,"tag":186,"props":296,"children":297},{},[298,303,305,310,312,317],{"type":40,"tag":60,"props":299,"children":300},{},[301],{"type":46,"value":302},"For series already flowing into Grafana Cloud that you can't fix at the source → Adaptive Metrics.",{"type":46,"value":304}," This is exactly what it is for: it aggregates series ",{"type":40,"tag":68,"props":306,"children":307},{},[308],{"type":46,"value":309},"correctly",{"type":46,"value":311}," — counter-reset-aware, with a recorded audit trail, and reversible — instead of blindly stripping labels. Route the user to the ",{"type":40,"tag":92,"props":313,"children":315},{"className":314},[],[316],{"type":46,"value":97},{"type":46,"value":318}," skill.",{"type":40,"tag":49,"props":320,"children":321},{},[322,327,329,334,336,341,343,349,351],{"type":40,"tag":92,"props":323,"children":325},{"className":324},[],[326],{"type":46,"value":150},{"type":46,"value":328}," has a couple of narrow, safe uses (dropping an ",{"type":40,"tag":68,"props":330,"children":331},{},[332],{"type":46,"value":333},"entire",{"type":46,"value":335}," unwanted metric; removing a label that ",{"type":40,"tag":68,"props":337,"children":338},{},[339],{"type":46,"value":340},"exactly duplicates",{"type":46,"value":342}," a target label) — covered in ",{"type":40,"tag":83,"props":344,"children":346},{"href":345},"#4-metric_relabel_configs-narrow-safe-uses-only",[347],{"type":46,"value":348},"Source-Side Prevention",{"type":46,"value":350}," — but ",{"type":40,"tag":60,"props":352,"children":353},{},[354],{"type":46,"value":355},"reducing cardinality by dropping a distinguishing label is never one of them.",{"type":40,"tag":109,"props":357,"children":358},{},[],{"type":40,"tag":113,"props":360,"children":362},{"id":361},"core-concepts",[363],{"type":46,"value":364},"Core Concepts",{"type":40,"tag":49,"props":366,"children":367},{},[368,373],{"type":40,"tag":60,"props":369,"children":370},{},[371],{"type":46,"value":372},"Series",{"type":46,"value":374}," are the fundamental unit in Prometheus. Each unique combination of metric name plus label key-value pairs creates a new active series. Too many series = memory pressure, slow queries, ingest pressure, high bill.",{"type":40,"tag":49,"props":376,"children":377},{},[378,383,385,390,392,398,400,406,408,414,416,421,423,428],{"type":40,"tag":60,"props":379,"children":380},{},[381],{"type":46,"value":382},"Cardinality",{"type":46,"value":384}," = the number of unique values a label can have. Total series for a metric ≈ the ",{"type":40,"tag":68,"props":386,"children":387},{},[388],{"type":46,"value":389},"product",{"type":46,"value":391}," of cardinalities across its labels. A metric with ",{"type":40,"tag":92,"props":393,"children":395},{"className":394},[],[396],{"type":46,"value":397},"path",{"type":46,"value":399}," (100 values), ",{"type":40,"tag":92,"props":401,"children":403},{"className":402},[],[404],{"type":46,"value":405},"status_code",{"type":46,"value":407}," (10 values), ",{"type":40,"tag":92,"props":409,"children":411},{"className":410},[],[412],{"type":46,"value":413},"method",{"type":46,"value":415}," (5 values), and ",{"type":40,"tag":92,"props":417,"children":419},{"className":418},[],[420],{"type":46,"value":142},{"type":46,"value":422}," (50 values) = ",{"type":40,"tag":60,"props":424,"children":425},{},[426],{"type":46,"value":427},"250,000 series per metric",{"type":46,"value":429},". Adding one more high-cardinality label often 10–100×s the count.",{"type":40,"tag":49,"props":431,"children":432},{},[433,438],{"type":40,"tag":60,"props":434,"children":435},{},[436],{"type":46,"value":437},"The dual impact rule",{"type":46,"value":439},": High-cardinality labels hurt on both paths:",{"type":40,"tag":182,"props":441,"children":442},{},[443,453],{"type":40,"tag":186,"props":444,"children":445},{},[446,451],{"type":40,"tag":60,"props":447,"children":448},{},[449],{"type":46,"value":450},"Ingestion path",{"type":46,"value":452},": More active series → larger head block, larger WAL, more memory, larger remote_write payloads, higher Grafana Cloud bill (Active Series + DPM)",{"type":40,"tag":186,"props":454,"children":455},{},[456,461,463,469,471,477],{"type":40,"tag":60,"props":457,"children":458},{},[459],{"type":46,"value":460},"Query path",{"type":46,"value":462},": PromQL operators (",{"type":40,"tag":92,"props":464,"children":466},{"className":465},[],[467],{"type":46,"value":468},"sum by",{"type":46,"value":470},", ",{"type":40,"tag":92,"props":472,"children":474},{"className":473},[],[475],{"type":46,"value":476},"rate",{"type":46,"value":478},", joins) must materialize matching series in memory. High cardinality balloons query memory and latency",{"type":40,"tag":49,"props":480,"children":481},{},[482,487,489,494],{"type":40,"tag":60,"props":483,"children":484},{},[485],{"type":46,"value":486},"Series churn",{"type":46,"value":488}," is the silent killer. If a label value changes frequently (deploy version, pod name, ephemeral IDs), every change creates a ",{"type":40,"tag":68,"props":490,"children":491},{},[492],{"type":46,"value":493},"new",{"type":46,"value":495}," series while the old one continues to age out. Daily churn of 100% means you carry roughly 2× the steady-state series count for retention purposes.",{"type":40,"tag":49,"props":497,"children":498},{},[499,504],{"type":40,"tag":60,"props":500,"children":501},{},[502],{"type":46,"value":503},"The key question for any proposed label",{"type":46,"value":505},": \"Will queries that use this metric reliably specify or aggregate on this label?\" If no → it should NOT be a label.",{"type":40,"tag":109,"props":507,"children":508},{},[],{"type":40,"tag":113,"props":510,"children":512},{"id":511},"label-evaluation-framework",[513],{"type":46,"value":514},"Label Evaluation Framework",{"type":40,"tag":49,"props":516,"children":517},{},[518],{"type":46,"value":519},"When auditing a label set, assess each label against these criteria.",{"type":40,"tag":521,"props":522,"children":524},"h3",{"id":523},"cardinality-scoring",[525],{"type":46,"value":526},"Cardinality Scoring",{"type":40,"tag":528,"props":529,"children":530},"table",{},[531,554],{"type":40,"tag":532,"props":533,"children":534},"thead",{},[535],{"type":40,"tag":536,"props":537,"children":538},"tr",{},[539,545,549],{"type":40,"tag":540,"props":541,"children":542},"th",{},[543],{"type":46,"value":544},"Label Example",{"type":40,"tag":540,"props":546,"children":547},{},[548],{"type":46,"value":382},{"type":40,"tag":540,"props":550,"children":551},{},[552],{"type":46,"value":553},"Verdict",{"type":40,"tag":555,"props":556,"children":557},"tbody",{},[558,583,606,634,658,693,716,752,791,827,863,892],{"type":40,"tag":536,"props":559,"children":560},{},[561,573,578],{"type":40,"tag":562,"props":563,"children":564},"td",{},[565,571],{"type":40,"tag":92,"props":566,"children":568},{"className":567},[],[569],{"type":46,"value":570},"env",{"type":46,"value":572}," (prod\u002Fstaging\u002Fdev)",{"type":40,"tag":562,"props":574,"children":575},{},[576],{"type":46,"value":577},"2–5 values",{"type":40,"tag":562,"props":579,"children":580},{},[581],{"type":46,"value":582},"✅ Good",{"type":40,"tag":536,"props":584,"children":585},{},[586,597,602],{"type":40,"tag":562,"props":587,"children":588},{},[589,595],{"type":40,"tag":92,"props":590,"children":592},{"className":591},[],[593],{"type":46,"value":594},"job",{"type":46,"value":596}," (Prometheus scrape job)",{"type":40,"tag":562,"props":598,"children":599},{},[600],{"type":46,"value":601},"5–50 values",{"type":40,"tag":562,"props":603,"children":604},{},[605],{"type":46,"value":582},{"type":40,"tag":536,"props":607,"children":608},{},[609,625,630],{"type":40,"tag":562,"props":610,"children":611},{},[612,618,619],{"type":40,"tag":92,"props":613,"children":615},{"className":614},[],[616],{"type":46,"value":617},"cluster",{"type":46,"value":470},{"type":40,"tag":92,"props":620,"children":622},{"className":621},[],[623],{"type":46,"value":624},"region",{"type":40,"tag":562,"props":626,"children":627},{},[628],{"type":46,"value":629},"Tens",{"type":40,"tag":562,"props":631,"children":632},{},[633],{"type":46,"value":582},{"type":40,"tag":536,"props":635,"children":636},{},[637,648,653],{"type":40,"tag":562,"props":638,"children":639},{},[640,646],{"type":40,"tag":92,"props":641,"children":643},{"className":642},[],[644],{"type":46,"value":645},"namespace",{"type":46,"value":647}," (K8s)",{"type":40,"tag":562,"props":649,"children":650},{},[651],{"type":46,"value":652},"Tens–low hundreds",{"type":40,"tag":562,"props":654,"children":655},{},[656],{"type":46,"value":657},"✅ Acceptable",{"type":40,"tag":536,"props":659,"children":660},{},[661,684,689],{"type":40,"tag":562,"props":662,"children":663},{},[664,670,671,677,678],{"type":40,"tag":92,"props":665,"children":667},{"className":666},[],[668],{"type":46,"value":669},"service",{"type":46,"value":470},{"type":40,"tag":92,"props":672,"children":674},{"className":673},[],[675],{"type":46,"value":676},"workload",{"type":46,"value":470},{"type":40,"tag":92,"props":679,"children":681},{"className":680},[],[682],{"type":46,"value":683},"container",{"type":40,"tag":562,"props":685,"children":686},{},[687],{"type":46,"value":688},"Tens–hundreds",{"type":40,"tag":562,"props":690,"children":691},{},[692],{"type":46,"value":657},{"type":40,"tag":536,"props":694,"children":695},{},[696,706,711],{"type":40,"tag":562,"props":697,"children":698},{},[699,704],{"type":40,"tag":92,"props":700,"children":702},{"className":701},[],[703],{"type":46,"value":142},{"type":46,"value":705}," (host:port)",{"type":40,"tag":562,"props":707,"children":708},{},[709],{"type":46,"value":710},"Hundreds–low thousands",{"type":40,"tag":562,"props":712,"children":713},{},[714],{"type":46,"value":715},"⚠️ Evaluate — fine on per-instance metrics, risky on aggregated ones",{"type":40,"tag":536,"props":717,"children":718},{},[719,728,733],{"type":40,"tag":562,"props":720,"children":721},{},[722,727],{"type":40,"tag":92,"props":723,"children":725},{"className":724},[],[726],{"type":46,"value":134},{"type":46,"value":647},{"type":40,"tag":562,"props":729,"children":730},{},[731],{"type":46,"value":732},"Thousands + transient = high churn",{"type":40,"tag":562,"props":734,"children":735},{},[736,738,743,745,750],{"type":46,"value":737},"⚠️ Required for K8s monitoring and series uniqueness — keep it. If ",{"type":40,"tag":92,"props":739,"children":741},{"className":740},[],[742],{"type":46,"value":134},{"type":46,"value":744},"-level series are too expensive, reduce them with Adaptive Metrics; ",{"type":40,"tag":60,"props":746,"children":747},{},[748],{"type":46,"value":749},"never",{"type":46,"value":751}," drop at scrape",{"type":40,"tag":536,"props":753,"children":754},{},[755,773,778],{"type":40,"tag":562,"props":756,"children":757},{},[758,763,765,771],{"type":40,"tag":92,"props":759,"children":761},{"className":760},[],[762],{"type":46,"value":397},{"type":46,"value":764}," \u002F ",{"type":40,"tag":92,"props":766,"children":768},{"className":767},[],[769],{"type":46,"value":770},"route",{"type":46,"value":772}," (HTTP)",{"type":40,"tag":562,"props":774,"children":775},{},[776],{"type":46,"value":777},"Bounded if templated; unbounded if raw URLs",{"type":40,"tag":562,"props":779,"children":780},{},[781,783,789],{"type":46,"value":782},"⚠️ Only with templated values (",{"type":40,"tag":92,"props":784,"children":786},{"className":785},[],[787],{"type":46,"value":788},"\u002Fusers\u002F:id",{"type":46,"value":790},")",{"type":40,"tag":536,"props":792,"children":793},{},[794,817,822],{"type":40,"tag":562,"props":795,"children":796},{},[797,803,804,810,811],{"type":40,"tag":92,"props":798,"children":800},{"className":799},[],[801],{"type":46,"value":802},"version",{"type":46,"value":470},{"type":40,"tag":92,"props":805,"children":807},{"className":806},[],[808],{"type":46,"value":809},"image_tag",{"type":46,"value":470},{"type":40,"tag":92,"props":812,"children":814},{"className":813},[],[815],{"type":46,"value":816},"git_sha",{"type":40,"tag":562,"props":818,"children":819},{},[820],{"type":46,"value":821},"Grows on every deploy → churn",{"type":40,"tag":562,"props":823,"children":824},{},[825],{"type":46,"value":826},"⚠️ Use sparingly; consider info-metric pattern",{"type":40,"tag":536,"props":828,"children":829},{},[830,853,858],{"type":40,"tag":562,"props":831,"children":832},{},[833,839,840,846,847],{"type":40,"tag":92,"props":834,"children":836},{"className":835},[],[837],{"type":46,"value":838},"user_id",{"type":46,"value":470},{"type":40,"tag":92,"props":841,"children":843},{"className":842},[],[844],{"type":46,"value":845},"request_id",{"type":46,"value":470},{"type":40,"tag":92,"props":848,"children":850},{"className":849},[],[851],{"type":46,"value":852},"trace_id",{"type":40,"tag":562,"props":854,"children":855},{},[856],{"type":46,"value":857},"Unbounded",{"type":40,"tag":562,"props":859,"children":860},{},[861],{"type":46,"value":862},"❌ Never as label — use exemplars",{"type":40,"tag":536,"props":864,"children":865},{},[866,882,887],{"type":40,"tag":562,"props":867,"children":868},{},[869,875,876],{"type":40,"tag":92,"props":870,"children":872},{"className":871},[],[873],{"type":46,"value":874},"customer_id",{"type":46,"value":470},{"type":40,"tag":92,"props":877,"children":879},{"className":878},[],[880],{"type":46,"value":881},"tenant_id",{"type":40,"tag":562,"props":883,"children":884},{},[885],{"type":46,"value":886},"Often unbounded",{"type":40,"tag":562,"props":888,"children":889},{},[890],{"type":46,"value":891},"❌ Only acceptable for small fixed tenant counts",{"type":40,"tag":536,"props":893,"children":894},{},[895,918,923],{"type":40,"tag":562,"props":896,"children":897},{},[898,904,905,911,912],{"type":40,"tag":92,"props":899,"children":901},{"className":900},[],[902],{"type":46,"value":903},"error_message",{"type":46,"value":470},{"type":40,"tag":92,"props":906,"children":908},{"className":907},[],[909],{"type":46,"value":910},"query",{"type":46,"value":470},{"type":40,"tag":92,"props":913,"children":915},{"className":914},[],[916],{"type":46,"value":917},"sql",{"type":40,"tag":562,"props":919,"children":920},{},[921],{"type":46,"value":922},"Unbounded text",{"type":40,"tag":562,"props":924,"children":925},{},[926],{"type":46,"value":927},"❌ Never",{"type":40,"tag":521,"props":929,"children":931},{"id":930},"access-pattern-alignment",[932],{"type":46,"value":933},"Access Pattern Alignment",{"type":40,"tag":49,"props":935,"children":936},{},[937],{"type":46,"value":938},"For each label, ask:",{"type":40,"tag":182,"props":940,"children":941},{},[942,947,952],{"type":40,"tag":186,"props":943,"children":944},{},[945],{"type":46,"value":946},"Do queries on this metric reliably aggregate by or filter on this label?",{"type":40,"tag":186,"props":948,"children":949},{},[950],{"type":46,"value":951},"Does this label logically segment the metric the way users think about it?",{"type":40,"tag":186,"props":953,"children":954},{},[955],{"type":46,"value":956},"Would removing this label force users to use exemplars, logs, or traces instead — and would that be acceptable for the rare lookup case?",{"type":40,"tag":521,"props":958,"children":960},{"id":959},"static-vs-dynamic-label-values",[961],{"type":46,"value":962},"Static vs. Dynamic Label Values",{"type":40,"tag":182,"props":964,"children":965},{},[966,1013],{"type":40,"tag":186,"props":967,"children":968},{},[969,974,976,982,984,990,991,997,998,1004,1006,1011],{"type":40,"tag":60,"props":970,"children":971},{},[972],{"type":46,"value":973},"Static \u002F target labels",{"type":46,"value":975}," (set once per scrape target via ",{"type":40,"tag":92,"props":977,"children":979},{"className":978},[],[980],{"type":46,"value":981},"relabel_configs",{"type":46,"value":983},", e.g., ",{"type":40,"tag":92,"props":985,"children":987},{"className":986},[],[988],{"type":46,"value":989},"env=prod",{"type":46,"value":470},{"type":40,"tag":92,"props":992,"children":994},{"className":993},[],[995],{"type":46,"value":996},"cluster=us-east",{"type":46,"value":470},{"type":40,"tag":92,"props":999,"children":1001},{"className":1000},[],[1002],{"type":46,"value":1003},"team=payments",{"type":46,"value":1005},") add cardinality proportional to ",{"type":40,"tag":68,"props":1007,"children":1008},{},[1009],{"type":46,"value":1010},"targets",{"type":46,"value":1012},", not requests. Cheap and high-value. Use freely.",{"type":40,"tag":186,"props":1014,"children":1015},{},[1016,1021,1023,1028,1029,1034,1035,1041,1043,1048,1050],{"type":40,"tag":60,"props":1017,"children":1018},{},[1019],{"type":46,"value":1020},"Dynamic \u002F sample labels",{"type":46,"value":1022}," (emitted by the application per measurement, e.g., ",{"type":40,"tag":92,"props":1024,"children":1026},{"className":1025},[],[1027],{"type":46,"value":405},{"type":46,"value":470},{"type":40,"tag":92,"props":1030,"children":1032},{"className":1031},[],[1033],{"type":46,"value":413},{"type":46,"value":470},{"type":40,"tag":92,"props":1036,"children":1038},{"className":1037},[],[1039],{"type":46,"value":1040},"cache_hit",{"type":46,"value":1042},") multiply cardinality by ",{"type":40,"tag":68,"props":1044,"children":1045},{},[1046],{"type":46,"value":1047},"value count",{"type":46,"value":1049},". Keep possible values in the single digits or low tens. ",{"type":40,"tag":60,"props":1051,"children":1052},{},[1053],{"type":46,"value":1054},"The application code is the source of truth — fix it there, not in Prometheus.",{"type":40,"tag":521,"props":1056,"children":1058},{"id":1057},"consistency-check",[1059],{"type":46,"value":1060},"Consistency Check",{"type":40,"tag":182,"props":1062,"children":1063},{},[1064,1098,1151,1164],{"type":40,"tag":186,"props":1065,"children":1066},{},[1067,1069,1074,1076,1082,1084,1089,1090,1096],{"type":46,"value":1068},"Label ",{"type":40,"tag":68,"props":1070,"children":1071},{},[1072],{"type":46,"value":1073},"names",{"type":46,"value":1075}," consistent across services? (",{"type":40,"tag":92,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":46,"value":1081},"status",{"type":46,"value":1083}," vs ",{"type":40,"tag":92,"props":1085,"children":1087},{"className":1086},[],[1088],{"type":46,"value":405},{"type":46,"value":1083},{"type":40,"tag":92,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":46,"value":1095},"http_status",{"type":46,"value":1097}," produces three separate label families — joins break)",{"type":40,"tag":186,"props":1099,"children":1100},{},[1101,1102,1107,1109,1115,1116,1122,1123,1129,1130,1136,1137,1143,1144,1150],{"type":46,"value":1068},{"type":40,"tag":68,"props":1103,"children":1104},{},[1105],{"type":46,"value":1106},"values",{"type":46,"value":1108}," normalized? (",{"type":40,"tag":92,"props":1110,"children":1112},{"className":1111},[],[1113],{"type":46,"value":1114},"200",{"type":46,"value":1083},{"type":40,"tag":92,"props":1117,"children":1119},{"className":1118},[],[1120],{"type":46,"value":1121},"\"200\"",{"type":46,"value":470},{"type":40,"tag":92,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":46,"value":1128},"GET",{"type":46,"value":1083},{"type":40,"tag":92,"props":1131,"children":1133},{"className":1132},[],[1134],{"type":46,"value":1135},"get",{"type":46,"value":470},{"type":40,"tag":92,"props":1138,"children":1140},{"className":1139},[],[1141],{"type":46,"value":1142},"Error",{"type":46,"value":1083},{"type":40,"tag":92,"props":1145,"children":1147},{"className":1146},[],[1148],{"type":46,"value":1149},"error",{"type":46,"value":790},{"type":40,"tag":186,"props":1152,"children":1153},{},[1154,1156,1162],{"type":46,"value":1155},"Naming convention consistent? Prometheus convention is ",{"type":40,"tag":92,"props":1157,"children":1159},{"className":1158},[],[1160],{"type":46,"value":1161},"snake_case",{"type":46,"value":1163}," for both metric and label names",{"type":40,"tag":186,"props":1165,"children":1166},{},[1167,1169,1174,1175,1181,1182,1188],{"type":46,"value":1168},"Same concept, same name across services? (",{"type":40,"tag":92,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":46,"value":669},{"type":46,"value":1083},{"type":40,"tag":92,"props":1176,"children":1178},{"className":1177},[],[1179],{"type":46,"value":1180},"svc",{"type":46,"value":1083},{"type":40,"tag":92,"props":1183,"children":1185},{"className":1184},[],[1186],{"type":46,"value":1187},"app_name",{"type":46,"value":790},{"type":40,"tag":521,"props":1190,"children":1192},{"id":1191},"histogram-bucket-discipline-critical-often-missed",[1193],{"type":46,"value":1194},"Histogram Bucket Discipline (critical, often missed)",{"type":40,"tag":49,"props":1196,"children":1197},{},[1198,1200,1205,1207,1213,1215,1221,1222,1228,1230,1236],{"type":46,"value":1199},"Every histogram metric multiplies its base cardinality by ",{"type":40,"tag":60,"props":1201,"children":1202},{},[1203],{"type":46,"value":1204},"(bucket count + 3)",{"type":46,"value":1206}," — buckets via ",{"type":40,"tag":92,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":46,"value":1212},"_bucket{le=\"...\"}",{"type":46,"value":1214}," plus ",{"type":40,"tag":92,"props":1216,"children":1218},{"className":1217},[],[1219],{"type":46,"value":1220},"_sum",{"type":46,"value":470},{"type":40,"tag":92,"props":1223,"children":1225},{"className":1224},[],[1226],{"type":46,"value":1227},"_count",{"type":46,"value":1229},", and ",{"type":40,"tag":92,"props":1231,"children":1233},{"className":1232},[],[1234],{"type":46,"value":1235},"_created",{"type":46,"value":1237}," (Prometheus 2.39+).",{"type":40,"tag":182,"props":1239,"children":1240},{},[1241,1259,1290,1300],{"type":40,"tag":186,"props":1242,"children":1243},{},[1244,1246,1252,1254],{"type":46,"value":1245},"Default ",{"type":40,"tag":92,"props":1247,"children":1249},{"className":1248},[],[1250],{"type":46,"value":1251},"prometheus.DefBuckets",{"type":46,"value":1253}," has 11 buckets → ",{"type":40,"tag":60,"props":1255,"children":1256},{},[1257],{"type":46,"value":1258},"14× multiplier",{"type":40,"tag":186,"props":1260,"children":1261},{},[1262,1264,1269,1270,1275,1276,1281,1283,1288],{"type":46,"value":1263},"A histogram with ",{"type":40,"tag":92,"props":1265,"children":1267},{"className":1266},[],[1268],{"type":46,"value":413},{"type":46,"value":470},{"type":40,"tag":92,"props":1271,"children":1273},{"className":1272},[],[1274],{"type":46,"value":397},{"type":46,"value":470},{"type":40,"tag":92,"props":1277,"children":1279},{"className":1278},[],[1280],{"type":46,"value":1081},{"type":46,"value":1282}," already at 1,000 series becomes ",{"type":40,"tag":60,"props":1284,"children":1285},{},[1286],{"type":46,"value":1287},"14,000 series",{"type":46,"value":1289}," after adding histogram cardinality",{"type":40,"tag":186,"props":1291,"children":1292},{},[1293,1298],{"type":40,"tag":60,"props":1294,"children":1295},{},[1296],{"type":46,"value":1297},"Always trim histogram label cardinality first",{"type":46,"value":1299}," — labels matter 14× more on histograms than on counters\u002Fgauges",{"type":40,"tag":186,"props":1301,"children":1302},{},[1303],{"type":46,"value":1304},"Consider native histograms (Prometheus 2.40+) which use a single sparse series instead of one-per-bucket — major cardinality reduction for high-resolution latency tracking",{"type":40,"tag":521,"props":1306,"children":1308},{"id":1307},"info-metric-pattern-for-high-churn-metadata",[1309],{"type":46,"value":1310},"Info-Metric Pattern (for high-churn metadata)",{"type":40,"tag":49,"props":1312,"children":1313},{},[1314,1316,1321,1323,1328,1329,1334,1335,1340],{"type":46,"value":1315},"When you want to ",{"type":40,"tag":68,"props":1317,"children":1318},{},[1319],{"type":46,"value":1320},"know",{"type":46,"value":1322}," about a label (e.g., ",{"type":40,"tag":92,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":46,"value":802},{"type":46,"value":470},{"type":40,"tag":92,"props":1330,"children":1332},{"className":1331},[],[1333],{"type":46,"value":816},{"type":46,"value":470},{"type":40,"tag":92,"props":1336,"children":1338},{"className":1337},[],[1339],{"type":46,"value":809},{"type":46,"value":1341},") without paying for it on every metric, use an info metric:",{"type":40,"tag":1343,"props":1344,"children":1348},"pre",{"className":1345,"code":1347,"language":46},[1346],"language-text","# A single low-cardinality counter\u002Fgauge of value 1, with the metadata attached\napp_build_info{app=\"payment-api\", version=\"2.4.1\", git_sha=\"a1b2c3\"} 1\n",[1349],{"type":40,"tag":92,"props":1350,"children":1352},{"__ignoreMap":1351},"",[1353],{"type":46,"value":1347},{"type":40,"tag":49,"props":1355,"children":1356},{},[1357,1359,1365],{"type":46,"value":1358},"Then join at query time. The classic approach is a vector match with ",{"type":40,"tag":92,"props":1360,"children":1362},{"className":1361},[],[1363],{"type":46,"value":1364},"group_left",{"type":46,"value":1366},":",{"type":40,"tag":1343,"props":1368,"children":1372},{"className":1369,"code":1370,"language":1371,"meta":1351,"style":1351},"language-promql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","sum by (version) (\n rate(http_requests_total{app=\"payment-api\"}[5m])\n * on (app) group_left (version) app_build_info\n)\n","promql",[1373],{"type":40,"tag":92,"props":1374,"children":1375},{"__ignoreMap":1351},[1376,1387,1396,1405],{"type":40,"tag":1377,"props":1378,"children":1381},"span",{"class":1379,"line":1380},"line",1,[1382],{"type":40,"tag":1377,"props":1383,"children":1384},{},[1385],{"type":46,"value":1386},"sum by (version) (\n",{"type":40,"tag":1377,"props":1388,"children":1390},{"class":1379,"line":1389},2,[1391],{"type":40,"tag":1377,"props":1392,"children":1393},{},[1394],{"type":46,"value":1395}," rate(http_requests_total{app=\"payment-api\"}[5m])\n",{"type":40,"tag":1377,"props":1397,"children":1399},{"class":1379,"line":1398},3,[1400],{"type":40,"tag":1377,"props":1401,"children":1402},{},[1403],{"type":46,"value":1404}," * on (app) group_left (version) app_build_info\n",{"type":40,"tag":1377,"props":1406,"children":1408},{"class":1379,"line":1407},4,[1409],{"type":40,"tag":1377,"props":1410,"children":1411},{},[1412],{"type":46,"value":1413},")\n",{"type":40,"tag":49,"props":1415,"children":1416},{},[1417,1419,1424],{"type":46,"value":1418},"The ",{"type":40,"tag":92,"props":1420,"children":1422},{"className":1421},[],[1423],{"type":46,"value":802},{"type":46,"value":1425}," label lives on exactly one series per build, not on every metric.",{"type":40,"tag":1427,"props":1428,"children":1430},"h4",{"id":1429},"the-info-function-simpler-join",[1431,1432,1438],{"type":46,"value":1418},{"type":40,"tag":92,"props":1433,"children":1435},{"className":1434},[],[1436],{"type":46,"value":1437},"info()",{"type":46,"value":1439}," function (simpler join)",{"type":40,"tag":49,"props":1441,"children":1442},{},[1443,1445,1450,1452,1458,1460,1466],{"type":46,"value":1444},"PromQL's ",{"type":40,"tag":92,"props":1446,"children":1448},{"className":1447},[],[1449],{"type":46,"value":1437},{"type":46,"value":1451}," function (experimental, Prometheus 3.0+; enable with ",{"type":40,"tag":92,"props":1453,"children":1455},{"className":1454},[],[1456],{"type":46,"value":1457},"--enable-feature=promql-experimental-functions",{"type":46,"value":1459},") automates the info-metric join so you don't have to hand-write the ",{"type":40,"tag":92,"props":1461,"children":1463},{"className":1462},[],[1464],{"type":46,"value":1465},"* on (...) group_left (...)",{"type":46,"value":1467}," match:",{"type":40,"tag":1343,"props":1469,"children":1471},{"className":1369,"code":1470,"language":1371,"meta":1351,"style":1351},"info(\n rate(http_requests_total{app=\"payment-api\"}[5m]),\n {version=~\".+\"}\n)\n",[1472],{"type":40,"tag":92,"props":1473,"children":1474},{"__ignoreMap":1351},[1475,1483,1491,1499],{"type":40,"tag":1377,"props":1476,"children":1477},{"class":1379,"line":1380},[1478],{"type":40,"tag":1377,"props":1479,"children":1480},{},[1481],{"type":46,"value":1482},"info(\n",{"type":40,"tag":1377,"props":1484,"children":1485},{"class":1379,"line":1389},[1486],{"type":40,"tag":1377,"props":1487,"children":1488},{},[1489],{"type":46,"value":1490}," rate(http_requests_total{app=\"payment-api\"}[5m]),\n",{"type":40,"tag":1377,"props":1492,"children":1493},{"class":1379,"line":1398},[1494],{"type":40,"tag":1377,"props":1495,"children":1496},{},[1497],{"type":46,"value":1498}," {version=~\".+\"}\n",{"type":40,"tag":1377,"props":1500,"children":1501},{"class":1379,"line":1407},[1502],{"type":40,"tag":1377,"props":1503,"children":1504},{},[1505],{"type":46,"value":1413},{"type":40,"tag":49,"props":1507,"children":1508},{},[1509,1515,1517,1523,1525,1530,1532,1537,1539,1545,1547,1552,1553,1558,1560,1565,1567,1573,1575,1580],{"type":40,"tag":92,"props":1510,"children":1512},{"className":1511},[],[1513],{"type":46,"value":1514},"info(v, [labelselector])",{"type":46,"value":1516}," takes a range\u002Finstant vector ",{"type":40,"tag":92,"props":1518,"children":1520},{"className":1519},[],[1521],{"type":46,"value":1522},"v",{"type":46,"value":1524}," and, for each series, finds matching info metrics and adds their labels. The optional second argument is a label-matcher restricting which info labels are attached (here, only ",{"type":40,"tag":92,"props":1526,"children":1528},{"className":1527},[],[1529],{"type":46,"value":802},{"type":46,"value":1531},"). By default ",{"type":40,"tag":92,"props":1533,"children":1535},{"className":1534},[],[1536],{"type":46,"value":1437},{"type":46,"value":1538}," joins against the conventional ",{"type":40,"tag":92,"props":1540,"children":1542},{"className":1541},[],[1543],{"type":46,"value":1544},"target_info",{"type":46,"value":1546}," metric and matches on identifying labels (e.g. ",{"type":40,"tag":92,"props":1548,"children":1550},{"className":1549},[],[1551],{"type":46,"value":142},{"type":46,"value":470},{"type":40,"tag":92,"props":1554,"children":1556},{"className":1555},[],[1557],{"type":46,"value":594},{"type":46,"value":1559},"), so it's especially ergonomic for OpenTelemetry-style ",{"type":40,"tag":92,"props":1561,"children":1563},{"className":1562},[],[1564],{"type":46,"value":1544},{"type":46,"value":1566},". For custom info metrics like ",{"type":40,"tag":92,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":46,"value":1572},"app_build_info",{"type":46,"value":1574}," the explicit ",{"type":40,"tag":92,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":46,"value":1364},{"type":46,"value":1581}," form above is still the most portable.",{"type":40,"tag":49,"props":1583,"children":1584},{},[1585,1587,1592,1594,1599,1601,1606],{"type":46,"value":1586},"Prefer ",{"type":40,"tag":92,"props":1588,"children":1590},{"className":1589},[],[1591],{"type":46,"value":1437},{"type":46,"value":1593}," when you're on Prometheus 3.x and joining against ",{"type":40,"tag":92,"props":1595,"children":1597},{"className":1596},[],[1598],{"type":46,"value":1544},{"type":46,"value":1600},"; fall back to the explicit ",{"type":40,"tag":92,"props":1602,"children":1604},{"className":1603},[],[1605],{"type":46,"value":1364},{"type":46,"value":1607}," match for older versions, custom info metrics, or when the experimental feature flag isn't enabled.",{"type":40,"tag":109,"props":1609,"children":1610},{},[],{"type":40,"tag":113,"props":1612,"children":1614},{"id":1613},"evaluation-output-format",[1615],{"type":46,"value":1616},"Evaluation Output Format",{"type":40,"tag":49,"props":1618,"children":1619},{},[1620],{"type":46,"value":1621},"When auditing a label set, produce a report in this structure:",{"type":40,"tag":1343,"props":1623,"children":1626},{"className":1624,"code":1625,"language":46},[1346],"## Prometheus Label Strategy Audit\n\n### Summary\n[1-2 sentence overall assessment — total estimated active series, biggest risks]\n\n### Per-Label Analysis\n| Metric Family | Label | Cardinality | Used in Queries? | Verdict | Action |\n|---|---|---|---|---|---|\n| http_requests_total | path | Unbounded (raw URLs) | Sometimes | ❌ Remove | Template in code: `\u002Fusers\u002F:id` not `\u002Fusers\u002F12345` |\n| http_requests_total | pod | High + churn | Rarely | ⚠️ Keep — makes the series unique | If too expensive, aggregate away with Adaptive Metrics; query by `workload` for the common case |\n\n### Histogram-Specific Findings\n[Highlight any histograms with high label cardinality — these are 14×+ amplified]\n\n### Estimated Impact\n- Active series reduction: [X series → Y series]\n- DPM reduction: [X DPM → Y DPM] (samples-per-minute = series × ~6 at 10s scrape)\n- Memory impact: [if measurable]\n\n### Recommended Label Set\n[Final recommended labels per metric family]\n\n### Implementation Plan\n1. [Code changes — instrumentation hygiene: stop emitting bad labels at the source]\n2. [Scrape target labels — relabel_configs (additive: env, cluster, team, workload)]\n3. [Post-ingest cost reduction on series you can't fix at the source — Adaptive Metrics]\n4. [Recording rules to materialize useful aggregates]\n",[1627],{"type":40,"tag":92,"props":1628,"children":1629},{"__ignoreMap":1351},[1630],{"type":46,"value":1625},{"type":40,"tag":109,"props":1632,"children":1633},{},[],{"type":40,"tag":113,"props":1635,"children":1637},{"id":1636},"recommended-common-target-labels",[1638],{"type":46,"value":1639},"Recommended Common Target Labels",{"type":40,"tag":49,"props":1641,"children":1642},{},[1643,1645,1650,1652,1657],{"type":46,"value":1644},"These should be set as ",{"type":40,"tag":60,"props":1646,"children":1647},{},[1648],{"type":46,"value":1649},"target labels",{"type":46,"value":1651}," (via ",{"type":40,"tag":92,"props":1653,"children":1655},{"className":1654},[],[1656],{"type":46,"value":981},{"type":46,"value":1658}," on the scrape job, NOT emitted by the app) — they're per-target, low cardinality, high query value:",{"type":40,"tag":528,"props":1660,"children":1661},{},[1662,1683],{"type":40,"tag":532,"props":1663,"children":1664},{},[1665],{"type":40,"tag":536,"props":1666,"children":1667},{},[1668,1673,1678],{"type":40,"tag":540,"props":1669,"children":1670},{},[1671],{"type":46,"value":1672},"Label",{"type":40,"tag":540,"props":1674,"children":1675},{},[1676],{"type":46,"value":1677},"Purpose",{"type":40,"tag":540,"props":1679,"children":1680},{},[1681],{"type":46,"value":1682},"Notes",{"type":40,"tag":555,"props":1684,"children":1685},{},[1686,1707,1742,1784,1805,1824,1851],{"type":40,"tag":536,"props":1687,"children":1688},{},[1689,1697,1702],{"type":40,"tag":562,"props":1690,"children":1691},{},[1692],{"type":40,"tag":92,"props":1693,"children":1695},{"className":1694},[],[1696],{"type":46,"value":594},{"type":40,"tag":562,"props":1698,"children":1699},{},[1700],{"type":46,"value":1701},"Prometheus scrape job name",{"type":40,"tag":562,"props":1703,"children":1704},{},[1705],{"type":46,"value":1706},"Set automatically by Prometheus",{"type":40,"tag":536,"props":1708,"children":1709},{},[1710,1718,1730],{"type":40,"tag":562,"props":1711,"children":1712},{},[1713],{"type":40,"tag":92,"props":1714,"children":1716},{"className":1715},[],[1717],{"type":46,"value":142},{"type":40,"tag":562,"props":1719,"children":1720},{},[1721,1723,1729],{"type":46,"value":1722},"Target endpoint (",{"type":40,"tag":92,"props":1724,"children":1726},{"className":1725},[],[1727],{"type":46,"value":1728},"host:port",{"type":46,"value":790},{"type":40,"tag":562,"props":1731,"children":1732},{},[1733,1735,1740],{"type":46,"value":1734},"Set automatically; rename via ",{"type":40,"tag":92,"props":1736,"children":1738},{"className":1737},[],[1739],{"type":46,"value":981},{"type":46,"value":1741}," to a friendlier value if needed",{"type":40,"tag":536,"props":1743,"children":1744},{},[1745,1753,1779],{"type":40,"tag":562,"props":1746,"children":1747},{},[1748],{"type":40,"tag":92,"props":1749,"children":1751},{"className":1750},[],[1752],{"type":46,"value":570},{"type":40,"tag":562,"props":1754,"children":1755},{},[1756,1758,1764,1765,1771,1772,1778],{"type":46,"value":1757},"Environment (",{"type":40,"tag":92,"props":1759,"children":1761},{"className":1760},[],[1762],{"type":46,"value":1763},"prod",{"type":46,"value":470},{"type":40,"tag":92,"props":1766,"children":1768},{"className":1767},[],[1769],{"type":46,"value":1770},"staging",{"type":46,"value":470},{"type":40,"tag":92,"props":1773,"children":1775},{"className":1774},[],[1776],{"type":46,"value":1777},"dev",{"type":46,"value":790},{"type":40,"tag":562,"props":1780,"children":1781},{},[1782],{"type":46,"value":1783},"Set via static_configs labels or service discovery",{"type":40,"tag":536,"props":1785,"children":1786},{},[1787,1795,1800],{"type":40,"tag":562,"props":1788,"children":1789},{},[1790],{"type":40,"tag":92,"props":1791,"children":1793},{"className":1792},[],[1794],{"type":46,"value":617},{"type":40,"tag":562,"props":1796,"children":1797},{},[1798],{"type":46,"value":1799},"Multi-cluster differentiation",{"type":40,"tag":562,"props":1801,"children":1802},{},[1803],{"type":46,"value":1804},"Critical for federation\u002FMimir multi-tenant",{"type":40,"tag":536,"props":1806,"children":1807},{},[1808,1816,1821],{"type":40,"tag":562,"props":1809,"children":1810},{},[1811],{"type":40,"tag":92,"props":1812,"children":1814},{"className":1813},[],[1815],{"type":46,"value":624},{"type":40,"tag":562,"props":1817,"children":1818},{},[1819],{"type":46,"value":1820},"Geographic region",{"type":40,"tag":562,"props":1822,"children":1823},{},[],{"type":40,"tag":536,"props":1825,"children":1826},{},[1827,1843,1848],{"type":40,"tag":562,"props":1828,"children":1829},{},[1830,1836,1837],{"type":40,"tag":92,"props":1831,"children":1833},{"className":1832},[],[1834],{"type":46,"value":1835},"team",{"type":46,"value":764},{"type":40,"tag":92,"props":1838,"children":1840},{"className":1839},[],[1841],{"type":46,"value":1842},"squad",{"type":40,"tag":562,"props":1844,"children":1845},{},[1846],{"type":46,"value":1847},"Ownership — also useful for access control",{"type":40,"tag":562,"props":1849,"children":1850},{},[],{"type":40,"tag":536,"props":1852,"children":1853},{},[1854,1862,1867],{"type":40,"tag":562,"props":1855,"children":1856},{},[1857],{"type":40,"tag":92,"props":1858,"children":1860},{"className":1859},[],[1861],{"type":46,"value":669},{"type":40,"tag":562,"props":1863,"children":1864},{},[1865],{"type":46,"value":1866},"Logical service identity",{"type":40,"tag":562,"props":1868,"children":1869},{},[1870],{"type":46,"value":1871},"One service may span multiple jobs",{"type":40,"tag":49,"props":1873,"children":1874},{},[1875,1877,1882,1884,1889,1891,1897],{"type":46,"value":1876},"These should ",{"type":40,"tag":60,"props":1878,"children":1879},{},[1880],{"type":46,"value":1881},"NOT",{"type":46,"value":1883}," be re-emitted by the application. If the app emits a ",{"type":40,"tag":92,"props":1885,"children":1887},{"className":1886},[],[1888],{"type":46,"value":617},{"type":46,"value":1890}," label, it duplicates the target label and creates collisions \u002F ",{"type":40,"tag":92,"props":1892,"children":1894},{"className":1893},[],[1895],{"type":46,"value":1896},"honor_labels",{"type":46,"value":1898}," decisions you don't want to make.",{"type":40,"tag":109,"props":1900,"children":1901},{},[],{"type":40,"tag":113,"props":1903,"children":1905},{"id":1904},"kubernetes-patterns",[1906],{"type":46,"value":1907},"Kubernetes Patterns",{"type":40,"tag":521,"props":1909,"children":1911},{"id":1910},"recommended-labels-from-kubernetes_sd_configs",[1912],{"type":46,"value":1913},"Recommended Labels (from kubernetes_sd_configs)",{"type":40,"tag":528,"props":1915,"children":1916},{},[1917,1936],{"type":40,"tag":532,"props":1918,"children":1919},{},[1920],{"type":40,"tag":536,"props":1921,"children":1922},{},[1923,1927,1932],{"type":40,"tag":540,"props":1924,"children":1925},{},[1926],{"type":46,"value":1672},{"type":40,"tag":540,"props":1928,"children":1929},{},[1930],{"type":46,"value":1931},"Source",{"type":40,"tag":540,"props":1933,"children":1934},{},[1935],{"type":46,"value":1682},{"type":40,"tag":555,"props":1937,"children":1938},{},[1939,1960,1981,2027],{"type":40,"tag":536,"props":1940,"children":1941},{},[1942,1950,1955],{"type":40,"tag":562,"props":1943,"children":1944},{},[1945],{"type":40,"tag":92,"props":1946,"children":1948},{"className":1947},[],[1949],{"type":46,"value":645},{"type":40,"tag":562,"props":1951,"children":1952},{},[1953],{"type":46,"value":1954},"Pod metadata",{"type":40,"tag":562,"props":1956,"children":1957},{},[1958],{"type":46,"value":1959},"Always keep",{"type":40,"tag":536,"props":1961,"children":1962},{},[1963,1971,1976],{"type":40,"tag":562,"props":1964,"children":1965},{},[1966],{"type":40,"tag":92,"props":1967,"children":1969},{"className":1968},[],[1970],{"type":46,"value":683},{"type":40,"tag":562,"props":1972,"children":1973},{},[1974],{"type":46,"value":1975},"Pod spec",{"type":40,"tag":562,"props":1977,"children":1978},{},[1979],{"type":46,"value":1980},"Low cardinality, useful for multi-container pods",{"type":40,"tag":536,"props":1982,"children":1983},{},[1984,1992,2003],{"type":40,"tag":562,"props":1985,"children":1986},{},[1987],{"type":40,"tag":92,"props":1988,"children":1990},{"className":1989},[],[1991],{"type":46,"value":676},{"type":40,"tag":562,"props":1993,"children":1994},{},[1995,1997],{"type":46,"value":1996},"Derived: ",{"type":40,"tag":92,"props":1998,"children":2000},{"className":1999},[],[2001],{"type":46,"value":2002},"{controller_kind}\u002F{controller_name}",{"type":40,"tag":562,"props":2004,"children":2005},{},[2006,2008,2013,2015,2020,2022],{"type":46,"value":2007},"Add as a stable aggregation key ",{"type":40,"tag":68,"props":2009,"children":2010},{},[2011],{"type":46,"value":2012},"alongside",{"type":46,"value":2014}," ",{"type":40,"tag":92,"props":2016,"children":2018},{"className":2017},[],[2019],{"type":46,"value":134},{"type":46,"value":2021}," — static, predictable. It's an addition, not a replacement: don't use it as an excuse to drop ",{"type":40,"tag":92,"props":2023,"children":2025},{"className":2024},[],[2026],{"type":46,"value":134},{"type":40,"tag":536,"props":2028,"children":2029},{},[2030,2038,2043],{"type":40,"tag":562,"props":2031,"children":2032},{},[2033],{"type":40,"tag":92,"props":2034,"children":2036},{"className":2035},[],[2037],{"type":46,"value":669},{"type":40,"tag":562,"props":2039,"children":2040},{},[2041],{"type":46,"value":2042},"K8s Service",{"type":40,"tag":562,"props":2044,"children":2045},{},[2046],{"type":46,"value":2047},"If scraping via Service",{"type":40,"tag":521,"props":2049,"children":2051},{"id":2050},"handling-the-pod-label",[2052,2054,2059],{"type":46,"value":2053},"Handling the ",{"type":40,"tag":92,"props":2055,"children":2057},{"className":2056},[],[2058],{"type":46,"value":134},{"type":46,"value":2060}," Label",{"type":40,"tag":49,"props":2062,"children":2063},{},[2064,2069,2071,2076,2078,2082,2084,2096,2098,2103],{"type":40,"tag":92,"props":2065,"children":2067},{"className":2066},[],[2068],{"type":46,"value":134},{"type":46,"value":2070}," is high-cardinality and transient — it rolls on every deploy and restart, so it dominates churn and series count. But it is also a label that ",{"type":40,"tag":60,"props":2072,"children":2073},{},[2074],{"type":46,"value":2075},"makes K8s series unique",{"type":46,"value":2077},", and Kubernetes monitoring (per-pod resource attribution, kube-state-metrics joins) depends on it. ",{"type":40,"tag":83,"props":2079,"children":2080},{"href":85},[2081],{"type":46,"value":88},{"type":46,"value":2083}," applies: ",{"type":40,"tag":60,"props":2085,"children":2086},{},[2087,2089,2094],{"type":46,"value":2088},"do not drop ",{"type":40,"tag":92,"props":2090,"children":2092},{"className":2091},[],[2093],{"type":46,"value":134},{"type":46,"value":2095}," at scrape time.",{"type":46,"value":2097}," Collapsing pods into one series mixes their counter resets and breaks ",{"type":40,"tag":92,"props":2099,"children":2101},{"className":2100},[],[2102],{"type":46,"value":201},{"type":46,"value":107},{"type":40,"tag":49,"props":2105,"children":2106},{},[2107],{"type":46,"value":2108},"Instead:",{"type":40,"tag":182,"props":2110,"children":2111},{},[2112,2162,2179],{"type":40,"tag":186,"props":2113,"children":2114},{},[2115,2125,2127,2132,2134,2138,2140,2145,2147,2153,2155,2160],{"type":40,"tag":60,"props":2116,"children":2117},{},[2118,2120],{"type":46,"value":2119},"Add ",{"type":40,"tag":92,"props":2121,"children":2123},{"className":2122},[],[2124],{"type":46,"value":676},{"type":46,"value":2126}," (",{"type":40,"tag":92,"props":2128,"children":2130},{"className":2129},[],[2131],{"type":46,"value":2002},{"type":46,"value":2133},") as a ",{"type":40,"tag":68,"props":2135,"children":2136},{},[2137],{"type":46,"value":72},{"type":46,"value":2139}," label via ",{"type":40,"tag":92,"props":2141,"children":2143},{"className":2142},[],[2144],{"type":46,"value":981},{"type":46,"value":2146},", so dashboards and alerts can aggregate on the stable workload identity (",{"type":40,"tag":92,"props":2148,"children":2150},{"className":2149},[],[2151],{"type":46,"value":2152},"sum by (workload)",{"type":46,"value":2154},") without touching ",{"type":40,"tag":92,"props":2156,"children":2158},{"className":2157},[],[2159],{"type":46,"value":134},{"type":46,"value":2161},". This is additive — it removes nothing.",{"type":40,"tag":186,"props":2163,"children":2164},{},[2165,2177],{"type":40,"tag":60,"props":2166,"children":2167},{},[2168,2170,2175],{"type":46,"value":2169},"Don't emit ",{"type":40,"tag":92,"props":2171,"children":2173},{"className":2172},[],[2174],{"type":46,"value":134},{"type":46,"value":2176}," from application code",{"type":46,"value":2178}," — let it come from Kubernetes service discovery, so there is exactly one source of truth (see below).",{"type":40,"tag":186,"props":2180,"children":2181},{},[2182,2194,2196,2201,2203,2208,2210,2214,2216,2221],{"type":40,"tag":60,"props":2183,"children":2184},{},[2185,2187,2192],{"type":46,"value":2186},"If ",{"type":40,"tag":92,"props":2188,"children":2190},{"className":2189},[],[2191],{"type":46,"value":134},{"type":46,"value":2193},"-level series are genuinely too expensive in Grafana Cloud",{"type":46,"value":2195},", reduce them with ",{"type":40,"tag":60,"props":2197,"children":2198},{},[2199],{"type":46,"value":2200},"Adaptive Metrics",{"type":46,"value":2202},", which aggregates ",{"type":40,"tag":92,"props":2204,"children":2206},{"className":2205},[],[2207],{"type":46,"value":134},{"type":46,"value":2209}," away ",{"type":40,"tag":68,"props":2211,"children":2212},{},[2213],{"type":46,"value":309},{"type":46,"value":2215}," (post-ingest, counter-reset-aware, reversible) rather than corrupting the raw data at scrape. Route to the ",{"type":40,"tag":92,"props":2217,"children":2219},{"className":2218},[],[2220],{"type":46,"value":97},{"type":46,"value":318},{"type":40,"tag":521,"props":2223,"children":2225},{"id":2224},"dont-map-ephemeral-fields-into-labels-in-the-first-place",[2226],{"type":46,"value":2227},"Don't Map Ephemeral Fields into Labels in the First Place",{"type":40,"tag":49,"props":2229,"children":2230},{},[2231,2240,2242,2247,2249,2254,2256,2262,2264,2270],{"type":40,"tag":60,"props":2232,"children":2233},{},[2234],{"type":40,"tag":92,"props":2235,"children":2237},{"className":2236},[],[2238],{"type":46,"value":2239},"uid",{"type":46,"value":2241}," regenerates on every pod recreation and has no legitimate query use. The fix is to ",{"type":40,"tag":60,"props":2243,"children":2244},{},[2245],{"type":46,"value":2246},"never map it into a label",{"type":46,"value":2248}," — leave it out of your ",{"type":40,"tag":92,"props":2250,"children":2252},{"className":2251},[],[2253],{"type":46,"value":981},{"type":46,"value":2255},". (It isn't in default ",{"type":40,"tag":92,"props":2257,"children":2259},{"className":2258},[],[2260],{"type":46,"value":2261},"kubernetes_sd_configs",{"type":46,"value":2263}," output unless you explicitly target it.) Don't try to ",{"type":40,"tag":92,"props":2265,"children":2267},{"className":2266},[],[2268],{"type":46,"value":2269},"labeldrop",{"type":46,"value":2271}," it after the fact — by then it's already distinguishing series, and removing it breaks the data exactly like dropping any other unique label.",{"type":40,"tag":521,"props":2273,"children":2275},{"id":2274},"one-source-of-truth-for-target-identity",[2276],{"type":46,"value":2277},"One Source of Truth for Target Identity",{"type":40,"tag":49,"props":2279,"children":2280},{},[2281,2286,2287,2292,2293,2299,2300,2306,2308,2313,2315,2320,2322,2327,2329,2334,2336,2341,2343,2348,2350,2355,2357,2361,2363,2367],{"type":40,"tag":92,"props":2282,"children":2284},{"className":2283},[],[2285],{"type":46,"value":142},{"type":46,"value":470},{"type":40,"tag":92,"props":2288,"children":2290},{"className":2289},[],[2291],{"type":46,"value":134},{"type":46,"value":470},{"type":40,"tag":92,"props":2294,"children":2296},{"className":2295},[],[2297],{"type":46,"value":2298},"node",{"type":46,"value":1229},{"type":40,"tag":92,"props":2301,"children":2303},{"className":2302},[],[2304],{"type":46,"value":2305},"host",{"type":46,"value":2307}," should come from ",{"type":40,"tag":60,"props":2309,"children":2310},{},[2311],{"type":46,"value":2312},"scrape target labels",{"type":46,"value":2314},", not from application code. If the app ",{"type":40,"tag":68,"props":2316,"children":2317},{},[2318],{"type":46,"value":2319},"also",{"type":46,"value":2321}," emits its own ",{"type":40,"tag":92,"props":2323,"children":2325},{"className":2324},[],[2326],{"type":46,"value":142},{"type":46,"value":2328},"\u002F",{"type":40,"tag":92,"props":2330,"children":2332},{"className":2331},[],[2333],{"type":46,"value":2298},{"type":46,"value":2335},", you get duplicates and ",{"type":40,"tag":92,"props":2337,"children":2339},{"className":2338},[],[2340],{"type":46,"value":1896},{"type":46,"value":2342}," collisions. The fix is ",{"type":40,"tag":60,"props":2344,"children":2345},{},[2346],{"type":46,"value":2347},"in the application",{"type":46,"value":2349}," — stop emitting them — not a scrape-time ",{"type":40,"tag":92,"props":2351,"children":2353},{"className":2352},[],[2354],{"type":46,"value":2269},{"type":46,"value":2356},". (Removing a label that ",{"type":40,"tag":68,"props":2358,"children":2359},{},[2360],{"type":46,"value":340},{"type":46,"value":2362}," a target label is the one narrow exception; see ",{"type":40,"tag":83,"props":2364,"children":2365},{"href":345},[2366],{"type":46,"value":150},{"type":46,"value":2368},".)",{"type":40,"tag":521,"props":2370,"children":2372},{"id":2371},"kube-state-metrics-label-propagation-️",[2373],{"type":46,"value":2374},"kube-state-metrics label propagation ⚠️",{"type":40,"tag":182,"props":2376,"children":2377},{},[2378,2389,2394],{"type":40,"tag":186,"props":2379,"children":2380},{},[2381,2387],{"type":40,"tag":92,"props":2382,"children":2384},{"className":2383},[],[2385],{"type":46,"value":2386},"kube_pod_labels{label_app_kubernetes_io_*=...}",{"type":46,"value":2388}," can carry dozens of metadata labels",{"type":40,"tag":186,"props":2390,"children":2391},{},[2392],{"type":46,"value":2393},"Each unique pod label combination is a new series",{"type":40,"tag":186,"props":2395,"children":2396},{},[2397,2399,2405,2407,2412],{"type":46,"value":2398},"Restrict at the source with kube-state-metrics' ",{"type":40,"tag":92,"props":2400,"children":2402},{"className":2401},[],[2403],{"type":46,"value":2404},"--metric-labels-allowlist",{"type":46,"value":2406}," — this controls what is ",{"type":40,"tag":68,"props":2408,"children":2409},{},[2410],{"type":46,"value":2411},"ever emitted",{"type":46,"value":2413},", so it's prevention, not destructive after-the-fact dropping",{"type":40,"tag":109,"props":2415,"children":2416},{},[],{"type":40,"tag":113,"props":2418,"children":2420},{"id":2419},"source-side-prevention-where-to-fix-what",[2421],{"type":46,"value":2422},"Source-Side Prevention: Where to Fix What",{"type":40,"tag":49,"props":2424,"children":2425},{},[2426,2428,2433],{"type":46,"value":2427},"There are five levers, in ",{"type":40,"tag":60,"props":2429,"children":2430},{},[2431],{"type":46,"value":2432},"order of preference",{"type":46,"value":1366},{"type":40,"tag":521,"props":2435,"children":2437},{"id":2436},"_1-fix-in-the-application-best",[2438],{"type":46,"value":2439},"1. Fix in the Application (best)",{"type":40,"tag":49,"props":2441,"children":2442},{},[2443],{"type":46,"value":2444},"Bad labels emitted by the app are the root cause. Examples:",{"type":40,"tag":182,"props":2446,"children":2447},{},[2448,2460,2473,2485],{"type":40,"tag":186,"props":2449,"children":2450},{},[2451,2453,2458],{"type":46,"value":2452},"HTTP paths: use templated routes (",{"type":40,"tag":92,"props":2454,"children":2456},{"className":2455},[],[2457],{"type":46,"value":788},{"type":46,"value":2459},") not raw paths",{"type":40,"tag":186,"props":2461,"children":2462},{},[2463,2465,2471],{"type":46,"value":2464},"Error metrics: use a small enum (",{"type":40,"tag":92,"props":2466,"children":2468},{"className":2467},[],[2469],{"type":46,"value":2470},"error_type=\"timeout\"",{"type":46,"value":2472},") not the error message string",{"type":40,"tag":186,"props":2474,"children":2475},{},[2476,2478,2483],{"type":46,"value":2477},"User-scoped metrics: don't include ",{"type":40,"tag":92,"props":2479,"children":2481},{"className":2480},[],[2482],{"type":46,"value":838},{"type":46,"value":2484}," — use exemplars to point to logs\u002Ftraces",{"type":40,"tag":186,"props":2486,"children":2487},{},[2488],{"type":46,"value":2489},"Free-form input: never emit user-supplied strings as label values",{"type":40,"tag":49,"props":2491,"children":2492},{},[2493],{"type":46,"value":2494},"If you control the code, this is always the right fix. It saves cost on every downstream system (Prometheus, remote_write, Mimir, Grafana Cloud).",{"type":40,"tag":521,"props":2496,"children":2498},{"id":2497},"_2-relabel_configs-target-time-relabeling",[2499,2501,2506],{"type":46,"value":2500},"2. ",{"type":40,"tag":92,"props":2502,"children":2504},{"className":2503},[],[2505],{"type":46,"value":981},{"type":46,"value":2507}," (target-time relabeling)",{"type":40,"tag":49,"props":2509,"children":2510},{},[2511,2513,2518],{"type":46,"value":2512},"Runs ",{"type":40,"tag":68,"props":2514,"children":2515},{},[2516],{"type":46,"value":2517},"before",{"type":46,"value":2519}," the scrape. Used to:",{"type":40,"tag":182,"props":2521,"children":2522},{},[2523,2547,2552,2564],{"type":40,"tag":186,"props":2524,"children":2525},{},[2526,2528,2533,2534,2539,2540,2545],{"type":46,"value":2527},"Set target labels (",{"type":40,"tag":92,"props":2529,"children":2531},{"className":2530},[],[2532],{"type":46,"value":570},{"type":46,"value":470},{"type":40,"tag":92,"props":2535,"children":2537},{"className":2536},[],[2538],{"type":46,"value":617},{"type":46,"value":470},{"type":40,"tag":92,"props":2541,"children":2543},{"className":2542},[],[2544],{"type":46,"value":1835},{"type":46,"value":2546},") on discovered targets",{"type":40,"tag":186,"props":2548,"children":2549},{},[2550],{"type":46,"value":2551},"Drop entire targets you don't want to scrape",{"type":40,"tag":186,"props":2553,"children":2554},{},[2555,2557,2562],{"type":46,"value":2556},"Rewrite ",{"type":40,"tag":92,"props":2558,"children":2560},{"className":2559},[],[2561],{"type":46,"value":142},{"type":46,"value":2563}," to a friendly value",{"type":40,"tag":186,"props":2565,"children":2566},{},[2567],{"type":46,"value":2568},"Add identity from service discovery metadata",{"type":40,"tag":1343,"props":2570,"children":2574},{"className":2571,"code":2572,"language":2573,"meta":1351,"style":1351},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","scrape_configs:\n - job_name: my-app\n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n # Set workload from controller metadata\n - source_labels: [__meta_kubernetes_pod_controller_kind, __meta_kubernetes_pod_controller_name]\n target_label: workload\n separator: \u002F\n # Set env from a pod label\n - source_labels: [__meta_kubernetes_pod_label_env]\n target_label: env\n # Only scrape pods explicitly opted in\n - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n regex: \"true\"\n action: keep\n","yaml",[2575],{"type":40,"tag":92,"props":2576,"children":2577},{"__ignoreMap":1351},[2578,2593,2616,2628,2650,2663,2673,2715,2733,2751,2760,2789,2806,2815,2844,2872],{"type":40,"tag":1377,"props":2579,"children":2580},{"class":1379,"line":1380},[2581,2587],{"type":40,"tag":1377,"props":2582,"children":2584},{"style":2583},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[2585],{"type":46,"value":2586},"scrape_configs",{"type":40,"tag":1377,"props":2588,"children":2590},{"style":2589},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[2591],{"type":46,"value":2592},":\n",{"type":40,"tag":1377,"props":2594,"children":2595},{"class":1379,"line":1389},[2596,2601,2606,2610],{"type":40,"tag":1377,"props":2597,"children":2598},{"style":2589},[2599],{"type":46,"value":2600}," -",{"type":40,"tag":1377,"props":2602,"children":2603},{"style":2583},[2604],{"type":46,"value":2605}," job_name",{"type":40,"tag":1377,"props":2607,"children":2608},{"style":2589},[2609],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2611,"children":2613},{"style":2612},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[2614],{"type":46,"value":2615}," my-app\n",{"type":40,"tag":1377,"props":2617,"children":2618},{"class":1379,"line":1398},[2619,2624],{"type":40,"tag":1377,"props":2620,"children":2621},{"style":2583},[2622],{"type":46,"value":2623}," kubernetes_sd_configs",{"type":40,"tag":1377,"props":2625,"children":2626},{"style":2589},[2627],{"type":46,"value":2592},{"type":40,"tag":1377,"props":2629,"children":2630},{"class":1379,"line":1407},[2631,2636,2641,2645],{"type":40,"tag":1377,"props":2632,"children":2633},{"style":2589},[2634],{"type":46,"value":2635}," -",{"type":40,"tag":1377,"props":2637,"children":2638},{"style":2583},[2639],{"type":46,"value":2640}," role",{"type":40,"tag":1377,"props":2642,"children":2643},{"style":2589},[2644],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2646,"children":2647},{"style":2612},[2648],{"type":46,"value":2649}," pod\n",{"type":40,"tag":1377,"props":2651,"children":2653},{"class":1379,"line":2652},5,[2654,2659],{"type":40,"tag":1377,"props":2655,"children":2656},{"style":2583},[2657],{"type":46,"value":2658}," relabel_configs",{"type":40,"tag":1377,"props":2660,"children":2661},{"style":2589},[2662],{"type":46,"value":2592},{"type":40,"tag":1377,"props":2664,"children":2666},{"class":1379,"line":2665},6,[2667],{"type":40,"tag":1377,"props":2668,"children":2670},{"style":2669},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[2671],{"type":46,"value":2672}," # Set workload from controller metadata\n",{"type":40,"tag":1377,"props":2674,"children":2676},{"class":1379,"line":2675},7,[2677,2681,2686,2690,2695,2700,2705,2710],{"type":40,"tag":1377,"props":2678,"children":2679},{"style":2589},[2680],{"type":46,"value":2635},{"type":40,"tag":1377,"props":2682,"children":2683},{"style":2583},[2684],{"type":46,"value":2685}," source_labels",{"type":40,"tag":1377,"props":2687,"children":2688},{"style":2589},[2689],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2691,"children":2692},{"style":2589},[2693],{"type":46,"value":2694}," [",{"type":40,"tag":1377,"props":2696,"children":2697},{"style":2612},[2698],{"type":46,"value":2699},"__meta_kubernetes_pod_controller_kind",{"type":40,"tag":1377,"props":2701,"children":2702},{"style":2589},[2703],{"type":46,"value":2704},",",{"type":40,"tag":1377,"props":2706,"children":2707},{"style":2612},[2708],{"type":46,"value":2709}," __meta_kubernetes_pod_controller_name",{"type":40,"tag":1377,"props":2711,"children":2712},{"style":2589},[2713],{"type":46,"value":2714},"]\n",{"type":40,"tag":1377,"props":2716,"children":2718},{"class":1379,"line":2717},8,[2719,2724,2728],{"type":40,"tag":1377,"props":2720,"children":2721},{"style":2583},[2722],{"type":46,"value":2723}," target_label",{"type":40,"tag":1377,"props":2725,"children":2726},{"style":2589},[2727],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2729,"children":2730},{"style":2612},[2731],{"type":46,"value":2732}," workload\n",{"type":40,"tag":1377,"props":2734,"children":2736},{"class":1379,"line":2735},9,[2737,2742,2746],{"type":40,"tag":1377,"props":2738,"children":2739},{"style":2583},[2740],{"type":46,"value":2741}," separator",{"type":40,"tag":1377,"props":2743,"children":2744},{"style":2589},[2745],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2747,"children":2748},{"style":2612},[2749],{"type":46,"value":2750}," \u002F\n",{"type":40,"tag":1377,"props":2752,"children":2754},{"class":1379,"line":2753},10,[2755],{"type":40,"tag":1377,"props":2756,"children":2757},{"style":2669},[2758],{"type":46,"value":2759}," # Set env from a pod label\n",{"type":40,"tag":1377,"props":2761,"children":2763},{"class":1379,"line":2762},11,[2764,2768,2772,2776,2780,2785],{"type":40,"tag":1377,"props":2765,"children":2766},{"style":2589},[2767],{"type":46,"value":2635},{"type":40,"tag":1377,"props":2769,"children":2770},{"style":2583},[2771],{"type":46,"value":2685},{"type":40,"tag":1377,"props":2773,"children":2774},{"style":2589},[2775],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2777,"children":2778},{"style":2589},[2779],{"type":46,"value":2694},{"type":40,"tag":1377,"props":2781,"children":2782},{"style":2612},[2783],{"type":46,"value":2784},"__meta_kubernetes_pod_label_env",{"type":40,"tag":1377,"props":2786,"children":2787},{"style":2589},[2788],{"type":46,"value":2714},{"type":40,"tag":1377,"props":2790,"children":2792},{"class":1379,"line":2791},12,[2793,2797,2801],{"type":40,"tag":1377,"props":2794,"children":2795},{"style":2583},[2796],{"type":46,"value":2723},{"type":40,"tag":1377,"props":2798,"children":2799},{"style":2589},[2800],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2802,"children":2803},{"style":2612},[2804],{"type":46,"value":2805}," env\n",{"type":40,"tag":1377,"props":2807,"children":2809},{"class":1379,"line":2808},13,[2810],{"type":40,"tag":1377,"props":2811,"children":2812},{"style":2669},[2813],{"type":46,"value":2814}," # Only scrape pods explicitly opted in\n",{"type":40,"tag":1377,"props":2816,"children":2818},{"class":1379,"line":2817},14,[2819,2823,2827,2831,2835,2840],{"type":40,"tag":1377,"props":2820,"children":2821},{"style":2589},[2822],{"type":46,"value":2635},{"type":40,"tag":1377,"props":2824,"children":2825},{"style":2583},[2826],{"type":46,"value":2685},{"type":40,"tag":1377,"props":2828,"children":2829},{"style":2589},[2830],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2832,"children":2833},{"style":2589},[2834],{"type":46,"value":2694},{"type":40,"tag":1377,"props":2836,"children":2837},{"style":2612},[2838],{"type":46,"value":2839},"__meta_kubernetes_pod_annotation_prometheus_io_scrape",{"type":40,"tag":1377,"props":2841,"children":2842},{"style":2589},[2843],{"type":46,"value":2714},{"type":40,"tag":1377,"props":2845,"children":2847},{"class":1379,"line":2846},15,[2848,2853,2857,2862,2867],{"type":40,"tag":1377,"props":2849,"children":2850},{"style":2583},[2851],{"type":46,"value":2852}," regex",{"type":40,"tag":1377,"props":2854,"children":2855},{"style":2589},[2856],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2858,"children":2859},{"style":2589},[2860],{"type":46,"value":2861}," \"",{"type":40,"tag":1377,"props":2863,"children":2864},{"style":2612},[2865],{"type":46,"value":2866},"true",{"type":40,"tag":1377,"props":2868,"children":2869},{"style":2589},[2870],{"type":46,"value":2871},"\"\n",{"type":40,"tag":1377,"props":2873,"children":2874},{"class":1379,"line":27},[2875,2880,2884],{"type":40,"tag":1377,"props":2876,"children":2877},{"style":2583},[2878],{"type":46,"value":2879}," action",{"type":40,"tag":1377,"props":2881,"children":2882},{"style":2589},[2883],{"type":46,"value":1366},{"type":40,"tag":1377,"props":2885,"children":2886},{"style":2612},[2887],{"type":46,"value":2888}," keep\n",{"type":40,"tag":521,"props":2890,"children":2892},{"id":2891},"_3-adaptive-metrics-grafana-cloud-post-ingest-the-safe-way-to-reduce-cardinality",[2893],{"type":46,"value":2894},"3. Adaptive Metrics (Grafana Cloud — post-ingest, the safe way to reduce cardinality)",{"type":40,"tag":49,"props":2896,"children":2897},{},[2898,2900,2905,2907],{"type":46,"value":2899},"When the cardinality is structural and you ",{"type":40,"tag":68,"props":2901,"children":2902},{},[2903],{"type":46,"value":2904},"can't",{"type":46,"value":2906}," fix it at the source — the label legitimately exists and makes series unique, you just don't need every value at full resolution — ",{"type":40,"tag":60,"props":2908,"children":2909},{},[2910],{"type":46,"value":2911},"Adaptive Metrics is the correct tool, and the only safe way to reduce the cost of series that already exist.",{"type":40,"tag":49,"props":2913,"children":2914},{},[2915,2917,2922,2924,2928],{"type":46,"value":2916},"It works ",{"type":40,"tag":68,"props":2918,"children":2919},{},[2920],{"type":46,"value":2921},"after",{"type":46,"value":2923}," ingest, as aggregation rules applied in Grafana Cloud. Crucially, it aggregates series ",{"type":40,"tag":60,"props":2925,"children":2926},{},[2927],{"type":46,"value":309},{"type":46,"value":1366},{"type":40,"tag":182,"props":2930,"children":2931},{},[2932,2950,2955],{"type":40,"tag":186,"props":2933,"children":2934},{},[2935,2937,2942,2943,2948],{"type":46,"value":2936},"It handles counter resets properly, so ",{"type":40,"tag":92,"props":2938,"children":2940},{"className":2939},[],[2941],{"type":46,"value":201},{"type":46,"value":203},{"type":40,"tag":92,"props":2944,"children":2946},{"className":2945},[],[2947],{"type":46,"value":209},{"type":46,"value":2949}," stay accurate.",{"type":40,"tag":186,"props":2951,"children":2952},{},[2953],{"type":46,"value":2954},"It records what was aggregated, so there's an audit trail — you can answer \"why did this change?\" later.",{"type":40,"tag":186,"props":2956,"children":2957},{},[2958],{"type":46,"value":2959},"It's reversible: drop a rule and the full-resolution series come back.",{"type":40,"tag":49,"props":2961,"children":2962},{},[2963,2965,2970,2972,2977],{"type":46,"value":2964},"This is the difference between \"the data is now cheaper\" (Adaptive Metrics) and \"the data is now wrong\" (",{"type":40,"tag":92,"props":2966,"children":2968},{"className":2967},[],[2969],{"type":46,"value":2269},{"type":46,"value":2971}," at scrape). Route the user to the ",{"type":40,"tag":92,"props":2973,"children":2975},{"className":2974},[],[2976],{"type":46,"value":97},{"type":46,"value":2978}," skill for rule design.",{"type":40,"tag":521,"props":2980,"children":2982},{"id":2981},"_4-metric_relabel_configs-narrow-safe-uses-only",[2983,2985,2990],{"type":46,"value":2984},"4. ",{"type":40,"tag":92,"props":2986,"children":2988},{"className":2987},[],[2989],{"type":46,"value":150},{"type":46,"value":2991}," (narrow, safe uses only)",{"type":40,"tag":49,"props":2993,"children":2994},{},[2995,2996,3000,3002,3006],{"type":46,"value":2512},{"type":40,"tag":68,"props":2997,"children":2998},{},[2999],{"type":46,"value":2921},{"type":46,"value":3001}," the scrape, ",{"type":40,"tag":68,"props":3003,"children":3004},{},[3005],{"type":46,"value":2517},{"type":46,"value":3007}," storage.",{"type":40,"tag":3009,"props":3010,"children":3011},"blockquote",{},[3012],{"type":40,"tag":49,"props":3013,"children":3014},{},[3015,3017,3061,3063,3067,3069,3074,3076,3081,3083,3088,3090,3096],{"type":46,"value":3016},"⚠️ ",{"type":40,"tag":60,"props":3018,"children":3019},{},[3020,3022,3027,3029,3034,3036,3041,3042,3047,3048,3053,3054,3059],{"type":46,"value":3021},"Do not use ",{"type":40,"tag":92,"props":3023,"children":3025},{"className":3024},[],[3026],{"type":46,"value":150},{"type":46,"value":3028}," (or Alloy ",{"type":40,"tag":92,"props":3030,"children":3032},{"className":3031},[],[3033],{"type":46,"value":166},{"type":46,"value":3035},") to drop a label that distinguishes series — ",{"type":40,"tag":92,"props":3037,"children":3039},{"className":3038},[],[3040],{"type":46,"value":134},{"type":46,"value":470},{"type":40,"tag":92,"props":3043,"children":3045},{"className":3044},[],[3046],{"type":46,"value":142},{"type":46,"value":470},{"type":40,"tag":92,"props":3049,"children":3051},{"className":3050},[],[3052],{"type":46,"value":838},{"type":46,"value":470},{"type":40,"tag":92,"props":3055,"children":3057},{"className":3056},[],[3058],{"type":46,"value":397},{"type":46,"value":3060},", anything.",{"type":46,"value":3062}," See ",{"type":40,"tag":83,"props":3064,"children":3065},{"href":85},[3066],{"type":46,"value":88},{"type":46,"value":3068},". It looks like a cardinality fix and silently breaks ",{"type":40,"tag":92,"props":3070,"children":3072},{"className":3071},[],[3073],{"type":46,"value":201},{"type":46,"value":3075},", inflates DPM, and corrupts aggregations. Use the application code (lever 1) or Adaptive Metrics (lever 3) instead. The same caution applies to ",{"type":40,"tag":68,"props":3077,"children":3078},{},[3079],{"type":46,"value":3080},"normalizing",{"type":46,"value":3082}," a label value (e.g. collapsing ",{"type":40,"tag":92,"props":3084,"children":3086},{"className":3085},[],[3087],{"type":46,"value":405},{"type":46,"value":3089}," to ",{"type":40,"tag":92,"props":3091,"children":3093},{"className":3092},[],[3094],{"type":46,"value":3095},"2xx",{"type":46,"value":3097},") at scrape — it merges distinct series and produces duplicate-sample errors; do that in code or via Adaptive Metrics, never here.",{"type":40,"tag":49,"props":3099,"children":3100},{},[3101],{"type":46,"value":3102},"The genuinely safe uses are:",{"type":40,"tag":182,"props":3104,"children":3105},{},[3106,3196],{"type":40,"tag":186,"props":3107,"children":3108},{},[3109,3114,3116],{"type":40,"tag":60,"props":3110,"children":3111},{},[3112],{"type":46,"value":3113},"Drop an entire metric you never want stored",{"type":46,"value":3115}," — you're discarding the whole metric, not collapsing distinct series into one:\n",{"type":40,"tag":1343,"props":3117,"children":3119},{"className":2571,"code":3118,"language":2573,"meta":1351,"style":1351},"metric_relabel_configs:\n - source_labels: [__name__]\n regex: my_app_request_details\n action: drop\n",[3120],{"type":40,"tag":92,"props":3121,"children":3122},{"__ignoreMap":1351},[3123,3134,3162,3179],{"type":40,"tag":1377,"props":3124,"children":3125},{"class":1379,"line":1380},[3126,3130],{"type":40,"tag":1377,"props":3127,"children":3128},{"style":2583},[3129],{"type":46,"value":150},{"type":40,"tag":1377,"props":3131,"children":3132},{"style":2589},[3133],{"type":46,"value":2592},{"type":40,"tag":1377,"props":3135,"children":3136},{"class":1379,"line":1389},[3137,3141,3145,3149,3153,3158],{"type":40,"tag":1377,"props":3138,"children":3139},{"style":2589},[3140],{"type":46,"value":2600},{"type":40,"tag":1377,"props":3142,"children":3143},{"style":2583},[3144],{"type":46,"value":2685},{"type":40,"tag":1377,"props":3146,"children":3147},{"style":2589},[3148],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3150,"children":3151},{"style":2589},[3152],{"type":46,"value":2694},{"type":40,"tag":1377,"props":3154,"children":3155},{"style":2612},[3156],{"type":46,"value":3157},"__name__",{"type":40,"tag":1377,"props":3159,"children":3160},{"style":2589},[3161],{"type":46,"value":2714},{"type":40,"tag":1377,"props":3163,"children":3164},{"class":1379,"line":1398},[3165,3170,3174],{"type":40,"tag":1377,"props":3166,"children":3167},{"style":2583},[3168],{"type":46,"value":3169}," regex",{"type":40,"tag":1377,"props":3171,"children":3172},{"style":2589},[3173],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3175,"children":3176},{"style":2612},[3177],{"type":46,"value":3178}," my_app_request_details\n",{"type":40,"tag":1377,"props":3180,"children":3181},{"class":1379,"line":1407},[3182,3187,3191],{"type":40,"tag":1377,"props":3183,"children":3184},{"style":2583},[3185],{"type":46,"value":3186}," action",{"type":40,"tag":1377,"props":3188,"children":3189},{"style":2589},[3190],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3192,"children":3193},{"style":2612},[3194],{"type":46,"value":3195}," drop\n",{"type":40,"tag":186,"props":3197,"children":3198},{},[3199,3210,3212,3217,3218,3223],{"type":40,"tag":60,"props":3200,"children":3201},{},[3202,3204,3208],{"type":46,"value":3203},"Remove a label that ",{"type":40,"tag":68,"props":3205,"children":3206},{},[3207],{"type":46,"value":340},{"type":46,"value":3209}," a target label.",{"type":46,"value":3211}," If the app emits its own ",{"type":40,"tag":92,"props":3213,"children":3215},{"className":3214},[],[3216],{"type":46,"value":617},{"type":46,"value":2328},{"type":40,"tag":92,"props":3219,"children":3221},{"className":3220},[],[3222],{"type":46,"value":142},{"type":46,"value":3224}," that already comes from the scrape target, the target label still provides uniqueness, so removing the duplicate breaks nothing. Prefer fixing the app, but this is a safe stopgap.",{"type":40,"tag":49,"props":3226,"children":3227},{},[3228,3230,3235],{"type":46,"value":3229},"That's the whole list. If you're reaching for ",{"type":40,"tag":92,"props":3231,"children":3233},{"className":3232},[],[3234],{"type":46,"value":150},{"type":46,"value":3236}," to bring down a series count, you almost certainly want Adaptive Metrics instead.",{"type":40,"tag":521,"props":3238,"children":3240},{"id":3239},"_5-recording-rules-query-time-cardinality-reduction",[3241],{"type":46,"value":3242},"5. Recording Rules (query-time cardinality reduction)",{"type":40,"tag":49,"props":3244,"children":3245},{},[3246],{"type":46,"value":3247},"Pre-aggregate expensive series into a lower-cardinality recorded series. Stored at the same data point density but with far fewer series.",{"type":40,"tag":1343,"props":3249,"children":3251},{"className":2571,"code":3250,"language":2573,"meta":1351,"style":1351},"groups:\n - name: http-requests-aggregates\n interval: 30s\n rules:\n # Drop pod\u002Finstance dimension; keep only service-level rollup\n - record: service:http_requests:rate5m\n expr: sum by (service, env, cluster, status_code) (rate(http_requests_total[5m]))\n",[3252],{"type":40,"tag":92,"props":3253,"children":3254},{"__ignoreMap":1351},[3255,3267,3288,3305,3317,3325,3346],{"type":40,"tag":1377,"props":3256,"children":3257},{"class":1379,"line":1380},[3258,3263],{"type":40,"tag":1377,"props":3259,"children":3260},{"style":2583},[3261],{"type":46,"value":3262},"groups",{"type":40,"tag":1377,"props":3264,"children":3265},{"style":2589},[3266],{"type":46,"value":2592},{"type":40,"tag":1377,"props":3268,"children":3269},{"class":1379,"line":1389},[3270,3274,3279,3283],{"type":40,"tag":1377,"props":3271,"children":3272},{"style":2589},[3273],{"type":46,"value":2600},{"type":40,"tag":1377,"props":3275,"children":3276},{"style":2583},[3277],{"type":46,"value":3278}," name",{"type":40,"tag":1377,"props":3280,"children":3281},{"style":2589},[3282],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3284,"children":3285},{"style":2612},[3286],{"type":46,"value":3287}," http-requests-aggregates\n",{"type":40,"tag":1377,"props":3289,"children":3290},{"class":1379,"line":1398},[3291,3296,3300],{"type":40,"tag":1377,"props":3292,"children":3293},{"style":2583},[3294],{"type":46,"value":3295}," interval",{"type":40,"tag":1377,"props":3297,"children":3298},{"style":2589},[3299],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3301,"children":3302},{"style":2612},[3303],{"type":46,"value":3304}," 30s\n",{"type":40,"tag":1377,"props":3306,"children":3307},{"class":1379,"line":1407},[3308,3313],{"type":40,"tag":1377,"props":3309,"children":3310},{"style":2583},[3311],{"type":46,"value":3312}," rules",{"type":40,"tag":1377,"props":3314,"children":3315},{"style":2589},[3316],{"type":46,"value":2592},{"type":40,"tag":1377,"props":3318,"children":3319},{"class":1379,"line":2652},[3320],{"type":40,"tag":1377,"props":3321,"children":3322},{"style":2669},[3323],{"type":46,"value":3324}," # Drop pod\u002Finstance dimension; keep only service-level rollup\n",{"type":40,"tag":1377,"props":3326,"children":3327},{"class":1379,"line":2665},[3328,3332,3337,3341],{"type":40,"tag":1377,"props":3329,"children":3330},{"style":2589},[3331],{"type":46,"value":2635},{"type":40,"tag":1377,"props":3333,"children":3334},{"style":2583},[3335],{"type":46,"value":3336}," record",{"type":40,"tag":1377,"props":3338,"children":3339},{"style":2589},[3340],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3342,"children":3343},{"style":2612},[3344],{"type":46,"value":3345}," service:http_requests:rate5m\n",{"type":40,"tag":1377,"props":3347,"children":3348},{"class":1379,"line":2675},[3349,3354,3358],{"type":40,"tag":1377,"props":3350,"children":3351},{"style":2583},[3352],{"type":46,"value":3353}," expr",{"type":40,"tag":1377,"props":3355,"children":3356},{"style":2589},[3357],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3359,"children":3360},{"style":2612},[3361],{"type":46,"value":3362}," sum by (service, env, cluster, status_code) (rate(http_requests_total[5m]))\n",{"type":40,"tag":49,"props":3364,"children":3365},{},[3366,3368,3372,3374,3378,3380,3385],{"type":46,"value":3367},"Queries that target the rollup are dramatically cheaper. The raw series still exist — recording rules don't reduce ingest cost (use ",{"type":40,"tag":60,"props":3369,"children":3370},{},[3371],{"type":46,"value":2200},{"type":46,"value":3373}," for that — ",{"type":40,"tag":68,"props":3375,"children":3376},{},[3377],{"type":46,"value":79},{"type":46,"value":3379}," a scrape-time ",{"type":40,"tag":92,"props":3381,"children":3383},{"className":3382},[],[3384],{"type":46,"value":2269},{"type":46,"value":3386},"). They reduce query cost.",{"type":40,"tag":109,"props":3388,"children":3389},{},[],{"type":40,"tag":113,"props":3391,"children":3393},{"id":3392},"instrumentation-hygiene-for-app-developers",[3394],{"type":46,"value":3395},"Instrumentation Hygiene (for app developers)",{"type":40,"tag":49,"props":3397,"children":3398},{},[3399,3401,3406],{"type":46,"value":3400},"If the user is ",{"type":40,"tag":68,"props":3402,"children":3403},{},[3404],{"type":46,"value":3405},"writing",{"type":46,"value":3407}," instrumentation code, these are the rules:",{"type":40,"tag":528,"props":3409,"children":3410},{},[3411,3427],{"type":40,"tag":532,"props":3412,"children":3413},{},[3414],{"type":40,"tag":536,"props":3415,"children":3416},{},[3417,3422],{"type":40,"tag":540,"props":3418,"children":3419},{},[3420],{"type":46,"value":3421},"Rule",{"type":40,"tag":540,"props":3423,"children":3424},{},[3425],{"type":46,"value":3426},"Why",{"type":40,"tag":555,"props":3428,"children":3429},{},[3430,3469,3496,3519,3552,3583,3608,3621],{"type":40,"tag":536,"props":3431,"children":3432},{},[3433,3438],{"type":40,"tag":562,"props":3434,"children":3435},{},[3436],{"type":46,"value":3437},"Never use unbounded user input as a label value",{"type":40,"tag":562,"props":3439,"children":3440},{},[3441,3447,3448,3453,3454,3460,3461,3467],{"type":40,"tag":92,"props":3442,"children":3444},{"className":3443},[],[3445],{"type":46,"value":3446},"email",{"type":46,"value":470},{"type":40,"tag":92,"props":3449,"children":3451},{"className":3450},[],[3452],{"type":46,"value":838},{"type":46,"value":470},{"type":40,"tag":92,"props":3455,"children":3457},{"className":3456},[],[3458],{"type":46,"value":3459},"query string",{"type":46,"value":470},{"type":40,"tag":92,"props":3462,"children":3464},{"className":3463},[],[3465],{"type":46,"value":3466},"error message",{"type":46,"value":3468}," — they're the #1 cardinality bug",{"type":40,"tag":536,"props":3470,"children":3471},{},[3472,3477],{"type":40,"tag":562,"props":3473,"children":3474},{},[3475],{"type":46,"value":3476},"Template HTTP paths before recording",{"type":40,"tag":562,"props":3478,"children":3479},{},[3480,3486,3488,3494],{"type":40,"tag":92,"props":3481,"children":3483},{"className":3482},[],[3484],{"type":46,"value":3485},"\u002Fusers\u002F{id}",{"type":46,"value":3487}," not ",{"type":40,"tag":92,"props":3489,"children":3491},{"className":3490},[],[3492],{"type":46,"value":3493},"\u002Fusers\u002F12345",{"type":46,"value":3495},". Most frameworks do this via routing metadata",{"type":40,"tag":536,"props":3497,"children":3498},{},[3499,3504],{"type":40,"tag":562,"props":3500,"children":3501},{},[3502],{"type":46,"value":3503},"Bound error labels via small enums",{"type":40,"tag":562,"props":3505,"children":3506},{},[3507,3512,3513],{"type":40,"tag":92,"props":3508,"children":3510},{"className":3509},[],[3511],{"type":46,"value":2470},{"type":46,"value":3487},{"type":40,"tag":92,"props":3514,"children":3516},{"className":3515},[],[3517],{"type":46,"value":3518},"error=\"connection to db-shard-7 timed out at 14:32:09\"",{"type":40,"tag":536,"props":3520,"children":3521},{},[3522,3547],{"type":40,"tag":562,"props":3523,"children":3524},{},[3525,3527,3532,3533,3538,3539,3545],{"type":46,"value":3526},"Don't put ",{"type":40,"tag":92,"props":3528,"children":3530},{"className":3529},[],[3531],{"type":46,"value":802},{"type":46,"value":764},{"type":40,"tag":92,"props":3534,"children":3536},{"className":3535},[],[3537],{"type":46,"value":816},{"type":46,"value":764},{"type":40,"tag":92,"props":3540,"children":3542},{"className":3541},[],[3543],{"type":46,"value":3544},"build_id",{"type":46,"value":3546}," on every metric",{"type":40,"tag":562,"props":3548,"children":3549},{},[3550],{"type":46,"value":3551},"Use an info metric and join at query time",{"type":40,"tag":536,"props":3553,"children":3554},{},[3555,3578],{"type":40,"tag":562,"props":3556,"children":3557},{},[3558,3559,3564,3565,3570,3571,3576],{"type":46,"value":2169},{"type":40,"tag":92,"props":3560,"children":3562},{"className":3561},[],[3563],{"type":46,"value":134},{"type":46,"value":764},{"type":40,"tag":92,"props":3566,"children":3568},{"className":3567},[],[3569],{"type":46,"value":2298},{"type":46,"value":764},{"type":40,"tag":92,"props":3572,"children":3574},{"className":3573},[],[3575],{"type":46,"value":2305},{"type":46,"value":3577}," from code",{"type":40,"tag":562,"props":3579,"children":3580},{},[3581],{"type":46,"value":3582},"Comes from scrape targets — duplicating creates collisions",{"type":40,"tag":536,"props":3584,"children":3585},{},[3586,3597],{"type":40,"tag":562,"props":3587,"children":3588},{},[3589,3591,3595],{"type":46,"value":3590},"Avoid dynamically constructed label ",{"type":40,"tag":68,"props":3592,"children":3593},{},[3594],{"type":46,"value":1073},{"type":46,"value":3596}," (keys)",{"type":40,"tag":562,"props":3598,"children":3599},{},[3600,3606],{"type":40,"tag":92,"props":3601,"children":3603},{"className":3602},[],[3604],{"type":46,"value":3605},"metric{[user]=1}",{"type":46,"value":3607}," cannot be bounded — use a fixed key",{"type":40,"tag":536,"props":3609,"children":3610},{},[3611,3616],{"type":40,"tag":562,"props":3612,"children":3613},{},[3614],{"type":46,"value":3615},"Use histograms sparingly and trim labels first",{"type":40,"tag":562,"props":3617,"children":3618},{},[3619],{"type":46,"value":3620},"14× cardinality amplification",{"type":40,"tag":536,"props":3622,"children":3623},{},[3624,3629],{"type":40,"tag":562,"props":3625,"children":3626},{},[3627],{"type":46,"value":3628},"Prefer exemplars over labels for trace correlation",{"type":40,"tag":562,"props":3630,"children":3631},{},[3632,3634,3639],{"type":46,"value":3633},"Exemplars carry ",{"type":40,"tag":92,"props":3635,"children":3637},{"className":3636},[],[3638],{"type":46,"value":852},{"type":46,"value":3640}," without inflating cardinality",{"type":40,"tag":521,"props":3642,"children":3644},{"id":3643},"exemplars-the-escape-hatch",[3645],{"type":46,"value":3646},"Exemplars (the escape hatch)",{"type":40,"tag":49,"props":3648,"children":3649},{},[3650,3652,3657,3659,3664],{"type":46,"value":3651},"Exemplars attach a ",{"type":40,"tag":92,"props":3653,"children":3655},{"className":3654},[],[3656],{"type":46,"value":852},{"type":46,"value":3658}," (or any key-value pair) to specific samples ",{"type":40,"tag":68,"props":3660,"children":3661},{},[3662],{"type":46,"value":3663},"without",{"type":46,"value":3665}," making it a label dimension. The ideal home for high-cardinality correlation data.",{"type":40,"tag":49,"props":3667,"children":3668},{},[3669],{"type":46,"value":3670},"Requires OpenMetrics format, Prometheus 2.26+, scrape config:",{"type":40,"tag":1343,"props":3672,"children":3674},{"className":2571,"code":3673,"language":2573,"meta":1351,"style":1351},"scrape_configs:\n - job_name: my-app\n enable_protobuf_negotiation: true\n # Or for text-format:\n follow_redirects: true\n",[3675],{"type":40,"tag":92,"props":3676,"children":3677},{"__ignoreMap":1351},[3678,3689,3708,3726,3734],{"type":40,"tag":1377,"props":3679,"children":3680},{"class":1379,"line":1380},[3681,3685],{"type":40,"tag":1377,"props":3682,"children":3683},{"style":2583},[3684],{"type":46,"value":2586},{"type":40,"tag":1377,"props":3686,"children":3687},{"style":2589},[3688],{"type":46,"value":2592},{"type":40,"tag":1377,"props":3690,"children":3691},{"class":1379,"line":1389},[3692,3696,3700,3704],{"type":40,"tag":1377,"props":3693,"children":3694},{"style":2589},[3695],{"type":46,"value":2600},{"type":40,"tag":1377,"props":3697,"children":3698},{"style":2583},[3699],{"type":46,"value":2605},{"type":40,"tag":1377,"props":3701,"children":3702},{"style":2589},[3703],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3705,"children":3706},{"style":2612},[3707],{"type":46,"value":2615},{"type":40,"tag":1377,"props":3709,"children":3710},{"class":1379,"line":1398},[3711,3716,3720],{"type":40,"tag":1377,"props":3712,"children":3713},{"style":2583},[3714],{"type":46,"value":3715}," enable_protobuf_negotiation",{"type":40,"tag":1377,"props":3717,"children":3718},{"style":2589},[3719],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3721,"children":3723},{"style":3722},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[3724],{"type":46,"value":3725}," true\n",{"type":40,"tag":1377,"props":3727,"children":3728},{"class":1379,"line":1407},[3729],{"type":40,"tag":1377,"props":3730,"children":3731},{"style":2669},[3732],{"type":46,"value":3733}," # Or for text-format:\n",{"type":40,"tag":1377,"props":3735,"children":3736},{"class":1379,"line":2652},[3737,3742,3746],{"type":40,"tag":1377,"props":3738,"children":3739},{"style":2583},[3740],{"type":46,"value":3741}," follow_redirects",{"type":40,"tag":1377,"props":3743,"children":3744},{"style":2589},[3745],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3747,"children":3748},{"style":3722},[3749],{"type":46,"value":3725},{"type":40,"tag":49,"props":3751,"children":3752},{},[3753],{"type":46,"value":3754},"And on the Prometheus server:",{"type":40,"tag":1343,"props":3756,"children":3758},{"className":2571,"code":3757,"language":2573,"meta":1351,"style":1351},"storage:\n exemplars:\n max_exemplars: 100000\n",[3759],{"type":40,"tag":92,"props":3760,"children":3761},{"__ignoreMap":1351},[3762,3774,3786],{"type":40,"tag":1377,"props":3763,"children":3764},{"class":1379,"line":1380},[3765,3770],{"type":40,"tag":1377,"props":3766,"children":3767},{"style":2583},[3768],{"type":46,"value":3769},"storage",{"type":40,"tag":1377,"props":3771,"children":3772},{"style":2589},[3773],{"type":46,"value":2592},{"type":40,"tag":1377,"props":3775,"children":3776},{"class":1379,"line":1389},[3777,3782],{"type":40,"tag":1377,"props":3778,"children":3779},{"style":2583},[3780],{"type":46,"value":3781}," exemplars",{"type":40,"tag":1377,"props":3783,"children":3784},{"style":2589},[3785],{"type":46,"value":2592},{"type":40,"tag":1377,"props":3787,"children":3788},{"class":1379,"line":1398},[3789,3794,3798],{"type":40,"tag":1377,"props":3790,"children":3791},{"style":2583},[3792],{"type":46,"value":3793}," max_exemplars",{"type":40,"tag":1377,"props":3795,"children":3796},{"style":2589},[3797],{"type":46,"value":1366},{"type":40,"tag":1377,"props":3799,"children":3801},{"style":3800},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[3802],{"type":46,"value":3803}," 100000\n",{"type":40,"tag":49,"props":3805,"children":3806},{},[3807],{"type":46,"value":3808},"Use exemplars for:",{"type":40,"tag":182,"props":3810,"children":3811},{},[3812,3822,3832],{"type":40,"tag":186,"props":3813,"children":3814},{},[3815,3820],{"type":40,"tag":92,"props":3816,"children":3818},{"className":3817},[],[3819],{"type":46,"value":852},{"type":46,"value":3821}," correlation (Tempo, Jaeger)",{"type":40,"tag":186,"props":3823,"children":3824},{},[3825,3830],{"type":40,"tag":92,"props":3826,"children":3828},{"className":3827},[],[3829],{"type":46,"value":845},{"type":46,"value":3831}," for specific debug lookups",{"type":40,"tag":186,"props":3833,"children":3834},{},[3835],{"type":46,"value":3836},"Any sparse \"useful when you need it\" key",{"type":40,"tag":49,"props":3838,"children":3839},{},[3840],{"type":46,"value":3841},"Query exemplars via Grafana's exemplars-on-graph feature, not via PromQL aggregation.",{"type":40,"tag":109,"props":3843,"children":3844},{},[],{"type":40,"tag":113,"props":3846,"children":3848},{"id":3847},"the-8020-rule",[3849],{"type":46,"value":3850},"The 80\u002F20 Rule",{"type":40,"tag":49,"props":3852,"children":3853},{},[3854],{"type":46,"value":3855},"The most impactful improvements almost always come from these five changes:",{"type":40,"tag":275,"props":3857,"children":3858},{},[3859,3889,3899,3954,3981],{"type":40,"tag":186,"props":3860,"children":3861},{},[3862,3867,3869,3874,3876,3881,3882,3887],{"type":40,"tag":60,"props":3863,"children":3864},{},[3865],{"type":46,"value":3866},"Drop unbounded labels at the app layer",{"type":46,"value":3868}," — ",{"type":40,"tag":92,"props":3870,"children":3872},{"className":3871},[],[3873],{"type":46,"value":397},{"type":46,"value":3875}," (untemplated), ",{"type":40,"tag":92,"props":3877,"children":3879},{"className":3878},[],[3880],{"type":46,"value":838},{"type":46,"value":470},{"type":40,"tag":92,"props":3883,"children":3885},{"className":3884},[],[3886],{"type":46,"value":903},{"type":46,"value":3888},". Single biggest win.",{"type":40,"tag":186,"props":3890,"children":3891},{},[3892,3897],{"type":40,"tag":60,"props":3893,"children":3894},{},[3895],{"type":46,"value":3896},"Trim histogram label cardinality before anything else",{"type":46,"value":3898}," — 14× amplification on every histogram.",{"type":40,"tag":186,"props":3900,"children":3901},{},[3902,3924,3926,3931,3933,3938,3940,3945,3947,3952],{"type":40,"tag":60,"props":3903,"children":3904},{},[3905,3906,3911,3912,3917,3918,3923],{"type":46,"value":2169},{"type":40,"tag":92,"props":3907,"children":3909},{"className":3908},[],[3910],{"type":46,"value":134},{"type":46,"value":2328},{"type":40,"tag":92,"props":3913,"children":3915},{"className":3914},[],[3916],{"type":46,"value":142},{"type":46,"value":2328},{"type":40,"tag":92,"props":3919,"children":3921},{"className":3920},[],[3922],{"type":46,"value":2298},{"type":46,"value":2176},{"type":46,"value":3925}," — let them come from scrape targets, and add a stable ",{"type":40,"tag":92,"props":3927,"children":3929},{"className":3928},[],[3930],{"type":46,"value":676},{"type":46,"value":3932}," target label to aggregate on. (Never ",{"type":40,"tag":68,"props":3934,"children":3935},{},[3936],{"type":46,"value":3937},"drop",{"type":46,"value":3939}," the real ",{"type":40,"tag":92,"props":3941,"children":3943},{"className":3942},[],[3944],{"type":46,"value":134},{"type":46,"value":3946}," at scrape to cut cardinality — if ",{"type":40,"tag":92,"props":3948,"children":3950},{"className":3949},[],[3951],{"type":46,"value":134},{"type":46,"value":3953},"-level series are too expensive, use Adaptive Metrics.)",{"type":40,"tag":186,"props":3955,"children":3956},{},[3957,3979],{"type":40,"tag":60,"props":3958,"children":3959},{},[3960,3962,3967,3968,3973,3974],{"type":46,"value":3961},"Use info metrics for ",{"type":40,"tag":92,"props":3963,"children":3965},{"className":3964},[],[3966],{"type":46,"value":802},{"type":46,"value":764},{"type":40,"tag":92,"props":3969,"children":3971},{"className":3970},[],[3972],{"type":46,"value":816},{"type":46,"value":764},{"type":40,"tag":92,"props":3975,"children":3977},{"className":3976},[],[3978],{"type":46,"value":809},{"type":46,"value":3980}," — eliminates deploy-driven churn.",{"type":40,"tag":186,"props":3982,"children":3983},{},[3984,3996,3997,4002,4003,4008,4009,4014,4015,4020],{"type":40,"tag":60,"props":3985,"children":3986},{},[3987,3989,3994],{"type":46,"value":3988},"Set target labels via ",{"type":40,"tag":92,"props":3990,"children":3992},{"className":3991},[],[3993],{"type":46,"value":981},{"type":46,"value":3995},", not app code",{"type":46,"value":3868},{"type":40,"tag":92,"props":3998,"children":4000},{"className":3999},[],[4001],{"type":46,"value":570},{"type":46,"value":470},{"type":40,"tag":92,"props":4004,"children":4006},{"className":4005},[],[4007],{"type":46,"value":617},{"type":46,"value":470},{"type":40,"tag":92,"props":4010,"children":4012},{"className":4011},[],[4013],{"type":46,"value":1835},{"type":46,"value":470},{"type":40,"tag":92,"props":4016,"children":4018},{"className":4017},[],[4019],{"type":46,"value":669},{"type":46,"value":4021}," should never be emitted by the application.",{"type":40,"tag":49,"props":4023,"children":4024},{},[4025],{"type":46,"value":4026},"Focus on these before anything else.",{"type":40,"tag":109,"props":4028,"children":4029},{},[],{"type":40,"tag":113,"props":4031,"children":4033},{"id":4032},"labels-to-avoid-quick-reference",[4034],{"type":46,"value":4035},"Labels to Avoid — Quick Reference",{"type":40,"tag":528,"props":4037,"children":4038},{},[4039,4058],{"type":40,"tag":532,"props":4040,"children":4041},{},[4042],{"type":40,"tag":536,"props":4043,"children":4044},{},[4045,4049,4053],{"type":40,"tag":540,"props":4046,"children":4047},{},[4048],{"type":46,"value":1672},{"type":40,"tag":540,"props":4050,"children":4051},{},[4052],{"type":46,"value":3426},{"type":40,"tag":540,"props":4054,"children":4055},{},[4056],{"type":46,"value":4057},"Alternative",{"type":40,"tag":555,"props":4059,"children":4060},{},[4061,4095,4121,4154,4194,4229,4268,4296,4338,4356],{"type":40,"tag":536,"props":4062,"children":4063},{},[4064,4080,4084],{"type":40,"tag":562,"props":4065,"children":4066},{},[4067,4072,4073,4078],{"type":40,"tag":92,"props":4068,"children":4070},{"className":4069},[],[4071],{"type":46,"value":838},{"type":46,"value":470},{"type":40,"tag":92,"props":4074,"children":4076},{"className":4075},[],[4077],{"type":46,"value":874},{"type":46,"value":4079}," (large tenant base)",{"type":40,"tag":562,"props":4081,"children":4082},{},[4083],{"type":46,"value":857},{"type":40,"tag":562,"props":4085,"children":4086},{},[4087,4089],{"type":46,"value":4088},"Exemplars; aggregate by ",{"type":40,"tag":92,"props":4090,"children":4092},{"className":4091},[],[4093],{"type":46,"value":4094},"tenant_tier",{"type":40,"tag":536,"props":4096,"children":4097},{},[4098,4112,4116],{"type":40,"tag":562,"props":4099,"children":4100},{},[4101,4106,4107],{"type":40,"tag":92,"props":4102,"children":4104},{"className":4103},[],[4105],{"type":46,"value":845},{"type":46,"value":470},{"type":40,"tag":92,"props":4108,"children":4110},{"className":4109},[],[4111],{"type":46,"value":852},{"type":40,"tag":562,"props":4113,"children":4114},{},[4115],{"type":46,"value":857},{"type":40,"tag":562,"props":4117,"children":4118},{},[4119],{"type":46,"value":4120},"Exemplars",{"type":40,"tag":536,"props":4122,"children":4123},{},[4124,4140,4144],{"type":40,"tag":562,"props":4125,"children":4126},{},[4127,4132,4133,4138],{"type":40,"tag":92,"props":4128,"children":4130},{"className":4129},[],[4131],{"type":46,"value":397},{"type":46,"value":764},{"type":40,"tag":92,"props":4134,"children":4136},{"className":4135},[],[4137],{"type":46,"value":770},{"type":46,"value":4139}," (raw URLs)",{"type":40,"tag":562,"props":4141,"children":4142},{},[4143],{"type":46,"value":857},{"type":40,"tag":562,"props":4145,"children":4146},{},[4147,4149],{"type":46,"value":4148},"Template in code: ",{"type":40,"tag":92,"props":4150,"children":4152},{"className":4151},[],[4153],{"type":46,"value":788},{"type":40,"tag":536,"props":4155,"children":4156},{},[4157,4177,4181],{"type":40,"tag":562,"props":4158,"children":4159},{},[4160,4165,4166,4171,4172],{"type":40,"tag":92,"props":4161,"children":4163},{"className":4162},[],[4164],{"type":46,"value":903},{"type":46,"value":470},{"type":40,"tag":92,"props":4167,"children":4169},{"className":4168},[],[4170],{"type":46,"value":910},{"type":46,"value":470},{"type":40,"tag":92,"props":4173,"children":4175},{"className":4174},[],[4176],{"type":46,"value":917},{"type":40,"tag":562,"props":4178,"children":4179},{},[4180],{"type":46,"value":922},{"type":40,"tag":562,"props":4182,"children":4183},{},[4184,4186,4192],{"type":46,"value":4185},"Bounded ",{"type":40,"tag":92,"props":4187,"children":4189},{"className":4188},[],[4190],{"type":46,"value":4191},"error_type",{"type":46,"value":4193}," enum",{"type":40,"tag":536,"props":4195,"children":4196},{},[4197,4219,4224],{"type":40,"tag":562,"props":4198,"children":4199},{},[4200,4205,4206,4211,4212,4217],{"type":40,"tag":92,"props":4201,"children":4203},{"className":4202},[],[4204],{"type":46,"value":802},{"type":46,"value":470},{"type":40,"tag":92,"props":4207,"children":4209},{"className":4208},[],[4210],{"type":46,"value":816},{"type":46,"value":470},{"type":40,"tag":92,"props":4213,"children":4215},{"className":4214},[],[4216],{"type":46,"value":809},{"type":46,"value":4218}," (on every metric)",{"type":40,"tag":562,"props":4220,"children":4221},{},[4222],{"type":46,"value":4223},"Churn on every deploy",{"type":40,"tag":562,"props":4225,"children":4226},{},[4227],{"type":46,"value":4228},"Info metric pattern",{"type":40,"tag":536,"props":4230,"children":4231},{},[4232,4244,4249],{"type":40,"tag":562,"props":4233,"children":4234},{},[4235,4237,4242],{"type":46,"value":4236},"App-emitted ",{"type":40,"tag":92,"props":4238,"children":4240},{"className":4239},[],[4241],{"type":46,"value":134},{"type":46,"value":4243}," (duplicating SD)",{"type":40,"tag":562,"props":4245,"children":4246},{},[4247],{"type":46,"value":4248},"Should come from K8s service discovery, not code",{"type":40,"tag":562,"props":4250,"children":4251},{},[4252,4254,4259,4261,4266],{"type":46,"value":4253},"Stop emitting it in code; keep the discovered ",{"type":40,"tag":92,"props":4255,"children":4257},{"className":4256},[],[4258],{"type":46,"value":134},{"type":46,"value":4260},". Never drop the real ",{"type":40,"tag":92,"props":4262,"children":4264},{"className":4263},[],[4265],{"type":46,"value":134},{"type":46,"value":4267}," to cut cardinality — use Adaptive Metrics",{"type":40,"tag":536,"props":4269,"children":4270},{},[4271,4280,4285],{"type":40,"tag":562,"props":4272,"children":4273},{},[4274,4279],{"type":40,"tag":92,"props":4275,"children":4277},{"className":4276},[],[4278],{"type":46,"value":2239},{"type":46,"value":647},{"type":40,"tag":562,"props":4281,"children":4282},{},[4283],{"type":46,"value":4284},"Unbounded; regenerates on restart",{"type":40,"tag":562,"props":4286,"children":4287},{},[4288,4290,4295],{"type":46,"value":4289},"Never map it into a label in the first place (leave it out of ",{"type":40,"tag":92,"props":4291,"children":4293},{"className":4292},[],[4294],{"type":46,"value":981},{"type":46,"value":790},{"type":40,"tag":536,"props":4297,"children":4298},{},[4299,4321,4326],{"type":40,"tag":562,"props":4300,"children":4301},{},[4302,4304,4309,4310,4315,4316],{"type":46,"value":4303},"Application-emitted ",{"type":40,"tag":92,"props":4305,"children":4307},{"className":4306},[],[4308],{"type":46,"value":142},{"type":46,"value":470},{"type":40,"tag":92,"props":4311,"children":4313},{"className":4312},[],[4314],{"type":46,"value":2298},{"type":46,"value":470},{"type":40,"tag":92,"props":4317,"children":4319},{"className":4318},[],[4320],{"type":46,"value":2305},{"type":40,"tag":562,"props":4322,"children":4323},{},[4324],{"type":46,"value":4325},"Should come from scrape target",{"type":40,"tag":562,"props":4327,"children":4328},{},[4329,4331,4336],{"type":46,"value":4330},"Stop emitting in code (removing an ",{"type":40,"tag":68,"props":4332,"children":4333},{},[4334],{"type":46,"value":4335},"exact",{"type":46,"value":4337}," target-label duplicate at scrape is the only safe drop)",{"type":40,"tag":536,"props":4339,"children":4340},{},[4341,4346,4351],{"type":40,"tag":562,"props":4342,"children":4343},{},[4344],{"type":46,"value":4345},"Dynamically-named label keys",{"type":40,"tag":562,"props":4347,"children":4348},{},[4349],{"type":46,"value":4350},"Cannot be bounded",{"type":40,"tag":562,"props":4352,"children":4353},{},[4354],{"type":46,"value":4355},"Use fixed keys with bounded values",{"type":40,"tag":536,"props":4357,"children":4358},{},[4359,4371,4376],{"type":40,"tag":562,"props":4360,"children":4361},{},[4362,4364,4369],{"type":46,"value":4363},"Raw ",{"type":40,"tag":92,"props":4365,"children":4367},{"className":4366},[],[4368],{"type":46,"value":405},{"type":46,"value":4370}," on histograms",{"type":40,"tag":562,"props":4372,"children":4373},{},[4374],{"type":46,"value":4375},"14× amplification",{"type":40,"tag":562,"props":4377,"children":4378},{},[4379,4381,4387,4388,4393,4394,4400,4401,4407],{"type":46,"value":4380},"Bucket to ",{"type":40,"tag":92,"props":4382,"children":4384},{"className":4383},[],[4385],{"type":46,"value":4386},"status_class",{"type":46,"value":2126},{"type":40,"tag":92,"props":4389,"children":4391},{"className":4390},[],[4392],{"type":46,"value":3095},{"type":46,"value":470},{"type":40,"tag":92,"props":4395,"children":4397},{"className":4396},[],[4398],{"type":46,"value":4399},"4xx",{"type":46,"value":470},{"type":40,"tag":92,"props":4402,"children":4404},{"className":4403},[],[4405],{"type":46,"value":4406},"5xx",{"type":46,"value":790},{"type":40,"tag":109,"props":4409,"children":4410},{},[],{"type":40,"tag":113,"props":4412,"children":4414},{"id":4413},"when-to-route-elsewhere",[4415],{"type":46,"value":4416},"When to Route Elsewhere",{"type":40,"tag":182,"props":4418,"children":4419},{},[4420,4450,4467,4482,4497],{"type":40,"tag":186,"props":4421,"children":4422},{},[4423,4428,4429,4434,4436,4441,4443,4448],{"type":40,"tag":60,"props":4424,"children":4425},{},[4426],{"type":46,"value":4427},"\"Reduce my Grafana Cloud bill\"",{"type":46,"value":764},{"type":40,"tag":60,"props":4430,"children":4431},{},[4432],{"type":46,"value":4433},"\"reduce cardinality on series already ingested\"",{"type":46,"value":4435}," → engage ",{"type":40,"tag":92,"props":4437,"children":4439},{"className":4438},[],[4440],{"type":46,"value":97},{"type":46,"value":4442}," skill (post-ingest aggregation rules — the safe, counter-reset-aware way; never ",{"type":40,"tag":92,"props":4444,"children":4446},{"className":4445},[],[4447],{"type":46,"value":2269},{"type":46,"value":4449}," distinguishing labels at scrape)",{"type":40,"tag":186,"props":4451,"children":4452},{},[4453,4458,4459,4465],{"type":40,"tag":60,"props":4454,"children":4455},{},[4456],{"type":46,"value":4457},"\"Which metrics are driving my DPM?\"",{"type":46,"value":4435},{"type":40,"tag":92,"props":4460,"children":4462},{"className":4461},[],[4463],{"type":46,"value":4464},"dpm-finder",{"type":46,"value":4466}," skill",{"type":40,"tag":186,"props":4468,"children":4469},{},[4470,4475,4476,4481],{"type":40,"tag":60,"props":4471,"children":4472},{},[4473],{"type":46,"value":4474},"\"My Prometheus is OOMing \u002F scraping is failing right now\"",{"type":46,"value":4435},{"type":40,"tag":92,"props":4477,"children":4479},{"className":4478},[],[4480],{"type":46,"value":105},{"type":46,"value":4466},{"type":40,"tag":186,"props":4483,"children":4484},{},[4485,4490,4491,4496],{"type":40,"tag":60,"props":4486,"children":4487},{},[4488],{"type":46,"value":4489},"\"How do I write the query to find the bad metric?\"",{"type":46,"value":4435},{"type":40,"tag":92,"props":4492,"children":4494},{"className":4493},[],[4495],{"type":46,"value":1371},{"type":46,"value":4466},{"type":40,"tag":186,"props":4498,"children":4499},{},[4500,4505,4506,4512],{"type":40,"tag":60,"props":4501,"children":4502},{},[4503],{"type":46,"value":4504},"\"How do I configure relabel rules in Alloy?\"",{"type":46,"value":4435},{"type":40,"tag":92,"props":4507,"children":4509},{"className":4508},[],[4510],{"type":46,"value":4511},"alloy",{"type":46,"value":4466},{"type":40,"tag":49,"props":4514,"children":4515},{},[4516,4518,4523,4525,4530,4531,4536],{"type":46,"value":4517},"This skill's lane is ",{"type":40,"tag":60,"props":4519,"children":4520},{},[4521],{"type":46,"value":4522},"strategy and design",{"type":46,"value":4524},". Other skills own ",{"type":40,"tag":60,"props":4526,"children":4527},{},[4528],{"type":46,"value":4529},"diagnosis",{"type":46,"value":203},{"type":40,"tag":60,"props":4532,"children":4533},{},[4534],{"type":46,"value":4535},"operational remediation",{"type":46,"value":107},{"type":40,"tag":4538,"props":4539,"children":4540},"style",{},[4541],{"type":46,"value":4542},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4544,"total":4734},[4545,4564,4583,4603,4618,4634,4647,4662,4679,4694,4707,4722],{"slug":4546,"name":4546,"fn":4547,"description":4548,"org":4549,"tags":4550,"stars":4561,"repoUrl":4562,"updatedAt":4563},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4551,4554,4557,4560],{"name":4552,"slug":4553,"type":15},"Distributed Tracing","distributed-tracing",{"name":4555,"slug":4556,"type":15},"Frontend","frontend",{"name":4558,"slug":4559,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":4565,"name":4565,"fn":4566,"description":4567,"org":4568,"tags":4569,"stars":4580,"repoUrl":4581,"updatedAt":4582},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4570,4573,4576,4579],{"name":4571,"slug":4572,"type":15},"API Development","api-development",{"name":4574,"slug":4575,"type":15},"Authentication","authentication",{"name":4577,"slug":4578,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":4584,"name":4584,"fn":4585,"description":4586,"org":4587,"tags":4588,"stars":4580,"repoUrl":4581,"updatedAt":4602},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4589,4592,4595,4596,4599],{"name":4590,"slug":4591,"type":15},"CSV","csv",{"name":4593,"slug":4594,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":4597,"slug":4598,"type":15},"GraphQL","graphql",{"name":4600,"slug":4601,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":4604,"name":4604,"fn":4605,"description":4606,"org":4607,"tags":4608,"stars":4615,"repoUrl":4616,"updatedAt":4617},"agento11y","manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4609,4612,4613,4614],{"name":4610,"slug":4611,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":4558,"slug":4559,"type":15},{"name":13,"slug":14,"type":15},430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-25T05:30:40.29622",{"slug":4619,"name":4619,"fn":4620,"description":4621,"org":4622,"tags":4623,"stars":4615,"repoUrl":4616,"updatedAt":4633},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4624,4625,4626,4629,4632],{"name":4610,"slug":4611,"type":15},{"name":9,"slug":8,"type":15},{"name":4627,"slug":4628,"type":15},"Instrumentation","instrumentation",{"name":4630,"slug":4631,"type":15},"LLM","llm",{"name":13,"slug":14,"type":15},"2026-07-31T05:53:52.580237",{"slug":4635,"name":4635,"fn":4636,"description":4637,"org":4638,"tags":4639,"stars":4615,"repoUrl":4616,"updatedAt":4646},"agento11y-prod-setup","setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4640,4641,4644,4645],{"name":4610,"slug":4611,"type":15},{"name":4642,"slug":4643,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:53.576347",{"slug":4648,"name":4648,"fn":4649,"description":4650,"org":4651,"tags":4652,"stars":4615,"repoUrl":4616,"updatedAt":4661},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4653,4654,4655,4658],{"name":4610,"slug":4611,"type":15},{"name":9,"slug":8,"type":15},{"name":4656,"slug":4657,"type":15},"QA","qa",{"name":4659,"slug":4660,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":4663,"name":4663,"fn":4664,"description":4665,"org":4666,"tags":4667,"stars":4615,"repoUrl":4616,"updatedAt":4678},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4668,4671,4674,4677],{"name":4669,"slug":4670,"type":15},"Dashboards","dashboards",{"name":4672,"slug":4673,"type":15},"Data Visualization","data-visualization",{"name":4675,"slug":4676,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":4680,"name":4680,"fn":4681,"description":4682,"org":4683,"tags":4684,"stars":4615,"repoUrl":4616,"updatedAt":4693},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4685,4688,4689,4692],{"name":4686,"slug":4687,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":4690,"slug":4691,"type":15},"Incident Response","incident-response",{"name":13,"slug":14,"type":15},"2026-07-18T05:11:10.445428",{"slug":4695,"name":4695,"fn":4696,"description":4697,"org":4698,"tags":4699,"stars":4615,"repoUrl":4616,"updatedAt":4706},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4700,4701,4702,4705],{"name":4686,"slug":4687,"type":15},{"name":9,"slug":8,"type":15},{"name":4703,"slug":4704,"type":15},"Graph Analysis","graph-analysis",{"name":13,"slug":14,"type":15},"2026-07-25T05:30:39.380934",{"slug":4708,"name":4708,"fn":4709,"description":4710,"org":4711,"tags":4712,"stars":4615,"repoUrl":4616,"updatedAt":4721},"gcx","manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4713,4716,4717,4718],{"name":4714,"slug":4715,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":4558,"slug":4559,"type":15},{"name":4719,"slug":4720,"type":15},"Operations","operations","2026-07-31T05:53:50.587304",{"slug":4723,"name":4723,"fn":4724,"description":4725,"org":4726,"tags":4727,"stars":4615,"repoUrl":4616,"updatedAt":4733},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4728,4729,4730],{"name":4714,"slug":4715,"type":15},{"name":9,"slug":8,"type":15},{"name":4731,"slug":4732,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80,{"items":4736,"total":4837},[4737,4749,4764,4780,4793,4808,4822],{"slug":97,"name":97,"fn":4738,"description":4739,"org":4740,"tags":4741,"stars":23,"repoUrl":24,"updatedAt":4748},"optimize Grafana Cloud metrics costs","Cut Grafana Cloud Metrics cost by shrinking active-series count with Adaptive Metrics aggregation rules — auto-recommendations from query history, custom exact\u002Fregex rules, label-drop config, unused-metric detection, and Alloy remote_write fallback. Use when investigating a high Mimir\u002FGrafana Cloud bill, hunting high-cardinality labels (`pod_uid`, `service_instance_id`, `version`), pre-aggregating counters\u002Fgauges, dropping unused metrics, or measuring `grafanacloud_instance_active_series` before\u002Fafter — even when the user says \"reduce cardinality\", \"too many series\", \"metrics spend\", \"active series count is exploding\", or \"drop the version label\" without naming Adaptive Metrics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4742,4745,4746,4747],{"name":4743,"slug":4744,"type":15},"Cost Optimization","cost-optimization",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},{"name":13,"slug":14,"type":15},"2026-07-12T07:44:27.451068",{"slug":4750,"name":4750,"fn":4751,"description":4752,"org":4753,"tags":4754,"stars":23,"repoUrl":24,"updatedAt":4763},"admin","manage Grafana Cloud accounts and RBAC","Manage Grafana Cloud accounts — organizations, stacks, RBAC roles and assignments, SSO\u002FSAML\u002FOAuth\u002FGitHub auth, service accounts for CI\u002FCD, user invites, team membership, and API-driven provisioning. Creates stacks via the Cloud API, mints service-account tokens, applies role assignments, configures SSO providers, and provisions teams\u002Ffolders\u002Fdashboards via Terraform. Use when managing Grafana Cloud access, configuring SSO\u002FSAML\u002FOAuth, setting up service accounts for Terraform\u002FCI\u002FCD, assigning RBAC roles, inviting users, managing multiple stacks or organizations, provisioning cloud resources via API or Terraform, or auditing admin actions — even when the user says \"set up SSO\", \"create a stack\", \"make a service account\", or \"onboard a team\" without explicitly saying \"admin\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4755,4758,4761,4762],{"name":4756,"slug":4757,"type":15},"Access Control","access-control",{"name":4759,"slug":4760,"type":15},"Auth","auth",{"name":9,"slug":8,"type":15},{"name":4719,"slug":4720,"type":15},"2026-07-12T07:44:12.078436",{"slug":4765,"name":4765,"fn":4766,"description":4767,"org":4768,"tags":4769,"stars":23,"repoUrl":24,"updatedAt":4779},"admission-control","implement admission control webhooks","Use when the user asks to \"write a validator\", \"add validation\", \"implement admission control\", \"write a mutating webhook\", \"add a mutation handler\", \"validate incoming resources\", \"implement admission logic\", \"add admission webhooks\", \"write ingress validation\", or asks how to validate or mutate resources before they are persisted in a grafana-app-sdk app. Provides guidance on implementing validation and mutation admission handlers for grafana-app-sdk apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4770,4773,4776],{"name":4771,"slug":4772,"type":15},"Architecture","architecture",{"name":4774,"slug":4775,"type":15},"Security","security",{"name":4777,"slug":4778,"type":15},"Validation","validation","2026-07-12T07:45:06.148973",{"slug":4781,"name":4781,"fn":4782,"description":4783,"org":4784,"tags":4785,"stars":23,"repoUrl":24,"updatedAt":4792},"alerting-irm","configure Grafana Alerting and Incident Management","Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack\u002FPagerDuty\u002Femail\u002Fwebhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty\u002FSlack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says \"page me on errors\", \"alert me when X happens\", \"route this to the platform team\", or \"set up an SLO\" without naming Alerting or IRM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4786,4789,4790,4791],{"name":4787,"slug":4788,"type":15},"Alerting","alerting",{"name":9,"slug":8,"type":15},{"name":4690,"slug":4691,"type":15},{"name":4558,"slug":4559,"type":15},"2026-07-12T07:44:02.393397",{"slug":4511,"name":4511,"fn":4794,"description":4795,"org":4796,"tags":4797,"stars":23,"repoUrl":24,"updatedAt":4807},"build unified telemetry pipelines with Grafana Alloy","Build a unified telemetry pipeline with Grafana Alloy — one OpenTelemetry-compatible binary that collects metrics, logs, traces, and profiles and ships to Grafana Cloud \u002F Prometheus \u002F Loki \u002F Tempo \u002F Pyroscope. Covers the Alloy config language (blocks, `sys.env`, component refs), `prometheus.scrape` → `remote_write`, `loki.source.file` + `loki.process` → `loki.write`, `otelcol.receiver.otlp` → `otelcol.exporter.otlp`, `pyroscope.scrape`, K8s \u002F Docker \u002F EC2 discovery, relabeling, modules (`import.file\u002Fgit\u002Fhttp`), clustering, Fleet Management `remotecfg`, the Alloy UI at `:12345`, and `alloy fmt` \u002F `alloy validate`. Use when writing a `config.alloy`, replacing Grafana Agent \u002F OTel Collector, scraping K8s pods, parsing logs, ingesting OTLP, or debugging \"Alloy isn't sending anything\" — even when the user says \"set up the agent\", \"write me a scrape config\", \"drop these logs before sending\", or \"OTel collector config\" without naming Alloy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4798,4799,4802,4803,4804],{"name":9,"slug":8,"type":15},{"name":4800,"slug":4801,"type":15},"Logs","logs",{"name":21,"slug":22,"type":15},{"name":13,"slug":14,"type":15},{"name":4805,"slug":4806,"type":15},"OpenTelemetry","opentelemetry","2026-07-12T07:43:54.817139",{"slug":4809,"name":4809,"fn":4810,"description":4811,"org":4812,"tags":4813,"stars":23,"repoUrl":24,"updatedAt":4821},"app-observability","monitor application performance in Grafana Cloud","Get RED metrics + service maps + frontend RUM + AI\u002FLLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50\u002Fp95\u002Fp99 latency, exemplar-to-trace, traces-to-logs \u002F profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token \u002F cost \u002F latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says \"set up APM\", \"show service map\", \"monitor browser perf\", \"session replay\", \"RUM SDK\", or \"watch our OpenAI bill\" without naming App \u002F Frontend \u002F AI Observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4814,4817,4818,4819,4820],{"name":4815,"slug":4816,"type":15},"APM","apm",{"name":4552,"slug":4553,"type":15},{"name":9,"slug":8,"type":15},{"name":4630,"slug":4631,"type":15},{"name":13,"slug":14,"type":15},"2026-07-12T07:44:34.500406",{"slug":4823,"name":4823,"fn":4824,"description":4825,"org":4826,"tags":4827,"stars":23,"repoUrl":24,"updatedAt":4836},"app-sdk-concepts","scaffold and configure Grafana apps","Use when starting any grafana-app-sdk work — scaffolding a Grafana app, initializing a Grafana App Platform app, picking a deployment mode (standalone operator \u002F grafana\u002Fapps \u002F frontend-only), wiring app-specific config, or onboarding to the SDK. Covers `grafana-app-sdk` CLI install, `project init` per deployment mode, project layout, the schema-centric workflow (CUE kinds → generated code → reconciler\u002Fadmission logic), and `SpecificConfig` for env-driven app configuration. Use even if the user just says \"make a new app\", \"Grafana app platform\", \"kinds and watchers\", \"operator scaffold\", or asks about `apps\u002F` inside the Grafana repo, without naming the SDK explicitly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4828,4829,4832,4835],{"name":4771,"slug":4772,"type":15},{"name":4830,"slug":4831,"type":15},"Deployment","deployment",{"name":4833,"slug":4834,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},"2026-07-12T07:45:08.595757",48]