[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-loki-label-analyzer":3,"mdc--ddm5ju-key":34,"related-repo-grafana-loki-label-analyzer":3896,"related-org-grafana-loki-label-analyzer":4010},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"loki-label-analyzer","audit and design Grafana Loki labels","Expert evaluator for Grafana Loki label strategy. Audits, designs, and improves label schemas using cardinality scoring, access-pattern alignment, static vs. dynamic label rules, and consistency checks. Use when the user asks to evaluate, audit, design, or improve a Loki label strategy — or asks why their Loki queries are slow.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Performance","performance",{"name":20,"slug":21,"type":15},"Logs","logs",{"name":9,"slug":8,"type":15},189,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills","2026-07-25T05:30:52.497271","Apache-2.0",16,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],null,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fgrafana-cloud\u002Floki-label-analyzer","---\nname: loki-label-analyzer\nlicense: Apache-2.0\ndescription: >\n  Expert evaluator for Grafana Loki label strategy. Audits, designs, and improves\n  label schemas using cardinality scoring, access-pattern alignment, static vs.\n  dynamic label rules, and consistency checks. Use when the user asks to evaluate,\n  audit, design, or improve a Loki label strategy — or asks why their Loki queries\n  are slow.\n---\n\n# Loki Label Strategy Evaluator\n\nYou are an expert in Grafana Loki label strategy. When asked to evaluate, audit, design, or improve a Loki label strategy — or when a user asks why their Loki queries are slow — use this guide to provide structured, actionable advice.\n\n---\n\n## Core Concepts\n\n**Streams** are the fundamental unit in Loki. Each unique combination of label key-value pairs creates a new stream. Too many streams = performance problems. Too few = broad, slow queries.\n\n**Cardinality** = the number of unique values a label can have. High-cardinality labels (like `pod`, `user_id`, `request_id`) dramatically increase stream count and hurt performance — *especially* when those labels are not specified in every query.\n\n**The dual impact rule**: High-cardinality labels hurt on both paths:\n- **Ingestion path**: More streams → larger index, higher storage costs\n- **Query path**: If a high-cardinality label exists but isn't in the query selector, Loki must scan ALL streams matching the other selectors — catastrophic for performance\n\n**The key question for any dynamic label**: \"Will this label be used in 9 out of 10 queries?\" If no → it should NOT be a label — **except** platform \u002F correlation labels (below).\n\n**Platform \u002F correlation labels are exempt from drop recommendations.** Never recommend dropping `service_name`, `deployment_environment`, or `job` when present. Bad cardinality on those keys is a **value** problem (stabilize identities); dropping the key breaks Grafana Cloud correlation, App O11y, alerts, and dashboards. Load [references\u002Fprotected-labels.md](references\u002Fprotected-labels.md) before any demote\u002F`label_keep` advice.\n\n---\n\n## Label Evaluation Framework\n\nWhen auditing a label strategy, assess each label against these criteria.\n\n### Cardinality Scoring\n\n| Label Example | Cardinality | Verdict |\n|---|---|---|\n| `service_name` \u002F `deployment_environment` \u002F `job` | Any | ✅ Keep key — remediate values if high-card (never drop) |\n| `env` (prod\u002Fstaging\u002Fdev) | 2–5 values | ✅ Good |\n| `level` (info\u002Fwarn\u002Ferror) | 3–6 values | ✅ Good |\n| `namespace` (K8s) | Tens | ✅ Acceptable |\n| `instance` \u002F `hostname` | Hundreds–thousands | ⚠️ Evaluate access patterns |\n| `pod` | Thousands + transient | ⚠️ Demote off index (structured metadata) — migrate selectors first |\n| `user_id`, `request_id` | Unbounded | ❌ Never use as label |\n\n### Access Pattern Alignment\nFor each label, ask:\n- Is this label on the protected allowlist? If yes → Keep key; remediate values only ([protected-labels.md](references\u002Fprotected-labels.md))\n- Is this label used as a selector in most queries targeting these logs?\n- Does this label logically segment data in the way users think about it?\n- Would demoting this label break alerts, dashboards, LBAC, or correlation without a migration plan?\n- Would demoting this label force users to scan dramatically more data?\n\n### Static vs. Dynamic Label Values\n- **Static labels** (values don't change per log line, e.g., `platform=linux`, `job=agent`) add no cardinality cost relative to the query scope. Use freely for LBAC, exploration, and alert routing.\n- **Dynamic labels** (values change per log line) must be bounded. Keep possible values in the single digits or low tens.\n\n### Consistency Check\n- Are label names consistent across services? (case-sensitive — `Level` ≠ `level`)\n- Are label values normalized? (`INFO`, `info`, `Info` should all become `info`)\n- Is there a naming convention? (pick one: `snake_case` or `camelCase` — be consistent)\n\n---\n\n## Evaluation Output Format\n\nWhen auditing a label set, produce a report in the structure below.\n\n**Hard requirements before finalizing any audit report:**\n\n1. **Disclaimer (mandatory, first body section):** Load [references\u002Fdisclaimer.md](references\u002Fdisclaimer.md) and paste its two paragraphs **verbatim** under a `### Disclaimer` heading. An empty Disclaimer heading is a failed report — do not ship the audit until both paragraphs are present. Never paraphrase, summarize, or omit this text.\n2. **Protected labels:** Before recommending demote\u002Fdrop for any label, load [references\u002Fprotected-labels.md](references\u002Fprotected-labels.md). Never recommend dropping `service_name`, `deployment_environment`, or `job` when present — only value remediation. Include a **Downstream dependency check** covering alerts, dashboards, LBAC, and correlation.\n3. **Cost Impact Analysis:** Include when Grafana Cloud usage metrics are available; if they are not, state what is missing and still give qualitative A\u002FB\u002FC guidance. Load [references\u002Fcost-impact.md](references\u002Fcost-impact.md) and follow its **Required report shape** (scenario cards). Do **not** paste markdown tables or panel\u002Fquery JSON into this section.\n\n**Report completion check:** Before delivering, confirm (a) the output contains the substring `Confidential Information of Raintank, Inc.` immediately after `### Disclaimer`, (b) Cost Impact Analysis uses scenario cards (A\u002FB\u002FC) with a **Billing note** opener and a bullet **Measured baseline** — not a scenario table and not `panelId`\u002F`targets` JSON, and (c) no Action cell recommends dropping an allowlisted correlation label. If (a) is missing, paste from [references\u002Fdisclaimer.md](references\u002Fdisclaimer.md) and re-emit. If (b) fails, rewrite Cost Impact from [references\u002Fcost-impact.md](references\u002Fcost-impact.md). If (c) fails, rewrite Actions per [references\u002Fprotected-labels.md](references\u002Fprotected-labels.md).\n\n```\n## Loki Label Strategy Audit\n\n### Disclaimer\n[Paste BOTH paragraphs from references\u002Fdisclaimer.md HERE — never leave this heading empty]\n\n### Summary\n[1-2 sentence overall assessment]\n\n### Downstream dependency check\n[Alerts \u002F dashboards \u002F LBAC \u002F correlation that select on labels proposed for demote or rename — or \"unknown; confirm with customer before cutover\"]\n\n### Label Analysis\n| Label | Cardinality | Used in Queries? | Verdict | Action |\n|---|---|---|---|---|\n| service_name | High (UUID values) | Always | ✅ Keep key | Stabilize values to durable service identity — do not drop label |\n| deployment_environment | Low | Often | ✅ Keep | — |\n| job | Low–medium | Often | ✅ Keep | — |\n| pod | Very High (transient)| Rarely | ⚠️ Demote | Move to structured metadata or embed; migrate selectors first |\n\n### Estimated Impact\n- Stream count reduction: [X streams → Y streams]\n- Query performance: [describe improvement]\n- Storage impact: [if log line changes are involved]\n- Correlation impact: [none if allowlist preserved; call out if aliases need dual-write]\n\n### Cost Impact Analysis\n[Follow references\u002Fcost-impact.md Required report shape — do not invent a table]\n\n**Billing note:** Label hygiene alone does not reduce billable ingest bytes.\nStream count and query cost improve; ingest $ drops only when volume is reduced.\n\n**Measured baseline** (Grafana Cloud usage metrics):\n- Active streams: [N]\n- Billable ingest: [rate]\n- Overage: [units or $]\n- Top ingest contributor: [name + rate] (omit if unavailable)\n\n**Scenario A — Label hygiene only (this audit)**\n- Actions \u002F stream impact \u002F volume=$0 \u002F overage unchanged\n\n**Scenario B — A + approved debug\u002Ftrace drop**\n- Actions \u002F volume % \u002F $ or overage estimate \u002F customer-approval guardrail\n\n**Scenario C — B + log-line compaction**\n- Actions \u002F additional volume % \u002F highest-value target\n\n**Attribution gap:** [...]\n**Caveats:** [...]\n\n### Recommended Label Set\n[Final recommended labels — must include service_name, deployment_environment, job when present]\n\n### Migration Notes\n[How to implement changes via Alloy\u002FAgent pipeline stages; dual-write \u002F selector updates for any demote or rename]\n```\n\n---\n\n## Recommended Common Labels\n\nEvery log source should consider these base labels — all low cardinality, high query value:\n\n| Label | Purpose |\n|---|---|\n| `service_name` | Identifying the generating application (OTel `service.name` — **required for Grafana Cloud correlation \u002F App O11y**) |\n| `deployment_environment` | Deployment environment (OTel `deployment.environment`) — keep when present |\n| `job` | Collector \u002F OTel job (`namespace\u002Fservice.name` pattern common on span metrics) — keep when present |\n| `app` \u002F `service` | Legacy aliases only — prefer aligning to `service_name`; do not delete without a migration plan |\n| `env` | Environment shorthand (prod, staging, dev) when `deployment_environment` is absent |\n| `cluster` | Multi-cluster differentiation |\n| `region` | Geographic region |\n| `level` | Log severity — normalize to: `info`, `warn`, `error`, `debug` |\n| `team` \u002F `squad` | Ownership (also useful for LBAC) |\n| `source` | Log origin type (`file`, `k8s-events`, `journal`, `syslog`, etc.) |\n| `classification` | Data sensitivity level — for LBAC policies |\n\nAlways include allowlisted correlation labels in any `label_keep` list — see [references\u002Fprotected-labels.md](references\u002Fprotected-labels.md).\n\n---\n\n## Kubernetes Pod Logs\n\n### Recommended Labels\n\n| Label | Description |\n|---|---|\n| `service_name` | Stable service identity (OTel `service.name`) — **keep**; remediate UUID\u002Fephemeral values |\n| `namespace` | K8s namespace — delineates isolation boundaries |\n| `container` | Container name — low cardinality, differentiates log formats |\n| `workload` | `{controller_kind}\u002F{controller_name}` e.g. `ReplicaSet\u002Fpayment-api` — **strongly recommended** |\n\n**Why `workload` beats `app` for K8s**: Derived from `{{controller_kind}}\u002F{{controller_name}}` — static values that never change like pod names do. Unlike `app` (which may aggregate multiple workload types), `workload` is precise and predictable. Users always know exactly what value to query. Still keep `service_name` for cross-signal correlation even when using `workload`.\n\n### Labels to demote in Kubernetes (not \"never existed\")\n\n**`pod` label** ⚠️\n- Highly transient: pod names change on every restart\u002Frollout\n- Very high cardinality: 5 pods × 2 containers = 10 streams; add `pod` → 10 × N streams\n- Users almost never query for a specific pod; they query for the *workload*\n- **Solution**: Use `workload` as the index label; store `pod` in structured metadata or embed in the log line. Migrate any alerts\u002Fdashboards that select on `pod` before demoting.\n\n**`filename` label (raw K8s path)** ⚠️\n- K8s log paths contain pod UID: `\u002Fvar\u002Flog\u002Fpods\u002F{namespace}_{pod}_{pod_id}\u002F{container}\u002F{rotation}.log`\n- The `pod_id` component makes this unbounded\n- **Solution**: Normalize to `\u002Fvar\u002Flog\u002Fpods\u002F{namespace}\u002F{controller_name}\u002F{container}.log` or demote entirely after checking selectors\n\n```alloy\n\u002F\u002F Normalize K8s filename to remove pod UID\nstage.replace {\n source = \"filename\"\n expression = \"\u002Fvar\u002Flog\u002Fpods\u002F([^\u002F]+)_[^_]+_[^\u002F]+\u002F([^\u002F]+)\u002F\\\\d+\\\\.log\"\n replace = \"\u002Fvar\u002Flog\u002Fpods\u002F$1\u002F$2\u002Fcurrent.log\"\n}\n```\n\n---\n\n## Host \u002F VM \u002F Bare Metal Labels\n\nIn addition to common labels, add:\n\n| Label | Description | Notes |\n|---|---|---|\n| `instance` | Hostname of the machine | Cardinality = number of machines; acceptable for fixed infrastructure |\n| `filename` | Full path to the file being tailed | Normalize rotating filenames — strip date suffixes |\n\n```alloy\n\u002F\u002F Remove date suffixes from rotating log file names\n\u002F\u002F \u002Fvar\u002Flog\u002Fmyapp\u002Flogfile-20230927.txt → \u002Fvar\u002Flog\u002Fmyapp\u002Flogfile.txt\nstage.replace {\n source = \"filename\"\n expression = \"-\\\\d{8}(\\\\.log|\\\\.txt)$\"\n replace = \"$1\"\n}\n```\n\n---\n\n## Journal Logs\n\nWhen collecting via `loki.source.journal`, many labels are auto-discovered under `__journal__*`:\n`boot_id`, `cap_effective`, `cmdline`, `comm`, `exe`, `gid`, `hostname`, `machine_id`, `pid`, `stream_id`, `systemd_cgroup`, `systemd_invocation_id`, `systemd_slice`, `systemd_unit`, `transport`, `uid`\n\nAlmost all are high-cardinality. **Keep** `instance` (hostname) and `unit` (`systemd_unit`, e.g. `nginx.service`), plus any allowlisted correlation labels present on the stream (`service_name`, `deployment_environment`, `job`).\n\nDrop other non-allowlisted high-cardinality journal labels (not platform keys):\n```alloy\nloki.process \"journal_labels\" {\n forward_to = [...]\n stage.label_keep {\n values = [\"instance\", \"unit\", \"env\", \"cluster\", \"service_name\", \"deployment_environment\", \"job\"]\n }\n}\n```\n\n---\n\n## Structured Metadata\n\nStructured metadata attaches key-value pairs to log entries *without* making them index labels. The ideal home for high-cardinality values users occasionally need.\n\n**Requires**: Loki 2.9+, Grafana Agent\u002FAlloy. Enable via `limits_config`:\n```yaml\nlimits_config:\n allow_structured_metadata: true\n```\n\n**Good candidates for structured metadata** (not labels):\n- `pod` — K8s pod name\n- `node` — K8s worker node\n- `version` \u002F `image` \u002F `tag`\n- `trace_id` \u002F `user_id`\n- `process_id`\n- `restarted` — pod restart timestamp\n\nQuery structured metadata at query time without a parser:\n```logql\n{service_name=\"payment-api\"} | pod=\"payment-api-7f9d4b-xk2r9\"\n```\n\n---\n\n## Embedding Metadata in Log Lines\n\nWhen structured metadata isn't available, embed high-cardinality values into the log line rather than using them as labels.\n\n### Method 1: stage.template (append to log line)\n\n```alloy\nloki.process \"embed_pod\" {\n forward_to = [...]\n\n \u002F\u002F For JSON logs\n stage.match {\n selector = \"{} |~ \\\"^\\\\s*\\\\{\\\"\"\n stage.replace {\n expression = \"\\\\}$\"\n replace = \"\"\n }\n stage.template {\n source = \"log_line\"\n template = \"{{ .Entry }},\\\"_pod\\\":\\\"{{ .pod }}\\\"}\"\n }\n }\n\n \u002F\u002F For text logs\n stage.match {\n selector = \"{} !~ \\\"^\\\\s*\\\\{\\\"\"\n stage.template {\n source = \"log_line\"\n template = \"{{ .Entry }} _pod={{ .pod }}\"\n }\n }\n\n stage.output { source = \"log_line\" }\n}\n```\n\nResult: `ts=... msg=\"...\" _pod=agent-logs-cqhfk`\n\nQuery by aggregate (normal use):\n```logql\nsum(count_over_time({workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"}[1m]))\n```\n\nQuery a specific pod (edge case debugging):\n```logql\n{workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"} |= `_pod=payment-api-3`\n```\n\n### Method 2: stage.pack (JSON envelope)\n\n```alloy\nloki.process \"pack_pod\" {\n forward_to = [...]\n stage.pack {\n labels = [\"pod\"]\n ingest_timestamp = false\n }\n}\n```\n\nPacked result: `{\"_entry\": \"original log line\", \"pod\": \"agent-logs-cqhfk\"}`\n\nUnpack at query time:\n```logql\n{workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"}\n |= `agent-logs-cqhfk`\n | unpack\n```\n\n---\n\n## Performance Bottleneck Diagnosis\n\nWhen a user reports slow queries, identify where time is spent using Querier `metrics.go` logs.\n\n### Four Query Stages\n\n| Stage | Metric | High Value Means | Fix |\n|---|---|---|---|\n| Queue | `queue_time` | Not enough Queriers | Add Queriers or reduce parallelism |\n| Index | `chunk_refs_fetch_time` | Need more Index Gateway instances | Scale index-gateways; check CPU |\n| Storage | `store_chunks_download_time` | Chunks too small OR storage bottleneck | Check avg chunk size: `total_bytes \u002F cache_chunk_req` |\n| Execution | `duration - chunk_refs_fetch_time - store_chunks_download_time` | CPU-intensive regex, or too many tiny log lines | Reduce regex; add CPU; increase parallelism |\n\n**Ideally, the majority of time is spent in Execution.** If not, that indicates infrastructure or label design problems.\n\n### Checking Chunk Size\n```\navg chunk size = total_bytes \u002F cache_chunk_req\n```\nIf the result is a few hundred bytes or kilobytes (instead of megabytes), chunks are too small. This means labels are over-splitting data into too many streams. Revisit cardinality — demote non-allowlisted high-card labels or stabilize protected-label values.\n\n### Common Label-Related Performance Problems\n\n**Problem: Query scans too many streams**\n- Cause: High-cardinality labels exist but aren't specified in the query selector\n- Fix: Demote the label off the index after a migration check, or ensure queries always include it as a filter. Never demote allowlisted correlation labels — stabilize their values instead ([protected-labels.md](references\u002Fprotected-labels.md))\n\n**Problem: High `post_filter_lines` discard ratio** (`post_filter_lines \u003C\u003C total_lines`)\n- Cause: Insufficient label selectivity; query scans and discards most logs\n- Fix: Add labels matching user access patterns (`level`, `workload`, `container`, `service_name`)\n\n**Problem: Small chunks**\n- Cause: Too many labels creating too many fine-grained streams\n- Fix: Demote high-cardinality non-allowlisted labels (e.g. `pod`) to consolidate streams; remediate protected-label *values* if they are the splitter\n\n### Query Optimization Quick Wins\n1. Add `container` or `workload` to narrow scope before line filters\n2. Add `level` label + always use it in queries (filters out 94%+ of logs when searching for errors)\n3. Demote `pod` off the index → reduces stream count by ~5× in typical K8s deployments (migrate selectors first)\n4. Replace regex line filters (`|~`) with exact filters (`|=`) where possible\n5. Keep `service_name` (and peers); if values are UUIDs\u002Fephemeral, normalize to a stable identity — do not drop the key\n\n---\n\n## Alloy \u002F Agent Configuration Patterns\n\n### Normalize Log Level\n\n```alloy\nloki.process \"normalize_level\" {\n forward_to = [...]\n stage.replace { source = \"level\"; expression = \"(?i)I(nfo)?\"; replace = \"info\" }\n stage.replace { source = \"level\"; expression = \"(?i)W(arn(ing)?)?\"; replace = \"warn\" }\n stage.replace { source = \"level\"; expression = \"(?i)E(rr(or)?)?\"; replace = \"error\" }\n stage.replace { source = \"level\"; expression = \"(?i)D(ebug?)?\"; replace = \"debug\" }\n stage.labels { values = { level = \"\" } }\n}\n```\n\n### Conditional Meta-Label Extraction\n\n```alloy\n\u002F\u002F Only extract when the relevant field is present — avoids unnecessary cardinality\nloki.process \"conditional_extraction\" {\n forward_to = [...]\n stage.match {\n selector = \"{app=\\\"loki\\\"} |= \\\"component\\\"\"\n stage.logfmt { mapping = { \"component\" = \"\" } }\n stage.labels { values = { component = \"\" } }\n }\n}\n```\n\n### Enforce Approved Label Set (always use as final stage)\n\nAlways include allowlisted correlation labels when present — never omit `service_name`, `deployment_environment`, or `job` from `label_keep` ([protected-labels.md](references\u002Fprotected-labels.md)):\n\n```alloy\nloki.process \"enforce_labels\" {\n forward_to = [loki.write.default.receiver]\n \u002F\u002F ... other stages ...\n stage.label_keep {\n values = [\n \"service_name\", \"deployment_environment\", \"job\",\n \"env\", \"cluster\", \"level\", \"namespace\", \"workload\", \"container\",\n ]\n }\n}\n```\n\n### Soft Enforcement (inject \"unknown\" for missing labels)\n\n```alloy\nstage.template {\n source = \"team\"\n template = \"{{ if .Value }}{{ .Value }}{{ else }}unknown{{ end }}\"\n}\nstage.labels { values = { team = \"\" } }\n```\n\n---\n\n## Log Line Optimization\n\nByte-level reductions (timestamps, ANSI, null JSON fields) for Scenario C savings — see [references\u002Flog-line-optimization.md](references\u002Flog-line-optimization.md).\n\n---\n\n## Security & LBAC\n\nGrafana Enterprise Logs (GEL) supports Label-Based Access Control (LBAC). Any label can serve as an access control selector.\n\n**Best labels for LBAC**:\n- `classification` — data sensitivity (`public`, `restricted`, `confidential`, `top-secret`)\n- `source` — controls which teams can see which log origins\n- `team` \u002F `squad` — ownership-based access\n- `env` — environment-level restrictions\n\nStatic aggregate labels like `owner=sysadmins` or `category=database` are particularly effective: one label value gates access to many log files, rather than requiring a long allowlist of filenames or streams.\n\n---\n\n## The 80\u002F20 Rule\n\nThe most impactful improvements almost always come from these four changes:\n\n1. **Demote `pod` off the index** (structured metadata) — biggest stream reduction in K8s; migrate selectors first\n2. **Add `level` as a label AND always specify it in queries** — can eliminate 94%+ of scanned data when searching for errors\n3. **Normalize label values** — eliminates phantom duplicate streams from inconsistent casing; for `service_name`, stabilize UUID\u002Fephemeral values (never drop the key)\n4. **Normalize or demote `filename`** in K8s — highly variable paths inflate stream count significantly\n\nFocus on these before anything else. Never \"fix\" cardinality by dropping `service_name`, `deployment_environment`, or `job`.\n\n---\n\n## Labels to Avoid — Quick Reference\n\n| Label | Why | Alternative |\n|---|---|---|\n| `pod` | Transient, high card | Demote: `workload` label + `pod` in structured metadata (migrate selectors) |\n| `user_id` | Unbounded — never valid as index label | Keep only in log content |\n| `request_id` \u002F `trace_id` | Unbounded — never valid as index label | Structured metadata |\n| `filename` (raw K8s path) | Contains pod UID | Normalize or demote after selector check |\n| Unnormalized `level` | `INFO`\u002F`info`\u002F`Info` = 3 streams | Normalize at collection time |\n| UUID \u002F ephemeral `service_name` *values* | Inflates streams; key is still required | Keep key; map values to stable service identity |\n| Any dynamically-named label key | Cannot be bounded | Use fixed keys with bounded values |\n\n**Never drop:** `service_name`, `deployment_environment`, `job` — see [references\u002Fprotected-labels.md](references\u002Fprotected-labels.md).\n\n---\n\n## Cost Impact Analysis\n\nLabel hygiene alone does not cut billable ingest bytes ($0 direct). Volume savings come from enabled `stage.drop` \u002F log-line cleanup. Load [references\u002Fcost-impact.md](references\u002Fcost-impact.md) when writing the report section: use its scenario-card shape, cite scalar metrics (optional short panel ID \u002F PromQL), and never paste the agent-only reference table or panel JSON into the customer report.\n",{"data":35,"body":36},{"name":4,"license":26,"description":6},{"type":37,"children":38},"root",[39,48,54,58,65,76,118,128,153,170,226,229,235,240,247,461,467,472,507,513,551,557,633,636,642,647,655,764,837,849,852,858,863,1180,1197,1200,1206,1212,1331,1390,1396,1411,1471,1486,1530,1595,1598,1604,1609,1676,1737,1740,1746,1877,1938,1943,1997,2000,2006,2018,2036,2076,2086,2167,2172,2188,2191,2197,2202,2208,2439,2450,2455,2469,2474,2488,2494,2554,2565,2570,2601,2604,2610,2623,2629,2775,2785,2791,2800,2805,2811,2819,2837,2861,2898,2906,2933,2939,3016,3019,3025,3031,3100,3106,3181,3187,3223,3307,3313,3359,3362,3368,3379,3382,3388,3393,3402,3479,3499,3502,3508,3513,3580,3603,3606,3612,3829,3862,3865,3871,3890],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"loki-label-strategy-evaluator",[45],{"type":46,"value":47},"text","Loki Label Strategy Evaluator",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52],{"type":46,"value":53},"You are an expert in Grafana Loki label strategy. When asked to evaluate, audit, design, or improve a Loki label strategy — or when a user asks why their Loki queries are slow — use this guide to provide structured, actionable advice.",{"type":40,"tag":55,"props":56,"children":57},"hr",{},[],{"type":40,"tag":59,"props":60,"children":62},"h2",{"id":61},"core-concepts",[63],{"type":46,"value":64},"Core Concepts",{"type":40,"tag":49,"props":66,"children":67},{},[68,74],{"type":40,"tag":69,"props":70,"children":71},"strong",{},[72],{"type":46,"value":73},"Streams",{"type":46,"value":75}," are the fundamental unit in Loki. Each unique combination of label key-value pairs creates a new stream. Too many streams = performance problems. Too few = broad, slow queries.",{"type":40,"tag":49,"props":77,"children":78},{},[79,84,86,93,95,101,102,108,110,116],{"type":40,"tag":69,"props":80,"children":81},{},[82],{"type":46,"value":83},"Cardinality",{"type":46,"value":85}," = the number of unique values a label can have. High-cardinality labels (like ",{"type":40,"tag":87,"props":88,"children":90},"code",{"className":89},[],[91],{"type":46,"value":92},"pod",{"type":46,"value":94},", ",{"type":40,"tag":87,"props":96,"children":98},{"className":97},[],[99],{"type":46,"value":100},"user_id",{"type":46,"value":94},{"type":40,"tag":87,"props":103,"children":105},{"className":104},[],[106],{"type":46,"value":107},"request_id",{"type":46,"value":109},") dramatically increase stream count and hurt performance — ",{"type":40,"tag":111,"props":112,"children":113},"em",{},[114],{"type":46,"value":115},"especially",{"type":46,"value":117}," when those labels are not specified in every query.",{"type":40,"tag":49,"props":119,"children":120},{},[121,126],{"type":40,"tag":69,"props":122,"children":123},{},[124],{"type":46,"value":125},"The dual impact rule",{"type":46,"value":127},": High-cardinality labels hurt on both paths:",{"type":40,"tag":129,"props":130,"children":131},"ul",{},[132,143],{"type":40,"tag":133,"props":134,"children":135},"li",{},[136,141],{"type":40,"tag":69,"props":137,"children":138},{},[139],{"type":46,"value":140},"Ingestion path",{"type":46,"value":142},": More streams → larger index, higher storage costs",{"type":40,"tag":133,"props":144,"children":145},{},[146,151],{"type":40,"tag":69,"props":147,"children":148},{},[149],{"type":46,"value":150},"Query path",{"type":46,"value":152},": If a high-cardinality label exists but isn't in the query selector, Loki must scan ALL streams matching the other selectors — catastrophic for performance",{"type":40,"tag":49,"props":154,"children":155},{},[156,161,163,168],{"type":40,"tag":69,"props":157,"children":158},{},[159],{"type":46,"value":160},"The key question for any dynamic label",{"type":46,"value":162},": \"Will this label be used in 9 out of 10 queries?\" If no → it should NOT be a label — ",{"type":40,"tag":69,"props":164,"children":165},{},[166],{"type":46,"value":167},"except",{"type":46,"value":169}," platform \u002F correlation labels (below).",{"type":40,"tag":49,"props":171,"children":172},{},[173,178,180,186,187,193,195,201,203,208,210,216,218,224],{"type":40,"tag":69,"props":174,"children":175},{},[176],{"type":46,"value":177},"Platform \u002F correlation labels are exempt from drop recommendations.",{"type":46,"value":179}," Never recommend dropping ",{"type":40,"tag":87,"props":181,"children":183},{"className":182},[],[184],{"type":46,"value":185},"service_name",{"type":46,"value":94},{"type":40,"tag":87,"props":188,"children":190},{"className":189},[],[191],{"type":46,"value":192},"deployment_environment",{"type":46,"value":194},", or ",{"type":40,"tag":87,"props":196,"children":198},{"className":197},[],[199],{"type":46,"value":200},"job",{"type":46,"value":202}," when present. Bad cardinality on those keys is a ",{"type":40,"tag":69,"props":204,"children":205},{},[206],{"type":46,"value":207},"value",{"type":46,"value":209}," problem (stabilize identities); dropping the key breaks Grafana Cloud correlation, App O11y, alerts, and dashboards. Load ",{"type":40,"tag":211,"props":212,"children":214},"a",{"href":213},"references\u002Fprotected-labels.md",[215],{"type":46,"value":213},{"type":46,"value":217}," before any demote\u002F",{"type":40,"tag":87,"props":219,"children":221},{"className":220},[],[222],{"type":46,"value":223},"label_keep",{"type":46,"value":225}," advice.",{"type":40,"tag":55,"props":227,"children":228},{},[],{"type":40,"tag":59,"props":230,"children":232},{"id":231},"label-evaluation-framework",[233],{"type":46,"value":234},"Label Evaluation Framework",{"type":40,"tag":49,"props":236,"children":237},{},[238],{"type":46,"value":239},"When auditing a label strategy, assess each label against these criteria.",{"type":40,"tag":241,"props":242,"children":244},"h3",{"id":243},"cardinality-scoring",[245],{"type":46,"value":246},"Cardinality Scoring",{"type":40,"tag":248,"props":249,"children":250},"table",{},[251,274],{"type":40,"tag":252,"props":253,"children":254},"thead",{},[255],{"type":40,"tag":256,"props":257,"children":258},"tr",{},[259,265,269],{"type":40,"tag":260,"props":261,"children":262},"th",{},[263],{"type":46,"value":264},"Label Example",{"type":40,"tag":260,"props":266,"children":267},{},[268],{"type":46,"value":83},{"type":40,"tag":260,"props":270,"children":271},{},[272],{"type":46,"value":273},"Verdict",{"type":40,"tag":275,"props":276,"children":277},"tbody",{},[278,313,337,360,384,413,434],{"type":40,"tag":256,"props":279,"children":280},{},[281,303,308],{"type":40,"tag":282,"props":283,"children":284},"td",{},[285,290,292,297,298],{"type":40,"tag":87,"props":286,"children":288},{"className":287},[],[289],{"type":46,"value":185},{"type":46,"value":291}," \u002F ",{"type":40,"tag":87,"props":293,"children":295},{"className":294},[],[296],{"type":46,"value":192},{"type":46,"value":291},{"type":40,"tag":87,"props":299,"children":301},{"className":300},[],[302],{"type":46,"value":200},{"type":40,"tag":282,"props":304,"children":305},{},[306],{"type":46,"value":307},"Any",{"type":40,"tag":282,"props":309,"children":310},{},[311],{"type":46,"value":312},"✅ Keep key — remediate values if high-card (never drop)",{"type":40,"tag":256,"props":314,"children":315},{},[316,327,332],{"type":40,"tag":282,"props":317,"children":318},{},[319,325],{"type":40,"tag":87,"props":320,"children":322},{"className":321},[],[323],{"type":46,"value":324},"env",{"type":46,"value":326}," (prod\u002Fstaging\u002Fdev)",{"type":40,"tag":282,"props":328,"children":329},{},[330],{"type":46,"value":331},"2–5 values",{"type":40,"tag":282,"props":333,"children":334},{},[335],{"type":46,"value":336},"✅ Good",{"type":40,"tag":256,"props":338,"children":339},{},[340,351,356],{"type":40,"tag":282,"props":341,"children":342},{},[343,349],{"type":40,"tag":87,"props":344,"children":346},{"className":345},[],[347],{"type":46,"value":348},"level",{"type":46,"value":350}," (info\u002Fwarn\u002Ferror)",{"type":40,"tag":282,"props":352,"children":353},{},[354],{"type":46,"value":355},"3–6 values",{"type":40,"tag":282,"props":357,"children":358},{},[359],{"type":46,"value":336},{"type":40,"tag":256,"props":361,"children":362},{},[363,374,379],{"type":40,"tag":282,"props":364,"children":365},{},[366,372],{"type":40,"tag":87,"props":367,"children":369},{"className":368},[],[370],{"type":46,"value":371},"namespace",{"type":46,"value":373}," (K8s)",{"type":40,"tag":282,"props":375,"children":376},{},[377],{"type":46,"value":378},"Tens",{"type":40,"tag":282,"props":380,"children":381},{},[382],{"type":46,"value":383},"✅ Acceptable",{"type":40,"tag":256,"props":385,"children":386},{},[387,403,408],{"type":40,"tag":282,"props":388,"children":389},{},[390,396,397],{"type":40,"tag":87,"props":391,"children":393},{"className":392},[],[394],{"type":46,"value":395},"instance",{"type":46,"value":291},{"type":40,"tag":87,"props":398,"children":400},{"className":399},[],[401],{"type":46,"value":402},"hostname",{"type":40,"tag":282,"props":404,"children":405},{},[406],{"type":46,"value":407},"Hundreds–thousands",{"type":40,"tag":282,"props":409,"children":410},{},[411],{"type":46,"value":412},"⚠️ Evaluate access patterns",{"type":40,"tag":256,"props":414,"children":415},{},[416,424,429],{"type":40,"tag":282,"props":417,"children":418},{},[419],{"type":40,"tag":87,"props":420,"children":422},{"className":421},[],[423],{"type":46,"value":92},{"type":40,"tag":282,"props":425,"children":426},{},[427],{"type":46,"value":428},"Thousands + transient",{"type":40,"tag":282,"props":430,"children":431},{},[432],{"type":46,"value":433},"⚠️ Demote off index (structured metadata) — migrate selectors first",{"type":40,"tag":256,"props":435,"children":436},{},[437,451,456],{"type":40,"tag":282,"props":438,"children":439},{},[440,445,446],{"type":40,"tag":87,"props":441,"children":443},{"className":442},[],[444],{"type":46,"value":100},{"type":46,"value":94},{"type":40,"tag":87,"props":447,"children":449},{"className":448},[],[450],{"type":46,"value":107},{"type":40,"tag":282,"props":452,"children":453},{},[454],{"type":46,"value":455},"Unbounded",{"type":40,"tag":282,"props":457,"children":458},{},[459],{"type":46,"value":460},"❌ Never use as label",{"type":40,"tag":241,"props":462,"children":464},{"id":463},"access-pattern-alignment",[465],{"type":46,"value":466},"Access Pattern Alignment",{"type":40,"tag":49,"props":468,"children":469},{},[470],{"type":46,"value":471},"For each label, ask:",{"type":40,"tag":129,"props":473,"children":474},{},[475,487,492,497,502],{"type":40,"tag":133,"props":476,"children":477},{},[478,480,485],{"type":46,"value":479},"Is this label on the protected allowlist? If yes → Keep key; remediate values only (",{"type":40,"tag":211,"props":481,"children":482},{"href":213},[483],{"type":46,"value":484},"protected-labels.md",{"type":46,"value":486},")",{"type":40,"tag":133,"props":488,"children":489},{},[490],{"type":46,"value":491},"Is this label used as a selector in most queries targeting these logs?",{"type":40,"tag":133,"props":493,"children":494},{},[495],{"type":46,"value":496},"Does this label logically segment data in the way users think about it?",{"type":40,"tag":133,"props":498,"children":499},{},[500],{"type":46,"value":501},"Would demoting this label break alerts, dashboards, LBAC, or correlation without a migration plan?",{"type":40,"tag":133,"props":503,"children":504},{},[505],{"type":46,"value":506},"Would demoting this label force users to scan dramatically more data?",{"type":40,"tag":241,"props":508,"children":510},{"id":509},"static-vs-dynamic-label-values",[511],{"type":46,"value":512},"Static vs. Dynamic Label Values",{"type":40,"tag":129,"props":514,"children":515},{},[516,541],{"type":40,"tag":133,"props":517,"children":518},{},[519,524,526,532,533,539],{"type":40,"tag":69,"props":520,"children":521},{},[522],{"type":46,"value":523},"Static labels",{"type":46,"value":525}," (values don't change per log line, e.g., ",{"type":40,"tag":87,"props":527,"children":529},{"className":528},[],[530],{"type":46,"value":531},"platform=linux",{"type":46,"value":94},{"type":40,"tag":87,"props":534,"children":536},{"className":535},[],[537],{"type":46,"value":538},"job=agent",{"type":46,"value":540},") add no cardinality cost relative to the query scope. Use freely for LBAC, exploration, and alert routing.",{"type":40,"tag":133,"props":542,"children":543},{},[544,549],{"type":40,"tag":69,"props":545,"children":546},{},[547],{"type":46,"value":548},"Dynamic labels",{"type":46,"value":550}," (values change per log line) must be bounded. Keep possible values in the single digits or low tens.",{"type":40,"tag":241,"props":552,"children":554},{"id":553},"consistency-check",[555],{"type":46,"value":556},"Consistency Check",{"type":40,"tag":129,"props":558,"children":559},{},[560,579,612],{"type":40,"tag":133,"props":561,"children":562},{},[563,565,571,573,578],{"type":46,"value":564},"Are label names consistent across services? (case-sensitive — ",{"type":40,"tag":87,"props":566,"children":568},{"className":567},[],[569],{"type":46,"value":570},"Level",{"type":46,"value":572}," ≠ ",{"type":40,"tag":87,"props":574,"children":576},{"className":575},[],[577],{"type":46,"value":348},{"type":46,"value":486},{"type":40,"tag":133,"props":580,"children":581},{},[582,584,590,591,597,598,604,606,611],{"type":46,"value":583},"Are label values normalized? (",{"type":40,"tag":87,"props":585,"children":587},{"className":586},[],[588],{"type":46,"value":589},"INFO",{"type":46,"value":94},{"type":40,"tag":87,"props":592,"children":594},{"className":593},[],[595],{"type":46,"value":596},"info",{"type":46,"value":94},{"type":40,"tag":87,"props":599,"children":601},{"className":600},[],[602],{"type":46,"value":603},"Info",{"type":46,"value":605}," should all become ",{"type":40,"tag":87,"props":607,"children":609},{"className":608},[],[610],{"type":46,"value":596},{"type":46,"value":486},{"type":40,"tag":133,"props":613,"children":614},{},[615,617,623,625,631],{"type":46,"value":616},"Is there a naming convention? (pick one: ",{"type":40,"tag":87,"props":618,"children":620},{"className":619},[],[621],{"type":46,"value":622},"snake_case",{"type":46,"value":624}," or ",{"type":40,"tag":87,"props":626,"children":628},{"className":627},[],[629],{"type":46,"value":630},"camelCase",{"type":46,"value":632}," — be consistent)",{"type":40,"tag":55,"props":634,"children":635},{},[],{"type":40,"tag":59,"props":637,"children":639},{"id":638},"evaluation-output-format",[640],{"type":46,"value":641},"Evaluation Output Format",{"type":40,"tag":49,"props":643,"children":644},{},[645],{"type":46,"value":646},"When auditing a label set, produce a report in the structure below.",{"type":40,"tag":49,"props":648,"children":649},{},[650],{"type":40,"tag":69,"props":651,"children":652},{},[653],{"type":46,"value":654},"Hard requirements before finalizing any audit report:",{"type":40,"tag":656,"props":657,"children":658},"ol",{},[659,691,733],{"type":40,"tag":133,"props":660,"children":661},{},[662,667,669,674,676,681,683,689],{"type":40,"tag":69,"props":663,"children":664},{},[665],{"type":46,"value":666},"Disclaimer (mandatory, first body section):",{"type":46,"value":668}," Load ",{"type":40,"tag":211,"props":670,"children":672},{"href":671},"references\u002Fdisclaimer.md",[673],{"type":46,"value":671},{"type":46,"value":675}," and paste its two paragraphs ",{"type":40,"tag":69,"props":677,"children":678},{},[679],{"type":46,"value":680},"verbatim",{"type":46,"value":682}," under a ",{"type":40,"tag":87,"props":684,"children":686},{"className":685},[],[687],{"type":46,"value":688},"### Disclaimer",{"type":46,"value":690}," heading. An empty Disclaimer heading is a failed report — do not ship the audit until both paragraphs are present. Never paraphrase, summarize, or omit this text.",{"type":40,"tag":133,"props":692,"children":693},{},[694,699,701,705,707,712,713,718,719,724,726,731],{"type":40,"tag":69,"props":695,"children":696},{},[697],{"type":46,"value":698},"Protected labels:",{"type":46,"value":700}," Before recommending demote\u002Fdrop for any label, load ",{"type":40,"tag":211,"props":702,"children":703},{"href":213},[704],{"type":46,"value":213},{"type":46,"value":706},". Never recommend dropping ",{"type":40,"tag":87,"props":708,"children":710},{"className":709},[],[711],{"type":46,"value":185},{"type":46,"value":94},{"type":40,"tag":87,"props":714,"children":716},{"className":715},[],[717],{"type":46,"value":192},{"type":46,"value":194},{"type":40,"tag":87,"props":720,"children":722},{"className":721},[],[723],{"type":46,"value":200},{"type":46,"value":725}," when present — only value remediation. Include a ",{"type":40,"tag":69,"props":727,"children":728},{},[729],{"type":46,"value":730},"Downstream dependency check",{"type":46,"value":732}," covering alerts, dashboards, LBAC, and correlation.",{"type":40,"tag":133,"props":734,"children":735},{},[736,741,743,748,750,755,757,762],{"type":40,"tag":69,"props":737,"children":738},{},[739],{"type":46,"value":740},"Cost Impact Analysis:",{"type":46,"value":742}," Include when Grafana Cloud usage metrics are available; if they are not, state what is missing and still give qualitative A\u002FB\u002FC guidance. Load ",{"type":40,"tag":211,"props":744,"children":746},{"href":745},"references\u002Fcost-impact.md",[747],{"type":46,"value":745},{"type":46,"value":749}," and follow its ",{"type":40,"tag":69,"props":751,"children":752},{},[753],{"type":46,"value":754},"Required report shape",{"type":46,"value":756}," (scenario cards). Do ",{"type":40,"tag":69,"props":758,"children":759},{},[760],{"type":46,"value":761},"not",{"type":46,"value":763}," paste markdown tables or panel\u002Fquery JSON into this section.",{"type":40,"tag":49,"props":765,"children":766},{},[767,772,774,780,782,787,789,794,796,801,803,809,811,817,819,823,825,829,831,835],{"type":40,"tag":69,"props":768,"children":769},{},[770],{"type":46,"value":771},"Report completion check:",{"type":46,"value":773}," Before delivering, confirm (a) the output contains the substring ",{"type":40,"tag":87,"props":775,"children":777},{"className":776},[],[778],{"type":46,"value":779},"Confidential Information of Raintank, Inc.",{"type":46,"value":781}," immediately after ",{"type":40,"tag":87,"props":783,"children":785},{"className":784},[],[786],{"type":46,"value":688},{"type":46,"value":788},", (b) Cost Impact Analysis uses scenario cards (A\u002FB\u002FC) with a ",{"type":40,"tag":69,"props":790,"children":791},{},[792],{"type":46,"value":793},"Billing note",{"type":46,"value":795}," opener and a bullet ",{"type":40,"tag":69,"props":797,"children":798},{},[799],{"type":46,"value":800},"Measured baseline",{"type":46,"value":802}," — not a scenario table and not ",{"type":40,"tag":87,"props":804,"children":806},{"className":805},[],[807],{"type":46,"value":808},"panelId",{"type":46,"value":810},"\u002F",{"type":40,"tag":87,"props":812,"children":814},{"className":813},[],[815],{"type":46,"value":816},"targets",{"type":46,"value":818}," JSON, and (c) no Action cell recommends dropping an allowlisted correlation label. If (a) is missing, paste from ",{"type":40,"tag":211,"props":820,"children":821},{"href":671},[822],{"type":46,"value":671},{"type":46,"value":824}," and re-emit. If (b) fails, rewrite Cost Impact from ",{"type":40,"tag":211,"props":826,"children":827},{"href":745},[828],{"type":46,"value":745},{"type":46,"value":830},". If (c) fails, rewrite Actions per ",{"type":40,"tag":211,"props":832,"children":833},{"href":213},[834],{"type":46,"value":213},{"type":46,"value":836},".",{"type":40,"tag":838,"props":839,"children":843},"pre",{"className":840,"code":842,"language":46},[841],"language-text","## Loki Label Strategy Audit\n\n### Disclaimer\n[Paste BOTH paragraphs from references\u002Fdisclaimer.md HERE — never leave this heading empty]\n\n### Summary\n[1-2 sentence overall assessment]\n\n### Downstream dependency check\n[Alerts \u002F dashboards \u002F LBAC \u002F correlation that select on labels proposed for demote or rename — or \"unknown; confirm with customer before cutover\"]\n\n### Label Analysis\n| Label | Cardinality | Used in Queries? | Verdict | Action |\n|---|---|---|---|---|\n| service_name | High (UUID values) | Always | ✅ Keep key | Stabilize values to durable service identity — do not drop label |\n| deployment_environment | Low | Often | ✅ Keep | — |\n| job | Low–medium | Often | ✅ Keep | — |\n| pod | Very High (transient)| Rarely | ⚠️ Demote | Move to structured metadata or embed; migrate selectors first |\n\n### Estimated Impact\n- Stream count reduction: [X streams → Y streams]\n- Query performance: [describe improvement]\n- Storage impact: [if log line changes are involved]\n- Correlation impact: [none if allowlist preserved; call out if aliases need dual-write]\n\n### Cost Impact Analysis\n[Follow references\u002Fcost-impact.md Required report shape — do not invent a table]\n\n**Billing note:** Label hygiene alone does not reduce billable ingest bytes.\nStream count and query cost improve; ingest $ drops only when volume is reduced.\n\n**Measured baseline** (Grafana Cloud usage metrics):\n- Active streams: [N]\n- Billable ingest: [rate]\n- Overage: [units or $]\n- Top ingest contributor: [name + rate] (omit if unavailable)\n\n**Scenario A — Label hygiene only (this audit)**\n- Actions \u002F stream impact \u002F volume=$0 \u002F overage unchanged\n\n**Scenario B — A + approved debug\u002Ftrace drop**\n- Actions \u002F volume % \u002F $ or overage estimate \u002F customer-approval guardrail\n\n**Scenario C — B + log-line compaction**\n- Actions \u002F additional volume % \u002F highest-value target\n\n**Attribution gap:** [...]\n**Caveats:** [...]\n\n### Recommended Label Set\n[Final recommended labels — must include service_name, deployment_environment, job when present]\n\n### Migration Notes\n[How to implement changes via Alloy\u002FAgent pipeline stages; dual-write \u002F selector updates for any demote or rename]\n",[844],{"type":40,"tag":87,"props":845,"children":847},{"__ignoreMap":846},"",[848],{"type":46,"value":842},{"type":40,"tag":55,"props":850,"children":851},{},[],{"type":40,"tag":59,"props":853,"children":855},{"id":854},"recommended-common-labels",[856],{"type":46,"value":857},"Recommended Common Labels",{"type":40,"tag":49,"props":859,"children":860},{},[861],{"type":46,"value":862},"Every log source should consider these base labels — all low cardinality, high query value:",{"type":40,"tag":248,"props":864,"children":865},{},[866,882],{"type":40,"tag":252,"props":867,"children":868},{},[869],{"type":40,"tag":256,"props":870,"children":871},{},[872,877],{"type":40,"tag":260,"props":873,"children":874},{},[875],{"type":46,"value":876},"Label",{"type":40,"tag":260,"props":878,"children":879},{},[880],{"type":46,"value":881},"Purpose",{"type":40,"tag":275,"props":883,"children":884},{},[885,915,939,963,994,1017,1034,1051,1093,1117,1163],{"type":40,"tag":256,"props":886,"children":887},{},[888,896],{"type":40,"tag":282,"props":889,"children":890},{},[891],{"type":40,"tag":87,"props":892,"children":894},{"className":893},[],[895],{"type":46,"value":185},{"type":40,"tag":282,"props":897,"children":898},{},[899,901,907,909,914],{"type":46,"value":900},"Identifying the generating application (OTel ",{"type":40,"tag":87,"props":902,"children":904},{"className":903},[],[905],{"type":46,"value":906},"service.name",{"type":46,"value":908}," — ",{"type":40,"tag":69,"props":910,"children":911},{},[912],{"type":46,"value":913},"required for Grafana Cloud correlation \u002F App O11y",{"type":46,"value":486},{"type":40,"tag":256,"props":916,"children":917},{},[918,926],{"type":40,"tag":282,"props":919,"children":920},{},[921],{"type":40,"tag":87,"props":922,"children":924},{"className":923},[],[925],{"type":46,"value":192},{"type":40,"tag":282,"props":927,"children":928},{},[929,931,937],{"type":46,"value":930},"Deployment environment (OTel ",{"type":40,"tag":87,"props":932,"children":934},{"className":933},[],[935],{"type":46,"value":936},"deployment.environment",{"type":46,"value":938},") — keep when present",{"type":40,"tag":256,"props":940,"children":941},{},[942,950],{"type":40,"tag":282,"props":943,"children":944},{},[945],{"type":40,"tag":87,"props":946,"children":948},{"className":947},[],[949],{"type":46,"value":200},{"type":40,"tag":282,"props":951,"children":952},{},[953,955,961],{"type":46,"value":954},"Collector \u002F OTel job (",{"type":40,"tag":87,"props":956,"children":958},{"className":957},[],[959],{"type":46,"value":960},"namespace\u002Fservice.name",{"type":46,"value":962}," pattern common on span metrics) — keep when present",{"type":40,"tag":256,"props":964,"children":965},{},[966,982],{"type":40,"tag":282,"props":967,"children":968},{},[969,975,976],{"type":40,"tag":87,"props":970,"children":972},{"className":971},[],[973],{"type":46,"value":974},"app",{"type":46,"value":291},{"type":40,"tag":87,"props":977,"children":979},{"className":978},[],[980],{"type":46,"value":981},"service",{"type":40,"tag":282,"props":983,"children":984},{},[985,987,992],{"type":46,"value":986},"Legacy aliases only — prefer aligning to ",{"type":40,"tag":87,"props":988,"children":990},{"className":989},[],[991],{"type":46,"value":185},{"type":46,"value":993},"; do not delete without a migration plan",{"type":40,"tag":256,"props":995,"children":996},{},[997,1005],{"type":40,"tag":282,"props":998,"children":999},{},[1000],{"type":40,"tag":87,"props":1001,"children":1003},{"className":1002},[],[1004],{"type":46,"value":324},{"type":40,"tag":282,"props":1006,"children":1007},{},[1008,1010,1015],{"type":46,"value":1009},"Environment shorthand (prod, staging, dev) when ",{"type":40,"tag":87,"props":1011,"children":1013},{"className":1012},[],[1014],{"type":46,"value":192},{"type":46,"value":1016}," is absent",{"type":40,"tag":256,"props":1018,"children":1019},{},[1020,1029],{"type":40,"tag":282,"props":1021,"children":1022},{},[1023],{"type":40,"tag":87,"props":1024,"children":1026},{"className":1025},[],[1027],{"type":46,"value":1028},"cluster",{"type":40,"tag":282,"props":1030,"children":1031},{},[1032],{"type":46,"value":1033},"Multi-cluster differentiation",{"type":40,"tag":256,"props":1035,"children":1036},{},[1037,1046],{"type":40,"tag":282,"props":1038,"children":1039},{},[1040],{"type":40,"tag":87,"props":1041,"children":1043},{"className":1042},[],[1044],{"type":46,"value":1045},"region",{"type":40,"tag":282,"props":1047,"children":1048},{},[1049],{"type":46,"value":1050},"Geographic region",{"type":40,"tag":256,"props":1052,"children":1053},{},[1054,1062],{"type":40,"tag":282,"props":1055,"children":1056},{},[1057],{"type":40,"tag":87,"props":1058,"children":1060},{"className":1059},[],[1061],{"type":46,"value":348},{"type":40,"tag":282,"props":1063,"children":1064},{},[1065,1067,1072,1073,1079,1080,1086,1087],{"type":46,"value":1066},"Log severity — normalize to: ",{"type":40,"tag":87,"props":1068,"children":1070},{"className":1069},[],[1071],{"type":46,"value":596},{"type":46,"value":94},{"type":40,"tag":87,"props":1074,"children":1076},{"className":1075},[],[1077],{"type":46,"value":1078},"warn",{"type":46,"value":94},{"type":40,"tag":87,"props":1081,"children":1083},{"className":1082},[],[1084],{"type":46,"value":1085},"error",{"type":46,"value":94},{"type":40,"tag":87,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":46,"value":1092},"debug",{"type":40,"tag":256,"props":1094,"children":1095},{},[1096,1112],{"type":40,"tag":282,"props":1097,"children":1098},{},[1099,1105,1106],{"type":40,"tag":87,"props":1100,"children":1102},{"className":1101},[],[1103],{"type":46,"value":1104},"team",{"type":46,"value":291},{"type":40,"tag":87,"props":1107,"children":1109},{"className":1108},[],[1110],{"type":46,"value":1111},"squad",{"type":40,"tag":282,"props":1113,"children":1114},{},[1115],{"type":46,"value":1116},"Ownership (also useful for LBAC)",{"type":40,"tag":256,"props":1118,"children":1119},{},[1120,1129],{"type":40,"tag":282,"props":1121,"children":1122},{},[1123],{"type":40,"tag":87,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":46,"value":1128},"source",{"type":40,"tag":282,"props":1130,"children":1131},{},[1132,1134,1140,1141,1147,1148,1154,1155,1161],{"type":46,"value":1133},"Log origin type (",{"type":40,"tag":87,"props":1135,"children":1137},{"className":1136},[],[1138],{"type":46,"value":1139},"file",{"type":46,"value":94},{"type":40,"tag":87,"props":1142,"children":1144},{"className":1143},[],[1145],{"type":46,"value":1146},"k8s-events",{"type":46,"value":94},{"type":40,"tag":87,"props":1149,"children":1151},{"className":1150},[],[1152],{"type":46,"value":1153},"journal",{"type":46,"value":94},{"type":40,"tag":87,"props":1156,"children":1158},{"className":1157},[],[1159],{"type":46,"value":1160},"syslog",{"type":46,"value":1162},", etc.)",{"type":40,"tag":256,"props":1164,"children":1165},{},[1166,1175],{"type":40,"tag":282,"props":1167,"children":1168},{},[1169],{"type":40,"tag":87,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":46,"value":1174},"classification",{"type":40,"tag":282,"props":1176,"children":1177},{},[1178],{"type":46,"value":1179},"Data sensitivity level — for LBAC policies",{"type":40,"tag":49,"props":1181,"children":1182},{},[1183,1185,1190,1192,1196],{"type":46,"value":1184},"Always include allowlisted correlation labels in any ",{"type":40,"tag":87,"props":1186,"children":1188},{"className":1187},[],[1189],{"type":46,"value":223},{"type":46,"value":1191}," list — see ",{"type":40,"tag":211,"props":1193,"children":1194},{"href":213},[1195],{"type":46,"value":213},{"type":46,"value":836},{"type":40,"tag":55,"props":1198,"children":1199},{},[],{"type":40,"tag":59,"props":1201,"children":1203},{"id":1202},"kubernetes-pod-logs",[1204],{"type":46,"value":1205},"Kubernetes Pod Logs",{"type":40,"tag":241,"props":1207,"children":1209},{"id":1208},"recommended-labels",[1210],{"type":46,"value":1211},"Recommended Labels",{"type":40,"tag":248,"props":1213,"children":1214},{},[1215,1230],{"type":40,"tag":252,"props":1216,"children":1217},{},[1218],{"type":40,"tag":256,"props":1219,"children":1220},{},[1221,1225],{"type":40,"tag":260,"props":1222,"children":1223},{},[1224],{"type":46,"value":876},{"type":40,"tag":260,"props":1226,"children":1227},{},[1228],{"type":46,"value":1229},"Description",{"type":40,"tag":275,"props":1231,"children":1232},{},[1233,1263,1279,1296],{"type":40,"tag":256,"props":1234,"children":1235},{},[1236,1244],{"type":40,"tag":282,"props":1237,"children":1238},{},[1239],{"type":40,"tag":87,"props":1240,"children":1242},{"className":1241},[],[1243],{"type":46,"value":185},{"type":40,"tag":282,"props":1245,"children":1246},{},[1247,1249,1254,1256,1261],{"type":46,"value":1248},"Stable service identity (OTel ",{"type":40,"tag":87,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":46,"value":906},{"type":46,"value":1255},") — ",{"type":40,"tag":69,"props":1257,"children":1258},{},[1259],{"type":46,"value":1260},"keep",{"type":46,"value":1262},"; remediate UUID\u002Fephemeral values",{"type":40,"tag":256,"props":1264,"children":1265},{},[1266,1274],{"type":40,"tag":282,"props":1267,"children":1268},{},[1269],{"type":40,"tag":87,"props":1270,"children":1272},{"className":1271},[],[1273],{"type":46,"value":371},{"type":40,"tag":282,"props":1275,"children":1276},{},[1277],{"type":46,"value":1278},"K8s namespace — delineates isolation boundaries",{"type":40,"tag":256,"props":1280,"children":1281},{},[1282,1291],{"type":40,"tag":282,"props":1283,"children":1284},{},[1285],{"type":40,"tag":87,"props":1286,"children":1288},{"className":1287},[],[1289],{"type":46,"value":1290},"container",{"type":40,"tag":282,"props":1292,"children":1293},{},[1294],{"type":46,"value":1295},"Container name — low cardinality, differentiates log formats",{"type":40,"tag":256,"props":1297,"children":1298},{},[1299,1308],{"type":40,"tag":282,"props":1300,"children":1301},{},[1302],{"type":40,"tag":87,"props":1303,"children":1305},{"className":1304},[],[1306],{"type":46,"value":1307},"workload",{"type":40,"tag":282,"props":1309,"children":1310},{},[1311,1317,1319,1325,1326],{"type":40,"tag":87,"props":1312,"children":1314},{"className":1313},[],[1315],{"type":46,"value":1316},"{controller_kind}\u002F{controller_name}",{"type":46,"value":1318}," e.g. ",{"type":40,"tag":87,"props":1320,"children":1322},{"className":1321},[],[1323],{"type":46,"value":1324},"ReplicaSet\u002Fpayment-api",{"type":46,"value":908},{"type":40,"tag":69,"props":1327,"children":1328},{},[1329],{"type":46,"value":1330},"strongly recommended",{"type":40,"tag":49,"props":1332,"children":1333},{},[1334,1353,1355,1361,1363,1368,1370,1375,1377,1382,1384,1389],{"type":40,"tag":69,"props":1335,"children":1336},{},[1337,1339,1344,1346,1351],{"type":46,"value":1338},"Why ",{"type":40,"tag":87,"props":1340,"children":1342},{"className":1341},[],[1343],{"type":46,"value":1307},{"type":46,"value":1345}," beats ",{"type":40,"tag":87,"props":1347,"children":1349},{"className":1348},[],[1350],{"type":46,"value":974},{"type":46,"value":1352}," for K8s",{"type":46,"value":1354},": Derived from ",{"type":40,"tag":87,"props":1356,"children":1358},{"className":1357},[],[1359],{"type":46,"value":1360},"{{controller_kind}}\u002F{{controller_name}}",{"type":46,"value":1362}," — static values that never change like pod names do. Unlike ",{"type":40,"tag":87,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":46,"value":974},{"type":46,"value":1369}," (which may aggregate multiple workload types), ",{"type":40,"tag":87,"props":1371,"children":1373},{"className":1372},[],[1374],{"type":46,"value":1307},{"type":46,"value":1376}," is precise and predictable. Users always know exactly what value to query. Still keep ",{"type":40,"tag":87,"props":1378,"children":1380},{"className":1379},[],[1381],{"type":46,"value":185},{"type":46,"value":1383}," for cross-signal correlation even when using ",{"type":40,"tag":87,"props":1385,"children":1387},{"className":1386},[],[1388],{"type":46,"value":1307},{"type":46,"value":836},{"type":40,"tag":241,"props":1391,"children":1393},{"id":1392},"labels-to-demote-in-kubernetes-not-never-existed",[1394],{"type":46,"value":1395},"Labels to demote in Kubernetes (not \"never existed\")",{"type":40,"tag":49,"props":1397,"children":1398},{},[1399,1409],{"type":40,"tag":69,"props":1400,"children":1401},{},[1402,1407],{"type":40,"tag":87,"props":1403,"children":1405},{"className":1404},[],[1406],{"type":46,"value":92},{"type":46,"value":1408}," label",{"type":46,"value":1410}," ⚠️",{"type":40,"tag":129,"props":1412,"children":1413},{},[1414,1419,1431,1440],{"type":40,"tag":133,"props":1415,"children":1416},{},[1417],{"type":46,"value":1418},"Highly transient: pod names change on every restart\u002Frollout",{"type":40,"tag":133,"props":1420,"children":1421},{},[1422,1424,1429],{"type":46,"value":1423},"Very high cardinality: 5 pods × 2 containers = 10 streams; add ",{"type":40,"tag":87,"props":1425,"children":1427},{"className":1426},[],[1428],{"type":46,"value":92},{"type":46,"value":1430}," → 10 × N streams",{"type":40,"tag":133,"props":1432,"children":1433},{},[1434,1436],{"type":46,"value":1435},"Users almost never query for a specific pod; they query for the ",{"type":40,"tag":111,"props":1437,"children":1438},{},[1439],{"type":46,"value":1307},{"type":40,"tag":133,"props":1441,"children":1442},{},[1443,1448,1450,1455,1457,1462,1464,1469],{"type":40,"tag":69,"props":1444,"children":1445},{},[1446],{"type":46,"value":1447},"Solution",{"type":46,"value":1449},": Use ",{"type":40,"tag":87,"props":1451,"children":1453},{"className":1452},[],[1454],{"type":46,"value":1307},{"type":46,"value":1456}," as the index label; store ",{"type":40,"tag":87,"props":1458,"children":1460},{"className":1459},[],[1461],{"type":46,"value":92},{"type":46,"value":1463}," in structured metadata or embed in the log line. Migrate any alerts\u002Fdashboards that select on ",{"type":40,"tag":87,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":46,"value":92},{"type":46,"value":1470}," before demoting.",{"type":40,"tag":49,"props":1472,"children":1473},{},[1474,1485],{"type":40,"tag":69,"props":1475,"children":1476},{},[1477,1483],{"type":40,"tag":87,"props":1478,"children":1480},{"className":1479},[],[1481],{"type":46,"value":1482},"filename",{"type":46,"value":1484}," label (raw K8s path)",{"type":46,"value":1410},{"type":40,"tag":129,"props":1487,"children":1488},{},[1489,1500,1513],{"type":40,"tag":133,"props":1490,"children":1491},{},[1492,1494],{"type":46,"value":1493},"K8s log paths contain pod UID: ",{"type":40,"tag":87,"props":1495,"children":1497},{"className":1496},[],[1498],{"type":46,"value":1499},"\u002Fvar\u002Flog\u002Fpods\u002F{namespace}_{pod}_{pod_id}\u002F{container}\u002F{rotation}.log",{"type":40,"tag":133,"props":1501,"children":1502},{},[1503,1505,1511],{"type":46,"value":1504},"The ",{"type":40,"tag":87,"props":1506,"children":1508},{"className":1507},[],[1509],{"type":46,"value":1510},"pod_id",{"type":46,"value":1512}," component makes this unbounded",{"type":40,"tag":133,"props":1514,"children":1515},{},[1516,1520,1522,1528],{"type":40,"tag":69,"props":1517,"children":1518},{},[1519],{"type":46,"value":1447},{"type":46,"value":1521},": Normalize to ",{"type":40,"tag":87,"props":1523,"children":1525},{"className":1524},[],[1526],{"type":46,"value":1527},"\u002Fvar\u002Flog\u002Fpods\u002F{namespace}\u002F{controller_name}\u002F{container}.log",{"type":46,"value":1529}," or demote entirely after checking selectors",{"type":40,"tag":838,"props":1531,"children":1535},{"className":1532,"code":1533,"language":1534,"meta":846,"style":846},"language-alloy shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\u002F\u002F Normalize K8s filename to remove pod UID\nstage.replace {\n source = \"filename\"\n expression = \"\u002Fvar\u002Flog\u002Fpods\u002F([^\u002F]+)_[^_]+_[^\u002F]+\u002F([^\u002F]+)\u002F\\\\d+\\\\.log\"\n replace = \"\u002Fvar\u002Flog\u002Fpods\u002F$1\u002F$2\u002Fcurrent.log\"\n}\n","alloy",[1536],{"type":40,"tag":87,"props":1537,"children":1538},{"__ignoreMap":846},[1539,1550,1559,1568,1577,1586],{"type":40,"tag":1540,"props":1541,"children":1544},"span",{"class":1542,"line":1543},"line",1,[1545],{"type":40,"tag":1540,"props":1546,"children":1547},{},[1548],{"type":46,"value":1549},"\u002F\u002F Normalize K8s filename to remove pod UID\n",{"type":40,"tag":1540,"props":1551,"children":1553},{"class":1542,"line":1552},2,[1554],{"type":40,"tag":1540,"props":1555,"children":1556},{},[1557],{"type":46,"value":1558},"stage.replace {\n",{"type":40,"tag":1540,"props":1560,"children":1562},{"class":1542,"line":1561},3,[1563],{"type":40,"tag":1540,"props":1564,"children":1565},{},[1566],{"type":46,"value":1567}," source = \"filename\"\n",{"type":40,"tag":1540,"props":1569,"children":1571},{"class":1542,"line":1570},4,[1572],{"type":40,"tag":1540,"props":1573,"children":1574},{},[1575],{"type":46,"value":1576}," expression = \"\u002Fvar\u002Flog\u002Fpods\u002F([^\u002F]+)_[^_]+_[^\u002F]+\u002F([^\u002F]+)\u002F\\\\d+\\\\.log\"\n",{"type":40,"tag":1540,"props":1578,"children":1580},{"class":1542,"line":1579},5,[1581],{"type":40,"tag":1540,"props":1582,"children":1583},{},[1584],{"type":46,"value":1585}," replace = \"\u002Fvar\u002Flog\u002Fpods\u002F$1\u002F$2\u002Fcurrent.log\"\n",{"type":40,"tag":1540,"props":1587,"children":1589},{"class":1542,"line":1588},6,[1590],{"type":40,"tag":1540,"props":1591,"children":1592},{},[1593],{"type":46,"value":1594},"}\n",{"type":40,"tag":55,"props":1596,"children":1597},{},[],{"type":40,"tag":59,"props":1599,"children":1601},{"id":1600},"host-vm-bare-metal-labels",[1602],{"type":46,"value":1603},"Host \u002F VM \u002F Bare Metal Labels",{"type":40,"tag":49,"props":1605,"children":1606},{},[1607],{"type":46,"value":1608},"In addition to common labels, add:",{"type":40,"tag":248,"props":1610,"children":1611},{},[1612,1631],{"type":40,"tag":252,"props":1613,"children":1614},{},[1615],{"type":40,"tag":256,"props":1616,"children":1617},{},[1618,1622,1626],{"type":40,"tag":260,"props":1619,"children":1620},{},[1621],{"type":46,"value":876},{"type":40,"tag":260,"props":1623,"children":1624},{},[1625],{"type":46,"value":1229},{"type":40,"tag":260,"props":1627,"children":1628},{},[1629],{"type":46,"value":1630},"Notes",{"type":40,"tag":275,"props":1632,"children":1633},{},[1634,1655],{"type":40,"tag":256,"props":1635,"children":1636},{},[1637,1645,1650],{"type":40,"tag":282,"props":1638,"children":1639},{},[1640],{"type":40,"tag":87,"props":1641,"children":1643},{"className":1642},[],[1644],{"type":46,"value":395},{"type":40,"tag":282,"props":1646,"children":1647},{},[1648],{"type":46,"value":1649},"Hostname of the machine",{"type":40,"tag":282,"props":1651,"children":1652},{},[1653],{"type":46,"value":1654},"Cardinality = number of machines; acceptable for fixed infrastructure",{"type":40,"tag":256,"props":1656,"children":1657},{},[1658,1666,1671],{"type":40,"tag":282,"props":1659,"children":1660},{},[1661],{"type":40,"tag":87,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":46,"value":1482},{"type":40,"tag":282,"props":1667,"children":1668},{},[1669],{"type":46,"value":1670},"Full path to the file being tailed",{"type":40,"tag":282,"props":1672,"children":1673},{},[1674],{"type":46,"value":1675},"Normalize rotating filenames — strip date suffixes",{"type":40,"tag":838,"props":1677,"children":1679},{"className":1532,"code":1678,"language":1534,"meta":846,"style":846},"\u002F\u002F Remove date suffixes from rotating log file names\n\u002F\u002F \u002Fvar\u002Flog\u002Fmyapp\u002Flogfile-20230927.txt → \u002Fvar\u002Flog\u002Fmyapp\u002Flogfile.txt\nstage.replace {\n source = \"filename\"\n expression = \"-\\\\d{8}(\\\\.log|\\\\.txt)$\"\n replace = \"$1\"\n}\n",[1680],{"type":40,"tag":87,"props":1681,"children":1682},{"__ignoreMap":846},[1683,1691,1699,1706,1713,1721,1729],{"type":40,"tag":1540,"props":1684,"children":1685},{"class":1542,"line":1543},[1686],{"type":40,"tag":1540,"props":1687,"children":1688},{},[1689],{"type":46,"value":1690},"\u002F\u002F Remove date suffixes from rotating log file names\n",{"type":40,"tag":1540,"props":1692,"children":1693},{"class":1542,"line":1552},[1694],{"type":40,"tag":1540,"props":1695,"children":1696},{},[1697],{"type":46,"value":1698},"\u002F\u002F \u002Fvar\u002Flog\u002Fmyapp\u002Flogfile-20230927.txt → \u002Fvar\u002Flog\u002Fmyapp\u002Flogfile.txt\n",{"type":40,"tag":1540,"props":1700,"children":1701},{"class":1542,"line":1561},[1702],{"type":40,"tag":1540,"props":1703,"children":1704},{},[1705],{"type":46,"value":1558},{"type":40,"tag":1540,"props":1707,"children":1708},{"class":1542,"line":1570},[1709],{"type":40,"tag":1540,"props":1710,"children":1711},{},[1712],{"type":46,"value":1567},{"type":40,"tag":1540,"props":1714,"children":1715},{"class":1542,"line":1579},[1716],{"type":40,"tag":1540,"props":1717,"children":1718},{},[1719],{"type":46,"value":1720}," expression = \"-\\\\d{8}(\\\\.log|\\\\.txt)$\"\n",{"type":40,"tag":1540,"props":1722,"children":1723},{"class":1542,"line":1588},[1724],{"type":40,"tag":1540,"props":1725,"children":1726},{},[1727],{"type":46,"value":1728}," replace = \"$1\"\n",{"type":40,"tag":1540,"props":1730,"children":1732},{"class":1542,"line":1731},7,[1733],{"type":40,"tag":1540,"props":1734,"children":1735},{},[1736],{"type":46,"value":1594},{"type":40,"tag":55,"props":1738,"children":1739},{},[],{"type":40,"tag":59,"props":1741,"children":1743},{"id":1742},"journal-logs",[1744],{"type":46,"value":1745},"Journal Logs",{"type":40,"tag":49,"props":1747,"children":1748},{},[1749,1751,1757,1759,1765,1767,1773,1774,1780,1781,1787,1788,1794,1795,1801,1802,1808,1809,1814,1815,1821,1822,1828,1829,1835,1836,1842,1843,1849,1850,1856,1857,1863,1864,1870,1871],{"type":46,"value":1750},"When collecting via ",{"type":40,"tag":87,"props":1752,"children":1754},{"className":1753},[],[1755],{"type":46,"value":1756},"loki.source.journal",{"type":46,"value":1758},", many labels are auto-discovered under ",{"type":40,"tag":87,"props":1760,"children":1762},{"className":1761},[],[1763],{"type":46,"value":1764},"__journal__*",{"type":46,"value":1766},":\n",{"type":40,"tag":87,"props":1768,"children":1770},{"className":1769},[],[1771],{"type":46,"value":1772},"boot_id",{"type":46,"value":94},{"type":40,"tag":87,"props":1775,"children":1777},{"className":1776},[],[1778],{"type":46,"value":1779},"cap_effective",{"type":46,"value":94},{"type":40,"tag":87,"props":1782,"children":1784},{"className":1783},[],[1785],{"type":46,"value":1786},"cmdline",{"type":46,"value":94},{"type":40,"tag":87,"props":1789,"children":1791},{"className":1790},[],[1792],{"type":46,"value":1793},"comm",{"type":46,"value":94},{"type":40,"tag":87,"props":1796,"children":1798},{"className":1797},[],[1799],{"type":46,"value":1800},"exe",{"type":46,"value":94},{"type":40,"tag":87,"props":1803,"children":1805},{"className":1804},[],[1806],{"type":46,"value":1807},"gid",{"type":46,"value":94},{"type":40,"tag":87,"props":1810,"children":1812},{"className":1811},[],[1813],{"type":46,"value":402},{"type":46,"value":94},{"type":40,"tag":87,"props":1816,"children":1818},{"className":1817},[],[1819],{"type":46,"value":1820},"machine_id",{"type":46,"value":94},{"type":40,"tag":87,"props":1823,"children":1825},{"className":1824},[],[1826],{"type":46,"value":1827},"pid",{"type":46,"value":94},{"type":40,"tag":87,"props":1830,"children":1832},{"className":1831},[],[1833],{"type":46,"value":1834},"stream_id",{"type":46,"value":94},{"type":40,"tag":87,"props":1837,"children":1839},{"className":1838},[],[1840],{"type":46,"value":1841},"systemd_cgroup",{"type":46,"value":94},{"type":40,"tag":87,"props":1844,"children":1846},{"className":1845},[],[1847],{"type":46,"value":1848},"systemd_invocation_id",{"type":46,"value":94},{"type":40,"tag":87,"props":1851,"children":1853},{"className":1852},[],[1854],{"type":46,"value":1855},"systemd_slice",{"type":46,"value":94},{"type":40,"tag":87,"props":1858,"children":1860},{"className":1859},[],[1861],{"type":46,"value":1862},"systemd_unit",{"type":46,"value":94},{"type":40,"tag":87,"props":1865,"children":1867},{"className":1866},[],[1868],{"type":46,"value":1869},"transport",{"type":46,"value":94},{"type":40,"tag":87,"props":1872,"children":1874},{"className":1873},[],[1875],{"type":46,"value":1876},"uid",{"type":40,"tag":49,"props":1878,"children":1879},{},[1880,1882,1887,1889,1894,1896,1902,1904,1909,1911,1917,1919,1924,1925,1930,1931,1936],{"type":46,"value":1881},"Almost all are high-cardinality. ",{"type":40,"tag":69,"props":1883,"children":1884},{},[1885],{"type":46,"value":1886},"Keep",{"type":46,"value":1888}," ",{"type":40,"tag":87,"props":1890,"children":1892},{"className":1891},[],[1893],{"type":46,"value":395},{"type":46,"value":1895}," (hostname) and ",{"type":40,"tag":87,"props":1897,"children":1899},{"className":1898},[],[1900],{"type":46,"value":1901},"unit",{"type":46,"value":1903}," (",{"type":40,"tag":87,"props":1905,"children":1907},{"className":1906},[],[1908],{"type":46,"value":1862},{"type":46,"value":1910},", e.g. ",{"type":40,"tag":87,"props":1912,"children":1914},{"className":1913},[],[1915],{"type":46,"value":1916},"nginx.service",{"type":46,"value":1918},"), plus any allowlisted correlation labels present on the stream (",{"type":40,"tag":87,"props":1920,"children":1922},{"className":1921},[],[1923],{"type":46,"value":185},{"type":46,"value":94},{"type":40,"tag":87,"props":1926,"children":1928},{"className":1927},[],[1929],{"type":46,"value":192},{"type":46,"value":94},{"type":40,"tag":87,"props":1932,"children":1934},{"className":1933},[],[1935],{"type":46,"value":200},{"type":46,"value":1937},").",{"type":40,"tag":49,"props":1939,"children":1940},{},[1941],{"type":46,"value":1942},"Drop other non-allowlisted high-cardinality journal labels (not platform keys):",{"type":40,"tag":838,"props":1944,"children":1946},{"className":1532,"code":1945,"language":1534,"meta":846,"style":846},"loki.process \"journal_labels\" {\n forward_to = [...]\n stage.label_keep {\n values = [\"instance\", \"unit\", \"env\", \"cluster\", \"service_name\", \"deployment_environment\", \"job\"]\n }\n}\n",[1947],{"type":40,"tag":87,"props":1948,"children":1949},{"__ignoreMap":846},[1950,1958,1966,1974,1982,1990],{"type":40,"tag":1540,"props":1951,"children":1952},{"class":1542,"line":1543},[1953],{"type":40,"tag":1540,"props":1954,"children":1955},{},[1956],{"type":46,"value":1957},"loki.process \"journal_labels\" {\n",{"type":40,"tag":1540,"props":1959,"children":1960},{"class":1542,"line":1552},[1961],{"type":40,"tag":1540,"props":1962,"children":1963},{},[1964],{"type":46,"value":1965}," forward_to = [...]\n",{"type":40,"tag":1540,"props":1967,"children":1968},{"class":1542,"line":1561},[1969],{"type":40,"tag":1540,"props":1970,"children":1971},{},[1972],{"type":46,"value":1973}," stage.label_keep {\n",{"type":40,"tag":1540,"props":1975,"children":1976},{"class":1542,"line":1570},[1977],{"type":40,"tag":1540,"props":1978,"children":1979},{},[1980],{"type":46,"value":1981}," values = [\"instance\", \"unit\", \"env\", \"cluster\", \"service_name\", \"deployment_environment\", \"job\"]\n",{"type":40,"tag":1540,"props":1983,"children":1984},{"class":1542,"line":1579},[1985],{"type":40,"tag":1540,"props":1986,"children":1987},{},[1988],{"type":46,"value":1989}," }\n",{"type":40,"tag":1540,"props":1991,"children":1992},{"class":1542,"line":1588},[1993],{"type":40,"tag":1540,"props":1994,"children":1995},{},[1996],{"type":46,"value":1594},{"type":40,"tag":55,"props":1998,"children":1999},{},[],{"type":40,"tag":59,"props":2001,"children":2003},{"id":2002},"structured-metadata",[2004],{"type":46,"value":2005},"Structured Metadata",{"type":40,"tag":49,"props":2007,"children":2008},{},[2009,2011,2016],{"type":46,"value":2010},"Structured metadata attaches key-value pairs to log entries ",{"type":40,"tag":111,"props":2012,"children":2013},{},[2014],{"type":46,"value":2015},"without",{"type":46,"value":2017}," making them index labels. The ideal home for high-cardinality values users occasionally need.",{"type":40,"tag":49,"props":2019,"children":2020},{},[2021,2026,2028,2034],{"type":40,"tag":69,"props":2022,"children":2023},{},[2024],{"type":46,"value":2025},"Requires",{"type":46,"value":2027},": Loki 2.9+, Grafana Agent\u002FAlloy. Enable via ",{"type":40,"tag":87,"props":2029,"children":2031},{"className":2030},[],[2032],{"type":46,"value":2033},"limits_config",{"type":46,"value":2035},":",{"type":40,"tag":838,"props":2037,"children":2041},{"className":2038,"code":2039,"language":2040,"meta":846,"style":846},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","limits_config:\n allow_structured_metadata: true\n","yaml",[2042],{"type":40,"tag":87,"props":2043,"children":2044},{"__ignoreMap":846},[2045,2058],{"type":40,"tag":1540,"props":2046,"children":2047},{"class":1542,"line":1543},[2048,2053],{"type":40,"tag":1540,"props":2049,"children":2051},{"style":2050},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[2052],{"type":46,"value":2033},{"type":40,"tag":1540,"props":2054,"children":2056},{"style":2055},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[2057],{"type":46,"value":1766},{"type":40,"tag":1540,"props":2059,"children":2060},{"class":1542,"line":1552},[2061,2066,2070],{"type":40,"tag":1540,"props":2062,"children":2063},{"style":2050},[2064],{"type":46,"value":2065}," allow_structured_metadata",{"type":40,"tag":1540,"props":2067,"children":2068},{"style":2055},[2069],{"type":46,"value":2035},{"type":40,"tag":1540,"props":2071,"children":2073},{"style":2072},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[2074],{"type":46,"value":2075}," true\n",{"type":40,"tag":49,"props":2077,"children":2078},{},[2079,2084],{"type":40,"tag":69,"props":2080,"children":2081},{},[2082],{"type":46,"value":2083},"Good candidates for structured metadata",{"type":46,"value":2085}," (not labels):",{"type":40,"tag":129,"props":2087,"children":2088},{},[2089,2099,2110,2132,2147,2156],{"type":40,"tag":133,"props":2090,"children":2091},{},[2092,2097],{"type":40,"tag":87,"props":2093,"children":2095},{"className":2094},[],[2096],{"type":46,"value":92},{"type":46,"value":2098}," — K8s pod name",{"type":40,"tag":133,"props":2100,"children":2101},{},[2102,2108],{"type":40,"tag":87,"props":2103,"children":2105},{"className":2104},[],[2106],{"type":46,"value":2107},"node",{"type":46,"value":2109}," — K8s worker node",{"type":40,"tag":133,"props":2111,"children":2112},{},[2113,2119,2120,2126,2127],{"type":40,"tag":87,"props":2114,"children":2116},{"className":2115},[],[2117],{"type":46,"value":2118},"version",{"type":46,"value":291},{"type":40,"tag":87,"props":2121,"children":2123},{"className":2122},[],[2124],{"type":46,"value":2125},"image",{"type":46,"value":291},{"type":40,"tag":87,"props":2128,"children":2130},{"className":2129},[],[2131],{"type":46,"value":15},{"type":40,"tag":133,"props":2133,"children":2134},{},[2135,2141,2142],{"type":40,"tag":87,"props":2136,"children":2138},{"className":2137},[],[2139],{"type":46,"value":2140},"trace_id",{"type":46,"value":291},{"type":40,"tag":87,"props":2143,"children":2145},{"className":2144},[],[2146],{"type":46,"value":100},{"type":40,"tag":133,"props":2148,"children":2149},{},[2150],{"type":40,"tag":87,"props":2151,"children":2153},{"className":2152},[],[2154],{"type":46,"value":2155},"process_id",{"type":40,"tag":133,"props":2157,"children":2158},{},[2159,2165],{"type":40,"tag":87,"props":2160,"children":2162},{"className":2161},[],[2163],{"type":46,"value":2164},"restarted",{"type":46,"value":2166}," — pod restart timestamp",{"type":40,"tag":49,"props":2168,"children":2169},{},[2170],{"type":46,"value":2171},"Query structured metadata at query time without a parser:",{"type":40,"tag":838,"props":2173,"children":2177},{"className":2174,"code":2175,"language":2176,"meta":846,"style":846},"language-logql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{service_name=\"payment-api\"} | pod=\"payment-api-7f9d4b-xk2r9\"\n","logql",[2178],{"type":40,"tag":87,"props":2179,"children":2180},{"__ignoreMap":846},[2181],{"type":40,"tag":1540,"props":2182,"children":2183},{"class":1542,"line":1543},[2184],{"type":40,"tag":1540,"props":2185,"children":2186},{},[2187],{"type":46,"value":2175},{"type":40,"tag":55,"props":2189,"children":2190},{},[],{"type":40,"tag":59,"props":2192,"children":2194},{"id":2193},"embedding-metadata-in-log-lines",[2195],{"type":46,"value":2196},"Embedding Metadata in Log Lines",{"type":40,"tag":49,"props":2198,"children":2199},{},[2200],{"type":46,"value":2201},"When structured metadata isn't available, embed high-cardinality values into the log line rather than using them as labels.",{"type":40,"tag":241,"props":2203,"children":2205},{"id":2204},"method-1-stagetemplate-append-to-log-line",[2206],{"type":46,"value":2207},"Method 1: stage.template (append to log line)",{"type":40,"tag":838,"props":2209,"children":2211},{"className":1532,"code":2210,"language":1534,"meta":846,"style":846},"loki.process \"embed_pod\" {\n forward_to = [...]\n\n \u002F\u002F For JSON logs\n stage.match {\n selector = \"{} |~ \\\"^\\\\s*\\\\{\\\"\"\n stage.replace {\n expression = \"\\\\}$\"\n replace = \"\"\n }\n stage.template {\n source = \"log_line\"\n template = \"{{ .Entry }},\\\"_pod\\\":\\\"{{ .pod }}\\\"}\"\n }\n }\n\n \u002F\u002F For text logs\n stage.match {\n selector = \"{} !~ \\\"^\\\\s*\\\\{\\\"\"\n stage.template {\n source = \"log_line\"\n template = \"{{ .Entry }} _pod={{ .pod }}\"\n }\n }\n\n stage.output { source = \"log_line\" }\n}\n",[2212],{"type":40,"tag":87,"props":2213,"children":2214},{"__ignoreMap":846},[2215,2223,2230,2239,2247,2255,2263,2271,2280,2289,2297,2306,2315,2324,2332,2340,2347,2356,2364,2373,2381,2389,2398,2406,2414,2422,2431],{"type":40,"tag":1540,"props":2216,"children":2217},{"class":1542,"line":1543},[2218],{"type":40,"tag":1540,"props":2219,"children":2220},{},[2221],{"type":46,"value":2222},"loki.process \"embed_pod\" {\n",{"type":40,"tag":1540,"props":2224,"children":2225},{"class":1542,"line":1552},[2226],{"type":40,"tag":1540,"props":2227,"children":2228},{},[2229],{"type":46,"value":1965},{"type":40,"tag":1540,"props":2231,"children":2232},{"class":1542,"line":1561},[2233],{"type":40,"tag":1540,"props":2234,"children":2236},{"emptyLinePlaceholder":2235},true,[2237],{"type":46,"value":2238},"\n",{"type":40,"tag":1540,"props":2240,"children":2241},{"class":1542,"line":1570},[2242],{"type":40,"tag":1540,"props":2243,"children":2244},{},[2245],{"type":46,"value":2246}," \u002F\u002F For JSON logs\n",{"type":40,"tag":1540,"props":2248,"children":2249},{"class":1542,"line":1579},[2250],{"type":40,"tag":1540,"props":2251,"children":2252},{},[2253],{"type":46,"value":2254}," stage.match {\n",{"type":40,"tag":1540,"props":2256,"children":2257},{"class":1542,"line":1588},[2258],{"type":40,"tag":1540,"props":2259,"children":2260},{},[2261],{"type":46,"value":2262}," selector = \"{} |~ \\\"^\\\\s*\\\\{\\\"\"\n",{"type":40,"tag":1540,"props":2264,"children":2265},{"class":1542,"line":1731},[2266],{"type":40,"tag":1540,"props":2267,"children":2268},{},[2269],{"type":46,"value":2270}," stage.replace {\n",{"type":40,"tag":1540,"props":2272,"children":2274},{"class":1542,"line":2273},8,[2275],{"type":40,"tag":1540,"props":2276,"children":2277},{},[2278],{"type":46,"value":2279}," expression = \"\\\\}$\"\n",{"type":40,"tag":1540,"props":2281,"children":2283},{"class":1542,"line":2282},9,[2284],{"type":40,"tag":1540,"props":2285,"children":2286},{},[2287],{"type":46,"value":2288}," replace = \"\"\n",{"type":40,"tag":1540,"props":2290,"children":2292},{"class":1542,"line":2291},10,[2293],{"type":40,"tag":1540,"props":2294,"children":2295},{},[2296],{"type":46,"value":1989},{"type":40,"tag":1540,"props":2298,"children":2300},{"class":1542,"line":2299},11,[2301],{"type":40,"tag":1540,"props":2302,"children":2303},{},[2304],{"type":46,"value":2305}," stage.template {\n",{"type":40,"tag":1540,"props":2307,"children":2309},{"class":1542,"line":2308},12,[2310],{"type":40,"tag":1540,"props":2311,"children":2312},{},[2313],{"type":46,"value":2314}," source = \"log_line\"\n",{"type":40,"tag":1540,"props":2316,"children":2318},{"class":1542,"line":2317},13,[2319],{"type":40,"tag":1540,"props":2320,"children":2321},{},[2322],{"type":46,"value":2323}," template = \"{{ .Entry }},\\\"_pod\\\":\\\"{{ .pod }}\\\"}\"\n",{"type":40,"tag":1540,"props":2325,"children":2327},{"class":1542,"line":2326},14,[2328],{"type":40,"tag":1540,"props":2329,"children":2330},{},[2331],{"type":46,"value":1989},{"type":40,"tag":1540,"props":2333,"children":2335},{"class":1542,"line":2334},15,[2336],{"type":40,"tag":1540,"props":2337,"children":2338},{},[2339],{"type":46,"value":1989},{"type":40,"tag":1540,"props":2341,"children":2342},{"class":1542,"line":27},[2343],{"type":40,"tag":1540,"props":2344,"children":2345},{"emptyLinePlaceholder":2235},[2346],{"type":46,"value":2238},{"type":40,"tag":1540,"props":2348,"children":2350},{"class":1542,"line":2349},17,[2351],{"type":40,"tag":1540,"props":2352,"children":2353},{},[2354],{"type":46,"value":2355}," \u002F\u002F For text logs\n",{"type":40,"tag":1540,"props":2357,"children":2359},{"class":1542,"line":2358},18,[2360],{"type":40,"tag":1540,"props":2361,"children":2362},{},[2363],{"type":46,"value":2254},{"type":40,"tag":1540,"props":2365,"children":2367},{"class":1542,"line":2366},19,[2368],{"type":40,"tag":1540,"props":2369,"children":2370},{},[2371],{"type":46,"value":2372}," selector = \"{} !~ \\\"^\\\\s*\\\\{\\\"\"\n",{"type":40,"tag":1540,"props":2374,"children":2376},{"class":1542,"line":2375},20,[2377],{"type":40,"tag":1540,"props":2378,"children":2379},{},[2380],{"type":46,"value":2305},{"type":40,"tag":1540,"props":2382,"children":2384},{"class":1542,"line":2383},21,[2385],{"type":40,"tag":1540,"props":2386,"children":2387},{},[2388],{"type":46,"value":2314},{"type":40,"tag":1540,"props":2390,"children":2392},{"class":1542,"line":2391},22,[2393],{"type":40,"tag":1540,"props":2394,"children":2395},{},[2396],{"type":46,"value":2397}," template = \"{{ .Entry }} _pod={{ .pod }}\"\n",{"type":40,"tag":1540,"props":2399,"children":2401},{"class":1542,"line":2400},23,[2402],{"type":40,"tag":1540,"props":2403,"children":2404},{},[2405],{"type":46,"value":1989},{"type":40,"tag":1540,"props":2407,"children":2409},{"class":1542,"line":2408},24,[2410],{"type":40,"tag":1540,"props":2411,"children":2412},{},[2413],{"type":46,"value":1989},{"type":40,"tag":1540,"props":2415,"children":2417},{"class":1542,"line":2416},25,[2418],{"type":40,"tag":1540,"props":2419,"children":2420},{"emptyLinePlaceholder":2235},[2421],{"type":46,"value":2238},{"type":40,"tag":1540,"props":2423,"children":2425},{"class":1542,"line":2424},26,[2426],{"type":40,"tag":1540,"props":2427,"children":2428},{},[2429],{"type":46,"value":2430}," stage.output { source = \"log_line\" }\n",{"type":40,"tag":1540,"props":2432,"children":2434},{"class":1542,"line":2433},27,[2435],{"type":40,"tag":1540,"props":2436,"children":2437},{},[2438],{"type":46,"value":1594},{"type":40,"tag":49,"props":2440,"children":2441},{},[2442,2444],{"type":46,"value":2443},"Result: ",{"type":40,"tag":87,"props":2445,"children":2447},{"className":2446},[],[2448],{"type":46,"value":2449},"ts=... msg=\"...\" _pod=agent-logs-cqhfk",{"type":40,"tag":49,"props":2451,"children":2452},{},[2453],{"type":46,"value":2454},"Query by aggregate (normal use):",{"type":40,"tag":838,"props":2456,"children":2458},{"className":2174,"code":2457,"language":2176,"meta":846,"style":846},"sum(count_over_time({workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"}[1m]))\n",[2459],{"type":40,"tag":87,"props":2460,"children":2461},{"__ignoreMap":846},[2462],{"type":40,"tag":1540,"props":2463,"children":2464},{"class":1542,"line":1543},[2465],{"type":40,"tag":1540,"props":2466,"children":2467},{},[2468],{"type":46,"value":2457},{"type":40,"tag":49,"props":2470,"children":2471},{},[2472],{"type":46,"value":2473},"Query a specific pod (edge case debugging):",{"type":40,"tag":838,"props":2475,"children":2477},{"className":2174,"code":2476,"language":2176,"meta":846,"style":846},"{workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"} |= `_pod=payment-api-3`\n",[2478],{"type":40,"tag":87,"props":2479,"children":2480},{"__ignoreMap":846},[2481],{"type":40,"tag":1540,"props":2482,"children":2483},{"class":1542,"line":1543},[2484],{"type":40,"tag":1540,"props":2485,"children":2486},{},[2487],{"type":46,"value":2476},{"type":40,"tag":241,"props":2489,"children":2491},{"id":2490},"method-2-stagepack-json-envelope",[2492],{"type":46,"value":2493},"Method 2: stage.pack (JSON envelope)",{"type":40,"tag":838,"props":2495,"children":2497},{"className":1532,"code":2496,"language":1534,"meta":846,"style":846},"loki.process \"pack_pod\" {\n forward_to = [...]\n stage.pack {\n labels = [\"pod\"]\n ingest_timestamp = false\n }\n}\n",[2498],{"type":40,"tag":87,"props":2499,"children":2500},{"__ignoreMap":846},[2501,2509,2516,2524,2532,2540,2547],{"type":40,"tag":1540,"props":2502,"children":2503},{"class":1542,"line":1543},[2504],{"type":40,"tag":1540,"props":2505,"children":2506},{},[2507],{"type":46,"value":2508},"loki.process \"pack_pod\" {\n",{"type":40,"tag":1540,"props":2510,"children":2511},{"class":1542,"line":1552},[2512],{"type":40,"tag":1540,"props":2513,"children":2514},{},[2515],{"type":46,"value":1965},{"type":40,"tag":1540,"props":2517,"children":2518},{"class":1542,"line":1561},[2519],{"type":40,"tag":1540,"props":2520,"children":2521},{},[2522],{"type":46,"value":2523}," stage.pack {\n",{"type":40,"tag":1540,"props":2525,"children":2526},{"class":1542,"line":1570},[2527],{"type":40,"tag":1540,"props":2528,"children":2529},{},[2530],{"type":46,"value":2531}," labels = [\"pod\"]\n",{"type":40,"tag":1540,"props":2533,"children":2534},{"class":1542,"line":1579},[2535],{"type":40,"tag":1540,"props":2536,"children":2537},{},[2538],{"type":46,"value":2539}," ingest_timestamp = false\n",{"type":40,"tag":1540,"props":2541,"children":2542},{"class":1542,"line":1588},[2543],{"type":40,"tag":1540,"props":2544,"children":2545},{},[2546],{"type":46,"value":1989},{"type":40,"tag":1540,"props":2548,"children":2549},{"class":1542,"line":1731},[2550],{"type":40,"tag":1540,"props":2551,"children":2552},{},[2553],{"type":46,"value":1594},{"type":40,"tag":49,"props":2555,"children":2556},{},[2557,2559],{"type":46,"value":2558},"Packed result: ",{"type":40,"tag":87,"props":2560,"children":2562},{"className":2561},[],[2563],{"type":46,"value":2564},"{\"_entry\": \"original log line\", \"pod\": \"agent-logs-cqhfk\"}",{"type":40,"tag":49,"props":2566,"children":2567},{},[2568],{"type":46,"value":2569},"Unpack at query time:",{"type":40,"tag":838,"props":2571,"children":2573},{"className":2174,"code":2572,"language":2176,"meta":846,"style":846},"{workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"}\n |= `agent-logs-cqhfk`\n | unpack\n",[2574],{"type":40,"tag":87,"props":2575,"children":2576},{"__ignoreMap":846},[2577,2585,2593],{"type":40,"tag":1540,"props":2578,"children":2579},{"class":1542,"line":1543},[2580],{"type":40,"tag":1540,"props":2581,"children":2582},{},[2583],{"type":46,"value":2584},"{workload=\"ReplicaSet\u002Fpayment-api\", level=\"error\"}\n",{"type":40,"tag":1540,"props":2586,"children":2587},{"class":1542,"line":1552},[2588],{"type":40,"tag":1540,"props":2589,"children":2590},{},[2591],{"type":46,"value":2592}," |= `agent-logs-cqhfk`\n",{"type":40,"tag":1540,"props":2594,"children":2595},{"class":1542,"line":1561},[2596],{"type":40,"tag":1540,"props":2597,"children":2598},{},[2599],{"type":46,"value":2600}," | unpack\n",{"type":40,"tag":55,"props":2602,"children":2603},{},[],{"type":40,"tag":59,"props":2605,"children":2607},{"id":2606},"performance-bottleneck-diagnosis",[2608],{"type":46,"value":2609},"Performance Bottleneck Diagnosis",{"type":40,"tag":49,"props":2611,"children":2612},{},[2613,2615,2621],{"type":46,"value":2614},"When a user reports slow queries, identify where time is spent using Querier ",{"type":40,"tag":87,"props":2616,"children":2618},{"className":2617},[],[2619],{"type":46,"value":2620},"metrics.go",{"type":46,"value":2622}," logs.",{"type":40,"tag":241,"props":2624,"children":2626},{"id":2625},"four-query-stages",[2627],{"type":46,"value":2628},"Four Query Stages",{"type":40,"tag":248,"props":2630,"children":2631},{},[2632,2658],{"type":40,"tag":252,"props":2633,"children":2634},{},[2635],{"type":40,"tag":256,"props":2636,"children":2637},{},[2638,2643,2648,2653],{"type":40,"tag":260,"props":2639,"children":2640},{},[2641],{"type":46,"value":2642},"Stage",{"type":40,"tag":260,"props":2644,"children":2645},{},[2646],{"type":46,"value":2647},"Metric",{"type":40,"tag":260,"props":2649,"children":2650},{},[2651],{"type":46,"value":2652},"High Value Means",{"type":40,"tag":260,"props":2654,"children":2655},{},[2656],{"type":46,"value":2657},"Fix",{"type":40,"tag":275,"props":2659,"children":2660},{},[2661,2688,2715,2748],{"type":40,"tag":256,"props":2662,"children":2663},{},[2664,2669,2678,2683],{"type":40,"tag":282,"props":2665,"children":2666},{},[2667],{"type":46,"value":2668},"Queue",{"type":40,"tag":282,"props":2670,"children":2671},{},[2672],{"type":40,"tag":87,"props":2673,"children":2675},{"className":2674},[],[2676],{"type":46,"value":2677},"queue_time",{"type":40,"tag":282,"props":2679,"children":2680},{},[2681],{"type":46,"value":2682},"Not enough Queriers",{"type":40,"tag":282,"props":2684,"children":2685},{},[2686],{"type":46,"value":2687},"Add Queriers or reduce parallelism",{"type":40,"tag":256,"props":2689,"children":2690},{},[2691,2696,2705,2710],{"type":40,"tag":282,"props":2692,"children":2693},{},[2694],{"type":46,"value":2695},"Index",{"type":40,"tag":282,"props":2697,"children":2698},{},[2699],{"type":40,"tag":87,"props":2700,"children":2702},{"className":2701},[],[2703],{"type":46,"value":2704},"chunk_refs_fetch_time",{"type":40,"tag":282,"props":2706,"children":2707},{},[2708],{"type":46,"value":2709},"Need more Index Gateway instances",{"type":40,"tag":282,"props":2711,"children":2712},{},[2713],{"type":46,"value":2714},"Scale index-gateways; check CPU",{"type":40,"tag":256,"props":2716,"children":2717},{},[2718,2723,2732,2737],{"type":40,"tag":282,"props":2719,"children":2720},{},[2721],{"type":46,"value":2722},"Storage",{"type":40,"tag":282,"props":2724,"children":2725},{},[2726],{"type":40,"tag":87,"props":2727,"children":2729},{"className":2728},[],[2730],{"type":46,"value":2731},"store_chunks_download_time",{"type":40,"tag":282,"props":2733,"children":2734},{},[2735],{"type":46,"value":2736},"Chunks too small OR storage bottleneck",{"type":40,"tag":282,"props":2738,"children":2739},{},[2740,2742],{"type":46,"value":2741},"Check avg chunk size: ",{"type":40,"tag":87,"props":2743,"children":2745},{"className":2744},[],[2746],{"type":46,"value":2747},"total_bytes \u002F cache_chunk_req",{"type":40,"tag":256,"props":2749,"children":2750},{},[2751,2756,2765,2770],{"type":40,"tag":282,"props":2752,"children":2753},{},[2754],{"type":46,"value":2755},"Execution",{"type":40,"tag":282,"props":2757,"children":2758},{},[2759],{"type":40,"tag":87,"props":2760,"children":2762},{"className":2761},[],[2763],{"type":46,"value":2764},"duration - chunk_refs_fetch_time - store_chunks_download_time",{"type":40,"tag":282,"props":2766,"children":2767},{},[2768],{"type":46,"value":2769},"CPU-intensive regex, or too many tiny log lines",{"type":40,"tag":282,"props":2771,"children":2772},{},[2773],{"type":46,"value":2774},"Reduce regex; add CPU; increase parallelism",{"type":40,"tag":49,"props":2776,"children":2777},{},[2778,2783],{"type":40,"tag":69,"props":2779,"children":2780},{},[2781],{"type":46,"value":2782},"Ideally, the majority of time is spent in Execution.",{"type":46,"value":2784}," If not, that indicates infrastructure or label design problems.",{"type":40,"tag":241,"props":2786,"children":2788},{"id":2787},"checking-chunk-size",[2789],{"type":46,"value":2790},"Checking Chunk Size",{"type":40,"tag":838,"props":2792,"children":2795},{"className":2793,"code":2794,"language":46},[841],"avg chunk size = total_bytes \u002F cache_chunk_req\n",[2796],{"type":40,"tag":87,"props":2797,"children":2798},{"__ignoreMap":846},[2799],{"type":46,"value":2794},{"type":40,"tag":49,"props":2801,"children":2802},{},[2803],{"type":46,"value":2804},"If the result is a few hundred bytes or kilobytes (instead of megabytes), chunks are too small. This means labels are over-splitting data into too many streams. Revisit cardinality — demote non-allowlisted high-card labels or stabilize protected-label values.",{"type":40,"tag":241,"props":2806,"children":2808},{"id":2807},"common-label-related-performance-problems",[2809],{"type":46,"value":2810},"Common Label-Related Performance Problems",{"type":40,"tag":49,"props":2812,"children":2813},{},[2814],{"type":40,"tag":69,"props":2815,"children":2816},{},[2817],{"type":46,"value":2818},"Problem: Query scans too many streams",{"type":40,"tag":129,"props":2820,"children":2821},{},[2822,2827],{"type":40,"tag":133,"props":2823,"children":2824},{},[2825],{"type":46,"value":2826},"Cause: High-cardinality labels exist but aren't specified in the query selector",{"type":40,"tag":133,"props":2828,"children":2829},{},[2830,2832,2836],{"type":46,"value":2831},"Fix: Demote the label off the index after a migration check, or ensure queries always include it as a filter. Never demote allowlisted correlation labels — stabilize their values instead (",{"type":40,"tag":211,"props":2833,"children":2834},{"href":213},[2835],{"type":46,"value":484},{"type":46,"value":486},{"type":40,"tag":49,"props":2838,"children":2839},{},[2840,2853,2854,2860],{"type":40,"tag":69,"props":2841,"children":2842},{},[2843,2845,2851],{"type":46,"value":2844},"Problem: High ",{"type":40,"tag":87,"props":2846,"children":2848},{"className":2847},[],[2849],{"type":46,"value":2850},"post_filter_lines",{"type":46,"value":2852}," discard ratio",{"type":46,"value":1903},{"type":40,"tag":87,"props":2855,"children":2857},{"className":2856},[],[2858],{"type":46,"value":2859},"post_filter_lines \u003C\u003C total_lines",{"type":46,"value":486},{"type":40,"tag":129,"props":2862,"children":2863},{},[2864,2869],{"type":40,"tag":133,"props":2865,"children":2866},{},[2867],{"type":46,"value":2868},"Cause: Insufficient label selectivity; query scans and discards most logs",{"type":40,"tag":133,"props":2870,"children":2871},{},[2872,2874,2879,2880,2885,2886,2891,2892,2897],{"type":46,"value":2873},"Fix: Add labels matching user access patterns (",{"type":40,"tag":87,"props":2875,"children":2877},{"className":2876},[],[2878],{"type":46,"value":348},{"type":46,"value":94},{"type":40,"tag":87,"props":2881,"children":2883},{"className":2882},[],[2884],{"type":46,"value":1307},{"type":46,"value":94},{"type":40,"tag":87,"props":2887,"children":2889},{"className":2888},[],[2890],{"type":46,"value":1290},{"type":46,"value":94},{"type":40,"tag":87,"props":2893,"children":2895},{"className":2894},[],[2896],{"type":46,"value":185},{"type":46,"value":486},{"type":40,"tag":49,"props":2899,"children":2900},{},[2901],{"type":40,"tag":69,"props":2902,"children":2903},{},[2904],{"type":46,"value":2905},"Problem: Small chunks",{"type":40,"tag":129,"props":2907,"children":2908},{},[2909,2914],{"type":40,"tag":133,"props":2910,"children":2911},{},[2912],{"type":46,"value":2913},"Cause: Too many labels creating too many fine-grained streams",{"type":40,"tag":133,"props":2915,"children":2916},{},[2917,2919,2924,2926,2931],{"type":46,"value":2918},"Fix: Demote high-cardinality non-allowlisted labels (e.g. ",{"type":40,"tag":87,"props":2920,"children":2922},{"className":2921},[],[2923],{"type":46,"value":92},{"type":46,"value":2925},") to consolidate streams; remediate protected-label ",{"type":40,"tag":111,"props":2927,"children":2928},{},[2929],{"type":46,"value":2930},"values",{"type":46,"value":2932}," if they are the splitter",{"type":40,"tag":241,"props":2934,"children":2936},{"id":2935},"query-optimization-quick-wins",[2937],{"type":46,"value":2938},"Query Optimization Quick Wins",{"type":40,"tag":656,"props":2940,"children":2941},{},[2942,2960,2971,2983,3004],{"type":40,"tag":133,"props":2943,"children":2944},{},[2945,2947,2952,2953,2958],{"type":46,"value":2946},"Add ",{"type":40,"tag":87,"props":2948,"children":2950},{"className":2949},[],[2951],{"type":46,"value":1290},{"type":46,"value":624},{"type":40,"tag":87,"props":2954,"children":2956},{"className":2955},[],[2957],{"type":46,"value":1307},{"type":46,"value":2959}," to narrow scope before line filters",{"type":40,"tag":133,"props":2961,"children":2962},{},[2963,2964,2969],{"type":46,"value":2946},{"type":40,"tag":87,"props":2965,"children":2967},{"className":2966},[],[2968],{"type":46,"value":348},{"type":46,"value":2970}," label + always use it in queries (filters out 94%+ of logs when searching for errors)",{"type":40,"tag":133,"props":2972,"children":2973},{},[2974,2976,2981],{"type":46,"value":2975},"Demote ",{"type":40,"tag":87,"props":2977,"children":2979},{"className":2978},[],[2980],{"type":46,"value":92},{"type":46,"value":2982}," off the index → reduces stream count by ~5× in typical K8s deployments (migrate selectors first)",{"type":40,"tag":133,"props":2984,"children":2985},{},[2986,2988,2994,2996,3002],{"type":46,"value":2987},"Replace regex line filters (",{"type":40,"tag":87,"props":2989,"children":2991},{"className":2990},[],[2992],{"type":46,"value":2993},"|~",{"type":46,"value":2995},") with exact filters (",{"type":40,"tag":87,"props":2997,"children":2999},{"className":2998},[],[3000],{"type":46,"value":3001},"|=",{"type":46,"value":3003},") where possible",{"type":40,"tag":133,"props":3005,"children":3006},{},[3007,3009,3014],{"type":46,"value":3008},"Keep ",{"type":40,"tag":87,"props":3010,"children":3012},{"className":3011},[],[3013],{"type":46,"value":185},{"type":46,"value":3015}," (and peers); if values are UUIDs\u002Fephemeral, normalize to a stable identity — do not drop the key",{"type":40,"tag":55,"props":3017,"children":3018},{},[],{"type":40,"tag":59,"props":3020,"children":3022},{"id":3021},"alloy-agent-configuration-patterns",[3023],{"type":46,"value":3024},"Alloy \u002F Agent Configuration Patterns",{"type":40,"tag":241,"props":3026,"children":3028},{"id":3027},"normalize-log-level",[3029],{"type":46,"value":3030},"Normalize Log Level",{"type":40,"tag":838,"props":3032,"children":3034},{"className":1532,"code":3033,"language":1534,"meta":846,"style":846},"loki.process \"normalize_level\" {\n forward_to = [...]\n stage.replace { source = \"level\"; expression = \"(?i)I(nfo)?\"; replace = \"info\" }\n stage.replace { source = \"level\"; expression = \"(?i)W(arn(ing)?)?\"; replace = \"warn\" }\n stage.replace { source = \"level\"; expression = \"(?i)E(rr(or)?)?\"; replace = \"error\" }\n stage.replace { source = \"level\"; expression = \"(?i)D(ebug?)?\"; replace = \"debug\" }\n stage.labels { values = { level = \"\" } }\n}\n",[3035],{"type":40,"tag":87,"props":3036,"children":3037},{"__ignoreMap":846},[3038,3046,3053,3061,3069,3077,3085,3093],{"type":40,"tag":1540,"props":3039,"children":3040},{"class":1542,"line":1543},[3041],{"type":40,"tag":1540,"props":3042,"children":3043},{},[3044],{"type":46,"value":3045},"loki.process \"normalize_level\" {\n",{"type":40,"tag":1540,"props":3047,"children":3048},{"class":1542,"line":1552},[3049],{"type":40,"tag":1540,"props":3050,"children":3051},{},[3052],{"type":46,"value":1965},{"type":40,"tag":1540,"props":3054,"children":3055},{"class":1542,"line":1561},[3056],{"type":40,"tag":1540,"props":3057,"children":3058},{},[3059],{"type":46,"value":3060}," stage.replace { source = \"level\"; expression = \"(?i)I(nfo)?\"; replace = \"info\" }\n",{"type":40,"tag":1540,"props":3062,"children":3063},{"class":1542,"line":1570},[3064],{"type":40,"tag":1540,"props":3065,"children":3066},{},[3067],{"type":46,"value":3068}," stage.replace { source = \"level\"; expression = \"(?i)W(arn(ing)?)?\"; replace = \"warn\" }\n",{"type":40,"tag":1540,"props":3070,"children":3071},{"class":1542,"line":1579},[3072],{"type":40,"tag":1540,"props":3073,"children":3074},{},[3075],{"type":46,"value":3076}," stage.replace { source = \"level\"; expression = \"(?i)E(rr(or)?)?\"; replace = \"error\" }\n",{"type":40,"tag":1540,"props":3078,"children":3079},{"class":1542,"line":1588},[3080],{"type":40,"tag":1540,"props":3081,"children":3082},{},[3083],{"type":46,"value":3084}," stage.replace { source = \"level\"; expression = \"(?i)D(ebug?)?\"; replace = \"debug\" }\n",{"type":40,"tag":1540,"props":3086,"children":3087},{"class":1542,"line":1731},[3088],{"type":40,"tag":1540,"props":3089,"children":3090},{},[3091],{"type":46,"value":3092}," stage.labels { values = { level = \"\" } }\n",{"type":40,"tag":1540,"props":3094,"children":3095},{"class":1542,"line":2273},[3096],{"type":40,"tag":1540,"props":3097,"children":3098},{},[3099],{"type":46,"value":1594},{"type":40,"tag":241,"props":3101,"children":3103},{"id":3102},"conditional-meta-label-extraction",[3104],{"type":46,"value":3105},"Conditional Meta-Label Extraction",{"type":40,"tag":838,"props":3107,"children":3109},{"className":1532,"code":3108,"language":1534,"meta":846,"style":846},"\u002F\u002F Only extract when the relevant field is present — avoids unnecessary cardinality\nloki.process \"conditional_extraction\" {\n forward_to = [...]\n stage.match {\n selector = \"{app=\\\"loki\\\"} |= \\\"component\\\"\"\n stage.logfmt { mapping = { \"component\" = \"\" } }\n stage.labels { values = { component = \"\" } }\n }\n}\n",[3110],{"type":40,"tag":87,"props":3111,"children":3112},{"__ignoreMap":846},[3113,3121,3129,3136,3143,3151,3159,3167,3174],{"type":40,"tag":1540,"props":3114,"children":3115},{"class":1542,"line":1543},[3116],{"type":40,"tag":1540,"props":3117,"children":3118},{},[3119],{"type":46,"value":3120},"\u002F\u002F Only extract when the relevant field is present — avoids unnecessary cardinality\n",{"type":40,"tag":1540,"props":3122,"children":3123},{"class":1542,"line":1552},[3124],{"type":40,"tag":1540,"props":3125,"children":3126},{},[3127],{"type":46,"value":3128},"loki.process \"conditional_extraction\" {\n",{"type":40,"tag":1540,"props":3130,"children":3131},{"class":1542,"line":1561},[3132],{"type":40,"tag":1540,"props":3133,"children":3134},{},[3135],{"type":46,"value":1965},{"type":40,"tag":1540,"props":3137,"children":3138},{"class":1542,"line":1570},[3139],{"type":40,"tag":1540,"props":3140,"children":3141},{},[3142],{"type":46,"value":2254},{"type":40,"tag":1540,"props":3144,"children":3145},{"class":1542,"line":1579},[3146],{"type":40,"tag":1540,"props":3147,"children":3148},{},[3149],{"type":46,"value":3150}," selector = \"{app=\\\"loki\\\"} |= \\\"component\\\"\"\n",{"type":40,"tag":1540,"props":3152,"children":3153},{"class":1542,"line":1588},[3154],{"type":40,"tag":1540,"props":3155,"children":3156},{},[3157],{"type":46,"value":3158}," stage.logfmt { mapping = { \"component\" = \"\" } }\n",{"type":40,"tag":1540,"props":3160,"children":3161},{"class":1542,"line":1731},[3162],{"type":40,"tag":1540,"props":3163,"children":3164},{},[3165],{"type":46,"value":3166}," stage.labels { values = { component = \"\" } }\n",{"type":40,"tag":1540,"props":3168,"children":3169},{"class":1542,"line":2273},[3170],{"type":40,"tag":1540,"props":3171,"children":3172},{},[3173],{"type":46,"value":1989},{"type":40,"tag":1540,"props":3175,"children":3176},{"class":1542,"line":2282},[3177],{"type":40,"tag":1540,"props":3178,"children":3179},{},[3180],{"type":46,"value":1594},{"type":40,"tag":241,"props":3182,"children":3184},{"id":3183},"enforce-approved-label-set-always-use-as-final-stage",[3185],{"type":46,"value":3186},"Enforce Approved Label Set (always use as final stage)",{"type":40,"tag":49,"props":3188,"children":3189},{},[3190,3192,3197,3198,3203,3204,3209,3211,3216,3217,3221],{"type":46,"value":3191},"Always include allowlisted correlation labels when present — never omit ",{"type":40,"tag":87,"props":3193,"children":3195},{"className":3194},[],[3196],{"type":46,"value":185},{"type":46,"value":94},{"type":40,"tag":87,"props":3199,"children":3201},{"className":3200},[],[3202],{"type":46,"value":192},{"type":46,"value":194},{"type":40,"tag":87,"props":3205,"children":3207},{"className":3206},[],[3208],{"type":46,"value":200},{"type":46,"value":3210}," from ",{"type":40,"tag":87,"props":3212,"children":3214},{"className":3213},[],[3215],{"type":46,"value":223},{"type":46,"value":1903},{"type":40,"tag":211,"props":3218,"children":3219},{"href":213},[3220],{"type":46,"value":484},{"type":46,"value":3222},"):",{"type":40,"tag":838,"props":3224,"children":3226},{"className":1532,"code":3225,"language":1534,"meta":846,"style":846},"loki.process \"enforce_labels\" {\n forward_to = [loki.write.default.receiver]\n \u002F\u002F ... other stages ...\n stage.label_keep {\n values = [\n \"service_name\", \"deployment_environment\", \"job\",\n \"env\", \"cluster\", \"level\", \"namespace\", \"workload\", \"container\",\n ]\n }\n}\n",[3227],{"type":40,"tag":87,"props":3228,"children":3229},{"__ignoreMap":846},[3230,3238,3246,3254,3261,3269,3277,3285,3293,3300],{"type":40,"tag":1540,"props":3231,"children":3232},{"class":1542,"line":1543},[3233],{"type":40,"tag":1540,"props":3234,"children":3235},{},[3236],{"type":46,"value":3237},"loki.process \"enforce_labels\" {\n",{"type":40,"tag":1540,"props":3239,"children":3240},{"class":1542,"line":1552},[3241],{"type":40,"tag":1540,"props":3242,"children":3243},{},[3244],{"type":46,"value":3245}," forward_to = [loki.write.default.receiver]\n",{"type":40,"tag":1540,"props":3247,"children":3248},{"class":1542,"line":1561},[3249],{"type":40,"tag":1540,"props":3250,"children":3251},{},[3252],{"type":46,"value":3253}," \u002F\u002F ... other stages ...\n",{"type":40,"tag":1540,"props":3255,"children":3256},{"class":1542,"line":1570},[3257],{"type":40,"tag":1540,"props":3258,"children":3259},{},[3260],{"type":46,"value":1973},{"type":40,"tag":1540,"props":3262,"children":3263},{"class":1542,"line":1579},[3264],{"type":40,"tag":1540,"props":3265,"children":3266},{},[3267],{"type":46,"value":3268}," values = [\n",{"type":40,"tag":1540,"props":3270,"children":3271},{"class":1542,"line":1588},[3272],{"type":40,"tag":1540,"props":3273,"children":3274},{},[3275],{"type":46,"value":3276}," \"service_name\", \"deployment_environment\", \"job\",\n",{"type":40,"tag":1540,"props":3278,"children":3279},{"class":1542,"line":1731},[3280],{"type":40,"tag":1540,"props":3281,"children":3282},{},[3283],{"type":46,"value":3284}," \"env\", \"cluster\", \"level\", \"namespace\", \"workload\", \"container\",\n",{"type":40,"tag":1540,"props":3286,"children":3287},{"class":1542,"line":2273},[3288],{"type":40,"tag":1540,"props":3289,"children":3290},{},[3291],{"type":46,"value":3292}," ]\n",{"type":40,"tag":1540,"props":3294,"children":3295},{"class":1542,"line":2282},[3296],{"type":40,"tag":1540,"props":3297,"children":3298},{},[3299],{"type":46,"value":1989},{"type":40,"tag":1540,"props":3301,"children":3302},{"class":1542,"line":2291},[3303],{"type":40,"tag":1540,"props":3304,"children":3305},{},[3306],{"type":46,"value":1594},{"type":40,"tag":241,"props":3308,"children":3310},{"id":3309},"soft-enforcement-inject-unknown-for-missing-labels",[3311],{"type":46,"value":3312},"Soft Enforcement (inject \"unknown\" for missing labels)",{"type":40,"tag":838,"props":3314,"children":3316},{"className":1532,"code":3315,"language":1534,"meta":846,"style":846},"stage.template {\n source = \"team\"\n template = \"{{ if .Value }}{{ .Value }}{{ else }}unknown{{ end }}\"\n}\nstage.labels { values = { team = \"\" } }\n",[3317],{"type":40,"tag":87,"props":3318,"children":3319},{"__ignoreMap":846},[3320,3328,3336,3344,3351],{"type":40,"tag":1540,"props":3321,"children":3322},{"class":1542,"line":1543},[3323],{"type":40,"tag":1540,"props":3324,"children":3325},{},[3326],{"type":46,"value":3327},"stage.template {\n",{"type":40,"tag":1540,"props":3329,"children":3330},{"class":1542,"line":1552},[3331],{"type":40,"tag":1540,"props":3332,"children":3333},{},[3334],{"type":46,"value":3335}," source = \"team\"\n",{"type":40,"tag":1540,"props":3337,"children":3338},{"class":1542,"line":1561},[3339],{"type":40,"tag":1540,"props":3340,"children":3341},{},[3342],{"type":46,"value":3343}," template = \"{{ if .Value }}{{ .Value }}{{ else }}unknown{{ end }}\"\n",{"type":40,"tag":1540,"props":3345,"children":3346},{"class":1542,"line":1570},[3347],{"type":40,"tag":1540,"props":3348,"children":3349},{},[3350],{"type":46,"value":1594},{"type":40,"tag":1540,"props":3352,"children":3353},{"class":1542,"line":1579},[3354],{"type":40,"tag":1540,"props":3355,"children":3356},{},[3357],{"type":46,"value":3358},"stage.labels { values = { team = \"\" } }\n",{"type":40,"tag":55,"props":3360,"children":3361},{},[],{"type":40,"tag":59,"props":3363,"children":3365},{"id":3364},"log-line-optimization",[3366],{"type":46,"value":3367},"Log Line Optimization",{"type":40,"tag":49,"props":3369,"children":3370},{},[3371,3373,3378],{"type":46,"value":3372},"Byte-level reductions (timestamps, ANSI, null JSON fields) for Scenario C savings — see ",{"type":40,"tag":211,"props":3374,"children":3376},{"href":3375},"references\u002Flog-line-optimization.md",[3377],{"type":46,"value":3375},{"type":46,"value":836},{"type":40,"tag":55,"props":3380,"children":3381},{},[],{"type":40,"tag":59,"props":3383,"children":3385},{"id":3384},"security-lbac",[3386],{"type":46,"value":3387},"Security & LBAC",{"type":40,"tag":49,"props":3389,"children":3390},{},[3391],{"type":46,"value":3392},"Grafana Enterprise Logs (GEL) supports Label-Based Access Control (LBAC). Any label can serve as an access control selector.",{"type":40,"tag":49,"props":3394,"children":3395},{},[3396,3401],{"type":40,"tag":69,"props":3397,"children":3398},{},[3399],{"type":46,"value":3400},"Best labels for LBAC",{"type":46,"value":2035},{"type":40,"tag":129,"props":3403,"children":3404},{},[3405,3443,3453,3469],{"type":40,"tag":133,"props":3406,"children":3407},{},[3408,3413,3415,3421,3422,3428,3429,3435,3436,3442],{"type":40,"tag":87,"props":3409,"children":3411},{"className":3410},[],[3412],{"type":46,"value":1174},{"type":46,"value":3414}," — data sensitivity (",{"type":40,"tag":87,"props":3416,"children":3418},{"className":3417},[],[3419],{"type":46,"value":3420},"public",{"type":46,"value":94},{"type":40,"tag":87,"props":3423,"children":3425},{"className":3424},[],[3426],{"type":46,"value":3427},"restricted",{"type":46,"value":94},{"type":40,"tag":87,"props":3430,"children":3432},{"className":3431},[],[3433],{"type":46,"value":3434},"confidential",{"type":46,"value":94},{"type":40,"tag":87,"props":3437,"children":3439},{"className":3438},[],[3440],{"type":46,"value":3441},"top-secret",{"type":46,"value":486},{"type":40,"tag":133,"props":3444,"children":3445},{},[3446,3451],{"type":40,"tag":87,"props":3447,"children":3449},{"className":3448},[],[3450],{"type":46,"value":1128},{"type":46,"value":3452}," — controls which teams can see which log origins",{"type":40,"tag":133,"props":3454,"children":3455},{},[3456,3461,3462,3467],{"type":40,"tag":87,"props":3457,"children":3459},{"className":3458},[],[3460],{"type":46,"value":1104},{"type":46,"value":291},{"type":40,"tag":87,"props":3463,"children":3465},{"className":3464},[],[3466],{"type":46,"value":1111},{"type":46,"value":3468}," — ownership-based access",{"type":40,"tag":133,"props":3470,"children":3471},{},[3472,3477],{"type":40,"tag":87,"props":3473,"children":3475},{"className":3474},[],[3476],{"type":46,"value":324},{"type":46,"value":3478}," — environment-level restrictions",{"type":40,"tag":49,"props":3480,"children":3481},{},[3482,3484,3490,3491,3497],{"type":46,"value":3483},"Static aggregate labels like ",{"type":40,"tag":87,"props":3485,"children":3487},{"className":3486},[],[3488],{"type":46,"value":3489},"owner=sysadmins",{"type":46,"value":624},{"type":40,"tag":87,"props":3492,"children":3494},{"className":3493},[],[3495],{"type":46,"value":3496},"category=database",{"type":46,"value":3498}," are particularly effective: one label value gates access to many log files, rather than requiring a long allowlist of filenames or streams.",{"type":40,"tag":55,"props":3500,"children":3501},{},[],{"type":40,"tag":59,"props":3503,"children":3505},{"id":3504},"the-8020-rule",[3506],{"type":46,"value":3507},"The 80\u002F20 Rule",{"type":40,"tag":49,"props":3509,"children":3510},{},[3511],{"type":46,"value":3512},"The most impactful improvements almost always come from these four changes:",{"type":40,"tag":656,"props":3514,"children":3515},{},[3516,3532,3548,3565],{"type":40,"tag":133,"props":3517,"children":3518},{},[3519,3530],{"type":40,"tag":69,"props":3520,"children":3521},{},[3522,3523,3528],{"type":46,"value":2975},{"type":40,"tag":87,"props":3524,"children":3526},{"className":3525},[],[3527],{"type":46,"value":92},{"type":46,"value":3529}," off the index",{"type":46,"value":3531}," (structured metadata) — biggest stream reduction in K8s; migrate selectors first",{"type":40,"tag":133,"props":3533,"children":3534},{},[3535,3546],{"type":40,"tag":69,"props":3536,"children":3537},{},[3538,3539,3544],{"type":46,"value":2946},{"type":40,"tag":87,"props":3540,"children":3542},{"className":3541},[],[3543],{"type":46,"value":348},{"type":46,"value":3545}," as a label AND always specify it in queries",{"type":46,"value":3547}," — can eliminate 94%+ of scanned data when searching for errors",{"type":40,"tag":133,"props":3549,"children":3550},{},[3551,3556,3558,3563],{"type":40,"tag":69,"props":3552,"children":3553},{},[3554],{"type":46,"value":3555},"Normalize label values",{"type":46,"value":3557}," — eliminates phantom duplicate streams from inconsistent casing; for ",{"type":40,"tag":87,"props":3559,"children":3561},{"className":3560},[],[3562],{"type":46,"value":185},{"type":46,"value":3564},", stabilize UUID\u002Fephemeral values (never drop the key)",{"type":40,"tag":133,"props":3566,"children":3567},{},[3568,3578],{"type":40,"tag":69,"props":3569,"children":3570},{},[3571,3573],{"type":46,"value":3572},"Normalize or demote ",{"type":40,"tag":87,"props":3574,"children":3576},{"className":3575},[],[3577],{"type":46,"value":1482},{"type":46,"value":3579}," in K8s — highly variable paths inflate stream count significantly",{"type":40,"tag":49,"props":3581,"children":3582},{},[3583,3585,3590,3591,3596,3597,3602],{"type":46,"value":3584},"Focus on these before anything else. Never \"fix\" cardinality by dropping ",{"type":40,"tag":87,"props":3586,"children":3588},{"className":3587},[],[3589],{"type":46,"value":185},{"type":46,"value":94},{"type":40,"tag":87,"props":3592,"children":3594},{"className":3593},[],[3595],{"type":46,"value":192},{"type":46,"value":194},{"type":40,"tag":87,"props":3598,"children":3600},{"className":3599},[],[3601],{"type":46,"value":200},{"type":46,"value":836},{"type":40,"tag":55,"props":3604,"children":3605},{},[],{"type":40,"tag":59,"props":3607,"children":3609},{"id":3608},"labels-to-avoid-quick-reference",[3610],{"type":46,"value":3611},"Labels to Avoid — Quick Reference",{"type":40,"tag":248,"props":3613,"children":3614},{},[3615,3635],{"type":40,"tag":252,"props":3616,"children":3617},{},[3618],{"type":40,"tag":256,"props":3619,"children":3620},{},[3621,3625,3630],{"type":40,"tag":260,"props":3622,"children":3623},{},[3624],{"type":46,"value":876},{"type":40,"tag":260,"props":3626,"children":3627},{},[3628],{"type":46,"value":3629},"Why",{"type":40,"tag":260,"props":3631,"children":3632},{},[3633],{"type":46,"value":3634},"Alternative",{"type":40,"tag":275,"props":3636,"children":3637},{},[3638,3673,3694,3720,3743,3783,3811],{"type":40,"tag":256,"props":3639,"children":3640},{},[3641,3649,3654],{"type":40,"tag":282,"props":3642,"children":3643},{},[3644],{"type":40,"tag":87,"props":3645,"children":3647},{"className":3646},[],[3648],{"type":46,"value":92},{"type":40,"tag":282,"props":3650,"children":3651},{},[3652],{"type":46,"value":3653},"Transient, high card",{"type":40,"tag":282,"props":3655,"children":3656},{},[3657,3659,3664,3666,3671],{"type":46,"value":3658},"Demote: ",{"type":40,"tag":87,"props":3660,"children":3662},{"className":3661},[],[3663],{"type":46,"value":1307},{"type":46,"value":3665}," label + ",{"type":40,"tag":87,"props":3667,"children":3669},{"className":3668},[],[3670],{"type":46,"value":92},{"type":46,"value":3672}," in structured metadata (migrate selectors)",{"type":40,"tag":256,"props":3674,"children":3675},{},[3676,3684,3689],{"type":40,"tag":282,"props":3677,"children":3678},{},[3679],{"type":40,"tag":87,"props":3680,"children":3682},{"className":3681},[],[3683],{"type":46,"value":100},{"type":40,"tag":282,"props":3685,"children":3686},{},[3687],{"type":46,"value":3688},"Unbounded — never valid as index label",{"type":40,"tag":282,"props":3690,"children":3691},{},[3692],{"type":46,"value":3693},"Keep only in log content",{"type":40,"tag":256,"props":3695,"children":3696},{},[3697,3711,3715],{"type":40,"tag":282,"props":3698,"children":3699},{},[3700,3705,3706],{"type":40,"tag":87,"props":3701,"children":3703},{"className":3702},[],[3704],{"type":46,"value":107},{"type":46,"value":291},{"type":40,"tag":87,"props":3707,"children":3709},{"className":3708},[],[3710],{"type":46,"value":2140},{"type":40,"tag":282,"props":3712,"children":3713},{},[3714],{"type":46,"value":3688},{"type":40,"tag":282,"props":3716,"children":3717},{},[3718],{"type":46,"value":3719},"Structured metadata",{"type":40,"tag":256,"props":3721,"children":3722},{},[3723,3733,3738],{"type":40,"tag":282,"props":3724,"children":3725},{},[3726,3731],{"type":40,"tag":87,"props":3727,"children":3729},{"className":3728},[],[3730],{"type":46,"value":1482},{"type":46,"value":3732}," (raw K8s path)",{"type":40,"tag":282,"props":3734,"children":3735},{},[3736],{"type":46,"value":3737},"Contains pod UID",{"type":40,"tag":282,"props":3739,"children":3740},{},[3741],{"type":46,"value":3742},"Normalize or demote after selector check",{"type":40,"tag":256,"props":3744,"children":3745},{},[3746,3756,3778],{"type":40,"tag":282,"props":3747,"children":3748},{},[3749,3751],{"type":46,"value":3750},"Unnormalized ",{"type":40,"tag":87,"props":3752,"children":3754},{"className":3753},[],[3755],{"type":46,"value":348},{"type":40,"tag":282,"props":3757,"children":3758},{},[3759,3764,3765,3770,3771,3776],{"type":40,"tag":87,"props":3760,"children":3762},{"className":3761},[],[3763],{"type":46,"value":589},{"type":46,"value":810},{"type":40,"tag":87,"props":3766,"children":3768},{"className":3767},[],[3769],{"type":46,"value":596},{"type":46,"value":810},{"type":40,"tag":87,"props":3772,"children":3774},{"className":3773},[],[3775],{"type":46,"value":603},{"type":46,"value":3777}," = 3 streams",{"type":40,"tag":282,"props":3779,"children":3780},{},[3781],{"type":46,"value":3782},"Normalize at collection time",{"type":40,"tag":256,"props":3784,"children":3785},{},[3786,3801,3806],{"type":40,"tag":282,"props":3787,"children":3788},{},[3789,3791,3796,3797],{"type":46,"value":3790},"UUID \u002F ephemeral ",{"type":40,"tag":87,"props":3792,"children":3794},{"className":3793},[],[3795],{"type":46,"value":185},{"type":46,"value":1888},{"type":40,"tag":111,"props":3798,"children":3799},{},[3800],{"type":46,"value":2930},{"type":40,"tag":282,"props":3802,"children":3803},{},[3804],{"type":46,"value":3805},"Inflates streams; key is still required",{"type":40,"tag":282,"props":3807,"children":3808},{},[3809],{"type":46,"value":3810},"Keep key; map values to stable service identity",{"type":40,"tag":256,"props":3812,"children":3813},{},[3814,3819,3824],{"type":40,"tag":282,"props":3815,"children":3816},{},[3817],{"type":46,"value":3818},"Any dynamically-named label key",{"type":40,"tag":282,"props":3820,"children":3821},{},[3822],{"type":46,"value":3823},"Cannot be bounded",{"type":40,"tag":282,"props":3825,"children":3826},{},[3827],{"type":46,"value":3828},"Use fixed keys with bounded values",{"type":40,"tag":49,"props":3830,"children":3831},{},[3832,3837,3838,3843,3844,3849,3850,3855,3857,3861],{"type":40,"tag":69,"props":3833,"children":3834},{},[3835],{"type":46,"value":3836},"Never drop:",{"type":46,"value":1888},{"type":40,"tag":87,"props":3839,"children":3841},{"className":3840},[],[3842],{"type":46,"value":185},{"type":46,"value":94},{"type":40,"tag":87,"props":3845,"children":3847},{"className":3846},[],[3848],{"type":46,"value":192},{"type":46,"value":94},{"type":40,"tag":87,"props":3851,"children":3853},{"className":3852},[],[3854],{"type":46,"value":200},{"type":46,"value":3856}," — see ",{"type":40,"tag":211,"props":3858,"children":3859},{"href":213},[3860],{"type":46,"value":213},{"type":46,"value":836},{"type":40,"tag":55,"props":3863,"children":3864},{},[],{"type":40,"tag":59,"props":3866,"children":3868},{"id":3867},"cost-impact-analysis",[3869],{"type":46,"value":3870},"Cost Impact Analysis",{"type":40,"tag":49,"props":3872,"children":3873},{},[3874,3876,3882,3884,3888],{"type":46,"value":3875},"Label hygiene alone does not cut billable ingest bytes ($0 direct). Volume savings come from enabled ",{"type":40,"tag":87,"props":3877,"children":3879},{"className":3878},[],[3880],{"type":46,"value":3881},"stage.drop",{"type":46,"value":3883}," \u002F log-line cleanup. Load ",{"type":40,"tag":211,"props":3885,"children":3886},{"href":745},[3887],{"type":46,"value":745},{"type":46,"value":3889}," when writing the report section: use its scenario-card shape, cite scalar metrics (optional short panel ID \u002F PromQL), and never paste the agent-only reference table or panel JSON into the customer report.",{"type":40,"tag":3891,"props":3892,"children":3893},"style",{},[3894],{"type":46,"value":3895},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3897,"total":4009},[3898,3913,3930,3946,3963,3976,3994],{"slug":3899,"name":3899,"fn":3900,"description":3901,"org":3902,"tags":3903,"stars":23,"repoUrl":24,"updatedAt":3912},"adaptive-metrics","optimize Grafana Cloud metrics costs","Cut Grafana Cloud Metrics cost by shrinking active-series count with Adaptive Metrics aggregation rules — auto-recommendations from query history, custom exact\u002Fregex rules, label-drop config, unused-metric detection, and Alloy remote_write fallback. Use when investigating a high Mimir\u002FGrafana Cloud bill, hunting high-cardinality labels (`pod_uid`, `service_instance_id`, `version`), pre-aggregating counters\u002Fgauges, dropping unused metrics, or measuring `grafanacloud_instance_active_series` before\u002Fafter — even when the user says \"reduce cardinality\", \"too many series\", \"metrics spend\", \"active series count is exploding\", or \"drop the version label\" without naming Adaptive Metrics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3904,3907,3908,3911],{"name":3905,"slug":3906,"type":15},"Cost Optimization","cost-optimization",{"name":9,"slug":8,"type":15},{"name":3909,"slug":3910,"type":15},"Metrics","metrics",{"name":13,"slug":14,"type":15},"2026-07-12T07:44:27.451068",{"slug":3914,"name":3914,"fn":3915,"description":3916,"org":3917,"tags":3918,"stars":23,"repoUrl":24,"updatedAt":3929},"admin","manage Grafana Cloud accounts and RBAC","Manage Grafana Cloud accounts — organizations, stacks, RBAC roles and assignments, SSO\u002FSAML\u002FOAuth\u002FGitHub auth, service accounts for CI\u002FCD, user invites, team membership, and API-driven provisioning. Creates stacks via the Cloud API, mints service-account tokens, applies role assignments, configures SSO providers, and provisions teams\u002Ffolders\u002Fdashboards via Terraform. Use when managing Grafana Cloud access, configuring SSO\u002FSAML\u002FOAuth, setting up service accounts for Terraform\u002FCI\u002FCD, assigning RBAC roles, inviting users, managing multiple stacks or organizations, provisioning cloud resources via API or Terraform, or auditing admin actions — even when the user says \"set up SSO\", \"create a stack\", \"make a service account\", or \"onboard a team\" without explicitly saying \"admin\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3919,3922,3925,3926],{"name":3920,"slug":3921,"type":15},"Access Control","access-control",{"name":3923,"slug":3924,"type":15},"Auth","auth",{"name":9,"slug":8,"type":15},{"name":3927,"slug":3928,"type":15},"Operations","operations","2026-07-12T07:44:12.078436",{"slug":3931,"name":3931,"fn":3932,"description":3933,"org":3934,"tags":3935,"stars":23,"repoUrl":24,"updatedAt":3945},"admission-control","implement admission control webhooks","Use when the user asks to \"write a validator\", \"add validation\", \"implement admission control\", \"write a mutating webhook\", \"add a mutation handler\", \"validate incoming resources\", \"implement admission logic\", \"add admission webhooks\", \"write ingress validation\", or asks how to validate or mutate resources before they are persisted in a grafana-app-sdk app. Provides guidance on implementing validation and mutation admission handlers for grafana-app-sdk apps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3936,3939,3942],{"name":3937,"slug":3938,"type":15},"Architecture","architecture",{"name":3940,"slug":3941,"type":15},"Security","security",{"name":3943,"slug":3944,"type":15},"Validation","validation","2026-07-12T07:45:06.148973",{"slug":3947,"name":3947,"fn":3948,"description":3949,"org":3950,"tags":3951,"stars":23,"repoUrl":24,"updatedAt":3962},"alerting-irm","configure Grafana Alerting and Incident Management","Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack\u002FPagerDuty\u002Femail\u002Fwebhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty\u002FSlack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says \"page me on errors\", \"alert me when X happens\", \"route this to the platform team\", or \"set up an SLO\" without naming Alerting or IRM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3952,3955,3956,3959],{"name":3953,"slug":3954,"type":15},"Alerting","alerting",{"name":9,"slug":8,"type":15},{"name":3957,"slug":3958,"type":15},"Incident Response","incident-response",{"name":3960,"slug":3961,"type":15},"Monitoring","monitoring","2026-07-12T07:44:02.393397",{"slug":1534,"name":1534,"fn":3964,"description":3965,"org":3966,"tags":3967,"stars":23,"repoUrl":24,"updatedAt":3975},"build unified telemetry pipelines with Grafana Alloy","Build a unified telemetry pipeline with Grafana Alloy — one OpenTelemetry-compatible binary that collects metrics, logs, traces, and profiles and ships to Grafana Cloud \u002F Prometheus \u002F Loki \u002F Tempo \u002F Pyroscope. Covers the Alloy config language (blocks, `sys.env`, component refs), `prometheus.scrape` → `remote_write`, `loki.source.file` + `loki.process` → `loki.write`, `otelcol.receiver.otlp` → `otelcol.exporter.otlp`, `pyroscope.scrape`, K8s \u002F Docker \u002F EC2 discovery, relabeling, modules (`import.file\u002Fgit\u002Fhttp`), clustering, Fleet Management `remotecfg`, the Alloy UI at `:12345`, and `alloy fmt` \u002F `alloy validate`. Use when writing a `config.alloy`, replacing Grafana Agent \u002F OTel Collector, scraping K8s pods, parsing logs, ingesting OTLP, or debugging \"Alloy isn't sending anything\" — even when the user says \"set up the agent\", \"write me a scrape config\", \"drop these logs before sending\", or \"OTel collector config\" without naming Alloy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3968,3969,3970,3971,3972],{"name":9,"slug":8,"type":15},{"name":20,"slug":21,"type":15},{"name":3909,"slug":3910,"type":15},{"name":13,"slug":14,"type":15},{"name":3973,"slug":3974,"type":15},"OpenTelemetry","opentelemetry","2026-07-12T07:43:54.817139",{"slug":3977,"name":3977,"fn":3978,"description":3979,"org":3980,"tags":3981,"stars":23,"repoUrl":24,"updatedAt":3993},"app-observability","monitor application performance in Grafana Cloud","Get RED metrics + service maps + frontend RUM + AI\u002FLLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50\u002Fp95\u002Fp99 latency, exemplar-to-trace, traces-to-logs \u002F profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token \u002F cost \u002F latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says \"set up APM\", \"show service map\", \"monitor browser perf\", \"session replay\", \"RUM SDK\", or \"watch our OpenAI bill\" without naming App \u002F Frontend \u002F AI Observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3982,3985,3988,3989,3992],{"name":3983,"slug":3984,"type":15},"APM","apm",{"name":3986,"slug":3987,"type":15},"Distributed Tracing","distributed-tracing",{"name":9,"slug":8,"type":15},{"name":3990,"slug":3991,"type":15},"LLM","llm",{"name":13,"slug":14,"type":15},"2026-07-12T07:44:34.500406",{"slug":3995,"name":3995,"fn":3996,"description":3997,"org":3998,"tags":3999,"stars":23,"repoUrl":24,"updatedAt":4008},"app-sdk-concepts","scaffold and configure Grafana apps","Use when starting any grafana-app-sdk work — scaffolding a Grafana app, initializing a Grafana App Platform app, picking a deployment mode (standalone operator \u002F grafana\u002Fapps \u002F frontend-only), wiring app-specific config, or onboarding to the SDK. Covers `grafana-app-sdk` CLI install, `project init` per deployment mode, project layout, the schema-centric workflow (CUE kinds → generated code → reconciler\u002Fadmission logic), and `SpecificConfig` for env-driven app configuration. Use even if the user just says \"make a new app\", \"Grafana app platform\", \"kinds and watchers\", \"operator scaffold\", or asks about `apps\u002F` inside the Grafana repo, without naming the SDK explicitly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4000,4001,4004,4007],{"name":3937,"slug":3938,"type":15},{"name":4002,"slug":4003,"type":15},"Deployment","deployment",{"name":4005,"slug":4006,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},"2026-07-12T07:45:08.595757",48,{"items":4011,"total":4191},[4012,4027,4046,4066,4081,4095,4108,4123,4140,4153,4166,4179],{"slug":4013,"name":4013,"fn":4014,"description":4015,"org":4016,"tags":4017,"stars":4024,"repoUrl":4025,"updatedAt":4026},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4018,4019,4022,4023],{"name":3986,"slug":3987,"type":15},{"name":4020,"slug":4021,"type":15},"Frontend","frontend",{"name":3960,"slug":3961,"type":15},{"name":13,"slug":14,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":4028,"name":4028,"fn":4029,"description":4030,"org":4031,"tags":4032,"stars":4043,"repoUrl":4044,"updatedAt":4045},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4033,4036,4039,4042],{"name":4034,"slug":4035,"type":15},"API Development","api-development",{"name":4037,"slug":4038,"type":15},"Authentication","authentication",{"name":4040,"slug":4041,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":4047,"name":4047,"fn":4048,"description":4049,"org":4050,"tags":4051,"stars":4043,"repoUrl":4044,"updatedAt":4065},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4052,4055,4058,4059,4062],{"name":4053,"slug":4054,"type":15},"CSV","csv",{"name":4056,"slug":4057,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":4060,"slug":4061,"type":15},"GraphQL","graphql",{"name":4063,"slug":4064,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":4067,"name":4067,"fn":4068,"description":4069,"org":4070,"tags":4071,"stars":4078,"repoUrl":4079,"updatedAt":4080},"agento11y","manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4072,4075,4076,4077],{"name":4073,"slug":4074,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":3960,"slug":3961,"type":15},{"name":13,"slug":14,"type":15},430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-25T05:30:40.29622",{"slug":4082,"name":4082,"fn":4083,"description":4084,"org":4085,"tags":4086,"stars":4078,"repoUrl":4079,"updatedAt":4094},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4087,4088,4089,4092,4093],{"name":4073,"slug":4074,"type":15},{"name":9,"slug":8,"type":15},{"name":4090,"slug":4091,"type":15},"Instrumentation","instrumentation",{"name":3990,"slug":3991,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:52.580237",{"slug":4096,"name":4096,"fn":4097,"description":4098,"org":4099,"tags":4100,"stars":4078,"repoUrl":4079,"updatedAt":4107},"agento11y-prod-setup","setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4101,4102,4105,4106],{"name":4073,"slug":4074,"type":15},{"name":4103,"slug":4104,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:53.576347",{"slug":4109,"name":4109,"fn":4110,"description":4111,"org":4112,"tags":4113,"stars":4078,"repoUrl":4079,"updatedAt":4122},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4114,4115,4116,4119],{"name":4073,"slug":4074,"type":15},{"name":9,"slug":8,"type":15},{"name":4117,"slug":4118,"type":15},"QA","qa",{"name":4120,"slug":4121,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":4124,"name":4124,"fn":4125,"description":4126,"org":4127,"tags":4128,"stars":4078,"repoUrl":4079,"updatedAt":4139},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4129,4132,4135,4138],{"name":4130,"slug":4131,"type":15},"Dashboards","dashboards",{"name":4133,"slug":4134,"type":15},"Data Visualization","data-visualization",{"name":4136,"slug":4137,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":4141,"name":4141,"fn":4142,"description":4143,"org":4144,"tags":4145,"stars":4078,"repoUrl":4079,"updatedAt":4152},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4146,4149,4150,4151],{"name":4147,"slug":4148,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":3957,"slug":3958,"type":15},{"name":13,"slug":14,"type":15},"2026-07-18T05:11:10.445428",{"slug":4154,"name":4154,"fn":4155,"description":4156,"org":4157,"tags":4158,"stars":4078,"repoUrl":4079,"updatedAt":4165},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4159,4160,4161,4164],{"name":4147,"slug":4148,"type":15},{"name":9,"slug":8,"type":15},{"name":4162,"slug":4163,"type":15},"Graph Analysis","graph-analysis",{"name":13,"slug":14,"type":15},"2026-07-25T05:30:39.380934",{"slug":4167,"name":4167,"fn":4168,"description":4169,"org":4170,"tags":4171,"stars":4078,"repoUrl":4079,"updatedAt":4178},"gcx","manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4172,4175,4176,4177],{"name":4173,"slug":4174,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":3960,"slug":3961,"type":15},{"name":3927,"slug":3928,"type":15},"2026-07-31T05:53:50.587304",{"slug":4180,"name":4180,"fn":4181,"description":4182,"org":4183,"tags":4184,"stars":4078,"repoUrl":4079,"updatedAt":4190},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4185,4186,4187],{"name":4173,"slug":4174,"type":15},{"name":9,"slug":8,"type":15},{"name":4188,"slug":4189,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80]