[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-elastic-observability-k8s-investigation":3,"mdc--krg93m-key":34,"related-repo-elastic-observability-k8s-investigation":3598,"related-org-elastic-observability-k8s-investigation":3693},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"observability-k8s-investigation","investigate Kubernetes issues with OTel telemetry","Investigate Kubernetes workload, node, and control-plane issues using OTel telemetry (EDOT). Use when diagnosing pod failures (CrashLoopBackOff, OOMKilled, Error), node pressure, resource exhaustion, image pull failures, admission rejections, autoscaling anomalies, or correlating K8s state with application signals. OTel ingest path only — the legacy ECS Kubernetes integration shape is out of scope.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"elastic","Elastic","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Felastic.png",[12,16,17,20],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Kubernetes","kubernetes",{"name":21,"slug":22,"type":15},"Debugging","debugging",531,"https:\u002F\u002Fgithub.com\u002Felastic\u002Fagent-skills","2026-07-12T07:47:31.290712",null,41,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"Official Elastic Skills","https:\u002F\u002Fgithub.com\u002Felastic\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fobservability\u002Fskills\u002Fk8s-investigation","---\nname: observability-k8s-investigation\ndescription: >\n  Investigate Kubernetes workload, node, and control-plane issues using OTel telemetry\n  (EDOT). Use when diagnosing pod failures (CrashLoopBackOff, OOMKilled, Error), node\n  pressure, resource exhaustion, image pull failures, admission rejections, autoscaling\n  anomalies, or correlating K8s state with application signals. OTel ingest path only\n  — the legacy ECS Kubernetes integration shape is out of scope.\nmetadata:\n  author: elastic\n  version: 0.2.0\n---\n\n# Kubernetes Investigation\n\nDiagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the\nkube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify\nroot cause across the workload, node, and control-plane layers.\n\n## Scope\n\n**In scope:** OTel-receiver-namespaced indices (`metrics-kubeletstatsreceiver.otel-*`,\n`metrics-k8sclusterreceiver.otel-*`, `logs-k8seventsreceiver.otel-*`, `logs-k8sobjectsreceiver.otel-*`) and OTel\nsemantic conventions (`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.restarts`).\n\n**Out of scope:**\n\n- The legacy Elastic Agent Kubernetes integration (`metrics-kubernetes.*`, `logs-kubernetes.*`, `kubernetes.*` fields).\n  Being deprecated — do not author queries against these paths.\n- APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain —\n  once a K8s root cause is ruled in or out, APM investigation continues outside this skill.\n- Cluster provisioning, capacity planning, cost optimization. Different domain.\n\n## Guidelines\n\nThese apply to every investigation. When in doubt, re-read them before writing the synthesis.\n\n**Absence of evidence is not evidence. Do not confabulate from empty results.** If log queries return 0 rows, logs are\nlikely not collected or the pod has no recent lines — this does _not_ mean \"dependency unavailable\" or any other\nspecific failure mode. Report `no_logs_available` and weight remaining signals accordingly.\n\n**Empty dependency data ≠ upstream healthy.** Services without APM instrumentation (load generators, workers) emit no\ndestination metrics. Report `insufficient_dependency_data`, not \"upstreams OK.\"\n\n**Co-symptoms are not causes.** Two services degrading simultaneously usually share an upstream, not a causal link. Only\nattribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5×\nerror rate, >3× latency).\n\n**OOMKilled ≠ memory leak by default.** The limit may simply be undersized for the workload's working set. Compare\nagainst a 7-day baseline at the same hour-of-day before claiming a leak.\n\n**Error-termination ≠ application bug by default.** Check `k8s.pod.cpu_limit_utilization` first. CFS throttling driving\nliveness probe timeouts is the most common misdiagnosis in this space.\n\n**Average CPU hides throttling.** A pod can look healthy at 40–60% average `cpu_limit_utilization` while being throttled\nseverely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads hit quota mid-period and stall. Look at\nmax and p95, not just average.\n\n**Restart count is boolean, not a counter.** `k8s.container.restarts` is pulled directly from the K8s API and may be\npruned by the kubelet at any time, so the absolute value is unreliable. Treat it as `== 0` (no recent restarts) vs `> 0`\n(recently restarting); do not derive backoff timing or \"linear vs exponential\" patterns from it. Confirm the restart\npattern via K8s `Killing` \u002F `BackOff` events instead.\n\n**Prefer to report uncertainty over manufacturing confidence.** If the evidence is ambiguous, the synthesis should say\nso. Competing hypotheses are a valid output.\n\n## Indices and fields\n\n### Where to look\n\n| Signal                | Index pattern                                       | Use                                                                 |\n| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- |\n| Pod\u002Fcontainer runtime | `metrics-kubeletstatsreceiver.otel-*`               | CPU, memory, network, filesystem. Utilization ratios.               |\n| Cluster state         | `metrics-k8sclusterreceiver.otel-*`                 | Restarts, phase, last-terminated reason, HPA, quota, node condition |\n| K8s events            | `logs-k8seventsreceiver.otel-*`                     | Killing, BackOff, FailedScheduling, Evicted, image pull events      |\n| K8s object snapshots  | `logs-k8sobjectsreceiver.otel-*`                    | Deployment\u002Fservice\u002Fconfigmap state over time                        |\n| Application logs      | `logs-*.otel-*`                                     | `body.text`, `severity_text`, filtered by `k8s.pod.name`            |\n| APM                   | `traces-*.otel-*`, `metrics-service_*.otel-default` | Correlate via `service.name` + K8s resource attrs                   |\n| ML anomalies          | `.ml-anomalies-*`                                   | Memory-growth, restart-rate, throttle jobs (if configured)          |\n\n### Key fields\n\nFlat OTel paths work in ES|QL. Prefer the flat form for readability; the nested `resource.attributes.*` form is for raw\nlog documents only.\n\n| Field                                            | Index                       | What it is                                              |\n| ------------------------------------------------ | --------------------------- | ------------------------------------------------------- |\n| `k8s.pod.name`                                   | all k8s                     | Pod name                                                |\n| `k8s.namespace.name`                             | all k8s                     | Namespace                                               |\n| `k8s.container.name`                             | all k8s                     | Container within pod                                    |\n| `k8s.deployment.name`                            | k8sclusterreceiver + others | Parent deployment                                       |\n| `k8s.pod.phase`                                  | k8sclusterreceiver          | Pending=1\u002FRunning=2\u002FSucceeded=3\u002FFailed=4\u002FUnknown=5      |\n| `k8s.container.restarts`                         | k8sclusterreceiver          | Total container restart count                           |\n| `k8s.container.status.last_terminated_reason`    | k8sclusterreceiver          | `OOMKilled`, `Error`, `Completed`, `ContainerCannotRun` |\n| `k8s.pod.status_reason`                          | k8sclusterreceiver          | Pod-level reason (`Evicted`, `NodeLost`)                |\n| `k8s.pod.memory_limit_utilization`               | kubeletstatsreceiver        | 0.0–1.0+ (can exceed 1 transiently before OOM)          |\n| `k8s.pod.cpu_limit_utilization`                  | kubeletstatsreceiver        | 0.0–N (frequently >1 under CFS throttling)              |\n| `k8s.pod.memory.usage` \u002F `.working_set`          | kubeletstatsreceiver        | Bytes                                                   |\n| `k8s.node.condition_memory_pressure`             | k8sclusterreceiver          | 1 = pressure, 0 = ok                                    |\n| `k8s.node.condition_ready`                       | k8sclusterreceiver          | 0 = NotReady                                            |\n| `k8s.hpa.current_replicas` \u002F `.desired_replicas` | k8sclusterreceiver          | HPA state                                               |\n| `attributes.k8s.event.reason`                    | k8seventsreceiver           | Event reason (filter on this)                           |\n| `body.text`                                      | k8seventsreceiver \u002F logs    | Event message \u002F log message                             |\n| `k8s.object.name`                                | k8seventsreceiver           | involvedObject name (log attribute, use flat form)      |\n\n### Field availability\n\nSeveral fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify\npresence before relying on them; if absent, fall back as noted and call out the substitution in the synthesis.\n\n| Field                                                        | Why it might be missing                                                                                    | Fall-back                                                                                                                     |\n| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |\n| `k8s.container.status.last_terminated_reason`                | Optional metric in k8sclusterreceiver; gated behind `metrics_collected.metadata` config.                   | Infer from K8s `Killing` \u002F `OOMKilling` events in `logs-k8seventsreceiver.otel-*` and exit codes in app logs.                 |\n| `k8s.pod.status_reason`                                      | Same — optional metric on k8sclusterreceiver.                                                              | Infer from events: `Evicted`, `NodeLost`, `Preempted`.                                                                        |\n| `k8s.pod.cpu_limit_utilization` \u002F `memory_limit_utilization` | Only emitted when the pod has the corresponding limit set, and the kubeletstatsreceiver metric is enabled. | Compute manually as `k8s.pod.cpu.usage \u002F \u003Climit>` from k8sclusterreceiver, or use absolute usage trending against a baseline. |\n| `k8s.node.condition_memory_pressure`                         | Gated behind k8sclusterreceiver `node_conditions_to_report` (default omits this).                          | Compare `k8s.node.memory.usage` against `k8s.node.allocatable_memory`, or look for `Evicted` events on the node.              |\n\nIf a fall-back is used, note it in the synthesis (e.g. `(via memory.usage; limit_utilization not collected)`) so the\nreader knows the signal is indirect.\n\n## ES|QL gotchas\n\nBefore writing queries, know these. Each of them silently produces wrong answers rather than failing loudly.\n\n**`VALUES()` returns scalar for single distinct value, array for multiple.** Templating that assumes array shape (e.g.\n`| first`) extracts the first character of the string when scalar. Use `MV_FIRST(VALUES(...))` or handle both.\n\n**`PERCENTILE` does not work on OTel `histogram` type** (as of 8.15). For APM duration percentiles, use `AVG` on the\n`aggregate_metric_double` summary field (`AVG(transaction.duration.summary)` divides sum by value_count). For true\npercentiles, fall back to Kibana Query DSL.\n\n**`COUNT(agg_metric_double)` returns `value_count` (events), not doc count.** `SUM(field)` gives the sum component;\n`AVG(field)` gives sum\u002Fvalue_count. Do not use `SUM(transaction.duration.summary)` as an event-count proxy — it returns\ntotal duration.\n\n**K8s metrics use flat OTel field paths in ES|QL.** `k8s.pod.name`, not `resource.attributes.k8s.pod.name`. The nested\nform is for raw log documents.\n\n## Failure-mode taxonomy\n\nVocabulary for classification, not a decision tree. Use the pivotal-signal column to recognize which mode you're looking\nat; use \"Investigate\" to know what else should corroborate.\n\n### Workload layer\n\n| Mode                                | Pivotal signal                                                                   | Investigate                                                                                                                                                                                                                                               |\n| ----------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **OOMKilled**                       | `last_terminated_reason == \"OOMKilled\"` + `memory_limit_utilization → 1.0`       | Monotonic rise (leak) vs. load-driven spike? Compare current trend to 7-day baseline. Check heap metrics (JVM, Go, Node) for GC pressure.                                                                                                                 |\n| **CPU throttling → Error exit**     | `cpu_limit_utilization > 1.0` + `last_terminated_reason == \"Error\"`              | Liveness\u002Freadiness probe timeouts from CFS throttling. Average CPU can look fine (40–60%) while p99 throttle is severe. Check probe timeouts vs observed startup\u002Fhealth latency.                                                                          |\n| **Liveness probe misconfiguration** | Restarts without resource pressure; `initialDelaySeconds` \u003C startup time         | K8s events show `Unhealthy` \u002F `Killing`. `kubectl logs --previous` typically shows healthy startup before kill.                                                                                                                                           |\n| **CrashLoopBackOff (generic)**      | `BackOff` events + rising `k8s.container.restarts`                               | Branch on `last_terminated_reason` — this is a meta-mode. OOMKilled → memory path; Error → logs + throttling; ContainerCannotRun → image\u002Fexec.                                                                                                            |\n| **ImagePullBackOff**                | K8s events `Failed` with image name + `429` or `not found`                       | Registry rate limit? Missing tag? Wrong imagePullSecret? Check recency of `Pulling`\u002F`Pulled` events.                                                                                                                                                      |\n| **Stuck rollout**                   | New pods `Pending`\u002Fnot-Ready > `progressDeadlineSeconds`; old pods still serving | Check `k8s.deployment.available` vs `.desired`. Admission rejection? Readiness probe failing on new pods? HPA not scaling?                                                                                                                                |\n| **Termination signal race**         | Brief 5xx bursts correlated with rolling deploys                                 | Endpoint removal races termination. New requests can hit the pod after SIGTERM starts. NGINX gotcha: `STOPSIGNAL SIGTERM` triggers _fast_ shutdown, not graceful — use `STOPSIGNAL SIGQUIT` for graceful drain. Check ingress 502 rate vs rollout timing. |\n\n### Node layer\n\n| Mode                                | Pivotal signal                                                          | Investigate                                                                                                                        |\n| ----------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |\n| **Node NotReady cascade**           | `k8s.node.condition_ready == 0` + mass `Evicted` events                 | Memory pressure? Disk pressure? Network partition from API server? Inspect kubelet logs, `k8s.node.condition_*` history.           |\n| **Resource eviction**               | `status_reason == \"Evicted\"` + `condition_memory_pressure == 1` on node | Node-level noisy neighbor. QoS order: BestEffort → Burstable → Guaranteed. Identify which pod drove node memory up.                |\n| **Node affinity\u002Fselector conflict** | Mass unschedulable pods after label change                              | K8s events show `FailedScheduling`. Often triggered by cluster upgrades (e.g. `node-role.kubernetes.io\u002Fmaster` → `control-plane`). |\n\n### Control plane\n\n| Mode                          | Pivotal signal                                                     | Investigate                                                                                                                             |\n| ----------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |\n| **etcd I\u002FO cascade**          | API server latency spike + cluster-wide kubelet heartbeat failures | Disk IOPS, fsync latency (must be \u003C10ms). Cloud-burst-credit exhaustion is common.                                                      |\n| **Admission webhook block**   | Mass `FailedCreate` across namespaces; deployments frozen          | `failurePolicy:Fail` webhook pod crashed. Check webhook pod health + API server TCP connection cache (caches dead connections ~15 min). |\n| **Priority preemption storm** | Production pods terminating with `preempted-by` annotation         | New `PriorityClass` with `globalDefault:true` caused cascade. Check `kube-scheduler` events.                                            |\n| **PDB drain deadlock**        | Node drain stuck indefinitely; HTTP 429 from Eviction API          | PDB `minAvailable`\u002F`maxUnavailable` too strict. No default drain timeout. Manual PDB deletion unblocks.                                 |\n\n### Autoscaling & admission\n\n| Mode                          | Pivotal signal                                                     | Investigate                                                                                                                                        |\n| ----------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **HPA unready-pod dampening** | Load rising, HPA not scaling; unready pods included in calculation | HPA averages CPU across all replicas including unready (0% contribution). Check `k8s.hpa.current_replicas` vs `.desired_replicas` + pod readiness. |\n| **Resource quota silent 403** | Deployment stuck at n-1\u002Fn; `FailedCreate` on ReplicaSet            | Namespace quota exhausted (often CronJob accumulation). Check `k8s.resource_quota.used` vs `.hard_limit`.                                          |\n\n### Networking\n\n| Mode                        | Pivotal signal                                           | Investigate                                                                                                           |\n| --------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |\n| **StatefulSet split-brain** | Duplicate pod identities across partitioned nodes        | Network partition + eviction timeout race. Two instances of same ordinal running. No fencing by default.              |\n| **CoreDNS OOMKill**         | CoreDNS restarts + cluster-wide DNS timeouts in app logs | Default CoreDNS memory (~170Mi) insufficient under query amplification (ndots:5, each external lookup → ~10 lookups). |\n\n### When classification is ambiguous\n\nReal incidents often match two modes. Examples:\n\n- OOMKilled pod with simultaneous CPU throttling — memory usually drives the kill, but verify by checking whether memory\n  or CPU hit limit first.\n- Stuck rollout with HPA dampening and resource quota near-exhaustion — both can freeze a deploy. Check which constraint\n  is binding.\n- Node NotReady with pods that were already crashing — the node issue may be incidental.\n\nWhen two modes fit, name both in the synthesis and say which one you believe is causal and why. Do not force a single\nhypothesis when the evidence supports two.\n\n## Signal interpretation\n\n### Memory\n\n- **Monotonic rise over 30–60 min** → leak. Check GC metrics for the language: JVM `jvm.gc.duration`, Go\n  `process.runtime.go.gc.pause_ns`, Node `v8js_gc_duration`. Rising GC frequency\u002Fpause with stable live-set is the\n  canonical leak signature.\n- **Diurnal \u002F load-correlated spikes** → load-driven, not leak. Consider HPA tuning or limit increase.\n- **Hits 1.0, then restart** → OOMKilled confirmed. Exit code 137 (SIGKILL) in app logs consistent.\n\n### CPU\n\n- `cpu_limit_utilization > 1.0` sustained → CFS throttling. Node has spare CPU; the pod is quota-blocked.\n- Symptoms of throttling (not the throttle metric itself): liveness probe timeouts, p99 latency 4–16× p50, queue\n  backpressure upstream, Error-reason container terminations.\n- Average can look healthy while p95 is throttled. Do not trust average alone.\n\n### Restart patterns\n\n- `restarts > 0` recently → workload has been restarting. Don't read magnitude into the count (see _Restart count is\n  boolean_); confirm the pattern from K8s `Killing` \u002F `BackOff` event timestamps in `logs-k8seventsreceiver.otel-*`.\n- Restarts correlated with memory pressure (`memory_limit_utilization → 1.0`) → OOMKilled path.\n- Restarts without memory\u002FCPU pressure → probe misconfig, app bug, or startup dependency failure. Pull events for\n  `Unhealthy` and `Killing`.\n\n### Termination reasons\n\n- `OOMKilled` → memory path.\n- `Error` → non-zero exit. Check app logs; if empty\u002Fminimal, check CPU throttling before attributing to app logic.\n- `Completed` → ran to completion. Normal for Jobs\u002FCronJobs\u002Finit containers; anomalous otherwise.\n- `ContainerCannotRun` → runtime\u002Fimage\u002Fexec issue. Check image pull events.\n\n## Investigation flow\n\n> An investigation is not a checklist. The sections below describe a _typical_ arc — **compress, skip, or revisit them\n> based on what you find.** Terminate as soon as you have enough evidence to synthesize at a known confidence. Chasing\n> signals past the point of diminishing returns is a failure mode, not thoroughness.\n\n### Orient\n\nResolve the target: `k8s.pod.name`, `k8s.namespace.name`, optionally `k8s.deployment.name` and `service.name`. If no\ntime window is given, default to the last hour for pod-level investigations, last 2 hours for event correlation, last 6\nhours for ongoing\u002Funresolved incidents.\n\nIf the alert payload already tells you the failure mode (e.g., it fires specifically on `OOMKilled`), note that and skip\nclassification; move to confirmation and baseline comparison.\n\n### Characterize\n\nGet the shape of the workload's recent behavior: restart count, termination reasons, phase, utilization. One or two\nqueries usually suffice.\n\n```esql\nFROM metrics-k8sclusterreceiver.otel-*\n| WHERE k8s.pod.name == \"\u003Cpod>\" AND k8s.namespace.name == \"\u003Cns>\"\n  AND @timestamp > NOW() - 1 hour\n| STATS restarts = MAX(k8s.container.restarts),\n        term_reasons = VALUES(k8s.container.status.last_terminated_reason),\n        phase = MAX(k8s.pod.phase)\n```\n\n```esql\nFROM metrics-kubeletstatsreceiver.otel-*\n| WHERE k8s.pod.name == \"\u003Cpod>\" AND @timestamp > NOW() - 15 minutes\n| STATS mem_pct = ROUND(MAX(k8s.pod.memory_limit_utilization) * 100, 1),\n        cpu_pct = ROUND(MAX(k8s.pod.cpu_limit_utilization) * 100, 1)\n```\n\n### Classify\n\nUse the taxonomy. The pivotal signal should match; the \"Investigate\" column tells you what corroboration to seek.\n\nWhen two modes fit, note both and proceed with the one that has the stronger pivotal signal. You may revise during\ncorroboration.\n\n### Corroborate\n\nPull the evidence your classification predicts you'll find. Typical sources:\n\n**K8s events** for the namespace and window:\n\n```esql\nFROM logs-k8seventsreceiver.otel-*\n| WHERE k8s.namespace.name == \"\u003Cns>\"\n  AND @timestamp > NOW() - 2 hours\n  AND attributes.k8s.event.reason IN (\n    \"BackOff\", \"Killing\", \"Unhealthy\", \"Failed\",\n    \"FailedScheduling\", \"Evicted\", \"SuccessfulRescale\",\n    \"Pulling\", \"Pulled\", \"Started\", \"Created\"\n  )\n| SORT @timestamp DESC\n| KEEP @timestamp, attributes.k8s.event.reason, body.text, k8s.object.name\n| LIMIT 30\n```\n\n**Application logs** if available — look at the 200 most recent lines before the termination timestamp. If absent, flag\n`no_logs_available`; do not invent a log pattern.\n\n**APM** if the pod runs an instrumented service — resolve `service.name` from pod resource attributes for later\ncorrelation. SLO \u002F latency \u002F error-rate analysis itself is APM-layer work and out of scope for this skill.\n\n**Baseline comparison** — for utilization-based findings, compare current values to 7-day-prior at the same hour-of-day.\n\"High memory\" is meaningful only relative to what's normal for this workload.\n\n### Check for upstream cause (conditional)\n\nOnly pursue if the symptom pattern suggests it. Threshold: upstream error rate >5× baseline _or_ latency >3× baseline,\nAND degradation started before the symptom on the target service. Co-symptoms do not establish causation.\n\nIf `metrics-service_destination.1m.otel-default` has no rows for the service, report `insufficient_dependency_data` —\nnot \"upstreams healthy.\"\n\n### Check for recent change (conditional)\n\n`SuccessfulCreate` \u002F `Pulled` events in the last 2 hours often correlate with deploys. `logs-k8sobjectsreceiver.otel-*`\nshows configmap\u002Fsecret\u002Fdeployment spec changes. A change within 15 minutes of the symptom onset is a strong correlation,\nbut still a correlation — verify it plausibly explains the mode you've classified.\n\n### Synthesize and stop\n\nSynthesize as soon as you have enough evidence to support a hypothesis at known confidence. You do not need to complete\nevery section above — investigation terminates when either:\n\n- You have a high-confidence hypothesis with corroboration, or\n- You have a low\u002Fmedium-confidence hypothesis and further queries are unlikely to change the picture (e.g., logs are\n  unavailable, APM isn't instrumented, no recent changes found).\n\n## Synthesis\n\nDefault structure:\n\n```text\nHYPOTHESIS (confidence: high | medium | low)\n\u003COne paragraph: service, symptom, most likely cause. Name the failure mode from the taxonomy.>\n\nEVIDENCE\n- \u003CFinding from characterization, with the concrete metric or value.>\n- \u003CFinding from events \u002F logs \u002F APM.>\n- \u003CFinding from baseline comparison, dependency check, or change correlation if pursued.>\n\nCONFIDENCE NOTE\n\u003COnly if not 'high'. What specific evidence is missing or ambiguous.>\n\nRECOMMENDED NEXT STEPS\n1. \u003CMost actionable — typically a config check or metric to observe.>\n2. \u003CSecondary.>\n\nDOWNSTREAM IMPACT\n\u003CServices depending on this workload, or 'No downstream dependencies identified.'>\n```\n\n**When two hypotheses are live:** replace HYPOTHESIS with COMPETING HYPOTHESES; list both, say which you lean toward and\nwhy, and list the evidence that would disambiguate them.\n\n**When no incident is found** (symptom resolved, or alert appears spurious): say so directly.\n`ALERT FIRED BUT SYSTEM APPEARS HEALTHY` is a valid output. List what you checked and what you didn't find.\n\n### Confidence calibration\n\nStart at **high** and downgrade based on what's missing:\n\n- Downgrade to **medium** if: primary signal is clear but corroboration is missing (no logs, no APM, no baseline\n  comparison possible). Or: two modes fit and you can't disambiguate.\n- Downgrade to **low** if: only a single signal supports the hypothesis, signals conflict, or the mode requires evidence\n  you couldn't fetch.\n\nNever return **high** when application log data was absent and the hypothesis depends on application behavior. Absence\nof evidence does not corroborate a hypothesis.\n\n## Query recipes\n\n### Most-restarting pods in a namespace\n\n```esql\nFROM metrics-k8sclusterreceiver.otel-*\n| WHERE k8s.namespace.name == \"\u003Cns>\" AND @timestamp > NOW() - 1 hour\n| STATS restarts = MAX(k8s.container.restarts) BY k8s.pod.name, k8s.container.status.last_terminated_reason\n| WHERE restarts > 0\n| SORT restarts DESC\n| LIMIT 20\n```\n\n### CPU throttling check for a pod\n\n```esql\nFROM metrics-kubeletstatsreceiver.otel-*\n| WHERE k8s.pod.name == \"\u003Cpod>\" AND @timestamp > NOW() - 30 minutes\n| STATS max_cpu_ratio = ROUND(MAX(k8s.pod.cpu_limit_utilization), 2),\n        avg_cpu_ratio = ROUND(AVG(k8s.pod.cpu_limit_utilization), 2),\n        max_cpu_cores = ROUND(MAX(k8s.pod.cpu.usage), 3)\n```\n\nSustained ratio >1.0 = throttling. Transient >1.0 with avg \u003C0.5 is usually benign burst.\n\n### Nodes under memory pressure (right now)\n\n```esql\nFROM metrics-k8sclusterreceiver.otel-*\n| WHERE @timestamp > NOW() - 15 minutes AND k8s.node.condition_memory_pressure == 1\n| STATS ts = MAX(@timestamp) BY k8s.node.name\n| SORT ts DESC\n```\n\n### Admission denials (webhook or quota) last hour\n\n```esql\nFROM logs-k8seventsreceiver.otel-*\n| WHERE @timestamp > NOW() - 1 hour\n  AND (attributes.k8s.event.reason == \"FailedCreate\"\n       OR body.text LIKE \"*admission webhook*\"\n       OR body.text LIKE \"*exceeded quota*\")\n| SORT @timestamp DESC\n| KEEP @timestamp, k8s.namespace.name, attributes.k8s.event.reason, body.text\n| LIMIT 30\n```\n\n### Firing K8s alerts\n\n```text\nGET \u002Fapi\u002Falerting\u002Frules\u002F_find?search=k8s&search_fields=tags&filter=alert.attributes.executionStatus.status:active\n```\n\n## Examples\n\n### \"Why is my pod CrashLoopBackOff-ing?\"\n\nCharacterize first: get restart count, termination reason, memory and CPU utilization.\n\n- If `last_terminated_reason == \"OOMKilled\"` and memory utilization hit 1.0 → memory path. Corroborate with 7-day\n  baseline: monotonic rise over days = leak; spiky = load-driven. Check GC metrics if language is known.\n- If `last_terminated_reason == \"Error\"` and `cpu_limit_utilization > 1.0` → CPU throttling path. Corroborate with\n  liveness probe config (initialDelaySeconds, timeoutSeconds) and K8s events for `Unhealthy`.\n- If `last_terminated_reason == \"Error\"` and CPU is fine → application-logic path. Pull recent logs before termination.\n- If `last_terminated_reason == \"ContainerCannotRun\"` → image\u002Fexec path. Check K8s events for `Failed` pull events.\n\nSynthesize with appropriate confidence. If logs were unavailable on the Error path, downgrade to medium and say so.\n\n### \"Is my rollout stuck?\"\n\nAuthoritative signal: `k8s.deployment.available \u003C k8s.deployment.desired` for > 10 minutes.\n\nDiagnose the constraint:\n\n- K8s events on the new ReplicaSet: `FailedCreate` → admission rejection (quota, webhook, PSP). `FailedScheduling` → no\n  node fits.\n- New-pod utilization: all at 0% memory → never started (image pull failure); high CPU with low memory → slow startup\n  hitting readiness probe.\n- HPA state: stable `current_replicas \u003C desired_replicas` under load → unready-pod dampening.\n\n### \"Alert fired but everything looks healthy\"\n\nPossible and worth naming explicitly. Check:\n\n- Has the symptom resolved? Compare current utilization\u002Frestart rate to the alert trigger point.\n- Was the alert a transient spike that's already decayed?\n- Is the alert tuned appropriately (e.g., too-short evaluation window)?\n\nOutput: `ALERT FIRED BUT SYSTEM APPEARS HEALTHY` with what you checked. Recommend alert tuning if the pattern is\nrecurrent.\n\n## Related\n\n- **Workflow:** `K8s CrashLoopBackOff Investigation` — alert-triggered automated version of the pod-level path above.\n  Runs deterministic ESQL + branches; this skill provides the interpretation layer the workflow lacks.\n- **Forge genome library:** 16 K8s failure scenarios (OOMKill cascade, CPU throttling, probe misconfig, node NotReady,\n  admission webhook block, etc.) validating this skill's coverage.\n",{"data":35,"body":38},{"name":4,"description":6,"metadata":36},{"author":8,"version":37},"0.2.0",{"type":39,"children":40},"root",[41,50,56,63,128,136,178,184,189,215,233,243,253,271,289,338,348,354,361,577,583,596,1035,1041,1046,1251,1264,1270,1275,1307,1355,1402,1426,1432,1437,1443,1780,1786,1929,1934,2102,2108,2208,2214,2280,2286,2291,2309,2314,2320,2326,2383,2389,2412,2418,2488,2494,2537,2543,2566,2572,2603,2615,2621,2626,2693,2732,2738,2743,2748,2754,2759,2768,2868,2884,2900,2910,2916,2928,2948,2954,2978,2984,2989,3002,3008,3013,3023,3033,3051,3057,3069,3095,3106,3112,3118,3172,3178,3224,3229,3235,3273,3279,3347,3353,3362,3368,3374,3379,3446,3451,3457,3470,3475,3515,3521,3526,3544,3556,3562,3592],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"kubernetes-investigation",[47],{"type":48,"value":49},"text","Kubernetes Investigation",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54],{"type":48,"value":55},"Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the\nkube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify\nroot cause across the workload, node, and control-plane layers.",{"type":42,"tag":57,"props":58,"children":60},"h2",{"id":59},"scope",[61],{"type":48,"value":62},"Scope",{"type":42,"tag":51,"props":64,"children":65},{},[66,72,74,81,83,89,91,97,98,104,106,112,113,119,120,126],{"type":42,"tag":67,"props":68,"children":69},"strong",{},[70],{"type":48,"value":71},"In scope:",{"type":48,"value":73}," OTel-receiver-namespaced indices (",{"type":42,"tag":75,"props":76,"children":78},"code",{"className":77},[],[79],{"type":48,"value":80},"metrics-kubeletstatsreceiver.otel-*",{"type":48,"value":82},",\n",{"type":42,"tag":75,"props":84,"children":86},{"className":85},[],[87],{"type":48,"value":88},"metrics-k8sclusterreceiver.otel-*",{"type":48,"value":90},", ",{"type":42,"tag":75,"props":92,"children":94},{"className":93},[],[95],{"type":48,"value":96},"logs-k8seventsreceiver.otel-*",{"type":48,"value":90},{"type":42,"tag":75,"props":99,"children":101},{"className":100},[],[102],{"type":48,"value":103},"logs-k8sobjectsreceiver.otel-*",{"type":48,"value":105},") and OTel\nsemantic conventions (",{"type":42,"tag":75,"props":107,"children":109},{"className":108},[],[110],{"type":48,"value":111},"k8s.pod.name",{"type":48,"value":90},{"type":42,"tag":75,"props":114,"children":116},{"className":115},[],[117],{"type":48,"value":118},"k8s.namespace.name",{"type":48,"value":90},{"type":42,"tag":75,"props":121,"children":123},{"className":122},[],[124],{"type":48,"value":125},"k8s.container.restarts",{"type":48,"value":127},").",{"type":42,"tag":51,"props":129,"children":130},{},[131],{"type":42,"tag":67,"props":132,"children":133},{},[134],{"type":48,"value":135},"Out of scope:",{"type":42,"tag":137,"props":138,"children":139},"ul",{},[140,168,173],{"type":42,"tag":141,"props":142,"children":143},"li",{},[144,146,152,153,159,160,166],{"type":48,"value":145},"The legacy Elastic Agent Kubernetes integration (",{"type":42,"tag":75,"props":147,"children":149},{"className":148},[],[150],{"type":48,"value":151},"metrics-kubernetes.*",{"type":48,"value":90},{"type":42,"tag":75,"props":154,"children":156},{"className":155},[],[157],{"type":48,"value":158},"logs-kubernetes.*",{"type":48,"value":90},{"type":42,"tag":75,"props":161,"children":163},{"className":162},[],[164],{"type":48,"value":165},"kubernetes.*",{"type":48,"value":167}," fields).\nBeing deprecated — do not author queries against these paths.",{"type":42,"tag":141,"props":169,"children":170},{},[171],{"type":48,"value":172},"APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain —\nonce a K8s root cause is ruled in or out, APM investigation continues outside this skill.",{"type":42,"tag":141,"props":174,"children":175},{},[176],{"type":48,"value":177},"Cluster provisioning, capacity planning, cost optimization. Different domain.",{"type":42,"tag":57,"props":179,"children":181},{"id":180},"guidelines",[182],{"type":48,"value":183},"Guidelines",{"type":42,"tag":51,"props":185,"children":186},{},[187],{"type":48,"value":188},"These apply to every investigation. When in doubt, re-read them before writing the synthesis.",{"type":42,"tag":51,"props":190,"children":191},{},[192,197,199,205,207,213],{"type":42,"tag":67,"props":193,"children":194},{},[195],{"type":48,"value":196},"Absence of evidence is not evidence. Do not confabulate from empty results.",{"type":48,"value":198}," If log queries return 0 rows, logs are\nlikely not collected or the pod has no recent lines — this does ",{"type":42,"tag":200,"props":201,"children":202},"em",{},[203],{"type":48,"value":204},"not",{"type":48,"value":206}," mean \"dependency unavailable\" or any other\nspecific failure mode. Report ",{"type":42,"tag":75,"props":208,"children":210},{"className":209},[],[211],{"type":48,"value":212},"no_logs_available",{"type":48,"value":214}," and weight remaining signals accordingly.",{"type":42,"tag":51,"props":216,"children":217},{},[218,223,225,231],{"type":42,"tag":67,"props":219,"children":220},{},[221],{"type":48,"value":222},"Empty dependency data ≠ upstream healthy.",{"type":48,"value":224}," Services without APM instrumentation (load generators, workers) emit no\ndestination metrics. Report ",{"type":42,"tag":75,"props":226,"children":228},{"className":227},[],[229],{"type":48,"value":230},"insufficient_dependency_data",{"type":48,"value":232},", not \"upstreams OK.\"",{"type":42,"tag":51,"props":234,"children":235},{},[236,241],{"type":42,"tag":67,"props":237,"children":238},{},[239],{"type":48,"value":240},"Co-symptoms are not causes.",{"type":48,"value":242}," Two services degrading simultaneously usually share an upstream, not a causal link. Only\nattribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5×\nerror rate, >3× latency).",{"type":42,"tag":51,"props":244,"children":245},{},[246,251],{"type":42,"tag":67,"props":247,"children":248},{},[249],{"type":48,"value":250},"OOMKilled ≠ memory leak by default.",{"type":48,"value":252}," The limit may simply be undersized for the workload's working set. Compare\nagainst a 7-day baseline at the same hour-of-day before claiming a leak.",{"type":42,"tag":51,"props":254,"children":255},{},[256,261,263,269],{"type":42,"tag":67,"props":257,"children":258},{},[259],{"type":48,"value":260},"Error-termination ≠ application bug by default.",{"type":48,"value":262}," Check ",{"type":42,"tag":75,"props":264,"children":266},{"className":265},[],[267],{"type":48,"value":268},"k8s.pod.cpu_limit_utilization",{"type":48,"value":270}," first. CFS throttling driving\nliveness probe timeouts is the most common misdiagnosis in this space.",{"type":42,"tag":51,"props":272,"children":273},{},[274,279,281,287],{"type":42,"tag":67,"props":275,"children":276},{},[277],{"type":48,"value":278},"Average CPU hides throttling.",{"type":48,"value":280}," A pod can look healthy at 40–60% average ",{"type":42,"tag":75,"props":282,"children":284},{"className":283},[],[285],{"type":48,"value":286},"cpu_limit_utilization",{"type":48,"value":288}," while being throttled\nseverely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads hit quota mid-period and stall. Look at\nmax and p95, not just average.",{"type":42,"tag":51,"props":290,"children":291},{},[292,297,299,304,306,312,314,320,322,328,330,336],{"type":42,"tag":67,"props":293,"children":294},{},[295],{"type":48,"value":296},"Restart count is boolean, not a counter.",{"type":48,"value":298}," ",{"type":42,"tag":75,"props":300,"children":302},{"className":301},[],[303],{"type":48,"value":125},{"type":48,"value":305}," is pulled directly from the K8s API and may be\npruned by the kubelet at any time, so the absolute value is unreliable. Treat it as ",{"type":42,"tag":75,"props":307,"children":309},{"className":308},[],[310],{"type":48,"value":311},"== 0",{"type":48,"value":313}," (no recent restarts) vs ",{"type":42,"tag":75,"props":315,"children":317},{"className":316},[],[318],{"type":48,"value":319},"> 0",{"type":48,"value":321},"\n(recently restarting); do not derive backoff timing or \"linear vs exponential\" patterns from it. Confirm the restart\npattern via K8s ",{"type":42,"tag":75,"props":323,"children":325},{"className":324},[],[326],{"type":48,"value":327},"Killing",{"type":48,"value":329}," \u002F ",{"type":42,"tag":75,"props":331,"children":333},{"className":332},[],[334],{"type":48,"value":335},"BackOff",{"type":48,"value":337}," events instead.",{"type":42,"tag":51,"props":339,"children":340},{},[341,346],{"type":42,"tag":67,"props":342,"children":343},{},[344],{"type":48,"value":345},"Prefer to report uncertainty over manufacturing confidence.",{"type":48,"value":347}," If the evidence is ambiguous, the synthesis should say\nso. Competing hypotheses are a valid output.",{"type":42,"tag":57,"props":349,"children":351},{"id":350},"indices-and-fields",[352],{"type":48,"value":353},"Indices and fields",{"type":42,"tag":355,"props":356,"children":358},"h3",{"id":357},"where-to-look",[359],{"type":48,"value":360},"Where to look",{"type":42,"tag":362,"props":363,"children":364},"table",{},[365,389],{"type":42,"tag":366,"props":367,"children":368},"thead",{},[369],{"type":42,"tag":370,"props":371,"children":372},"tr",{},[373,379,384],{"type":42,"tag":374,"props":375,"children":376},"th",{},[377],{"type":48,"value":378},"Signal",{"type":42,"tag":374,"props":380,"children":381},{},[382],{"type":48,"value":383},"Index pattern",{"type":42,"tag":374,"props":385,"children":386},{},[387],{"type":48,"value":388},"Use",{"type":42,"tag":390,"props":391,"children":392},"tbody",{},[393,415,436,457,478,518,555],{"type":42,"tag":370,"props":394,"children":395},{},[396,402,410],{"type":42,"tag":397,"props":398,"children":399},"td",{},[400],{"type":48,"value":401},"Pod\u002Fcontainer runtime",{"type":42,"tag":397,"props":403,"children":404},{},[405],{"type":42,"tag":75,"props":406,"children":408},{"className":407},[],[409],{"type":48,"value":80},{"type":42,"tag":397,"props":411,"children":412},{},[413],{"type":48,"value":414},"CPU, memory, network, filesystem. Utilization ratios.",{"type":42,"tag":370,"props":416,"children":417},{},[418,423,431],{"type":42,"tag":397,"props":419,"children":420},{},[421],{"type":48,"value":422},"Cluster state",{"type":42,"tag":397,"props":424,"children":425},{},[426],{"type":42,"tag":75,"props":427,"children":429},{"className":428},[],[430],{"type":48,"value":88},{"type":42,"tag":397,"props":432,"children":433},{},[434],{"type":48,"value":435},"Restarts, phase, last-terminated reason, HPA, quota, node condition",{"type":42,"tag":370,"props":437,"children":438},{},[439,444,452],{"type":42,"tag":397,"props":440,"children":441},{},[442],{"type":48,"value":443},"K8s events",{"type":42,"tag":397,"props":445,"children":446},{},[447],{"type":42,"tag":75,"props":448,"children":450},{"className":449},[],[451],{"type":48,"value":96},{"type":42,"tag":397,"props":453,"children":454},{},[455],{"type":48,"value":456},"Killing, BackOff, FailedScheduling, Evicted, image pull events",{"type":42,"tag":370,"props":458,"children":459},{},[460,465,473],{"type":42,"tag":397,"props":461,"children":462},{},[463],{"type":48,"value":464},"K8s object snapshots",{"type":42,"tag":397,"props":466,"children":467},{},[468],{"type":42,"tag":75,"props":469,"children":471},{"className":470},[],[472],{"type":48,"value":103},{"type":42,"tag":397,"props":474,"children":475},{},[476],{"type":48,"value":477},"Deployment\u002Fservice\u002Fconfigmap state over time",{"type":42,"tag":370,"props":479,"children":480},{},[481,486,495],{"type":42,"tag":397,"props":482,"children":483},{},[484],{"type":48,"value":485},"Application logs",{"type":42,"tag":397,"props":487,"children":488},{},[489],{"type":42,"tag":75,"props":490,"children":492},{"className":491},[],[493],{"type":48,"value":494},"logs-*.otel-*",{"type":42,"tag":397,"props":496,"children":497},{},[498,504,505,511,513],{"type":42,"tag":75,"props":499,"children":501},{"className":500},[],[502],{"type":48,"value":503},"body.text",{"type":48,"value":90},{"type":42,"tag":75,"props":506,"children":508},{"className":507},[],[509],{"type":48,"value":510},"severity_text",{"type":48,"value":512},", filtered by ",{"type":42,"tag":75,"props":514,"children":516},{"className":515},[],[517],{"type":48,"value":111},{"type":42,"tag":370,"props":519,"children":520},{},[521,526,542],{"type":42,"tag":397,"props":522,"children":523},{},[524],{"type":48,"value":525},"APM",{"type":42,"tag":397,"props":527,"children":528},{},[529,535,536],{"type":42,"tag":75,"props":530,"children":532},{"className":531},[],[533],{"type":48,"value":534},"traces-*.otel-*",{"type":48,"value":90},{"type":42,"tag":75,"props":537,"children":539},{"className":538},[],[540],{"type":48,"value":541},"metrics-service_*.otel-default",{"type":42,"tag":397,"props":543,"children":544},{},[545,547,553],{"type":48,"value":546},"Correlate via ",{"type":42,"tag":75,"props":548,"children":550},{"className":549},[],[551],{"type":48,"value":552},"service.name",{"type":48,"value":554}," + K8s resource attrs",{"type":42,"tag":370,"props":556,"children":557},{},[558,563,572],{"type":42,"tag":397,"props":559,"children":560},{},[561],{"type":48,"value":562},"ML anomalies",{"type":42,"tag":397,"props":564,"children":565},{},[566],{"type":42,"tag":75,"props":567,"children":569},{"className":568},[],[570],{"type":48,"value":571},".ml-anomalies-*",{"type":42,"tag":397,"props":573,"children":574},{},[575],{"type":48,"value":576},"Memory-growth, restart-rate, throttle jobs (if configured)",{"type":42,"tag":355,"props":578,"children":580},{"id":579},"key-fields",[581],{"type":48,"value":582},"Key fields",{"type":42,"tag":51,"props":584,"children":585},{},[586,588,594],{"type":48,"value":587},"Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested ",{"type":42,"tag":75,"props":589,"children":591},{"className":590},[],[592],{"type":48,"value":593},"resource.attributes.*",{"type":48,"value":595}," form is for raw\nlog documents only.",{"type":42,"tag":362,"props":597,"children":598},{},[599,620],{"type":42,"tag":366,"props":600,"children":601},{},[602],{"type":42,"tag":370,"props":603,"children":604},{},[605,610,615],{"type":42,"tag":374,"props":606,"children":607},{},[608],{"type":48,"value":609},"Field",{"type":42,"tag":374,"props":611,"children":612},{},[613],{"type":48,"value":614},"Index",{"type":42,"tag":374,"props":616,"children":617},{},[618],{"type":48,"value":619},"What it is",{"type":42,"tag":390,"props":621,"children":622},{},[623,644,664,685,707,729,749,795,831,853,873,901,922,943,971,993,1014],{"type":42,"tag":370,"props":624,"children":625},{},[626,634,639],{"type":42,"tag":397,"props":627,"children":628},{},[629],{"type":42,"tag":75,"props":630,"children":632},{"className":631},[],[633],{"type":48,"value":111},{"type":42,"tag":397,"props":635,"children":636},{},[637],{"type":48,"value":638},"all k8s",{"type":42,"tag":397,"props":640,"children":641},{},[642],{"type":48,"value":643},"Pod name",{"type":42,"tag":370,"props":645,"children":646},{},[647,655,659],{"type":42,"tag":397,"props":648,"children":649},{},[650],{"type":42,"tag":75,"props":651,"children":653},{"className":652},[],[654],{"type":48,"value":118},{"type":42,"tag":397,"props":656,"children":657},{},[658],{"type":48,"value":638},{"type":42,"tag":397,"props":660,"children":661},{},[662],{"type":48,"value":663},"Namespace",{"type":42,"tag":370,"props":665,"children":666},{},[667,676,680],{"type":42,"tag":397,"props":668,"children":669},{},[670],{"type":42,"tag":75,"props":671,"children":673},{"className":672},[],[674],{"type":48,"value":675},"k8s.container.name",{"type":42,"tag":397,"props":677,"children":678},{},[679],{"type":48,"value":638},{"type":42,"tag":397,"props":681,"children":682},{},[683],{"type":48,"value":684},"Container within pod",{"type":42,"tag":370,"props":686,"children":687},{},[688,697,702],{"type":42,"tag":397,"props":689,"children":690},{},[691],{"type":42,"tag":75,"props":692,"children":694},{"className":693},[],[695],{"type":48,"value":696},"k8s.deployment.name",{"type":42,"tag":397,"props":698,"children":699},{},[700],{"type":48,"value":701},"k8sclusterreceiver + others",{"type":42,"tag":397,"props":703,"children":704},{},[705],{"type":48,"value":706},"Parent deployment",{"type":42,"tag":370,"props":708,"children":709},{},[710,719,724],{"type":42,"tag":397,"props":711,"children":712},{},[713],{"type":42,"tag":75,"props":714,"children":716},{"className":715},[],[717],{"type":48,"value":718},"k8s.pod.phase",{"type":42,"tag":397,"props":720,"children":721},{},[722],{"type":48,"value":723},"k8sclusterreceiver",{"type":42,"tag":397,"props":725,"children":726},{},[727],{"type":48,"value":728},"Pending=1\u002FRunning=2\u002FSucceeded=3\u002FFailed=4\u002FUnknown=5",{"type":42,"tag":370,"props":730,"children":731},{},[732,740,744],{"type":42,"tag":397,"props":733,"children":734},{},[735],{"type":42,"tag":75,"props":736,"children":738},{"className":737},[],[739],{"type":48,"value":125},{"type":42,"tag":397,"props":741,"children":742},{},[743],{"type":48,"value":723},{"type":42,"tag":397,"props":745,"children":746},{},[747],{"type":48,"value":748},"Total container restart count",{"type":42,"tag":370,"props":750,"children":751},{},[752,761,765],{"type":42,"tag":397,"props":753,"children":754},{},[755],{"type":42,"tag":75,"props":756,"children":758},{"className":757},[],[759],{"type":48,"value":760},"k8s.container.status.last_terminated_reason",{"type":42,"tag":397,"props":762,"children":763},{},[764],{"type":48,"value":723},{"type":42,"tag":397,"props":766,"children":767},{},[768,774,775,781,782,788,789],{"type":42,"tag":75,"props":769,"children":771},{"className":770},[],[772],{"type":48,"value":773},"OOMKilled",{"type":48,"value":90},{"type":42,"tag":75,"props":776,"children":778},{"className":777},[],[779],{"type":48,"value":780},"Error",{"type":48,"value":90},{"type":42,"tag":75,"props":783,"children":785},{"className":784},[],[786],{"type":48,"value":787},"Completed",{"type":48,"value":90},{"type":42,"tag":75,"props":790,"children":792},{"className":791},[],[793],{"type":48,"value":794},"ContainerCannotRun",{"type":42,"tag":370,"props":796,"children":797},{},[798,807,811],{"type":42,"tag":397,"props":799,"children":800},{},[801],{"type":42,"tag":75,"props":802,"children":804},{"className":803},[],[805],{"type":48,"value":806},"k8s.pod.status_reason",{"type":42,"tag":397,"props":808,"children":809},{},[810],{"type":48,"value":723},{"type":42,"tag":397,"props":812,"children":813},{},[814,816,822,823,829],{"type":48,"value":815},"Pod-level reason (",{"type":42,"tag":75,"props":817,"children":819},{"className":818},[],[820],{"type":48,"value":821},"Evicted",{"type":48,"value":90},{"type":42,"tag":75,"props":824,"children":826},{"className":825},[],[827],{"type":48,"value":828},"NodeLost",{"type":48,"value":830},")",{"type":42,"tag":370,"props":832,"children":833},{},[834,843,848],{"type":42,"tag":397,"props":835,"children":836},{},[837],{"type":42,"tag":75,"props":838,"children":840},{"className":839},[],[841],{"type":48,"value":842},"k8s.pod.memory_limit_utilization",{"type":42,"tag":397,"props":844,"children":845},{},[846],{"type":48,"value":847},"kubeletstatsreceiver",{"type":42,"tag":397,"props":849,"children":850},{},[851],{"type":48,"value":852},"0.0–1.0+ (can exceed 1 transiently before OOM)",{"type":42,"tag":370,"props":854,"children":855},{},[856,864,868],{"type":42,"tag":397,"props":857,"children":858},{},[859],{"type":42,"tag":75,"props":860,"children":862},{"className":861},[],[863],{"type":48,"value":268},{"type":42,"tag":397,"props":865,"children":866},{},[867],{"type":48,"value":847},{"type":42,"tag":397,"props":869,"children":870},{},[871],{"type":48,"value":872},"0.0–N (frequently >1 under CFS throttling)",{"type":42,"tag":370,"props":874,"children":875},{},[876,892,896],{"type":42,"tag":397,"props":877,"children":878},{},[879,885,886],{"type":42,"tag":75,"props":880,"children":882},{"className":881},[],[883],{"type":48,"value":884},"k8s.pod.memory.usage",{"type":48,"value":329},{"type":42,"tag":75,"props":887,"children":889},{"className":888},[],[890],{"type":48,"value":891},".working_set",{"type":42,"tag":397,"props":893,"children":894},{},[895],{"type":48,"value":847},{"type":42,"tag":397,"props":897,"children":898},{},[899],{"type":48,"value":900},"Bytes",{"type":42,"tag":370,"props":902,"children":903},{},[904,913,917],{"type":42,"tag":397,"props":905,"children":906},{},[907],{"type":42,"tag":75,"props":908,"children":910},{"className":909},[],[911],{"type":48,"value":912},"k8s.node.condition_memory_pressure",{"type":42,"tag":397,"props":914,"children":915},{},[916],{"type":48,"value":723},{"type":42,"tag":397,"props":918,"children":919},{},[920],{"type":48,"value":921},"1 = pressure, 0 = ok",{"type":42,"tag":370,"props":923,"children":924},{},[925,934,938],{"type":42,"tag":397,"props":926,"children":927},{},[928],{"type":42,"tag":75,"props":929,"children":931},{"className":930},[],[932],{"type":48,"value":933},"k8s.node.condition_ready",{"type":42,"tag":397,"props":935,"children":936},{},[937],{"type":48,"value":723},{"type":42,"tag":397,"props":939,"children":940},{},[941],{"type":48,"value":942},"0 = NotReady",{"type":42,"tag":370,"props":944,"children":945},{},[946,962,966],{"type":42,"tag":397,"props":947,"children":948},{},[949,955,956],{"type":42,"tag":75,"props":950,"children":952},{"className":951},[],[953],{"type":48,"value":954},"k8s.hpa.current_replicas",{"type":48,"value":329},{"type":42,"tag":75,"props":957,"children":959},{"className":958},[],[960],{"type":48,"value":961},".desired_replicas",{"type":42,"tag":397,"props":963,"children":964},{},[965],{"type":48,"value":723},{"type":42,"tag":397,"props":967,"children":968},{},[969],{"type":48,"value":970},"HPA state",{"type":42,"tag":370,"props":972,"children":973},{},[974,983,988],{"type":42,"tag":397,"props":975,"children":976},{},[977],{"type":42,"tag":75,"props":978,"children":980},{"className":979},[],[981],{"type":48,"value":982},"attributes.k8s.event.reason",{"type":42,"tag":397,"props":984,"children":985},{},[986],{"type":48,"value":987},"k8seventsreceiver",{"type":42,"tag":397,"props":989,"children":990},{},[991],{"type":48,"value":992},"Event reason (filter on this)",{"type":42,"tag":370,"props":994,"children":995},{},[996,1004,1009],{"type":42,"tag":397,"props":997,"children":998},{},[999],{"type":42,"tag":75,"props":1000,"children":1002},{"className":1001},[],[1003],{"type":48,"value":503},{"type":42,"tag":397,"props":1005,"children":1006},{},[1007],{"type":48,"value":1008},"k8seventsreceiver \u002F logs",{"type":42,"tag":397,"props":1010,"children":1011},{},[1012],{"type":48,"value":1013},"Event message \u002F log message",{"type":42,"tag":370,"props":1015,"children":1016},{},[1017,1026,1030],{"type":42,"tag":397,"props":1018,"children":1019},{},[1020],{"type":42,"tag":75,"props":1021,"children":1023},{"className":1022},[],[1024],{"type":48,"value":1025},"k8s.object.name",{"type":42,"tag":397,"props":1027,"children":1028},{},[1029],{"type":48,"value":987},{"type":42,"tag":397,"props":1031,"children":1032},{},[1033],{"type":48,"value":1034},"involvedObject name (log attribute, use flat form)",{"type":42,"tag":355,"props":1036,"children":1038},{"id":1037},"field-availability",[1039],{"type":48,"value":1040},"Field availability",{"type":42,"tag":51,"props":1042,"children":1043},{},[1044],{"type":48,"value":1045},"Several fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify\npresence before relying on them; if absent, fall back as noted and call out the substitution in the synthesis.",{"type":42,"tag":362,"props":1047,"children":1048},{},[1049,1069],{"type":42,"tag":366,"props":1050,"children":1051},{},[1052],{"type":42,"tag":370,"props":1053,"children":1054},{},[1055,1059,1064],{"type":42,"tag":374,"props":1056,"children":1057},{},[1058],{"type":48,"value":609},{"type":42,"tag":374,"props":1060,"children":1061},{},[1062],{"type":48,"value":1063},"Why it might be missing",{"type":42,"tag":374,"props":1065,"children":1066},{},[1067],{"type":48,"value":1068},"Fall-back",{"type":42,"tag":390,"props":1070,"children":1071},{},[1072,1122,1163,1199],{"type":42,"tag":370,"props":1073,"children":1074},{},[1075,1083,1096],{"type":42,"tag":397,"props":1076,"children":1077},{},[1078],{"type":42,"tag":75,"props":1079,"children":1081},{"className":1080},[],[1082],{"type":48,"value":760},{"type":42,"tag":397,"props":1084,"children":1085},{},[1086,1088,1094],{"type":48,"value":1087},"Optional metric in k8sclusterreceiver; gated behind ",{"type":42,"tag":75,"props":1089,"children":1091},{"className":1090},[],[1092],{"type":48,"value":1093},"metrics_collected.metadata",{"type":48,"value":1095}," config.",{"type":42,"tag":397,"props":1097,"children":1098},{},[1099,1101,1106,1107,1113,1115,1120],{"type":48,"value":1100},"Infer from K8s ",{"type":42,"tag":75,"props":1102,"children":1104},{"className":1103},[],[1105],{"type":48,"value":327},{"type":48,"value":329},{"type":42,"tag":75,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":48,"value":1112},"OOMKilling",{"type":48,"value":1114}," events in ",{"type":42,"tag":75,"props":1116,"children":1118},{"className":1117},[],[1119],{"type":48,"value":96},{"type":48,"value":1121}," and exit codes in app logs.",{"type":42,"tag":370,"props":1123,"children":1124},{},[1125,1133,1138],{"type":42,"tag":397,"props":1126,"children":1127},{},[1128],{"type":42,"tag":75,"props":1129,"children":1131},{"className":1130},[],[1132],{"type":48,"value":806},{"type":42,"tag":397,"props":1134,"children":1135},{},[1136],{"type":48,"value":1137},"Same — optional metric on k8sclusterreceiver.",{"type":42,"tag":397,"props":1139,"children":1140},{},[1141,1143,1148,1149,1154,1155,1161],{"type":48,"value":1142},"Infer from events: ",{"type":42,"tag":75,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":48,"value":821},{"type":48,"value":90},{"type":42,"tag":75,"props":1150,"children":1152},{"className":1151},[],[1153],{"type":48,"value":828},{"type":48,"value":90},{"type":42,"tag":75,"props":1156,"children":1158},{"className":1157},[],[1159],{"type":48,"value":1160},"Preempted",{"type":48,"value":1162},".",{"type":42,"tag":370,"props":1164,"children":1165},{},[1166,1181,1186],{"type":42,"tag":397,"props":1167,"children":1168},{},[1169,1174,1175],{"type":42,"tag":75,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":48,"value":268},{"type":48,"value":329},{"type":42,"tag":75,"props":1176,"children":1178},{"className":1177},[],[1179],{"type":48,"value":1180},"memory_limit_utilization",{"type":42,"tag":397,"props":1182,"children":1183},{},[1184],{"type":48,"value":1185},"Only emitted when the pod has the corresponding limit set, and the kubeletstatsreceiver metric is enabled.",{"type":42,"tag":397,"props":1187,"children":1188},{},[1189,1191,1197],{"type":48,"value":1190},"Compute manually as ",{"type":42,"tag":75,"props":1192,"children":1194},{"className":1193},[],[1195],{"type":48,"value":1196},"k8s.pod.cpu.usage \u002F \u003Climit>",{"type":48,"value":1198}," from k8sclusterreceiver, or use absolute usage trending against a baseline.",{"type":42,"tag":370,"props":1200,"children":1201},{},[1202,1210,1223],{"type":42,"tag":397,"props":1203,"children":1204},{},[1205],{"type":42,"tag":75,"props":1206,"children":1208},{"className":1207},[],[1209],{"type":48,"value":912},{"type":42,"tag":397,"props":1211,"children":1212},{},[1213,1215,1221],{"type":48,"value":1214},"Gated behind k8sclusterreceiver ",{"type":42,"tag":75,"props":1216,"children":1218},{"className":1217},[],[1219],{"type":48,"value":1220},"node_conditions_to_report",{"type":48,"value":1222}," (default omits this).",{"type":42,"tag":397,"props":1224,"children":1225},{},[1226,1228,1234,1236,1242,1244,1249],{"type":48,"value":1227},"Compare ",{"type":42,"tag":75,"props":1229,"children":1231},{"className":1230},[],[1232],{"type":48,"value":1233},"k8s.node.memory.usage",{"type":48,"value":1235}," against ",{"type":42,"tag":75,"props":1237,"children":1239},{"className":1238},[],[1240],{"type":48,"value":1241},"k8s.node.allocatable_memory",{"type":48,"value":1243},", or look for ",{"type":42,"tag":75,"props":1245,"children":1247},{"className":1246},[],[1248],{"type":48,"value":821},{"type":48,"value":1250}," events on the node.",{"type":42,"tag":51,"props":1252,"children":1253},{},[1254,1256,1262],{"type":48,"value":1255},"If a fall-back is used, note it in the synthesis (e.g. ",{"type":42,"tag":75,"props":1257,"children":1259},{"className":1258},[],[1260],{"type":48,"value":1261},"(via memory.usage; limit_utilization not collected)",{"type":48,"value":1263},") so the\nreader knows the signal is indirect.",{"type":42,"tag":57,"props":1265,"children":1267},{"id":1266},"esql-gotchas",[1268],{"type":48,"value":1269},"ES|QL gotchas",{"type":42,"tag":51,"props":1271,"children":1272},{},[1273],{"type":48,"value":1274},"Before writing queries, know these. Each of them silently produces wrong answers rather than failing loudly.",{"type":42,"tag":51,"props":1276,"children":1277},{},[1278,1289,1291,1297,1299,1305],{"type":42,"tag":67,"props":1279,"children":1280},{},[1281,1287],{"type":42,"tag":75,"props":1282,"children":1284},{"className":1283},[],[1285],{"type":48,"value":1286},"VALUES()",{"type":48,"value":1288}," returns scalar for single distinct value, array for multiple.",{"type":48,"value":1290}," Templating that assumes array shape (e.g.\n",{"type":42,"tag":75,"props":1292,"children":1294},{"className":1293},[],[1295],{"type":48,"value":1296},"| first",{"type":48,"value":1298},") extracts the first character of the string when scalar. Use ",{"type":42,"tag":75,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":48,"value":1304},"MV_FIRST(VALUES(...))",{"type":48,"value":1306}," or handle both.",{"type":42,"tag":51,"props":1308,"children":1309},{},[1310,1329,1331,1337,1339,1345,1347,1353],{"type":42,"tag":67,"props":1311,"children":1312},{},[1313,1319,1321,1327],{"type":42,"tag":75,"props":1314,"children":1316},{"className":1315},[],[1317],{"type":48,"value":1318},"PERCENTILE",{"type":48,"value":1320}," does not work on OTel ",{"type":42,"tag":75,"props":1322,"children":1324},{"className":1323},[],[1325],{"type":48,"value":1326},"histogram",{"type":48,"value":1328}," type",{"type":48,"value":1330}," (as of 8.15). For APM duration percentiles, use ",{"type":42,"tag":75,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":48,"value":1336},"AVG",{"type":48,"value":1338}," on the\n",{"type":42,"tag":75,"props":1340,"children":1342},{"className":1341},[],[1343],{"type":48,"value":1344},"aggregate_metric_double",{"type":48,"value":1346}," summary field (",{"type":42,"tag":75,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":48,"value":1352},"AVG(transaction.duration.summary)",{"type":48,"value":1354}," divides sum by value_count). For true\npercentiles, fall back to Kibana Query DSL.",{"type":42,"tag":51,"props":1356,"children":1357},{},[1358,1377,1378,1384,1386,1392,1394,1400],{"type":42,"tag":67,"props":1359,"children":1360},{},[1361,1367,1369,1375],{"type":42,"tag":75,"props":1362,"children":1364},{"className":1363},[],[1365],{"type":48,"value":1366},"COUNT(agg_metric_double)",{"type":48,"value":1368}," returns ",{"type":42,"tag":75,"props":1370,"children":1372},{"className":1371},[],[1373],{"type":48,"value":1374},"value_count",{"type":48,"value":1376}," (events), not doc count.",{"type":48,"value":298},{"type":42,"tag":75,"props":1379,"children":1381},{"className":1380},[],[1382],{"type":48,"value":1383},"SUM(field)",{"type":48,"value":1385}," gives the sum component;\n",{"type":42,"tag":75,"props":1387,"children":1389},{"className":1388},[],[1390],{"type":48,"value":1391},"AVG(field)",{"type":48,"value":1393}," gives sum\u002Fvalue_count. Do not use ",{"type":42,"tag":75,"props":1395,"children":1397},{"className":1396},[],[1398],{"type":48,"value":1399},"SUM(transaction.duration.summary)",{"type":48,"value":1401}," as an event-count proxy — it returns\ntotal duration.",{"type":42,"tag":51,"props":1403,"children":1404},{},[1405,1410,1411,1416,1418,1424],{"type":42,"tag":67,"props":1406,"children":1407},{},[1408],{"type":48,"value":1409},"K8s metrics use flat OTel field paths in ES|QL.",{"type":48,"value":298},{"type":42,"tag":75,"props":1412,"children":1414},{"className":1413},[],[1415],{"type":48,"value":111},{"type":48,"value":1417},", not ",{"type":42,"tag":75,"props":1419,"children":1421},{"className":1420},[],[1422],{"type":48,"value":1423},"resource.attributes.k8s.pod.name",{"type":48,"value":1425},". The nested\nform is for raw log documents.",{"type":42,"tag":57,"props":1427,"children":1429},{"id":1428},"failure-mode-taxonomy",[1430],{"type":48,"value":1431},"Failure-mode taxonomy",{"type":42,"tag":51,"props":1433,"children":1434},{},[1435],{"type":48,"value":1436},"Vocabulary for classification, not a decision tree. Use the pivotal-signal column to recognize which mode you're looking\nat; use \"Investigate\" to know what else should corroborate.",{"type":42,"tag":355,"props":1438,"children":1440},{"id":1439},"workload-layer",[1441],{"type":48,"value":1442},"Workload layer",{"type":42,"tag":362,"props":1444,"children":1445},{},[1446,1467],{"type":42,"tag":366,"props":1447,"children":1448},{},[1449],{"type":42,"tag":370,"props":1450,"children":1451},{},[1452,1457,1462],{"type":42,"tag":374,"props":1453,"children":1454},{},[1455],{"type":48,"value":1456},"Mode",{"type":42,"tag":374,"props":1458,"children":1459},{},[1460],{"type":48,"value":1461},"Pivotal signal",{"type":42,"tag":374,"props":1463,"children":1464},{},[1465],{"type":48,"value":1466},"Investigate",{"type":42,"tag":390,"props":1468,"children":1469},{},[1470,1502,1534,1585,1624,1683,1736],{"type":42,"tag":370,"props":1471,"children":1472},{},[1473,1480,1497],{"type":42,"tag":397,"props":1474,"children":1475},{},[1476],{"type":42,"tag":67,"props":1477,"children":1478},{},[1479],{"type":48,"value":773},{"type":42,"tag":397,"props":1481,"children":1482},{},[1483,1489,1491],{"type":42,"tag":75,"props":1484,"children":1486},{"className":1485},[],[1487],{"type":48,"value":1488},"last_terminated_reason == \"OOMKilled\"",{"type":48,"value":1490}," + ",{"type":42,"tag":75,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":48,"value":1496},"memory_limit_utilization → 1.0",{"type":42,"tag":397,"props":1498,"children":1499},{},[1500],{"type":48,"value":1501},"Monotonic rise (leak) vs. load-driven spike? Compare current trend to 7-day baseline. Check heap metrics (JVM, Go, Node) for GC pressure.",{"type":42,"tag":370,"props":1503,"children":1504},{},[1505,1513,1529],{"type":42,"tag":397,"props":1506,"children":1507},{},[1508],{"type":42,"tag":67,"props":1509,"children":1510},{},[1511],{"type":48,"value":1512},"CPU throttling → Error exit",{"type":42,"tag":397,"props":1514,"children":1515},{},[1516,1522,1523],{"type":42,"tag":75,"props":1517,"children":1519},{"className":1518},[],[1520],{"type":48,"value":1521},"cpu_limit_utilization > 1.0",{"type":48,"value":1490},{"type":42,"tag":75,"props":1524,"children":1526},{"className":1525},[],[1527],{"type":48,"value":1528},"last_terminated_reason == \"Error\"",{"type":42,"tag":397,"props":1530,"children":1531},{},[1532],{"type":48,"value":1533},"Liveness\u002Freadiness probe timeouts from CFS throttling. Average CPU can look fine (40–60%) while p99 throttle is severe. Check probe timeouts vs observed startup\u002Fhealth latency.",{"type":42,"tag":370,"props":1535,"children":1536},{},[1537,1545,1558],{"type":42,"tag":397,"props":1538,"children":1539},{},[1540],{"type":42,"tag":67,"props":1541,"children":1542},{},[1543],{"type":48,"value":1544},"Liveness probe misconfiguration",{"type":42,"tag":397,"props":1546,"children":1547},{},[1548,1550,1556],{"type":48,"value":1549},"Restarts without resource pressure; ",{"type":42,"tag":75,"props":1551,"children":1553},{"className":1552},[],[1554],{"type":48,"value":1555},"initialDelaySeconds",{"type":48,"value":1557}," \u003C startup time",{"type":42,"tag":397,"props":1559,"children":1560},{},[1561,1563,1569,1570,1575,1577,1583],{"type":48,"value":1562},"K8s events show ",{"type":42,"tag":75,"props":1564,"children":1566},{"className":1565},[],[1567],{"type":48,"value":1568},"Unhealthy",{"type":48,"value":329},{"type":42,"tag":75,"props":1571,"children":1573},{"className":1572},[],[1574],{"type":48,"value":327},{"type":48,"value":1576},". ",{"type":42,"tag":75,"props":1578,"children":1580},{"className":1579},[],[1581],{"type":48,"value":1582},"kubectl logs --previous",{"type":48,"value":1584}," typically shows healthy startup before kill.",{"type":42,"tag":370,"props":1586,"children":1587},{},[1588,1596,1611],{"type":42,"tag":397,"props":1589,"children":1590},{},[1591],{"type":42,"tag":67,"props":1592,"children":1593},{},[1594],{"type":48,"value":1595},"CrashLoopBackOff (generic)",{"type":42,"tag":397,"props":1597,"children":1598},{},[1599,1604,1606],{"type":42,"tag":75,"props":1600,"children":1602},{"className":1601},[],[1603],{"type":48,"value":335},{"type":48,"value":1605}," events + rising ",{"type":42,"tag":75,"props":1607,"children":1609},{"className":1608},[],[1610],{"type":48,"value":125},{"type":42,"tag":397,"props":1612,"children":1613},{},[1614,1616,1622],{"type":48,"value":1615},"Branch on ",{"type":42,"tag":75,"props":1617,"children":1619},{"className":1618},[],[1620],{"type":48,"value":1621},"last_terminated_reason",{"type":48,"value":1623}," — this is a meta-mode. OOMKilled → memory path; Error → logs + throttling; ContainerCannotRun → image\u002Fexec.",{"type":42,"tag":370,"props":1625,"children":1626},{},[1627,1635,1662],{"type":42,"tag":397,"props":1628,"children":1629},{},[1630],{"type":42,"tag":67,"props":1631,"children":1632},{},[1633],{"type":48,"value":1634},"ImagePullBackOff",{"type":42,"tag":397,"props":1636,"children":1637},{},[1638,1640,1646,1648,1654,1656],{"type":48,"value":1639},"K8s events ",{"type":42,"tag":75,"props":1641,"children":1643},{"className":1642},[],[1644],{"type":48,"value":1645},"Failed",{"type":48,"value":1647}," with image name + ",{"type":42,"tag":75,"props":1649,"children":1651},{"className":1650},[],[1652],{"type":48,"value":1653},"429",{"type":48,"value":1655}," or ",{"type":42,"tag":75,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":48,"value":1661},"not found",{"type":42,"tag":397,"props":1663,"children":1664},{},[1665,1667,1673,1675,1681],{"type":48,"value":1666},"Registry rate limit? Missing tag? Wrong imagePullSecret? Check recency of ",{"type":42,"tag":75,"props":1668,"children":1670},{"className":1669},[],[1671],{"type":48,"value":1672},"Pulling",{"type":48,"value":1674},"\u002F",{"type":42,"tag":75,"props":1676,"children":1678},{"className":1677},[],[1679],{"type":48,"value":1680},"Pulled",{"type":48,"value":1682}," events.",{"type":42,"tag":370,"props":1684,"children":1685},{},[1686,1694,1715],{"type":42,"tag":397,"props":1687,"children":1688},{},[1689],{"type":42,"tag":67,"props":1690,"children":1691},{},[1692],{"type":48,"value":1693},"Stuck rollout",{"type":42,"tag":397,"props":1695,"children":1696},{},[1697,1699,1705,1707,1713],{"type":48,"value":1698},"New pods ",{"type":42,"tag":75,"props":1700,"children":1702},{"className":1701},[],[1703],{"type":48,"value":1704},"Pending",{"type":48,"value":1706},"\u002Fnot-Ready > ",{"type":42,"tag":75,"props":1708,"children":1710},{"className":1709},[],[1711],{"type":48,"value":1712},"progressDeadlineSeconds",{"type":48,"value":1714},"; old pods still serving",{"type":42,"tag":397,"props":1716,"children":1717},{},[1718,1720,1726,1728,1734],{"type":48,"value":1719},"Check ",{"type":42,"tag":75,"props":1721,"children":1723},{"className":1722},[],[1724],{"type":48,"value":1725},"k8s.deployment.available",{"type":48,"value":1727}," vs ",{"type":42,"tag":75,"props":1729,"children":1731},{"className":1730},[],[1732],{"type":48,"value":1733},".desired",{"type":48,"value":1735},". Admission rejection? Readiness probe failing on new pods? HPA not scaling?",{"type":42,"tag":370,"props":1737,"children":1738},{},[1739,1747,1752],{"type":42,"tag":397,"props":1740,"children":1741},{},[1742],{"type":42,"tag":67,"props":1743,"children":1744},{},[1745],{"type":48,"value":1746},"Termination signal race",{"type":42,"tag":397,"props":1748,"children":1749},{},[1750],{"type":48,"value":1751},"Brief 5xx bursts correlated with rolling deploys",{"type":42,"tag":397,"props":1753,"children":1754},{},[1755,1757,1763,1765,1770,1772,1778],{"type":48,"value":1756},"Endpoint removal races termination. New requests can hit the pod after SIGTERM starts. NGINX gotcha: ",{"type":42,"tag":75,"props":1758,"children":1760},{"className":1759},[],[1761],{"type":48,"value":1762},"STOPSIGNAL SIGTERM",{"type":48,"value":1764}," triggers ",{"type":42,"tag":200,"props":1766,"children":1767},{},[1768],{"type":48,"value":1769},"fast",{"type":48,"value":1771}," shutdown, not graceful — use ",{"type":42,"tag":75,"props":1773,"children":1775},{"className":1774},[],[1776],{"type":48,"value":1777},"STOPSIGNAL SIGQUIT",{"type":48,"value":1779}," for graceful drain. Check ingress 502 rate vs rollout timing.",{"type":42,"tag":355,"props":1781,"children":1783},{"id":1782},"node-layer",[1784],{"type":48,"value":1785},"Node layer",{"type":42,"tag":362,"props":1787,"children":1788},{},[1789,1807],{"type":42,"tag":366,"props":1790,"children":1791},{},[1792],{"type":42,"tag":370,"props":1793,"children":1794},{},[1795,1799,1803],{"type":42,"tag":374,"props":1796,"children":1797},{},[1798],{"type":48,"value":1456},{"type":42,"tag":374,"props":1800,"children":1801},{},[1802],{"type":48,"value":1461},{"type":42,"tag":374,"props":1804,"children":1805},{},[1806],{"type":48,"value":1466},{"type":42,"tag":390,"props":1808,"children":1809},{},[1810,1852,1886],{"type":42,"tag":370,"props":1811,"children":1812},{},[1813,1821,1839],{"type":42,"tag":397,"props":1814,"children":1815},{},[1816],{"type":42,"tag":67,"props":1817,"children":1818},{},[1819],{"type":48,"value":1820},"Node NotReady cascade",{"type":42,"tag":397,"props":1822,"children":1823},{},[1824,1830,1832,1837],{"type":42,"tag":75,"props":1825,"children":1827},{"className":1826},[],[1828],{"type":48,"value":1829},"k8s.node.condition_ready == 0",{"type":48,"value":1831}," + mass ",{"type":42,"tag":75,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":48,"value":821},{"type":48,"value":1838}," events",{"type":42,"tag":397,"props":1840,"children":1841},{},[1842,1844,1850],{"type":48,"value":1843},"Memory pressure? Disk pressure? Network partition from API server? Inspect kubelet logs, ",{"type":42,"tag":75,"props":1845,"children":1847},{"className":1846},[],[1848],{"type":48,"value":1849},"k8s.node.condition_*",{"type":48,"value":1851}," history.",{"type":42,"tag":370,"props":1853,"children":1854},{},[1855,1863,1881],{"type":42,"tag":397,"props":1856,"children":1857},{},[1858],{"type":42,"tag":67,"props":1859,"children":1860},{},[1861],{"type":48,"value":1862},"Resource eviction",{"type":42,"tag":397,"props":1864,"children":1865},{},[1866,1872,1873,1879],{"type":42,"tag":75,"props":1867,"children":1869},{"className":1868},[],[1870],{"type":48,"value":1871},"status_reason == \"Evicted\"",{"type":48,"value":1490},{"type":42,"tag":75,"props":1874,"children":1876},{"className":1875},[],[1877],{"type":48,"value":1878},"condition_memory_pressure == 1",{"type":48,"value":1880}," on node",{"type":42,"tag":397,"props":1882,"children":1883},{},[1884],{"type":48,"value":1885},"Node-level noisy neighbor. QoS order: BestEffort → Burstable → Guaranteed. Identify which pod drove node memory up.",{"type":42,"tag":370,"props":1887,"children":1888},{},[1889,1897,1902],{"type":42,"tag":397,"props":1890,"children":1891},{},[1892],{"type":42,"tag":67,"props":1893,"children":1894},{},[1895],{"type":48,"value":1896},"Node affinity\u002Fselector conflict",{"type":42,"tag":397,"props":1898,"children":1899},{},[1900],{"type":48,"value":1901},"Mass unschedulable pods after label change",{"type":42,"tag":397,"props":1903,"children":1904},{},[1905,1906,1912,1914,1920,1922,1928],{"type":48,"value":1562},{"type":42,"tag":75,"props":1907,"children":1909},{"className":1908},[],[1910],{"type":48,"value":1911},"FailedScheduling",{"type":48,"value":1913},". Often triggered by cluster upgrades (e.g. ",{"type":42,"tag":75,"props":1915,"children":1917},{"className":1916},[],[1918],{"type":48,"value":1919},"node-role.kubernetes.io\u002Fmaster",{"type":48,"value":1921}," → ",{"type":42,"tag":75,"props":1923,"children":1925},{"className":1924},[],[1926],{"type":48,"value":1927},"control-plane",{"type":48,"value":127},{"type":42,"tag":355,"props":1930,"children":1931},{"id":1927},[1932],{"type":48,"value":1933},"Control plane",{"type":42,"tag":362,"props":1935,"children":1936},{},[1937,1955],{"type":42,"tag":366,"props":1938,"children":1939},{},[1940],{"type":42,"tag":370,"props":1941,"children":1942},{},[1943,1947,1951],{"type":42,"tag":374,"props":1944,"children":1945},{},[1946],{"type":48,"value":1456},{"type":42,"tag":374,"props":1948,"children":1949},{},[1950],{"type":48,"value":1461},{"type":42,"tag":374,"props":1952,"children":1953},{},[1954],{"type":48,"value":1466},{"type":42,"tag":390,"props":1956,"children":1957},{},[1958,1979,2014,2066],{"type":42,"tag":370,"props":1959,"children":1960},{},[1961,1969,1974],{"type":42,"tag":397,"props":1962,"children":1963},{},[1964],{"type":42,"tag":67,"props":1965,"children":1966},{},[1967],{"type":48,"value":1968},"etcd I\u002FO cascade",{"type":42,"tag":397,"props":1970,"children":1971},{},[1972],{"type":48,"value":1973},"API server latency spike + cluster-wide kubelet heartbeat failures",{"type":42,"tag":397,"props":1975,"children":1976},{},[1977],{"type":48,"value":1978},"Disk IOPS, fsync latency (must be \u003C10ms). Cloud-burst-credit exhaustion is common.",{"type":42,"tag":370,"props":1980,"children":1981},{},[1982,1990,2003],{"type":42,"tag":397,"props":1983,"children":1984},{},[1985],{"type":42,"tag":67,"props":1986,"children":1987},{},[1988],{"type":48,"value":1989},"Admission webhook block",{"type":42,"tag":397,"props":1991,"children":1992},{},[1993,1995,2001],{"type":48,"value":1994},"Mass ",{"type":42,"tag":75,"props":1996,"children":1998},{"className":1997},[],[1999],{"type":48,"value":2000},"FailedCreate",{"type":48,"value":2002}," across namespaces; deployments frozen",{"type":42,"tag":397,"props":2004,"children":2005},{},[2006,2012],{"type":42,"tag":75,"props":2007,"children":2009},{"className":2008},[],[2010],{"type":48,"value":2011},"failurePolicy:Fail",{"type":48,"value":2013}," webhook pod crashed. Check webhook pod health + API server TCP connection cache (caches dead connections ~15 min).",{"type":42,"tag":370,"props":2015,"children":2016},{},[2017,2025,2038],{"type":42,"tag":397,"props":2018,"children":2019},{},[2020],{"type":42,"tag":67,"props":2021,"children":2022},{},[2023],{"type":48,"value":2024},"Priority preemption storm",{"type":42,"tag":397,"props":2026,"children":2027},{},[2028,2030,2036],{"type":48,"value":2029},"Production pods terminating with ",{"type":42,"tag":75,"props":2031,"children":2033},{"className":2032},[],[2034],{"type":48,"value":2035},"preempted-by",{"type":48,"value":2037}," annotation",{"type":42,"tag":397,"props":2039,"children":2040},{},[2041,2043,2049,2051,2057,2059,2065],{"type":48,"value":2042},"New ",{"type":42,"tag":75,"props":2044,"children":2046},{"className":2045},[],[2047],{"type":48,"value":2048},"PriorityClass",{"type":48,"value":2050}," with ",{"type":42,"tag":75,"props":2052,"children":2054},{"className":2053},[],[2055],{"type":48,"value":2056},"globalDefault:true",{"type":48,"value":2058}," caused cascade. Check ",{"type":42,"tag":75,"props":2060,"children":2062},{"className":2061},[],[2063],{"type":48,"value":2064},"kube-scheduler",{"type":48,"value":1682},{"type":42,"tag":370,"props":2067,"children":2068},{},[2069,2077,2082],{"type":42,"tag":397,"props":2070,"children":2071},{},[2072],{"type":42,"tag":67,"props":2073,"children":2074},{},[2075],{"type":48,"value":2076},"PDB drain deadlock",{"type":42,"tag":397,"props":2078,"children":2079},{},[2080],{"type":48,"value":2081},"Node drain stuck indefinitely; HTTP 429 from Eviction API",{"type":42,"tag":397,"props":2083,"children":2084},{},[2085,2087,2093,2094,2100],{"type":48,"value":2086},"PDB ",{"type":42,"tag":75,"props":2088,"children":2090},{"className":2089},[],[2091],{"type":48,"value":2092},"minAvailable",{"type":48,"value":1674},{"type":42,"tag":75,"props":2095,"children":2097},{"className":2096},[],[2098],{"type":48,"value":2099},"maxUnavailable",{"type":48,"value":2101}," too strict. No default drain timeout. Manual PDB deletion unblocks.",{"type":42,"tag":355,"props":2103,"children":2105},{"id":2104},"autoscaling-admission",[2106],{"type":48,"value":2107},"Autoscaling & admission",{"type":42,"tag":362,"props":2109,"children":2110},{},[2111,2129],{"type":42,"tag":366,"props":2112,"children":2113},{},[2114],{"type":42,"tag":370,"props":2115,"children":2116},{},[2117,2121,2125],{"type":42,"tag":374,"props":2118,"children":2119},{},[2120],{"type":48,"value":1456},{"type":42,"tag":374,"props":2122,"children":2123},{},[2124],{"type":48,"value":1461},{"type":42,"tag":374,"props":2126,"children":2127},{},[2128],{"type":48,"value":1466},{"type":42,"tag":390,"props":2130,"children":2131},{},[2132,2166],{"type":42,"tag":370,"props":2133,"children":2134},{},[2135,2143,2148],{"type":42,"tag":397,"props":2136,"children":2137},{},[2138],{"type":42,"tag":67,"props":2139,"children":2140},{},[2141],{"type":48,"value":2142},"HPA unready-pod dampening",{"type":42,"tag":397,"props":2144,"children":2145},{},[2146],{"type":48,"value":2147},"Load rising, HPA not scaling; unready pods included in calculation",{"type":42,"tag":397,"props":2149,"children":2150},{},[2151,2153,2158,2159,2164],{"type":48,"value":2152},"HPA averages CPU across all replicas including unready (0% contribution). Check ",{"type":42,"tag":75,"props":2154,"children":2156},{"className":2155},[],[2157],{"type":48,"value":954},{"type":48,"value":1727},{"type":42,"tag":75,"props":2160,"children":2162},{"className":2161},[],[2163],{"type":48,"value":961},{"type":48,"value":2165}," + pod readiness.",{"type":42,"tag":370,"props":2167,"children":2168},{},[2169,2177,2189],{"type":42,"tag":397,"props":2170,"children":2171},{},[2172],{"type":42,"tag":67,"props":2173,"children":2174},{},[2175],{"type":48,"value":2176},"Resource quota silent 403",{"type":42,"tag":397,"props":2178,"children":2179},{},[2180,2182,2187],{"type":48,"value":2181},"Deployment stuck at n-1\u002Fn; ",{"type":42,"tag":75,"props":2183,"children":2185},{"className":2184},[],[2186],{"type":48,"value":2000},{"type":48,"value":2188}," on ReplicaSet",{"type":42,"tag":397,"props":2190,"children":2191},{},[2192,2194,2200,2201,2207],{"type":48,"value":2193},"Namespace quota exhausted (often CronJob accumulation). Check ",{"type":42,"tag":75,"props":2195,"children":2197},{"className":2196},[],[2198],{"type":48,"value":2199},"k8s.resource_quota.used",{"type":48,"value":1727},{"type":42,"tag":75,"props":2202,"children":2204},{"className":2203},[],[2205],{"type":48,"value":2206},".hard_limit",{"type":48,"value":1162},{"type":42,"tag":355,"props":2209,"children":2211},{"id":2210},"networking",[2212],{"type":48,"value":2213},"Networking",{"type":42,"tag":362,"props":2215,"children":2216},{},[2217,2235],{"type":42,"tag":366,"props":2218,"children":2219},{},[2220],{"type":42,"tag":370,"props":2221,"children":2222},{},[2223,2227,2231],{"type":42,"tag":374,"props":2224,"children":2225},{},[2226],{"type":48,"value":1456},{"type":42,"tag":374,"props":2228,"children":2229},{},[2230],{"type":48,"value":1461},{"type":42,"tag":374,"props":2232,"children":2233},{},[2234],{"type":48,"value":1466},{"type":42,"tag":390,"props":2236,"children":2237},{},[2238,2259],{"type":42,"tag":370,"props":2239,"children":2240},{},[2241,2249,2254],{"type":42,"tag":397,"props":2242,"children":2243},{},[2244],{"type":42,"tag":67,"props":2245,"children":2246},{},[2247],{"type":48,"value":2248},"StatefulSet split-brain",{"type":42,"tag":397,"props":2250,"children":2251},{},[2252],{"type":48,"value":2253},"Duplicate pod identities across partitioned nodes",{"type":42,"tag":397,"props":2255,"children":2256},{},[2257],{"type":48,"value":2258},"Network partition + eviction timeout race. Two instances of same ordinal running. No fencing by default.",{"type":42,"tag":370,"props":2260,"children":2261},{},[2262,2270,2275],{"type":42,"tag":397,"props":2263,"children":2264},{},[2265],{"type":42,"tag":67,"props":2266,"children":2267},{},[2268],{"type":48,"value":2269},"CoreDNS OOMKill",{"type":42,"tag":397,"props":2271,"children":2272},{},[2273],{"type":48,"value":2274},"CoreDNS restarts + cluster-wide DNS timeouts in app logs",{"type":42,"tag":397,"props":2276,"children":2277},{},[2278],{"type":48,"value":2279},"Default CoreDNS memory (~170Mi) insufficient under query amplification (ndots:5, each external lookup → ~10 lookups).",{"type":42,"tag":355,"props":2281,"children":2283},{"id":2282},"when-classification-is-ambiguous",[2284],{"type":48,"value":2285},"When classification is ambiguous",{"type":42,"tag":51,"props":2287,"children":2288},{},[2289],{"type":48,"value":2290},"Real incidents often match two modes. Examples:",{"type":42,"tag":137,"props":2292,"children":2293},{},[2294,2299,2304],{"type":42,"tag":141,"props":2295,"children":2296},{},[2297],{"type":48,"value":2298},"OOMKilled pod with simultaneous CPU throttling — memory usually drives the kill, but verify by checking whether memory\nor CPU hit limit first.",{"type":42,"tag":141,"props":2300,"children":2301},{},[2302],{"type":48,"value":2303},"Stuck rollout with HPA dampening and resource quota near-exhaustion — both can freeze a deploy. Check which constraint\nis binding.",{"type":42,"tag":141,"props":2305,"children":2306},{},[2307],{"type":48,"value":2308},"Node NotReady with pods that were already crashing — the node issue may be incidental.",{"type":42,"tag":51,"props":2310,"children":2311},{},[2312],{"type":48,"value":2313},"When two modes fit, name both in the synthesis and say which one you believe is causal and why. Do not force a single\nhypothesis when the evidence supports two.",{"type":42,"tag":57,"props":2315,"children":2317},{"id":2316},"signal-interpretation",[2318],{"type":48,"value":2319},"Signal interpretation",{"type":42,"tag":355,"props":2321,"children":2323},{"id":2322},"memory",[2324],{"type":48,"value":2325},"Memory",{"type":42,"tag":137,"props":2327,"children":2328},{},[2329,2363,2373],{"type":42,"tag":141,"props":2330,"children":2331},{},[2332,2337,2339,2345,2347,2353,2355,2361],{"type":42,"tag":67,"props":2333,"children":2334},{},[2335],{"type":48,"value":2336},"Monotonic rise over 30–60 min",{"type":48,"value":2338}," → leak. Check GC metrics for the language: JVM ",{"type":42,"tag":75,"props":2340,"children":2342},{"className":2341},[],[2343],{"type":48,"value":2344},"jvm.gc.duration",{"type":48,"value":2346},", Go\n",{"type":42,"tag":75,"props":2348,"children":2350},{"className":2349},[],[2351],{"type":48,"value":2352},"process.runtime.go.gc.pause_ns",{"type":48,"value":2354},", Node ",{"type":42,"tag":75,"props":2356,"children":2358},{"className":2357},[],[2359],{"type":48,"value":2360},"v8js_gc_duration",{"type":48,"value":2362},". Rising GC frequency\u002Fpause with stable live-set is the\ncanonical leak signature.",{"type":42,"tag":141,"props":2364,"children":2365},{},[2366,2371],{"type":42,"tag":67,"props":2367,"children":2368},{},[2369],{"type":48,"value":2370},"Diurnal \u002F load-correlated spikes",{"type":48,"value":2372}," → load-driven, not leak. Consider HPA tuning or limit increase.",{"type":42,"tag":141,"props":2374,"children":2375},{},[2376,2381],{"type":42,"tag":67,"props":2377,"children":2378},{},[2379],{"type":48,"value":2380},"Hits 1.0, then restart",{"type":48,"value":2382}," → OOMKilled confirmed. Exit code 137 (SIGKILL) in app logs consistent.",{"type":42,"tag":355,"props":2384,"children":2386},{"id":2385},"cpu",[2387],{"type":48,"value":2388},"CPU",{"type":42,"tag":137,"props":2390,"children":2391},{},[2392,2402,2407],{"type":42,"tag":141,"props":2393,"children":2394},{},[2395,2400],{"type":42,"tag":75,"props":2396,"children":2398},{"className":2397},[],[2399],{"type":48,"value":1521},{"type":48,"value":2401}," sustained → CFS throttling. Node has spare CPU; the pod is quota-blocked.",{"type":42,"tag":141,"props":2403,"children":2404},{},[2405],{"type":48,"value":2406},"Symptoms of throttling (not the throttle metric itself): liveness probe timeouts, p99 latency 4–16× p50, queue\nbackpressure upstream, Error-reason container terminations.",{"type":42,"tag":141,"props":2408,"children":2409},{},[2410],{"type":48,"value":2411},"Average can look healthy while p95 is throttled. Do not trust average alone.",{"type":42,"tag":355,"props":2413,"children":2415},{"id":2414},"restart-patterns",[2416],{"type":48,"value":2417},"Restart patterns",{"type":42,"tag":137,"props":2419,"children":2420},{},[2421,2458,2470],{"type":42,"tag":141,"props":2422,"children":2423},{},[2424,2430,2432,2437,2439,2444,2445,2450,2452,2457],{"type":42,"tag":75,"props":2425,"children":2427},{"className":2426},[],[2428],{"type":48,"value":2429},"restarts > 0",{"type":48,"value":2431}," recently → workload has been restarting. Don't read magnitude into the count (see ",{"type":42,"tag":200,"props":2433,"children":2434},{},[2435],{"type":48,"value":2436},"Restart count is\nboolean",{"type":48,"value":2438},"); confirm the pattern from K8s ",{"type":42,"tag":75,"props":2440,"children":2442},{"className":2441},[],[2443],{"type":48,"value":327},{"type":48,"value":329},{"type":42,"tag":75,"props":2446,"children":2448},{"className":2447},[],[2449],{"type":48,"value":335},{"type":48,"value":2451}," event timestamps in ",{"type":42,"tag":75,"props":2453,"children":2455},{"className":2454},[],[2456],{"type":48,"value":96},{"type":48,"value":1162},{"type":42,"tag":141,"props":2459,"children":2460},{},[2461,2463,2468],{"type":48,"value":2462},"Restarts correlated with memory pressure (",{"type":42,"tag":75,"props":2464,"children":2466},{"className":2465},[],[2467],{"type":48,"value":1496},{"type":48,"value":2469},") → OOMKilled path.",{"type":42,"tag":141,"props":2471,"children":2472},{},[2473,2475,2480,2482,2487],{"type":48,"value":2474},"Restarts without memory\u002FCPU pressure → probe misconfig, app bug, or startup dependency failure. Pull events for\n",{"type":42,"tag":75,"props":2476,"children":2478},{"className":2477},[],[2479],{"type":48,"value":1568},{"type":48,"value":2481}," and ",{"type":42,"tag":75,"props":2483,"children":2485},{"className":2484},[],[2486],{"type":48,"value":327},{"type":48,"value":1162},{"type":42,"tag":355,"props":2489,"children":2491},{"id":2490},"termination-reasons",[2492],{"type":48,"value":2493},"Termination reasons",{"type":42,"tag":137,"props":2495,"children":2496},{},[2497,2507,2517,2527],{"type":42,"tag":141,"props":2498,"children":2499},{},[2500,2505],{"type":42,"tag":75,"props":2501,"children":2503},{"className":2502},[],[2504],{"type":48,"value":773},{"type":48,"value":2506}," → memory path.",{"type":42,"tag":141,"props":2508,"children":2509},{},[2510,2515],{"type":42,"tag":75,"props":2511,"children":2513},{"className":2512},[],[2514],{"type":48,"value":780},{"type":48,"value":2516}," → non-zero exit. Check app logs; if empty\u002Fminimal, check CPU throttling before attributing to app logic.",{"type":42,"tag":141,"props":2518,"children":2519},{},[2520,2525],{"type":42,"tag":75,"props":2521,"children":2523},{"className":2522},[],[2524],{"type":48,"value":787},{"type":48,"value":2526}," → ran to completion. Normal for Jobs\u002FCronJobs\u002Finit containers; anomalous otherwise.",{"type":42,"tag":141,"props":2528,"children":2529},{},[2530,2535],{"type":42,"tag":75,"props":2531,"children":2533},{"className":2532},[],[2534],{"type":48,"value":794},{"type":48,"value":2536}," → runtime\u002Fimage\u002Fexec issue. Check image pull events.",{"type":42,"tag":57,"props":2538,"children":2540},{"id":2539},"investigation-flow",[2541],{"type":48,"value":2542},"Investigation flow",{"type":42,"tag":2544,"props":2545,"children":2546},"blockquote",{},[2547],{"type":42,"tag":51,"props":2548,"children":2549},{},[2550,2552,2557,2559,2564],{"type":48,"value":2551},"An investigation is not a checklist. The sections below describe a ",{"type":42,"tag":200,"props":2553,"children":2554},{},[2555],{"type":48,"value":2556},"typical",{"type":48,"value":2558}," arc — ",{"type":42,"tag":67,"props":2560,"children":2561},{},[2562],{"type":48,"value":2563},"compress, skip, or revisit them\nbased on what you find.",{"type":48,"value":2565}," Terminate as soon as you have enough evidence to synthesize at a known confidence. Chasing\nsignals past the point of diminishing returns is a failure mode, not thoroughness.",{"type":42,"tag":355,"props":2567,"children":2569},{"id":2568},"orient",[2570],{"type":48,"value":2571},"Orient",{"type":42,"tag":51,"props":2573,"children":2574},{},[2575,2577,2582,2583,2588,2590,2595,2596,2601],{"type":48,"value":2576},"Resolve the target: ",{"type":42,"tag":75,"props":2578,"children":2580},{"className":2579},[],[2581],{"type":48,"value":111},{"type":48,"value":90},{"type":42,"tag":75,"props":2584,"children":2586},{"className":2585},[],[2587],{"type":48,"value":118},{"type":48,"value":2589},", optionally ",{"type":42,"tag":75,"props":2591,"children":2593},{"className":2592},[],[2594],{"type":48,"value":696},{"type":48,"value":2481},{"type":42,"tag":75,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":48,"value":552},{"type":48,"value":2602},". If no\ntime window is given, default to the last hour for pod-level investigations, last 2 hours for event correlation, last 6\nhours for ongoing\u002Funresolved incidents.",{"type":42,"tag":51,"props":2604,"children":2605},{},[2606,2608,2613],{"type":48,"value":2607},"If the alert payload already tells you the failure mode (e.g., it fires specifically on ",{"type":42,"tag":75,"props":2609,"children":2611},{"className":2610},[],[2612],{"type":48,"value":773},{"type":48,"value":2614},"), note that and skip\nclassification; move to confirmation and baseline comparison.",{"type":42,"tag":355,"props":2616,"children":2618},{"id":2617},"characterize",[2619],{"type":48,"value":2620},"Characterize",{"type":42,"tag":51,"props":2622,"children":2623},{},[2624],{"type":48,"value":2625},"Get the shape of the workload's recent behavior: restart count, termination reasons, phase, utilization. One or two\nqueries usually suffice.",{"type":42,"tag":2627,"props":2628,"children":2633},"pre",{"className":2629,"code":2630,"language":2631,"meta":2632,"style":2632},"language-esql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","FROM metrics-k8sclusterreceiver.otel-*\n| WHERE k8s.pod.name == \"\u003Cpod>\" AND k8s.namespace.name == \"\u003Cns>\"\n  AND @timestamp > NOW() - 1 hour\n| STATS restarts = MAX(k8s.container.restarts),\n        term_reasons = VALUES(k8s.container.status.last_terminated_reason),\n        phase = MAX(k8s.pod.phase)\n","esql","",[2634],{"type":42,"tag":75,"props":2635,"children":2636},{"__ignoreMap":2632},[2637,2648,2657,2666,2675,2684],{"type":42,"tag":2638,"props":2639,"children":2642},"span",{"class":2640,"line":2641},"line",1,[2643],{"type":42,"tag":2638,"props":2644,"children":2645},{},[2646],{"type":48,"value":2647},"FROM metrics-k8sclusterreceiver.otel-*\n",{"type":42,"tag":2638,"props":2649,"children":2651},{"class":2640,"line":2650},2,[2652],{"type":42,"tag":2638,"props":2653,"children":2654},{},[2655],{"type":48,"value":2656},"| WHERE k8s.pod.name == \"\u003Cpod>\" AND k8s.namespace.name == \"\u003Cns>\"\n",{"type":42,"tag":2638,"props":2658,"children":2660},{"class":2640,"line":2659},3,[2661],{"type":42,"tag":2638,"props":2662,"children":2663},{},[2664],{"type":48,"value":2665},"  AND @timestamp > NOW() - 1 hour\n",{"type":42,"tag":2638,"props":2667,"children":2669},{"class":2640,"line":2668},4,[2670],{"type":42,"tag":2638,"props":2671,"children":2672},{},[2673],{"type":48,"value":2674},"| STATS restarts = MAX(k8s.container.restarts),\n",{"type":42,"tag":2638,"props":2676,"children":2678},{"class":2640,"line":2677},5,[2679],{"type":42,"tag":2638,"props":2680,"children":2681},{},[2682],{"type":48,"value":2683},"        term_reasons = VALUES(k8s.container.status.last_terminated_reason),\n",{"type":42,"tag":2638,"props":2685,"children":2687},{"class":2640,"line":2686},6,[2688],{"type":42,"tag":2638,"props":2689,"children":2690},{},[2691],{"type":48,"value":2692},"        phase = MAX(k8s.pod.phase)\n",{"type":42,"tag":2627,"props":2694,"children":2696},{"className":2629,"code":2695,"language":2631,"meta":2632,"style":2632},"FROM metrics-kubeletstatsreceiver.otel-*\n| WHERE k8s.pod.name == \"\u003Cpod>\" AND @timestamp > NOW() - 15 minutes\n| STATS mem_pct = ROUND(MAX(k8s.pod.memory_limit_utilization) * 100, 1),\n        cpu_pct = ROUND(MAX(k8s.pod.cpu_limit_utilization) * 100, 1)\n",[2697],{"type":42,"tag":75,"props":2698,"children":2699},{"__ignoreMap":2632},[2700,2708,2716,2724],{"type":42,"tag":2638,"props":2701,"children":2702},{"class":2640,"line":2641},[2703],{"type":42,"tag":2638,"props":2704,"children":2705},{},[2706],{"type":48,"value":2707},"FROM metrics-kubeletstatsreceiver.otel-*\n",{"type":42,"tag":2638,"props":2709,"children":2710},{"class":2640,"line":2650},[2711],{"type":42,"tag":2638,"props":2712,"children":2713},{},[2714],{"type":48,"value":2715},"| WHERE k8s.pod.name == \"\u003Cpod>\" AND @timestamp > NOW() - 15 minutes\n",{"type":42,"tag":2638,"props":2717,"children":2718},{"class":2640,"line":2659},[2719],{"type":42,"tag":2638,"props":2720,"children":2721},{},[2722],{"type":48,"value":2723},"| STATS mem_pct = ROUND(MAX(k8s.pod.memory_limit_utilization) * 100, 1),\n",{"type":42,"tag":2638,"props":2725,"children":2726},{"class":2640,"line":2668},[2727],{"type":42,"tag":2638,"props":2728,"children":2729},{},[2730],{"type":48,"value":2731},"        cpu_pct = ROUND(MAX(k8s.pod.cpu_limit_utilization) * 100, 1)\n",{"type":42,"tag":355,"props":2733,"children":2735},{"id":2734},"classify",[2736],{"type":48,"value":2737},"Classify",{"type":42,"tag":51,"props":2739,"children":2740},{},[2741],{"type":48,"value":2742},"Use the taxonomy. The pivotal signal should match; the \"Investigate\" column tells you what corroboration to seek.",{"type":42,"tag":51,"props":2744,"children":2745},{},[2746],{"type":48,"value":2747},"When two modes fit, note both and proceed with the one that has the stronger pivotal signal. You may revise during\ncorroboration.",{"type":42,"tag":355,"props":2749,"children":2751},{"id":2750},"corroborate",[2752],{"type":48,"value":2753},"Corroborate",{"type":42,"tag":51,"props":2755,"children":2756},{},[2757],{"type":48,"value":2758},"Pull the evidence your classification predicts you'll find. Typical sources:",{"type":42,"tag":51,"props":2760,"children":2761},{},[2762,2766],{"type":42,"tag":67,"props":2763,"children":2764},{},[2765],{"type":48,"value":443},{"type":48,"value":2767}," for the namespace and window:",{"type":42,"tag":2627,"props":2769,"children":2771},{"className":2629,"code":2770,"language":2631,"meta":2632,"style":2632},"FROM logs-k8seventsreceiver.otel-*\n| WHERE k8s.namespace.name == \"\u003Cns>\"\n  AND @timestamp > NOW() - 2 hours\n  AND attributes.k8s.event.reason IN (\n    \"BackOff\", \"Killing\", \"Unhealthy\", \"Failed\",\n    \"FailedScheduling\", \"Evicted\", \"SuccessfulRescale\",\n    \"Pulling\", \"Pulled\", \"Started\", \"Created\"\n  )\n| SORT @timestamp DESC\n| KEEP @timestamp, attributes.k8s.event.reason, body.text, k8s.object.name\n| LIMIT 30\n",[2772],{"type":42,"tag":75,"props":2773,"children":2774},{"__ignoreMap":2632},[2775,2783,2791,2799,2807,2815,2823,2832,2841,2850,2859],{"type":42,"tag":2638,"props":2776,"children":2777},{"class":2640,"line":2641},[2778],{"type":42,"tag":2638,"props":2779,"children":2780},{},[2781],{"type":48,"value":2782},"FROM logs-k8seventsreceiver.otel-*\n",{"type":42,"tag":2638,"props":2784,"children":2785},{"class":2640,"line":2650},[2786],{"type":42,"tag":2638,"props":2787,"children":2788},{},[2789],{"type":48,"value":2790},"| WHERE k8s.namespace.name == \"\u003Cns>\"\n",{"type":42,"tag":2638,"props":2792,"children":2793},{"class":2640,"line":2659},[2794],{"type":42,"tag":2638,"props":2795,"children":2796},{},[2797],{"type":48,"value":2798},"  AND @timestamp > NOW() - 2 hours\n",{"type":42,"tag":2638,"props":2800,"children":2801},{"class":2640,"line":2668},[2802],{"type":42,"tag":2638,"props":2803,"children":2804},{},[2805],{"type":48,"value":2806},"  AND attributes.k8s.event.reason IN (\n",{"type":42,"tag":2638,"props":2808,"children":2809},{"class":2640,"line":2677},[2810],{"type":42,"tag":2638,"props":2811,"children":2812},{},[2813],{"type":48,"value":2814},"    \"BackOff\", \"Killing\", \"Unhealthy\", \"Failed\",\n",{"type":42,"tag":2638,"props":2816,"children":2817},{"class":2640,"line":2686},[2818],{"type":42,"tag":2638,"props":2819,"children":2820},{},[2821],{"type":48,"value":2822},"    \"FailedScheduling\", \"Evicted\", \"SuccessfulRescale\",\n",{"type":42,"tag":2638,"props":2824,"children":2826},{"class":2640,"line":2825},7,[2827],{"type":42,"tag":2638,"props":2828,"children":2829},{},[2830],{"type":48,"value":2831},"    \"Pulling\", \"Pulled\", \"Started\", \"Created\"\n",{"type":42,"tag":2638,"props":2833,"children":2835},{"class":2640,"line":2834},8,[2836],{"type":42,"tag":2638,"props":2837,"children":2838},{},[2839],{"type":48,"value":2840},"  )\n",{"type":42,"tag":2638,"props":2842,"children":2844},{"class":2640,"line":2843},9,[2845],{"type":42,"tag":2638,"props":2846,"children":2847},{},[2848],{"type":48,"value":2849},"| SORT @timestamp DESC\n",{"type":42,"tag":2638,"props":2851,"children":2853},{"class":2640,"line":2852},10,[2854],{"type":42,"tag":2638,"props":2855,"children":2856},{},[2857],{"type":48,"value":2858},"| KEEP @timestamp, attributes.k8s.event.reason, body.text, k8s.object.name\n",{"type":42,"tag":2638,"props":2860,"children":2862},{"class":2640,"line":2861},11,[2863],{"type":42,"tag":2638,"props":2864,"children":2865},{},[2866],{"type":48,"value":2867},"| LIMIT 30\n",{"type":42,"tag":51,"props":2869,"children":2870},{},[2871,2875,2877,2882],{"type":42,"tag":67,"props":2872,"children":2873},{},[2874],{"type":48,"value":485},{"type":48,"value":2876}," if available — look at the 200 most recent lines before the termination timestamp. If absent, flag\n",{"type":42,"tag":75,"props":2878,"children":2880},{"className":2879},[],[2881],{"type":48,"value":212},{"type":48,"value":2883},"; do not invent a log pattern.",{"type":42,"tag":51,"props":2885,"children":2886},{},[2887,2891,2893,2898],{"type":42,"tag":67,"props":2888,"children":2889},{},[2890],{"type":48,"value":525},{"type":48,"value":2892}," if the pod runs an instrumented service — resolve ",{"type":42,"tag":75,"props":2894,"children":2896},{"className":2895},[],[2897],{"type":48,"value":552},{"type":48,"value":2899}," from pod resource attributes for later\ncorrelation. SLO \u002F latency \u002F error-rate analysis itself is APM-layer work and out of scope for this skill.",{"type":42,"tag":51,"props":2901,"children":2902},{},[2903,2908],{"type":42,"tag":67,"props":2904,"children":2905},{},[2906],{"type":48,"value":2907},"Baseline comparison",{"type":48,"value":2909}," — for utilization-based findings, compare current values to 7-day-prior at the same hour-of-day.\n\"High memory\" is meaningful only relative to what's normal for this workload.",{"type":42,"tag":355,"props":2911,"children":2913},{"id":2912},"check-for-upstream-cause-conditional",[2914],{"type":48,"value":2915},"Check for upstream cause (conditional)",{"type":42,"tag":51,"props":2917,"children":2918},{},[2919,2921,2926],{"type":48,"value":2920},"Only pursue if the symptom pattern suggests it. Threshold: upstream error rate >5× baseline ",{"type":42,"tag":200,"props":2922,"children":2923},{},[2924],{"type":48,"value":2925},"or",{"type":48,"value":2927}," latency >3× baseline,\nAND degradation started before the symptom on the target service. Co-symptoms do not establish causation.",{"type":42,"tag":51,"props":2929,"children":2930},{},[2931,2933,2939,2941,2946],{"type":48,"value":2932},"If ",{"type":42,"tag":75,"props":2934,"children":2936},{"className":2935},[],[2937],{"type":48,"value":2938},"metrics-service_destination.1m.otel-default",{"type":48,"value":2940}," has no rows for the service, report ",{"type":42,"tag":75,"props":2942,"children":2944},{"className":2943},[],[2945],{"type":48,"value":230},{"type":48,"value":2947}," —\nnot \"upstreams healthy.\"",{"type":42,"tag":355,"props":2949,"children":2951},{"id":2950},"check-for-recent-change-conditional",[2952],{"type":48,"value":2953},"Check for recent change (conditional)",{"type":42,"tag":51,"props":2955,"children":2956},{},[2957,2963,2964,2969,2971,2976],{"type":42,"tag":75,"props":2958,"children":2960},{"className":2959},[],[2961],{"type":48,"value":2962},"SuccessfulCreate",{"type":48,"value":329},{"type":42,"tag":75,"props":2965,"children":2967},{"className":2966},[],[2968],{"type":48,"value":1680},{"type":48,"value":2970}," events in the last 2 hours often correlate with deploys. ",{"type":42,"tag":75,"props":2972,"children":2974},{"className":2973},[],[2975],{"type":48,"value":103},{"type":48,"value":2977},"\nshows configmap\u002Fsecret\u002Fdeployment spec changes. A change within 15 minutes of the symptom onset is a strong correlation,\nbut still a correlation — verify it plausibly explains the mode you've classified.",{"type":42,"tag":355,"props":2979,"children":2981},{"id":2980},"synthesize-and-stop",[2982],{"type":48,"value":2983},"Synthesize and stop",{"type":42,"tag":51,"props":2985,"children":2986},{},[2987],{"type":48,"value":2988},"Synthesize as soon as you have enough evidence to support a hypothesis at known confidence. You do not need to complete\nevery section above — investigation terminates when either:",{"type":42,"tag":137,"props":2990,"children":2991},{},[2992,2997],{"type":42,"tag":141,"props":2993,"children":2994},{},[2995],{"type":48,"value":2996},"You have a high-confidence hypothesis with corroboration, or",{"type":42,"tag":141,"props":2998,"children":2999},{},[3000],{"type":48,"value":3001},"You have a low\u002Fmedium-confidence hypothesis and further queries are unlikely to change the picture (e.g., logs are\nunavailable, APM isn't instrumented, no recent changes found).",{"type":42,"tag":57,"props":3003,"children":3005},{"id":3004},"synthesis",[3006],{"type":48,"value":3007},"Synthesis",{"type":42,"tag":51,"props":3009,"children":3010},{},[3011],{"type":48,"value":3012},"Default structure:",{"type":42,"tag":2627,"props":3014,"children":3018},{"className":3015,"code":3017,"language":48,"meta":2632},[3016],"language-text","HYPOTHESIS (confidence: high | medium | low)\n\u003COne paragraph: service, symptom, most likely cause. Name the failure mode from the taxonomy.>\n\nEVIDENCE\n- \u003CFinding from characterization, with the concrete metric or value.>\n- \u003CFinding from events \u002F logs \u002F APM.>\n- \u003CFinding from baseline comparison, dependency check, or change correlation if pursued.>\n\nCONFIDENCE NOTE\n\u003COnly if not 'high'. What specific evidence is missing or ambiguous.>\n\nRECOMMENDED NEXT STEPS\n1. \u003CMost actionable — typically a config check or metric to observe.>\n2. \u003CSecondary.>\n\nDOWNSTREAM IMPACT\n\u003CServices depending on this workload, or 'No downstream dependencies identified.'>\n",[3019],{"type":42,"tag":75,"props":3020,"children":3021},{"__ignoreMap":2632},[3022],{"type":48,"value":3017},{"type":42,"tag":51,"props":3024,"children":3025},{},[3026,3031],{"type":42,"tag":67,"props":3027,"children":3028},{},[3029],{"type":48,"value":3030},"When two hypotheses are live:",{"type":48,"value":3032}," replace HYPOTHESIS with COMPETING HYPOTHESES; list both, say which you lean toward and\nwhy, and list the evidence that would disambiguate them.",{"type":42,"tag":51,"props":3034,"children":3035},{},[3036,3041,3043,3049],{"type":42,"tag":67,"props":3037,"children":3038},{},[3039],{"type":48,"value":3040},"When no incident is found",{"type":48,"value":3042}," (symptom resolved, or alert appears spurious): say so directly.\n",{"type":42,"tag":75,"props":3044,"children":3046},{"className":3045},[],[3047],{"type":48,"value":3048},"ALERT FIRED BUT SYSTEM APPEARS HEALTHY",{"type":48,"value":3050}," is a valid output. List what you checked and what you didn't find.",{"type":42,"tag":355,"props":3052,"children":3054},{"id":3053},"confidence-calibration",[3055],{"type":48,"value":3056},"Confidence calibration",{"type":42,"tag":51,"props":3058,"children":3059},{},[3060,3062,3067],{"type":48,"value":3061},"Start at ",{"type":42,"tag":67,"props":3063,"children":3064},{},[3065],{"type":48,"value":3066},"high",{"type":48,"value":3068}," and downgrade based on what's missing:",{"type":42,"tag":137,"props":3070,"children":3071},{},[3072,3084],{"type":42,"tag":141,"props":3073,"children":3074},{},[3075,3077,3082],{"type":48,"value":3076},"Downgrade to ",{"type":42,"tag":67,"props":3078,"children":3079},{},[3080],{"type":48,"value":3081},"medium",{"type":48,"value":3083}," if: primary signal is clear but corroboration is missing (no logs, no APM, no baseline\ncomparison possible). Or: two modes fit and you can't disambiguate.",{"type":42,"tag":141,"props":3085,"children":3086},{},[3087,3088,3093],{"type":48,"value":3076},{"type":42,"tag":67,"props":3089,"children":3090},{},[3091],{"type":48,"value":3092},"low",{"type":48,"value":3094}," if: only a single signal supports the hypothesis, signals conflict, or the mode requires evidence\nyou couldn't fetch.",{"type":42,"tag":51,"props":3096,"children":3097},{},[3098,3100,3104],{"type":48,"value":3099},"Never return ",{"type":42,"tag":67,"props":3101,"children":3102},{},[3103],{"type":48,"value":3066},{"type":48,"value":3105}," when application log data was absent and the hypothesis depends on application behavior. Absence\nof evidence does not corroborate a hypothesis.",{"type":42,"tag":57,"props":3107,"children":3109},{"id":3108},"query-recipes",[3110],{"type":48,"value":3111},"Query recipes",{"type":42,"tag":355,"props":3113,"children":3115},{"id":3114},"most-restarting-pods-in-a-namespace",[3116],{"type":48,"value":3117},"Most-restarting pods in a namespace",{"type":42,"tag":2627,"props":3119,"children":3121},{"className":2629,"code":3120,"language":2631,"meta":2632,"style":2632},"FROM metrics-k8sclusterreceiver.otel-*\n| WHERE k8s.namespace.name == \"\u003Cns>\" AND @timestamp > NOW() - 1 hour\n| STATS restarts = MAX(k8s.container.restarts) BY k8s.pod.name, k8s.container.status.last_terminated_reason\n| WHERE restarts > 0\n| SORT restarts DESC\n| LIMIT 20\n",[3122],{"type":42,"tag":75,"props":3123,"children":3124},{"__ignoreMap":2632},[3125,3132,3140,3148,3156,3164],{"type":42,"tag":2638,"props":3126,"children":3127},{"class":2640,"line":2641},[3128],{"type":42,"tag":2638,"props":3129,"children":3130},{},[3131],{"type":48,"value":2647},{"type":42,"tag":2638,"props":3133,"children":3134},{"class":2640,"line":2650},[3135],{"type":42,"tag":2638,"props":3136,"children":3137},{},[3138],{"type":48,"value":3139},"| WHERE k8s.namespace.name == \"\u003Cns>\" AND @timestamp > NOW() - 1 hour\n",{"type":42,"tag":2638,"props":3141,"children":3142},{"class":2640,"line":2659},[3143],{"type":42,"tag":2638,"props":3144,"children":3145},{},[3146],{"type":48,"value":3147},"| STATS restarts = MAX(k8s.container.restarts) BY k8s.pod.name, k8s.container.status.last_terminated_reason\n",{"type":42,"tag":2638,"props":3149,"children":3150},{"class":2640,"line":2668},[3151],{"type":42,"tag":2638,"props":3152,"children":3153},{},[3154],{"type":48,"value":3155},"| WHERE restarts > 0\n",{"type":42,"tag":2638,"props":3157,"children":3158},{"class":2640,"line":2677},[3159],{"type":42,"tag":2638,"props":3160,"children":3161},{},[3162],{"type":48,"value":3163},"| SORT restarts DESC\n",{"type":42,"tag":2638,"props":3165,"children":3166},{"class":2640,"line":2686},[3167],{"type":42,"tag":2638,"props":3168,"children":3169},{},[3170],{"type":48,"value":3171},"| LIMIT 20\n",{"type":42,"tag":355,"props":3173,"children":3175},{"id":3174},"cpu-throttling-check-for-a-pod",[3176],{"type":48,"value":3177},"CPU throttling check for a pod",{"type":42,"tag":2627,"props":3179,"children":3181},{"className":2629,"code":3180,"language":2631,"meta":2632,"style":2632},"FROM metrics-kubeletstatsreceiver.otel-*\n| WHERE k8s.pod.name == \"\u003Cpod>\" AND @timestamp > NOW() - 30 minutes\n| STATS max_cpu_ratio = ROUND(MAX(k8s.pod.cpu_limit_utilization), 2),\n        avg_cpu_ratio = ROUND(AVG(k8s.pod.cpu_limit_utilization), 2),\n        max_cpu_cores = ROUND(MAX(k8s.pod.cpu.usage), 3)\n",[3182],{"type":42,"tag":75,"props":3183,"children":3184},{"__ignoreMap":2632},[3185,3192,3200,3208,3216],{"type":42,"tag":2638,"props":3186,"children":3187},{"class":2640,"line":2641},[3188],{"type":42,"tag":2638,"props":3189,"children":3190},{},[3191],{"type":48,"value":2707},{"type":42,"tag":2638,"props":3193,"children":3194},{"class":2640,"line":2650},[3195],{"type":42,"tag":2638,"props":3196,"children":3197},{},[3198],{"type":48,"value":3199},"| WHERE k8s.pod.name == \"\u003Cpod>\" AND @timestamp > NOW() - 30 minutes\n",{"type":42,"tag":2638,"props":3201,"children":3202},{"class":2640,"line":2659},[3203],{"type":42,"tag":2638,"props":3204,"children":3205},{},[3206],{"type":48,"value":3207},"| STATS max_cpu_ratio = ROUND(MAX(k8s.pod.cpu_limit_utilization), 2),\n",{"type":42,"tag":2638,"props":3209,"children":3210},{"class":2640,"line":2668},[3211],{"type":42,"tag":2638,"props":3212,"children":3213},{},[3214],{"type":48,"value":3215},"        avg_cpu_ratio = ROUND(AVG(k8s.pod.cpu_limit_utilization), 2),\n",{"type":42,"tag":2638,"props":3217,"children":3218},{"class":2640,"line":2677},[3219],{"type":42,"tag":2638,"props":3220,"children":3221},{},[3222],{"type":48,"value":3223},"        max_cpu_cores = ROUND(MAX(k8s.pod.cpu.usage), 3)\n",{"type":42,"tag":51,"props":3225,"children":3226},{},[3227],{"type":48,"value":3228},"Sustained ratio >1.0 = throttling. Transient >1.0 with avg \u003C0.5 is usually benign burst.",{"type":42,"tag":355,"props":3230,"children":3232},{"id":3231},"nodes-under-memory-pressure-right-now",[3233],{"type":48,"value":3234},"Nodes under memory pressure (right now)",{"type":42,"tag":2627,"props":3236,"children":3238},{"className":2629,"code":3237,"language":2631,"meta":2632,"style":2632},"FROM metrics-k8sclusterreceiver.otel-*\n| WHERE @timestamp > NOW() - 15 minutes AND k8s.node.condition_memory_pressure == 1\n| STATS ts = MAX(@timestamp) BY k8s.node.name\n| SORT ts DESC\n",[3239],{"type":42,"tag":75,"props":3240,"children":3241},{"__ignoreMap":2632},[3242,3249,3257,3265],{"type":42,"tag":2638,"props":3243,"children":3244},{"class":2640,"line":2641},[3245],{"type":42,"tag":2638,"props":3246,"children":3247},{},[3248],{"type":48,"value":2647},{"type":42,"tag":2638,"props":3250,"children":3251},{"class":2640,"line":2650},[3252],{"type":42,"tag":2638,"props":3253,"children":3254},{},[3255],{"type":48,"value":3256},"| WHERE @timestamp > NOW() - 15 minutes AND k8s.node.condition_memory_pressure == 1\n",{"type":42,"tag":2638,"props":3258,"children":3259},{"class":2640,"line":2659},[3260],{"type":42,"tag":2638,"props":3261,"children":3262},{},[3263],{"type":48,"value":3264},"| STATS ts = MAX(@timestamp) BY k8s.node.name\n",{"type":42,"tag":2638,"props":3266,"children":3267},{"class":2640,"line":2668},[3268],{"type":42,"tag":2638,"props":3269,"children":3270},{},[3271],{"type":48,"value":3272},"| SORT ts DESC\n",{"type":42,"tag":355,"props":3274,"children":3276},{"id":3275},"admission-denials-webhook-or-quota-last-hour",[3277],{"type":48,"value":3278},"Admission denials (webhook or quota) last hour",{"type":42,"tag":2627,"props":3280,"children":3282},{"className":2629,"code":3281,"language":2631,"meta":2632,"style":2632},"FROM logs-k8seventsreceiver.otel-*\n| WHERE @timestamp > NOW() - 1 hour\n  AND (attributes.k8s.event.reason == \"FailedCreate\"\n       OR body.text LIKE \"*admission webhook*\"\n       OR body.text LIKE \"*exceeded quota*\")\n| SORT @timestamp DESC\n| KEEP @timestamp, k8s.namespace.name, attributes.k8s.event.reason, body.text\n| LIMIT 30\n",[3283],{"type":42,"tag":75,"props":3284,"children":3285},{"__ignoreMap":2632},[3286,3293,3301,3309,3317,3325,3332,3340],{"type":42,"tag":2638,"props":3287,"children":3288},{"class":2640,"line":2641},[3289],{"type":42,"tag":2638,"props":3290,"children":3291},{},[3292],{"type":48,"value":2782},{"type":42,"tag":2638,"props":3294,"children":3295},{"class":2640,"line":2650},[3296],{"type":42,"tag":2638,"props":3297,"children":3298},{},[3299],{"type":48,"value":3300},"| WHERE @timestamp > NOW() - 1 hour\n",{"type":42,"tag":2638,"props":3302,"children":3303},{"class":2640,"line":2659},[3304],{"type":42,"tag":2638,"props":3305,"children":3306},{},[3307],{"type":48,"value":3308},"  AND (attributes.k8s.event.reason == \"FailedCreate\"\n",{"type":42,"tag":2638,"props":3310,"children":3311},{"class":2640,"line":2668},[3312],{"type":42,"tag":2638,"props":3313,"children":3314},{},[3315],{"type":48,"value":3316},"       OR body.text LIKE \"*admission webhook*\"\n",{"type":42,"tag":2638,"props":3318,"children":3319},{"class":2640,"line":2677},[3320],{"type":42,"tag":2638,"props":3321,"children":3322},{},[3323],{"type":48,"value":3324},"       OR body.text LIKE \"*exceeded quota*\")\n",{"type":42,"tag":2638,"props":3326,"children":3327},{"class":2640,"line":2686},[3328],{"type":42,"tag":2638,"props":3329,"children":3330},{},[3331],{"type":48,"value":2849},{"type":42,"tag":2638,"props":3333,"children":3334},{"class":2640,"line":2825},[3335],{"type":42,"tag":2638,"props":3336,"children":3337},{},[3338],{"type":48,"value":3339},"| KEEP @timestamp, k8s.namespace.name, attributes.k8s.event.reason, body.text\n",{"type":42,"tag":2638,"props":3341,"children":3342},{"class":2640,"line":2834},[3343],{"type":42,"tag":2638,"props":3344,"children":3345},{},[3346],{"type":48,"value":2867},{"type":42,"tag":355,"props":3348,"children":3350},{"id":3349},"firing-k8s-alerts",[3351],{"type":48,"value":3352},"Firing K8s alerts",{"type":42,"tag":2627,"props":3354,"children":3357},{"className":3355,"code":3356,"language":48,"meta":2632},[3016],"GET \u002Fapi\u002Falerting\u002Frules\u002F_find?search=k8s&search_fields=tags&filter=alert.attributes.executionStatus.status:active\n",[3358],{"type":42,"tag":75,"props":3359,"children":3360},{"__ignoreMap":2632},[3361],{"type":48,"value":3356},{"type":42,"tag":57,"props":3363,"children":3365},{"id":3364},"examples",[3366],{"type":48,"value":3367},"Examples",{"type":42,"tag":355,"props":3369,"children":3371},{"id":3370},"why-is-my-pod-crashloopbackoff-ing",[3372],{"type":48,"value":3373},"\"Why is my pod CrashLoopBackOff-ing?\"",{"type":42,"tag":51,"props":3375,"children":3376},{},[3377],{"type":48,"value":3378},"Characterize first: get restart count, termination reason, memory and CPU utilization.",{"type":42,"tag":137,"props":3380,"children":3381},{},[3382,3393,3416,3427],{"type":42,"tag":141,"props":3383,"children":3384},{},[3385,3386,3391],{"type":48,"value":2932},{"type":42,"tag":75,"props":3387,"children":3389},{"className":3388},[],[3390],{"type":48,"value":1488},{"type":48,"value":3392}," and memory utilization hit 1.0 → memory path. Corroborate with 7-day\nbaseline: monotonic rise over days = leak; spiky = load-driven. Check GC metrics if language is known.",{"type":42,"tag":141,"props":3394,"children":3395},{},[3396,3397,3402,3403,3408,3410,3415],{"type":48,"value":2932},{"type":42,"tag":75,"props":3398,"children":3400},{"className":3399},[],[3401],{"type":48,"value":1528},{"type":48,"value":2481},{"type":42,"tag":75,"props":3404,"children":3406},{"className":3405},[],[3407],{"type":48,"value":1521},{"type":48,"value":3409}," → CPU throttling path. Corroborate with\nliveness probe config (initialDelaySeconds, timeoutSeconds) and K8s events for ",{"type":42,"tag":75,"props":3411,"children":3413},{"className":3412},[],[3414],{"type":48,"value":1568},{"type":48,"value":1162},{"type":42,"tag":141,"props":3417,"children":3418},{},[3419,3420,3425],{"type":48,"value":2932},{"type":42,"tag":75,"props":3421,"children":3423},{"className":3422},[],[3424],{"type":48,"value":1528},{"type":48,"value":3426}," and CPU is fine → application-logic path. Pull recent logs before termination.",{"type":42,"tag":141,"props":3428,"children":3429},{},[3430,3431,3437,3439,3444],{"type":48,"value":2932},{"type":42,"tag":75,"props":3432,"children":3434},{"className":3433},[],[3435],{"type":48,"value":3436},"last_terminated_reason == \"ContainerCannotRun\"",{"type":48,"value":3438}," → image\u002Fexec path. Check K8s events for ",{"type":42,"tag":75,"props":3440,"children":3442},{"className":3441},[],[3443],{"type":48,"value":1645},{"type":48,"value":3445}," pull events.",{"type":42,"tag":51,"props":3447,"children":3448},{},[3449],{"type":48,"value":3450},"Synthesize with appropriate confidence. If logs were unavailable on the Error path, downgrade to medium and say so.",{"type":42,"tag":355,"props":3452,"children":3454},{"id":3453},"is-my-rollout-stuck",[3455],{"type":48,"value":3456},"\"Is my rollout stuck?\"",{"type":42,"tag":51,"props":3458,"children":3459},{},[3460,3462,3468],{"type":48,"value":3461},"Authoritative signal: ",{"type":42,"tag":75,"props":3463,"children":3465},{"className":3464},[],[3466],{"type":48,"value":3467},"k8s.deployment.available \u003C k8s.deployment.desired",{"type":48,"value":3469}," for > 10 minutes.",{"type":42,"tag":51,"props":3471,"children":3472},{},[3473],{"type":48,"value":3474},"Diagnose the constraint:",{"type":42,"tag":137,"props":3476,"children":3477},{},[3478,3497,3502],{"type":42,"tag":141,"props":3479,"children":3480},{},[3481,3483,3488,3490,3495],{"type":48,"value":3482},"K8s events on the new ReplicaSet: ",{"type":42,"tag":75,"props":3484,"children":3486},{"className":3485},[],[3487],{"type":48,"value":2000},{"type":48,"value":3489}," → admission rejection (quota, webhook, PSP). ",{"type":42,"tag":75,"props":3491,"children":3493},{"className":3492},[],[3494],{"type":48,"value":1911},{"type":48,"value":3496}," → no\nnode fits.",{"type":42,"tag":141,"props":3498,"children":3499},{},[3500],{"type":48,"value":3501},"New-pod utilization: all at 0% memory → never started (image pull failure); high CPU with low memory → slow startup\nhitting readiness probe.",{"type":42,"tag":141,"props":3503,"children":3504},{},[3505,3507,3513],{"type":48,"value":3506},"HPA state: stable ",{"type":42,"tag":75,"props":3508,"children":3510},{"className":3509},[],[3511],{"type":48,"value":3512},"current_replicas \u003C desired_replicas",{"type":48,"value":3514}," under load → unready-pod dampening.",{"type":42,"tag":355,"props":3516,"children":3518},{"id":3517},"alert-fired-but-everything-looks-healthy",[3519],{"type":48,"value":3520},"\"Alert fired but everything looks healthy\"",{"type":42,"tag":51,"props":3522,"children":3523},{},[3524],{"type":48,"value":3525},"Possible and worth naming explicitly. Check:",{"type":42,"tag":137,"props":3527,"children":3528},{},[3529,3534,3539],{"type":42,"tag":141,"props":3530,"children":3531},{},[3532],{"type":48,"value":3533},"Has the symptom resolved? Compare current utilization\u002Frestart rate to the alert trigger point.",{"type":42,"tag":141,"props":3535,"children":3536},{},[3537],{"type":48,"value":3538},"Was the alert a transient spike that's already decayed?",{"type":42,"tag":141,"props":3540,"children":3541},{},[3542],{"type":48,"value":3543},"Is the alert tuned appropriately (e.g., too-short evaluation window)?",{"type":42,"tag":51,"props":3545,"children":3546},{},[3547,3549,3554],{"type":48,"value":3548},"Output: ",{"type":42,"tag":75,"props":3550,"children":3552},{"className":3551},[],[3553],{"type":48,"value":3048},{"type":48,"value":3555}," with what you checked. Recommend alert tuning if the pattern is\nrecurrent.",{"type":42,"tag":57,"props":3557,"children":3559},{"id":3558},"related",[3560],{"type":48,"value":3561},"Related",{"type":42,"tag":137,"props":3563,"children":3564},{},[3565,3582],{"type":42,"tag":141,"props":3566,"children":3567},{},[3568,3573,3574,3580],{"type":42,"tag":67,"props":3569,"children":3570},{},[3571],{"type":48,"value":3572},"Workflow:",{"type":48,"value":298},{"type":42,"tag":75,"props":3575,"children":3577},{"className":3576},[],[3578],{"type":48,"value":3579},"K8s CrashLoopBackOff Investigation",{"type":48,"value":3581}," — alert-triggered automated version of the pod-level path above.\nRuns deterministic ESQL + branches; this skill provides the interpretation layer the workflow lacks.",{"type":42,"tag":141,"props":3583,"children":3584},{},[3585,3590],{"type":42,"tag":67,"props":3586,"children":3587},{},[3588],{"type":48,"value":3589},"Forge genome library:",{"type":48,"value":3591}," 16 K8s failure scenarios (OOMKill cascade, CPU throttling, probe misconfig, node NotReady,\nadmission webhook block, etc.) validating this skill's coverage.",{"type":42,"tag":3593,"props":3594,"children":3595},"style",{},[3596],{"type":48,"value":3597},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3599,"total":3692},[3600,3617,3631,3641,3654,3666,3681],{"slug":3601,"name":3601,"fn":3602,"description":3603,"org":3604,"tags":3605,"stars":23,"repoUrl":24,"updatedAt":3616},"cloud-access-management","manage Elastic Cloud organization access","Manage Elastic Cloud organization access: invite users, assign roles to Serverless projects, and create or revoke Cloud API keys. Use when granting, modifying, or auditing user access.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3606,3609,3610,3613],{"name":3607,"slug":3608,"type":15},"Cloud","cloud",{"name":9,"slug":8,"type":15},{"name":3611,"slug":3612,"type":15},"Operations","operations",{"name":3614,"slug":3615,"type":15},"Permissions","permissions","2026-07-12T07:46:44.946285",{"slug":3618,"name":3618,"fn":3619,"description":3620,"org":3621,"tags":3622,"stars":23,"repoUrl":24,"updatedAt":3630},"cloud-create-project","create Elastic Cloud Serverless projects","Creates Elastic Cloud Serverless projects (Elasticsearch, Observability, or Security) via the REST API, saves credentials to file, and bootstraps a scoped Elasticsearch API key. Use when creating a new serverless project, provisioning a search or observability environment, or spinning up a new Elastic Cloud project.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3623,3624,3627],{"name":3607,"slug":3608,"type":15},{"name":3625,"slug":3626,"type":15},"Deployment","deployment",{"name":3628,"slug":3629,"type":15},"Elasticsearch","elasticsearch","2026-07-12T07:46:42.353362",{"slug":3632,"name":3632,"fn":3633,"description":3634,"org":3635,"tags":3636,"stars":23,"repoUrl":24,"updatedAt":3640},"cloud-manage-project","manage Elastic Cloud Serverless projects","Manages existing Elastic Cloud Serverless projects: list, get, update, delete, reset credentials, resume, and load saved credentials. Connects to existing projects by resolving endpoints and acquiring scoped Elasticsearch API keys. Use when performing day-2 operations on serverless projects, connecting to an existing project, loading or resetting project credentials, or looking up project details.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3637,3638,3639],{"name":3607,"slug":3608,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3611,"slug":3612,"type":15},"2026-07-12T07:46:41.097412",{"slug":3642,"name":3642,"fn":3643,"description":3644,"org":3645,"tags":3646,"stars":23,"repoUrl":24,"updatedAt":3653},"cloud-network-security","manage Elastic Cloud network security","Manage Serverless network security (traffic filters): create, update, and delete IP filters and AWS PrivateLink VPC filters. Use when restricting network access or configuring private connectivity.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3647,3648,3649,3650],{"name":3607,"slug":3608,"type":15},{"name":3628,"slug":3629,"type":15},{"name":2213,"slug":2210,"type":15},{"name":3651,"slug":3652,"type":15},"Security","security","2026-07-12T07:46:43.675992",{"slug":3655,"name":3655,"fn":3656,"description":3657,"org":3658,"tags":3659,"stars":23,"repoUrl":24,"updatedAt":3665},"cloud-setup","configure Elastic Cloud authentication","Configures Elastic Cloud authentication and environment defaults. Use when setting up EC_API_KEY, configuring Cloud API access, or when another cloud skill requires credentials.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3660,3663,3664],{"name":3661,"slug":3662,"type":15},"Authentication","authentication",{"name":3607,"slug":3608,"type":15},{"name":3628,"slug":3629,"type":15},"2026-07-12T07:46:39.783105",{"slug":3667,"name":3667,"fn":3668,"description":3669,"org":3670,"tags":3671,"stars":23,"repoUrl":24,"updatedAt":3680},"elasticsearch-audit","configure Elasticsearch security audit logs","Enable, configure, and query Elasticsearch security audit logs. Use when the task involves audit logging setup, event filtering, or investigating security incidents like failed logins.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3672,3675,3676,3679],{"name":3673,"slug":3674,"type":15},"Audit","audit",{"name":3628,"slug":3629,"type":15},{"name":3677,"slug":3678,"type":15},"Logs","logs",{"name":3651,"slug":3652,"type":15},"2026-07-12T07:47:35.092599",{"slug":3682,"name":3682,"fn":3683,"description":3684,"org":3685,"tags":3686,"stars":23,"repoUrl":24,"updatedAt":3691},"elasticsearch-authn","configure Elasticsearch authentication realms","Authenticate to Elasticsearch using native, file-based, LDAP\u002FAD, SAML, OIDC, Kerberos, JWT, or certificate realms. Use when connecting with credentials, choosing a realm, or managing API keys. Assumes the target realms are already configured.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3687,3688,3689,3690],{"name":3661,"slug":3662,"type":15},{"name":9,"slug":8,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3651,"slug":3652,"type":15},"2026-07-12T07:47:41.474547",35,{"items":3694,"total":3814},[3695,3714,3729,3742,3749,3755,3761,3768,3774,3781,3788,3801],{"slug":3696,"name":3696,"fn":3697,"description":3698,"org":3699,"tags":3700,"stars":3711,"repoUrl":3712,"updatedAt":3713},"accessing-benchmark-results","retrieve and analyze Rally benchmark results","Retrieve Rally benchmark results from an external Elasticsearch metrics store. Use to list past races, get a single race's overall (per-task) results, chart a metric's trend across multiple runs, compare two races, or check whether a run converged — e.g. \"show me recent geonames races\", \"what's the service_time trend for nyc_taxis over the last 30 days?\", \"compare these two race-ids\". Applies when datastore.type = elasticsearch is set in ~\u002F.rally\u002Frally.ini.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3701,3704,3707,3708],{"name":3702,"slug":3703,"type":15},"Analytics","analytics",{"name":3705,"slug":3706,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":3709,"slug":3710,"type":15},"Performance","performance",2027,"https:\u002F\u002Fgithub.com\u002Felastic\u002Frally","2026-07-12T07:46:38.54144",{"slug":3715,"name":3715,"fn":3716,"description":3717,"org":3718,"tags":3719,"stars":3711,"repoUrl":3712,"updatedAt":3728},"developing-rally","develop and debug Rally source code","Work on Rally's own codebase, not running benchmarks with it. Use when setting up the dev environment, running Rally's tests or linters, navigating its source, debugging Rally's own code, or making changes to Rally itself.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3720,3721,3722,3725],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":3723,"slug":3724,"type":15},"Engineering","engineering",{"name":3726,"slug":3727,"type":15},"Local Development","local-development","2026-07-12T07:46:35.976807",{"slug":3730,"name":3730,"fn":3731,"description":3732,"org":3733,"tags":3734,"stars":3711,"repoUrl":3712,"updatedAt":3741},"running-benchmarks","run Rally benchmarks against Elasticsearch","Run Rally benchmarks (races) against Elasticsearch — an existing\u002Fexternal cluster or a Rally-provisioned distribution — and read the summary report. Use when running a race (any pipeline, track, challenge, target-hosts, or auth) or when interpreting throughput, latency, and service_time results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3735,3736,3737,3738],{"name":9,"slug":8,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3709,"slug":3710,"type":15},{"name":3739,"slug":3740,"type":15},"Testing","testing","2026-07-12T07:46:37.277964",{"slug":3601,"name":3601,"fn":3602,"description":3603,"org":3743,"tags":3744,"stars":23,"repoUrl":24,"updatedAt":3616},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3745,3746,3747,3748],{"name":3607,"slug":3608,"type":15},{"name":9,"slug":8,"type":15},{"name":3611,"slug":3612,"type":15},{"name":3614,"slug":3615,"type":15},{"slug":3618,"name":3618,"fn":3619,"description":3620,"org":3750,"tags":3751,"stars":23,"repoUrl":24,"updatedAt":3630},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3752,3753,3754],{"name":3607,"slug":3608,"type":15},{"name":3625,"slug":3626,"type":15},{"name":3628,"slug":3629,"type":15},{"slug":3632,"name":3632,"fn":3633,"description":3634,"org":3756,"tags":3757,"stars":23,"repoUrl":24,"updatedAt":3640},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3758,3759,3760],{"name":3607,"slug":3608,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3611,"slug":3612,"type":15},{"slug":3642,"name":3642,"fn":3643,"description":3644,"org":3762,"tags":3763,"stars":23,"repoUrl":24,"updatedAt":3653},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3764,3765,3766,3767],{"name":3607,"slug":3608,"type":15},{"name":3628,"slug":3629,"type":15},{"name":2213,"slug":2210,"type":15},{"name":3651,"slug":3652,"type":15},{"slug":3655,"name":3655,"fn":3656,"description":3657,"org":3769,"tags":3770,"stars":23,"repoUrl":24,"updatedAt":3665},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3771,3772,3773],{"name":3661,"slug":3662,"type":15},{"name":3607,"slug":3608,"type":15},{"name":3628,"slug":3629,"type":15},{"slug":3667,"name":3667,"fn":3668,"description":3669,"org":3775,"tags":3776,"stars":23,"repoUrl":24,"updatedAt":3680},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3777,3778,3779,3780],{"name":3673,"slug":3674,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3677,"slug":3678,"type":15},{"name":3651,"slug":3652,"type":15},{"slug":3682,"name":3682,"fn":3683,"description":3684,"org":3782,"tags":3783,"stars":23,"repoUrl":24,"updatedAt":3691},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3784,3785,3786,3787],{"name":3661,"slug":3662,"type":15},{"name":9,"slug":8,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3651,"slug":3652,"type":15},{"slug":3789,"name":3789,"fn":3790,"description":3791,"org":3792,"tags":3793,"stars":23,"repoUrl":24,"updatedAt":3800},"elasticsearch-authz","manage Elasticsearch RBAC and security roles","Manage Elasticsearch RBAC: native users, roles, role mappings, document- and field-level security. Use when creating users or roles, assigning privileges, or mapping external realms like LDAP\u002FSAML.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3794,3795,3796,3799],{"name":9,"slug":8,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3797,"slug":3798,"type":15},"RBAC","rbac",{"name":3651,"slug":3652,"type":15},"2026-07-12T07:47:36.394177",{"slug":3802,"name":3802,"fn":3803,"description":3804,"org":3805,"tags":3806,"stars":23,"repoUrl":24,"updatedAt":3813},"elasticsearch-esql","query Elasticsearch data with ES|QL","Execute ES|QL (Elasticsearch Query Language) queries, use when the user wants to query Elasticsearch data, analyze logs, aggregate metrics, explore data, or create charts and dashboards from ES|QL results.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3807,3808,3809,3810],{"name":3702,"slug":3703,"type":15},{"name":3705,"slug":3706,"type":15},{"name":3628,"slug":3629,"type":15},{"name":3811,"slug":3812,"type":15},"SQL","sql","2026-07-12T07:47:40.249533",86]