[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-elastic-observability-service-health":3,"mdc--66ysmv-key":36,"related-org-elastic-observability-service-health":2232,"related-repo-elastic-observability-service-health":2399},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":34,"mdContent":35},"observability-service-health","assess Elastic APM service health","Assess APM service health using SLOs, alerts, ML, throughput, latency, error rate, and dependencies. Use when checking service status, performance, or when the user asks about service health.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"elastic","Elastic","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Felastic.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Performance","performance",{"name":20,"slug":21,"type":15},"Monitoring","monitoring",{"name":23,"slug":24,"type":15},"Elasticsearch","elasticsearch",531,"https:\u002F\u002Fgithub.com\u002Felastic\u002Fagent-skills","2026-07-12T07:47:18.484792",null,41,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":33},[],"Official Elastic Skills","https:\u002F\u002Fgithub.com\u002Felastic\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fobservability\u002Fskills\u002Fservice-health","---\nname: observability-service-health\ndescription: >\n  Assess APM service health using SLOs, alerts, ML, throughput, latency, error rate,\n  and dependencies. Use when checking service status, performance, or when the user\n  asks about service health.\nmetadata:\n  author: elastic\n  version: 0.1.0\n---\n\n# APM Service Health\n\nAssess APM service health using [Observability APIs](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapis),\n**ES|QL** against APM indices, Elasticsearch APIs, and (for correlation and APM-specific logic) the Kibana repo. Use\nSLOs, firing alerts, ML anomalies, throughput, latency (avg\u002Fp95\u002Fp99), error rate, and dependency health.\n\n## Where to look\n\n- **Observability APIs** ([Observability APIs](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapis)): Use the\n  **SLOs API** ([Stack](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-slo) |\n  [Serverless](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-slo)) to get SLO definitions, status, burn\n  rate, and error budget. Use the **Alerting API**\n  ([Stack](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-alerting) |\n  [Serverless](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-alerting)) to list and manage alerting\n  rules and their alerts for the service. Use **APM annotations API** to create or search annotations when needed.\n- **ES|QL and Elasticsearch:** Query `traces*apm*,traces*otel*` and `metrics*apm*,metrics*otel*` with **ES|QL** (see\n  [Using ES|QL for APM metrics](#using-esql-for-apm-metrics)) for throughput, latency, error rate, and dependency-style\n  aggregations. Use Elasticsearch APIs (e.g. `POST _query` for ES|QL, or Query DSL) as documented in the Elasticsearch\n  repo for indices and search.\n- **APM Correlations:** Run the **apm-correlations** script to get attributes that correlate with high-latency or failed\n  transactions for a given service. It tries the Kibana internal APM correlations API first, then falls back to\n  Elasticsearch significant_terms on `traces*apm*,traces*otel*`. See\n  [APM Correlations script](#apm-correlations-script).\n- **Infrastructure:** Correlate via **resource attributes** (e.g. `k8s.pod.name`, `container.id`, `host.name`) in\n  traces; query infrastructure or metrics indices with ES|QL\u002FElasticsearch for CPU and memory. **OOM** and **CPU\n  throttling** directly impact APM health.\n- **Logs:** Use **ES|QL** or Elasticsearch search on log indices filtered by `service.name` or `trace.id` to explain\n  behavior and root cause.\n- **Observability Labs:** [Observability Labs](https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs) and\n  [APM tag](https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs\u002Fblog\u002Ftag\u002Fapm) for patterns and troubleshooting.\n\n## Health criteria\n\nSynthesize health from all of the following when available:\n\n| Signal                | What to check                                                             |\n| --------------------- | ------------------------------------------------------------------------- |\n| **SLOs**              | Burn rate, status (healthy\u002Fdegrading\u002Fviolated), error budget.             |\n| **Firing alerts**     | Open or recently fired alerts for the service or dependencies.            |\n| **ML anomalies**      | Anomaly jobs; score and severity for latency, throughput, or error rate.  |\n| **Throughput**        | Request rate; compare to baseline or previous period.                     |\n| **Latency**           | Avg, p95, p99; compare to SLO targets or history.                         |\n| **Error rate**        | Failed\u002Ftotal requests; spikes or sustained elevation.                     |\n| **Dependency health** | Downstream latency, error rate, availability (ES\\|QL, APIs, Kibana repo). |\n| **Infrastructure**    | CPU usage, memory; OOM and CPU throttling on pods\u002Fcontainers\u002Fhosts.       |\n| **Logs**              | App logs filtered by service or trace ID for context and root cause.      |\n\nTreat a service as **unhealthy** if SLOs are violated, critical alerts are firing, or ML anomalies indicate severe\ndegradation. Correlate with infrastructure (OOM, CPU throttling), dependencies, and logs (service\u002Ftrace context) to\nexplain _why_ and suggest next steps.\n\n## Using ES|QL for APM metrics\n\nWhen querying APM data from Elasticsearch (`traces*apm*,traces*otel*`, `metrics*apm*,metrics*otel*`), use **ES|QL by\ndefault** where available.\n\n- **Availability:** ES|QL is available in **Elasticsearch 8.11+** (technical preview; GA in 8.14). It is **always\n  available** in\n  [Elastic Observability Serverless Complete tier](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fobservability-serverless-feature-tiers).\n- **Scoping to a service:** Always filter by `service.name` (and `service.environment` when relevant). Combine with a\n  time range on `@timestamp`:\n\n```esql\nWHERE service.name == \"my-service-name\" AND service.environment == \"production\"\n  AND @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n```\n\n- **Example patterns:** Throughput, latency, and error rate over time: see Kibana `trace_charts_definition.ts`\n  (`getThroughputChart`, `getLatencyChart`, `getErrorRateChart`). Use `from(index)` → `where(...)` → `stats(...)` \u002F\n  `evaluate(...)` with `BUCKET(@timestamp, ...)` and `WHERE service.name == \"\u003Cservice_name>\"`.\n- **Performance:** Add `LIMIT n` to cap rows and token usage. Prefer coarser `BUCKET(@timestamp, ...)` (e.g. 1 hour)\n  when only trends are needed; finer buckets increase work and result size.\n\n## APM Correlations script\n\nWhen only a **subpopulation** of transactions has high latency or failures, run the **apm-correlations** script to list\nattributes that correlate with those transactions (e.g. host, service version, pod, region). The script tries the Kibana\ninternal APM correlations API first; if unavailable (e.g. 404), it falls back to Elasticsearch significant_terms on\n`traces*apm*,traces*otel*`.\n\n```bash\n# Latency correlations (attributes over-represented in slow transactions)\nnode skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js latency-correlations --service-name \u003Cname> [--start \u003Ciso>] [--end \u003Ciso>] [--last-minutes 60] [--transaction-type \u003Ct>] [--transaction-name \u003Cn>] [--space \u003Cid>] [--json]\n\n# Failed transaction correlations\nnode skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js failed-correlations --service-name \u003Cname> [--start \u003Ciso>] [--end \u003Ciso>] [--last-minutes 60] [--transaction-type \u003Ct>] [--transaction-name \u003Cn>] [--space \u003Cid>] [--json]\n\n# Test Kibana connection\nnode skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js test [--space \u003Cid>]\n```\n\n**Environment:** `KIBANA_URL` and `KIBANA_API_KEY` (or `KIBANA_USERNAME`\u002F`KIBANA_PASSWORD`) for Kibana; for fallback,\n`ELASTICSEARCH_URL` and `ELASTICSEARCH_API_KEY`. Use the same time range as the investigation.\n\n## Workflow\n\n```text\nService health progress:\n- [ ] Step 1: Identify the service (and time range)\n- [ ] Step 2: Check SLOs and firing alerts\n- [ ] Step 3: Check ML anomalies (if configured)\n- [ ] Step 4: Review throughput, latency (avg\u002Fp95\u002Fp99), error rate\n- [ ] Step 5: Assess dependency health (ES|QL\u002FAPIs \u002F Kibana repo)\n- [ ] Step 6: Correlate with infrastructure and logs\n- [ ] Step 7: Summarize health and recommend actions\n```\n\n### Step 1: Identify the service\n\nConfirm service name and time range. Resolve the service from the request; if multiple are in scope, target the most\nrelevant. Use ES|QL on `traces*apm*,traces*otel*` or `metrics*apm*,metrics*otel*` (e.g.\n`WHERE service.name == \"\u003Cname>\"`) or Kibana repo APM routes to obtain service-level data. If the user has not provided\nthe time range, assume last hour.\n\n### Step 2: Check SLOs and firing alerts\n\n**SLOs:** Call the **SLOs API** to get SLO definitions and status for the service (latency, availability),\nhealthy\u002Fdegrading\u002Fviolated, burn rate, error budget. **Alerts:** For active APM alerts, call\n`\u002Fapi\u002Falerting\u002Frules\u002F_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active`.\nWhen checking one service, include both rules where `params.serviceName` matches the service and rules where\n`params.serviceName` is absent (all-services rules). Do not query `.alerts*` indices for active-state checks. Correlate\nwith SLO violations or metric changes.\n\n### Step 3: Check ML anomalies\n\nIf ML anomaly detection is used, query ML job results or anomaly records (via Elasticsearch ML APIs or indices) for the\nservice and time range. Note high-severity anomalies (latency, throughput, error rate); use anomaly time windows to\nnarrow Steps 4–5.\n\n### Step 4: Review throughput, latency, and error rate\n\nUse **ES|QL** against `traces*apm*,traces*otel*` or `metrics*apm*,metrics*otel*` for the service and time range to get\n**throughput** (e.g. req\u002Fmin), **latency** (avg, p95, p99), **error rate** (failed\u002Ftotal or 5xx\u002Ftotal). Example:\n`FROM traces*apm*,traces*otel* | WHERE service.name == \"\u003Cservice_name>\" AND @timestamp >= ... AND @timestamp \u003C= ... | STATS ...`.\nCompare to prior period or SLO targets. See [Using ES|QL for APM metrics](#using-esql-for-apm-metrics).\n\n### Step 5: Assess dependency health\n\nObtain dependency and service-map data via **ES|QL** on `traces*apm*,traces*otel*`\u002F`metrics*apm*,metrics*otel*` (e.g.\ndownstream service\u002Fspan aggregations) or via APM route handlers in the **Kibana repo** that expose\ndependency\u002Fservice-map data. For the service and time range, note downstream latency and error rate; flag slow or\nfailing dependencies as likely causes.\n\n### Step 6: Correlate with infrastructure and logs\n\n- **APM Correlations (when only a subpopulation is affected):** Run\n  `node skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js latency-correlations|failed-correlations --service-name \u003Cname> [--start ...] [--end ...]`\n  to get correlated attributes. Filter by those attributes and fetch trace samples or errors to confirm root cause. See\n  [APM Correlations script](#apm-correlations-script).\n- **Infrastructure:** Use **resource attributes** from traces (e.g. `k8s.pod.name`, `container.id`, `host.name`) and\n  query infrastructure\u002Fmetrics indices with **ES|QL** or Elasticsearch for **CPU** and **memory**. **OOM** and **CPU\n  throttling** directly impact APM health; correlate their time windows with APM degradation.\n- **Logs:** Use **ES|QL** or Elasticsearch on log indices with `service.name == \"\u003Cservice_name>\"` or\n  `trace.id == \"\u003Ctrace_id>\"` to explain behavior and root cause (exceptions, timeouts, restarts).\n\n### Step 7: Summarize and recommend\n\nState health (**healthy** \u002F **degraded** \u002F **unhealthy**) with reasons; list concrete next steps.\n\n## Examples\n\n### Example: ES|QL for a specific service\n\nScope with `WHERE service.name == \"\u003Cservice_name>\"` and time range. Throughput and error rate (1-hour buckets; `LIMIT`\ncaps rows and tokens):\n\n```esql\nFROM traces*apm*,traces*otel*\n| WHERE service.name == \"api-gateway\"\n  AND @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == \"failure\" BY BUCKET(@timestamp, 1 hour)\n| EVAL error_rate = failures \u002F request_count\n| SORT @timestamp\n| LIMIT 500\n```\n\nLatency percentiles and exact field names: see Kibana `trace_charts_definition.ts`.\n\n### Example: \"Is service X healthy?\"\n\n1. Resolve service X and time range. Call **SLOs API** and **Alerting API**; run **ES|QL** on\n   `traces*apm*,traces*otel*`\u002F`metrics*apm*,metrics*otel*` for throughput, latency, error rate; query\n   dependency\u002Fservice-map data (ES|QL or Kibana repo).\n2. Evaluate SLO status (violated\u002Fdegrading?), firing rules, ML anomalies, and dependency health.\n3. Answer: Healthy \u002F Degraded \u002F Unhealthy with reasons and next steps (e.g.\n   [Observability Labs](https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs)).\n\n### Example: \"Why is service Y slow?\"\n\n1. Service Y and slowness time range. Call **SLOs API** and **Alerting API**; run **ES|QL** for Y and dependencies;\n   query ML anomaly results.\n2. Compare latency (avg\u002Fp95\u002Fp99) to prior period via ES|QL; from dependency data identify high-latency or failing deps.\n3. Summarize (e.g. p99 up; dependency Z elevated) and recommend (investigate Z; Observability Labs for latency).\n\n### Example: Correlate service to infrastructure (OpenTelemetry)\n\nUse **resource attributes** on spans\u002Ftraces to get the runtimes (pods, containers, hosts) for the service. Then check\nCPU and memory for those resources in the same time window as the APM issue:\n\n- From the service’s traces or metrics, read resource attributes such as `k8s.pod.name`, `k8s.namespace.name`,\n  `container.id`, or `host.name`.\n- Run **ES|QL** or Elasticsearch search on infrastructure\u002Fmetrics indices filtered by those resource values and the\n  incident time range. Check **CPU usage** and **memory consumption** (e.g. `system.cpu.total.norm.pct`); look for\n  **OOMKilled** events, **CPU throttling**, or sustained high CPU\u002Fmemory that align with APM latency or error spikes.\n\n### Example: Filter logs by service or trace ID\n\nTo understand behavior for a specific service or a single trace, filter logs accordingly:\n\n- **By service:** Run **ES|QL** or Elasticsearch search on log indices with `service.name == \"\u003Cservice_name>\"` and time\n  range to get application logs (errors, warnings, restarts) in the service context.\n- **By trace ID:** When investigating a specific request, take the `trace.id` from the APM trace and filter logs by\n  `trace.id == \"\u003Ctrace_id>\"` (or equivalent field in your log schema). Logs with that trace ID show the full request\n  path and help explain failures or latency.\n\n## Guidelines\n\n- Use **Observability APIs** ([SLOs API](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-slo),\n  [Alerting API](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-alerting)) and **ES|QL** on\n  `traces*apm*,traces*otel*`\u002F`metrics*apm*,metrics*otel*` (8.11+ or Serverless), filtering by `service.name` (and\n  `service.environment` when relevant). For active APM alerts, call\n  `\u002Fapi\u002Falerting\u002Frules\u002F_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active`.\n  When checking one service, evaluate both rule types: rules where `params.serviceName` matches the target service, and\n  rules where `params.serviceName` is absent (all-services rules). Treat either as applicable to the service before\n  declaring health. Do not query `.alerts*` indices when determining currently active alerts; use the Alerting API\n  response above as the source of truth. For APM correlations, run the **apm-correlations** script (see\n  [APM Correlations script](#apm-correlations-script)); for dependency\u002Fservice-map data, use ES|QL or Kibana repo route\n  handlers. For Elasticsearch index and search behavior, see the **Elasticsearch** APIs in the Elasticsearch repo.\n- Always use the **user's time range**; avoid assuming \"last 1 hour\" if the issue is historical.\n- When SLOs exist, anchor the health summary to SLO status and burn rate; when they do not, rely on alerts, anomalies,\n  throughput, latency, error rate, and dependencies.\n- When analyzing **only application metrics ingested via OpenTelemetry**, use the ES|QL **TS** (time series) command for\n  efficient metrics queries. The TS command is available in **Elasticsearch 9.3+** and is **always available** in\n  Elastic Observability Serverless.\n- Summary: one short health verdict plus bullet points for evidence and next steps.\n",{"data":37,"body":40},{"name":4,"description":6,"metadata":38},{"author":8,"version":39},"0.1.0",{"type":41,"children":42},"root",[43,52,77,84,352,358,363,535,555,560,585,653,684,795,800,824,1268,1323,1329,1339,1346,1372,1378,1432,1438,1443,1449,1507,1513,1544,1550,1669,1675,1699,1705,1711,1731,1793,1804,1810,1866,1872,1906,1912,1922,2001,2007,2012,2062,2068,2226],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"apm-service-health",[49],{"type":50,"value":51},"text","APM Service Health",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,67,69,75],{"type":50,"value":57},"Assess APM service health using ",{"type":44,"tag":59,"props":60,"children":64},"a",{"href":61,"rel":62},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapis",[63],"nofollow",[65],{"type":50,"value":66},"Observability APIs",{"type":50,"value":68},",\n",{"type":44,"tag":70,"props":71,"children":72},"strong",{},[73],{"type":50,"value":74},"ES|QL",{"type":50,"value":76}," against APM indices, Elasticsearch APIs, and (for correlation and APM-specific logic) the Kibana repo. Use\nSLOs, firing alerts, ML anomalies, throughput, latency (avg\u002Fp95\u002Fp99), error rate, and dependency health.",{"type":44,"tag":78,"props":79,"children":81},"h2",{"id":80},"where-to-look",[82],{"type":50,"value":83},"Where to look",{"type":44,"tag":85,"props":86,"children":87},"ul",{},[88,158,207,239,292,324],{"type":44,"tag":89,"props":90,"children":91},"li",{},[92,96,98,103,105,110,111,118,120,127,129,134,136,142,143,149,151,156],{"type":44,"tag":70,"props":93,"children":94},{},[95],{"type":50,"value":66},{"type":50,"value":97}," (",{"type":44,"tag":59,"props":99,"children":101},{"href":61,"rel":100},[63],[102],{"type":50,"value":66},{"type":50,"value":104},"): Use the\n",{"type":44,"tag":70,"props":106,"children":107},{},[108],{"type":50,"value":109},"SLOs API",{"type":50,"value":97},{"type":44,"tag":59,"props":112,"children":115},{"href":113,"rel":114},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-slo",[63],[116],{"type":50,"value":117},"Stack",{"type":50,"value":119}," |\n",{"type":44,"tag":59,"props":121,"children":124},{"href":122,"rel":123},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-slo",[63],[125],{"type":50,"value":126},"Serverless",{"type":50,"value":128},") to get SLO definitions, status, burn\nrate, and error budget. Use the ",{"type":44,"tag":70,"props":130,"children":131},{},[132],{"type":50,"value":133},"Alerting API",{"type":50,"value":135},"\n(",{"type":44,"tag":59,"props":137,"children":140},{"href":138,"rel":139},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-alerting",[63],[141],{"type":50,"value":117},{"type":50,"value":119},{"type":44,"tag":59,"props":144,"children":147},{"href":145,"rel":146},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-alerting",[63],[148],{"type":50,"value":126},{"type":50,"value":150},") to list and manage alerting\nrules and their alerts for the service. Use ",{"type":44,"tag":70,"props":152,"children":153},{},[154],{"type":50,"value":155},"APM annotations API",{"type":50,"value":157}," to create or search annotations when needed.",{"type":44,"tag":89,"props":159,"children":160},{},[161,166,168,175,177,183,185,189,191,197,199,205],{"type":44,"tag":70,"props":162,"children":163},{},[164],{"type":50,"value":165},"ES|QL and Elasticsearch:",{"type":50,"value":167}," Query ",{"type":44,"tag":169,"props":170,"children":172},"code",{"className":171},[],[173],{"type":50,"value":174},"traces*apm*,traces*otel*",{"type":50,"value":176}," and ",{"type":44,"tag":169,"props":178,"children":180},{"className":179},[],[181],{"type":50,"value":182},"metrics*apm*,metrics*otel*",{"type":50,"value":184}," with ",{"type":44,"tag":70,"props":186,"children":187},{},[188],{"type":50,"value":74},{"type":50,"value":190}," (see\n",{"type":44,"tag":59,"props":192,"children":194},{"href":193},"#using-esql-for-apm-metrics",[195],{"type":50,"value":196},"Using ES|QL for APM metrics",{"type":50,"value":198},") for throughput, latency, error rate, and dependency-style\naggregations. Use Elasticsearch APIs (e.g. ",{"type":44,"tag":169,"props":200,"children":202},{"className":201},[],[203],{"type":50,"value":204},"POST _query",{"type":50,"value":206}," for ES|QL, or Query DSL) as documented in the Elasticsearch\nrepo for indices and search.",{"type":44,"tag":89,"props":208,"children":209},{},[210,215,217,222,224,229,231,237],{"type":44,"tag":70,"props":211,"children":212},{},[213],{"type":50,"value":214},"APM Correlations:",{"type":50,"value":216}," Run the ",{"type":44,"tag":70,"props":218,"children":219},{},[220],{"type":50,"value":221},"apm-correlations",{"type":50,"value":223}," script to get attributes that correlate with high-latency or failed\ntransactions for a given service. It tries the Kibana internal APM correlations API first, then falls back to\nElasticsearch significant_terms on ",{"type":44,"tag":169,"props":225,"children":227},{"className":226},[],[228],{"type":50,"value":174},{"type":50,"value":230},". See\n",{"type":44,"tag":59,"props":232,"children":234},{"href":233},"#apm-correlations-script",[235],{"type":50,"value":236},"APM Correlations script",{"type":50,"value":238},".",{"type":44,"tag":89,"props":240,"children":241},{},[242,247,249,254,256,262,264,270,271,277,279,284,285,290],{"type":44,"tag":70,"props":243,"children":244},{},[245],{"type":50,"value":246},"Infrastructure:",{"type":50,"value":248}," Correlate via ",{"type":44,"tag":70,"props":250,"children":251},{},[252],{"type":50,"value":253},"resource attributes",{"type":50,"value":255}," (e.g. ",{"type":44,"tag":169,"props":257,"children":259},{"className":258},[],[260],{"type":50,"value":261},"k8s.pod.name",{"type":50,"value":263},", ",{"type":44,"tag":169,"props":265,"children":267},{"className":266},[],[268],{"type":50,"value":269},"container.id",{"type":50,"value":263},{"type":44,"tag":169,"props":272,"children":274},{"className":273},[],[275],{"type":50,"value":276},"host.name",{"type":50,"value":278},") in\ntraces; query infrastructure or metrics indices with ES|QL\u002FElasticsearch for CPU and memory. ",{"type":44,"tag":70,"props":280,"children":281},{},[282],{"type":50,"value":283},"OOM",{"type":50,"value":176},{"type":44,"tag":70,"props":286,"children":287},{},[288],{"type":50,"value":289},"CPU\nthrottling",{"type":50,"value":291}," directly impact APM health.",{"type":44,"tag":89,"props":293,"children":294},{},[295,300,302,306,308,314,316,322],{"type":44,"tag":70,"props":296,"children":297},{},[298],{"type":50,"value":299},"Logs:",{"type":50,"value":301}," Use ",{"type":44,"tag":70,"props":303,"children":304},{},[305],{"type":50,"value":74},{"type":50,"value":307}," or Elasticsearch search on log indices filtered by ",{"type":44,"tag":169,"props":309,"children":311},{"className":310},[],[312],{"type":50,"value":313},"service.name",{"type":50,"value":315}," or ",{"type":44,"tag":169,"props":317,"children":319},{"className":318},[],[320],{"type":50,"value":321},"trace.id",{"type":50,"value":323}," to explain\nbehavior and root cause.",{"type":44,"tag":89,"props":325,"children":326},{},[327,332,334,341,343,350],{"type":44,"tag":70,"props":328,"children":329},{},[330],{"type":50,"value":331},"Observability Labs:",{"type":50,"value":333}," ",{"type":44,"tag":59,"props":335,"children":338},{"href":336,"rel":337},"https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs",[63],[339],{"type":50,"value":340},"Observability Labs",{"type":50,"value":342}," and\n",{"type":44,"tag":59,"props":344,"children":347},{"href":345,"rel":346},"https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs\u002Fblog\u002Ftag\u002Fapm",[63],[348],{"type":50,"value":349},"APM tag",{"type":50,"value":351}," for patterns and troubleshooting.",{"type":44,"tag":78,"props":353,"children":355},{"id":354},"health-criteria",[356],{"type":50,"value":357},"Health criteria",{"type":44,"tag":53,"props":359,"children":360},{},[361],{"type":50,"value":362},"Synthesize health from all of the following when available:",{"type":44,"tag":364,"props":365,"children":366},"table",{},[367,386],{"type":44,"tag":368,"props":369,"children":370},"thead",{},[371],{"type":44,"tag":372,"props":373,"children":374},"tr",{},[375,381],{"type":44,"tag":376,"props":377,"children":378},"th",{},[379],{"type":50,"value":380},"Signal",{"type":44,"tag":376,"props":382,"children":383},{},[384],{"type":50,"value":385},"What to check",{"type":44,"tag":387,"props":388,"children":389},"tbody",{},[390,407,423,439,455,471,487,503,519],{"type":44,"tag":372,"props":391,"children":392},{},[393,402],{"type":44,"tag":394,"props":395,"children":396},"td",{},[397],{"type":44,"tag":70,"props":398,"children":399},{},[400],{"type":50,"value":401},"SLOs",{"type":44,"tag":394,"props":403,"children":404},{},[405],{"type":50,"value":406},"Burn rate, status (healthy\u002Fdegrading\u002Fviolated), error budget.",{"type":44,"tag":372,"props":408,"children":409},{},[410,418],{"type":44,"tag":394,"props":411,"children":412},{},[413],{"type":44,"tag":70,"props":414,"children":415},{},[416],{"type":50,"value":417},"Firing alerts",{"type":44,"tag":394,"props":419,"children":420},{},[421],{"type":50,"value":422},"Open or recently fired alerts for the service or dependencies.",{"type":44,"tag":372,"props":424,"children":425},{},[426,434],{"type":44,"tag":394,"props":427,"children":428},{},[429],{"type":44,"tag":70,"props":430,"children":431},{},[432],{"type":50,"value":433},"ML anomalies",{"type":44,"tag":394,"props":435,"children":436},{},[437],{"type":50,"value":438},"Anomaly jobs; score and severity for latency, throughput, or error rate.",{"type":44,"tag":372,"props":440,"children":441},{},[442,450],{"type":44,"tag":394,"props":443,"children":444},{},[445],{"type":44,"tag":70,"props":446,"children":447},{},[448],{"type":50,"value":449},"Throughput",{"type":44,"tag":394,"props":451,"children":452},{},[453],{"type":50,"value":454},"Request rate; compare to baseline or previous period.",{"type":44,"tag":372,"props":456,"children":457},{},[458,466],{"type":44,"tag":394,"props":459,"children":460},{},[461],{"type":44,"tag":70,"props":462,"children":463},{},[464],{"type":50,"value":465},"Latency",{"type":44,"tag":394,"props":467,"children":468},{},[469],{"type":50,"value":470},"Avg, p95, p99; compare to SLO targets or history.",{"type":44,"tag":372,"props":472,"children":473},{},[474,482],{"type":44,"tag":394,"props":475,"children":476},{},[477],{"type":44,"tag":70,"props":478,"children":479},{},[480],{"type":50,"value":481},"Error rate",{"type":44,"tag":394,"props":483,"children":484},{},[485],{"type":50,"value":486},"Failed\u002Ftotal requests; spikes or sustained elevation.",{"type":44,"tag":372,"props":488,"children":489},{},[490,498],{"type":44,"tag":394,"props":491,"children":492},{},[493],{"type":44,"tag":70,"props":494,"children":495},{},[496],{"type":50,"value":497},"Dependency health",{"type":44,"tag":394,"props":499,"children":500},{},[501],{"type":50,"value":502},"Downstream latency, error rate, availability (ES|QL, APIs, Kibana repo).",{"type":44,"tag":372,"props":504,"children":505},{},[506,514],{"type":44,"tag":394,"props":507,"children":508},{},[509],{"type":44,"tag":70,"props":510,"children":511},{},[512],{"type":50,"value":513},"Infrastructure",{"type":44,"tag":394,"props":515,"children":516},{},[517],{"type":50,"value":518},"CPU usage, memory; OOM and CPU throttling on pods\u002Fcontainers\u002Fhosts.",{"type":44,"tag":372,"props":520,"children":521},{},[522,530],{"type":44,"tag":394,"props":523,"children":524},{},[525],{"type":44,"tag":70,"props":526,"children":527},{},[528],{"type":50,"value":529},"Logs",{"type":44,"tag":394,"props":531,"children":532},{},[533],{"type":50,"value":534},"App logs filtered by service or trace ID for context and root cause.",{"type":44,"tag":53,"props":536,"children":537},{},[538,540,545,547,553],{"type":50,"value":539},"Treat a service as ",{"type":44,"tag":70,"props":541,"children":542},{},[543],{"type":50,"value":544},"unhealthy",{"type":50,"value":546}," if SLOs are violated, critical alerts are firing, or ML anomalies indicate severe\ndegradation. Correlate with infrastructure (OOM, CPU throttling), dependencies, and logs (service\u002Ftrace context) to\nexplain ",{"type":44,"tag":548,"props":549,"children":550},"em",{},[551],{"type":50,"value":552},"why",{"type":50,"value":554}," and suggest next steps.",{"type":44,"tag":78,"props":556,"children":558},{"id":557},"using-esql-for-apm-metrics",[559],{"type":50,"value":196},{"type":44,"tag":53,"props":561,"children":562},{},[563,565,570,571,576,578,583],{"type":50,"value":564},"When querying APM data from Elasticsearch (",{"type":44,"tag":169,"props":566,"children":568},{"className":567},[],[569],{"type":50,"value":174},{"type":50,"value":263},{"type":44,"tag":169,"props":572,"children":574},{"className":573},[],[575],{"type":50,"value":182},{"type":50,"value":577},"), use ",{"type":44,"tag":70,"props":579,"children":580},{},[581],{"type":50,"value":582},"ES|QL by\ndefault",{"type":50,"value":584}," where available.",{"type":44,"tag":85,"props":586,"children":587},{},[588,620],{"type":44,"tag":89,"props":589,"children":590},{},[591,596,598,603,605,610,612,619],{"type":44,"tag":70,"props":592,"children":593},{},[594],{"type":50,"value":595},"Availability:",{"type":50,"value":597}," ES|QL is available in ",{"type":44,"tag":70,"props":599,"children":600},{},[601],{"type":50,"value":602},"Elasticsearch 8.11+",{"type":50,"value":604}," (technical preview; GA in 8.14). It is ",{"type":44,"tag":70,"props":606,"children":607},{},[608],{"type":50,"value":609},"always\navailable",{"type":50,"value":611}," in\n",{"type":44,"tag":59,"props":613,"children":616},{"href":614,"rel":615},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fobservability-serverless-feature-tiers",[63],[617],{"type":50,"value":618},"Elastic Observability Serverless Complete tier",{"type":50,"value":238},{"type":44,"tag":89,"props":621,"children":622},{},[623,628,630,635,637,643,645,651],{"type":44,"tag":70,"props":624,"children":625},{},[626],{"type":50,"value":627},"Scoping to a service:",{"type":50,"value":629}," Always filter by ",{"type":44,"tag":169,"props":631,"children":633},{"className":632},[],[634],{"type":50,"value":313},{"type":50,"value":636}," (and ",{"type":44,"tag":169,"props":638,"children":640},{"className":639},[],[641],{"type":50,"value":642},"service.environment",{"type":50,"value":644}," when relevant). Combine with a\ntime range on ",{"type":44,"tag":169,"props":646,"children":648},{"className":647},[],[649],{"type":50,"value":650},"@timestamp",{"type":50,"value":652},":",{"type":44,"tag":654,"props":655,"children":660},"pre",{"className":656,"code":657,"language":658,"meta":659,"style":659},"language-esql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","WHERE service.name == \"my-service-name\" AND service.environment == \"production\"\n  AND @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n","esql","",[661],{"type":44,"tag":169,"props":662,"children":663},{"__ignoreMap":659},[664,675],{"type":44,"tag":665,"props":666,"children":669},"span",{"class":667,"line":668},"line",1,[670],{"type":44,"tag":665,"props":671,"children":672},{},[673],{"type":50,"value":674},"WHERE service.name == \"my-service-name\" AND service.environment == \"production\"\n",{"type":44,"tag":665,"props":676,"children":678},{"class":667,"line":677},2,[679],{"type":44,"tag":665,"props":680,"children":681},{},[682],{"type":50,"value":683},"  AND @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n",{"type":44,"tag":85,"props":685,"children":686},{},[687,770],{"type":44,"tag":89,"props":688,"children":689},{},[690,695,697,703,704,710,711,717,718,724,726,732,734,740,741,747,749,755,756,762,763,769],{"type":44,"tag":70,"props":691,"children":692},{},[693],{"type":50,"value":694},"Example patterns:",{"type":50,"value":696}," Throughput, latency, and error rate over time: see Kibana ",{"type":44,"tag":169,"props":698,"children":700},{"className":699},[],[701],{"type":50,"value":702},"trace_charts_definition.ts",{"type":50,"value":135},{"type":44,"tag":169,"props":705,"children":707},{"className":706},[],[708],{"type":50,"value":709},"getThroughputChart",{"type":50,"value":263},{"type":44,"tag":169,"props":712,"children":714},{"className":713},[],[715],{"type":50,"value":716},"getLatencyChart",{"type":50,"value":263},{"type":44,"tag":169,"props":719,"children":721},{"className":720},[],[722],{"type":50,"value":723},"getErrorRateChart",{"type":50,"value":725},"). Use ",{"type":44,"tag":169,"props":727,"children":729},{"className":728},[],[730],{"type":50,"value":731},"from(index)",{"type":50,"value":733}," → ",{"type":44,"tag":169,"props":735,"children":737},{"className":736},[],[738],{"type":50,"value":739},"where(...)",{"type":50,"value":733},{"type":44,"tag":169,"props":742,"children":744},{"className":743},[],[745],{"type":50,"value":746},"stats(...)",{"type":50,"value":748}," \u002F\n",{"type":44,"tag":169,"props":750,"children":752},{"className":751},[],[753],{"type":50,"value":754},"evaluate(...)",{"type":50,"value":184},{"type":44,"tag":169,"props":757,"children":759},{"className":758},[],[760],{"type":50,"value":761},"BUCKET(@timestamp, ...)",{"type":50,"value":176},{"type":44,"tag":169,"props":764,"children":766},{"className":765},[],[767],{"type":50,"value":768},"WHERE service.name == \"\u003Cservice_name>\"",{"type":50,"value":238},{"type":44,"tag":89,"props":771,"children":772},{},[773,778,780,786,788,793],{"type":44,"tag":70,"props":774,"children":775},{},[776],{"type":50,"value":777},"Performance:",{"type":50,"value":779}," Add ",{"type":44,"tag":169,"props":781,"children":783},{"className":782},[],[784],{"type":50,"value":785},"LIMIT n",{"type":50,"value":787}," to cap rows and token usage. Prefer coarser ",{"type":44,"tag":169,"props":789,"children":791},{"className":790},[],[792],{"type":50,"value":761},{"type":50,"value":794}," (e.g. 1 hour)\nwhen only trends are needed; finer buckets increase work and result size.",{"type":44,"tag":78,"props":796,"children":798},{"id":797},"apm-correlations-script",[799],{"type":50,"value":236},{"type":44,"tag":53,"props":801,"children":802},{},[803,805,810,812,816,818,823],{"type":50,"value":804},"When only a ",{"type":44,"tag":70,"props":806,"children":807},{},[808],{"type":50,"value":809},"subpopulation",{"type":50,"value":811}," of transactions has high latency or failures, run the ",{"type":44,"tag":70,"props":813,"children":814},{},[815],{"type":50,"value":221},{"type":50,"value":817}," script to list\nattributes that correlate with those transactions (e.g. host, service version, pod, region). The script tries the Kibana\ninternal APM correlations API first; if unavailable (e.g. 404), it falls back to Elasticsearch significant_terms on\n",{"type":44,"tag":169,"props":819,"children":821},{"className":820},[],[822],{"type":50,"value":174},{"type":50,"value":238},{"type":44,"tag":654,"props":825,"children":829},{"className":826,"code":827,"language":828,"meta":659,"style":659},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Latency correlations (attributes over-represented in slow transactions)\nnode skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js latency-correlations --service-name \u003Cname> [--start \u003Ciso>] [--end \u003Ciso>] [--last-minutes 60] [--transaction-type \u003Ct>] [--transaction-name \u003Cn>] [--space \u003Cid>] [--json]\n\n# Failed transaction correlations\nnode skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js failed-correlations --service-name \u003Cname> [--start \u003Ciso>] [--end \u003Ciso>] [--last-minutes 60] [--transaction-type \u003Ct>] [--transaction-name \u003Cn>] [--space \u003Cid>] [--json]\n\n# Test Kibana connection\nnode skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js test [--space \u003Cid>]\n","bash",[830],{"type":44,"tag":169,"props":831,"children":832},{"__ignoreMap":659},[833,842,1029,1039,1048,1209,1217,1226],{"type":44,"tag":665,"props":834,"children":835},{"class":667,"line":668},[836],{"type":44,"tag":665,"props":837,"children":839},{"style":838},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[840],{"type":50,"value":841},"# Latency correlations (attributes over-represented in slow transactions)\n",{"type":44,"tag":665,"props":843,"children":844},{"class":667,"line":677},[845,851,857,862,867,873,878,884,889,894,899,904,909,913,918,923,927,931,935,939,943,948,953,958,962,967,971,975,980,984,989,993,997,1002,1006,1011,1016,1020,1024],{"type":44,"tag":665,"props":846,"children":848},{"style":847},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[849],{"type":50,"value":850},"node",{"type":44,"tag":665,"props":852,"children":854},{"style":853},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[855],{"type":50,"value":856}," skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js",{"type":44,"tag":665,"props":858,"children":859},{"style":853},[860],{"type":50,"value":861}," latency-correlations",{"type":44,"tag":665,"props":863,"children":864},{"style":853},[865],{"type":50,"value":866}," --service-name",{"type":44,"tag":665,"props":868,"children":870},{"style":869},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[871],{"type":50,"value":872}," \u003C",{"type":44,"tag":665,"props":874,"children":875},{"style":853},[876],{"type":50,"value":877},"nam",{"type":44,"tag":665,"props":879,"children":881},{"style":880},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[882],{"type":50,"value":883},"e",{"type":44,"tag":665,"props":885,"children":886},{"style":869},[887],{"type":50,"value":888},">",{"type":44,"tag":665,"props":890,"children":891},{"style":880},[892],{"type":50,"value":893}," [--start ",{"type":44,"tag":665,"props":895,"children":896},{"style":869},[897],{"type":50,"value":898},"\u003C",{"type":44,"tag":665,"props":900,"children":901},{"style":853},[902],{"type":50,"value":903},"is",{"type":44,"tag":665,"props":905,"children":906},{"style":880},[907],{"type":50,"value":908},"o",{"type":44,"tag":665,"props":910,"children":911},{"style":869},[912],{"type":50,"value":888},{"type":44,"tag":665,"props":914,"children":915},{"style":853},[916],{"type":50,"value":917},"]",{"type":44,"tag":665,"props":919,"children":920},{"style":880},[921],{"type":50,"value":922}," [--end ",{"type":44,"tag":665,"props":924,"children":925},{"style":869},[926],{"type":50,"value":898},{"type":44,"tag":665,"props":928,"children":929},{"style":853},[930],{"type":50,"value":903},{"type":44,"tag":665,"props":932,"children":933},{"style":880},[934],{"type":50,"value":908},{"type":44,"tag":665,"props":936,"children":937},{"style":869},[938],{"type":50,"value":888},{"type":44,"tag":665,"props":940,"children":941},{"style":853},[942],{"type":50,"value":917},{"type":44,"tag":665,"props":944,"children":945},{"style":880},[946],{"type":50,"value":947}," [--last-minutes ",{"type":44,"tag":665,"props":949,"children":950},{"style":853},[951],{"type":50,"value":952},"60]",{"type":44,"tag":665,"props":954,"children":955},{"style":880},[956],{"type":50,"value":957}," [--transaction-type ",{"type":44,"tag":665,"props":959,"children":960},{"style":869},[961],{"type":50,"value":898},{"type":44,"tag":665,"props":963,"children":964},{"style":880},[965],{"type":50,"value":966},"t",{"type":44,"tag":665,"props":968,"children":969},{"style":869},[970],{"type":50,"value":888},{"type":44,"tag":665,"props":972,"children":973},{"style":853},[974],{"type":50,"value":917},{"type":44,"tag":665,"props":976,"children":977},{"style":880},[978],{"type":50,"value":979}," [--transaction-name ",{"type":44,"tag":665,"props":981,"children":982},{"style":869},[983],{"type":50,"value":898},{"type":44,"tag":665,"props":985,"children":986},{"style":880},[987],{"type":50,"value":988},"n",{"type":44,"tag":665,"props":990,"children":991},{"style":869},[992],{"type":50,"value":888},{"type":44,"tag":665,"props":994,"children":995},{"style":853},[996],{"type":50,"value":917},{"type":44,"tag":665,"props":998,"children":999},{"style":880},[1000],{"type":50,"value":1001}," [--space ",{"type":44,"tag":665,"props":1003,"children":1004},{"style":869},[1005],{"type":50,"value":898},{"type":44,"tag":665,"props":1007,"children":1008},{"style":853},[1009],{"type":50,"value":1010},"i",{"type":44,"tag":665,"props":1012,"children":1013},{"style":880},[1014],{"type":50,"value":1015},"d",{"type":44,"tag":665,"props":1017,"children":1018},{"style":869},[1019],{"type":50,"value":888},{"type":44,"tag":665,"props":1021,"children":1022},{"style":853},[1023],{"type":50,"value":917},{"type":44,"tag":665,"props":1025,"children":1026},{"style":880},[1027],{"type":50,"value":1028}," [--json]\n",{"type":44,"tag":665,"props":1030,"children":1032},{"class":667,"line":1031},3,[1033],{"type":44,"tag":665,"props":1034,"children":1036},{"emptyLinePlaceholder":1035},true,[1037],{"type":50,"value":1038},"\n",{"type":44,"tag":665,"props":1040,"children":1042},{"class":667,"line":1041},4,[1043],{"type":44,"tag":665,"props":1044,"children":1045},{"style":838},[1046],{"type":50,"value":1047},"# Failed transaction correlations\n",{"type":44,"tag":665,"props":1049,"children":1051},{"class":667,"line":1050},5,[1052,1056,1060,1065,1069,1073,1077,1081,1085,1089,1093,1097,1101,1105,1109,1113,1117,1121,1125,1129,1133,1137,1141,1145,1149,1153,1157,1161,1165,1169,1173,1177,1181,1185,1189,1193,1197,1201,1205],{"type":44,"tag":665,"props":1053,"children":1054},{"style":847},[1055],{"type":50,"value":850},{"type":44,"tag":665,"props":1057,"children":1058},{"style":853},[1059],{"type":50,"value":856},{"type":44,"tag":665,"props":1061,"children":1062},{"style":853},[1063],{"type":50,"value":1064}," failed-correlations",{"type":44,"tag":665,"props":1066,"children":1067},{"style":853},[1068],{"type":50,"value":866},{"type":44,"tag":665,"props":1070,"children":1071},{"style":869},[1072],{"type":50,"value":872},{"type":44,"tag":665,"props":1074,"children":1075},{"style":853},[1076],{"type":50,"value":877},{"type":44,"tag":665,"props":1078,"children":1079},{"style":880},[1080],{"type":50,"value":883},{"type":44,"tag":665,"props":1082,"children":1083},{"style":869},[1084],{"type":50,"value":888},{"type":44,"tag":665,"props":1086,"children":1087},{"style":880},[1088],{"type":50,"value":893},{"type":44,"tag":665,"props":1090,"children":1091},{"style":869},[1092],{"type":50,"value":898},{"type":44,"tag":665,"props":1094,"children":1095},{"style":853},[1096],{"type":50,"value":903},{"type":44,"tag":665,"props":1098,"children":1099},{"style":880},[1100],{"type":50,"value":908},{"type":44,"tag":665,"props":1102,"children":1103},{"style":869},[1104],{"type":50,"value":888},{"type":44,"tag":665,"props":1106,"children":1107},{"style":853},[1108],{"type":50,"value":917},{"type":44,"tag":665,"props":1110,"children":1111},{"style":880},[1112],{"type":50,"value":922},{"type":44,"tag":665,"props":1114,"children":1115},{"style":869},[1116],{"type":50,"value":898},{"type":44,"tag":665,"props":1118,"children":1119},{"style":853},[1120],{"type":50,"value":903},{"type":44,"tag":665,"props":1122,"children":1123},{"style":880},[1124],{"type":50,"value":908},{"type":44,"tag":665,"props":1126,"children":1127},{"style":869},[1128],{"type":50,"value":888},{"type":44,"tag":665,"props":1130,"children":1131},{"style":853},[1132],{"type":50,"value":917},{"type":44,"tag":665,"props":1134,"children":1135},{"style":880},[1136],{"type":50,"value":947},{"type":44,"tag":665,"props":1138,"children":1139},{"style":853},[1140],{"type":50,"value":952},{"type":44,"tag":665,"props":1142,"children":1143},{"style":880},[1144],{"type":50,"value":957},{"type":44,"tag":665,"props":1146,"children":1147},{"style":869},[1148],{"type":50,"value":898},{"type":44,"tag":665,"props":1150,"children":1151},{"style":880},[1152],{"type":50,"value":966},{"type":44,"tag":665,"props":1154,"children":1155},{"style":869},[1156],{"type":50,"value":888},{"type":44,"tag":665,"props":1158,"children":1159},{"style":853},[1160],{"type":50,"value":917},{"type":44,"tag":665,"props":1162,"children":1163},{"style":880},[1164],{"type":50,"value":979},{"type":44,"tag":665,"props":1166,"children":1167},{"style":869},[1168],{"type":50,"value":898},{"type":44,"tag":665,"props":1170,"children":1171},{"style":880},[1172],{"type":50,"value":988},{"type":44,"tag":665,"props":1174,"children":1175},{"style":869},[1176],{"type":50,"value":888},{"type":44,"tag":665,"props":1178,"children":1179},{"style":853},[1180],{"type":50,"value":917},{"type":44,"tag":665,"props":1182,"children":1183},{"style":880},[1184],{"type":50,"value":1001},{"type":44,"tag":665,"props":1186,"children":1187},{"style":869},[1188],{"type":50,"value":898},{"type":44,"tag":665,"props":1190,"children":1191},{"style":853},[1192],{"type":50,"value":1010},{"type":44,"tag":665,"props":1194,"children":1195},{"style":880},[1196],{"type":50,"value":1015},{"type":44,"tag":665,"props":1198,"children":1199},{"style":869},[1200],{"type":50,"value":888},{"type":44,"tag":665,"props":1202,"children":1203},{"style":853},[1204],{"type":50,"value":917},{"type":44,"tag":665,"props":1206,"children":1207},{"style":880},[1208],{"type":50,"value":1028},{"type":44,"tag":665,"props":1210,"children":1212},{"class":667,"line":1211},6,[1213],{"type":44,"tag":665,"props":1214,"children":1215},{"emptyLinePlaceholder":1035},[1216],{"type":50,"value":1038},{"type":44,"tag":665,"props":1218,"children":1220},{"class":667,"line":1219},7,[1221],{"type":44,"tag":665,"props":1222,"children":1223},{"style":838},[1224],{"type":50,"value":1225},"# Test Kibana connection\n",{"type":44,"tag":665,"props":1227,"children":1229},{"class":667,"line":1228},8,[1230,1234,1238,1243,1247,1251,1255,1259,1263],{"type":44,"tag":665,"props":1231,"children":1232},{"style":847},[1233],{"type":50,"value":850},{"type":44,"tag":665,"props":1235,"children":1236},{"style":853},[1237],{"type":50,"value":856},{"type":44,"tag":665,"props":1239,"children":1240},{"style":853},[1241],{"type":50,"value":1242}," test",{"type":44,"tag":665,"props":1244,"children":1245},{"style":880},[1246],{"type":50,"value":1001},{"type":44,"tag":665,"props":1248,"children":1249},{"style":869},[1250],{"type":50,"value":898},{"type":44,"tag":665,"props":1252,"children":1253},{"style":853},[1254],{"type":50,"value":1010},{"type":44,"tag":665,"props":1256,"children":1257},{"style":880},[1258],{"type":50,"value":1015},{"type":44,"tag":665,"props":1260,"children":1261},{"style":869},[1262],{"type":50,"value":888},{"type":44,"tag":665,"props":1264,"children":1265},{"style":853},[1266],{"type":50,"value":1267},"]\n",{"type":44,"tag":53,"props":1269,"children":1270},{},[1271,1276,1277,1283,1284,1290,1292,1298,1300,1306,1308,1314,1315,1321],{"type":44,"tag":70,"props":1272,"children":1273},{},[1274],{"type":50,"value":1275},"Environment:",{"type":50,"value":333},{"type":44,"tag":169,"props":1278,"children":1280},{"className":1279},[],[1281],{"type":50,"value":1282},"KIBANA_URL",{"type":50,"value":176},{"type":44,"tag":169,"props":1285,"children":1287},{"className":1286},[],[1288],{"type":50,"value":1289},"KIBANA_API_KEY",{"type":50,"value":1291}," (or ",{"type":44,"tag":169,"props":1293,"children":1295},{"className":1294},[],[1296],{"type":50,"value":1297},"KIBANA_USERNAME",{"type":50,"value":1299},"\u002F",{"type":44,"tag":169,"props":1301,"children":1303},{"className":1302},[],[1304],{"type":50,"value":1305},"KIBANA_PASSWORD",{"type":50,"value":1307},") for Kibana; for fallback,\n",{"type":44,"tag":169,"props":1309,"children":1311},{"className":1310},[],[1312],{"type":50,"value":1313},"ELASTICSEARCH_URL",{"type":50,"value":176},{"type":44,"tag":169,"props":1316,"children":1318},{"className":1317},[],[1319],{"type":50,"value":1320},"ELASTICSEARCH_API_KEY",{"type":50,"value":1322},". Use the same time range as the investigation.",{"type":44,"tag":78,"props":1324,"children":1326},{"id":1325},"workflow",[1327],{"type":50,"value":1328},"Workflow",{"type":44,"tag":654,"props":1330,"children":1334},{"className":1331,"code":1333,"language":50,"meta":659},[1332],"language-text","Service health progress:\n- [ ] Step 1: Identify the service (and time range)\n- [ ] Step 2: Check SLOs and firing alerts\n- [ ] Step 3: Check ML anomalies (if configured)\n- [ ] Step 4: Review throughput, latency (avg\u002Fp95\u002Fp99), error rate\n- [ ] Step 5: Assess dependency health (ES|QL\u002FAPIs \u002F Kibana repo)\n- [ ] Step 6: Correlate with infrastructure and logs\n- [ ] Step 7: Summarize health and recommend actions\n",[1335],{"type":44,"tag":169,"props":1336,"children":1337},{"__ignoreMap":659},[1338],{"type":50,"value":1333},{"type":44,"tag":1340,"props":1341,"children":1343},"h3",{"id":1342},"step-1-identify-the-service",[1344],{"type":50,"value":1345},"Step 1: Identify the service",{"type":44,"tag":53,"props":1347,"children":1348},{},[1349,1351,1356,1357,1362,1364,1370],{"type":50,"value":1350},"Confirm service name and time range. Resolve the service from the request; if multiple are in scope, target the most\nrelevant. Use ES|QL on ",{"type":44,"tag":169,"props":1352,"children":1354},{"className":1353},[],[1355],{"type":50,"value":174},{"type":50,"value":315},{"type":44,"tag":169,"props":1358,"children":1360},{"className":1359},[],[1361],{"type":50,"value":182},{"type":50,"value":1363}," (e.g.\n",{"type":44,"tag":169,"props":1365,"children":1367},{"className":1366},[],[1368],{"type":50,"value":1369},"WHERE service.name == \"\u003Cname>\"",{"type":50,"value":1371},") or Kibana repo APM routes to obtain service-level data. If the user has not provided\nthe time range, assume last hour.",{"type":44,"tag":1340,"props":1373,"children":1375},{"id":1374},"step-2-check-slos-and-firing-alerts",[1376],{"type":50,"value":1377},"Step 2: Check SLOs and firing alerts",{"type":44,"tag":53,"props":1379,"children":1380},{},[1381,1386,1388,1392,1394,1399,1401,1407,1409,1415,1417,1422,1424,1430],{"type":44,"tag":70,"props":1382,"children":1383},{},[1384],{"type":50,"value":1385},"SLOs:",{"type":50,"value":1387}," Call the ",{"type":44,"tag":70,"props":1389,"children":1390},{},[1391],{"type":50,"value":109},{"type":50,"value":1393}," to get SLO definitions and status for the service (latency, availability),\nhealthy\u002Fdegrading\u002Fviolated, burn rate, error budget. ",{"type":44,"tag":70,"props":1395,"children":1396},{},[1397],{"type":50,"value":1398},"Alerts:",{"type":50,"value":1400}," For active APM alerts, call\n",{"type":44,"tag":169,"props":1402,"children":1404},{"className":1403},[],[1405],{"type":50,"value":1406},"\u002Fapi\u002Falerting\u002Frules\u002F_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active",{"type":50,"value":1408},".\nWhen checking one service, include both rules where ",{"type":44,"tag":169,"props":1410,"children":1412},{"className":1411},[],[1413],{"type":50,"value":1414},"params.serviceName",{"type":50,"value":1416}," matches the service and rules where\n",{"type":44,"tag":169,"props":1418,"children":1420},{"className":1419},[],[1421],{"type":50,"value":1414},{"type":50,"value":1423}," is absent (all-services rules). Do not query ",{"type":44,"tag":169,"props":1425,"children":1427},{"className":1426},[],[1428],{"type":50,"value":1429},".alerts*",{"type":50,"value":1431}," indices for active-state checks. Correlate\nwith SLO violations or metric changes.",{"type":44,"tag":1340,"props":1433,"children":1435},{"id":1434},"step-3-check-ml-anomalies",[1436],{"type":50,"value":1437},"Step 3: Check ML anomalies",{"type":44,"tag":53,"props":1439,"children":1440},{},[1441],{"type":50,"value":1442},"If ML anomaly detection is used, query ML job results or anomaly records (via Elasticsearch ML APIs or indices) for the\nservice and time range. Note high-severity anomalies (latency, throughput, error rate); use anomaly time windows to\nnarrow Steps 4–5.",{"type":44,"tag":1340,"props":1444,"children":1446},{"id":1445},"step-4-review-throughput-latency-and-error-rate",[1447],{"type":50,"value":1448},"Step 4: Review throughput, latency, and error rate",{"type":44,"tag":53,"props":1450,"children":1451},{},[1452,1454,1458,1460,1465,1466,1471,1473,1478,1480,1485,1487,1492,1494,1500,1502,1506],{"type":50,"value":1453},"Use ",{"type":44,"tag":70,"props":1455,"children":1456},{},[1457],{"type":50,"value":74},{"type":50,"value":1459}," against ",{"type":44,"tag":169,"props":1461,"children":1463},{"className":1462},[],[1464],{"type":50,"value":174},{"type":50,"value":315},{"type":44,"tag":169,"props":1467,"children":1469},{"className":1468},[],[1470],{"type":50,"value":182},{"type":50,"value":1472}," for the service and time range to get\n",{"type":44,"tag":70,"props":1474,"children":1475},{},[1476],{"type":50,"value":1477},"throughput",{"type":50,"value":1479}," (e.g. req\u002Fmin), ",{"type":44,"tag":70,"props":1481,"children":1482},{},[1483],{"type":50,"value":1484},"latency",{"type":50,"value":1486}," (avg, p95, p99), ",{"type":44,"tag":70,"props":1488,"children":1489},{},[1490],{"type":50,"value":1491},"error rate",{"type":50,"value":1493}," (failed\u002Ftotal or 5xx\u002Ftotal). Example:\n",{"type":44,"tag":169,"props":1495,"children":1497},{"className":1496},[],[1498],{"type":50,"value":1499},"FROM traces*apm*,traces*otel* | WHERE service.name == \"\u003Cservice_name>\" AND @timestamp >= ... AND @timestamp \u003C= ... | STATS ...",{"type":50,"value":1501},".\nCompare to prior period or SLO targets. See ",{"type":44,"tag":59,"props":1503,"children":1504},{"href":193},[1505],{"type":50,"value":196},{"type":50,"value":238},{"type":44,"tag":1340,"props":1508,"children":1510},{"id":1509},"step-5-assess-dependency-health",[1511],{"type":50,"value":1512},"Step 5: Assess dependency health",{"type":44,"tag":53,"props":1514,"children":1515},{},[1516,1518,1522,1524,1529,1530,1535,1537,1542],{"type":50,"value":1517},"Obtain dependency and service-map data via ",{"type":44,"tag":70,"props":1519,"children":1520},{},[1521],{"type":50,"value":74},{"type":50,"value":1523}," on ",{"type":44,"tag":169,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":50,"value":174},{"type":50,"value":1299},{"type":44,"tag":169,"props":1531,"children":1533},{"className":1532},[],[1534],{"type":50,"value":182},{"type":50,"value":1536}," (e.g.\ndownstream service\u002Fspan aggregations) or via APM route handlers in the ",{"type":44,"tag":70,"props":1538,"children":1539},{},[1540],{"type":50,"value":1541},"Kibana repo",{"type":50,"value":1543}," that expose\ndependency\u002Fservice-map data. For the service and time range, note downstream latency and error rate; flag slow or\nfailing dependencies as likely causes.",{"type":44,"tag":1340,"props":1545,"children":1547},{"id":1546},"step-6-correlate-with-infrastructure-and-logs",[1548],{"type":50,"value":1549},"Step 6: Correlate with infrastructure and logs",{"type":44,"tag":85,"props":1551,"children":1552},{},[1553,1576,1639],{"type":44,"tag":89,"props":1554,"children":1555},{},[1556,1561,1563,1569,1571,1575],{"type":44,"tag":70,"props":1557,"children":1558},{},[1559],{"type":50,"value":1560},"APM Correlations (when only a subpopulation is affected):",{"type":50,"value":1562}," Run\n",{"type":44,"tag":169,"props":1564,"children":1566},{"className":1565},[],[1567],{"type":50,"value":1568},"node skills\u002Fobservability\u002Fservice-health\u002Fscripts\u002Fapm-correlations.js latency-correlations|failed-correlations --service-name \u003Cname> [--start ...] [--end ...]",{"type":50,"value":1570},"\nto get correlated attributes. Filter by those attributes and fetch trace samples or errors to confirm root cause. See\n",{"type":44,"tag":59,"props":1572,"children":1573},{"href":233},[1574],{"type":50,"value":236},{"type":50,"value":238},{"type":44,"tag":89,"props":1577,"children":1578},{},[1579,1583,1584,1588,1590,1595,1596,1601,1602,1607,1609,1613,1615,1620,1621,1626,1628,1632,1633,1637],{"type":44,"tag":70,"props":1580,"children":1581},{},[1582],{"type":50,"value":246},{"type":50,"value":301},{"type":44,"tag":70,"props":1585,"children":1586},{},[1587],{"type":50,"value":253},{"type":50,"value":1589}," from traces (e.g. ",{"type":44,"tag":169,"props":1591,"children":1593},{"className":1592},[],[1594],{"type":50,"value":261},{"type":50,"value":263},{"type":44,"tag":169,"props":1597,"children":1599},{"className":1598},[],[1600],{"type":50,"value":269},{"type":50,"value":263},{"type":44,"tag":169,"props":1603,"children":1605},{"className":1604},[],[1606],{"type":50,"value":276},{"type":50,"value":1608},") and\nquery infrastructure\u002Fmetrics indices with ",{"type":44,"tag":70,"props":1610,"children":1611},{},[1612],{"type":50,"value":74},{"type":50,"value":1614}," or Elasticsearch for ",{"type":44,"tag":70,"props":1616,"children":1617},{},[1618],{"type":50,"value":1619},"CPU",{"type":50,"value":176},{"type":44,"tag":70,"props":1622,"children":1623},{},[1624],{"type":50,"value":1625},"memory",{"type":50,"value":1627},". ",{"type":44,"tag":70,"props":1629,"children":1630},{},[1631],{"type":50,"value":283},{"type":50,"value":176},{"type":44,"tag":70,"props":1634,"children":1635},{},[1636],{"type":50,"value":289},{"type":50,"value":1638}," directly impact APM health; correlate their time windows with APM degradation.",{"type":44,"tag":89,"props":1640,"children":1641},{},[1642,1646,1647,1651,1653,1659,1661,1667],{"type":44,"tag":70,"props":1643,"children":1644},{},[1645],{"type":50,"value":299},{"type":50,"value":301},{"type":44,"tag":70,"props":1648,"children":1649},{},[1650],{"type":50,"value":74},{"type":50,"value":1652}," or Elasticsearch on log indices with ",{"type":44,"tag":169,"props":1654,"children":1656},{"className":1655},[],[1657],{"type":50,"value":1658},"service.name == \"\u003Cservice_name>\"",{"type":50,"value":1660}," or\n",{"type":44,"tag":169,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":50,"value":1666},"trace.id == \"\u003Ctrace_id>\"",{"type":50,"value":1668}," to explain behavior and root cause (exceptions, timeouts, restarts).",{"type":44,"tag":1340,"props":1670,"children":1672},{"id":1671},"step-7-summarize-and-recommend",[1673],{"type":50,"value":1674},"Step 7: Summarize and recommend",{"type":44,"tag":53,"props":1676,"children":1677},{},[1678,1680,1685,1687,1692,1693,1697],{"type":50,"value":1679},"State health (",{"type":44,"tag":70,"props":1681,"children":1682},{},[1683],{"type":50,"value":1684},"healthy",{"type":50,"value":1686}," \u002F ",{"type":44,"tag":70,"props":1688,"children":1689},{},[1690],{"type":50,"value":1691},"degraded",{"type":50,"value":1686},{"type":44,"tag":70,"props":1694,"children":1695},{},[1696],{"type":50,"value":544},{"type":50,"value":1698},") with reasons; list concrete next steps.",{"type":44,"tag":78,"props":1700,"children":1702},{"id":1701},"examples",[1703],{"type":50,"value":1704},"Examples",{"type":44,"tag":1340,"props":1706,"children":1708},{"id":1707},"example-esql-for-a-specific-service",[1709],{"type":50,"value":1710},"Example: ES|QL for a specific service",{"type":44,"tag":53,"props":1712,"children":1713},{},[1714,1716,1721,1723,1729],{"type":50,"value":1715},"Scope with ",{"type":44,"tag":169,"props":1717,"children":1719},{"className":1718},[],[1720],{"type":50,"value":768},{"type":50,"value":1722}," and time range. Throughput and error rate (1-hour buckets; ",{"type":44,"tag":169,"props":1724,"children":1726},{"className":1725},[],[1727],{"type":50,"value":1728},"LIMIT",{"type":50,"value":1730},"\ncaps rows and tokens):",{"type":44,"tag":654,"props":1732,"children":1734},{"className":656,"code":1733,"language":658,"meta":659,"style":659},"FROM traces*apm*,traces*otel*\n| WHERE service.name == \"api-gateway\"\n  AND @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == \"failure\" BY BUCKET(@timestamp, 1 hour)\n| EVAL error_rate = failures \u002F request_count\n| SORT @timestamp\n| LIMIT 500\n",[1735],{"type":44,"tag":169,"props":1736,"children":1737},{"__ignoreMap":659},[1738,1746,1754,1761,1769,1777,1785],{"type":44,"tag":665,"props":1739,"children":1740},{"class":667,"line":668},[1741],{"type":44,"tag":665,"props":1742,"children":1743},{},[1744],{"type":50,"value":1745},"FROM traces*apm*,traces*otel*\n",{"type":44,"tag":665,"props":1747,"children":1748},{"class":667,"line":677},[1749],{"type":44,"tag":665,"props":1750,"children":1751},{},[1752],{"type":50,"value":1753},"| WHERE service.name == \"api-gateway\"\n",{"type":44,"tag":665,"props":1755,"children":1756},{"class":667,"line":1031},[1757],{"type":44,"tag":665,"props":1758,"children":1759},{},[1760],{"type":50,"value":683},{"type":44,"tag":665,"props":1762,"children":1763},{"class":667,"line":1041},[1764],{"type":44,"tag":665,"props":1765,"children":1766},{},[1767],{"type":50,"value":1768},"| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == \"failure\" BY BUCKET(@timestamp, 1 hour)\n",{"type":44,"tag":665,"props":1770,"children":1771},{"class":667,"line":1050},[1772],{"type":44,"tag":665,"props":1773,"children":1774},{},[1775],{"type":50,"value":1776},"| EVAL error_rate = failures \u002F request_count\n",{"type":44,"tag":665,"props":1778,"children":1779},{"class":667,"line":1211},[1780],{"type":44,"tag":665,"props":1781,"children":1782},{},[1783],{"type":50,"value":1784},"| SORT @timestamp\n",{"type":44,"tag":665,"props":1786,"children":1787},{"class":667,"line":1219},[1788],{"type":44,"tag":665,"props":1789,"children":1790},{},[1791],{"type":50,"value":1792},"| LIMIT 500\n",{"type":44,"tag":53,"props":1794,"children":1795},{},[1796,1798,1803],{"type":50,"value":1797},"Latency percentiles and exact field names: see Kibana ",{"type":44,"tag":169,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":50,"value":702},{"type":50,"value":238},{"type":44,"tag":1340,"props":1805,"children":1807},{"id":1806},"example-is-service-x-healthy",[1808],{"type":50,"value":1809},"Example: \"Is service X healthy?\"",{"type":44,"tag":1811,"props":1812,"children":1813},"ol",{},[1814,1849,1854],{"type":44,"tag":89,"props":1815,"children":1816},{},[1817,1819,1823,1824,1828,1830,1834,1836,1841,1842,1847],{"type":50,"value":1818},"Resolve service X and time range. Call ",{"type":44,"tag":70,"props":1820,"children":1821},{},[1822],{"type":50,"value":109},{"type":50,"value":176},{"type":44,"tag":70,"props":1825,"children":1826},{},[1827],{"type":50,"value":133},{"type":50,"value":1829},"; run ",{"type":44,"tag":70,"props":1831,"children":1832},{},[1833],{"type":50,"value":74},{"type":50,"value":1835}," on\n",{"type":44,"tag":169,"props":1837,"children":1839},{"className":1838},[],[1840],{"type":50,"value":174},{"type":50,"value":1299},{"type":44,"tag":169,"props":1843,"children":1845},{"className":1844},[],[1846],{"type":50,"value":182},{"type":50,"value":1848}," for throughput, latency, error rate; query\ndependency\u002Fservice-map data (ES|QL or Kibana repo).",{"type":44,"tag":89,"props":1850,"children":1851},{},[1852],{"type":50,"value":1853},"Evaluate SLO status (violated\u002Fdegrading?), firing rules, ML anomalies, and dependency health.",{"type":44,"tag":89,"props":1855,"children":1856},{},[1857,1859,1864],{"type":50,"value":1858},"Answer: Healthy \u002F Degraded \u002F Unhealthy with reasons and next steps (e.g.\n",{"type":44,"tag":59,"props":1860,"children":1862},{"href":336,"rel":1861},[63],[1863],{"type":50,"value":340},{"type":50,"value":1865},").",{"type":44,"tag":1340,"props":1867,"children":1869},{"id":1868},"example-why-is-service-y-slow",[1870],{"type":50,"value":1871},"Example: \"Why is service Y slow?\"",{"type":44,"tag":1811,"props":1873,"children":1874},{},[1875,1896,1901],{"type":44,"tag":89,"props":1876,"children":1877},{},[1878,1880,1884,1885,1889,1890,1894],{"type":50,"value":1879},"Service Y and slowness time range. Call ",{"type":44,"tag":70,"props":1881,"children":1882},{},[1883],{"type":50,"value":109},{"type":50,"value":176},{"type":44,"tag":70,"props":1886,"children":1887},{},[1888],{"type":50,"value":133},{"type":50,"value":1829},{"type":44,"tag":70,"props":1891,"children":1892},{},[1893],{"type":50,"value":74},{"type":50,"value":1895}," for Y and dependencies;\nquery ML anomaly results.",{"type":44,"tag":89,"props":1897,"children":1898},{},[1899],{"type":50,"value":1900},"Compare latency (avg\u002Fp95\u002Fp99) to prior period via ES|QL; from dependency data identify high-latency or failing deps.",{"type":44,"tag":89,"props":1902,"children":1903},{},[1904],{"type":50,"value":1905},"Summarize (e.g. p99 up; dependency Z elevated) and recommend (investigate Z; Observability Labs for latency).",{"type":44,"tag":1340,"props":1907,"children":1909},{"id":1908},"example-correlate-service-to-infrastructure-opentelemetry",[1910],{"type":50,"value":1911},"Example: Correlate service to infrastructure (OpenTelemetry)",{"type":44,"tag":53,"props":1913,"children":1914},{},[1915,1916,1920],{"type":50,"value":1453},{"type":44,"tag":70,"props":1917,"children":1918},{},[1919],{"type":50,"value":253},{"type":50,"value":1921}," on spans\u002Ftraces to get the runtimes (pods, containers, hosts) for the service. Then check\nCPU and memory for those resources in the same time window as the APM issue:",{"type":44,"tag":85,"props":1923,"children":1924},{},[1925,1956],{"type":44,"tag":89,"props":1926,"children":1927},{},[1928,1930,1935,1936,1942,1943,1948,1950,1955],{"type":50,"value":1929},"From the service’s traces or metrics, read resource attributes such as ",{"type":44,"tag":169,"props":1931,"children":1933},{"className":1932},[],[1934],{"type":50,"value":261},{"type":50,"value":263},{"type":44,"tag":169,"props":1937,"children":1939},{"className":1938},[],[1940],{"type":50,"value":1941},"k8s.namespace.name",{"type":50,"value":68},{"type":44,"tag":169,"props":1944,"children":1946},{"className":1945},[],[1947],{"type":50,"value":269},{"type":50,"value":1949},", or ",{"type":44,"tag":169,"props":1951,"children":1953},{"className":1952},[],[1954],{"type":50,"value":276},{"type":50,"value":238},{"type":44,"tag":89,"props":1957,"children":1958},{},[1959,1961,1965,1967,1972,1973,1978,1979,1985,1987,1992,1994,1999],{"type":50,"value":1960},"Run ",{"type":44,"tag":70,"props":1962,"children":1963},{},[1964],{"type":50,"value":74},{"type":50,"value":1966}," or Elasticsearch search on infrastructure\u002Fmetrics indices filtered by those resource values and the\nincident time range. Check ",{"type":44,"tag":70,"props":1968,"children":1969},{},[1970],{"type":50,"value":1971},"CPU usage",{"type":50,"value":176},{"type":44,"tag":70,"props":1974,"children":1975},{},[1976],{"type":50,"value":1977},"memory consumption",{"type":50,"value":255},{"type":44,"tag":169,"props":1980,"children":1982},{"className":1981},[],[1983],{"type":50,"value":1984},"system.cpu.total.norm.pct",{"type":50,"value":1986},"); look for\n",{"type":44,"tag":70,"props":1988,"children":1989},{},[1990],{"type":50,"value":1991},"OOMKilled",{"type":50,"value":1993}," events, ",{"type":44,"tag":70,"props":1995,"children":1996},{},[1997],{"type":50,"value":1998},"CPU throttling",{"type":50,"value":2000},", or sustained high CPU\u002Fmemory that align with APM latency or error spikes.",{"type":44,"tag":1340,"props":2002,"children":2004},{"id":2003},"example-filter-logs-by-service-or-trace-id",[2005],{"type":50,"value":2006},"Example: Filter logs by service or trace ID",{"type":44,"tag":53,"props":2008,"children":2009},{},[2010],{"type":50,"value":2011},"To understand behavior for a specific service or a single trace, filter logs accordingly:",{"type":44,"tag":85,"props":2013,"children":2014},{},[2015,2038],{"type":44,"tag":89,"props":2016,"children":2017},{},[2018,2023,2025,2029,2031,2036],{"type":44,"tag":70,"props":2019,"children":2020},{},[2021],{"type":50,"value":2022},"By service:",{"type":50,"value":2024}," Run ",{"type":44,"tag":70,"props":2026,"children":2027},{},[2028],{"type":50,"value":74},{"type":50,"value":2030}," or Elasticsearch search on log indices with ",{"type":44,"tag":169,"props":2032,"children":2034},{"className":2033},[],[2035],{"type":50,"value":1658},{"type":50,"value":2037}," and time\nrange to get application logs (errors, warnings, restarts) in the service context.",{"type":44,"tag":89,"props":2039,"children":2040},{},[2041,2046,2048,2053,2055,2060],{"type":44,"tag":70,"props":2042,"children":2043},{},[2044],{"type":50,"value":2045},"By trace ID:",{"type":50,"value":2047}," When investigating a specific request, take the ",{"type":44,"tag":169,"props":2049,"children":2051},{"className":2050},[],[2052],{"type":50,"value":321},{"type":50,"value":2054}," from the APM trace and filter logs by\n",{"type":44,"tag":169,"props":2056,"children":2058},{"className":2057},[],[2059],{"type":50,"value":1666},{"type":50,"value":2061}," (or equivalent field in your log schema). Logs with that trace ID show the full request\npath and help explain failures or latency.",{"type":44,"tag":78,"props":2063,"children":2065},{"id":2064},"guidelines",[2066],{"type":50,"value":2067},"Guidelines",{"type":44,"tag":85,"props":2069,"children":2070},{},[2071,2171,2183,2188,2221],{"type":44,"tag":89,"props":2072,"children":2073},{},[2074,2075,2079,2080,2085,2086,2091,2093,2097,2098,2103,2104,2109,2111,2116,2118,2123,2125,2130,2132,2137,2139,2144,2146,2151,2153,2157,2159,2163,2165,2169],{"type":50,"value":1453},{"type":44,"tag":70,"props":2076,"children":2077},{},[2078],{"type":50,"value":66},{"type":50,"value":97},{"type":44,"tag":59,"props":2081,"children":2083},{"href":113,"rel":2082},[63],[2084],{"type":50,"value":109},{"type":50,"value":68},{"type":44,"tag":59,"props":2087,"children":2089},{"href":138,"rel":2088},[63],[2090],{"type":50,"value":133},{"type":50,"value":2092},") and ",{"type":44,"tag":70,"props":2094,"children":2095},{},[2096],{"type":50,"value":74},{"type":50,"value":1835},{"type":44,"tag":169,"props":2099,"children":2101},{"className":2100},[],[2102],{"type":50,"value":174},{"type":50,"value":1299},{"type":44,"tag":169,"props":2105,"children":2107},{"className":2106},[],[2108],{"type":50,"value":182},{"type":50,"value":2110}," (8.11+ or Serverless), filtering by ",{"type":44,"tag":169,"props":2112,"children":2114},{"className":2113},[],[2115],{"type":50,"value":313},{"type":50,"value":2117}," (and\n",{"type":44,"tag":169,"props":2119,"children":2121},{"className":2120},[],[2122],{"type":50,"value":642},{"type":50,"value":2124}," when relevant). For active APM alerts, call\n",{"type":44,"tag":169,"props":2126,"children":2128},{"className":2127},[],[2129],{"type":50,"value":1406},{"type":50,"value":2131},".\nWhen checking one service, evaluate both rule types: rules where ",{"type":44,"tag":169,"props":2133,"children":2135},{"className":2134},[],[2136],{"type":50,"value":1414},{"type":50,"value":2138}," matches the target service, and\nrules where ",{"type":44,"tag":169,"props":2140,"children":2142},{"className":2141},[],[2143],{"type":50,"value":1414},{"type":50,"value":2145}," is absent (all-services rules). Treat either as applicable to the service before\ndeclaring health. Do not query ",{"type":44,"tag":169,"props":2147,"children":2149},{"className":2148},[],[2150],{"type":50,"value":1429},{"type":50,"value":2152}," indices when determining currently active alerts; use the Alerting API\nresponse above as the source of truth. For APM correlations, run the ",{"type":44,"tag":70,"props":2154,"children":2155},{},[2156],{"type":50,"value":221},{"type":50,"value":2158}," script (see\n",{"type":44,"tag":59,"props":2160,"children":2161},{"href":233},[2162],{"type":50,"value":236},{"type":50,"value":2164},"); for dependency\u002Fservice-map data, use ES|QL or Kibana repo route\nhandlers. For Elasticsearch index and search behavior, see the ",{"type":44,"tag":70,"props":2166,"children":2167},{},[2168],{"type":50,"value":23},{"type":50,"value":2170}," APIs in the Elasticsearch repo.",{"type":44,"tag":89,"props":2172,"children":2173},{},[2174,2176,2181],{"type":50,"value":2175},"Always use the ",{"type":44,"tag":70,"props":2177,"children":2178},{},[2179],{"type":50,"value":2180},"user's time range",{"type":50,"value":2182},"; avoid assuming \"last 1 hour\" if the issue is historical.",{"type":44,"tag":89,"props":2184,"children":2185},{},[2186],{"type":50,"value":2187},"When SLOs exist, anchor the health summary to SLO status and burn rate; when they do not, rely on alerts, anomalies,\nthroughput, latency, error rate, and dependencies.",{"type":44,"tag":89,"props":2189,"children":2190},{},[2191,2193,2198,2200,2205,2207,2212,2214,2219],{"type":50,"value":2192},"When analyzing ",{"type":44,"tag":70,"props":2194,"children":2195},{},[2196],{"type":50,"value":2197},"only application metrics ingested via OpenTelemetry",{"type":50,"value":2199},", use the ES|QL ",{"type":44,"tag":70,"props":2201,"children":2202},{},[2203],{"type":50,"value":2204},"TS",{"type":50,"value":2206}," (time series) command for\nefficient metrics queries. The TS command is available in ",{"type":44,"tag":70,"props":2208,"children":2209},{},[2210],{"type":50,"value":2211},"Elasticsearch 9.3+",{"type":50,"value":2213}," and is ",{"type":44,"tag":70,"props":2215,"children":2216},{},[2217],{"type":50,"value":2218},"always available",{"type":50,"value":2220}," in\nElastic Observability Serverless.",{"type":44,"tag":89,"props":2222,"children":2223},{},[2224],{"type":50,"value":2225},"Summary: one short health verdict plus bullet points for evidence and next steps.",{"type":44,"tag":2227,"props":2228,"children":2229},"style",{},[2230],{"type":50,"value":2231},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2233,"total":2398},[2234,2251,2268,2281,2298,2310,2320,2335,2347,2361,2372,2385],{"slug":2235,"name":2235,"fn":2236,"description":2237,"org":2238,"tags":2239,"stars":2248,"repoUrl":2249,"updatedAt":2250},"accessing-benchmark-results","retrieve and analyze Rally benchmark results","Retrieve Rally benchmark results from an external Elasticsearch metrics store. Use to list past races, get a single race's overall (per-task) results, chart a metric's trend across multiple runs, compare two races, or check whether a run converged — e.g. \"show me recent geonames races\", \"what's the service_time trend for nyc_taxis over the last 30 days?\", \"compare these two race-ids\". Applies when datastore.type = elasticsearch is set in ~\u002F.rally\u002Frally.ini.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2240,2243,2246,2247],{"name":2241,"slug":2242,"type":15},"Analytics","analytics",{"name":2244,"slug":2245,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},2027,"https:\u002F\u002Fgithub.com\u002Felastic\u002Frally","2026-07-12T07:46:38.54144",{"slug":2252,"name":2252,"fn":2253,"description":2254,"org":2255,"tags":2256,"stars":2248,"repoUrl":2249,"updatedAt":2267},"developing-rally","develop and debug Rally source code","Work on Rally's own codebase, not running benchmarks with it. Use when setting up the dev environment, running Rally's tests or linters, navigating its source, debugging Rally's own code, or making changes to Rally itself.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2257,2260,2261,2264],{"name":2258,"slug":2259,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":2262,"slug":2263,"type":15},"Engineering","engineering",{"name":2265,"slug":2266,"type":15},"Local Development","local-development","2026-07-12T07:46:35.976807",{"slug":2269,"name":2269,"fn":2270,"description":2271,"org":2272,"tags":2273,"stars":2248,"repoUrl":2249,"updatedAt":2280},"running-benchmarks","run Rally benchmarks against Elasticsearch","Run Rally benchmarks (races) against Elasticsearch — an existing\u002Fexternal cluster or a Rally-provisioned distribution — and read the summary report. Use when running a race (any pipeline, track, challenge, target-hosts, or auth) or when interpreting throughput, latency, and service_time results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2274,2275,2276,2277],{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":2278,"slug":2279,"type":15},"Testing","testing","2026-07-12T07:46:37.277964",{"slug":2282,"name":2282,"fn":2283,"description":2284,"org":2285,"tags":2286,"stars":25,"repoUrl":26,"updatedAt":2297},"cloud-access-management","manage Elastic Cloud organization access","Manage Elastic Cloud organization access: invite users, assign roles to Serverless projects, and create or revoke Cloud API keys. Use when granting, modifying, or auditing user access.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2287,2290,2291,2294],{"name":2288,"slug":2289,"type":15},"Cloud","cloud",{"name":9,"slug":8,"type":15},{"name":2292,"slug":2293,"type":15},"Operations","operations",{"name":2295,"slug":2296,"type":15},"Permissions","permissions","2026-07-12T07:46:44.946285",{"slug":2299,"name":2299,"fn":2300,"description":2301,"org":2302,"tags":2303,"stars":25,"repoUrl":26,"updatedAt":2309},"cloud-create-project","create Elastic Cloud Serverless projects","Creates Elastic Cloud Serverless projects (Elasticsearch, Observability, or Security) via the REST API, saves credentials to file, and bootstraps a scoped Elasticsearch API key. Use when creating a new serverless project, provisioning a search or observability environment, or spinning up a new Elastic Cloud project.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2304,2305,2308],{"name":2288,"slug":2289,"type":15},{"name":2306,"slug":2307,"type":15},"Deployment","deployment",{"name":23,"slug":24,"type":15},"2026-07-12T07:46:42.353362",{"slug":2311,"name":2311,"fn":2312,"description":2313,"org":2314,"tags":2315,"stars":25,"repoUrl":26,"updatedAt":2319},"cloud-manage-project","manage Elastic Cloud Serverless projects","Manages existing Elastic Cloud Serverless projects: list, get, update, delete, reset credentials, resume, and load saved credentials. Connects to existing projects by resolving endpoints and acquiring scoped Elasticsearch API keys. Use when performing day-2 operations on serverless projects, connecting to an existing project, loading or resetting project credentials, or looking up project details.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2316,2317,2318],{"name":2288,"slug":2289,"type":15},{"name":23,"slug":24,"type":15},{"name":2292,"slug":2293,"type":15},"2026-07-12T07:46:41.097412",{"slug":2321,"name":2321,"fn":2322,"description":2323,"org":2324,"tags":2325,"stars":25,"repoUrl":26,"updatedAt":2334},"cloud-network-security","manage Elastic Cloud network security","Manage Serverless network security (traffic filters): create, update, and delete IP filters and AWS PrivateLink VPC filters. Use when restricting network access or configuring private connectivity.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2326,2327,2328,2331],{"name":2288,"slug":2289,"type":15},{"name":23,"slug":24,"type":15},{"name":2329,"slug":2330,"type":15},"Networking","networking",{"name":2332,"slug":2333,"type":15},"Security","security","2026-07-12T07:46:43.675992",{"slug":2336,"name":2336,"fn":2337,"description":2338,"org":2339,"tags":2340,"stars":25,"repoUrl":26,"updatedAt":2346},"cloud-setup","configure Elastic Cloud authentication","Configures Elastic Cloud authentication and environment defaults. Use when setting up EC_API_KEY, configuring Cloud API access, or when another cloud skill requires credentials.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2341,2344,2345],{"name":2342,"slug":2343,"type":15},"Authentication","authentication",{"name":2288,"slug":2289,"type":15},{"name":23,"slug":24,"type":15},"2026-07-12T07:46:39.783105",{"slug":2348,"name":2348,"fn":2349,"description":2350,"org":2351,"tags":2352,"stars":25,"repoUrl":26,"updatedAt":2360},"elasticsearch-audit","configure Elasticsearch security audit logs","Enable, configure, and query Elasticsearch security audit logs. Use when the task involves audit logging setup, event filtering, or investigating security incidents like failed logins.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2353,2356,2357,2359],{"name":2354,"slug":2355,"type":15},"Audit","audit",{"name":23,"slug":24,"type":15},{"name":529,"slug":2358,"type":15},"logs",{"name":2332,"slug":2333,"type":15},"2026-07-12T07:47:35.092599",{"slug":2362,"name":2362,"fn":2363,"description":2364,"org":2365,"tags":2366,"stars":25,"repoUrl":26,"updatedAt":2371},"elasticsearch-authn","configure Elasticsearch authentication realms","Authenticate to Elasticsearch using native, file-based, LDAP\u002FAD, SAML, OIDC, Kerberos, JWT, or certificate realms. Use when connecting with credentials, choosing a realm, or managing API keys. Assumes the target realms are already configured.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2367,2368,2369,2370],{"name":2342,"slug":2343,"type":15},{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":2332,"slug":2333,"type":15},"2026-07-12T07:47:41.474547",{"slug":2373,"name":2373,"fn":2374,"description":2375,"org":2376,"tags":2377,"stars":25,"repoUrl":26,"updatedAt":2384},"elasticsearch-authz","manage Elasticsearch RBAC and security roles","Manage Elasticsearch RBAC: native users, roles, role mappings, document- and field-level security. Use when creating users or roles, assigning privileges, or mapping external realms like LDAP\u002FSAML.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2378,2379,2380,2383],{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":2381,"slug":2382,"type":15},"RBAC","rbac",{"name":2332,"slug":2333,"type":15},"2026-07-12T07:47:36.394177",{"slug":2386,"name":2386,"fn":2387,"description":2388,"org":2389,"tags":2390,"stars":25,"repoUrl":26,"updatedAt":2397},"elasticsearch-esql","query Elasticsearch data with ES|QL","Execute ES|QL (Elasticsearch Query Language) queries, use when the user wants to query Elasticsearch data, analyze logs, aggregate metrics, explore data, or create charts and dashboards from ES|QL results.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2391,2392,2393,2394],{"name":2241,"slug":2242,"type":15},{"name":2244,"slug":2245,"type":15},{"name":23,"slug":24,"type":15},{"name":2395,"slug":2396,"type":15},"SQL","sql","2026-07-12T07:47:40.249533",86,{"items":2400,"total":2447},[2401,2408,2414,2420,2427,2433,2440],{"slug":2282,"name":2282,"fn":2283,"description":2284,"org":2402,"tags":2403,"stars":25,"repoUrl":26,"updatedAt":2297},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2404,2405,2406,2407],{"name":2288,"slug":2289,"type":15},{"name":9,"slug":8,"type":15},{"name":2292,"slug":2293,"type":15},{"name":2295,"slug":2296,"type":15},{"slug":2299,"name":2299,"fn":2300,"description":2301,"org":2409,"tags":2410,"stars":25,"repoUrl":26,"updatedAt":2309},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2411,2412,2413],{"name":2288,"slug":2289,"type":15},{"name":2306,"slug":2307,"type":15},{"name":23,"slug":24,"type":15},{"slug":2311,"name":2311,"fn":2312,"description":2313,"org":2415,"tags":2416,"stars":25,"repoUrl":26,"updatedAt":2319},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2417,2418,2419],{"name":2288,"slug":2289,"type":15},{"name":23,"slug":24,"type":15},{"name":2292,"slug":2293,"type":15},{"slug":2321,"name":2321,"fn":2322,"description":2323,"org":2421,"tags":2422,"stars":25,"repoUrl":26,"updatedAt":2334},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2423,2424,2425,2426],{"name":2288,"slug":2289,"type":15},{"name":23,"slug":24,"type":15},{"name":2329,"slug":2330,"type":15},{"name":2332,"slug":2333,"type":15},{"slug":2336,"name":2336,"fn":2337,"description":2338,"org":2428,"tags":2429,"stars":25,"repoUrl":26,"updatedAt":2346},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2430,2431,2432],{"name":2342,"slug":2343,"type":15},{"name":2288,"slug":2289,"type":15},{"name":23,"slug":24,"type":15},{"slug":2348,"name":2348,"fn":2349,"description":2350,"org":2434,"tags":2435,"stars":25,"repoUrl":26,"updatedAt":2360},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2436,2437,2438,2439],{"name":2354,"slug":2355,"type":15},{"name":23,"slug":24,"type":15},{"name":529,"slug":2358,"type":15},{"name":2332,"slug":2333,"type":15},{"slug":2362,"name":2362,"fn":2363,"description":2364,"org":2441,"tags":2442,"stars":25,"repoUrl":26,"updatedAt":2371},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2443,2444,2445,2446],{"name":2342,"slug":2343,"type":15},{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":2332,"slug":2333,"type":15},35]