[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-elastic-observability-llm-obs":3,"mdc-3k71ee-key":34,"related-org-elastic-observability-llm-obs":1759,"related-repo-elastic-observability-llm-obs":1929},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"observability-llm-obs","monitor LLMs and agentic applications","Monitor LLMs and agentic apps: performance, token\u002Fcost, response quality, and workflow orchestration. Use when the user asks about LLM monitoring, GenAI observability, or AI cost\u002Fquality.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"elastic","Elastic","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Felastic.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Agents","agents",531,"https:\u002F\u002Fgithub.com\u002Felastic\u002Fagent-skills","2026-07-12T07:47:21.047897",null,41,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"Official Elastic Skills","https:\u002F\u002Fgithub.com\u002Felastic\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fobservability\u002Fskills\u002Fllm-obs","---\nname: observability-llm-obs\ndescription: >\n  Monitor LLMs and agentic apps: performance, token\u002Fcost, response quality, and workflow\n  orchestration. Use when the user asks about LLM monitoring, GenAI observability,\n  or AI cost\u002Fquality.\nmetadata:\n  author: elastic\n  version: 0.1.0\n---\n\n# LLM and Agentic Observability\n\nAnswer user questions about monitoring LLMs and agentic components using **data ingested into Elastic** only. Focus on\nLLM performance, cost and token utilization, response quality, and call chaining or agentic workflow orchestration. Use\n**ES|QL**, Elasticsearch APIs, and (where needed) Kibana APIs. Do not rely on Kibana UI; the skill works without it. A\ngiven deployment typically uses **one or more** ingestion paths (APM\u002FOTLP traces **and\u002For** integration metrics\u002Flogs)—\ndiscover what is available before querying.\n\n## Where to look\n\n- **Trace and metrics data (APM \u002F OTel):** Trace data in Elastic is stored in **`traces*`** when collected by the\n  Elastic APM Agent, and in **`traces-generic.otel-default`** (and similar) when collected by OpenTelemetry. Use the\n  generic pattern **`traces*`** to find all trace data regardless of source. When the application is instrumented with\n  OpenTelemetry (e.g. Elastic\n  [Distributions of OpenTelemetry (EDOT)](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fget-started\u002Fopentelemetry\u002Fuse-cases\u002Fllms),\n  OpenLLMetry, OpenLIT, Langtrace exporting to OTLP), LLM and agent spans land in these trace data streams; metrics may\n  land in **`metrics-apm*`** or metrics-generic. Query **`traces*`** and **`metrics*`** data streams for per-request and\n  aggregated LLM signals.\n- **Integration metrics and logs:** When the user collects data via\n  [Elastic LLM integrations](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapplications\u002Fllm-observability)\n  (OpenAI, Azure OpenAI, Azure AI Foundry, Amazon Bedrock, Bedrock AgentCore, GCP Vertex AI, etc.), metrics and logs go\n  to **integration data streams** (e.g. `metrics*`, `logs*` with dataset\u002Fnamespace per integration). Check which data\n  streams exist.\n- **Discover first:** Use Elasticsearch to list data streams or indices (e.g. `GET _data_stream`, or\n  `GET traces*\u002F_mapping`, `GET metrics*\u002F_mapping`) and optionally sample a document to see which LLM-related fields are\n  present. Do not assume both APM and integration data exist.\n- **ES|QL:** Use the **elasticsearch-esql** skill for ES|QL syntax, commands, and query patterns when building queries\n  against `traces*` or metrics data streams.\n- **Alerts and SLOs:** Use the [Observability APIs](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapis) **SLOs\n  API** ([Stack](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-slo) |\n  [Serverless](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-slo)) and **Alerting API**\n  ([Stack](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-alerting) |\n  [Serverless](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-alerting)) to find SLOs and alerting rules\n  that target LLM-related data (e.g. services backed by `traces*`, or integration metrics). Firing alerts or\n  violated\u002Fdegrading SLOs point to potential degraded performance.\n\n## Data available in Elastic\n\n### From traces and metrics (traces*, metrics-apm* \u002F metrics-generic)\n\nSpans from OTel\u002FEDOT (and compatible SDKs) carry **span attributes** that may follow\n[OpenTelemetry GenAI semantic conventions](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fsemconv\u002Fgen-ai\u002Fgen-ai-spans\u002F) or\nprovider-specific names. In Elasticsearch, attributes typically appear under `span.attributes` (exact key names depend\non ingestion). Common attributes:\n\n| Purpose              | Example attribute names (OTel GenAI)                      |\n| -------------------- | --------------------------------------------------------- |\n| Operation \u002F provider | `gen_ai.operation.name`, `gen_ai.provider.name`           |\n| Model                | `gen_ai.request.model`, `gen_ai.response.model`           |\n| Token usage          | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |\n| Request config       | `gen_ai.request.temperature`, `gen_ai.request.max_tokens` |\n| Errors               | `error.type`                                              |\n| Conversation \u002F agent | `gen_ai.conversation.id`; tool\u002Fagent spans as child spans |\n\nCost is **not** in the OTel spec; some instrumentations add custom attributes (e.g. `llm.response.cost.usd_estimate`).\nDiscover actual field names from the index mapping or a sample document (e.g. `span.attributes.*` or flattened keys).\n\nUse **duration** and **event.outcome** on spans for latency and success\u002Ffailure. Use **trace.id**, **span.id**, and\nparent\u002Fchild span relationships to analyze **call chaining** and agentic workflows (e.g. one root span, multiple LLM or\ntool-call child spans).\n\n### From LLM integrations\n\nIntegrations (OpenAI, Azure OpenAI, Azure AI Foundry, Bedrock, Bedrock AgentCore, Vertex AI, etc.) ship **metrics** (and\nwhere supported **logs**) to Elastic. Metrics typically include token usage, request counts, latency, and—where the\nintegration supports it—cost-related fields. Logs may include prompt\u002Fresponse or guardrail events. Exact field names and\ndata streams are defined by each integration package; discover them from the integration docs or from the target data\nstream mapping.\n\n## Determine what data is available\n\n1. **List data streams:** `GET _data_stream` and filter for `traces*`, `metrics-apm*` (or `metrics*`), and `metrics-*` \u002F\n   `logs-*` that match known LLM integration datasets (e.g. from\n   [Elastic LLM observability](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapplications\u002Fllm-observability)).\n2. **Inspect trace indices:** For `traces*`, run a small search or use mapping to see if spans contain `gen_ai.*` or\n   `llm.*` (or similar) attributes. Confirm presence of token, model, and duration fields.\n3. **Inspect integration indices:** For metrics\u002Flogs data streams, check mapping or one document to see token, cost,\n   latency, and model dimensions.\n4. **Use one source per use case:** If both APM and integration data exist, prefer one consistent source for a given\n   question (e.g. use traces for per-request chain analysis, integration metrics for aggregate token\u002Fcost).\n5. **Check alerts and SLOs:** Use the SLOs API and Alerting API to list SLOs and alerting rules that target LLM-related\n   services or integration metrics, and to get open or recently fired alerts. Firing alerts or SLOs in\n   degrading\u002Fviolated status point to potential degraded performance.\n\n## Use cases and query patterns\n\n### LLM performance (latency, throughput, errors)\n\n- **Traces:** ES|QL on `traces*` filtered by span attributes (e.g. `gen_ai.operation.name` or `gen_ai.provider.name`\n  when present). Compute throughput (count per time bucket), latency (e.g. `duration.us` or span duration), and error\n  rate (`event.outcome == \"failure\"`) by model, service, or time.\n- **Integrations:** Query integration metrics for request rate, latency, and error metrics by model\u002Fdimension as exposed\n  by the integration.\n\n### Cost and token utilization\n\n- **Traces:** Aggregate from spans in `traces*`: sum `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` (or\n  equivalent attribute names) by time, model, or service. If a cost attribute exists (e.g. custom\n  `llm.response.cost.*`), sum it for cost views.\n- **Integrations:** Use integration metrics that expose token counts and\u002For cost; aggregate by time and model.\n\n### Response quality and safety\n\n- **Traces:** Use `event.outcome`, `error.type`, and span attributes (e.g. `gen_ai.response.finish_reasons`) in\n  `traces*` to identify failures, timeouts, or content filters. Correlate with prompts\u002Fresponses if captured in\n  attributes (e.g. `gen_ai.input.messages`, `gen_ai.output.messages`) and not redacted.\n- **Integrations:** Query integration logs for guardrail blocks, content filter events, or policy violations (e.g.\n  [Bedrock Guardrails](https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs\u002Fblog\u002Fllm-observability-amazon-bedrock-guardrails))\n  using the fields defined by that integration.\n\n### Call chaining and agentic workflow orchestration\n\n- **Traces only:** Use **trace hierarchy** in `traces*`. Filter by root service or trace attributes; group by `trace.id`\n  and use parent\u002Fchild span relationships (e.g. `parent.id`, `span.id`) to reconstruct chains (e.g. orchestration span →\n  multiple LLM or tool-call spans). Aggregate by span name or `gen_ai.operation.name` to see distribution of steps (e.g.\n  retrieval, LLM, tool use). Duration per span and per trace gives bottleneck and end-to-end latency.\n\n## Using ES|QL for LLM data\n\n- **Availability:** ES|QL is available in Elasticsearch 8.11+ (GA in 8.14) and in Elastic Observability Serverless.\n- **Scoping:** Always restrict by time range (`@timestamp`). When present, add `service.name` and optionally\n  `service.environment`. For LLM-specific spans, filter by span attributes once you know the field names (e.g. a keyword\n  field for `gen_ai.provider.name` or `gen_ai.operation.name`).\n- **Performance:** Use `LIMIT`, coarse time buckets when only trends are needed, and avoid full scans over large\n  windows.\n\n## Workflow\n\n```text\nLLM observability progress:\n- [ ] Step 1: Determine available data (traces*, metrics-apm* or metrics*, or integration data streams)\n- [ ] Step 2: Discover LLM-related field names (mapping or sample doc)\n- [ ] Step 3: Run ES|QL or Elasticsearch queries for the user's question (performance, cost, quality, orchestration)\n- [ ] Step 4: Check for active alerts or SLOs defined on LLM-related data (Alerting API, SLOs API); field names from\n        Step 2 help identify related rules; firing alerts or violated\u002Fdegrading SLOs indicate potential degraded performance\n- [ ] Step 5: Summarize findings from ingested data only; include alert\u002FSLO status when relevant\n```\n\n## Examples\n\n### Example: Token usage over time from traces\n\nAssume span attributes are available as `span.attributes.gen_ai.usage.input_tokens` and\n`span.attributes.gen_ai.usage.output_tokens` (adjust to actual field names from mapping):\n\n```esql\nFROM traces*\n| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n  AND span.attributes.gen_ai.provider.name IS NOT NULL\n| STATS\n    input_tokens = SUM(span.attributes.gen_ai.usage.input_tokens),\n    output_tokens = SUM(span.attributes.gen_ai.usage.output_tokens)\n  BY BUCKET(@timestamp, 1 hour), span.attributes.gen_ai.request.model\n| SORT @timestamp\n| LIMIT 500\n```\n\n### Example: Latency and error rate by model\n\n```esql\nFROM traces*\n| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n  AND span.attributes.gen_ai.request.model IS NOT NULL\n| STATS\n    request_count = COUNT(*),\n    failures = COUNT(*) WHERE event.outcome == \"failure\",\n    avg_duration_us = AVG(span.duration.us)\n  BY span.attributes.gen_ai.request.model\n| EVAL error_rate = failures \u002F request_count\n| LIMIT 100\n```\n\n### Example: Agentic workflow (trace-level view)\n\nGet trace IDs that contain at least one LLM span and count spans per trace to see chain length:\n\n```esql\nFROM traces*\n| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n  AND span.attributes.gen_ai.operation.name IS NOT NULL\n| STATS span_count = COUNT(*), total_duration_us = SUM(span.duration.us) BY trace.id\n| WHERE span_count > 1\n| SORT total_duration_us DESC\n| LIMIT 50\n```\n\n### Example: Integration metrics (Amazon Bedrock AgentCore)\n\nThe [Amazon Bedrock AgentCore integration](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Freference\u002Fintegrations\u002Faws_bedrock_agentcore)\nships metrics to the `metrics-aws_bedrock_agentcore.metrics-*` data stream (time series index). Use **`TS`** for\naggregations on time series data streams (Elasticsearch 9.2+); use a time range with **`TRANGE`** (9.3+). The\nintegration’s dashboards and\n[alerting rule templates](https:\u002F\u002Fgithub.com\u002Felastic\u002Fintegrations\u002Ftree\u002Fmain\u002Fpackages\u002Faws_bedrock_agentcore\u002Fkibana\u002Falerting_rule_template)\nExample: token usage (counter), invocations (counter), and average latency (gauge) by hour and agent:\n\n```esql\nTS metrics-aws_bedrock_agentcore.metrics-*\n| WHERE TRANGE(7 days)\n  AND aws.dimensions.Operation == \"InvokeAgentRuntime\"\n| STATS\n    total_tokens = SUM(RATE(aws.bedrock_agentcore.metrics.TokenCount.sum)),\n    total_invocations = SUM(RATE(aws.bedrock_agentcore.metrics.Invocations.sum)),\n    avg_latency_ms = AVG(AVG_OVER_TIME(aws.bedrock_agentcore.metrics.Latency.avg))\n  BY TBUCKET(1 hour), aws.bedrock_agentcore.agent_name\n| SORT TBUCKET(1 hour) DESC\n```\n\nFor Elasticsearch 8.x or when `TS` is not available, use `FROM` with `BUCKET(@timestamp, 1 hour)` and `SUM`\u002F`AVG` over\nthe metric fields (as in the integration's alert rule templates). For other LLM integrations (OpenAI, Azure OpenAI,\nVertex AI, etc.), use that integration’s data stream index pattern and field names from its package (see\n[Elastic LLM observability](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapplications\u002Fllm-observability)).\n\n## Guidelines\n\n- **Data only in Elastic:** Use only data collected and stored in Elastic (traces in `traces*`, metrics, or integration\n  metrics\u002Flogs). Do not describe or rely on other vendors’ UIs or products.\n- **One technology per customer:** Assume a single ingestion path per deployment when answering; discover which (traces\n  vs integration) exists and use it consistently for the question.\n- **Discover field names:** Before writing ES|QL or Query DSL, confirm LLM-related attribute or metric names from\n  `_mapping` or a sample document; naming may differ (e.g. `gen_ai.*` vs `llm.*` or integration-specific fields).\n- **No Kibana UI dependency:** Prefer ES|QL and Elasticsearch APIs; use Kibana APIs only when needed (e.g. SLO,\n  alerting). Do not instruct the user to open Kibana UI.\n- **References:**\n  [LLM and agentic AI observability](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapplications\u002Fllm-observability),\n  [Observability Labs – LLM Observability](https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs\u002Fblog\u002Ftag\u002Fllmobs),\n  [OpenTelemetry GenAI spans](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fsemconv\u002Fgen-ai\u002Fgen-ai-spans\u002F). For ES|QL syntax and\n  query patterns, use the **elasticsearch-esql** skill, or look through\n  [ES|QL TS command reference](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Freference\u002Fquery-languages\u002Fesql\u002Fcommands\u002Fts) for Elastic v9.3\n  or higher and for Serverless, and look through\n  [ES|QL FROM command reference](https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Freference\u002Fquery-languages\u002Fesql\u002Fcommands\u002Ffrom) for other\n  Elastic versions.\n",{"data":35,"body":38},{"name":4,"description":6,"metadata":36},{"author":8,"version":37},"0.1.0",{"type":39,"children":40},"root",[41,50,85,92,353,359,366,395,555,583,621,627,646,652,779,785,791,851,857,906,912,985,991,1045,1051,1127,1133,1145,1151,1157,1178,1270,1276,1361,1367,1372,1433,1439,1492,1570,1619,1625,1753],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"llm-and-agentic-observability",[47],{"type":48,"value":49},"text","LLM and Agentic Observability",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54,56,62,64,69,71,76,78,83],{"type":48,"value":55},"Answer user questions about monitoring LLMs and agentic components using ",{"type":42,"tag":57,"props":58,"children":59},"strong",{},[60],{"type":48,"value":61},"data ingested into Elastic",{"type":48,"value":63}," only. Focus on\nLLM performance, cost and token utilization, response quality, and call chaining or agentic workflow orchestration. Use\n",{"type":42,"tag":57,"props":65,"children":66},{},[67],{"type":48,"value":68},"ES|QL",{"type":48,"value":70},", Elasticsearch APIs, and (where needed) Kibana APIs. Do not rely on Kibana UI; the skill works without it. A\ngiven deployment typically uses ",{"type":42,"tag":57,"props":72,"children":73},{},[74],{"type":48,"value":75},"one or more",{"type":48,"value":77}," ingestion paths (APM\u002FOTLP traces ",{"type":42,"tag":57,"props":79,"children":80},{},[81],{"type":48,"value":82},"and\u002For",{"type":48,"value":84}," integration metrics\u002Flogs)—\ndiscover what is available before querying.",{"type":42,"tag":86,"props":87,"children":89},"h2",{"id":88},"where-to-look",[90],{"type":48,"value":91},"Where to look",{"type":42,"tag":93,"props":94,"children":95},"ul",{},[96,183,224,257,281],{"type":42,"tag":97,"props":98,"children":99},"li",{},[100,105,107,117,119,128,130,138,140,149,151,160,162,170,172,181],{"type":42,"tag":57,"props":101,"children":102},{},[103],{"type":48,"value":104},"Trace and metrics data (APM \u002F OTel):",{"type":48,"value":106}," Trace data in Elastic is stored in ",{"type":42,"tag":57,"props":108,"children":109},{},[110],{"type":42,"tag":111,"props":112,"children":114},"code",{"className":113},[],[115],{"type":48,"value":116},"traces*",{"type":48,"value":118}," when collected by the\nElastic APM Agent, and in ",{"type":42,"tag":57,"props":120,"children":121},{},[122],{"type":42,"tag":111,"props":123,"children":125},{"className":124},[],[126],{"type":48,"value":127},"traces-generic.otel-default",{"type":48,"value":129}," (and similar) when collected by OpenTelemetry. Use the\ngeneric pattern ",{"type":42,"tag":57,"props":131,"children":132},{},[133],{"type":42,"tag":111,"props":134,"children":136},{"className":135},[],[137],{"type":48,"value":116},{"type":48,"value":139}," to find all trace data regardless of source. When the application is instrumented with\nOpenTelemetry (e.g. Elastic\n",{"type":42,"tag":141,"props":142,"children":146},"a",{"href":143,"rel":144},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fget-started\u002Fopentelemetry\u002Fuse-cases\u002Fllms",[145],"nofollow",[147],{"type":48,"value":148},"Distributions of OpenTelemetry (EDOT)",{"type":48,"value":150},",\nOpenLLMetry, OpenLIT, Langtrace exporting to OTLP), LLM and agent spans land in these trace data streams; metrics may\nland in ",{"type":42,"tag":57,"props":152,"children":153},{},[154],{"type":42,"tag":111,"props":155,"children":157},{"className":156},[],[158],{"type":48,"value":159},"metrics-apm*",{"type":48,"value":161}," or metrics-generic. Query ",{"type":42,"tag":57,"props":163,"children":164},{},[165],{"type":42,"tag":111,"props":166,"children":168},{"className":167},[],[169],{"type":48,"value":116},{"type":48,"value":171}," and ",{"type":42,"tag":57,"props":173,"children":174},{},[175],{"type":42,"tag":111,"props":176,"children":178},{"className":177},[],[179],{"type":48,"value":180},"metrics*",{"type":48,"value":182}," data streams for per-request and\naggregated LLM signals.",{"type":42,"tag":97,"props":184,"children":185},{},[186,191,193,200,202,207,209,214,216,222],{"type":42,"tag":57,"props":187,"children":188},{},[189],{"type":48,"value":190},"Integration metrics and logs:",{"type":48,"value":192}," When the user collects data via\n",{"type":42,"tag":141,"props":194,"children":197},{"href":195,"rel":196},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapplications\u002Fllm-observability",[145],[198],{"type":48,"value":199},"Elastic LLM integrations",{"type":48,"value":201},"\n(OpenAI, Azure OpenAI, Azure AI Foundry, Amazon Bedrock, Bedrock AgentCore, GCP Vertex AI, etc.), metrics and logs go\nto ",{"type":42,"tag":57,"props":203,"children":204},{},[205],{"type":48,"value":206},"integration data streams",{"type":48,"value":208}," (e.g. ",{"type":42,"tag":111,"props":210,"children":212},{"className":211},[],[213],{"type":48,"value":180},{"type":48,"value":215},", ",{"type":42,"tag":111,"props":217,"children":219},{"className":218},[],[220],{"type":48,"value":221},"logs*",{"type":48,"value":223}," with dataset\u002Fnamespace per integration). Check which data\nstreams exist.",{"type":42,"tag":97,"props":225,"children":226},{},[227,232,234,240,242,248,249,255],{"type":42,"tag":57,"props":228,"children":229},{},[230],{"type":48,"value":231},"Discover first:",{"type":48,"value":233}," Use Elasticsearch to list data streams or indices (e.g. ",{"type":42,"tag":111,"props":235,"children":237},{"className":236},[],[238],{"type":48,"value":239},"GET _data_stream",{"type":48,"value":241},", or\n",{"type":42,"tag":111,"props":243,"children":245},{"className":244},[],[246],{"type":48,"value":247},"GET traces*\u002F_mapping",{"type":48,"value":215},{"type":42,"tag":111,"props":250,"children":252},{"className":251},[],[253],{"type":48,"value":254},"GET metrics*\u002F_mapping",{"type":48,"value":256},") and optionally sample a document to see which LLM-related fields are\npresent. Do not assume both APM and integration data exist.",{"type":42,"tag":97,"props":258,"children":259},{},[260,265,267,272,274,279],{"type":42,"tag":57,"props":261,"children":262},{},[263],{"type":48,"value":264},"ES|QL:",{"type":48,"value":266}," Use the ",{"type":42,"tag":57,"props":268,"children":269},{},[270],{"type":48,"value":271},"elasticsearch-esql",{"type":48,"value":273}," skill for ES|QL syntax, commands, and query patterns when building queries\nagainst ",{"type":42,"tag":111,"props":275,"children":277},{"className":276},[],[278],{"type":48,"value":116},{"type":48,"value":280}," or metrics data streams.",{"type":42,"tag":97,"props":282,"children":283},{},[284,289,290,297,299,304,306,313,315,322,324,329,331,337,338,344,346,351],{"type":42,"tag":57,"props":285,"children":286},{},[287],{"type":48,"value":288},"Alerts and SLOs:",{"type":48,"value":266},{"type":42,"tag":141,"props":291,"children":294},{"href":292,"rel":293},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fsolutions\u002Fobservability\u002Fapis",[145],[295],{"type":48,"value":296},"Observability APIs",{"type":48,"value":298}," ",{"type":42,"tag":57,"props":300,"children":301},{},[302],{"type":48,"value":303},"SLOs\nAPI",{"type":48,"value":305}," (",{"type":42,"tag":141,"props":307,"children":310},{"href":308,"rel":309},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-slo",[145],[311],{"type":48,"value":312},"Stack",{"type":48,"value":314}," |\n",{"type":42,"tag":141,"props":316,"children":319},{"href":317,"rel":318},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-slo",[145],[320],{"type":48,"value":321},"Serverless",{"type":48,"value":323},") and ",{"type":42,"tag":57,"props":325,"children":326},{},[327],{"type":48,"value":328},"Alerting API",{"type":48,"value":330},"\n(",{"type":42,"tag":141,"props":332,"children":335},{"href":333,"rel":334},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fkibana\u002Fgroup\u002Fendpoint-alerting",[145],[336],{"type":48,"value":312},{"type":48,"value":314},{"type":42,"tag":141,"props":339,"children":342},{"href":340,"rel":341},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fapi\u002Fdoc\u002Fserverless\u002Fgroup\u002Fendpoint-alerting",[145],[343],{"type":48,"value":321},{"type":48,"value":345},") to find SLOs and alerting rules\nthat target LLM-related data (e.g. services backed by ",{"type":42,"tag":111,"props":347,"children":349},{"className":348},[],[350],{"type":48,"value":116},{"type":48,"value":352},", or integration metrics). Firing alerts or\nviolated\u002Fdegrading SLOs point to potential degraded performance.",{"type":42,"tag":86,"props":354,"children":356},{"id":355},"data-available-in-elastic",[357],{"type":48,"value":358},"Data available in Elastic",{"type":42,"tag":360,"props":361,"children":363},"h3",{"id":362},"from-traces-and-metrics-traces-metrics-apm-metrics-generic",[364],{"type":48,"value":365},"From traces and metrics (traces*, metrics-apm* \u002F metrics-generic)",{"type":42,"tag":51,"props":367,"children":368},{},[369,371,376,378,385,387,393],{"type":48,"value":370},"Spans from OTel\u002FEDOT (and compatible SDKs) carry ",{"type":42,"tag":57,"props":372,"children":373},{},[374],{"type":48,"value":375},"span attributes",{"type":48,"value":377}," that may follow\n",{"type":42,"tag":141,"props":379,"children":382},{"href":380,"rel":381},"https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fsemconv\u002Fgen-ai\u002Fgen-ai-spans\u002F",[145],[383],{"type":48,"value":384},"OpenTelemetry GenAI semantic conventions",{"type":48,"value":386}," or\nprovider-specific names. In Elasticsearch, attributes typically appear under ",{"type":42,"tag":111,"props":388,"children":390},{"className":389},[],[391],{"type":48,"value":392},"span.attributes",{"type":48,"value":394}," (exact key names depend\non ingestion). Common attributes:",{"type":42,"tag":396,"props":397,"children":398},"table",{},[399,418],{"type":42,"tag":400,"props":401,"children":402},"thead",{},[403],{"type":42,"tag":404,"props":405,"children":406},"tr",{},[407,413],{"type":42,"tag":408,"props":409,"children":410},"th",{},[411],{"type":48,"value":412},"Purpose",{"type":42,"tag":408,"props":414,"children":415},{},[416],{"type":48,"value":417},"Example attribute names (OTel GenAI)",{"type":42,"tag":419,"props":420,"children":421},"tbody",{},[422,447,471,495,519,536],{"type":42,"tag":404,"props":423,"children":424},{},[425,431],{"type":42,"tag":426,"props":427,"children":428},"td",{},[429],{"type":48,"value":430},"Operation \u002F provider",{"type":42,"tag":426,"props":432,"children":433},{},[434,440,441],{"type":42,"tag":111,"props":435,"children":437},{"className":436},[],[438],{"type":48,"value":439},"gen_ai.operation.name",{"type":48,"value":215},{"type":42,"tag":111,"props":442,"children":444},{"className":443},[],[445],{"type":48,"value":446},"gen_ai.provider.name",{"type":42,"tag":404,"props":448,"children":449},{},[450,455],{"type":42,"tag":426,"props":451,"children":452},{},[453],{"type":48,"value":454},"Model",{"type":42,"tag":426,"props":456,"children":457},{},[458,464,465],{"type":42,"tag":111,"props":459,"children":461},{"className":460},[],[462],{"type":48,"value":463},"gen_ai.request.model",{"type":48,"value":215},{"type":42,"tag":111,"props":466,"children":468},{"className":467},[],[469],{"type":48,"value":470},"gen_ai.response.model",{"type":42,"tag":404,"props":472,"children":473},{},[474,479],{"type":42,"tag":426,"props":475,"children":476},{},[477],{"type":48,"value":478},"Token usage",{"type":42,"tag":426,"props":480,"children":481},{},[482,488,489],{"type":42,"tag":111,"props":483,"children":485},{"className":484},[],[486],{"type":48,"value":487},"gen_ai.usage.input_tokens",{"type":48,"value":215},{"type":42,"tag":111,"props":490,"children":492},{"className":491},[],[493],{"type":48,"value":494},"gen_ai.usage.output_tokens",{"type":42,"tag":404,"props":496,"children":497},{},[498,503],{"type":42,"tag":426,"props":499,"children":500},{},[501],{"type":48,"value":502},"Request config",{"type":42,"tag":426,"props":504,"children":505},{},[506,512,513],{"type":42,"tag":111,"props":507,"children":509},{"className":508},[],[510],{"type":48,"value":511},"gen_ai.request.temperature",{"type":48,"value":215},{"type":42,"tag":111,"props":514,"children":516},{"className":515},[],[517],{"type":48,"value":518},"gen_ai.request.max_tokens",{"type":42,"tag":404,"props":520,"children":521},{},[522,527],{"type":42,"tag":426,"props":523,"children":524},{},[525],{"type":48,"value":526},"Errors",{"type":42,"tag":426,"props":528,"children":529},{},[530],{"type":42,"tag":111,"props":531,"children":533},{"className":532},[],[534],{"type":48,"value":535},"error.type",{"type":42,"tag":404,"props":537,"children":538},{},[539,544],{"type":42,"tag":426,"props":540,"children":541},{},[542],{"type":48,"value":543},"Conversation \u002F agent",{"type":42,"tag":426,"props":545,"children":546},{},[547,553],{"type":42,"tag":111,"props":548,"children":550},{"className":549},[],[551],{"type":48,"value":552},"gen_ai.conversation.id",{"type":48,"value":554},"; tool\u002Fagent spans as child spans",{"type":42,"tag":51,"props":556,"children":557},{},[558,560,565,567,573,575,581],{"type":48,"value":559},"Cost is ",{"type":42,"tag":57,"props":561,"children":562},{},[563],{"type":48,"value":564},"not",{"type":48,"value":566}," in the OTel spec; some instrumentations add custom attributes (e.g. ",{"type":42,"tag":111,"props":568,"children":570},{"className":569},[],[571],{"type":48,"value":572},"llm.response.cost.usd_estimate",{"type":48,"value":574},").\nDiscover actual field names from the index mapping or a sample document (e.g. ",{"type":42,"tag":111,"props":576,"children":578},{"className":577},[],[579],{"type":48,"value":580},"span.attributes.*",{"type":48,"value":582}," or flattened keys).",{"type":42,"tag":51,"props":584,"children":585},{},[586,588,593,594,599,601,606,607,612,614,619],{"type":48,"value":587},"Use ",{"type":42,"tag":57,"props":589,"children":590},{},[591],{"type":48,"value":592},"duration",{"type":48,"value":171},{"type":42,"tag":57,"props":595,"children":596},{},[597],{"type":48,"value":598},"event.outcome",{"type":48,"value":600}," on spans for latency and success\u002Ffailure. Use ",{"type":42,"tag":57,"props":602,"children":603},{},[604],{"type":48,"value":605},"trace.id",{"type":48,"value":215},{"type":42,"tag":57,"props":608,"children":609},{},[610],{"type":48,"value":611},"span.id",{"type":48,"value":613},", and\nparent\u002Fchild span relationships to analyze ",{"type":42,"tag":57,"props":615,"children":616},{},[617],{"type":48,"value":618},"call chaining",{"type":48,"value":620}," and agentic workflows (e.g. one root span, multiple LLM or\ntool-call child spans).",{"type":42,"tag":360,"props":622,"children":624},{"id":623},"from-llm-integrations",[625],{"type":48,"value":626},"From LLM integrations",{"type":42,"tag":51,"props":628,"children":629},{},[630,632,637,639,644],{"type":48,"value":631},"Integrations (OpenAI, Azure OpenAI, Azure AI Foundry, Bedrock, Bedrock AgentCore, Vertex AI, etc.) ship ",{"type":42,"tag":57,"props":633,"children":634},{},[635],{"type":48,"value":636},"metrics",{"type":48,"value":638}," (and\nwhere supported ",{"type":42,"tag":57,"props":640,"children":641},{},[642],{"type":48,"value":643},"logs",{"type":48,"value":645},") to Elastic. Metrics typically include token usage, request counts, latency, and—where the\nintegration supports it—cost-related fields. Logs may include prompt\u002Fresponse or guardrail events. Exact field names and\ndata streams are defined by each integration package; discover them from the integration docs or from the target data\nstream mapping.",{"type":42,"tag":86,"props":647,"children":649},{"id":648},"determine-what-data-is-available",[650],{"type":48,"value":651},"Determine what data is available",{"type":42,"tag":653,"props":654,"children":655},"ol",{},[656,716,749,759,769],{"type":42,"tag":97,"props":657,"children":658},{},[659,664,665,670,672,677,678,683,685,690,692,698,700,706,708,714],{"type":42,"tag":57,"props":660,"children":661},{},[662],{"type":48,"value":663},"List data streams:",{"type":48,"value":298},{"type":42,"tag":111,"props":666,"children":668},{"className":667},[],[669],{"type":48,"value":239},{"type":48,"value":671}," and filter for ",{"type":42,"tag":111,"props":673,"children":675},{"className":674},[],[676],{"type":48,"value":116},{"type":48,"value":215},{"type":42,"tag":111,"props":679,"children":681},{"className":680},[],[682],{"type":48,"value":159},{"type":48,"value":684}," (or ",{"type":42,"tag":111,"props":686,"children":688},{"className":687},[],[689],{"type":48,"value":180},{"type":48,"value":691},"), and ",{"type":42,"tag":111,"props":693,"children":695},{"className":694},[],[696],{"type":48,"value":697},"metrics-*",{"type":48,"value":699}," \u002F\n",{"type":42,"tag":111,"props":701,"children":703},{"className":702},[],[704],{"type":48,"value":705},"logs-*",{"type":48,"value":707}," that match known LLM integration datasets (e.g. from\n",{"type":42,"tag":141,"props":709,"children":711},{"href":195,"rel":710},[145],[712],{"type":48,"value":713},"Elastic LLM observability",{"type":48,"value":715},").",{"type":42,"tag":97,"props":717,"children":718},{},[719,724,726,731,733,739,741,747],{"type":42,"tag":57,"props":720,"children":721},{},[722],{"type":48,"value":723},"Inspect trace indices:",{"type":48,"value":725}," For ",{"type":42,"tag":111,"props":727,"children":729},{"className":728},[],[730],{"type":48,"value":116},{"type":48,"value":732},", run a small search or use mapping to see if spans contain ",{"type":42,"tag":111,"props":734,"children":736},{"className":735},[],[737],{"type":48,"value":738},"gen_ai.*",{"type":48,"value":740}," or\n",{"type":42,"tag":111,"props":742,"children":744},{"className":743},[],[745],{"type":48,"value":746},"llm.*",{"type":48,"value":748}," (or similar) attributes. Confirm presence of token, model, and duration fields.",{"type":42,"tag":97,"props":750,"children":751},{},[752,757],{"type":42,"tag":57,"props":753,"children":754},{},[755],{"type":48,"value":756},"Inspect integration indices:",{"type":48,"value":758}," For metrics\u002Flogs data streams, check mapping or one document to see token, cost,\nlatency, and model dimensions.",{"type":42,"tag":97,"props":760,"children":761},{},[762,767],{"type":42,"tag":57,"props":763,"children":764},{},[765],{"type":48,"value":766},"Use one source per use case:",{"type":48,"value":768}," If both APM and integration data exist, prefer one consistent source for a given\nquestion (e.g. use traces for per-request chain analysis, integration metrics for aggregate token\u002Fcost).",{"type":42,"tag":97,"props":770,"children":771},{},[772,777],{"type":42,"tag":57,"props":773,"children":774},{},[775],{"type":48,"value":776},"Check alerts and SLOs:",{"type":48,"value":778}," Use the SLOs API and Alerting API to list SLOs and alerting rules that target LLM-related\nservices or integration metrics, and to get open or recently fired alerts. Firing alerts or SLOs in\ndegrading\u002Fviolated status point to potential degraded performance.",{"type":42,"tag":86,"props":780,"children":782},{"id":781},"use-cases-and-query-patterns",[783],{"type":48,"value":784},"Use cases and query patterns",{"type":42,"tag":360,"props":786,"children":788},{"id":787},"llm-performance-latency-throughput-errors",[789],{"type":48,"value":790},"LLM performance (latency, throughput, errors)",{"type":42,"tag":93,"props":792,"children":793},{},[794,841],{"type":42,"tag":97,"props":795,"children":796},{},[797,802,804,809,811,816,818,823,825,831,833,839],{"type":42,"tag":57,"props":798,"children":799},{},[800],{"type":48,"value":801},"Traces:",{"type":48,"value":803}," ES|QL on ",{"type":42,"tag":111,"props":805,"children":807},{"className":806},[],[808],{"type":48,"value":116},{"type":48,"value":810}," filtered by span attributes (e.g. ",{"type":42,"tag":111,"props":812,"children":814},{"className":813},[],[815],{"type":48,"value":439},{"type":48,"value":817}," or ",{"type":42,"tag":111,"props":819,"children":821},{"className":820},[],[822],{"type":48,"value":446},{"type":48,"value":824},"\nwhen present). Compute throughput (count per time bucket), latency (e.g. ",{"type":42,"tag":111,"props":826,"children":828},{"className":827},[],[829],{"type":48,"value":830},"duration.us",{"type":48,"value":832}," or span duration), and error\nrate (",{"type":42,"tag":111,"props":834,"children":836},{"className":835},[],[837],{"type":48,"value":838},"event.outcome == \"failure\"",{"type":48,"value":840},") by model, service, or time.",{"type":42,"tag":97,"props":842,"children":843},{},[844,849],{"type":42,"tag":57,"props":845,"children":846},{},[847],{"type":48,"value":848},"Integrations:",{"type":48,"value":850}," Query integration metrics for request rate, latency, and error metrics by model\u002Fdimension as exposed\nby the integration.",{"type":42,"tag":360,"props":852,"children":854},{"id":853},"cost-and-token-utilization",[855],{"type":48,"value":856},"Cost and token utilization",{"type":42,"tag":93,"props":858,"children":859},{},[860,897],{"type":42,"tag":97,"props":861,"children":862},{},[863,867,869,874,876,881,882,887,889,895],{"type":42,"tag":57,"props":864,"children":865},{},[866],{"type":48,"value":801},{"type":48,"value":868}," Aggregate from spans in ",{"type":42,"tag":111,"props":870,"children":872},{"className":871},[],[873],{"type":48,"value":116},{"type":48,"value":875},": sum ",{"type":42,"tag":111,"props":877,"children":879},{"className":878},[],[880],{"type":48,"value":487},{"type":48,"value":171},{"type":42,"tag":111,"props":883,"children":885},{"className":884},[],[886],{"type":48,"value":494},{"type":48,"value":888}," (or\nequivalent attribute names) by time, model, or service. If a cost attribute exists (e.g. custom\n",{"type":42,"tag":111,"props":890,"children":892},{"className":891},[],[893],{"type":48,"value":894},"llm.response.cost.*",{"type":48,"value":896},"), sum it for cost views.",{"type":42,"tag":97,"props":898,"children":899},{},[900,904],{"type":42,"tag":57,"props":901,"children":902},{},[903],{"type":48,"value":848},{"type":48,"value":905}," Use integration metrics that expose token counts and\u002For cost; aggregate by time and model.",{"type":42,"tag":360,"props":907,"children":909},{"id":908},"response-quality-and-safety",[910],{"type":48,"value":911},"Response quality and safety",{"type":42,"tag":93,"props":913,"children":914},{},[915,967],{"type":42,"tag":97,"props":916,"children":917},{},[918,922,924,929,930,935,937,943,945,950,952,958,959,965],{"type":42,"tag":57,"props":919,"children":920},{},[921],{"type":48,"value":801},{"type":48,"value":923}," Use ",{"type":42,"tag":111,"props":925,"children":927},{"className":926},[],[928],{"type":48,"value":598},{"type":48,"value":215},{"type":42,"tag":111,"props":931,"children":933},{"className":932},[],[934],{"type":48,"value":535},{"type":48,"value":936},", and span attributes (e.g. ",{"type":42,"tag":111,"props":938,"children":940},{"className":939},[],[941],{"type":48,"value":942},"gen_ai.response.finish_reasons",{"type":48,"value":944},") in\n",{"type":42,"tag":111,"props":946,"children":948},{"className":947},[],[949],{"type":48,"value":116},{"type":48,"value":951}," to identify failures, timeouts, or content filters. Correlate with prompts\u002Fresponses if captured in\nattributes (e.g. ",{"type":42,"tag":111,"props":953,"children":955},{"className":954},[],[956],{"type":48,"value":957},"gen_ai.input.messages",{"type":48,"value":215},{"type":42,"tag":111,"props":960,"children":962},{"className":961},[],[963],{"type":48,"value":964},"gen_ai.output.messages",{"type":48,"value":966},") and not redacted.",{"type":42,"tag":97,"props":968,"children":969},{},[970,974,976,983],{"type":42,"tag":57,"props":971,"children":972},{},[973],{"type":48,"value":848},{"type":48,"value":975}," Query integration logs for guardrail blocks, content filter events, or policy violations (e.g.\n",{"type":42,"tag":141,"props":977,"children":980},{"href":978,"rel":979},"https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs\u002Fblog\u002Fllm-observability-amazon-bedrock-guardrails",[145],[981],{"type":48,"value":982},"Bedrock Guardrails",{"type":48,"value":984},")\nusing the fields defined by that integration.",{"type":42,"tag":360,"props":986,"children":988},{"id":987},"call-chaining-and-agentic-workflow-orchestration",[989],{"type":48,"value":990},"Call chaining and agentic workflow orchestration",{"type":42,"tag":93,"props":992,"children":993},{},[994],{"type":42,"tag":97,"props":995,"children":996},{},[997,1002,1003,1008,1010,1015,1017,1022,1024,1030,1031,1036,1038,1043],{"type":42,"tag":57,"props":998,"children":999},{},[1000],{"type":48,"value":1001},"Traces only:",{"type":48,"value":923},{"type":42,"tag":57,"props":1004,"children":1005},{},[1006],{"type":48,"value":1007},"trace hierarchy",{"type":48,"value":1009}," in ",{"type":42,"tag":111,"props":1011,"children":1013},{"className":1012},[],[1014],{"type":48,"value":116},{"type":48,"value":1016},". Filter by root service or trace attributes; group by ",{"type":42,"tag":111,"props":1018,"children":1020},{"className":1019},[],[1021],{"type":48,"value":605},{"type":48,"value":1023},"\nand use parent\u002Fchild span relationships (e.g. ",{"type":42,"tag":111,"props":1025,"children":1027},{"className":1026},[],[1028],{"type":48,"value":1029},"parent.id",{"type":48,"value":215},{"type":42,"tag":111,"props":1032,"children":1034},{"className":1033},[],[1035],{"type":48,"value":611},{"type":48,"value":1037},") to reconstruct chains (e.g. orchestration span →\nmultiple LLM or tool-call spans). Aggregate by span name or ",{"type":42,"tag":111,"props":1039,"children":1041},{"className":1040},[],[1042],{"type":48,"value":439},{"type":48,"value":1044}," to see distribution of steps (e.g.\nretrieval, LLM, tool use). Duration per span and per trace gives bottleneck and end-to-end latency.",{"type":42,"tag":86,"props":1046,"children":1048},{"id":1047},"using-esql-for-llm-data",[1049],{"type":48,"value":1050},"Using ES|QL for LLM data",{"type":42,"tag":93,"props":1052,"children":1053},{},[1054,1064,1110],{"type":42,"tag":97,"props":1055,"children":1056},{},[1057,1062],{"type":42,"tag":57,"props":1058,"children":1059},{},[1060],{"type":48,"value":1061},"Availability:",{"type":48,"value":1063}," ES|QL is available in Elasticsearch 8.11+ (GA in 8.14) and in Elastic Observability Serverless.",{"type":42,"tag":97,"props":1065,"children":1066},{},[1067,1072,1074,1080,1082,1088,1090,1096,1098,1103,1104,1109],{"type":42,"tag":57,"props":1068,"children":1069},{},[1070],{"type":48,"value":1071},"Scoping:",{"type":48,"value":1073}," Always restrict by time range (",{"type":42,"tag":111,"props":1075,"children":1077},{"className":1076},[],[1078],{"type":48,"value":1079},"@timestamp",{"type":48,"value":1081},"). When present, add ",{"type":42,"tag":111,"props":1083,"children":1085},{"className":1084},[],[1086],{"type":48,"value":1087},"service.name",{"type":48,"value":1089}," and optionally\n",{"type":42,"tag":111,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":48,"value":1095},"service.environment",{"type":48,"value":1097},". For LLM-specific spans, filter by span attributes once you know the field names (e.g. a keyword\nfield for ",{"type":42,"tag":111,"props":1099,"children":1101},{"className":1100},[],[1102],{"type":48,"value":446},{"type":48,"value":817},{"type":42,"tag":111,"props":1105,"children":1107},{"className":1106},[],[1108],{"type":48,"value":439},{"type":48,"value":715},{"type":42,"tag":97,"props":1111,"children":1112},{},[1113,1118,1119,1125],{"type":42,"tag":57,"props":1114,"children":1115},{},[1116],{"type":48,"value":1117},"Performance:",{"type":48,"value":923},{"type":42,"tag":111,"props":1120,"children":1122},{"className":1121},[],[1123],{"type":48,"value":1124},"LIMIT",{"type":48,"value":1126},", coarse time buckets when only trends are needed, and avoid full scans over large\nwindows.",{"type":42,"tag":86,"props":1128,"children":1130},{"id":1129},"workflow",[1131],{"type":48,"value":1132},"Workflow",{"type":42,"tag":1134,"props":1135,"children":1140},"pre",{"className":1136,"code":1138,"language":48,"meta":1139},[1137],"language-text","LLM observability progress:\n- [ ] Step 1: Determine available data (traces*, metrics-apm* or metrics*, or integration data streams)\n- [ ] Step 2: Discover LLM-related field names (mapping or sample doc)\n- [ ] Step 3: Run ES|QL or Elasticsearch queries for the user's question (performance, cost, quality, orchestration)\n- [ ] Step 4: Check for active alerts or SLOs defined on LLM-related data (Alerting API, SLOs API); field names from\n        Step 2 help identify related rules; firing alerts or violated\u002Fdegrading SLOs indicate potential degraded performance\n- [ ] Step 5: Summarize findings from ingested data only; include alert\u002FSLO status when relevant\n","",[1141],{"type":42,"tag":111,"props":1142,"children":1143},{"__ignoreMap":1139},[1144],{"type":48,"value":1138},{"type":42,"tag":86,"props":1146,"children":1148},{"id":1147},"examples",[1149],{"type":48,"value":1150},"Examples",{"type":42,"tag":360,"props":1152,"children":1154},{"id":1153},"example-token-usage-over-time-from-traces",[1155],{"type":48,"value":1156},"Example: Token usage over time from traces",{"type":42,"tag":51,"props":1158,"children":1159},{},[1160,1162,1168,1170,1176],{"type":48,"value":1161},"Assume span attributes are available as ",{"type":42,"tag":111,"props":1163,"children":1165},{"className":1164},[],[1166],{"type":48,"value":1167},"span.attributes.gen_ai.usage.input_tokens",{"type":48,"value":1169}," and\n",{"type":42,"tag":111,"props":1171,"children":1173},{"className":1172},[],[1174],{"type":48,"value":1175},"span.attributes.gen_ai.usage.output_tokens",{"type":48,"value":1177}," (adjust to actual field names from mapping):",{"type":42,"tag":1134,"props":1179,"children":1183},{"className":1180,"code":1181,"language":1182,"meta":1139,"style":1139},"language-esql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","FROM traces*\n| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n  AND span.attributes.gen_ai.provider.name IS NOT NULL\n| STATS\n    input_tokens = SUM(span.attributes.gen_ai.usage.input_tokens),\n    output_tokens = SUM(span.attributes.gen_ai.usage.output_tokens)\n  BY BUCKET(@timestamp, 1 hour), span.attributes.gen_ai.request.model\n| SORT @timestamp\n| LIMIT 500\n","esql",[1184],{"type":42,"tag":111,"props":1185,"children":1186},{"__ignoreMap":1139},[1187,1198,1207,1216,1225,1234,1243,1252,1261],{"type":42,"tag":1188,"props":1189,"children":1192},"span",{"class":1190,"line":1191},"line",1,[1193],{"type":42,"tag":1188,"props":1194,"children":1195},{},[1196],{"type":48,"value":1197},"FROM traces*\n",{"type":42,"tag":1188,"props":1199,"children":1201},{"class":1190,"line":1200},2,[1202],{"type":42,"tag":1188,"props":1203,"children":1204},{},[1205],{"type":48,"value":1206},"| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n",{"type":42,"tag":1188,"props":1208,"children":1210},{"class":1190,"line":1209},3,[1211],{"type":42,"tag":1188,"props":1212,"children":1213},{},[1214],{"type":48,"value":1215},"  AND span.attributes.gen_ai.provider.name IS NOT NULL\n",{"type":42,"tag":1188,"props":1217,"children":1219},{"class":1190,"line":1218},4,[1220],{"type":42,"tag":1188,"props":1221,"children":1222},{},[1223],{"type":48,"value":1224},"| STATS\n",{"type":42,"tag":1188,"props":1226,"children":1228},{"class":1190,"line":1227},5,[1229],{"type":42,"tag":1188,"props":1230,"children":1231},{},[1232],{"type":48,"value":1233},"    input_tokens = SUM(span.attributes.gen_ai.usage.input_tokens),\n",{"type":42,"tag":1188,"props":1235,"children":1237},{"class":1190,"line":1236},6,[1238],{"type":42,"tag":1188,"props":1239,"children":1240},{},[1241],{"type":48,"value":1242},"    output_tokens = SUM(span.attributes.gen_ai.usage.output_tokens)\n",{"type":42,"tag":1188,"props":1244,"children":1246},{"class":1190,"line":1245},7,[1247],{"type":42,"tag":1188,"props":1248,"children":1249},{},[1250],{"type":48,"value":1251},"  BY BUCKET(@timestamp, 1 hour), span.attributes.gen_ai.request.model\n",{"type":42,"tag":1188,"props":1253,"children":1255},{"class":1190,"line":1254},8,[1256],{"type":42,"tag":1188,"props":1257,"children":1258},{},[1259],{"type":48,"value":1260},"| SORT @timestamp\n",{"type":42,"tag":1188,"props":1262,"children":1264},{"class":1190,"line":1263},9,[1265],{"type":42,"tag":1188,"props":1266,"children":1267},{},[1268],{"type":48,"value":1269},"| LIMIT 500\n",{"type":42,"tag":360,"props":1271,"children":1273},{"id":1272},"example-latency-and-error-rate-by-model",[1274],{"type":48,"value":1275},"Example: Latency and error rate by model",{"type":42,"tag":1134,"props":1277,"children":1279},{"className":1180,"code":1278,"language":1182,"meta":1139,"style":1139},"FROM traces*\n| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n  AND span.attributes.gen_ai.request.model IS NOT NULL\n| STATS\n    request_count = COUNT(*),\n    failures = COUNT(*) WHERE event.outcome == \"failure\",\n    avg_duration_us = AVG(span.duration.us)\n  BY span.attributes.gen_ai.request.model\n| EVAL error_rate = failures \u002F request_count\n| LIMIT 100\n",[1280],{"type":42,"tag":111,"props":1281,"children":1282},{"__ignoreMap":1139},[1283,1290,1297,1305,1312,1320,1328,1336,1344,1352],{"type":42,"tag":1188,"props":1284,"children":1285},{"class":1190,"line":1191},[1286],{"type":42,"tag":1188,"props":1287,"children":1288},{},[1289],{"type":48,"value":1197},{"type":42,"tag":1188,"props":1291,"children":1292},{"class":1190,"line":1200},[1293],{"type":42,"tag":1188,"props":1294,"children":1295},{},[1296],{"type":48,"value":1206},{"type":42,"tag":1188,"props":1298,"children":1299},{"class":1190,"line":1209},[1300],{"type":42,"tag":1188,"props":1301,"children":1302},{},[1303],{"type":48,"value":1304},"  AND span.attributes.gen_ai.request.model IS NOT NULL\n",{"type":42,"tag":1188,"props":1306,"children":1307},{"class":1190,"line":1218},[1308],{"type":42,"tag":1188,"props":1309,"children":1310},{},[1311],{"type":48,"value":1224},{"type":42,"tag":1188,"props":1313,"children":1314},{"class":1190,"line":1227},[1315],{"type":42,"tag":1188,"props":1316,"children":1317},{},[1318],{"type":48,"value":1319},"    request_count = COUNT(*),\n",{"type":42,"tag":1188,"props":1321,"children":1322},{"class":1190,"line":1236},[1323],{"type":42,"tag":1188,"props":1324,"children":1325},{},[1326],{"type":48,"value":1327},"    failures = COUNT(*) WHERE event.outcome == \"failure\",\n",{"type":42,"tag":1188,"props":1329,"children":1330},{"class":1190,"line":1245},[1331],{"type":42,"tag":1188,"props":1332,"children":1333},{},[1334],{"type":48,"value":1335},"    avg_duration_us = AVG(span.duration.us)\n",{"type":42,"tag":1188,"props":1337,"children":1338},{"class":1190,"line":1254},[1339],{"type":42,"tag":1188,"props":1340,"children":1341},{},[1342],{"type":48,"value":1343},"  BY span.attributes.gen_ai.request.model\n",{"type":42,"tag":1188,"props":1345,"children":1346},{"class":1190,"line":1263},[1347],{"type":42,"tag":1188,"props":1348,"children":1349},{},[1350],{"type":48,"value":1351},"| EVAL error_rate = failures \u002F request_count\n",{"type":42,"tag":1188,"props":1353,"children":1355},{"class":1190,"line":1354},10,[1356],{"type":42,"tag":1188,"props":1357,"children":1358},{},[1359],{"type":48,"value":1360},"| LIMIT 100\n",{"type":42,"tag":360,"props":1362,"children":1364},{"id":1363},"example-agentic-workflow-trace-level-view",[1365],{"type":48,"value":1366},"Example: Agentic workflow (trace-level view)",{"type":42,"tag":51,"props":1368,"children":1369},{},[1370],{"type":48,"value":1371},"Get trace IDs that contain at least one LLM span and count spans per trace to see chain length:",{"type":42,"tag":1134,"props":1373,"children":1375},{"className":1180,"code":1374,"language":1182,"meta":1139,"style":1139},"FROM traces*\n| WHERE @timestamp >= \"2025-03-01T00:00:00Z\" AND @timestamp \u003C= \"2025-03-01T23:59:59Z\"\n  AND span.attributes.gen_ai.operation.name IS NOT NULL\n| STATS span_count = COUNT(*), total_duration_us = SUM(span.duration.us) BY trace.id\n| WHERE span_count > 1\n| SORT total_duration_us DESC\n| LIMIT 50\n",[1376],{"type":42,"tag":111,"props":1377,"children":1378},{"__ignoreMap":1139},[1379,1386,1393,1401,1409,1417,1425],{"type":42,"tag":1188,"props":1380,"children":1381},{"class":1190,"line":1191},[1382],{"type":42,"tag":1188,"props":1383,"children":1384},{},[1385],{"type":48,"value":1197},{"type":42,"tag":1188,"props":1387,"children":1388},{"class":1190,"line":1200},[1389],{"type":42,"tag":1188,"props":1390,"children":1391},{},[1392],{"type":48,"value":1206},{"type":42,"tag":1188,"props":1394,"children":1395},{"class":1190,"line":1209},[1396],{"type":42,"tag":1188,"props":1397,"children":1398},{},[1399],{"type":48,"value":1400},"  AND span.attributes.gen_ai.operation.name IS NOT NULL\n",{"type":42,"tag":1188,"props":1402,"children":1403},{"class":1190,"line":1218},[1404],{"type":42,"tag":1188,"props":1405,"children":1406},{},[1407],{"type":48,"value":1408},"| STATS span_count = COUNT(*), total_duration_us = SUM(span.duration.us) BY trace.id\n",{"type":42,"tag":1188,"props":1410,"children":1411},{"class":1190,"line":1227},[1412],{"type":42,"tag":1188,"props":1413,"children":1414},{},[1415],{"type":48,"value":1416},"| WHERE span_count > 1\n",{"type":42,"tag":1188,"props":1418,"children":1419},{"class":1190,"line":1236},[1420],{"type":42,"tag":1188,"props":1421,"children":1422},{},[1423],{"type":48,"value":1424},"| SORT total_duration_us DESC\n",{"type":42,"tag":1188,"props":1426,"children":1427},{"class":1190,"line":1245},[1428],{"type":42,"tag":1188,"props":1429,"children":1430},{},[1431],{"type":48,"value":1432},"| LIMIT 50\n",{"type":42,"tag":360,"props":1434,"children":1436},{"id":1435},"example-integration-metrics-amazon-bedrock-agentcore",[1437],{"type":48,"value":1438},"Example: Integration metrics (Amazon Bedrock AgentCore)",{"type":42,"tag":51,"props":1440,"children":1441},{},[1442,1444,1451,1453,1459,1461,1470,1472,1481,1483,1490],{"type":48,"value":1443},"The ",{"type":42,"tag":141,"props":1445,"children":1448},{"href":1446,"rel":1447},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Freference\u002Fintegrations\u002Faws_bedrock_agentcore",[145],[1449],{"type":48,"value":1450},"Amazon Bedrock AgentCore integration",{"type":48,"value":1452},"\nships metrics to the ",{"type":42,"tag":111,"props":1454,"children":1456},{"className":1455},[],[1457],{"type":48,"value":1458},"metrics-aws_bedrock_agentcore.metrics-*",{"type":48,"value":1460}," data stream (time series index). Use ",{"type":42,"tag":57,"props":1462,"children":1463},{},[1464],{"type":42,"tag":111,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":48,"value":1469},"TS",{"type":48,"value":1471}," for\naggregations on time series data streams (Elasticsearch 9.2+); use a time range with ",{"type":42,"tag":57,"props":1473,"children":1474},{},[1475],{"type":42,"tag":111,"props":1476,"children":1478},{"className":1477},[],[1479],{"type":48,"value":1480},"TRANGE",{"type":48,"value":1482}," (9.3+). The\nintegration’s dashboards and\n",{"type":42,"tag":141,"props":1484,"children":1487},{"href":1485,"rel":1486},"https:\u002F\u002Fgithub.com\u002Felastic\u002Fintegrations\u002Ftree\u002Fmain\u002Fpackages\u002Faws_bedrock_agentcore\u002Fkibana\u002Falerting_rule_template",[145],[1488],{"type":48,"value":1489},"alerting rule templates",{"type":48,"value":1491},"\nExample: token usage (counter), invocations (counter), and average latency (gauge) by hour and agent:",{"type":42,"tag":1134,"props":1493,"children":1495},{"className":1180,"code":1494,"language":1182,"meta":1139,"style":1139},"TS metrics-aws_bedrock_agentcore.metrics-*\n| WHERE TRANGE(7 days)\n  AND aws.dimensions.Operation == \"InvokeAgentRuntime\"\n| STATS\n    total_tokens = SUM(RATE(aws.bedrock_agentcore.metrics.TokenCount.sum)),\n    total_invocations = SUM(RATE(aws.bedrock_agentcore.metrics.Invocations.sum)),\n    avg_latency_ms = AVG(AVG_OVER_TIME(aws.bedrock_agentcore.metrics.Latency.avg))\n  BY TBUCKET(1 hour), aws.bedrock_agentcore.agent_name\n| SORT TBUCKET(1 hour) DESC\n",[1496],{"type":42,"tag":111,"props":1497,"children":1498},{"__ignoreMap":1139},[1499,1507,1515,1523,1530,1538,1546,1554,1562],{"type":42,"tag":1188,"props":1500,"children":1501},{"class":1190,"line":1191},[1502],{"type":42,"tag":1188,"props":1503,"children":1504},{},[1505],{"type":48,"value":1506},"TS metrics-aws_bedrock_agentcore.metrics-*\n",{"type":42,"tag":1188,"props":1508,"children":1509},{"class":1190,"line":1200},[1510],{"type":42,"tag":1188,"props":1511,"children":1512},{},[1513],{"type":48,"value":1514},"| WHERE TRANGE(7 days)\n",{"type":42,"tag":1188,"props":1516,"children":1517},{"class":1190,"line":1209},[1518],{"type":42,"tag":1188,"props":1519,"children":1520},{},[1521],{"type":48,"value":1522},"  AND aws.dimensions.Operation == \"InvokeAgentRuntime\"\n",{"type":42,"tag":1188,"props":1524,"children":1525},{"class":1190,"line":1218},[1526],{"type":42,"tag":1188,"props":1527,"children":1528},{},[1529],{"type":48,"value":1224},{"type":42,"tag":1188,"props":1531,"children":1532},{"class":1190,"line":1227},[1533],{"type":42,"tag":1188,"props":1534,"children":1535},{},[1536],{"type":48,"value":1537},"    total_tokens = SUM(RATE(aws.bedrock_agentcore.metrics.TokenCount.sum)),\n",{"type":42,"tag":1188,"props":1539,"children":1540},{"class":1190,"line":1236},[1541],{"type":42,"tag":1188,"props":1542,"children":1543},{},[1544],{"type":48,"value":1545},"    total_invocations = SUM(RATE(aws.bedrock_agentcore.metrics.Invocations.sum)),\n",{"type":42,"tag":1188,"props":1547,"children":1548},{"class":1190,"line":1245},[1549],{"type":42,"tag":1188,"props":1550,"children":1551},{},[1552],{"type":48,"value":1553},"    avg_latency_ms = AVG(AVG_OVER_TIME(aws.bedrock_agentcore.metrics.Latency.avg))\n",{"type":42,"tag":1188,"props":1555,"children":1556},{"class":1190,"line":1254},[1557],{"type":42,"tag":1188,"props":1558,"children":1559},{},[1560],{"type":48,"value":1561},"  BY TBUCKET(1 hour), aws.bedrock_agentcore.agent_name\n",{"type":42,"tag":1188,"props":1563,"children":1564},{"class":1190,"line":1263},[1565],{"type":42,"tag":1188,"props":1566,"children":1567},{},[1568],{"type":48,"value":1569},"| SORT TBUCKET(1 hour) DESC\n",{"type":42,"tag":51,"props":1571,"children":1572},{},[1573,1575,1580,1582,1588,1590,1596,1597,1603,1605,1611,1613,1618],{"type":48,"value":1574},"For Elasticsearch 8.x or when ",{"type":42,"tag":111,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":48,"value":1469},{"type":48,"value":1581}," is not available, use ",{"type":42,"tag":111,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":48,"value":1587},"FROM",{"type":48,"value":1589}," with ",{"type":42,"tag":111,"props":1591,"children":1593},{"className":1592},[],[1594],{"type":48,"value":1595},"BUCKET(@timestamp, 1 hour)",{"type":48,"value":171},{"type":42,"tag":111,"props":1598,"children":1600},{"className":1599},[],[1601],{"type":48,"value":1602},"SUM",{"type":48,"value":1604},"\u002F",{"type":42,"tag":111,"props":1606,"children":1608},{"className":1607},[],[1609],{"type":48,"value":1610},"AVG",{"type":48,"value":1612}," over\nthe metric fields (as in the integration's alert rule templates). For other LLM integrations (OpenAI, Azure OpenAI,\nVertex AI, etc.), use that integration’s data stream index pattern and field names from its package (see\n",{"type":42,"tag":141,"props":1614,"children":1616},{"href":195,"rel":1615},[145],[1617],{"type":48,"value":713},{"type":48,"value":715},{"type":42,"tag":86,"props":1620,"children":1622},{"id":1621},"guidelines",[1623],{"type":48,"value":1624},"Guidelines",{"type":42,"tag":93,"props":1626,"children":1627},{},[1628,1645,1655,1687,1697],{"type":42,"tag":97,"props":1629,"children":1630},{},[1631,1636,1638,1643],{"type":42,"tag":57,"props":1632,"children":1633},{},[1634],{"type":48,"value":1635},"Data only in Elastic:",{"type":48,"value":1637}," Use only data collected and stored in Elastic (traces in ",{"type":42,"tag":111,"props":1639,"children":1641},{"className":1640},[],[1642],{"type":48,"value":116},{"type":48,"value":1644},", metrics, or integration\nmetrics\u002Flogs). Do not describe or rely on other vendors’ UIs or products.",{"type":42,"tag":97,"props":1646,"children":1647},{},[1648,1653],{"type":42,"tag":57,"props":1649,"children":1650},{},[1651],{"type":48,"value":1652},"One technology per customer:",{"type":48,"value":1654}," Assume a single ingestion path per deployment when answering; discover which (traces\nvs integration) exists and use it consistently for the question.",{"type":42,"tag":97,"props":1656,"children":1657},{},[1658,1663,1665,1671,1673,1678,1680,1685],{"type":42,"tag":57,"props":1659,"children":1660},{},[1661],{"type":48,"value":1662},"Discover field names:",{"type":48,"value":1664}," Before writing ES|QL or Query DSL, confirm LLM-related attribute or metric names from\n",{"type":42,"tag":111,"props":1666,"children":1668},{"className":1667},[],[1669],{"type":48,"value":1670},"_mapping",{"type":48,"value":1672}," or a sample document; naming may differ (e.g. ",{"type":42,"tag":111,"props":1674,"children":1676},{"className":1675},[],[1677],{"type":48,"value":738},{"type":48,"value":1679}," vs ",{"type":42,"tag":111,"props":1681,"children":1683},{"className":1682},[],[1684],{"type":48,"value":746},{"type":48,"value":1686}," or integration-specific fields).",{"type":42,"tag":97,"props":1688,"children":1689},{},[1690,1695],{"type":42,"tag":57,"props":1691,"children":1692},{},[1693],{"type":48,"value":1694},"No Kibana UI dependency:",{"type":48,"value":1696}," Prefer ES|QL and Elasticsearch APIs; use Kibana APIs only when needed (e.g. SLO,\nalerting). Do not instruct the user to open Kibana UI.",{"type":42,"tag":97,"props":1698,"children":1699},{},[1700,1705,1711,1713,1720,1721,1727,1729,1733,1735,1742,1744,1751],{"type":42,"tag":57,"props":1701,"children":1702},{},[1703],{"type":48,"value":1704},"References:",{"type":42,"tag":141,"props":1706,"children":1708},{"href":195,"rel":1707},[145],[1709],{"type":48,"value":1710},"LLM and agentic AI observability",{"type":48,"value":1712},",\n",{"type":42,"tag":141,"props":1714,"children":1717},{"href":1715,"rel":1716},"https:\u002F\u002Fwww.elastic.co\u002Fobservability-labs\u002Fblog\u002Ftag\u002Fllmobs",[145],[1718],{"type":48,"value":1719},"Observability Labs – LLM Observability",{"type":48,"value":1712},{"type":42,"tag":141,"props":1722,"children":1724},{"href":380,"rel":1723},[145],[1725],{"type":48,"value":1726},"OpenTelemetry GenAI spans",{"type":48,"value":1728},". For ES|QL syntax and\nquery patterns, use the ",{"type":42,"tag":57,"props":1730,"children":1731},{},[1732],{"type":48,"value":271},{"type":48,"value":1734}," skill, or look through\n",{"type":42,"tag":141,"props":1736,"children":1739},{"href":1737,"rel":1738},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Freference\u002Fquery-languages\u002Fesql\u002Fcommands\u002Fts",[145],[1740],{"type":48,"value":1741},"ES|QL TS command reference",{"type":48,"value":1743}," for Elastic v9.3\nor higher and for Serverless, and look through\n",{"type":42,"tag":141,"props":1745,"children":1748},{"href":1746,"rel":1747},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Freference\u002Fquery-languages\u002Fesql\u002Fcommands\u002Ffrom",[145],[1749],{"type":48,"value":1750},"ES|QL FROM command reference",{"type":48,"value":1752}," for other\nElastic versions.",{"type":42,"tag":1754,"props":1755,"children":1756},"style",{},[1757],{"type":48,"value":1758},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1760,"total":1928},[1761,1780,1797,1812,1829,1841,1851,1866,1878,1892,1903,1916],{"slug":1762,"name":1762,"fn":1763,"description":1764,"org":1765,"tags":1766,"stars":1777,"repoUrl":1778,"updatedAt":1779},"accessing-benchmark-results","retrieve and analyze Rally benchmark results","Retrieve Rally benchmark results from an external Elasticsearch metrics store. Use to list past races, get a single race's overall (per-task) results, chart a metric's trend across multiple runs, compare two races, or check whether a run converged — e.g. \"show me recent geonames races\", \"what's the service_time trend for nyc_taxis over the last 30 days?\", \"compare these two race-ids\". Applies when datastore.type = elasticsearch is set in ~\u002F.rally\u002Frally.ini.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1767,1770,1773,1774],{"name":1768,"slug":1769,"type":15},"Analytics","analytics",{"name":1771,"slug":1772,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":1775,"slug":1776,"type":15},"Performance","performance",2027,"https:\u002F\u002Fgithub.com\u002Felastic\u002Frally","2026-07-12T07:46:38.54144",{"slug":1781,"name":1781,"fn":1782,"description":1783,"org":1784,"tags":1785,"stars":1777,"repoUrl":1778,"updatedAt":1796},"developing-rally","develop and debug Rally source code","Work on Rally's own codebase, not running benchmarks with it. Use when setting up the dev environment, running Rally's tests or linters, navigating its source, debugging Rally's own code, or making changes to Rally itself.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1786,1789,1790,1793],{"name":1787,"slug":1788,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":1791,"slug":1792,"type":15},"Engineering","engineering",{"name":1794,"slug":1795,"type":15},"Local Development","local-development","2026-07-12T07:46:35.976807",{"slug":1798,"name":1798,"fn":1799,"description":1800,"org":1801,"tags":1802,"stars":1777,"repoUrl":1778,"updatedAt":1811},"running-benchmarks","run Rally benchmarks against Elasticsearch","Run Rally benchmarks (races) against Elasticsearch — an existing\u002Fexternal cluster or a Rally-provisioned distribution — and read the summary report. Use when running a race (any pipeline, track, challenge, target-hosts, or auth) or when interpreting throughput, latency, and service_time results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1803,1804,1807,1808],{"name":9,"slug":8,"type":15},{"name":1805,"slug":1806,"type":15},"Elasticsearch","elasticsearch",{"name":1775,"slug":1776,"type":15},{"name":1809,"slug":1810,"type":15},"Testing","testing","2026-07-12T07:46:37.277964",{"slug":1813,"name":1813,"fn":1814,"description":1815,"org":1816,"tags":1817,"stars":23,"repoUrl":24,"updatedAt":1828},"cloud-access-management","manage Elastic Cloud organization access","Manage Elastic Cloud organization access: invite users, assign roles to Serverless projects, and create or revoke Cloud API keys. Use when granting, modifying, or auditing user access.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1818,1821,1822,1825],{"name":1819,"slug":1820,"type":15},"Cloud","cloud",{"name":9,"slug":8,"type":15},{"name":1823,"slug":1824,"type":15},"Operations","operations",{"name":1826,"slug":1827,"type":15},"Permissions","permissions","2026-07-12T07:46:44.946285",{"slug":1830,"name":1830,"fn":1831,"description":1832,"org":1833,"tags":1834,"stars":23,"repoUrl":24,"updatedAt":1840},"cloud-create-project","create Elastic Cloud Serverless projects","Creates Elastic Cloud Serverless projects (Elasticsearch, Observability, or Security) via the REST API, saves credentials to file, and bootstraps a scoped Elasticsearch API key. Use when creating a new serverless project, provisioning a search or observability environment, or spinning up a new Elastic Cloud project.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1835,1836,1839],{"name":1819,"slug":1820,"type":15},{"name":1837,"slug":1838,"type":15},"Deployment","deployment",{"name":1805,"slug":1806,"type":15},"2026-07-12T07:46:42.353362",{"slug":1842,"name":1842,"fn":1843,"description":1844,"org":1845,"tags":1846,"stars":23,"repoUrl":24,"updatedAt":1850},"cloud-manage-project","manage Elastic Cloud Serverless projects","Manages existing Elastic Cloud Serverless projects: list, get, update, delete, reset credentials, resume, and load saved credentials. Connects to existing projects by resolving endpoints and acquiring scoped Elasticsearch API keys. Use when performing day-2 operations on serverless projects, connecting to an existing project, loading or resetting project credentials, or looking up project details.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1847,1848,1849],{"name":1819,"slug":1820,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1823,"slug":1824,"type":15},"2026-07-12T07:46:41.097412",{"slug":1852,"name":1852,"fn":1853,"description":1854,"org":1855,"tags":1856,"stars":23,"repoUrl":24,"updatedAt":1865},"cloud-network-security","manage Elastic Cloud network security","Manage Serverless network security (traffic filters): create, update, and delete IP filters and AWS PrivateLink VPC filters. Use when restricting network access or configuring private connectivity.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1857,1858,1859,1862],{"name":1819,"slug":1820,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1860,"slug":1861,"type":15},"Networking","networking",{"name":1863,"slug":1864,"type":15},"Security","security","2026-07-12T07:46:43.675992",{"slug":1867,"name":1867,"fn":1868,"description":1869,"org":1870,"tags":1871,"stars":23,"repoUrl":24,"updatedAt":1877},"cloud-setup","configure Elastic Cloud authentication","Configures Elastic Cloud authentication and environment defaults. Use when setting up EC_API_KEY, configuring Cloud API access, or when another cloud skill requires credentials.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1872,1875,1876],{"name":1873,"slug":1874,"type":15},"Authentication","authentication",{"name":1819,"slug":1820,"type":15},{"name":1805,"slug":1806,"type":15},"2026-07-12T07:46:39.783105",{"slug":1879,"name":1879,"fn":1880,"description":1881,"org":1882,"tags":1883,"stars":23,"repoUrl":24,"updatedAt":1891},"elasticsearch-audit","configure Elasticsearch security audit logs","Enable, configure, and query Elasticsearch security audit logs. Use when the task involves audit logging setup, event filtering, or investigating security incidents like failed logins.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1884,1887,1888,1890],{"name":1885,"slug":1886,"type":15},"Audit","audit",{"name":1805,"slug":1806,"type":15},{"name":1889,"slug":643,"type":15},"Logs",{"name":1863,"slug":1864,"type":15},"2026-07-12T07:47:35.092599",{"slug":1893,"name":1893,"fn":1894,"description":1895,"org":1896,"tags":1897,"stars":23,"repoUrl":24,"updatedAt":1902},"elasticsearch-authn","configure Elasticsearch authentication realms","Authenticate to Elasticsearch using native, file-based, LDAP\u002FAD, SAML, OIDC, Kerberos, JWT, or certificate realms. Use when connecting with credentials, choosing a realm, or managing API keys. Assumes the target realms are already configured.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1898,1899,1900,1901],{"name":1873,"slug":1874,"type":15},{"name":9,"slug":8,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1863,"slug":1864,"type":15},"2026-07-12T07:47:41.474547",{"slug":1904,"name":1904,"fn":1905,"description":1906,"org":1907,"tags":1908,"stars":23,"repoUrl":24,"updatedAt":1915},"elasticsearch-authz","manage Elasticsearch RBAC and security roles","Manage Elasticsearch RBAC: native users, roles, role mappings, document- and field-level security. Use when creating users or roles, assigning privileges, or mapping external realms like LDAP\u002FSAML.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1909,1910,1911,1914],{"name":9,"slug":8,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1912,"slug":1913,"type":15},"RBAC","rbac",{"name":1863,"slug":1864,"type":15},"2026-07-12T07:47:36.394177",{"slug":271,"name":271,"fn":1917,"description":1918,"org":1919,"tags":1920,"stars":23,"repoUrl":24,"updatedAt":1927},"query Elasticsearch data with ES|QL","Execute ES|QL (Elasticsearch Query Language) queries, use when the user wants to query Elasticsearch data, analyze logs, aggregate metrics, explore data, or create charts and dashboards from ES|QL results.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1921,1922,1923,1924],{"name":1768,"slug":1769,"type":15},{"name":1771,"slug":1772,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1925,"slug":1926,"type":15},"SQL","sql","2026-07-12T07:47:40.249533",86,{"items":1930,"total":1977},[1931,1938,1944,1950,1957,1963,1970],{"slug":1813,"name":1813,"fn":1814,"description":1815,"org":1932,"tags":1933,"stars":23,"repoUrl":24,"updatedAt":1828},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1934,1935,1936,1937],{"name":1819,"slug":1820,"type":15},{"name":9,"slug":8,"type":15},{"name":1823,"slug":1824,"type":15},{"name":1826,"slug":1827,"type":15},{"slug":1830,"name":1830,"fn":1831,"description":1832,"org":1939,"tags":1940,"stars":23,"repoUrl":24,"updatedAt":1840},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1941,1942,1943],{"name":1819,"slug":1820,"type":15},{"name":1837,"slug":1838,"type":15},{"name":1805,"slug":1806,"type":15},{"slug":1842,"name":1842,"fn":1843,"description":1844,"org":1945,"tags":1946,"stars":23,"repoUrl":24,"updatedAt":1850},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1947,1948,1949],{"name":1819,"slug":1820,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1823,"slug":1824,"type":15},{"slug":1852,"name":1852,"fn":1853,"description":1854,"org":1951,"tags":1952,"stars":23,"repoUrl":24,"updatedAt":1865},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1953,1954,1955,1956],{"name":1819,"slug":1820,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1860,"slug":1861,"type":15},{"name":1863,"slug":1864,"type":15},{"slug":1867,"name":1867,"fn":1868,"description":1869,"org":1958,"tags":1959,"stars":23,"repoUrl":24,"updatedAt":1877},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1960,1961,1962],{"name":1873,"slug":1874,"type":15},{"name":1819,"slug":1820,"type":15},{"name":1805,"slug":1806,"type":15},{"slug":1879,"name":1879,"fn":1880,"description":1881,"org":1964,"tags":1965,"stars":23,"repoUrl":24,"updatedAt":1891},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1966,1967,1968,1969],{"name":1885,"slug":1886,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1889,"slug":643,"type":15},{"name":1863,"slug":1864,"type":15},{"slug":1893,"name":1893,"fn":1894,"description":1895,"org":1971,"tags":1972,"stars":23,"repoUrl":24,"updatedAt":1902},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1973,1974,1975,1976],{"name":1873,"slug":1874,"type":15},{"name":9,"slug":8,"type":15},{"name":1805,"slug":1806,"type":15},{"name":1863,"slug":1864,"type":15},35]