[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-datadog-labs-dd-audit-cost-spike-investigation":3,"mdc-g2iqih-key":39,"related-repo-datadog-labs-dd-audit-cost-spike-investigation":1517,"related-org-datadog-labs-dd-audit-cost-spike-investigation":1624},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":28,"repoUrl":29,"updatedAt":30,"license":31,"forks":32,"topics":33,"repo":34,"sourceUrl":37,"mdContent":38},"dd-audit-cost-spike-investigation","investigate Datadog cost spikes","Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when\u002Fwhat spiked) with Audit Trail config changes (who changed what in the preceding window).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"datadog-labs","Datadog Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatadog-labs.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Operations","operations",{"name":20,"slug":21,"type":15},"Cost Optimization","cost-optimization",{"name":23,"slug":24,"type":15},"Datadog","datadog",{"name":26,"slug":27,"type":15},"Audit","audit",145,"https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills","2026-05-08T05:05:13.142629",null,19,[],{"repoUrl":29,"stars":28,"forks":32,"topics":35,"description":36},[],"Public repository for Datadog Agent Skills","https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fdd-audit\u002Fcost-spike-investigation","---\nname: dd-audit-cost-spike-investigation\ndescription: Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when\u002Fwhat spiked) with Audit Trail config changes (who changed what in the preceding window).\nmetadata:\n  version: \"0.1.0\"\n  author: datadog-labs\n  repository: https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills\n  tags: datadog,audit,cost,usage,spike,finops,dd-audit\n  alwaysApply: \"false\"\n---\n\n# Audit Trail: Cost \u002F Usage Spike Investigation\n\nIdentify what caused a Datadog usage spike by correlating billing data with configuration change history.\n\nThe causal chain is: **someone changed something → that change increased data volume → usage spiked → cost went up**. Usage Metering tells you when and what; Audit Trail tells you who made the change.\n\n## Prerequisites\n\n```bash\npup auth login   # OAuth2 (recommended) — covers audit queries\n# Usage Metering queries also need DD_API_KEY + DD_APP_KEY\nexport DD_API_KEY=\u003Cyour-api-key>\nexport DD_APP_KEY=\u003Cyour-app-key>\nexport DD_SITE=datadoghq.com\n```\n\n## Scope Boundary\n\nThis skill identifies **configuration changes** that may have caused a spike. It does not identify which specific user or process *submitted* the data (e.g., which service sent the LLM spans). For per-submission attribution, use LLM Observability traces or APM instrumentation.\n\n## Investigation Workflow\n\n### Step 1 — Identify the spike window and product family\n\n```bash\nSTART=$(date -u -v-7d +\"%Y-%m-%dT%H:%M:%SZ\" 2>\u002Fdev\u002Fnull || date -u -d \"7 days ago\" +\"%Y-%m-%dT%H:%M:%SZ\")\nEND=$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")\n\ncurl -s -G \"https:\u002F\u002Fapi.${DD_SITE}\u002Fapi\u002Fv2\u002Fusage\u002Fhourly_usage\" \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\" \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" \\\n  --data-urlencode \"filter[timestamp][start]=${START}\" \\\n  --data-urlencode \"filter[timestamp][end]=${END}\" \\\n  --data-urlencode \"filter[product_families]=all\" \\\n  | jq '[.data[] | {\n      timestamp: .attributes.timestamp,\n      product: .attributes.product_family,\n      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]\n    }]'\n```\n\n**Product families with LLM\u002FAI coverage:** `llm_observability`, `bits_ai`, `logs`, `apm`\n\n### Step 2 — Pinpoint the spike\n\nFrom Step 1, identify the hour\u002Fday where volume jumped. Note the timestamp as `SPIKE_TIME`.\n\n### Step 3 — Search Audit Trail for config changes in the 24h preceding the spike\n\n```bash\npup audit-logs search \\\n  --query \"@action:(created OR modified OR deleted)\" \\\n  --from \"SPIKE_TIME_MINUS_24H\" \\\n  --to \"SPIKE_TIME\" \\\n  --limit 200 \\\n  -o json \\\n  | jq '[.data[] | {\n      timestamp: .attributes.timestamp,\n      user: .attributes.attributes.usr.email,\n      actor_type: .attributes.attributes.evt.actor.type,\n      action: .attributes.attributes.action,\n      event_category: .attributes.attributes.evt.name,\n      resource_type: .attributes.attributes.asset.type,\n      resource_id: .attributes.attributes.asset.id\n    }]'\n```\n\n> **Note:** `--from` and `--to` accept ISO timestamps (e.g., `2026-05-01T14:00:00Z`) or relative values (`1h`, `24h`, `7d`).\n\n### Step 4 — Narrow to product-relevant config changes\n\nFilter to the audit categories most likely to affect the spiking product:\n\n| If this product spiked | Add to query |\n|------------------------|-------------|\n| `llm_observability` | `@evt.name:(Integration OR APM OR \"Log Management\")` |\n| `logs` \u002F `indexed_logs` | `@evt.name:\"Log Management\" @asset.type:(pipeline OR index OR exclusion_filter)` |\n| `apm` \u002F `indexed_spans` | `@evt.name:APM @asset.type:(retention_filter OR sampling_rate)` |\n| `rum` | `@evt.name:RUM` |\n| `metrics` | `@evt.name:Metrics` |\n\nExample for LLM Observability spike:\n\n```bash\npup audit-logs search \\\n  --query \"@evt.name:(Integration OR APM OR \\\"Log Management\\\") @action:(created OR modified)\" \\\n  --from \"SPIKE_TIME_MINUS_24H\" \\\n  --to \"SPIKE_TIME\" \\\n  --limit 100 \\\n  -o json \\\n  | jq '[.data[] | {\n      timestamp: .attributes.timestamp,\n      user: .attributes.attributes.usr.email,\n      action: .attributes.attributes.action,\n      category: .attributes.attributes.evt.name,\n      resource_type: .attributes.attributes.asset.type,\n      resource_id: .attributes.attributes.asset.id\n    }]'\n```\n\n## Output Format\n\n```\nUsage spike detected:\n  Product: \u003Cproduct_family>\n  Spike time: \u003CSPIKE_TIME>\n  Volume: \u003Cbaseline> → \u003Cspike_value> (\u003Cmagnitude>×)\n\nConfiguration changes in 24h preceding spike:\n  \u003Ctimestamp> | \u003Cuser_email> | \u003Caction> \u003Cresource_type> \u003Cresource_id> | \u003Ccategory>\n\nLikely causal change: \u003Cmost-proximate change matching the product family>\n\nConfidence: HIGH (single clear change) \u002F MEDIUM (multiple candidates) \u002F LOW (no matching changes)\n\nNext steps:\n  - Confirm with \u003Cuser_email> whether the change was intentional\n  - If unintentional: revert \u003Cresource_id> and monitor volume\n  - If intentional: update cost forecasts and alert thresholds\n```\n\n## When No Causal Change Is Found\n\n1. The change may predate the 24h window — expand to 72h\n2. The increase may be from application-side instrumentation changes — check deploys\n3. The increase may be organic traffic growth — correlate with product launch or traffic event\n\n## References\n\n- [Usage Metering API](https:\u002F\u002Fdocs.datadoghq.com\u002Fapi\u002Flatest\u002Fusage-metering\u002F)\n- [Audit Trail API](https:\u002F\u002Fdocs.datadoghq.com\u002Fapi\u002Flatest\u002Faudit\u002F)\n- [LLM Observability](https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002F)\n",{"data":40,"body":45},{"name":4,"description":6,"metadata":41},{"version":42,"author":8,"repository":29,"tags":43,"alwaysApply":44},"0.1.0","datadog,audit,cost,usage,spike,finops,dd-audit","false",{"type":46,"children":47},"root",[48,57,63,76,83,214,220,240,246,253,702,740,746,759,765,988,1047,1053,1058,1203,1208,1427,1433,1443,1449,1469,1475,1511],{"type":49,"tag":50,"props":51,"children":53},"element","h1",{"id":52},"audit-trail-cost-usage-spike-investigation",[54],{"type":55,"value":56},"text","Audit Trail: Cost \u002F Usage Spike Investigation",{"type":49,"tag":58,"props":59,"children":60},"p",{},[61],{"type":55,"value":62},"Identify what caused a Datadog usage spike by correlating billing data with configuration change history.",{"type":49,"tag":58,"props":64,"children":65},{},[66,68,74],{"type":55,"value":67},"The causal chain is: ",{"type":49,"tag":69,"props":70,"children":71},"strong",{},[72],{"type":55,"value":73},"someone changed something → that change increased data volume → usage spiked → cost went up",{"type":55,"value":75},". Usage Metering tells you when and what; Audit Trail tells you who made the change.",{"type":49,"tag":77,"props":78,"children":80},"h2",{"id":79},"prerequisites",[81],{"type":55,"value":82},"Prerequisites",{"type":49,"tag":84,"props":85,"children":90},"pre",{"className":86,"code":87,"language":88,"meta":89,"style":89},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","pup auth login   # OAuth2 (recommended) — covers audit queries\n# Usage Metering queries also need DD_API_KEY + DD_APP_KEY\nexport DD_API_KEY=\u003Cyour-api-key>\nexport DD_APP_KEY=\u003Cyour-app-key>\nexport DD_SITE=datadoghq.com\n","bash","",[91],{"type":49,"tag":92,"props":93,"children":94},"code",{"__ignoreMap":89},[95,124,133,165,191],{"type":49,"tag":96,"props":97,"children":100},"span",{"class":98,"line":99},"line",1,[101,107,113,118],{"type":49,"tag":96,"props":102,"children":104},{"style":103},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[105],{"type":55,"value":106},"pup",{"type":49,"tag":96,"props":108,"children":110},{"style":109},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[111],{"type":55,"value":112}," auth",{"type":49,"tag":96,"props":114,"children":115},{"style":109},[116],{"type":55,"value":117}," login",{"type":49,"tag":96,"props":119,"children":121},{"style":120},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[122],{"type":55,"value":123},"   # OAuth2 (recommended) — covers audit queries\n",{"type":49,"tag":96,"props":125,"children":127},{"class":98,"line":126},2,[128],{"type":49,"tag":96,"props":129,"children":130},{"style":120},[131],{"type":55,"value":132},"# Usage Metering queries also need DD_API_KEY + DD_APP_KEY\n",{"type":49,"tag":96,"props":134,"children":136},{"class":98,"line":135},3,[137,143,149,155,160],{"type":49,"tag":96,"props":138,"children":140},{"style":139},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[141],{"type":55,"value":142},"export",{"type":49,"tag":96,"props":144,"children":146},{"style":145},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[147],{"type":55,"value":148}," DD_API_KEY",{"type":49,"tag":96,"props":150,"children":152},{"style":151},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[153],{"type":55,"value":154},"=\u003C",{"type":49,"tag":96,"props":156,"children":157},{"style":145},[158],{"type":55,"value":159},"your-api-key",{"type":49,"tag":96,"props":161,"children":162},{"style":151},[163],{"type":55,"value":164},">\n",{"type":49,"tag":96,"props":166,"children":168},{"class":98,"line":167},4,[169,173,178,182,187],{"type":49,"tag":96,"props":170,"children":171},{"style":139},[172],{"type":55,"value":142},{"type":49,"tag":96,"props":174,"children":175},{"style":145},[176],{"type":55,"value":177}," DD_APP_KEY",{"type":49,"tag":96,"props":179,"children":180},{"style":151},[181],{"type":55,"value":154},{"type":49,"tag":96,"props":183,"children":184},{"style":145},[185],{"type":55,"value":186},"your-app-key",{"type":49,"tag":96,"props":188,"children":189},{"style":151},[190],{"type":55,"value":164},{"type":49,"tag":96,"props":192,"children":194},{"class":98,"line":193},5,[195,199,204,209],{"type":49,"tag":96,"props":196,"children":197},{"style":139},[198],{"type":55,"value":142},{"type":49,"tag":96,"props":200,"children":201},{"style":145},[202],{"type":55,"value":203}," DD_SITE",{"type":49,"tag":96,"props":205,"children":206},{"style":151},[207],{"type":55,"value":208},"=",{"type":49,"tag":96,"props":210,"children":211},{"style":145},[212],{"type":55,"value":213},"datadoghq.com\n",{"type":49,"tag":77,"props":215,"children":217},{"id":216},"scope-boundary",[218],{"type":55,"value":219},"Scope Boundary",{"type":49,"tag":58,"props":221,"children":222},{},[223,225,230,232,238],{"type":55,"value":224},"This skill identifies ",{"type":49,"tag":69,"props":226,"children":227},{},[228],{"type":55,"value":229},"configuration changes",{"type":55,"value":231}," that may have caused a spike. It does not identify which specific user or process ",{"type":49,"tag":233,"props":234,"children":235},"em",{},[236],{"type":55,"value":237},"submitted",{"type":55,"value":239}," the data (e.g., which service sent the LLM spans). For per-submission attribution, use LLM Observability traces or APM instrumentation.",{"type":49,"tag":77,"props":241,"children":243},{"id":242},"investigation-workflow",[244],{"type":55,"value":245},"Investigation Workflow",{"type":49,"tag":247,"props":248,"children":250},"h3",{"id":249},"step-1-identify-the-spike-window-and-product-family",[251],{"type":55,"value":252},"Step 1 — Identify the spike window and product family",{"type":49,"tag":84,"props":254,"children":256},{"className":86,"code":255,"language":88,"meta":89,"style":89},"START=$(date -u -v-7d +\"%Y-%m-%dT%H:%M:%SZ\" 2>\u002Fdev\u002Fnull || date -u -d \"7 days ago\" +\"%Y-%m-%dT%H:%M:%SZ\")\nEND=$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")\n\ncurl -s -G \"https:\u002F\u002Fapi.${DD_SITE}\u002Fapi\u002Fv2\u002Fusage\u002Fhourly_usage\" \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\" \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\" \\\n  --data-urlencode \"filter[timestamp][start]=${START}\" \\\n  --data-urlencode \"filter[timestamp][end]=${END}\" \\\n  --data-urlencode \"filter[product_families]=all\" \\\n  | jq '[.data[] | {\n      timestamp: .attributes.timestamp,\n      product: .attributes.product_family,\n      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]\n    }]'\n",[257],{"type":49,"tag":92,"props":258,"children":259},{"__ignoreMap":89},[260,371,411,420,476,511,545,579,612,637,661,670,679,688],{"type":49,"tag":96,"props":261,"children":262},{"class":98,"line":99},[263,268,273,278,283,288,293,298,303,307,312,317,322,327,331,336,341,346,350,354,358,362,366],{"type":49,"tag":96,"props":264,"children":265},{"style":145},[266],{"type":55,"value":267},"START",{"type":49,"tag":96,"props":269,"children":270},{"style":151},[271],{"type":55,"value":272},"=$(",{"type":49,"tag":96,"props":274,"children":275},{"style":103},[276],{"type":55,"value":277},"date",{"type":49,"tag":96,"props":279,"children":280},{"style":109},[281],{"type":55,"value":282}," -u",{"type":49,"tag":96,"props":284,"children":285},{"style":109},[286],{"type":55,"value":287}," -v-7d",{"type":49,"tag":96,"props":289,"children":290},{"style":109},[291],{"type":55,"value":292}," +",{"type":49,"tag":96,"props":294,"children":295},{"style":151},[296],{"type":55,"value":297},"\"",{"type":49,"tag":96,"props":299,"children":300},{"style":109},[301],{"type":55,"value":302},"%Y-%m-%dT%H:%M:%SZ",{"type":49,"tag":96,"props":304,"children":305},{"style":151},[306],{"type":55,"value":297},{"type":49,"tag":96,"props":308,"children":309},{"style":151},[310],{"type":55,"value":311}," 2>",{"type":49,"tag":96,"props":313,"children":314},{"style":109},[315],{"type":55,"value":316},"\u002Fdev\u002Fnull",{"type":49,"tag":96,"props":318,"children":319},{"style":151},[320],{"type":55,"value":321}," ||",{"type":49,"tag":96,"props":323,"children":324},{"style":103},[325],{"type":55,"value":326}," date",{"type":49,"tag":96,"props":328,"children":329},{"style":109},[330],{"type":55,"value":282},{"type":49,"tag":96,"props":332,"children":333},{"style":109},[334],{"type":55,"value":335}," -d",{"type":49,"tag":96,"props":337,"children":338},{"style":151},[339],{"type":55,"value":340}," \"",{"type":49,"tag":96,"props":342,"children":343},{"style":109},[344],{"type":55,"value":345},"7 days ago",{"type":49,"tag":96,"props":347,"children":348},{"style":151},[349],{"type":55,"value":297},{"type":49,"tag":96,"props":351,"children":352},{"style":109},[353],{"type":55,"value":292},{"type":49,"tag":96,"props":355,"children":356},{"style":151},[357],{"type":55,"value":297},{"type":49,"tag":96,"props":359,"children":360},{"style":109},[361],{"type":55,"value":302},{"type":49,"tag":96,"props":363,"children":364},{"style":151},[365],{"type":55,"value":297},{"type":49,"tag":96,"props":367,"children":368},{"style":151},[369],{"type":55,"value":370},")\n",{"type":49,"tag":96,"props":372,"children":373},{"class":98,"line":126},[374,379,383,387,391,395,399,403,407],{"type":49,"tag":96,"props":375,"children":376},{"style":145},[377],{"type":55,"value":378},"END",{"type":49,"tag":96,"props":380,"children":381},{"style":151},[382],{"type":55,"value":272},{"type":49,"tag":96,"props":384,"children":385},{"style":103},[386],{"type":55,"value":277},{"type":49,"tag":96,"props":388,"children":389},{"style":109},[390],{"type":55,"value":282},{"type":49,"tag":96,"props":392,"children":393},{"style":109},[394],{"type":55,"value":292},{"type":49,"tag":96,"props":396,"children":397},{"style":151},[398],{"type":55,"value":297},{"type":49,"tag":96,"props":400,"children":401},{"style":109},[402],{"type":55,"value":302},{"type":49,"tag":96,"props":404,"children":405},{"style":151},[406],{"type":55,"value":297},{"type":49,"tag":96,"props":408,"children":409},{"style":151},[410],{"type":55,"value":370},{"type":49,"tag":96,"props":412,"children":413},{"class":98,"line":135},[414],{"type":49,"tag":96,"props":415,"children":417},{"emptyLinePlaceholder":416},true,[418],{"type":55,"value":419},"\n",{"type":49,"tag":96,"props":421,"children":422},{"class":98,"line":167},[423,428,433,438,442,447,452,457,462,467,471],{"type":49,"tag":96,"props":424,"children":425},{"style":103},[426],{"type":55,"value":427},"curl",{"type":49,"tag":96,"props":429,"children":430},{"style":109},[431],{"type":55,"value":432}," -s",{"type":49,"tag":96,"props":434,"children":435},{"style":109},[436],{"type":55,"value":437}," -G",{"type":49,"tag":96,"props":439,"children":440},{"style":151},[441],{"type":55,"value":340},{"type":49,"tag":96,"props":443,"children":444},{"style":109},[445],{"type":55,"value":446},"https:\u002F\u002Fapi.",{"type":49,"tag":96,"props":448,"children":449},{"style":151},[450],{"type":55,"value":451},"${",{"type":49,"tag":96,"props":453,"children":454},{"style":145},[455],{"type":55,"value":456},"DD_SITE",{"type":49,"tag":96,"props":458,"children":459},{"style":151},[460],{"type":55,"value":461},"}",{"type":49,"tag":96,"props":463,"children":464},{"style":109},[465],{"type":55,"value":466},"\u002Fapi\u002Fv2\u002Fusage\u002Fhourly_usage",{"type":49,"tag":96,"props":468,"children":469},{"style":151},[470],{"type":55,"value":297},{"type":49,"tag":96,"props":472,"children":473},{"style":145},[474],{"type":55,"value":475}," \\\n",{"type":49,"tag":96,"props":477,"children":478},{"class":98,"line":193},[479,484,488,493,497,502,507],{"type":49,"tag":96,"props":480,"children":481},{"style":109},[482],{"type":55,"value":483},"  -H",{"type":49,"tag":96,"props":485,"children":486},{"style":151},[487],{"type":55,"value":340},{"type":49,"tag":96,"props":489,"children":490},{"style":109},[491],{"type":55,"value":492},"DD-API-KEY: ",{"type":49,"tag":96,"props":494,"children":495},{"style":151},[496],{"type":55,"value":451},{"type":49,"tag":96,"props":498,"children":499},{"style":145},[500],{"type":55,"value":501},"DD_API_KEY",{"type":49,"tag":96,"props":503,"children":504},{"style":151},[505],{"type":55,"value":506},"}\"",{"type":49,"tag":96,"props":508,"children":509},{"style":145},[510],{"type":55,"value":475},{"type":49,"tag":96,"props":512,"children":514},{"class":98,"line":513},6,[515,519,523,528,532,537,541],{"type":49,"tag":96,"props":516,"children":517},{"style":109},[518],{"type":55,"value":483},{"type":49,"tag":96,"props":520,"children":521},{"style":151},[522],{"type":55,"value":340},{"type":49,"tag":96,"props":524,"children":525},{"style":109},[526],{"type":55,"value":527},"DD-APPLICATION-KEY: ",{"type":49,"tag":96,"props":529,"children":530},{"style":151},[531],{"type":55,"value":451},{"type":49,"tag":96,"props":533,"children":534},{"style":145},[535],{"type":55,"value":536},"DD_APP_KEY",{"type":49,"tag":96,"props":538,"children":539},{"style":151},[540],{"type":55,"value":506},{"type":49,"tag":96,"props":542,"children":543},{"style":145},[544],{"type":55,"value":475},{"type":49,"tag":96,"props":546,"children":548},{"class":98,"line":547},7,[549,554,558,563,567,571,575],{"type":49,"tag":96,"props":550,"children":551},{"style":109},[552],{"type":55,"value":553},"  --data-urlencode",{"type":49,"tag":96,"props":555,"children":556},{"style":151},[557],{"type":55,"value":340},{"type":49,"tag":96,"props":559,"children":560},{"style":109},[561],{"type":55,"value":562},"filter[timestamp][start]=",{"type":49,"tag":96,"props":564,"children":565},{"style":151},[566],{"type":55,"value":451},{"type":49,"tag":96,"props":568,"children":569},{"style":145},[570],{"type":55,"value":267},{"type":49,"tag":96,"props":572,"children":573},{"style":151},[574],{"type":55,"value":506},{"type":49,"tag":96,"props":576,"children":577},{"style":145},[578],{"type":55,"value":475},{"type":49,"tag":96,"props":580,"children":582},{"class":98,"line":581},8,[583,587,591,596,600,604,608],{"type":49,"tag":96,"props":584,"children":585},{"style":109},[586],{"type":55,"value":553},{"type":49,"tag":96,"props":588,"children":589},{"style":151},[590],{"type":55,"value":340},{"type":49,"tag":96,"props":592,"children":593},{"style":109},[594],{"type":55,"value":595},"filter[timestamp][end]=",{"type":49,"tag":96,"props":597,"children":598},{"style":151},[599],{"type":55,"value":451},{"type":49,"tag":96,"props":601,"children":602},{"style":145},[603],{"type":55,"value":378},{"type":49,"tag":96,"props":605,"children":606},{"style":151},[607],{"type":55,"value":506},{"type":49,"tag":96,"props":609,"children":610},{"style":145},[611],{"type":55,"value":475},{"type":49,"tag":96,"props":613,"children":615},{"class":98,"line":614},9,[616,620,624,629,633],{"type":49,"tag":96,"props":617,"children":618},{"style":109},[619],{"type":55,"value":553},{"type":49,"tag":96,"props":621,"children":622},{"style":151},[623],{"type":55,"value":340},{"type":49,"tag":96,"props":625,"children":626},{"style":109},[627],{"type":55,"value":628},"filter[product_families]=all",{"type":49,"tag":96,"props":630,"children":631},{"style":151},[632],{"type":55,"value":297},{"type":49,"tag":96,"props":634,"children":635},{"style":145},[636],{"type":55,"value":475},{"type":49,"tag":96,"props":638,"children":640},{"class":98,"line":639},10,[641,646,651,656],{"type":49,"tag":96,"props":642,"children":643},{"style":151},[644],{"type":55,"value":645},"  |",{"type":49,"tag":96,"props":647,"children":648},{"style":103},[649],{"type":55,"value":650}," jq",{"type":49,"tag":96,"props":652,"children":653},{"style":151},[654],{"type":55,"value":655}," '",{"type":49,"tag":96,"props":657,"children":658},{"style":109},[659],{"type":55,"value":660},"[.data[] | {\n",{"type":49,"tag":96,"props":662,"children":664},{"class":98,"line":663},11,[665],{"type":49,"tag":96,"props":666,"children":667},{"style":109},[668],{"type":55,"value":669},"      timestamp: .attributes.timestamp,\n",{"type":49,"tag":96,"props":671,"children":673},{"class":98,"line":672},12,[674],{"type":49,"tag":96,"props":675,"children":676},{"style":109},[677],{"type":55,"value":678},"      product: .attributes.product_family,\n",{"type":49,"tag":96,"props":680,"children":682},{"class":98,"line":681},13,[683],{"type":49,"tag":96,"props":684,"children":685},{"style":109},[686],{"type":55,"value":687},"      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]\n",{"type":49,"tag":96,"props":689,"children":691},{"class":98,"line":690},14,[692,697],{"type":49,"tag":96,"props":693,"children":694},{"style":109},[695],{"type":55,"value":696},"    }]",{"type":49,"tag":96,"props":698,"children":699},{"style":151},[700],{"type":55,"value":701},"'\n",{"type":49,"tag":58,"props":703,"children":704},{},[705,710,712,718,720,726,727,733,734],{"type":49,"tag":69,"props":706,"children":707},{},[708],{"type":55,"value":709},"Product families with LLM\u002FAI coverage:",{"type":55,"value":711}," ",{"type":49,"tag":92,"props":713,"children":715},{"className":714},[],[716],{"type":55,"value":717},"llm_observability",{"type":55,"value":719},", ",{"type":49,"tag":92,"props":721,"children":723},{"className":722},[],[724],{"type":55,"value":725},"bits_ai",{"type":55,"value":719},{"type":49,"tag":92,"props":728,"children":730},{"className":729},[],[731],{"type":55,"value":732},"logs",{"type":55,"value":719},{"type":49,"tag":92,"props":735,"children":737},{"className":736},[],[738],{"type":55,"value":739},"apm",{"type":49,"tag":247,"props":741,"children":743},{"id":742},"step-2-pinpoint-the-spike",[744],{"type":55,"value":745},"Step 2 — Pinpoint the spike",{"type":49,"tag":58,"props":747,"children":748},{},[749,751,757],{"type":55,"value":750},"From Step 1, identify the hour\u002Fday where volume jumped. Note the timestamp as ",{"type":49,"tag":92,"props":752,"children":754},{"className":753},[],[755],{"type":55,"value":756},"SPIKE_TIME",{"type":55,"value":758},".",{"type":49,"tag":247,"props":760,"children":762},{"id":761},"step-3-search-audit-trail-for-config-changes-in-the-24h-preceding-the-spike",[763],{"type":55,"value":764},"Step 3 — Search Audit Trail for config changes in the 24h preceding the spike",{"type":49,"tag":84,"props":766,"children":768},{"className":86,"code":767,"language":88,"meta":89,"style":89},"pup audit-logs search \\\n  --query \"@action:(created OR modified OR deleted)\" \\\n  --from \"SPIKE_TIME_MINUS_24H\" \\\n  --to \"SPIKE_TIME\" \\\n  --limit 200 \\\n  -o json \\\n  | jq '[.data[] | {\n      timestamp: .attributes.timestamp,\n      user: .attributes.attributes.usr.email,\n      actor_type: .attributes.attributes.evt.actor.type,\n      action: .attributes.attributes.action,\n      event_category: .attributes.attributes.evt.name,\n      resource_type: .attributes.attributes.asset.type,\n      resource_id: .attributes.attributes.asset.id\n    }]'\n",[769],{"type":49,"tag":92,"props":770,"children":771},{"__ignoreMap":89},[772,793,818,843,867,885,902,921,928,936,944,952,960,968,976],{"type":49,"tag":96,"props":773,"children":774},{"class":98,"line":99},[775,779,784,789],{"type":49,"tag":96,"props":776,"children":777},{"style":103},[778],{"type":55,"value":106},{"type":49,"tag":96,"props":780,"children":781},{"style":109},[782],{"type":55,"value":783}," audit-logs",{"type":49,"tag":96,"props":785,"children":786},{"style":109},[787],{"type":55,"value":788}," search",{"type":49,"tag":96,"props":790,"children":791},{"style":145},[792],{"type":55,"value":475},{"type":49,"tag":96,"props":794,"children":795},{"class":98,"line":126},[796,801,805,810,814],{"type":49,"tag":96,"props":797,"children":798},{"style":109},[799],{"type":55,"value":800},"  --query",{"type":49,"tag":96,"props":802,"children":803},{"style":151},[804],{"type":55,"value":340},{"type":49,"tag":96,"props":806,"children":807},{"style":109},[808],{"type":55,"value":809},"@action:(created OR modified OR deleted)",{"type":49,"tag":96,"props":811,"children":812},{"style":151},[813],{"type":55,"value":297},{"type":49,"tag":96,"props":815,"children":816},{"style":145},[817],{"type":55,"value":475},{"type":49,"tag":96,"props":819,"children":820},{"class":98,"line":135},[821,826,830,835,839],{"type":49,"tag":96,"props":822,"children":823},{"style":109},[824],{"type":55,"value":825},"  --from",{"type":49,"tag":96,"props":827,"children":828},{"style":151},[829],{"type":55,"value":340},{"type":49,"tag":96,"props":831,"children":832},{"style":109},[833],{"type":55,"value":834},"SPIKE_TIME_MINUS_24H",{"type":49,"tag":96,"props":836,"children":837},{"style":151},[838],{"type":55,"value":297},{"type":49,"tag":96,"props":840,"children":841},{"style":145},[842],{"type":55,"value":475},{"type":49,"tag":96,"props":844,"children":845},{"class":98,"line":167},[846,851,855,859,863],{"type":49,"tag":96,"props":847,"children":848},{"style":109},[849],{"type":55,"value":850},"  --to",{"type":49,"tag":96,"props":852,"children":853},{"style":151},[854],{"type":55,"value":340},{"type":49,"tag":96,"props":856,"children":857},{"style":109},[858],{"type":55,"value":756},{"type":49,"tag":96,"props":860,"children":861},{"style":151},[862],{"type":55,"value":297},{"type":49,"tag":96,"props":864,"children":865},{"style":145},[866],{"type":55,"value":475},{"type":49,"tag":96,"props":868,"children":869},{"class":98,"line":193},[870,875,881],{"type":49,"tag":96,"props":871,"children":872},{"style":109},[873],{"type":55,"value":874},"  --limit",{"type":49,"tag":96,"props":876,"children":878},{"style":877},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[879],{"type":55,"value":880}," 200",{"type":49,"tag":96,"props":882,"children":883},{"style":145},[884],{"type":55,"value":475},{"type":49,"tag":96,"props":886,"children":887},{"class":98,"line":513},[888,893,898],{"type":49,"tag":96,"props":889,"children":890},{"style":109},[891],{"type":55,"value":892},"  -o",{"type":49,"tag":96,"props":894,"children":895},{"style":109},[896],{"type":55,"value":897}," json",{"type":49,"tag":96,"props":899,"children":900},{"style":145},[901],{"type":55,"value":475},{"type":49,"tag":96,"props":903,"children":904},{"class":98,"line":547},[905,909,913,917],{"type":49,"tag":96,"props":906,"children":907},{"style":151},[908],{"type":55,"value":645},{"type":49,"tag":96,"props":910,"children":911},{"style":103},[912],{"type":55,"value":650},{"type":49,"tag":96,"props":914,"children":915},{"style":151},[916],{"type":55,"value":655},{"type":49,"tag":96,"props":918,"children":919},{"style":109},[920],{"type":55,"value":660},{"type":49,"tag":96,"props":922,"children":923},{"class":98,"line":581},[924],{"type":49,"tag":96,"props":925,"children":926},{"style":109},[927],{"type":55,"value":669},{"type":49,"tag":96,"props":929,"children":930},{"class":98,"line":614},[931],{"type":49,"tag":96,"props":932,"children":933},{"style":109},[934],{"type":55,"value":935},"      user: .attributes.attributes.usr.email,\n",{"type":49,"tag":96,"props":937,"children":938},{"class":98,"line":639},[939],{"type":49,"tag":96,"props":940,"children":941},{"style":109},[942],{"type":55,"value":943},"      actor_type: .attributes.attributes.evt.actor.type,\n",{"type":49,"tag":96,"props":945,"children":946},{"class":98,"line":663},[947],{"type":49,"tag":96,"props":948,"children":949},{"style":109},[950],{"type":55,"value":951},"      action: .attributes.attributes.action,\n",{"type":49,"tag":96,"props":953,"children":954},{"class":98,"line":672},[955],{"type":49,"tag":96,"props":956,"children":957},{"style":109},[958],{"type":55,"value":959},"      event_category: .attributes.attributes.evt.name,\n",{"type":49,"tag":96,"props":961,"children":962},{"class":98,"line":681},[963],{"type":49,"tag":96,"props":964,"children":965},{"style":109},[966],{"type":55,"value":967},"      resource_type: .attributes.attributes.asset.type,\n",{"type":49,"tag":96,"props":969,"children":970},{"class":98,"line":690},[971],{"type":49,"tag":96,"props":972,"children":973},{"style":109},[974],{"type":55,"value":975},"      resource_id: .attributes.attributes.asset.id\n",{"type":49,"tag":96,"props":977,"children":979},{"class":98,"line":978},15,[980,984],{"type":49,"tag":96,"props":981,"children":982},{"style":109},[983],{"type":55,"value":696},{"type":49,"tag":96,"props":985,"children":986},{"style":151},[987],{"type":55,"value":701},{"type":49,"tag":989,"props":990,"children":991},"blockquote",{},[992],{"type":49,"tag":58,"props":993,"children":994},{},[995,1000,1001,1007,1009,1015,1017,1023,1025,1031,1032,1038,1039,1045],{"type":49,"tag":69,"props":996,"children":997},{},[998],{"type":55,"value":999},"Note:",{"type":55,"value":711},{"type":49,"tag":92,"props":1002,"children":1004},{"className":1003},[],[1005],{"type":55,"value":1006},"--from",{"type":55,"value":1008}," and ",{"type":49,"tag":92,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":55,"value":1014},"--to",{"type":55,"value":1016}," accept ISO timestamps (e.g., ",{"type":49,"tag":92,"props":1018,"children":1020},{"className":1019},[],[1021],{"type":55,"value":1022},"2026-05-01T14:00:00Z",{"type":55,"value":1024},") or relative values (",{"type":49,"tag":92,"props":1026,"children":1028},{"className":1027},[],[1029],{"type":55,"value":1030},"1h",{"type":55,"value":719},{"type":49,"tag":92,"props":1033,"children":1035},{"className":1034},[],[1036],{"type":55,"value":1037},"24h",{"type":55,"value":719},{"type":49,"tag":92,"props":1040,"children":1042},{"className":1041},[],[1043],{"type":55,"value":1044},"7d",{"type":55,"value":1046},").",{"type":49,"tag":247,"props":1048,"children":1050},{"id":1049},"step-4-narrow-to-product-relevant-config-changes",[1051],{"type":55,"value":1052},"Step 4 — Narrow to product-relevant config changes",{"type":49,"tag":58,"props":1054,"children":1055},{},[1056],{"type":55,"value":1057},"Filter to the audit categories most likely to affect the spiking product:",{"type":49,"tag":1059,"props":1060,"children":1061},"table",{},[1062,1081],{"type":49,"tag":1063,"props":1064,"children":1065},"thead",{},[1066],{"type":49,"tag":1067,"props":1068,"children":1069},"tr",{},[1070,1076],{"type":49,"tag":1071,"props":1072,"children":1073},"th",{},[1074],{"type":55,"value":1075},"If this product spiked",{"type":49,"tag":1071,"props":1077,"children":1078},{},[1079],{"type":55,"value":1080},"Add to query",{"type":49,"tag":1082,"props":1083,"children":1084},"tbody",{},[1085,1106,1134,1161,1182],{"type":49,"tag":1067,"props":1086,"children":1087},{},[1088,1097],{"type":49,"tag":1089,"props":1090,"children":1091},"td",{},[1092],{"type":49,"tag":92,"props":1093,"children":1095},{"className":1094},[],[1096],{"type":55,"value":717},{"type":49,"tag":1089,"props":1098,"children":1099},{},[1100],{"type":49,"tag":92,"props":1101,"children":1103},{"className":1102},[],[1104],{"type":55,"value":1105},"@evt.name:(Integration OR APM OR \"Log Management\")",{"type":49,"tag":1067,"props":1107,"children":1108},{},[1109,1125],{"type":49,"tag":1089,"props":1110,"children":1111},{},[1112,1117,1119],{"type":49,"tag":92,"props":1113,"children":1115},{"className":1114},[],[1116],{"type":55,"value":732},{"type":55,"value":1118}," \u002F ",{"type":49,"tag":92,"props":1120,"children":1122},{"className":1121},[],[1123],{"type":55,"value":1124},"indexed_logs",{"type":49,"tag":1089,"props":1126,"children":1127},{},[1128],{"type":49,"tag":92,"props":1129,"children":1131},{"className":1130},[],[1132],{"type":55,"value":1133},"@evt.name:\"Log Management\" @asset.type:(pipeline OR index OR exclusion_filter)",{"type":49,"tag":1067,"props":1135,"children":1136},{},[1137,1152],{"type":49,"tag":1089,"props":1138,"children":1139},{},[1140,1145,1146],{"type":49,"tag":92,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":55,"value":739},{"type":55,"value":1118},{"type":49,"tag":92,"props":1147,"children":1149},{"className":1148},[],[1150],{"type":55,"value":1151},"indexed_spans",{"type":49,"tag":1089,"props":1153,"children":1154},{},[1155],{"type":49,"tag":92,"props":1156,"children":1158},{"className":1157},[],[1159],{"type":55,"value":1160},"@evt.name:APM @asset.type:(retention_filter OR sampling_rate)",{"type":49,"tag":1067,"props":1162,"children":1163},{},[1164,1173],{"type":49,"tag":1089,"props":1165,"children":1166},{},[1167],{"type":49,"tag":92,"props":1168,"children":1170},{"className":1169},[],[1171],{"type":55,"value":1172},"rum",{"type":49,"tag":1089,"props":1174,"children":1175},{},[1176],{"type":49,"tag":92,"props":1177,"children":1179},{"className":1178},[],[1180],{"type":55,"value":1181},"@evt.name:RUM",{"type":49,"tag":1067,"props":1183,"children":1184},{},[1185,1194],{"type":49,"tag":1089,"props":1186,"children":1187},{},[1188],{"type":49,"tag":92,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":55,"value":1193},"metrics",{"type":49,"tag":1089,"props":1195,"children":1196},{},[1197],{"type":49,"tag":92,"props":1198,"children":1200},{"className":1199},[],[1201],{"type":55,"value":1202},"@evt.name:Metrics",{"type":49,"tag":58,"props":1204,"children":1205},{},[1206],{"type":55,"value":1207},"Example for LLM Observability spike:",{"type":49,"tag":84,"props":1209,"children":1211},{"className":86,"code":1210,"language":88,"meta":89,"style":89},"pup audit-logs search \\\n  --query \"@evt.name:(Integration OR APM OR \\\"Log Management\\\") @action:(created OR modified)\" \\\n  --from \"SPIKE_TIME_MINUS_24H\" \\\n  --to \"SPIKE_TIME\" \\\n  --limit 100 \\\n  -o json \\\n  | jq '[.data[] | {\n      timestamp: .attributes.timestamp,\n      user: .attributes.attributes.usr.email,\n      action: .attributes.attributes.action,\n      category: .attributes.attributes.evt.name,\n      resource_type: .attributes.attributes.asset.type,\n      resource_id: .attributes.attributes.asset.id\n    }]'\n",[1212],{"type":49,"tag":92,"props":1213,"children":1214},{"__ignoreMap":89},[1215,1234,1277,1300,1323,1339,1354,1373,1380,1387,1394,1402,1409,1416],{"type":49,"tag":96,"props":1216,"children":1217},{"class":98,"line":99},[1218,1222,1226,1230],{"type":49,"tag":96,"props":1219,"children":1220},{"style":103},[1221],{"type":55,"value":106},{"type":49,"tag":96,"props":1223,"children":1224},{"style":109},[1225],{"type":55,"value":783},{"type":49,"tag":96,"props":1227,"children":1228},{"style":109},[1229],{"type":55,"value":788},{"type":49,"tag":96,"props":1231,"children":1232},{"style":145},[1233],{"type":55,"value":475},{"type":49,"tag":96,"props":1235,"children":1236},{"class":98,"line":126},[1237,1241,1245,1250,1255,1260,1264,1269,1273],{"type":49,"tag":96,"props":1238,"children":1239},{"style":109},[1240],{"type":55,"value":800},{"type":49,"tag":96,"props":1242,"children":1243},{"style":151},[1244],{"type":55,"value":340},{"type":49,"tag":96,"props":1246,"children":1247},{"style":109},[1248],{"type":55,"value":1249},"@evt.name:(Integration OR APM OR ",{"type":49,"tag":96,"props":1251,"children":1252},{"style":145},[1253],{"type":55,"value":1254},"\\\"",{"type":49,"tag":96,"props":1256,"children":1257},{"style":109},[1258],{"type":55,"value":1259},"Log Management",{"type":49,"tag":96,"props":1261,"children":1262},{"style":145},[1263],{"type":55,"value":1254},{"type":49,"tag":96,"props":1265,"children":1266},{"style":109},[1267],{"type":55,"value":1268},") @action:(created OR modified)",{"type":49,"tag":96,"props":1270,"children":1271},{"style":151},[1272],{"type":55,"value":297},{"type":49,"tag":96,"props":1274,"children":1275},{"style":145},[1276],{"type":55,"value":475},{"type":49,"tag":96,"props":1278,"children":1279},{"class":98,"line":135},[1280,1284,1288,1292,1296],{"type":49,"tag":96,"props":1281,"children":1282},{"style":109},[1283],{"type":55,"value":825},{"type":49,"tag":96,"props":1285,"children":1286},{"style":151},[1287],{"type":55,"value":340},{"type":49,"tag":96,"props":1289,"children":1290},{"style":109},[1291],{"type":55,"value":834},{"type":49,"tag":96,"props":1293,"children":1294},{"style":151},[1295],{"type":55,"value":297},{"type":49,"tag":96,"props":1297,"children":1298},{"style":145},[1299],{"type":55,"value":475},{"type":49,"tag":96,"props":1301,"children":1302},{"class":98,"line":167},[1303,1307,1311,1315,1319],{"type":49,"tag":96,"props":1304,"children":1305},{"style":109},[1306],{"type":55,"value":850},{"type":49,"tag":96,"props":1308,"children":1309},{"style":151},[1310],{"type":55,"value":340},{"type":49,"tag":96,"props":1312,"children":1313},{"style":109},[1314],{"type":55,"value":756},{"type":49,"tag":96,"props":1316,"children":1317},{"style":151},[1318],{"type":55,"value":297},{"type":49,"tag":96,"props":1320,"children":1321},{"style":145},[1322],{"type":55,"value":475},{"type":49,"tag":96,"props":1324,"children":1325},{"class":98,"line":193},[1326,1330,1335],{"type":49,"tag":96,"props":1327,"children":1328},{"style":109},[1329],{"type":55,"value":874},{"type":49,"tag":96,"props":1331,"children":1332},{"style":877},[1333],{"type":55,"value":1334}," 100",{"type":49,"tag":96,"props":1336,"children":1337},{"style":145},[1338],{"type":55,"value":475},{"type":49,"tag":96,"props":1340,"children":1341},{"class":98,"line":513},[1342,1346,1350],{"type":49,"tag":96,"props":1343,"children":1344},{"style":109},[1345],{"type":55,"value":892},{"type":49,"tag":96,"props":1347,"children":1348},{"style":109},[1349],{"type":55,"value":897},{"type":49,"tag":96,"props":1351,"children":1352},{"style":145},[1353],{"type":55,"value":475},{"type":49,"tag":96,"props":1355,"children":1356},{"class":98,"line":547},[1357,1361,1365,1369],{"type":49,"tag":96,"props":1358,"children":1359},{"style":151},[1360],{"type":55,"value":645},{"type":49,"tag":96,"props":1362,"children":1363},{"style":103},[1364],{"type":55,"value":650},{"type":49,"tag":96,"props":1366,"children":1367},{"style":151},[1368],{"type":55,"value":655},{"type":49,"tag":96,"props":1370,"children":1371},{"style":109},[1372],{"type":55,"value":660},{"type":49,"tag":96,"props":1374,"children":1375},{"class":98,"line":581},[1376],{"type":49,"tag":96,"props":1377,"children":1378},{"style":109},[1379],{"type":55,"value":669},{"type":49,"tag":96,"props":1381,"children":1382},{"class":98,"line":614},[1383],{"type":49,"tag":96,"props":1384,"children":1385},{"style":109},[1386],{"type":55,"value":935},{"type":49,"tag":96,"props":1388,"children":1389},{"class":98,"line":639},[1390],{"type":49,"tag":96,"props":1391,"children":1392},{"style":109},[1393],{"type":55,"value":951},{"type":49,"tag":96,"props":1395,"children":1396},{"class":98,"line":663},[1397],{"type":49,"tag":96,"props":1398,"children":1399},{"style":109},[1400],{"type":55,"value":1401},"      category: .attributes.attributes.evt.name,\n",{"type":49,"tag":96,"props":1403,"children":1404},{"class":98,"line":672},[1405],{"type":49,"tag":96,"props":1406,"children":1407},{"style":109},[1408],{"type":55,"value":967},{"type":49,"tag":96,"props":1410,"children":1411},{"class":98,"line":681},[1412],{"type":49,"tag":96,"props":1413,"children":1414},{"style":109},[1415],{"type":55,"value":975},{"type":49,"tag":96,"props":1417,"children":1418},{"class":98,"line":690},[1419,1423],{"type":49,"tag":96,"props":1420,"children":1421},{"style":109},[1422],{"type":55,"value":696},{"type":49,"tag":96,"props":1424,"children":1425},{"style":151},[1426],{"type":55,"value":701},{"type":49,"tag":77,"props":1428,"children":1430},{"id":1429},"output-format",[1431],{"type":55,"value":1432},"Output Format",{"type":49,"tag":84,"props":1434,"children":1438},{"className":1435,"code":1437,"language":55},[1436],"language-text","Usage spike detected:\n  Product: \u003Cproduct_family>\n  Spike time: \u003CSPIKE_TIME>\n  Volume: \u003Cbaseline> → \u003Cspike_value> (\u003Cmagnitude>×)\n\nConfiguration changes in 24h preceding spike:\n  \u003Ctimestamp> | \u003Cuser_email> | \u003Caction> \u003Cresource_type> \u003Cresource_id> | \u003Ccategory>\n\nLikely causal change: \u003Cmost-proximate change matching the product family>\n\nConfidence: HIGH (single clear change) \u002F MEDIUM (multiple candidates) \u002F LOW (no matching changes)\n\nNext steps:\n  - Confirm with \u003Cuser_email> whether the change was intentional\n  - If unintentional: revert \u003Cresource_id> and monitor volume\n  - If intentional: update cost forecasts and alert thresholds\n",[1439],{"type":49,"tag":92,"props":1440,"children":1441},{"__ignoreMap":89},[1442],{"type":55,"value":1437},{"type":49,"tag":77,"props":1444,"children":1446},{"id":1445},"when-no-causal-change-is-found",[1447],{"type":55,"value":1448},"When No Causal Change Is Found",{"type":49,"tag":1450,"props":1451,"children":1452},"ol",{},[1453,1459,1464],{"type":49,"tag":1454,"props":1455,"children":1456},"li",{},[1457],{"type":55,"value":1458},"The change may predate the 24h window — expand to 72h",{"type":49,"tag":1454,"props":1460,"children":1461},{},[1462],{"type":55,"value":1463},"The increase may be from application-side instrumentation changes — check deploys",{"type":49,"tag":1454,"props":1465,"children":1466},{},[1467],{"type":55,"value":1468},"The increase may be organic traffic growth — correlate with product launch or traffic event",{"type":49,"tag":77,"props":1470,"children":1472},{"id":1471},"references",[1473],{"type":55,"value":1474},"References",{"type":49,"tag":1476,"props":1477,"children":1478},"ul",{},[1479,1491,1501],{"type":49,"tag":1454,"props":1480,"children":1481},{},[1482],{"type":49,"tag":1483,"props":1484,"children":1488},"a",{"href":1485,"rel":1486},"https:\u002F\u002Fdocs.datadoghq.com\u002Fapi\u002Flatest\u002Fusage-metering\u002F",[1487],"nofollow",[1489],{"type":55,"value":1490},"Usage Metering API",{"type":49,"tag":1454,"props":1492,"children":1493},{},[1494],{"type":49,"tag":1483,"props":1495,"children":1498},{"href":1496,"rel":1497},"https:\u002F\u002Fdocs.datadoghq.com\u002Fapi\u002Flatest\u002Faudit\u002F",[1487],[1499],{"type":55,"value":1500},"Audit Trail API",{"type":49,"tag":1454,"props":1502,"children":1503},{},[1504],{"type":49,"tag":1483,"props":1505,"children":1508},{"href":1506,"rel":1507},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002F",[1487],[1509],{"type":55,"value":1510},"LLM Observability",{"type":49,"tag":1512,"props":1513,"children":1514},"style",{},[1515],{"type":55,"value":1516},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1518,"total":1623},[1519,1534,1552,1563,1579,1593,1609],{"slug":1520,"name":1520,"fn":1521,"description":1522,"org":1523,"tags":1524,"stars":28,"repoUrl":29,"updatedAt":1533},"agent-install","install Datadog Agent on Kubernetes","Install the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1525,1526,1529,1532],{"name":23,"slug":24,"type":15},{"name":1527,"slug":1528,"type":15},"Deployment","deployment",{"name":1530,"slug":1531,"type":15},"Kubernetes","kubernetes",{"name":13,"slug":14,"type":15},"2026-04-15T04:57:27.489805",{"slug":1535,"name":1535,"fn":1536,"description":1537,"org":1538,"tags":1539,"stars":28,"repoUrl":29,"updatedAt":1551},"agent-observability-auto-experiment","run iterative code improvements using Datadog data","Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says \"run an auto experiment\", \"hill-climb this code\", \"iteratively improve X and measure the delta\", \"optimize this prompt\u002Ffile against my traces\", \"auto-optimize against LLM-Obs\", or wants the local equivalent of the auto_experiments worker. Works from a local dataset file, an ml_app, a dataset_id, or a list of trace_ids.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1540,1541,1544,1547,1550],{"name":23,"slug":24,"type":15},{"name":1542,"slug":1543,"type":15},"Debugging","debugging",{"name":1545,"slug":1546,"type":15},"Evals","evals",{"name":1548,"slug":1549,"type":15},"LLM","llm",{"name":13,"slug":14,"type":15},"2026-07-31T05:52:13.711906",{"slug":1553,"name":1553,"fn":1554,"description":1555,"org":1556,"tags":1557,"stars":28,"repoUrl":29,"updatedAt":1562},"agent-observability-eval-bootstrap","bootstrap evaluators from production traces","Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says \"bootstrap evaluators\", \"generate evaluators\", \"create evals from traces\", \"eval bootstrap\", \"write evaluators\", \"build eval suite\", \"publish evaluators\", or wants to generate BaseEvaluator\u002FLLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1558,1559,1560,1561],{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:19.259734",{"slug":1564,"name":1564,"fn":1565,"description":1566,"org":1567,"tags":1568,"stars":28,"repoUrl":29,"updatedAt":1578},"agent-observability-eval-pipeline","run agent observability and evaluation pipelines","End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a \"continue\" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-py-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says \"run the eval pipeline\", \"go from traces to evals\", \"bootstrap evals end to end\", \"classify then RCA then bootstrap\", \"build an eval set from scratch\", \"onboard me to datasets and experiments\", \"walk me through experiments\", \"I have an ml_app, now what\", \"Agent Observability onboarding\", \"guided experiment setup\", \"from traces to experiments\", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after \u003Cphase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at \u003Cphase>`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1569,1572,1575,1576,1577],{"name":1570,"slug":1571,"type":15},"Agents","agents",{"name":1573,"slug":1574,"type":15},"Data Pipeline","data-pipeline",{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:17.231423",{"slug":1580,"name":1580,"fn":1581,"description":1582,"org":1583,"tags":1584,"stars":28,"repoUrl":29,"updatedAt":1592},"agent-observability-experiment-analyzer","analyze LLM experiment results","Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says \"analyze experiment\", \"compare experiments\", \"analyze against baseline\", or provides one or two experiment IDs for analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1585,1588,1589,1590,1591],{"name":1586,"slug":1587,"type":15},"Analytics","analytics",{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:21.212498",{"slug":1594,"name":1594,"fn":1595,"description":1596,"org":1597,"tags":1598,"stars":28,"repoUrl":29,"updatedAt":1608},"agent-observability-experiment-py-bootstrap","generate Python experiment clients for LLM observability","Generates a self-contained Python experiment client that uses the ddtrace.llmobs SDK. Emits either a runnable .py script or a Jupyter .ipynb notebook matching the canonical DataDog reference notebook style. Use when the user says \"generate Python experiment\", \"write an SDK experiment\", \"create a ddtrace experiment\", \"Python notebook experiment\", \"use the Agent Observability SDK\", or has `ddtrace` installed and wants idiomatic SDK code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1599,1600,1603,1604,1605],{"name":23,"slug":24,"type":15},{"name":1601,"slug":1602,"type":15},"Jupyter","jupyter",{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"name":1606,"slug":1607,"type":15},"Python","python","2026-06-19T09:04:22.640384",{"slug":1610,"name":1610,"fn":1611,"description":1612,"org":1613,"tags":1614,"stars":28,"repoUrl":29,"updatedAt":1622},"agent-observability-replay-trace","iterate on LLM traces with local code","Use when a developer wants to iterate on ONE specific Agent Observability \u002F LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as \u002Fagent-observability-replay-trace \u003Ctrace-id> [changes to test]. Signals: \"replay this trace\"; \"iterate on a trace\"; \"this trace's output is wrong, fix it and re-run\"; \"re-run trace \u003Cid> with \u003Cchange>\"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace \u002F LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser \"Replay\" button (that's agent-observability-replay-experiment), building an experiment from a dataset\u002FCSV, writing evaluators, root-causing failed traces, or RUM\u002FHTTP session replay.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1615,1616,1617,1618,1619],{"name":23,"slug":24,"type":15},{"name":1542,"slug":1543,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"name":1620,"slug":1621,"type":15},"Tracing","tracing","2026-07-31T05:52:08.594182",34,{"items":1625,"total":1751},[1626,1633,1641,1648,1656,1664,1672,1680,1692,1704,1716,1736],{"slug":1520,"name":1520,"fn":1521,"description":1522,"org":1627,"tags":1628,"stars":28,"repoUrl":29,"updatedAt":1533},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1629,1630,1631,1632],{"name":23,"slug":24,"type":15},{"name":1527,"slug":1528,"type":15},{"name":1530,"slug":1531,"type":15},{"name":13,"slug":14,"type":15},{"slug":1535,"name":1535,"fn":1536,"description":1537,"org":1634,"tags":1635,"stars":28,"repoUrl":29,"updatedAt":1551},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1636,1637,1638,1639,1640],{"name":23,"slug":24,"type":15},{"name":1542,"slug":1543,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"slug":1553,"name":1553,"fn":1554,"description":1555,"org":1642,"tags":1643,"stars":28,"repoUrl":29,"updatedAt":1562},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1644,1645,1646,1647],{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"slug":1564,"name":1564,"fn":1565,"description":1566,"org":1649,"tags":1650,"stars":28,"repoUrl":29,"updatedAt":1578},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1651,1652,1653,1654,1655],{"name":1570,"slug":1571,"type":15},{"name":1573,"slug":1574,"type":15},{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":13,"slug":14,"type":15},{"slug":1580,"name":1580,"fn":1581,"description":1582,"org":1657,"tags":1658,"stars":28,"repoUrl":29,"updatedAt":1592},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1659,1660,1661,1662,1663],{"name":1586,"slug":1587,"type":15},{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"slug":1594,"name":1594,"fn":1595,"description":1596,"org":1665,"tags":1666,"stars":28,"repoUrl":29,"updatedAt":1608},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1667,1668,1669,1670,1671],{"name":23,"slug":24,"type":15},{"name":1601,"slug":1602,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"name":1606,"slug":1607,"type":15},{"slug":1610,"name":1610,"fn":1611,"description":1612,"org":1673,"tags":1674,"stars":28,"repoUrl":29,"updatedAt":1622},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1675,1676,1677,1678,1679],{"name":23,"slug":24,"type":15},{"name":1542,"slug":1543,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},{"name":1620,"slug":1621,"type":15},{"slug":1681,"name":1681,"fn":1682,"description":1683,"org":1684,"tags":1685,"stars":28,"repoUrl":29,"updatedAt":1691},"agent-observability-session-classify","evaluate user intent satisfaction in sessions","Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions\u002Ftraces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1686,1687,1688,1689,1690],{"name":1586,"slug":1587,"type":15},{"name":23,"slug":24,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:26.341497",{"slug":1693,"name":1693,"fn":1694,"description":1695,"org":1696,"tags":1697,"stars":28,"repoUrl":29,"updatedAt":1703},"agent-observability-trace-rca","diagnose LLM application failures from traces","Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says \"what's wrong with my app\", \"why is my eval failing\", \"analyze errors\", \"root cause analysis\", \"diagnose failures\", or wants to understand production failure patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1698,1699,1700,1701,1702],{"name":23,"slug":24,"type":15},{"name":1542,"slug":1543,"type":15},{"name":1545,"slug":1546,"type":15},{"name":1548,"slug":1549,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:24.316244",{"slug":1705,"name":1705,"fn":1706,"description":1707,"org":1708,"tags":1709,"stars":28,"repoUrl":29,"updatedAt":1715},"agent-skills","use Datadog observability skills","Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1710,1711,1714],{"name":23,"slug":24,"type":15},{"name":1712,"slug":1713,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},"2026-04-06T18:08:33.337476",{"slug":1717,"name":1717,"fn":1718,"description":1719,"org":1720,"tags":1721,"stars":28,"repoUrl":29,"updatedAt":1735},"datadog-app","build and manage Datadog applications","Guides developers building Datadog Apps with TypeScript, React, the @datadog\u002Fapps scaffolder, and @datadog\u002Fvite-plugin. Use when a user wants to scaffold, run, debug, upgrade, build, upload, publish, upload without publishing (draft upload), add an upload-no-publish script, set up CI\u002FCD, use OAuth or API\u002Fapplication key auth, trigger\u002Fpoll Workflow Automation, choose DDSQL or Action Catalog for backend data access, or query app datastores with DDSQL, including backend function troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1722,1723,1726,1729,1732],{"name":23,"slug":24,"type":15},{"name":1724,"slug":1725,"type":15},"Frontend","frontend",{"name":1727,"slug":1728,"type":15},"React","react",{"name":1730,"slug":1731,"type":15},"TypeScript","typescript",{"name":1733,"slug":1734,"type":15},"Vite","vite","2026-06-18T08:01:32.562331",{"slug":1737,"name":1737,"fn":1738,"description":1739,"org":1740,"tags":1741,"stars":28,"repoUrl":29,"updatedAt":1750},"dd-apm","query Datadog APM traces","APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1742,1743,1746,1747],{"name":23,"slug":24,"type":15},{"name":1744,"slug":1745,"type":15},"Distributed Tracing","distributed-tracing",{"name":13,"slug":14,"type":15},{"name":1748,"slug":1749,"type":15},"Performance","performance","2026-04-06T18:08:34.575282",35]