[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-datadog-labs-agent-observability-experiment-bootstrap":3,"mdc-jzmx0z-key":39,"related-org-datadog-labs-agent-observability-experiment-bootstrap":789,"related-repo-datadog-labs-agent-observability-experiment-bootstrap":952},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":28,"repoUrl":29,"updatedAt":30,"license":31,"forks":32,"topics":33,"repo":34,"sourceUrl":37,"mdContent":38},"agent-observability-experiment-bootstrap","bootstrap LLM observability experiments","Bootstrap a reproducible LLM Observability experiment through the Python ddtrace SDK or the Node dd-trace SDK. Use for experiment, dataset, evaluator, benchmark, regression, or LLM-as-a-judge scaffolding. The legacy Python invocation remains supported.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"datadog-labs","Datadog Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatadog-labs.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":20,"slug":21,"type":15},"Datadog","datadog",{"name":23,"slug":24,"type":15},"Evals","evals",{"name":26,"slug":27,"type":15},"Experiments","experiments",145,"https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills","2026-08-22T03:56:54.562327",null,19,[],{"repoUrl":29,"stars":28,"forks":32,"topics":35,"description":36},[],"Public repository for Datadog Agent Skills","https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fagent-observability\u002Fagent-observability-experiment-bootstrap","---\nname: agent-observability-experiment-bootstrap\ndescription: Bootstrap a reproducible LLM Observability experiment through the Python ddtrace SDK or the Node dd-trace SDK. Use for experiment, dataset, evaluator, benchmark, regression, or LLM-as-a-judge scaffolding. The legacy Python invocation remains supported.\n---\n\n# LLM Observability Experiment Bootstrap\n\nGenerate one reproducible experiment artifact. The artifact evaluates a task over a versioned dataset, records outputs and evaluator metrics, carries configuration and provenance, and prints a result link or identifiers when possible.\n\nThis skill is adapter-independent. Each adapter owns a language-specific directory under `references\u002F`; load only the selected adapter contract. The directories are intentionally symmetric even when one adapter currently has fewer supporting references.\n\n## Invocation and compatibility\n\nThe installed directory and legacy invocation remain valid:\n\n```text\n\u002Fagent-observability-experiment-bootstrap [--purpose TEXT] [--format py|ipynb|mjs]\n  [--dataset PATH | --dataset-name NAME] [--dataset-version N]\n  [--project-name NAME] [--evaluator-style function|class|remote]\n  [--jobs N] [--output PATH] [--task-source module:function]\n  [--placeholder-task] [--app-root PATH] [--env-file PATH]\n```\n\nGeneral options:\n\n```text\n--adapter python|node             # default: python\n--format py|ipynb|mjs             # Python: py\u002Fipynb; Node: mjs\n--site SITE                      # otherwise DD_SITE or datadoghq.com\n```\n\nDo not prompt for optional defaults. Resolve a non-empty purpose from `--purpose`, the request, or a focused question. Keep the purpose as reasoning context, not a fixed taxonomy.\n\n## Mandatory context loading\n\nLoad context in this order:\n\n1. Parse the adapter.\n2. Read exactly one adapter reference:\n   - Python SDK → `references\u002Fpython\u002Fpython.md`\n   - Node SDK → `references\u002Fnode\u002Fnodejs.md`\n3. For Python task generation, read only the selected provider reference under `references\u002Fpython\u002Fproviders\u002F`.\n4. For Python task generation, read only the selected evaluator reference under `references\u002Fpython\u002Fevaluator-styles\u002F`.\n\nDo not load all provider, evaluator, Python, and Node references “for completeness.” The selected reference is the source of truth for syntax and API behavior.\n\n## Adapter selection\n\nUse Python when the application or requested artifact is Python, or when no adapter is specified. Use Node when the application is JavaScript\u002FTypeScript and the local `dd-trace` package exposes `tracer.llmobs.experiments`.\n\nNever mix the Python and Node SDKs in one generated artifact. Do not use private SDK modules or invent a missing symbol. If local source and an installed package disagree, report the discrepancy and generate against the selected version.\n\n## Shared experiment model\n\nEvery adapter must represent the following concepts:\n\n1. **Project** — resolve an explicit project name, configured service metadata, or a clearly documented generated fallback. Never silently use an unrelated project.\n2. **Dataset** — records with input, optional expected output, optional metadata, and tags. Pin a remote dataset version when supplied.\n3. **Task** — a deterministic adapter from record input to the application under test. Keep evaluation logic outside the task.\n4. **Evaluators** — named row-level or summary-level metrics. Use deterministic checks for contracts and judges only where semantic evaluation is needed.\n5. **Run state** — preserve task errors, evaluator errors, completion state, result rows, and partial failures separately.\n6. **Provenance** — include purpose, adapter, skill name\u002Fversion, project, dataset identity\u002Fversion, task source, evaluator labels\u002Frubrics, model\u002Fconfiguration, Git revision, and generation timestamp.\n\n`expected_output` is optional and must not be synthesized from an observed production output without explicit validation. Distinguish a missing value from an intentionally empty object. Dataset tags must use the backend’s validated `key:value` form where the selected reference requires it.\n\n## Generation workflow\n\n### 1. Resolve purpose and project\n\nDerive the purpose and project without guessing across product boundaries. A project is not automatically the same as an `ml_app`, service, dataset, or repository name. Record how each value was resolved.\n\n### 2. Resolve the dataset\n\nSupport:\n\n- inline records;\n- local JSON or CSV;\n- a named remote dataset and optional version; and\n- an explicitly approved trace\u002Fannotation export.\n\nFor local JSON, require a top-level array, validate the selected adapter’s record shape, scrub obvious PII and credential-like values, and report affected record indices. Do not invent canonical or remote record IDs.\n\nFor CSV, preserve the runtime path and document the dependency. Use the Python CSV column contract from `references\u002Fpython\u002Fpython.md`; Node generation must not pretend that a Python-only CSV helper exists.\n\n### 3. Resolve the task\n\nUse `--task-source` when provided. Otherwise use the selected language’s bounded application discovery rules:\n\n- Python: inspect the resolved app root and rank real callable candidates.\n- Node: prefer an explicit import\u002Fmodule function and emit a clearly marked placeholder when absent.\n\nNever claim that an invented import is wired. Preserve side-effect warnings for network, database, filesystem, environment, or tool calls.\n\n### 4. Select evaluators\n\nSelect two or three evaluators based on purpose and available signals. Keep labels unique and stable.\n\n- Accuracy: exact\u002Fnear match plus a richer rule or judge when needed.\n- Tool use: inspect structured tool calls; state the limitation when the task does not expose them.\n- Structured output: parse and validate the schema.\n- Retrieval: evaluate groundedness only when retrieved context is available.\n- Regression: prefer deterministic checks and explicit thresholds.\n- Exploration: include diagnostics or taxonomy metrics, not only a pass\u002Ffail score.\n\nEvaluator failures must not become passing values. Summary evaluators must remain distinct from row evaluators.\n\n### 5. Emit the artifact\n\nUse the selected adapter reference for the exact generated code. Include:\n\n- purpose and project resolution;\n- dataset source and version;\n- real task source or a prominent placeholder warning;\n- evaluator labels and rubrics;\n- configuration and provenance;\n- credential instructions without literal secrets; and\n- a result URL\u002FID placeholder and next steps.\n\nPreserve the historical Python section ordering and evaluator\u002Fprovider reference behavior when using the Python adapter.\n\n### 6. Validate locally\n\nBefore presenting the artifact:\n\n- Python `.py`: `python -m py_compile \u003Cpath>`.\n- Python `.ipynb`: parse JSON and require code\u002Fmarkdown cells.\n- Node `.mjs`: `node --check \u003Cpath>`.\n\nFor every adapter, check for private imports, literal credentials, malformed tags, missing provenance, mismatched dataset versions, fabricated IDs, and task\u002Fevaluator errors that were collapsed into false or pass.\n\n### 7. Report completion\n\nUse this compact structure:\n\n```text\nGenerated LLM Observability experiment: \u003Cadapter>\u002F\u003Cformat>\nPath: \u003Cpath>\nPurpose: \"\u003Cpurpose>\"\nProject: \u003Cproject>\nDataset: \u003Clocal path | name>, version=\u003Cversion or latest>\nTask: \u003Cwired source | placeholder>\nEvaluators: \u003Clabels>\nProvenance: generated_by=claude-code, adapter=\u003Cadapter>, skill=agent-observability-experiment-bootstrap\nValidation: \u003Ccommands and pass\u002Ffail>\nResult link: \u003CURL or pending until run>\n\nNext steps:\n1. Verify the task source and evaluator semantics.\n2. Set the credentials required by the selected SDK.\n3. Install the selected SDK and run the generated artifact.\n4. Review per-row errors before treating metrics as a successful run.\n```\n\n## Safety and uncertainty\n\n- Do not modify application source code unless explicitly asked.\n- Do not write credentials into generated files or artifacts.\n- Do not publish prompts, outputs, traces, datasets, or evaluations without explicit user approval.\n- Do not use production data as ground truth without labeling and validation.\n- Do not retry non-idempotent writes automatically unless the selected SDK explicitly supports it.\n- On partial publication, preserve IDs and failed rows and provide a reconciliation path.\n\n## Reference maintenance\n\nEach adapter reference must identify the public source links and branch used to verify it. Re-check the reference when the SDK version changes. The Python reference uses the public `dd-trace-py` `main` branch; the Node reference uses the public `dd-trace-js` `master` branch.\n\nKeep shared workflow guidance here and language-specific syntax in the references. If a detail is only true for one SDK, do not duplicate it in this file.\n\n## Existing references\n\n- `references\u002Fpython\u002F` — Python `ddtrace.llmobs` API, providers, evaluator styles, environment template, and legacy compatibility.\n- `references\u002Fnode\u002F` — Node `tracer.llmobs.experiments` API and future Node-specific references.\n\n\nDo not modify `dd-trace-py` or `dd-trace-js` while updating this skill.\n",{"data":40,"body":41},{"name":4,"description":6},{"type":42,"children":43},"root",[44,53,59,73,80,85,97,102,111,124,130,135,201,206,212,232,237,243,248,312,331,337,344,357,363,368,391,396,408,414,427,440,445,451,456,489,494,500,505,543,548,554,559,613,618,624,629,638,644,677,683,719,724,730,770],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"llm-observability-experiment-bootstrap",[50],{"type":51,"value":52},"text","LLM Observability Experiment Bootstrap",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57],{"type":51,"value":58},"Generate one reproducible experiment artifact. The artifact evaluates a task over a versioned dataset, records outputs and evaluator metrics, carries configuration and provenance, and prints a result link or identifiers when possible.",{"type":45,"tag":54,"props":60,"children":61},{},[62,64,71],{"type":51,"value":63},"This skill is adapter-independent. Each adapter owns a language-specific directory under ",{"type":45,"tag":65,"props":66,"children":68},"code",{"className":67},[],[69],{"type":51,"value":70},"references\u002F",{"type":51,"value":72},"; load only the selected adapter contract. The directories are intentionally symmetric even when one adapter currently has fewer supporting references.",{"type":45,"tag":74,"props":75,"children":77},"h2",{"id":76},"invocation-and-compatibility",[78],{"type":51,"value":79},"Invocation and compatibility",{"type":45,"tag":54,"props":81,"children":82},{},[83],{"type":51,"value":84},"The installed directory and legacy invocation remain valid:",{"type":45,"tag":86,"props":87,"children":92},"pre",{"className":88,"code":90,"language":51,"meta":91},[89],"language-text","\u002Fagent-observability-experiment-bootstrap [--purpose TEXT] [--format py|ipynb|mjs]\n  [--dataset PATH | --dataset-name NAME] [--dataset-version N]\n  [--project-name NAME] [--evaluator-style function|class|remote]\n  [--jobs N] [--output PATH] [--task-source module:function]\n  [--placeholder-task] [--app-root PATH] [--env-file PATH]\n","",[93],{"type":45,"tag":65,"props":94,"children":95},{"__ignoreMap":91},[96],{"type":51,"value":90},{"type":45,"tag":54,"props":98,"children":99},{},[100],{"type":51,"value":101},"General options:",{"type":45,"tag":86,"props":103,"children":106},{"className":104,"code":105,"language":51,"meta":91},[89],"--adapter python|node             # default: python\n--format py|ipynb|mjs             # Python: py\u002Fipynb; Node: mjs\n--site SITE                      # otherwise DD_SITE or datadoghq.com\n",[107],{"type":45,"tag":65,"props":108,"children":109},{"__ignoreMap":91},[110],{"type":51,"value":105},{"type":45,"tag":54,"props":112,"children":113},{},[114,116,122],{"type":51,"value":115},"Do not prompt for optional defaults. Resolve a non-empty purpose from ",{"type":45,"tag":65,"props":117,"children":119},{"className":118},[],[120],{"type":51,"value":121},"--purpose",{"type":51,"value":123},", the request, or a focused question. Keep the purpose as reasoning context, not a fixed taxonomy.",{"type":45,"tag":74,"props":125,"children":127},{"id":126},"mandatory-context-loading",[128],{"type":51,"value":129},"Mandatory context loading",{"type":45,"tag":54,"props":131,"children":132},{},[133],{"type":51,"value":134},"Load context in this order:",{"type":45,"tag":136,"props":137,"children":138},"ol",{},[139,145,176,189],{"type":45,"tag":140,"props":141,"children":142},"li",{},[143],{"type":51,"value":144},"Parse the adapter.",{"type":45,"tag":140,"props":146,"children":147},{},[148,150],{"type":51,"value":149},"Read exactly one adapter reference:\n",{"type":45,"tag":151,"props":152,"children":153},"ul",{},[154,165],{"type":45,"tag":140,"props":155,"children":156},{},[157,159],{"type":51,"value":158},"Python SDK → ",{"type":45,"tag":65,"props":160,"children":162},{"className":161},[],[163],{"type":51,"value":164},"references\u002Fpython\u002Fpython.md",{"type":45,"tag":140,"props":166,"children":167},{},[168,170],{"type":51,"value":169},"Node SDK → ",{"type":45,"tag":65,"props":171,"children":173},{"className":172},[],[174],{"type":51,"value":175},"references\u002Fnode\u002Fnodejs.md",{"type":45,"tag":140,"props":177,"children":178},{},[179,181,187],{"type":51,"value":180},"For Python task generation, read only the selected provider reference under ",{"type":45,"tag":65,"props":182,"children":184},{"className":183},[],[185],{"type":51,"value":186},"references\u002Fpython\u002Fproviders\u002F",{"type":51,"value":188},".",{"type":45,"tag":140,"props":190,"children":191},{},[192,194,200],{"type":51,"value":193},"For Python task generation, read only the selected evaluator reference under ",{"type":45,"tag":65,"props":195,"children":197},{"className":196},[],[198],{"type":51,"value":199},"references\u002Fpython\u002Fevaluator-styles\u002F",{"type":51,"value":188},{"type":45,"tag":54,"props":202,"children":203},{},[204],{"type":51,"value":205},"Do not load all provider, evaluator, Python, and Node references “for completeness.” The selected reference is the source of truth for syntax and API behavior.",{"type":45,"tag":74,"props":207,"children":209},{"id":208},"adapter-selection",[210],{"type":51,"value":211},"Adapter selection",{"type":45,"tag":54,"props":213,"children":214},{},[215,217,223,225,231],{"type":51,"value":216},"Use Python when the application or requested artifact is Python, or when no adapter is specified. Use Node when the application is JavaScript\u002FTypeScript and the local ",{"type":45,"tag":65,"props":218,"children":220},{"className":219},[],[221],{"type":51,"value":222},"dd-trace",{"type":51,"value":224}," package exposes ",{"type":45,"tag":65,"props":226,"children":228},{"className":227},[],[229],{"type":51,"value":230},"tracer.llmobs.experiments",{"type":51,"value":188},{"type":45,"tag":54,"props":233,"children":234},{},[235],{"type":51,"value":236},"Never mix the Python and Node SDKs in one generated artifact. Do not use private SDK modules or invent a missing symbol. If local source and an installed package disagree, report the discrepancy and generate against the selected version.",{"type":45,"tag":74,"props":238,"children":240},{"id":239},"shared-experiment-model",[241],{"type":51,"value":242},"Shared experiment model",{"type":45,"tag":54,"props":244,"children":245},{},[246],{"type":51,"value":247},"Every adapter must represent the following concepts:",{"type":45,"tag":136,"props":249,"children":250},{},[251,262,272,282,292,302],{"type":45,"tag":140,"props":252,"children":253},{},[254,260],{"type":45,"tag":255,"props":256,"children":257},"strong",{},[258],{"type":51,"value":259},"Project",{"type":51,"value":261}," — resolve an explicit project name, configured service metadata, or a clearly documented generated fallback. Never silently use an unrelated project.",{"type":45,"tag":140,"props":263,"children":264},{},[265,270],{"type":45,"tag":255,"props":266,"children":267},{},[268],{"type":51,"value":269},"Dataset",{"type":51,"value":271}," — records with input, optional expected output, optional metadata, and tags. Pin a remote dataset version when supplied.",{"type":45,"tag":140,"props":273,"children":274},{},[275,280],{"type":45,"tag":255,"props":276,"children":277},{},[278],{"type":51,"value":279},"Task",{"type":51,"value":281}," — a deterministic adapter from record input to the application under test. Keep evaluation logic outside the task.",{"type":45,"tag":140,"props":283,"children":284},{},[285,290],{"type":45,"tag":255,"props":286,"children":287},{},[288],{"type":51,"value":289},"Evaluators",{"type":51,"value":291}," — named row-level or summary-level metrics. Use deterministic checks for contracts and judges only where semantic evaluation is needed.",{"type":45,"tag":140,"props":293,"children":294},{},[295,300],{"type":45,"tag":255,"props":296,"children":297},{},[298],{"type":51,"value":299},"Run state",{"type":51,"value":301}," — preserve task errors, evaluator errors, completion state, result rows, and partial failures separately.",{"type":45,"tag":140,"props":303,"children":304},{},[305,310],{"type":45,"tag":255,"props":306,"children":307},{},[308],{"type":51,"value":309},"Provenance",{"type":51,"value":311}," — include purpose, adapter, skill name\u002Fversion, project, dataset identity\u002Fversion, task source, evaluator labels\u002Frubrics, model\u002Fconfiguration, Git revision, and generation timestamp.",{"type":45,"tag":54,"props":313,"children":314},{},[315,321,323,329],{"type":45,"tag":65,"props":316,"children":318},{"className":317},[],[319],{"type":51,"value":320},"expected_output",{"type":51,"value":322}," is optional and must not be synthesized from an observed production output without explicit validation. Distinguish a missing value from an intentionally empty object. Dataset tags must use the backend’s validated ",{"type":45,"tag":65,"props":324,"children":326},{"className":325},[],[327],{"type":51,"value":328},"key:value",{"type":51,"value":330}," form where the selected reference requires it.",{"type":45,"tag":74,"props":332,"children":334},{"id":333},"generation-workflow",[335],{"type":51,"value":336},"Generation workflow",{"type":45,"tag":338,"props":339,"children":341},"h3",{"id":340},"_1-resolve-purpose-and-project",[342],{"type":51,"value":343},"1. Resolve purpose and project",{"type":45,"tag":54,"props":345,"children":346},{},[347,349,355],{"type":51,"value":348},"Derive the purpose and project without guessing across product boundaries. A project is not automatically the same as an ",{"type":45,"tag":65,"props":350,"children":352},{"className":351},[],[353],{"type":51,"value":354},"ml_app",{"type":51,"value":356},", service, dataset, or repository name. Record how each value was resolved.",{"type":45,"tag":338,"props":358,"children":360},{"id":359},"_2-resolve-the-dataset",[361],{"type":51,"value":362},"2. Resolve the dataset",{"type":45,"tag":54,"props":364,"children":365},{},[366],{"type":51,"value":367},"Support:",{"type":45,"tag":151,"props":369,"children":370},{},[371,376,381,386],{"type":45,"tag":140,"props":372,"children":373},{},[374],{"type":51,"value":375},"inline records;",{"type":45,"tag":140,"props":377,"children":378},{},[379],{"type":51,"value":380},"local JSON or CSV;",{"type":45,"tag":140,"props":382,"children":383},{},[384],{"type":51,"value":385},"a named remote dataset and optional version; and",{"type":45,"tag":140,"props":387,"children":388},{},[389],{"type":51,"value":390},"an explicitly approved trace\u002Fannotation export.",{"type":45,"tag":54,"props":392,"children":393},{},[394],{"type":51,"value":395},"For local JSON, require a top-level array, validate the selected adapter’s record shape, scrub obvious PII and credential-like values, and report affected record indices. Do not invent canonical or remote record IDs.",{"type":45,"tag":54,"props":397,"children":398},{},[399,401,406],{"type":51,"value":400},"For CSV, preserve the runtime path and document the dependency. Use the Python CSV column contract from ",{"type":45,"tag":65,"props":402,"children":404},{"className":403},[],[405],{"type":51,"value":164},{"type":51,"value":407},"; Node generation must not pretend that a Python-only CSV helper exists.",{"type":45,"tag":338,"props":409,"children":411},{"id":410},"_3-resolve-the-task",[412],{"type":51,"value":413},"3. Resolve the task",{"type":45,"tag":54,"props":415,"children":416},{},[417,419,425],{"type":51,"value":418},"Use ",{"type":45,"tag":65,"props":420,"children":422},{"className":421},[],[423],{"type":51,"value":424},"--task-source",{"type":51,"value":426}," when provided. Otherwise use the selected language’s bounded application discovery rules:",{"type":45,"tag":151,"props":428,"children":429},{},[430,435],{"type":45,"tag":140,"props":431,"children":432},{},[433],{"type":51,"value":434},"Python: inspect the resolved app root and rank real callable candidates.",{"type":45,"tag":140,"props":436,"children":437},{},[438],{"type":51,"value":439},"Node: prefer an explicit import\u002Fmodule function and emit a clearly marked placeholder when absent.",{"type":45,"tag":54,"props":441,"children":442},{},[443],{"type":51,"value":444},"Never claim that an invented import is wired. Preserve side-effect warnings for network, database, filesystem, environment, or tool calls.",{"type":45,"tag":338,"props":446,"children":448},{"id":447},"_4-select-evaluators",[449],{"type":51,"value":450},"4. Select evaluators",{"type":45,"tag":54,"props":452,"children":453},{},[454],{"type":51,"value":455},"Select two or three evaluators based on purpose and available signals. Keep labels unique and stable.",{"type":45,"tag":151,"props":457,"children":458},{},[459,464,469,474,479,484],{"type":45,"tag":140,"props":460,"children":461},{},[462],{"type":51,"value":463},"Accuracy: exact\u002Fnear match plus a richer rule or judge when needed.",{"type":45,"tag":140,"props":465,"children":466},{},[467],{"type":51,"value":468},"Tool use: inspect structured tool calls; state the limitation when the task does not expose them.",{"type":45,"tag":140,"props":470,"children":471},{},[472],{"type":51,"value":473},"Structured output: parse and validate the schema.",{"type":45,"tag":140,"props":475,"children":476},{},[477],{"type":51,"value":478},"Retrieval: evaluate groundedness only when retrieved context is available.",{"type":45,"tag":140,"props":480,"children":481},{},[482],{"type":51,"value":483},"Regression: prefer deterministic checks and explicit thresholds.",{"type":45,"tag":140,"props":485,"children":486},{},[487],{"type":51,"value":488},"Exploration: include diagnostics or taxonomy metrics, not only a pass\u002Ffail score.",{"type":45,"tag":54,"props":490,"children":491},{},[492],{"type":51,"value":493},"Evaluator failures must not become passing values. Summary evaluators must remain distinct from row evaluators.",{"type":45,"tag":338,"props":495,"children":497},{"id":496},"_5-emit-the-artifact",[498],{"type":51,"value":499},"5. Emit the artifact",{"type":45,"tag":54,"props":501,"children":502},{},[503],{"type":51,"value":504},"Use the selected adapter reference for the exact generated code. Include:",{"type":45,"tag":151,"props":506,"children":507},{},[508,513,518,523,528,533,538],{"type":45,"tag":140,"props":509,"children":510},{},[511],{"type":51,"value":512},"purpose and project resolution;",{"type":45,"tag":140,"props":514,"children":515},{},[516],{"type":51,"value":517},"dataset source and version;",{"type":45,"tag":140,"props":519,"children":520},{},[521],{"type":51,"value":522},"real task source or a prominent placeholder warning;",{"type":45,"tag":140,"props":524,"children":525},{},[526],{"type":51,"value":527},"evaluator labels and rubrics;",{"type":45,"tag":140,"props":529,"children":530},{},[531],{"type":51,"value":532},"configuration and provenance;",{"type":45,"tag":140,"props":534,"children":535},{},[536],{"type":51,"value":537},"credential instructions without literal secrets; and",{"type":45,"tag":140,"props":539,"children":540},{},[541],{"type":51,"value":542},"a result URL\u002FID placeholder and next steps.",{"type":45,"tag":54,"props":544,"children":545},{},[546],{"type":51,"value":547},"Preserve the historical Python section ordering and evaluator\u002Fprovider reference behavior when using the Python adapter.",{"type":45,"tag":338,"props":549,"children":551},{"id":550},"_6-validate-locally",[552],{"type":51,"value":553},"6. Validate locally",{"type":45,"tag":54,"props":555,"children":556},{},[557],{"type":51,"value":558},"Before presenting the artifact:",{"type":45,"tag":151,"props":560,"children":561},{},[562,582,594],{"type":45,"tag":140,"props":563,"children":564},{},[565,567,573,575,581],{"type":51,"value":566},"Python ",{"type":45,"tag":65,"props":568,"children":570},{"className":569},[],[571],{"type":51,"value":572},".py",{"type":51,"value":574},": ",{"type":45,"tag":65,"props":576,"children":578},{"className":577},[],[579],{"type":51,"value":580},"python -m py_compile \u003Cpath>",{"type":51,"value":188},{"type":45,"tag":140,"props":583,"children":584},{},[585,586,592],{"type":51,"value":566},{"type":45,"tag":65,"props":587,"children":589},{"className":588},[],[590],{"type":51,"value":591},".ipynb",{"type":51,"value":593},": parse JSON and require code\u002Fmarkdown cells.",{"type":45,"tag":140,"props":595,"children":596},{},[597,599,605,606,612],{"type":51,"value":598},"Node ",{"type":45,"tag":65,"props":600,"children":602},{"className":601},[],[603],{"type":51,"value":604},".mjs",{"type":51,"value":574},{"type":45,"tag":65,"props":607,"children":609},{"className":608},[],[610],{"type":51,"value":611},"node --check \u003Cpath>",{"type":51,"value":188},{"type":45,"tag":54,"props":614,"children":615},{},[616],{"type":51,"value":617},"For every adapter, check for private imports, literal credentials, malformed tags, missing provenance, mismatched dataset versions, fabricated IDs, and task\u002Fevaluator errors that were collapsed into false or pass.",{"type":45,"tag":338,"props":619,"children":621},{"id":620},"_7-report-completion",[622],{"type":51,"value":623},"7. Report completion",{"type":45,"tag":54,"props":625,"children":626},{},[627],{"type":51,"value":628},"Use this compact structure:",{"type":45,"tag":86,"props":630,"children":633},{"className":631,"code":632,"language":51,"meta":91},[89],"Generated LLM Observability experiment: \u003Cadapter>\u002F\u003Cformat>\nPath: \u003Cpath>\nPurpose: \"\u003Cpurpose>\"\nProject: \u003Cproject>\nDataset: \u003Clocal path | name>, version=\u003Cversion or latest>\nTask: \u003Cwired source | placeholder>\nEvaluators: \u003Clabels>\nProvenance: generated_by=claude-code, adapter=\u003Cadapter>, skill=agent-observability-experiment-bootstrap\nValidation: \u003Ccommands and pass\u002Ffail>\nResult link: \u003CURL or pending until run>\n\nNext steps:\n1. Verify the task source and evaluator semantics.\n2. Set the credentials required by the selected SDK.\n3. Install the selected SDK and run the generated artifact.\n4. Review per-row errors before treating metrics as a successful run.\n",[634],{"type":45,"tag":65,"props":635,"children":636},{"__ignoreMap":91},[637],{"type":51,"value":632},{"type":45,"tag":74,"props":639,"children":641},{"id":640},"safety-and-uncertainty",[642],{"type":51,"value":643},"Safety and uncertainty",{"type":45,"tag":151,"props":645,"children":646},{},[647,652,657,662,667,672],{"type":45,"tag":140,"props":648,"children":649},{},[650],{"type":51,"value":651},"Do not modify application source code unless explicitly asked.",{"type":45,"tag":140,"props":653,"children":654},{},[655],{"type":51,"value":656},"Do not write credentials into generated files or artifacts.",{"type":45,"tag":140,"props":658,"children":659},{},[660],{"type":51,"value":661},"Do not publish prompts, outputs, traces, datasets, or evaluations without explicit user approval.",{"type":45,"tag":140,"props":663,"children":664},{},[665],{"type":51,"value":666},"Do not use production data as ground truth without labeling and validation.",{"type":45,"tag":140,"props":668,"children":669},{},[670],{"type":51,"value":671},"Do not retry non-idempotent writes automatically unless the selected SDK explicitly supports it.",{"type":45,"tag":140,"props":673,"children":674},{},[675],{"type":51,"value":676},"On partial publication, preserve IDs and failed rows and provide a reconciliation path.",{"type":45,"tag":74,"props":678,"children":680},{"id":679},"reference-maintenance",[681],{"type":51,"value":682},"Reference maintenance",{"type":45,"tag":54,"props":684,"children":685},{},[686,688,694,696,702,704,710,711,717],{"type":51,"value":687},"Each adapter reference must identify the public source links and branch used to verify it. Re-check the reference when the SDK version changes. The Python reference uses the public ",{"type":45,"tag":65,"props":689,"children":691},{"className":690},[],[692],{"type":51,"value":693},"dd-trace-py",{"type":51,"value":695}," ",{"type":45,"tag":65,"props":697,"children":699},{"className":698},[],[700],{"type":51,"value":701},"main",{"type":51,"value":703}," branch; the Node reference uses the public ",{"type":45,"tag":65,"props":705,"children":707},{"className":706},[],[708],{"type":51,"value":709},"dd-trace-js",{"type":51,"value":695},{"type":45,"tag":65,"props":712,"children":714},{"className":713},[],[715],{"type":51,"value":716},"master",{"type":51,"value":718}," branch.",{"type":45,"tag":54,"props":720,"children":721},{},[722],{"type":51,"value":723},"Keep shared workflow guidance here and language-specific syntax in the references. If a detail is only true for one SDK, do not duplicate it in this file.",{"type":45,"tag":74,"props":725,"children":727},{"id":726},"existing-references",[728],{"type":51,"value":729},"Existing references",{"type":45,"tag":151,"props":731,"children":732},{},[733,752],{"type":45,"tag":140,"props":734,"children":735},{},[736,742,744,750],{"type":45,"tag":65,"props":737,"children":739},{"className":738},[],[740],{"type":51,"value":741},"references\u002Fpython\u002F",{"type":51,"value":743}," — Python ",{"type":45,"tag":65,"props":745,"children":747},{"className":746},[],[748],{"type":51,"value":749},"ddtrace.llmobs",{"type":51,"value":751}," API, providers, evaluator styles, environment template, and legacy compatibility.",{"type":45,"tag":140,"props":753,"children":754},{},[755,761,763,768],{"type":45,"tag":65,"props":756,"children":758},{"className":757},[],[759],{"type":51,"value":760},"references\u002Fnode\u002F",{"type":51,"value":762}," — Node ",{"type":45,"tag":65,"props":764,"children":766},{"className":765},[],[767],{"type":51,"value":230},{"type":51,"value":769}," API and future Node-specific references.",{"type":45,"tag":54,"props":771,"children":772},{},[773,775,780,782,787],{"type":51,"value":774},"Do not modify ",{"type":45,"tag":65,"props":776,"children":778},{"className":777},[],[779],{"type":51,"value":693},{"type":51,"value":781}," or ",{"type":45,"tag":65,"props":783,"children":785},{"className":784},[],[786],{"type":51,"value":709},{"type":51,"value":788}," while updating this skill.",{"items":790,"total":951},[791,806,820,831,847,861,869,883,895,907,919,939],{"slug":792,"name":792,"fn":793,"description":794,"org":795,"tags":796,"stars":28,"repoUrl":29,"updatedAt":805},"agent-install","install Datadog Agent on Kubernetes","Install the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[797,798,801,804],{"name":20,"slug":21,"type":15},{"name":799,"slug":800,"type":15},"Deployment","deployment",{"name":802,"slug":803,"type":15},"Kubernetes","kubernetes",{"name":13,"slug":14,"type":15},"2026-04-15T04:57:27.489805",{"slug":807,"name":807,"fn":808,"description":809,"org":810,"tags":811,"stars":28,"repoUrl":29,"updatedAt":819},"agent-observability-auto-experiment","run iterative code improvements using Datadog data","Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says \"run an auto experiment\", \"hill-climb this code\", \"iteratively improve X and measure the delta\", \"optimize this prompt\u002Ffile against my traces\", \"auto-optimize against LLM-Obs\", or wants the local equivalent of the auto_experiments worker. Works from a local dataset file, an ml_app, a dataset_id, or a list of trace_ids.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[812,813,816,817,818],{"name":20,"slug":21,"type":15},{"name":814,"slug":815,"type":15},"Debugging","debugging",{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-08-19T03:27:28.037126",{"slug":821,"name":821,"fn":822,"description":823,"org":824,"tags":825,"stars":28,"repoUrl":29,"updatedAt":830},"agent-observability-eval-bootstrap","bootstrap evaluators from production traces","Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says \"bootstrap evaluators\", \"generate evaluators\", \"create evals from traces\", \"eval bootstrap\", \"write evaluators\", \"build eval suite\", \"publish evaluators\", or wants to generate BaseEvaluator\u002FLLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[826,827,828,829],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-08-22T03:24:04.870382",{"slug":832,"name":832,"fn":833,"description":834,"org":835,"tags":836,"stars":28,"repoUrl":29,"updatedAt":846},"agent-observability-eval-pipeline","run agent observability and evaluation pipelines","End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a \"continue\" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says \"run the eval pipeline\", \"go from traces to evals\", \"bootstrap evals end to end\", \"classify then RCA then bootstrap\", \"build an eval set from scratch\", \"onboard me to datasets and experiments\", \"walk me through experiments\", \"I have an ml_app, now what\", \"Agent Observability onboarding\", \"guided experiment setup\", \"from traces to experiments\", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after \u003Cphase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at \u003Cphase>`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[837,840,843,844,845],{"name":838,"slug":839,"type":15},"Agents","agents",{"name":841,"slug":842,"type":15},"Data Pipeline","data-pipeline",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},"2026-08-22T03:24:05.862875",{"slug":848,"name":848,"fn":849,"description":850,"org":851,"tags":852,"stars":28,"repoUrl":29,"updatedAt":860},"agent-observability-experiment-analyzer","analyze LLM experiment results","Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says \"analyze experiment\", \"compare experiments\", \"analyze against baseline\", or provides one or two experiment IDs for analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[853,856,857,858,859],{"name":854,"slug":855,"type":15},"Analytics","analytics",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:21.212498",{"slug":4,"name":4,"fn":5,"description":6,"org":862,"tags":863,"stars":28,"repoUrl":29,"updatedAt":30},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[864,865,866,867,868],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":870,"name":870,"fn":871,"description":872,"org":873,"tags":874,"stars":28,"repoUrl":29,"updatedAt":882},"agent-observability-replay-trace","iterate on LLM traces with local code","Use when a developer wants to iterate on ONE specific Agent Observability \u002F LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as \u002Fagent-observability-replay-trace \u003Ctrace-id> [changes to test]. Signals: \"replay this trace\"; \"iterate on a trace\"; \"this trace's output is wrong, fix it and re-run\"; \"re-run trace \u003Cid> with \u003Cchange>\"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace \u002F LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser \"Replay\" button (that's agent-observability-replay-experiment), building an experiment from a dataset\u002FCSV, writing evaluators, root-causing failed traces, or RUM\u002FHTTP session replay.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[875,876,877,878,879],{"name":20,"slug":21,"type":15},{"name":814,"slug":815,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":880,"slug":881,"type":15},"Tracing","tracing","2026-08-06T05:38:08.57055",{"slug":884,"name":884,"fn":885,"description":886,"org":887,"tags":888,"stars":28,"repoUrl":29,"updatedAt":894},"agent-observability-session-classify","evaluate user intent satisfaction in sessions","Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions\u002Ftraces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[889,890,891,892,893],{"name":854,"slug":855,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:26.341497",{"slug":896,"name":896,"fn":897,"description":898,"org":899,"tags":900,"stars":28,"repoUrl":29,"updatedAt":906},"agent-observability-trace-rca","diagnose LLM application failures from traces","Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says \"what's wrong with my app\", \"why is my eval failing\", \"analyze errors\", \"root cause analysis\", \"diagnose failures\", or wants to understand production failure patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[901,902,903,904,905],{"name":20,"slug":21,"type":15},{"name":814,"slug":815,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:24.316244",{"slug":908,"name":908,"fn":909,"description":910,"org":911,"tags":912,"stars":28,"repoUrl":29,"updatedAt":918},"agent-skills","use Datadog observability skills","Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[913,914,917],{"name":20,"slug":21,"type":15},{"name":915,"slug":916,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},"2026-08-16T03:30:06.569153",{"slug":920,"name":920,"fn":921,"description":922,"org":923,"tags":924,"stars":28,"repoUrl":29,"updatedAt":938},"datadog-app","build and manage Datadog applications","Guides developers building Datadog Apps with TypeScript, React, the @datadog\u002Fapps scaffolder, and @datadog\u002Fvite-plugin. Use when a user wants to scaffold, run, debug, upgrade, build, upload, publish, upload without publishing (draft upload), add an upload-no-publish script, set up CI\u002FCD, use OAuth or API\u002Fapplication key auth, trigger\u002Fpoll Workflow Automation, choose DDSQL or Action Catalog for backend data access, or query app datastores with DDSQL, including backend function troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[925,926,929,932,935],{"name":20,"slug":21,"type":15},{"name":927,"slug":928,"type":15},"Frontend","frontend",{"name":930,"slug":931,"type":15},"React","react",{"name":933,"slug":934,"type":15},"TypeScript","typescript",{"name":936,"slug":937,"type":15},"Vite","vite","2026-06-18T08:01:32.562331",{"slug":940,"name":940,"fn":941,"description":942,"org":943,"tags":944,"stars":28,"repoUrl":29,"updatedAt":950},"dd-account-setup","configure Datadog account authentication","Ensure the user has an authenticated Datadog account with a valid DD_API_KEY on the right region before any Datadog setup or instrumentation. Detects existing DD_API_KEY \u002F DD_APP_KEY \u002F DD_SITE, validates them against the Datadog API, and fixes the common wrong-region 403. If no usable key exists, signs the user in (OAuth) or creates a new account, then obtains and validates a key. Use this whenever a user needs a Datadog account or API key, hits a 403 \u002F wrong-region error, or is about to run any Datadog *-setup or instrumentation skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[945,948,949],{"name":946,"slug":947,"type":15},"Configuration","configuration",{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},"2026-08-16T04:01:04.151248",38,{"items":953,"total":1008},[954,961,969,976,984,992,1000],{"slug":792,"name":792,"fn":793,"description":794,"org":955,"tags":956,"stars":28,"repoUrl":29,"updatedAt":805},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[957,958,959,960],{"name":20,"slug":21,"type":15},{"name":799,"slug":800,"type":15},{"name":802,"slug":803,"type":15},{"name":13,"slug":14,"type":15},{"slug":807,"name":807,"fn":808,"description":809,"org":962,"tags":963,"stars":28,"repoUrl":29,"updatedAt":819},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[964,965,966,967,968],{"name":20,"slug":21,"type":15},{"name":814,"slug":815,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":821,"name":821,"fn":822,"description":823,"org":970,"tags":971,"stars":28,"repoUrl":29,"updatedAt":830},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[972,973,974,975],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":832,"name":832,"fn":833,"description":834,"org":977,"tags":978,"stars":28,"repoUrl":29,"updatedAt":846},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[979,980,981,982,983],{"name":838,"slug":839,"type":15},{"name":841,"slug":842,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":848,"name":848,"fn":849,"description":850,"org":985,"tags":986,"stars":28,"repoUrl":29,"updatedAt":860},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[987,988,989,990,991],{"name":854,"slug":855,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":993,"tags":994,"stars":28,"repoUrl":29,"updatedAt":30},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[995,996,997,998,999],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":870,"name":870,"fn":871,"description":872,"org":1001,"tags":1002,"stars":28,"repoUrl":29,"updatedAt":882},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1003,1004,1005,1006,1007],{"name":20,"slug":21,"type":15},{"name":814,"slug":815,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":880,"slug":881,"type":15},36]