[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-datadog-labs-agent-observability-experiment-py-bootstrap":3,"mdc--ob6szo-key":39,"related-org-datadog-labs-agent-observability-experiment-py-bootstrap":8605,"related-repo-datadog-labs-agent-observability-experiment-py-bootstrap":8771},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":28,"repoUrl":29,"updatedAt":30,"license":31,"forks":32,"topics":33,"repo":34,"sourceUrl":37,"mdContent":38},"agent-observability-experiment-py-bootstrap","generate Python experiment clients for LLM observability","Generates a self-contained Python experiment client that uses the ddtrace.llmobs SDK. Emits either a runnable .py script or a Jupyter .ipynb notebook matching the canonical DataDog reference notebook style. Use when the user says \"generate Python experiment\", \"write an SDK experiment\", \"create a ddtrace experiment\", \"Python notebook experiment\", \"use the Agent Observability SDK\", or has `ddtrace` installed and wants idiomatic SDK code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"datadog-labs","Datadog Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatadog-labs.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":20,"slug":21,"type":15},"Datadog","datadog",{"name":23,"slug":24,"type":15},"Python","python",{"name":26,"slug":27,"type":15},"Jupyter","jupyter",145,"https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills","2026-06-19T09:04:22.640384",null,19,[],{"repoUrl":29,"stars":28,"forks":32,"topics":35,"description":36},[],"Public repository for Datadog Agent Skills","https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fagent-observability\u002Fagent-observability-experiment-py-bootstrap","---\nname: agent-observability-experiment-py-bootstrap\ndescription: Generates a self-contained Python experiment client that uses the ddtrace.llmobs SDK. Emits either a runnable .py script or a Jupyter .ipynb notebook matching the canonical DataDog reference notebook style. Use when the user says \"generate Python experiment\", \"write an SDK experiment\", \"create a ddtrace experiment\", \"Python notebook experiment\", \"use the Agent Observability SDK\", or has `ddtrace` installed and wants idiomatic SDK code.\n---\n\n## Backend\n\n**Detection** — At the start of every invocation, before taking any action, determine which backend (if any) to use for the startup beacon call:\n\n1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** for the beacon. Skip steps 2–3.\n2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__list_llmobs_evals` appears in your available tools. If present → use **MCP mode**.\n3. If MCP tools are absent → check whether `pup` is executable via `pup --version`. If responding → use **pup mode**.\n4. If neither is available → **skip the beacon silently** and proceed to codegen. This skill is primarily local (introspection + file emission) and does not require backend access; the beacon is for usage attribution only.\n\n**Invocation ID:** At the very start of each invocation, before any MCP\u002Fpup call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.\n\n**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:agent-observability-experiment-py-bootstrap[\u003Cinv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only** (the startup beacon below), use `skill:agent-observability-experiment-py-bootstrap:start[\u003Cinv_id>] — ` instead (note the `:start` suffix).\n\n**Startup beacon:** Immediately after parsing arguments (workflow step 1, before dataset resolution in step 2), issue exactly one beacon call to register skill usage and validate backend connectivity. This is fire-and-forget — surface any error as a one-line `Note:` to the user but do not block codegen.\n\n- **MCP mode:** call `mcp__datadog-llmo-mcp__list_llmobs_evals` with `telemetry.intent = \"skill:agent-observability-experiment-py-bootstrap:start[\u003Cinv_id>] — Skill startup: register usage and verify Datadog connectivity\"`. Discard the response payload; the call's purpose is the telemetry tag.\n- **pup mode:** run `pup llm-obs evals list --limit 1` via Bash. Pup carries its own telemetry; no intent prefix needed.\n- **No backend:** print one line `(Telemetry beacon skipped — no Datadog backend detected; this is informational only and does not affect codegen.)` and proceed.\n\nThe beacon **must not** fail the skill. If the call errors (auth, network, etc.), surface a one-line note and continue.\n\n---\n\n# Agent Observability Experiment (Python) Bootstrap — Generate a Python Experiment Using `ddtrace.llmobs`\n\nProduce a single self-contained Python experiment that uses the official **`ddtrace.llmobs` SDK**. Output is either a `.py` script or an `.ipynb` notebook. The generated code mirrors the patterns shown in DataDog's reference notebooks at \u003Chttps:\u002F\u002Fgithub.com\u002FDataDog\u002Fllm-observability\u002Ftree\u002Fmain\u002Fexperiments\u002Fnotebooks>.\n\nThe SDK handles lazy project\u002Fexperiment creation, dataset push diffing, the 5 MB \u002F 1000-record bulk threshold, eval metric streaming, and the status state machine on the user's behalf. This skill must therefore **never re-implement those primitives** — it just imports `LLMObs` and trusts it.\n\n## Usage\n\n```\n\u002Fagent-observability-experiment-py-bootstrap [--purpose \u003Cfree text>] [--format py|ipynb] [--dataset \u003Cpath>] [--dataset-name \u003Cname>] [--dataset-version \u003Cint>] [--project-name \u003Cname>] [--evaluator-style function|class|remote] [--jobs \u003Cn>] [--output \u003Cpath>] [--task-source \u003Cmodule:function>] [--placeholder-task] [--app-root \u003Cpath>] [--env-file \u003Cpath>]\n```\n\nArguments: $ARGUMENTS\n\n### Inputs\n\nAll inputs are optional. If the user omits a flag, fall back to the default — never block on prompting for `--jobs`, `--format`, etc.\n\n| Input | Default | Description |\n|---|---|---|\n| `--format` | `py` | `py` (single `.py` file) or `ipynb` (Jupyter notebook with one cell per section). |\n| `--dataset` | none — emit a sample 3-record `records=[...]` inline so the file is runnable as-is | Path to a local `DatasetRecordRaw[]` JSON or CSV. JSON → `create_dataset(records=...)`; CSV → `create_dataset_from_csv(...)`. Mutually exclusive with `--dataset-name`. |\n| `--dataset-name` | none | Name of an existing Datadog dataset to fetch at runtime via `LLMObs.pull_dataset(...)`. Use this when the dataset already lives in Datadog (e.g. created in the UI or by a prior run) — no local file required. Mutually exclusive with `--dataset`. |\n| `--dataset-version` | none (latest) | Pin to a specific dataset version when using `--dataset-name`. Passed through as `pull_dataset(version=N)`. Ignored if `--dataset-name` is not set. |\n| `--project-name` | `experiment-\u003Cservice-name>` — derived from the codebase (see Workflow step 1); falls back to `experiment-sdk-default` only if nothing resolves | Datadog project name (visible in the LLM Experiments UI). The SDK's `ml_app` tag falls back to this automatically — no separate flag needed. |\n| `--evaluator-style` | `function` | `function` (plain functions — notebook default), `class` (`BaseEvaluator` subclasses), or `remote` (`RemoteEvaluator` instances). |\n| `--jobs` | `10` | Passed to `experiment.run(jobs=N)`. |\n| `--output` | `.\u002Fexperiments\u002Fexperiment.\u003Cext>` | File extension derives from `--format`: `.py` or `.ipynb`. |\n| `--task-source` | auto — discovered by application introspection (see Workflow step 2.5) | Explicit override: `\u003Cdotted.module.path>:\u003Cfunction_name>` for the function to wrap as `task_fn`. Use when you already know the entry point and want to skip the introspection scan. |\n| `--placeholder-task` | off | Opt out of application introspection and emit the generic `# TODO(user)` placeholder task. Use when scaffolding without a real app, in tests, or when the user explicitly wants to fill in the task themselves. |\n| `--app-root` | resolved from `pyproject.toml` \u002F `setup.cfg` \u002F `setup.py` \u002F cwd | Root directory the introspection scan is restricted to. Skipped if `--placeholder-task` or `--task-source` is set. |\n| `--env-file` | none — generated file auto-discovers `.env` files at runtime (see Workflow step 4, section 1) | Explicit absolute path to a `.env`-style file. Generated code preloads this path **first** before the auto-discovery walk. Use when your credentials live in a non-standard location (e.g. `~\u002F.config\u002Fdd\u002Fstaging.env`). |\n| `--purpose` | auto — prompted via `AskUserQuestion` in Workflow step 2.0 if not set or inferable from the invocation message | Free-form string describing what the experiment is meant to validate (e.g. `\"test that the agent picks the right tool for ambiguous user requests\"`, `\"verify SQL output always parses\"`, `\"regression-test prompt v3 against prod baseline\"`). Used as reasoning context — biases candidate ranking in Workflow step 2.5, shapes the wrapper return type in 2.5d, seeds evaluator selection in step 3, and is embedded in the generated file's header comment. NOT a fixed taxonomy — Claude reads the string and decides effects dynamically per invocation. |\n\n---\n\n## SDK Surface (Cited)\n\nThese are the public symbols the generated code uses. All come from `ddtrace.llmobs` (the public package — never from `ddtrace.llmobs._experiment` or other underscore-prefixed modules).\n\n| Import | Source | What it gives you |\n|---|---|---|\n| `LLMObs` | `ddtrace\u002Fllmobs\u002F__init__.py` re-exports `_llmobs.py` | `.enable()`, `.create_dataset()`, `.create_dataset_from_csv()`, `.pull_dataset(dataset_name, project_name, version)`, `.experiment()`, `.async_experiment()` |\n| `RemoteEvaluator`, `EvaluatorContext` | `ddtrace\u002Fllmobs\u002F__init__.py` | LLM-as-Judge that runs server-side; preferred over inline `LLMJudge` |\n| `BaseEvaluator`, `EvaluatorResult` | `ddtrace\u002Fllmobs\u002F__init__.py` | Class-based evaluator path (advanced) |\n| `LLMJudge` | `ddtrace\u002Fllmobs\u002F_evaluators\u002Fllm_judge.py` (re-exported) | Inline LLM-as-Judge with prompt template support |\n\n**Canonical call signatures** (must match the generated code exactly):\n\n```python\nLLMObs.enable(\n    api_key=os.getenv(\"DD_API_KEY\"),\n    app_key=os.getenv(\"DD_APPLICATION_KEY\"),\n    site=os.getenv(\"DD_SITE\", \"datadoghq.com\"),  # required for non-prod sites (e.g. datad0g.com, datadoghq.eu)\n    project_name=\"\u003Cproject>\",\n    agentless_enabled=True,  # required when not running behind the dd-agent\n)\n# Note: ml_app is not a separate input. The SDK derives it from project_name\n# when not supplied. If a user really wants to override it later, they can\n# add `ml_app=\"...\"` to enable() themselves.\n\ndataset = LLMObs.create_dataset(\n    dataset_name=\"\u003Cname>\",\n    description=\"\u003Coptional>\",\n    records=[\n        # Per-record `tags` MUST be a list of \"key:value\" strings (e.g. \"env:smoke\"),\n        # never bare strings — the SDK rejects malformed tags with a ValueError on append.\n        {\"input_data\": {\"\u003Ck>\": \"\u003Cv>\"}, \"expected_output\": \"\u003Cv>\", \"metadata\": {}, \"tags\": [\"env:\u003Cenv>\"]},\n        # ...\n    ],\n)\n# OR\ndataset = LLMObs.create_dataset_from_csv(\n    csv_path=\"\u003Cpath>\",\n    dataset_name=\"\u003Cname>\",\n    input_data_columns=[\"\u003Ccol1>\", \"\u003Ccol2>\"],\n    expected_output_columns=[\"\u003Ccol>\"],\n)\n# OR pull an existing Datadog dataset by name (no local file needed)\ndataset = LLMObs.pull_dataset(\n    dataset_name=\"\u003Cname>\",\n    project_name=\"\u003Cproject>\",   # optional — defaults to the project on enable()\n    version=2,                  # optional — pin a version; omit for the latest\n)\n\ndef task_fn(input_data: dict, config: dict):\n    # In real output, this is wired to the user's discovered LLM call site via\n    # Workflow step 2.5. Only emits as a generic placeholder (with # TODO(user))\n    # when --placeholder-task is set or introspection found nothing.\n    ...\n\n# Plain function evaluator (default style)\ndef exact_match(input_data, output_data, expected_output) -> bool:\n    return output_data == expected_output\n\nexperiment = LLMObs.experiment(\n    name=\"\u003Cexperiment_name>\",\n    dataset=dataset,\n    task=task_fn,\n    evaluators=[exact_match],\n    config={\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.0,\n        # Provenance also lives in `config` so it renders in the\n        # experiment's Configuration view alongside model\u002Ftemperature.\n        # `tags=` below only reaches metadata.tags, which the current UI\n        # does not surface as chips — config is what users actually see.\n        \"generated_by\": \"claude-code\",\n        \"skill\": \"agent-observability-experiment-py-bootstrap\",\n        \"purpose\": \"\u003Cone-line purpose from step 2.4>\",\n    },\n    description=\"\u003Cone-line purpose from step 2.4>\",\n    tags={\n        # Same provenance, sent to experiment metadata.tags for any future\n        # tag-filter UI \u002F API consumers. Always emitted alongside the\n        # config copy — never one without the other.\n        \"generated_by\": \"claude-code\",\n        \"skill\": \"agent-observability-experiment-py-bootstrap\",\n        \"purpose\": \"\u003Cone-line purpose from step 2.4>\",\n    },\n)\n\nexperiment.run(jobs=10)\nprint(experiment.url)\n```\n\n---\n\n## Evaluator Styles\n\nGenerated code uses **one** of three evaluator surfaces, picked by `--evaluator-style`. The detail for each lives in `references\u002Fevaluator-styles\u002F\u003Cstyle>.md` and is loaded on demand — only the chosen style is consulted at generation time. The single piece of guidance that applies to all three is the `EvaluatorResult` rule below.\n\n### Return `EvaluatorResult`, not bare values\n\nPlain functions are allowed to return `bool` \u002F `float` \u002F `dict`, and `BaseEvaluator.evaluate()` is allowed to return raw `JSONType`. The SDK accepts both — but `EvaluatorResult` carries fields the Datadog UI surfaces in ways the raw value cannot:\n\n| Field | Type | Used by Datadog UI for |\n|---|---|---|\n| `value` | `bool` \u002F `float` \u002F `str` \u002F `dict` (JSONType) | The score itself — shown on the experiment metric. **Required.** |\n| `reasoning` | `str` | Per-record explanation shown in the compare UI; lets reviewers see *why* an evaluator passed\u002Ffailed without re-running the LLM. |\n| `assessment` | `str` (e.g. `\"pass\"` \u002F `\"fail\"` \u002F `\"partial\"`) | Determines whether a metric trend going up vs. down is an improvement; the UI uses this to color baseline-vs-candidate comparisons. |\n| `metadata` | `dict[str, JSONType]` | Free-form per-record context (e.g. `{\"confidence\": 0.95}`); shown in record drill-down. |\n| `tags` | `dict[str, JSONType]` | Used to slice experiment results in the UI (e.g. `{\"category\": \"accuracy\"}`). |\n\nDefault to `EvaluatorResult` for any evaluator richer than a one-line equality check. Trivial checks like `exact_match` and `length_under_500` are the only cases where a bare `bool` is acceptable.\n\n### Style router\n\nWhen `--evaluator-style` is resolved (default `function`), read the matching reference file from `\u003Cthis-skill-dir>\u002Freferences\u002Fevaluator-styles\u002F` and emit section 5 of the generated file using the code template it contains:\n\n| `--evaluator-style` value | Read | Best for |\n|---|---|---|\n| `function` *(default)* | `references\u002Fevaluator-styles\u002Ffunction.md` | Most cases — matches the canonical notebook style |\n| `class` | `references\u002Fevaluator-styles\u002Fclass.md` | Evaluators that need persistent state or an async client |\n| `remote` | `references\u002Fevaluator-styles\u002Fremote.md` | Server-side LLM-as-Judge configured in the Datadog UI, reused across experiments |\n\nDo **not** load all three. Each reference file is self-contained — code template + when-to-use + when-not-to-use guidance.\n\n---\n\n## Generated File Structure\n\nThe same section sequence in both formats. In `.py` these become comment banners; in `.ipynb` each becomes one markdown cell + one code cell.\n\n```\n0. File header docstring — name, generated_at, purpose (from step 2.4), provider, wired task source.\n1. Env setup           — auto-discover .env files (cwd, app root, parent walk,\n                         ~\u002F.datadog\u002Fcredentials), then os.getenv reads + hard-assert\n                         required keys. NO python-dotenv dependency. Shell env wins\n                         over file-loaded values. Only the provider keys actually\n                         needed by the wired task_fn are asserted (see step 2.5).\n2. LLMObs.enable()     — explicit api_key\u002Fapp_key\u002Fproject_name\u002Fagentless_enabled\n3. Dataset             — inline records OR create_dataset_from_csv\n4. Task function       — REAL task function imported from the user's app (via Workflow step 2.5\n                         introspection) and adapted to the SDK signature. Only falls back to a\n                         placeholder OpenAI call with # TODO(user) if --placeholder-task is set or\n                         no LLM call site was found in the project.\n5. Evaluators          — 2-3 in the requested style, semantics seeded by the purpose (step 3)\n6. Experiment          — LLMObs.experiment(config={..., \"generated_by\": \"claude-code\", \"purpose\": \"...\", ...}, tags={...})\n7. Run                 — experiment.run(jobs=N); print(experiment.url)\n8. Results inspection  — experiment.as_dataframe() if pandas, else print\n```\n\n---\n\n## Workflow\n\n1. **Parse arguments**. Default `--format py`. Resolve `--output` extension from `--format`.\n\n   If `--project-name` is not provided, resolve a default of the form `experiment-\u003Cservice-name>` by walking these sources in order, taking the first match:\n   1. `pyproject.toml` → `[project] name` (PEP 621) or `[tool.poetry] name`.\n   2. `setup.cfg` → `[metadata] name`.\n   3. `setup.py` → first `name=\"...\"` argument to `setup(...)`.\n   4. `package.json` → `\"name\"` (useful when the LLM app lives in a TS\u002FJS monorepo Python package).\n   5. The basename of the current working directory, lowercased and slugified (`\u002F^[a-z0-9-]+$\u002F` — replace non-matching chars with `-`).\n\n   The final project name is `experiment-\u003Cservice-name>`. Strip a leading `experiment-` from `\u003Cservice-name>` if it already starts with one (so a package literally named `experiment-foo` yields `experiment-foo`, not `experiment-experiment-foo`). If none of the five sources resolve to a non-empty string, fall back to `experiment-sdk-default` and emit a warning in the next-steps output that the user should set `--project-name` explicitly.\n\n   Embed the resolved name as a string literal in the generated `PROJECT_NAME = \"...\"` line — don't emit runtime `os.getcwd()` lookups, since the user may run the file from a different directory than where the skill resolved it.\n\n1.5. **Startup beacon.** Per the **Backend** section above: generate an 8-character hex invocation ID, then issue the single beacon call (MCP `list_llmobs_evals` tagged with `skill:agent-observability-experiment-py-bootstrap:start[\u003Cinv_id>] — Skill startup: ...`, or pup equivalent, or skip if no backend). Surface any error as a one-line `Note:` and proceed regardless — the beacon is for usage attribution and connectivity validation only; it never blocks codegen.\n\n2. **Resolve the dataset source.** Error out if both `--dataset` and `--dataset-name` are passed — they're mutually exclusive.\n\n   - **`--dataset \u003Cpath>` (local file → inline records or CSV loader)**:\n     - Read the file. If JSON, validate top-level array of `DatasetRecordRaw` shape (`input_data`, optional `expected_output`, `metadata`, `tags`). If CSV, parse header and auto-detect columns using the `dataset-bootstrap` heuristics: `prompt|input|query|question` → input, `expected|gold|truth|answer` → expected.\n     - Run a PII scrub (email\u002Fphone\u002FSSN\u002FAPI-key regexes) on all string values; replace matches with `\u003CREDACTED:pii-type>` and surface a warning listing affected indices.\n     - **For JSON datasets**, embed the records inline in the generated file (`records=[...]`) so the user has a single self-contained artifact. **For CSV datasets**, emit `LLMObs.create_dataset_from_csv(csv_path=\"\u003Cabsolute path>\", ...)` and tell the user the CSV needs to be present at runtime.\n\n   - **`--dataset-name \u003Cname>` (existing Datadog dataset → runtime pull)**:\n     - Emit `LLMObs.pull_dataset(dataset_name=\"\u003Cname>\", project_name=\"\u003Cproject>\"[, version=\u003Cn>])` in place of any `create_dataset*` call. The fetch happens when the generated experiment runs — the skill itself does not call Datadog.\n     - Pass `version=\u003Cn>` through only if `--dataset-version` was set; otherwise omit it so the SDK resolves the latest.\n     - Add a one-line comment above the call documenting what's being pulled, e.g. `# Pulled from Datadog: dataset_name=\"qa_v3\", version=latest`.\n     - Skip the PII scrub and the inline-records emission — there are no local records to scrub.\n\n   - **Neither flag given**:\n     - Fall back to the inline 3-record sample described under `--dataset`'s default, so the generated file remains runnable as-is.\n\n   **Note on dataset IDs.** The public SDK's `LLMObs.pull_dataset(...)` takes a name, not an ID — so there's no `--dataset-id` flag. If a user only has a dataset ID from a Datadog UI URL (`\u002Fllm\u002Fdatasets\u002F\u003Cid>`), the workflow is: open that URL in the UI, copy the dataset name, and pass it as `--dataset-name`. The skill must not import `ddtrace.llmobs._experiment` or any other underscore module to work around this.\n\n2.4. **Determine the experiment's purpose.** Capture a one-sentence statement of what the experiment is meant to validate. This becomes a *reasoning input* that biases every downstream step (introspection ranking in 2.5, wrapper shape in 2.5d, evaluator selection in step 3, file header in step 4). It is **not** a fixed taxonomy — the user types whatever describes their goal, and Claude reads it and applies judgment per invocation.\n\n   **Resolution order**:\n\n   1. If `--purpose \"\u003Ctext>\"` was passed → use it verbatim. Skip to 2.5.\n   2. If the user's original invocation message contains a clear purpose statement (e.g. *\"set up an experiment to test my tool selection logic\"*, *\"validate that SQL output is always parseable\"*, *\"regression-test prompt v3\"*), extract it and present it back for confirmation via `AskUserQuestion` with the extracted text as a single option labeled \"Use the purpose I described\" plus \"Pick a different purpose\" and \"Other\". If confirmed, use the extracted text. Skip to 2.5.\n   3. Otherwise, prompt with `AskUserQuestion`. The options are **seed prompts**, not a constrained taxonomy — they exist to give the user something to react to. Each \"label\" maps to a starter sentence that the user can refine; selecting an option uses its starter sentence as `purpose` unless the user provides notes\u002Ffree text instead.\n\n   **`AskUserQuestion` payload** (use these 5 options verbatim — they cover the common cases without locking the user into them):\n\n   ```\n   question: \"What is this experiment meant to validate?\"\n   header:   \"Purpose\"\n   options:\n     - label: \"Output accuracy \u002F answer quality\"\n       description: \"Verify the model produces correct answers compared to expected outputs. Most common starting point.\"\n     - label: \"Tool call correctness\"\n       description: \"For agent apps — validate the agent picks the right tool with the right arguments. Useful when tool routing is the failure surface.\"\n     - label: \"Structured output \u002F schema validity\"\n       description: \"Verify the output always conforms to a required shape (JSON, SQL, citation format, etc.).\"\n     - label: \"Retrieval \u002F RAG faithfulness\"\n       description: \"Verify the answer is grounded in retrieved documents — no hallucinations beyond the retrieved context.\"\n     - label: \"Refactor \u002F regression test\"\n       description: \"Check whether a prompt or code change preserves observed behavior. Uses the dataset's expected_output as the ground-truth baseline.\"\n   ```\n\n   The user picks an option (which becomes the starter sentence), or picks \"Other\" and writes their own. Either way, the resulting string is the `purpose` carried forward.\n\n   **What to do with the purpose downstream** — Claude reads it and reasons about effects, no static mapping:\n\n   - **In step 2.5 (introspection ranking)**: read the purpose; if it mentions tools \u002F agents → boost candidates with `@LLMObs.agent`, `@workflow`, LangChain `ReActAgent`, or tool-using shapes. If it mentions retrieval \u002F RAG \u002F grounding → boost candidates that import vector stores or call `retrieve`\u002F`query_engine`. If it mentions schema \u002F JSON \u002F SQL → boost candidates that use `response_format`, pydantic, or structured-output APIs. Apply judgment — there's no hardcoded mapping.\n   - **In step 2.5d (wrapper generation)**: if the purpose needs richer signal than just the final output (tool calls, retrieved docs, intermediate state), emit a wrapper that captures it *if the user's function exposes it*. Otherwise emit a `# Note:` comment explaining what the wrapper would ideally return and why a richer return shape would help — do not refactor the user's actual function.\n   - **In step 3 (evaluator template)**: read the purpose and seed the evaluator list to match. Accuracy → standard `exact_match` + LLMJudge. Tool calls → an LLMJudge with a tool-correctness rubric, or a `RemoteEvaluator` if the user has one configured. Schema → `JSONEvaluator(required_keys=[...])` or `RegexMatchEvaluator`. Retrieval → an LLMJudge for groundedness. Regression → `exact_match` + a near-match check. The `--evaluator-style` flag (function \u002F class \u002F remote) still picks the SURFACE; the purpose picks the SEMANTICS.\n   - **In step 4 (file emission)**: include the purpose as a `# Purpose:` comment in the file's docstring header so the user (and reviewers) can see what the experiment was scaffolded for.\n   - **In Output (next-steps)**: surface the resolved purpose in the next-steps block so it's part of the run-summary record.\n\n   **If the user is silent \u002F picks the default option without notes**, use the starter sentence as-is. Never block on this step — there should always be a `purpose` string moving forward, even if it's a generic \"Output accuracy \u002F answer quality\".\n\n---\n\n2.5. **Discover the task function from the user's application code.** This step replaces the old \"placeholder `task_fn`\" behavior — an onboarding-grade experiment needs to actually exercise the user's real LLM logic. The skill must do the work, not push it onto the user.\n\n   **Skip this entire step if** `--placeholder-task` is set, OR if `--task-source \u003Cmodule>:\u003Cfunction>` is set (in which case use the provided source directly — jump to substep 2.5d). Otherwise:\n\n   **2.5a — Resolve the app root.** If `--app-root \u003Cpath>` was supplied, use it. Otherwise use the directory containing whichever of these resolved during step 1 (in order): `pyproject.toml`, `setup.cfg`, `setup.py`, `package.json`. If none of those resolved, use the current working directory. **Hard cap** the scan to that directory tree; do **not** traverse `node_modules`, `.venv`, `venv`, `__pycache__`, `.git`, `dist`, `build`, `target`, `vendor`, `third_party`, or any directory matched by `.gitignore` if present. Refuse to scan if the resolved root is `\u002F` or `~` — that means resolution failed; treat as no candidate found.\n\n   **2.5b — Candidate discovery.** Use Grep \u002F Bash to find Python files inside the app root and identify call sites of these LLM SDKs (the union of common LLM clients):\n\n   | Module \u002F call pattern | Signal |\n   |---|---|\n   | `openai.ChatCompletion.create`, `openai.chat.completions.create`, `client.chat.completions.create` | OpenAI chat |\n   | `openai.completions.create`, `client.completions.create` | OpenAI text |\n   | `anthropic.messages.create`, `client.messages.create`, `Anthropic(...).messages.create` | Anthropic |\n   | `litellm.completion`, `litellm.acompletion` | LiteLLM (router) |\n   | `langchain.*.invoke`, `ChatOpenAI(...)`, `ChatAnthropic(...)`, `LLMChain(...)` | LangChain |\n   | `from llama_index`, `as_query_engine`, `as_chat_engine` | LlamaIndex |\n   | `google.generativeai.GenerativeModel(...).generate_content` | Vertex\u002FGemini |\n   | `boto3.client(\"bedrock-runtime\").invoke_model` | AWS Bedrock |\n   | `@LLMObs.llm`, `@LLMObs.agent`, `@LLMObs.workflow`, `@LLMObs.task`, `@workflow`, `@agent` | Already instrumented |\n\n   For each match, walk **up** to the enclosing function (`def` \u002F `async def`) — call it the *call-site function*. Record `{file, line, function_name, is_async, signature, enclosing_class}`.\n\n   **Skip** files matching `test_*.py`, `*_test.py`, `tests\u002F**`, `conftest.py`, `*_fixtures.py`. Skip private helpers (`def _foo` where the function is the *only* one in the file and looks like a utility — judge by whether it has a typed string input).\n\n   **2.5c — Score and rank candidates.** Score each candidate; higher is more likely to be the \"core\" entry point:\n\n   | Signal | Score delta |\n   |---|---|\n   | Function name in {`generate`, `chat`, `complete`, `respond`, `answer`, `handle_request`, `process_query`, `process_message`, `run`, `predict`, `infer`, `call_llm`, `query`, `agent_loop`, `main`} | **+5** |\n   | Function takes exactly one or two parameters, at least one being a `str` or `dict` | **+3** |\n   | Function is decorated with any `@LLMObs.*` \u002F `@workflow` \u002F `@agent` decorator | **+5** (this is the *intended* instrumentation point) |\n   | Function is at the top of its module (first non-import block) | **+2** |\n   | Module path matches `**\u002F{main,app,api,handlers,server,routes,agent,bot,chat}.py` | **+3** |\n   | Function is a class method (`self` first arg) but the class name matches `*Agent`, `*Bot`, `*Handler`, `*Service` | **+2** |\n   | Function name starts with `_` and module has other non-underscore candidates | **−3** (likely a helper) |\n   | Function file is under `examples\u002F`, `scripts\u002F`, `notebooks\u002F` | **−2** |\n   | Function uses LLM SDK at multiple lines (looks like a multi-step orchestration, not just a single call) | **+1** |\n\n   **Apply the experiment purpose from step 2.4 as an additional soft bias on top of these scores.** Read the purpose string and bump candidates whose shape matches what the user is trying to test — agent \u002F tool-using functions for tool-correctness purposes; retrieval \u002F vector-store users for RAG \u002F grounding purposes; structured-output users for schema purposes; etc. Use judgment, not a hardcoded mapping. A purpose-aligned candidate should typically beat a generically-better-named one (+3 to +5 effective bias is reasonable). See step 2.4 \"What to do with the purpose downstream\" for examples.\n\n   Pick the **top 3** candidates and present them to the user as a single short prompt (no `AskUserQuestion` — just a checkpoint prompt in chat):\n\n   ```\n   ## Task function discovery\n\n   Scanned `\u003Capp_root>` and found \u003CN> LLM-calling functions. Top candidates:\n\n   1. `\u003Cmodule.path>:\u003Cfunction_name>`   score \u003CS>   (\u003Cfile>:\u003Cline>, args=\u003Csig>)\n   2. ...\n   3. ...\n\n   I'll wire candidate **1** as the experiment's task function unless you say otherwise.\n   Reply with the number to pick a different candidate, or \"placeholder\" to emit a\n   generic placeholder task instead.\n   ```\n\n   Wait for confirmation. If the user is silent \u002F says \"go\" \u002F says \"1\", use #1. If they pick another number, use that. If they say \"placeholder\", set `--placeholder-task` semantics and skip to step 3. If the user supplies a different `module:function`, validate it exists, then use it.\n\n   **If zero candidates were found**, do not block the user — print a one-line note (\"No LLM call sites detected under `\u003Capp_root>`. Emitting a generic placeholder task — replace it before running.\") and fall through with placeholder semantics.\n\n   **2.5d — Generate the task function wrapper.** Once a target `\u003Cmodule>:\u003Cfunction>` is locked in, emit a real `task_fn` that imports and calls the user's function. This is **section 4** of the generated file (\"Task function\") and must NOT include `# TODO(user)` markers if introspection succeeded — the whole point is to remove that burden.\n\n   Wrapper construction rules:\n\n   - **Import**: emit `from \u003Cmodule> import \u003Cfunction_name>` at the top of section 4 (NOT at the top of the file — keeping it local makes the section self-contained and easy to swap). If the function is a class method, also import the class and instantiate it lazily inside `task_fn` (default `__init__()`, with a `# TODO: pass constructor args if needed` comment only if the constructor takes required arguments).\n   - **Signature adaptation**: the SDK's task function signature is `task_fn(input_data: dict, config: dict) -> Any`. The user's function probably has a different signature. Build a small adapter:\n\n     - If the user's function takes one parameter and the dataset's `input_data` looks like `{\"\u003Ckey>\": \u003Cvalue>}` with one key, call `\u003Cfunction_name>(input_data[\"\u003Ckey>\"])`.\n     - If the user's function takes multiple parameters, map them by name from `input_data` keys. If a name doesn't match, fall back to positional via `input_data.values()` order but emit a one-line comment flagging the assumption.\n     - If the user's function takes `**kwargs` or a single dict, pass `input_data` through unmodified.\n     - If the user's function is `async`, wrap with `asyncio.run(...)` inside a sync `task_fn` (simplest path; lets `LLMObs.experiment(...).run()` stay sync). Alternative: emit an `async def task_fn` and use `LLMObs.async_experiment(...)` — only do this if the rest of the experiment is already async.\n   - **Config passthrough**: never silently drop the `config` argument. If the user's function takes a `config` \u002F `model` \u002F `temperature` parameter, wire `config.get(\"...\")` into it.\n   - **Comment header**: emit a comment block above `task_fn` that names the source function, file, line, the experiment purpose (from step 2.4), and any adaptation choices, e.g.:\n\n     ```python\n     # Task function wired to: \u003Cmodule.path>:\u003Cfunction_name>\n     #   source:  \u003Cfile>:\u003Cline>\n     #   purpose: \u003Cone-line purpose from step 2.4>\n     #   adapter: input_data[\"\u003Ckey>\"] -> \u003Cfunction_name>(\u003Ckey>=...)\n     #\n     # To experiment with prompt \u002F model variants without editing your app, inline\n     # the call here instead of importing.\n     ```\n\n   - **Richer-return shape when the purpose demands it**: read the purpose from step 2.4. If validating just the final output string is too lossy for what the user is testing (e.g. they're checking tool-call selection — the final string says \"I'll book that for you\" but the *interesting* signal is which tool was called with which args), emit a wrapper that captures the richer signal **only if the user's function exposes it**. Two cases:\n\n     - **The function already returns richer data** (e.g. `{\"output\": str, \"tool_calls\": [...]}`, or a LangChain `AgentExecutor.invoke()` result with `intermediate_steps`): wire `task_fn` to return that shape directly. Evaluators in step 3 will receive it as `output_data`.\n     - **The function returns just a string** but the purpose needs more: emit a `# Note:` comment above `task_fn` explaining what richer shape would help, and how the user could expose it (e.g. *\"To capture tool calls for evaluation, refactor `respond(query)` to also return the `intermediate_steps` from your AgentExecutor.\"*). **Do not refactor the user's function** — emit a note, ship the simple wrapper, let the user choose.\n\n   Never invent a richer return shape that the user's function doesn't actually provide. The wrapper is plumbing, not surgery.\n\n   - **Side-effect warning**: scan the chosen function (and its immediate calls within the same module) for these patterns. If any are present, print a `WARNING:` line in the next-steps output, but do NOT alter the import:\n\n     | Pattern | Warning |\n     |---|---|\n     | `os.environ[...]` reads beyond LLM API keys | \"Task reads env vars beyond LLM credentials — make sure they're set when running the experiment.\" |\n     | `requests.`, `httpx.`, `aiohttp.` calls to non-LLM-provider URLs | \"Task makes external HTTP calls — running the experiment will hit those endpoints.\" |\n     | DB drivers (`psycopg2`, `sqlalchemy`, `pymongo`, `redis`, `boto3` ≠ bedrock) | \"Task hits a database — point at a non-prod instance before running.\" |\n     | File I\u002FO writes (`open(..., 'w')`, `Path.write_*`) | \"Task writes to disk — make sure the path is safe in your experiment env.\" |\n\n   - **Fallback to placeholder**: if `--placeholder-task` is set OR the introspection picked nothing, emit the original placeholder block (generic OpenAI call with `# TODO(user)`). This is the only path where `TODO(user)` is allowed in the task section.\n\n2.6. **Determine required credentials and emit the env-setup section.** The generated experiment must work without the user pre-exporting anything in their shell, *as long as* a discoverable `.env` lives in a standard location. Don't push setup work onto the user that the file can do itself.\n\n   **Required Datadog keys** (always): `DD_API_KEY` AND (`DD_APPLICATION_KEY` OR `DD_APP_KEY`). `DD_SITE` is optional (defaults to `datadoghq.com`).\n\n   **Required provider keys** — depend on what step 2.5 picked. Inspect the imported call site to identify the provider, then **read the matching reference file** for the exact assert lines, adapter notes, and gotchas. Only load the one that applies; do not load all of them.\n\n   | Detected SDK in task | Read |\n   |---|---|\n   | OpenAI (`openai.*`, `client.chat.completions.create`) — also the placeholder fallback | `references\u002Fproviders\u002Fopenai.md` |\n   | Azure OpenAI (`AzureOpenAI(...)`, `azure_endpoint=`) | `references\u002Fproviders\u002Fopenai.md` (Azure section) |\n   | Anthropic (`anthropic.*`, `Anthropic().messages.create`) | `references\u002Fproviders\u002Fanthropic.md` |\n   | LiteLLM (`litellm.completion`, `litellm.acompletion`) | `references\u002Fproviders\u002Flitellm.md` |\n   | LangChain (`ChatOpenAI` \u002F `ChatAnthropic` \u002F etc.) | `references\u002Fproviders\u002Flangchain.md` (walks one level deeper to the underlying provider) |\n   | LlamaIndex | `references\u002Fproviders\u002Fllamaindex.md` |\n   | Google Gemini (`google.generativeai`) \u002F Vertex AI | `references\u002Fproviders\u002Fgemini.md` |\n   | AWS Bedrock (`boto3.client(\"bedrock-runtime\")`) | `references\u002Fproviders\u002Fbedrock.md` |\n   | Custom \u002F not-recognized SDK | No reference file — emit a `# TODO(user): set the API key(s) your task_fn needs in your .env or shell.` comment instead of an assert. Do NOT fabricate provider keys. |\n\n   **Emit section 1 by reading the shipped template at `\u003Cthis-skill-dir>\u002Fscripts\u002Fenv_setup_template.py` and substituting two placeholders.** The template ships alongside this SKILL.md (`cp -r` install handles it) and is the canonical source for the loader + assert shape — do **not** re-derive it inline. Read the file, perform the substitutions below, and emit the result verbatim as section 1 of the generated experiment file.\n\n   | Placeholder | Replacement | Example |\n   |---|---|---|\n   | `{{ENV_FILE_OVERRIDE_LIST}}` | A Python list literal of absolute paths from `--env-file` (repeatable; empty list if not passed) | `[]` or `[\"\u002FUsers\u002Fme\u002F.config\u002Fdd\u002Fstaging.env\"]` |\n   | `{{PROVIDER_ASSERTS}}` | Zero or more `assert os.getenv(...)` lines derived from what step 2.5 wired — per the SDK-to-key table above. One line per required provider key. Empty string if the task uses LiteLLM (which auto-routes) or a custom\u002Funrecognized SDK. | `assert os.getenv(\"ANTHROPIC_API_KEY\"), \"ANTHROPIC_API_KEY is required for the wired task_fn.\"` |\n\n   The exact assert line(s) to substitute into `{{PROVIDER_ASSERTS}}` live in each provider's reference file under the `## {{PROVIDER_ASSERTS}} substitution` heading. Read only the file that matches the detected SDK from the table above.\n\n   The template's discovery walk (env file → script dir → cwd → parent dirs → `~\u002F.datadog\u002Fcredentials`) and shell-overrides-file contract are stable — they live in the template, not in this skill body. If you need to change discovery semantics, edit `scripts\u002Fenv_setup_template.py` directly; do not re-derive them in-prose here.\n\n3. **Pick evaluator template** based on `--evaluator-style` AND the purpose from step 2.4.\n\n   `--evaluator-style` decides the **surface** (function \u002F class \u002F remote — see `references\u002Fevaluator-styles\u002F\u003Cstyle>.md` for the syntax shape). The purpose decides the **semantics** — what each evaluator actually measures. Both inputs apply together.\n\n   Default shape (purpose: accuracy \u002F unspecified):\n   - `function`: 3 plain functions — one trivial boolean (`exact_match`-style, bare `bool` OK), one richer rule-based check returning `EvaluatorResult` with `reasoning` + `assessment`, and one LLM-as-Judge surrogate. If `--dataset` had structured `expected_output`, add a JSON-shape check (also returning `EvaluatorResult`).\n   - `class`: 2 `BaseEvaluator` subclasses with `evaluate(self, context: EvaluatorContext) -> EvaluatorResult`. Always return `EvaluatorResult` (never a bare value) — state-bearing evaluators have richer signal to surface.\n   - `remote`: 1-2 `RemoteEvaluator(eval_name=...)` instances with a comment instructing the user to create the judge in the Datadog UI first.\n\n   **Purpose-driven amendments** (read the purpose string from step 2.4 and adjust the default seed accordingly — use natural-language judgment, not a hardcoded lookup):\n\n   - If the purpose mentions **tool calls \u002F agent routing \u002F tool selection**: replace the LLM-as-Judge surrogate with a `tool_calls_correct` LLMJudge whose rubric checks the right-tool-with-right-args question. The rubric should reference `output_data[\"tool_calls\"]` if the wrapper from 2.5d captured them; otherwise check the textual output for the tool name.\n   - If the purpose mentions **groundedness \u002F faithfulness \u002F RAG**: add a `groundedness` LLMJudge that takes `input_data` (the question + retrieved docs) and `output_data` (the answer) and checks that every claim in the answer traces to the docs. Make the rubric explicit about hallucinations being a fail.\n   - If the purpose mentions **structured output \u002F JSON \u002F SQL \u002F schema**: lean into `JSONEvaluator(required_keys=[...])` or `RegexMatchEvaluator(pattern=r\"...\")` from the built-in set. Add a richer `EvaluatorResult` check for semantic schema-fit if the user's structured output has constraints beyond shape (e.g. enum values, numeric ranges).\n   - If the purpose mentions **regression \u002F refactor \u002F preserve behavior**: prefer `exact_match` as the primary signal, plus a near-match check (`difflib.SequenceMatcher.ratio()` ≥ 0.95 returning `EvaluatorResult` with assessment=\"partial\" between 0.7 and 0.95). Drop the LLMJudge surrogate — regression tests want determinism, not model judgment.\n   - For purposes the SKILL.md doesn't recognize verbatim: read the user's purpose string, reason about what evaluator shape would best measure it, and write the evaluator from scratch. Cite the purpose in a comment above the evaluator so the user can see the link. Never invent a `RemoteEvaluator(eval_name=...)` for a name that may not exist in their Datadog org — when in doubt emit an LLMJudge with an inline rubric instead.\n\n   Whatever you emit, the **count of evaluators stays 2–3** to keep the generated file lean. The user can add more after the first run.\n\n   **In all styles**: any evaluator with non-trivial logic must return `EvaluatorResult` populating at minimum `value` + `reasoning` + `assessment` (see the \"Return `EvaluatorResult`, not bare values\" section). The compare UI uses `reasoning` for per-record drill-downs and `assessment` to determine whether a metric trend is an improvement.\n\n4. **Emit the file**.\n\n   **For `.py`** — single file, one blank line between sections, banner comments like:\n   ```python\n   # ─── 3. Dataset ───────────────────────────────────────────────────────────────\n   ```\n   Use `from __future__ import annotations` and `from typing import Any, Dict` at the top. Type-hint task and evaluator function signatures.\n\n   **For `.ipynb`** — valid Jupyter notebook JSON. Schema:\n   ```json\n   {\n     \"cells\": [\n       {\"cell_type\": \"markdown\", \"metadata\": {}, \"source\": [\"## 1. Env setup\\n\", \"...\"]},\n       {\"cell_type\": \"code\", \"execution_count\": null, \"metadata\": {}, \"outputs\": [], \"source\": [\"...\"]},\n       ...\n     ],\n     \"metadata\": {\n       \"kernelspec\": {\"display_name\": \"Python 3\", \"language\": \"python\", \"name\": \"python3\"},\n       \"language_info\": {\"name\": \"python\", \"version\": \"3.10\"}\n     },\n     \"nbformat\": 4,\n     \"nbformat_minor\": 5\n   }\n   ```\n   One markdown cell + one code cell per section. Keep each code cell self-contained enough that re-running it in isolation makes sense.\n\n5. **Best-effort syntax check** via Bash. Don't fail the skill if the toolchain is missing — just report.\n   - `.py`: `python -m py_compile \u003Cpath>`\n   - `.ipynb`: `python -c \"import json; nb = json.load(open('\u003Cpath>')); assert nb.get('cells'); print(f'cells={len(nb[\\\"cells\\\"])}')\"`\n\n6. **Print next-steps** (see Output section).\n\n---\n\n## What the Generated Code MUST NOT Do\n\nA reviewer should be able to run these `grep` checks against the generated file and get zero matches:\n\n| `grep` pattern | Why it's wrong |\n|---|---|\n| `uuid4`, `uuid.uuid4` | `record_id` is minted by the SDK on `dataset.append()`; never client-generate. |\n| `PATCH `, `batch_update`, `records\u002Fupload` | Status state machine and dataset diff are SDK responsibilities. |\n| `from ddtrace.llmobs._` | Private import paths. Always use `from ddtrace.llmobs import ...`. |\n| `\"record_id\"`, `\"canonical_id\"` (as dict keys in records) | The SDK owns them. |\n| `DD_API_KEY = \"\u003Cactual key>\"` | Always read from `os.environ`. |\n| `requests.post`, `httpx.post` | The skill produces SDK-only code. Direct HTTP calls bypass the SDK's lazy creation, push-diff, and bulk-threshold handling. |\n\nIf any of those slip into the output, the skill is wrong — re-emit.\n\n---\n\n## Output\n\nAfter writing, print:\n\n```\nGenerated SDK experiment: \u003Cformat>\nPath: \u003Cpath>\nLines: \u003Ccount>   (or Cells: \u003Ccount> for .ipynb)\n\nPurpose:\n  \"\u003Cresolved purpose string from step 2.4>\"\n  (sourced via: --purpose flag | extracted from invocation message | AskUserQuestion default | AskUserQuestion + free text)\n\nSDK calls used:\n  ✓ LLMObs.enable(...)                       (line\u002Fcell ~\u003CN>)\n  ✓ LLMObs.\u003Ccreate_dataset|create_dataset_from_csv|pull_dataset>(...)  (line\u002Fcell ~\u003CN>)\n  ✓ task_fn(input_data, config)              (line\u002Fcell ~\u003CN>)\n  ✓ \u003CN> evaluators (style: \u003Cfunction|class|remote>, semantics seeded by purpose)\n  ✓ LLMObs.experiment(...).run(jobs=\u003CN>)     (line\u002Fcell ~\u003CN>)\n  ✓ Provenance (in config + tags): generated_by=claude-code, skill=agent-observability-experiment-py-bootstrap, purpose=...\n\nTask function source:\n  ✓ Wired to: \u003Cmodule.path>:\u003Cfunction_name>   (source: \u003Cfile>:\u003Cline>)\n  ✓ Adapter: \u003Cone line describing the input_data → call shape mapping>\n  ✓ Return shape: \u003Cplain string | {output, tool_calls} | {answer, retrieved_docs} | etc.>\n  ✓ Sync\u002Fasync: \u003Csync | async (wrapped with asyncio.run)>\n  [WARNING lines from the side-effect scan in Workflow step 2.5d, if any]\n  [Note line if the purpose requested richer return shape but the function only exposes a string]\n\n(If --placeholder-task was used or introspection found nothing:)\nTask function source:\n  ⚠ Placeholder task emitted (no real LLM call site found \u002F opted out).\n    Replace `task_fn` with your actual LLM call before running.\n\nSyntax check: \u003Cpass | skipped: toolchain missing | fail with details>\n\nInstall:\n  pip install \"ddtrace>=4.7\" \u003Cprovider-sdk-if-needed>\n  # python-dotenv is NOT required — the generated file ships its own .env loader.\n\nCredentials:\n  The generated file auto-discovers .env files at runtime. Discovery order\n  (first non-empty value wins per key; shell env always overrides files):\n    1. --env-file path baked in as ENV_FILE_OVERRIDE (if --env-file was passed)\n    2. \u003Coutput-file's-directory>\u002F.env\n    3. \u003Coutput-file's-directory>\u002F.env.local\n    4. \u003Ccwd>\u002F.env  and  \u003Ccwd>\u002F.env.local\n    5. parent-walk from cwd up to \u002F\n    6. ~\u002F.datadog\u002Fcredentials\n\n  Drop a .env at any of those locations with at minimum:\n    DD_API_KEY=...\n    DD_APPLICATION_KEY=...\n    DD_SITE=datadoghq.com           # only if not the US1 prod site\n    \u003CPROVIDER>_API_KEY=...           # the provider key the wired task needs\n  Or override on a per-run basis by exporting them in your shell — the loader\n  never overwrites a value that is already in os.environ.\n\nRun:\n  python \u003Cpath>                  # for --format py\n  jupyter notebook \u003Cpath>        # for --format ipynb\n\nNext steps:\n1. Confirm the wired task_fn matches the entry point you want to evaluate (see \"Task function source\" above).\n   Edit the import in section 4 of the generated file if you'd rather inline the call or pick a different function.\n2. Confirm the purpose (\"\u003Cpurpose string>\") matches what you actually want to measure — section 0 of the\n   file documents it, section 5's evaluators were seeded against it, and section 6's experiment carries\n   it as a config tag. Re-run this skill with `--purpose \"\u003Cnew text>\"` to regenerate against a different\n   target without changing your code.\n3. Adjust the evaluators if needed (or wire up RemoteEvaluator names you created in the Datadog UI).\n4. Run it. The script prints experiment.url at the end.\n5. Watch the experiment: https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Fexperiments\n```\n\n---\n\n## Reference Notebook Patterns (use as templates)\n\nThe canonical set lives at \u003Chttps:\u002F\u002Fgithub.com\u002FDataDog\u002Fllm-observability\u002Ftree\u002Fmain\u002Fexperiments\u002Fnotebooks> and serves as the style reference — the generated code should feel like it could have come from this set.\n\n| Notebook | Pattern demonstrated |\n|---|---|\n| `00-basic-datasets.ipynb` | Dataset create\u002Fappend\u002Fpush lifecycle |\n| `01-basic-experiments.ipynb` | Minimum viable experiment — inline records, OpenAI task, 2 boolean evaluators |\n| `02-extra-data.ipynb` | CSV-loaded dataset, multi-value task output, confidence-based evaluators |\n| `04-multi-span-experiments.ipynb` | Two-step LLM pipelines inside a single `task_fn` |\n| `07-remote-evaluators.ipynb` | `RemoteEvaluator` with custom `transform_fn` |\n\nWhen `--evaluator-style remote`, lean toward the `07` style. When `--dataset` is a CSV, lean toward `02`. Default (no `--dataset`, `--evaluator-style function`) is the `01` style.\n\n---\n\n## Datadog Documentation\n\nThese are the canonical reference pages on \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002F>. Use them to ground answers about Agent Observability features and to look up details that aren't covered in this skill.\n\n| Topic | URL | Use when |\n|---|---|---|\n| Agent Observability overview | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002F> | Establishing what the product covers, terminology |\n| Setup | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fsetup\u002F> | API\u002Fapp key creation, project + ml_app setup, region\u002Fsite selection |\n| Instrumentation overview | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Finstrumentation\u002F> | Auto-instrumentation, manual SDK usage, span model |\n| Python SDK reference | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Finstrumentation\u002Fsdk\u002F> | Public symbol list, decorator semantics, span kinds, annotate\u002Fenable signatures |\n| Experiments | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fexperiments\u002F> | `LLMObs.experiment(...)`, dataset lifecycle, eval streaming, status states |\n| Evaluations | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002F> | Evaluator concepts, managed vs custom evaluators |\n| Custom LLM-as-a-judge evaluations | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fcustom_llm_as_a_judge_evaluations\u002F> | `RemoteEvaluator` payload shape and rubric design |\n| Managed evaluations | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fmanaged_evaluations\u002F> | Pre-built judges (faithfulness, toxicity, etc.) |\n| Monitoring | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fmonitoring\u002F> | Alerts, dashboards, span-level monitors |\n| Terms \u002F glossary | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fterms\u002F> | Span kinds, sessions, traces, ml_app |\n| Evaluation developer guide | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fguide\u002Fevaluation_developer_guide\u002F> | Writing offline evaluators, validation strategy |\n| Claude Code skills guide | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fguide\u002Fclaude_code_skills\u002F> | How this skill fits alongside the rest of the `agent-observability` set |\n| MCP server | \u003Chttps:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fmcp_server\u002F> | Connecting MCP-compatible clients to Agent Observability data |\n| Reference notebooks (GitHub) | \u003Chttps:\u002F\u002Fgithub.com\u002FDataDog\u002Fllm-observability\u002Ftree\u002Fmain\u002Fexperiments\u002Fnotebooks> | Style-of-life examples for the generated `.py` \u002F `.ipynb` |\n\n### Researching features the skill does not cover\n\nIf the user asks about an Agent Observability feature the skill's body doesn't address (e.g., specific span kinds, dataset versioning semantics, an evaluator type not covered above), fetch the relevant page from `docs.datadoghq.com` rather than guessing:\n\n1. **Pick the most specific URL** from the table above. Most Agent Observability questions resolve under `\u002Fllm_observability\u002F{experiments,evaluations,instrumentation,monitoring}\u002F`.\n2. **Use `WebFetch`** on that URL with a focused query (e.g., `\"How does Dataset.push() handle the 5 MB threshold?\"`). Prefer `WebFetch` over generic web search — the canonical page is almost always under `docs.datadoghq.com\u002Fllm_observability\u002F`.\n3. **Fall back to `WebSearch`** with `site:docs.datadoghq.com\u002Fllm_observability` if you don't know which subpage owns the topic.\n4. **Cite the page** in the answer with its URL so the user can verify and bookmark.\n\nNever invent symbols or behaviors not present in this skill body or the docs above. If the docs don't cover the question either, say so explicitly and suggest filing an issue on `DataDog\u002Fllm-observability` rather than fabricating a workaround.\n\n---\n\n## Operating Rules\n\n- **SDK only.** No `requests.post`, no manual JSON:API envelope construction, no manual ID generation. If a feature seems to require those, you're solving the wrong problem — the SDK already covers it.\n- **Public imports only.** `from ddtrace.llmobs import ...`. Never `_experiment`, `_llmobs`, or any underscore-prefixed module.\n- **Env vars, not literals.** Credentials always read from `os.environ`. The generated `main()` (or the env-setup cell) must `assert` they're set with a clear message.\n- **Auto-discover, don't push setup work onto the user.** Section 1 always emits the `_load_env_files` helper (no `python-dotenv` dependency). It walks the discovery order documented in Workflow step 2.6 and prints which file(s) it loaded. Never substitute `load_dotenv()` from the third-party `python-dotenv` package — the inline helper has zero dependencies and is identical in behavior. Shell env vars always win over file-loaded values so the user can override any auto-discovered value by `export \u003CKEY>=...` before re-running.\n- **Provider-key asserts must match the wired task.** Generate the assert for `OPENAI_API_KEY` only if the task imports \u002F calls OpenAI; same for Anthropic \u002F Gemini \u002F Bedrock \u002F etc. Per the Workflow step 2.6 table. Never emit asserts for provider keys the task does not actually need — they're confusing and cause spurious \"missing key\" failures.\n- **Always pass `site=` to `LLMObs.enable()`.** Read it from `os.getenv(\"DD_SITE\", \"datadoghq.com\")`. Omitting `site=` silently defaults to US1 prod, which breaks every non-prod org (e.g. staging `datad0g.com`, `datadoghq.eu`). The canonical signature already includes it — never drop it.\n- **Per-record `tags` are `\"key:value\"` strings.** When inlining records (whether from `--dataset` JSON, CSV, or the default sample), each entry in a record's `\"tags\"` list must be a `\"key:value\"` string like `\"env:prod\"`, `\"source:traces\"`, `\"category:geography\"`. Bare strings (`\"smoke\"`, `\"baseline\"`) trigger `ValueError: Tag '\u003Cname>' is malformed.` at `Dataset.append()` time. If the source data has bare-string tags, namespace them — e.g. wrap `\"smoke\"` as `\"tag:smoke\"` rather than dropping it.\n- **Always resolve a purpose before generating.** Step 2.4 must produce a non-empty `purpose` string before steps 2.5 \u002F 3 \u002F 4 run. If the user provides one via `--purpose` or in their invocation, use it. Otherwise prompt via `AskUserQuestion` with the 5 seed options + Other. Never skip the prompt; never proceed with a blank purpose. A weak purpose (\"test it\") is still better than no purpose — generic accuracy semantics will at least seed reasonable defaults.\n- **Treat the purpose as reasoning input, not a switch statement.** There is no hardcoded mapping from purpose strings to evaluator code or wrapper shape. Read the purpose, reason about what's being measured, and emit appropriately. The same purpose string may produce different output for two different apps (a tool-call purpose in a LangChain app generates different wrapper code than in a raw OpenAI app) — that's expected.\n- **Introspect first, placeholder last.** The default behavior is Workflow step 2.5 — scan the user's app, find the LLM entry point, wire `task_fn` to it. A `# TODO(user)` marker in the task section is only acceptable when introspection genuinely found nothing or the user passed `--placeholder-task`. Never emit a placeholder task when a real candidate exists in the project — that's the failure mode this skill exists to fix.\n- **`# TODO(user)` markers on at least one evaluator** so reviewers can't ship un-customized evaluators by accident. (Evaluators stay user-owned even when the task is auto-wired.)\n- **Introspection is bounded.** The scan in Workflow step 2.5 must respect `--app-root` (or its default-resolved value), `.gitignore` if present, and the directory blocklist (`node_modules`, `.venv`, `__pycache__`, etc.). Refuse to scan `\u002F` or `~`. If a scan would touch more than ~10k Python files, narrow the root or ask the user to point at the relevant subdirectory.\n- **Match notebook conventions.** Plain function evaluators by default; class-based only when the user opts in. Print `experiment.url` at the end of every generated file.\n- **Tag every experiment with provenance + purpose — in both `config` and `tags`.** Every `LLMObs.experiment(...)` call **must** carry `\"generated_by\": \"claude-code\"`, `\"skill\": \"agent-observability-experiment-py-bootstrap\"`, AND `\"purpose\": \"\u003Cstep 2.4 string>\"` as keys in **both** the `config={...}` dict (so they render in the experiment's Configuration view, which is where users actually look) **and** the `tags={...}` dict (which the SDK serializes into `metadata.tags` for future tag-filter consumers). The `tags=` path alone is not enough: the current LLM Experiments UI does not surface `metadata.tags` as filterable chips, so users won't see the values unless they're also in `config`. The `purpose` field is what makes future runs of the same experiment discoverable by intent — without it, users see ten experiments with cryptic names and no idea what each was testing. Also set `description=\"\u003Cpurpose>\"` on the experiment so the UI list view shows it.\n- **PII scrub at the door.** If `--dataset` is given, scrub before inlining into the generated file. Never embed a record that contains an unmasked email\u002Fphone\u002FSSN\u002FAPI-key pattern.\n- **Don't generate `requirements.txt` or `pyproject.toml`.** Print the `pip install` command in the next-steps message instead — most users already have a venv.\n- **No silent fallbacks.** If `--format` is unsupported, error out with the valid choices.\n- **Python only.** If a user passes `--language typescript` (or any non-Python language flag), error out — this skill produces Python `ddtrace.llmobs` SDK code only.\n- **Research, don't invent.** If the user asks about an Agent Observability feature, span kind, evaluator type, or SDK symbol that is not documented in this skill body, `WebFetch` the relevant `docs.datadoghq.com\u002Fllm_observability\u002F*` page (see the Datadog Documentation table above for the canonical URLs) before answering. Cite the page URL in the response. If the docs don't cover the topic, say so explicitly — never fabricate symbols, flags, or behaviors.\n",{"data":40,"body":41},{"name":4,"description":6},{"type":42,"children":43},"root",[44,53,65,149,167,216,234,298,310,314,327,369,389,395,407,412,419,440,1072,1075,1081,1101,1295,1305,1967,1970,1976,2010,2023,2073,2299,2334,2340,2367,2481,2493,2496,2502,2521,2530,2533,2539,2786,2827,3132,3157,3166,3239,3254,3263,3275,3285,3451,3468,3471,3490,3514,3663,3673,3958,3999,4060,4070,4449,4459,4478,4487,4507,4525,4564,4569,5006,5011,5214,5240,5288,5305,5569,5601,5702,5722,5743,6910,6913,6919,6932,7126,7131,7134,7140,7145,7154,7157,7163,7175,7298,7355,7358,7364,7377,7741,7747,7760,7850,7863,7866,7872,8599],{"type":45,"tag":46,"props":47,"children":49},"element","h2",{"id":48},"backend",[50],{"type":51,"value":52},"text","Backend",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57,63],{"type":45,"tag":58,"props":59,"children":60},"strong",{},[61],{"type":51,"value":62},"Detection",{"type":51,"value":64}," — At the start of every invocation, before taking any action, determine which backend (if any) to use for the startup beacon call:",{"type":45,"tag":66,"props":67,"children":68},"ol",{},[69,91,111,137],{"type":45,"tag":70,"props":71,"children":72},"li",{},[73,75,82,84,89],{"type":51,"value":74},"If the user passed ",{"type":45,"tag":76,"props":77,"children":79},"code",{"className":78},[],[80],{"type":51,"value":81},"--backend pup",{"type":51,"value":83}," anywhere in their invocation → use ",{"type":45,"tag":58,"props":85,"children":86},{},[87],{"type":51,"value":88},"pup mode",{"type":51,"value":90}," for the beacon. Skip steps 2–3.",{"type":45,"tag":70,"props":92,"children":93},{},[94,96,102,104,109],{"type":51,"value":95},"Check whether MCP tools are present in your active tool list. The canonical signal is whether ",{"type":45,"tag":76,"props":97,"children":99},{"className":98},[],[100],{"type":51,"value":101},"mcp__datadog-llmo-mcp__list_llmobs_evals",{"type":51,"value":103}," appears in your available tools. If present → use ",{"type":45,"tag":58,"props":105,"children":106},{},[107],{"type":51,"value":108},"MCP mode",{"type":51,"value":110},".",{"type":45,"tag":70,"props":112,"children":113},{},[114,116,122,124,130,132,136],{"type":51,"value":115},"If MCP tools are absent → check whether ",{"type":45,"tag":76,"props":117,"children":119},{"className":118},[],[120],{"type":51,"value":121},"pup",{"type":51,"value":123}," is executable via ",{"type":45,"tag":76,"props":125,"children":127},{"className":126},[],[128],{"type":51,"value":129},"pup --version",{"type":51,"value":131},". If responding → use ",{"type":45,"tag":58,"props":133,"children":134},{},[135],{"type":51,"value":88},{"type":51,"value":110},{"type":45,"tag":70,"props":138,"children":139},{},[140,142,147],{"type":51,"value":141},"If neither is available → ",{"type":45,"tag":58,"props":143,"children":144},{},[145],{"type":51,"value":146},"skip the beacon silently",{"type":51,"value":148}," and proceed to codegen. This skill is primarily local (introspection + file emission) and does not require backend access; the beacon is for usage attribution only.",{"type":45,"tag":54,"props":150,"children":151},{},[152,157,159,165],{"type":45,"tag":58,"props":153,"children":154},{},[155],{"type":51,"value":156},"Invocation ID:",{"type":51,"value":158}," At the very start of each invocation, before any MCP\u002Fpup call, generate an 8-character hex invocation ID (e.g., ",{"type":45,"tag":76,"props":160,"children":162},{"className":161},[],[163],{"type":51,"value":164},"3a9f1c2b",{"type":51,"value":166},"). Keep it constant for the entire invocation.",{"type":45,"tag":54,"props":168,"children":169},{},[170,175,177,183,185,191,193,198,200,206,208,214],{"type":45,"tag":58,"props":171,"children":172},{},[173],{"type":51,"value":174},"Intent tagging:",{"type":51,"value":176}," On every MCP tool call, prefix ",{"type":45,"tag":76,"props":178,"children":180},{"className":179},[],[181],{"type":51,"value":182},"telemetry.intent",{"type":51,"value":184}," with ",{"type":45,"tag":76,"props":186,"children":188},{"className":187},[],[189],{"type":51,"value":190},"skill:agent-observability-experiment-py-bootstrap[\u003Cinv_id>] — ",{"type":51,"value":192}," followed by a description of why the tool is being called. On the ",{"type":45,"tag":58,"props":194,"children":195},{},[196],{"type":51,"value":197},"first MCP tool call only",{"type":51,"value":199}," (the startup beacon below), use ",{"type":45,"tag":76,"props":201,"children":203},{"className":202},[],[204],{"type":51,"value":205},"skill:agent-observability-experiment-py-bootstrap:start[\u003Cinv_id>] — ",{"type":51,"value":207}," instead (note the ",{"type":45,"tag":76,"props":209,"children":211},{"className":210},[],[212],{"type":51,"value":213},":start",{"type":51,"value":215}," suffix).",{"type":45,"tag":54,"props":217,"children":218},{},[219,224,226,232],{"type":45,"tag":58,"props":220,"children":221},{},[222],{"type":51,"value":223},"Startup beacon:",{"type":51,"value":225}," Immediately after parsing arguments (workflow step 1, before dataset resolution in step 2), issue exactly one beacon call to register skill usage and validate backend connectivity. This is fire-and-forget — surface any error as a one-line ",{"type":45,"tag":76,"props":227,"children":229},{"className":228},[],[230],{"type":51,"value":231},"Note:",{"type":51,"value":233}," to the user but do not block codegen.",{"type":45,"tag":235,"props":236,"children":237},"ul",{},[238,262,280],{"type":45,"tag":70,"props":239,"children":240},{},[241,246,248,253,254,260],{"type":45,"tag":58,"props":242,"children":243},{},[244],{"type":51,"value":245},"MCP mode:",{"type":51,"value":247}," call ",{"type":45,"tag":76,"props":249,"children":251},{"className":250},[],[252],{"type":51,"value":101},{"type":51,"value":184},{"type":45,"tag":76,"props":255,"children":257},{"className":256},[],[258],{"type":51,"value":259},"telemetry.intent = \"skill:agent-observability-experiment-py-bootstrap:start[\u003Cinv_id>] — Skill startup: register usage and verify Datadog connectivity\"",{"type":51,"value":261},". Discard the response payload; the call's purpose is the telemetry tag.",{"type":45,"tag":70,"props":263,"children":264},{},[265,270,272,278],{"type":45,"tag":58,"props":266,"children":267},{},[268],{"type":51,"value":269},"pup mode:",{"type":51,"value":271}," run ",{"type":45,"tag":76,"props":273,"children":275},{"className":274},[],[276],{"type":51,"value":277},"pup llm-obs evals list --limit 1",{"type":51,"value":279}," via Bash. Pup carries its own telemetry; no intent prefix needed.",{"type":45,"tag":70,"props":281,"children":282},{},[283,288,290,296],{"type":45,"tag":58,"props":284,"children":285},{},[286],{"type":51,"value":287},"No backend:",{"type":51,"value":289}," print one line ",{"type":45,"tag":76,"props":291,"children":293},{"className":292},[],[294],{"type":51,"value":295},"(Telemetry beacon skipped — no Datadog backend detected; this is informational only and does not affect codegen.)",{"type":51,"value":297}," and proceed.",{"type":45,"tag":54,"props":299,"children":300},{},[301,303,308],{"type":51,"value":302},"The beacon ",{"type":45,"tag":58,"props":304,"children":305},{},[306],{"type":51,"value":307},"must not",{"type":51,"value":309}," fail the skill. If the call errors (auth, network, etc.), surface a one-line note and continue.",{"type":45,"tag":311,"props":312,"children":313},"hr",{},[],{"type":45,"tag":315,"props":316,"children":318},"h1",{"id":317},"agent-observability-experiment-python-bootstrap-generate-a-python-experiment-using-ddtracellmobs",[319,321],{"type":51,"value":320},"Agent Observability Experiment (Python) Bootstrap — Generate a Python Experiment Using ",{"type":45,"tag":76,"props":322,"children":324},{"className":323},[],[325],{"type":51,"value":326},"ddtrace.llmobs",{"type":45,"tag":54,"props":328,"children":329},{},[330,332,342,344,350,352,358,360,368],{"type":51,"value":331},"Produce a single self-contained Python experiment that uses the official ",{"type":45,"tag":58,"props":333,"children":334},{},[335,340],{"type":45,"tag":76,"props":336,"children":338},{"className":337},[],[339],{"type":51,"value":326},{"type":51,"value":341}," SDK",{"type":51,"value":343},". Output is either a ",{"type":45,"tag":76,"props":345,"children":347},{"className":346},[],[348],{"type":51,"value":349},".py",{"type":51,"value":351}," script or an ",{"type":45,"tag":76,"props":353,"children":355},{"className":354},[],[356],{"type":51,"value":357},".ipynb",{"type":51,"value":359}," notebook. The generated code mirrors the patterns shown in DataDog's reference notebooks at ",{"type":45,"tag":361,"props":362,"children":366},"a",{"href":363,"rel":364},"https:\u002F\u002Fgithub.com\u002FDataDog\u002Fllm-observability\u002Ftree\u002Fmain\u002Fexperiments\u002Fnotebooks",[365],"nofollow",[367],{"type":51,"value":363},{"type":51,"value":110},{"type":45,"tag":54,"props":370,"children":371},{},[372,374,379,381,387],{"type":51,"value":373},"The SDK handles lazy project\u002Fexperiment creation, dataset push diffing, the 5 MB \u002F 1000-record bulk threshold, eval metric streaming, and the status state machine on the user's behalf. This skill must therefore ",{"type":45,"tag":58,"props":375,"children":376},{},[377],{"type":51,"value":378},"never re-implement those primitives",{"type":51,"value":380}," — it just imports ",{"type":45,"tag":76,"props":382,"children":384},{"className":383},[],[385],{"type":51,"value":386},"LLMObs",{"type":51,"value":388}," and trusts it.",{"type":45,"tag":46,"props":390,"children":392},{"id":391},"usage",[393],{"type":51,"value":394},"Usage",{"type":45,"tag":396,"props":397,"children":401},"pre",{"className":398,"code":400,"language":51},[399],"language-text","\u002Fagent-observability-experiment-py-bootstrap [--purpose \u003Cfree text>] [--format py|ipynb] [--dataset \u003Cpath>] [--dataset-name \u003Cname>] [--dataset-version \u003Cint>] [--project-name \u003Cname>] [--evaluator-style function|class|remote] [--jobs \u003Cn>] [--output \u003Cpath>] [--task-source \u003Cmodule:function>] [--placeholder-task] [--app-root \u003Cpath>] [--env-file \u003Cpath>]\n",[402],{"type":45,"tag":76,"props":403,"children":405},{"__ignoreMap":404},"",[406],{"type":51,"value":400},{"type":45,"tag":54,"props":408,"children":409},{},[410],{"type":51,"value":411},"Arguments: $ARGUMENTS",{"type":45,"tag":413,"props":414,"children":416},"h3",{"id":415},"inputs",[417],{"type":51,"value":418},"Inputs",{"type":45,"tag":54,"props":420,"children":421},{},[422,424,430,432,438],{"type":51,"value":423},"All inputs are optional. If the user omits a flag, fall back to the default — never block on prompting for ",{"type":45,"tag":76,"props":425,"children":427},{"className":426},[],[428],{"type":51,"value":429},"--jobs",{"type":51,"value":431},", ",{"type":45,"tag":76,"props":433,"children":435},{"className":434},[],[436],{"type":51,"value":437},"--format",{"type":51,"value":439},", etc.",{"type":45,"tag":441,"props":442,"children":443},"table",{},[444,468],{"type":45,"tag":445,"props":446,"children":447},"thead",{},[448],{"type":45,"tag":449,"props":450,"children":451},"tr",{},[452,458,463],{"type":45,"tag":453,"props":454,"children":455},"th",{},[456],{"type":51,"value":457},"Input",{"type":45,"tag":453,"props":459,"children":460},{},[461],{"type":51,"value":462},"Default",{"type":45,"tag":453,"props":464,"children":465},{},[466],{"type":51,"value":467},"Description",{"type":45,"tag":469,"props":470,"children":471},"tbody",{},[472,518,579,614,658,702,764,796,842,880,910,968,1020],{"type":45,"tag":449,"props":473,"children":474},{},[475,484,493],{"type":45,"tag":476,"props":477,"children":478},"td",{},[479],{"type":45,"tag":76,"props":480,"children":482},{"className":481},[],[483],{"type":51,"value":437},{"type":45,"tag":476,"props":485,"children":486},{},[487],{"type":45,"tag":76,"props":488,"children":490},{"className":489},[],[491],{"type":51,"value":492},"py",{"type":45,"tag":476,"props":494,"children":495},{},[496,501,503,508,510,516],{"type":45,"tag":76,"props":497,"children":499},{"className":498},[],[500],{"type":51,"value":492},{"type":51,"value":502}," (single ",{"type":45,"tag":76,"props":504,"children":506},{"className":505},[],[507],{"type":51,"value":349},{"type":51,"value":509}," file) or ",{"type":45,"tag":76,"props":511,"children":513},{"className":512},[],[514],{"type":51,"value":515},"ipynb",{"type":51,"value":517}," (Jupyter notebook with one cell per section).",{"type":45,"tag":449,"props":519,"children":520},{},[521,530,543],{"type":45,"tag":476,"props":522,"children":523},{},[524],{"type":45,"tag":76,"props":525,"children":527},{"className":526},[],[528],{"type":51,"value":529},"--dataset",{"type":45,"tag":476,"props":531,"children":532},{},[533,535,541],{"type":51,"value":534},"none — emit a sample 3-record ",{"type":45,"tag":76,"props":536,"children":538},{"className":537},[],[539],{"type":51,"value":540},"records=[...]",{"type":51,"value":542}," inline so the file is runnable as-is",{"type":45,"tag":476,"props":544,"children":545},{},[546,548,554,556,562,564,570,572,578],{"type":51,"value":547},"Path to a local ",{"type":45,"tag":76,"props":549,"children":551},{"className":550},[],[552],{"type":51,"value":553},"DatasetRecordRaw[]",{"type":51,"value":555}," JSON or CSV. JSON → ",{"type":45,"tag":76,"props":557,"children":559},{"className":558},[],[560],{"type":51,"value":561},"create_dataset(records=...)",{"type":51,"value":563},"; CSV → ",{"type":45,"tag":76,"props":565,"children":567},{"className":566},[],[568],{"type":51,"value":569},"create_dataset_from_csv(...)",{"type":51,"value":571},". Mutually exclusive with ",{"type":45,"tag":76,"props":573,"children":575},{"className":574},[],[576],{"type":51,"value":577},"--dataset-name",{"type":51,"value":110},{"type":45,"tag":449,"props":580,"children":581},{},[582,590,595],{"type":45,"tag":476,"props":583,"children":584},{},[585],{"type":45,"tag":76,"props":586,"children":588},{"className":587},[],[589],{"type":51,"value":577},{"type":45,"tag":476,"props":591,"children":592},{},[593],{"type":51,"value":594},"none",{"type":45,"tag":476,"props":596,"children":597},{},[598,600,606,608,613],{"type":51,"value":599},"Name of an existing Datadog dataset to fetch at runtime via ",{"type":45,"tag":76,"props":601,"children":603},{"className":602},[],[604],{"type":51,"value":605},"LLMObs.pull_dataset(...)",{"type":51,"value":607},". Use this when the dataset already lives in Datadog (e.g. created in the UI or by a prior run) — no local file required. Mutually exclusive with ",{"type":45,"tag":76,"props":609,"children":611},{"className":610},[],[612],{"type":51,"value":529},{"type":51,"value":110},{"type":45,"tag":449,"props":615,"children":616},{},[617,626,631],{"type":45,"tag":476,"props":618,"children":619},{},[620],{"type":45,"tag":76,"props":621,"children":623},{"className":622},[],[624],{"type":51,"value":625},"--dataset-version",{"type":45,"tag":476,"props":627,"children":628},{},[629],{"type":51,"value":630},"none (latest)",{"type":45,"tag":476,"props":632,"children":633},{},[634,636,641,643,649,651,656],{"type":51,"value":635},"Pin to a specific dataset version when using ",{"type":45,"tag":76,"props":637,"children":639},{"className":638},[],[640],{"type":51,"value":577},{"type":51,"value":642},". Passed through as ",{"type":45,"tag":76,"props":644,"children":646},{"className":645},[],[647],{"type":51,"value":648},"pull_dataset(version=N)",{"type":51,"value":650},". Ignored if ",{"type":45,"tag":76,"props":652,"children":654},{"className":653},[],[655],{"type":51,"value":577},{"type":51,"value":657}," is not set.",{"type":45,"tag":449,"props":659,"children":660},{},[661,670,689],{"type":45,"tag":476,"props":662,"children":663},{},[664],{"type":45,"tag":76,"props":665,"children":667},{"className":666},[],[668],{"type":51,"value":669},"--project-name",{"type":45,"tag":476,"props":671,"children":672},{},[673,679,681,687],{"type":45,"tag":76,"props":674,"children":676},{"className":675},[],[677],{"type":51,"value":678},"experiment-\u003Cservice-name>",{"type":51,"value":680}," — derived from the codebase (see Workflow step 1); falls back to ",{"type":45,"tag":76,"props":682,"children":684},{"className":683},[],[685],{"type":51,"value":686},"experiment-sdk-default",{"type":51,"value":688}," only if nothing resolves",{"type":45,"tag":476,"props":690,"children":691},{},[692,694,700],{"type":51,"value":693},"Datadog project name (visible in the LLM Experiments UI). The SDK's ",{"type":45,"tag":76,"props":695,"children":697},{"className":696},[],[698],{"type":51,"value":699},"ml_app",{"type":51,"value":701}," tag falls back to this automatically — no separate flag needed.",{"type":45,"tag":449,"props":703,"children":704},{},[705,714,723],{"type":45,"tag":476,"props":706,"children":707},{},[708],{"type":45,"tag":76,"props":709,"children":711},{"className":710},[],[712],{"type":51,"value":713},"--evaluator-style",{"type":45,"tag":476,"props":715,"children":716},{},[717],{"type":45,"tag":76,"props":718,"children":720},{"className":719},[],[721],{"type":51,"value":722},"function",{"type":45,"tag":476,"props":724,"children":725},{},[726,731,733,739,741,747,749,755,756,762],{"type":45,"tag":76,"props":727,"children":729},{"className":728},[],[730],{"type":51,"value":722},{"type":51,"value":732}," (plain functions — notebook default), ",{"type":45,"tag":76,"props":734,"children":736},{"className":735},[],[737],{"type":51,"value":738},"class",{"type":51,"value":740}," (",{"type":45,"tag":76,"props":742,"children":744},{"className":743},[],[745],{"type":51,"value":746},"BaseEvaluator",{"type":51,"value":748}," subclasses), or ",{"type":45,"tag":76,"props":750,"children":752},{"className":751},[],[753],{"type":51,"value":754},"remote",{"type":51,"value":740},{"type":45,"tag":76,"props":757,"children":759},{"className":758},[],[760],{"type":51,"value":761},"RemoteEvaluator",{"type":51,"value":763}," instances).",{"type":45,"tag":449,"props":765,"children":766},{},[767,775,784],{"type":45,"tag":476,"props":768,"children":769},{},[770],{"type":45,"tag":76,"props":771,"children":773},{"className":772},[],[774],{"type":51,"value":429},{"type":45,"tag":476,"props":776,"children":777},{},[778],{"type":45,"tag":76,"props":779,"children":781},{"className":780},[],[782],{"type":51,"value":783},"10",{"type":45,"tag":476,"props":785,"children":786},{},[787,789,795],{"type":51,"value":788},"Passed to ",{"type":45,"tag":76,"props":790,"children":792},{"className":791},[],[793],{"type":51,"value":794},"experiment.run(jobs=N)",{"type":51,"value":110},{"type":45,"tag":449,"props":797,"children":798},{},[799,808,817],{"type":45,"tag":476,"props":800,"children":801},{},[802],{"type":45,"tag":76,"props":803,"children":805},{"className":804},[],[806],{"type":51,"value":807},"--output",{"type":45,"tag":476,"props":809,"children":810},{},[811],{"type":45,"tag":76,"props":812,"children":814},{"className":813},[],[815],{"type":51,"value":816},".\u002Fexperiments\u002Fexperiment.\u003Cext>",{"type":45,"tag":476,"props":818,"children":819},{},[820,822,827,829,834,836,841],{"type":51,"value":821},"File extension derives from ",{"type":45,"tag":76,"props":823,"children":825},{"className":824},[],[826],{"type":51,"value":437},{"type":51,"value":828},": ",{"type":45,"tag":76,"props":830,"children":832},{"className":831},[],[833],{"type":51,"value":349},{"type":51,"value":835}," or ",{"type":45,"tag":76,"props":837,"children":839},{"className":838},[],[840],{"type":51,"value":357},{"type":51,"value":110},{"type":45,"tag":449,"props":843,"children":844},{},[845,854,859],{"type":45,"tag":476,"props":846,"children":847},{},[848],{"type":45,"tag":76,"props":849,"children":851},{"className":850},[],[852],{"type":51,"value":853},"--task-source",{"type":45,"tag":476,"props":855,"children":856},{},[857],{"type":51,"value":858},"auto — discovered by application introspection (see Workflow step 2.5)",{"type":45,"tag":476,"props":860,"children":861},{},[862,864,870,872,878],{"type":51,"value":863},"Explicit override: ",{"type":45,"tag":76,"props":865,"children":867},{"className":866},[],[868],{"type":51,"value":869},"\u003Cdotted.module.path>:\u003Cfunction_name>",{"type":51,"value":871}," for the function to wrap as ",{"type":45,"tag":76,"props":873,"children":875},{"className":874},[],[876],{"type":51,"value":877},"task_fn",{"type":51,"value":879},". Use when you already know the entry point and want to skip the introspection scan.",{"type":45,"tag":449,"props":881,"children":882},{},[883,892,897],{"type":45,"tag":476,"props":884,"children":885},{},[886],{"type":45,"tag":76,"props":887,"children":889},{"className":888},[],[890],{"type":51,"value":891},"--placeholder-task",{"type":45,"tag":476,"props":893,"children":894},{},[895],{"type":51,"value":896},"off",{"type":45,"tag":476,"props":898,"children":899},{},[900,902,908],{"type":51,"value":901},"Opt out of application introspection and emit the generic ",{"type":45,"tag":76,"props":903,"children":905},{"className":904},[],[906],{"type":51,"value":907},"# TODO(user)",{"type":51,"value":909}," placeholder task. Use when scaffolding without a real app, in tests, or when the user explicitly wants to fill in the task themselves.",{"type":45,"tag":449,"props":911,"children":912},{},[913,922,950],{"type":45,"tag":476,"props":914,"children":915},{},[916],{"type":45,"tag":76,"props":917,"children":919},{"className":918},[],[920],{"type":51,"value":921},"--app-root",{"type":45,"tag":476,"props":923,"children":924},{},[925,927,933,935,941,942,948],{"type":51,"value":926},"resolved from ",{"type":45,"tag":76,"props":928,"children":930},{"className":929},[],[931],{"type":51,"value":932},"pyproject.toml",{"type":51,"value":934}," \u002F ",{"type":45,"tag":76,"props":936,"children":938},{"className":937},[],[939],{"type":51,"value":940},"setup.cfg",{"type":51,"value":934},{"type":45,"tag":76,"props":943,"children":945},{"className":944},[],[946],{"type":51,"value":947},"setup.py",{"type":51,"value":949}," \u002F cwd",{"type":45,"tag":476,"props":951,"children":952},{},[953,955,960,961,966],{"type":51,"value":954},"Root directory the introspection scan is restricted to. Skipped if ",{"type":45,"tag":76,"props":956,"children":958},{"className":957},[],[959],{"type":51,"value":891},{"type":51,"value":835},{"type":45,"tag":76,"props":962,"children":964},{"className":963},[],[965],{"type":51,"value":853},{"type":51,"value":967}," is set.",{"type":45,"tag":449,"props":969,"children":970},{},[971,980,993],{"type":45,"tag":476,"props":972,"children":973},{},[974],{"type":45,"tag":76,"props":975,"children":977},{"className":976},[],[978],{"type":51,"value":979},"--env-file",{"type":45,"tag":476,"props":981,"children":982},{},[983,985,991],{"type":51,"value":984},"none — generated file auto-discovers ",{"type":45,"tag":76,"props":986,"children":988},{"className":987},[],[989],{"type":51,"value":990},".env",{"type":51,"value":992}," files at runtime (see Workflow step 4, section 1)",{"type":45,"tag":476,"props":994,"children":995},{},[996,998,1003,1005,1010,1012,1018],{"type":51,"value":997},"Explicit absolute path to a ",{"type":45,"tag":76,"props":999,"children":1001},{"className":1000},[],[1002],{"type":51,"value":990},{"type":51,"value":1004},"-style file. Generated code preloads this path ",{"type":45,"tag":58,"props":1006,"children":1007},{},[1008],{"type":51,"value":1009},"first",{"type":51,"value":1011}," before the auto-discovery walk. Use when your credentials live in a non-standard location (e.g. ",{"type":45,"tag":76,"props":1013,"children":1015},{"className":1014},[],[1016],{"type":51,"value":1017},"~\u002F.config\u002Fdd\u002Fstaging.env",{"type":51,"value":1019},").",{"type":45,"tag":449,"props":1021,"children":1022},{},[1023,1032,1045],{"type":45,"tag":476,"props":1024,"children":1025},{},[1026],{"type":45,"tag":76,"props":1027,"children":1029},{"className":1028},[],[1030],{"type":51,"value":1031},"--purpose",{"type":45,"tag":476,"props":1033,"children":1034},{},[1035,1037,1043],{"type":51,"value":1036},"auto — prompted via ",{"type":45,"tag":76,"props":1038,"children":1040},{"className":1039},[],[1041],{"type":51,"value":1042},"AskUserQuestion",{"type":51,"value":1044}," in Workflow step 2.0 if not set or inferable from the invocation message",{"type":45,"tag":476,"props":1046,"children":1047},{},[1048,1050,1056,1057,1063,1064,1070],{"type":51,"value":1049},"Free-form string describing what the experiment is meant to validate (e.g. ",{"type":45,"tag":76,"props":1051,"children":1053},{"className":1052},[],[1054],{"type":51,"value":1055},"\"test that the agent picks the right tool for ambiguous user requests\"",{"type":51,"value":431},{"type":45,"tag":76,"props":1058,"children":1060},{"className":1059},[],[1061],{"type":51,"value":1062},"\"verify SQL output always parses\"",{"type":51,"value":431},{"type":45,"tag":76,"props":1065,"children":1067},{"className":1066},[],[1068],{"type":51,"value":1069},"\"regression-test prompt v3 against prod baseline\"",{"type":51,"value":1071},"). Used as reasoning context — biases candidate ranking in Workflow step 2.5, shapes the wrapper return type in 2.5d, seeds evaluator selection in step 3, and is embedded in the generated file's header comment. NOT a fixed taxonomy — Claude reads the string and decides effects dynamically per invocation.",{"type":45,"tag":311,"props":1073,"children":1074},{},[],{"type":45,"tag":46,"props":1076,"children":1078},{"id":1077},"sdk-surface-cited",[1079],{"type":51,"value":1080},"SDK Surface (Cited)",{"type":45,"tag":54,"props":1082,"children":1083},{},[1084,1086,1091,1093,1099],{"type":51,"value":1085},"These are the public symbols the generated code uses. All come from ",{"type":45,"tag":76,"props":1087,"children":1089},{"className":1088},[],[1090],{"type":51,"value":326},{"type":51,"value":1092}," (the public package — never from ",{"type":45,"tag":76,"props":1094,"children":1096},{"className":1095},[],[1097],{"type":51,"value":1098},"ddtrace.llmobs._experiment",{"type":51,"value":1100}," or other underscore-prefixed modules).",{"type":45,"tag":441,"props":1102,"children":1103},{},[1104,1125],{"type":45,"tag":445,"props":1105,"children":1106},{},[1107],{"type":45,"tag":449,"props":1108,"children":1109},{},[1110,1115,1120],{"type":45,"tag":453,"props":1111,"children":1112},{},[1113],{"type":51,"value":1114},"Import",{"type":45,"tag":453,"props":1116,"children":1117},{},[1118],{"type":51,"value":1119},"Source",{"type":45,"tag":453,"props":1121,"children":1122},{},[1123],{"type":51,"value":1124},"What it gives you",{"type":45,"tag":469,"props":1126,"children":1127},{},[1128,1200,1237,1268],{"type":45,"tag":449,"props":1129,"children":1130},{},[1131,1139,1156],{"type":45,"tag":476,"props":1132,"children":1133},{},[1134],{"type":45,"tag":76,"props":1135,"children":1137},{"className":1136},[],[1138],{"type":51,"value":386},{"type":45,"tag":476,"props":1140,"children":1141},{},[1142,1148,1150],{"type":45,"tag":76,"props":1143,"children":1145},{"className":1144},[],[1146],{"type":51,"value":1147},"ddtrace\u002Fllmobs\u002F__init__.py",{"type":51,"value":1149}," re-exports ",{"type":45,"tag":76,"props":1151,"children":1153},{"className":1152},[],[1154],{"type":51,"value":1155},"_llmobs.py",{"type":45,"tag":476,"props":1157,"children":1158},{},[1159,1165,1166,1172,1173,1179,1180,1186,1187,1193,1194],{"type":45,"tag":76,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":51,"value":1164},".enable()",{"type":51,"value":431},{"type":45,"tag":76,"props":1167,"children":1169},{"className":1168},[],[1170],{"type":51,"value":1171},".create_dataset()",{"type":51,"value":431},{"type":45,"tag":76,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":51,"value":1178},".create_dataset_from_csv()",{"type":51,"value":431},{"type":45,"tag":76,"props":1181,"children":1183},{"className":1182},[],[1184],{"type":51,"value":1185},".pull_dataset(dataset_name, project_name, version)",{"type":51,"value":431},{"type":45,"tag":76,"props":1188,"children":1190},{"className":1189},[],[1191],{"type":51,"value":1192},".experiment()",{"type":51,"value":431},{"type":45,"tag":76,"props":1195,"children":1197},{"className":1196},[],[1198],{"type":51,"value":1199},".async_experiment()",{"type":45,"tag":449,"props":1201,"children":1202},{},[1203,1218,1226],{"type":45,"tag":476,"props":1204,"children":1205},{},[1206,1211,1212],{"type":45,"tag":76,"props":1207,"children":1209},{"className":1208},[],[1210],{"type":51,"value":761},{"type":51,"value":431},{"type":45,"tag":76,"props":1213,"children":1215},{"className":1214},[],[1216],{"type":51,"value":1217},"EvaluatorContext",{"type":45,"tag":476,"props":1219,"children":1220},{},[1221],{"type":45,"tag":76,"props":1222,"children":1224},{"className":1223},[],[1225],{"type":51,"value":1147},{"type":45,"tag":476,"props":1227,"children":1228},{},[1229,1231],{"type":51,"value":1230},"LLM-as-Judge that runs server-side; preferred over inline ",{"type":45,"tag":76,"props":1232,"children":1234},{"className":1233},[],[1235],{"type":51,"value":1236},"LLMJudge",{"type":45,"tag":449,"props":1238,"children":1239},{},[1240,1255,1263],{"type":45,"tag":476,"props":1241,"children":1242},{},[1243,1248,1249],{"type":45,"tag":76,"props":1244,"children":1246},{"className":1245},[],[1247],{"type":51,"value":746},{"type":51,"value":431},{"type":45,"tag":76,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":51,"value":1254},"EvaluatorResult",{"type":45,"tag":476,"props":1256,"children":1257},{},[1258],{"type":45,"tag":76,"props":1259,"children":1261},{"className":1260},[],[1262],{"type":51,"value":1147},{"type":45,"tag":476,"props":1264,"children":1265},{},[1266],{"type":51,"value":1267},"Class-based evaluator path (advanced)",{"type":45,"tag":449,"props":1269,"children":1270},{},[1271,1279,1290],{"type":45,"tag":476,"props":1272,"children":1273},{},[1274],{"type":45,"tag":76,"props":1275,"children":1277},{"className":1276},[],[1278],{"type":51,"value":1236},{"type":45,"tag":476,"props":1280,"children":1281},{},[1282,1288],{"type":45,"tag":76,"props":1283,"children":1285},{"className":1284},[],[1286],{"type":51,"value":1287},"ddtrace\u002Fllmobs\u002F_evaluators\u002Fllm_judge.py",{"type":51,"value":1289}," (re-exported)",{"type":45,"tag":476,"props":1291,"children":1292},{},[1293],{"type":51,"value":1294},"Inline LLM-as-Judge with prompt template support",{"type":45,"tag":54,"props":1296,"children":1297},{},[1298,1303],{"type":45,"tag":58,"props":1299,"children":1300},{},[1301],{"type":51,"value":1302},"Canonical call signatures",{"type":51,"value":1304}," (must match the generated code exactly):",{"type":45,"tag":396,"props":1306,"children":1309},{"className":1307,"code":1308,"language":24,"meta":404,"style":404},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","LLMObs.enable(\n    api_key=os.getenv(\"DD_API_KEY\"),\n    app_key=os.getenv(\"DD_APPLICATION_KEY\"),\n    site=os.getenv(\"DD_SITE\", \"datadoghq.com\"),  # required for non-prod sites (e.g. datad0g.com, datadoghq.eu)\n    project_name=\"\u003Cproject>\",\n    agentless_enabled=True,  # required when not running behind the dd-agent\n)\n# Note: ml_app is not a separate input. The SDK derives it from project_name\n# when not supplied. If a user really wants to override it later, they can\n# add `ml_app=\"...\"` to enable() themselves.\n\ndataset = LLMObs.create_dataset(\n    dataset_name=\"\u003Cname>\",\n    description=\"\u003Coptional>\",\n    records=[\n        # Per-record `tags` MUST be a list of \"key:value\" strings (e.g. \"env:smoke\"),\n        # never bare strings — the SDK rejects malformed tags with a ValueError on append.\n        {\"input_data\": {\"\u003Ck>\": \"\u003Cv>\"}, \"expected_output\": \"\u003Cv>\", \"metadata\": {}, \"tags\": [\"env:\u003Cenv>\"]},\n        # ...\n    ],\n)\n# OR\ndataset = LLMObs.create_dataset_from_csv(\n    csv_path=\"\u003Cpath>\",\n    dataset_name=\"\u003Cname>\",\n    input_data_columns=[\"\u003Ccol1>\", \"\u003Ccol2>\"],\n    expected_output_columns=[\"\u003Ccol>\"],\n)\n# OR pull an existing Datadog dataset by name (no local file needed)\ndataset = LLMObs.pull_dataset(\n    dataset_name=\"\u003Cname>\",\n    project_name=\"\u003Cproject>\",   # optional — defaults to the project on enable()\n    version=2,                  # optional — pin a version; omit for the latest\n)\n\ndef task_fn(input_data: dict, config: dict):\n    # In real output, this is wired to the user's discovered LLM call site via\n    # Workflow step 2.5. Only emits as a generic placeholder (with # TODO(user))\n    # when --placeholder-task is set or introspection found nothing.\n    ...\n\n# Plain function evaluator (default style)\ndef exact_match(input_data, output_data, expected_output) -> bool:\n    return output_data == expected_output\n\nexperiment = LLMObs.experiment(\n    name=\"\u003Cexperiment_name>\",\n    dataset=dataset,\n    task=task_fn,\n    evaluators=[exact_match],\n    config={\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.0,\n        # Provenance also lives in `config` so it renders in the\n        # experiment's Configuration view alongside model\u002Ftemperature.\n        # `tags=` below only reaches metadata.tags, which the current UI\n        # does not surface as chips — config is what users actually see.\n        \"generated_by\": \"claude-code\",\n        \"skill\": \"agent-observability-experiment-py-bootstrap\",\n        \"purpose\": \"\u003Cone-line purpose from step 2.4>\",\n    },\n    description=\"\u003Cone-line purpose from step 2.4>\",\n    tags={\n        # Same provenance, sent to experiment metadata.tags for any future\n        # tag-filter UI \u002F API consumers. Always emitted alongside the\n        # config copy — never one without the other.\n        \"generated_by\": \"claude-code\",\n        \"skill\": \"agent-observability-experiment-py-bootstrap\",\n        \"purpose\": \"\u003Cone-line purpose from step 2.4>\",\n    },\n)\n\nexperiment.run(jobs=10)\nprint(experiment.url)\n",[1310],{"type":45,"tag":76,"props":1311,"children":1312},{"__ignoreMap":404},[1313,1324,1333,1342,1351,1360,1369,1378,1387,1396,1405,1415,1424,1433,1442,1451,1460,1469,1478,1486,1495,1503,1512,1521,1530,1538,1547,1556,1564,1573,1582,1590,1599,1608,1616,1624,1633,1642,1651,1660,1669,1677,1686,1695,1704,1712,1721,1730,1739,1748,1757,1766,1775,1784,1793,1802,1811,1820,1829,1838,1847,1856,1865,1874,1883,1892,1901,1909,1917,1925,1933,1941,1949,1958],{"type":45,"tag":1314,"props":1315,"children":1318},"span",{"class":1316,"line":1317},"line",1,[1319],{"type":45,"tag":1314,"props":1320,"children":1321},{},[1322],{"type":51,"value":1323},"LLMObs.enable(\n",{"type":45,"tag":1314,"props":1325,"children":1327},{"class":1316,"line":1326},2,[1328],{"type":45,"tag":1314,"props":1329,"children":1330},{},[1331],{"type":51,"value":1332},"    api_key=os.getenv(\"DD_API_KEY\"),\n",{"type":45,"tag":1314,"props":1334,"children":1336},{"class":1316,"line":1335},3,[1337],{"type":45,"tag":1314,"props":1338,"children":1339},{},[1340],{"type":51,"value":1341},"    app_key=os.getenv(\"DD_APPLICATION_KEY\"),\n",{"type":45,"tag":1314,"props":1343,"children":1345},{"class":1316,"line":1344},4,[1346],{"type":45,"tag":1314,"props":1347,"children":1348},{},[1349],{"type":51,"value":1350},"    site=os.getenv(\"DD_SITE\", \"datadoghq.com\"),  # required for non-prod sites (e.g. datad0g.com, datadoghq.eu)\n",{"type":45,"tag":1314,"props":1352,"children":1354},{"class":1316,"line":1353},5,[1355],{"type":45,"tag":1314,"props":1356,"children":1357},{},[1358],{"type":51,"value":1359},"    project_name=\"\u003Cproject>\",\n",{"type":45,"tag":1314,"props":1361,"children":1363},{"class":1316,"line":1362},6,[1364],{"type":45,"tag":1314,"props":1365,"children":1366},{},[1367],{"type":51,"value":1368},"    agentless_enabled=True,  # required when not running behind the dd-agent\n",{"type":45,"tag":1314,"props":1370,"children":1372},{"class":1316,"line":1371},7,[1373],{"type":45,"tag":1314,"props":1374,"children":1375},{},[1376],{"type":51,"value":1377},")\n",{"type":45,"tag":1314,"props":1379,"children":1381},{"class":1316,"line":1380},8,[1382],{"type":45,"tag":1314,"props":1383,"children":1384},{},[1385],{"type":51,"value":1386},"# Note: ml_app is not a separate input. The SDK derives it from project_name\n",{"type":45,"tag":1314,"props":1388,"children":1390},{"class":1316,"line":1389},9,[1391],{"type":45,"tag":1314,"props":1392,"children":1393},{},[1394],{"type":51,"value":1395},"# when not supplied. If a user really wants to override it later, they can\n",{"type":45,"tag":1314,"props":1397,"children":1399},{"class":1316,"line":1398},10,[1400],{"type":45,"tag":1314,"props":1401,"children":1402},{},[1403],{"type":51,"value":1404},"# add `ml_app=\"...\"` to enable() themselves.\n",{"type":45,"tag":1314,"props":1406,"children":1408},{"class":1316,"line":1407},11,[1409],{"type":45,"tag":1314,"props":1410,"children":1412},{"emptyLinePlaceholder":1411},true,[1413],{"type":51,"value":1414},"\n",{"type":45,"tag":1314,"props":1416,"children":1418},{"class":1316,"line":1417},12,[1419],{"type":45,"tag":1314,"props":1420,"children":1421},{},[1422],{"type":51,"value":1423},"dataset = LLMObs.create_dataset(\n",{"type":45,"tag":1314,"props":1425,"children":1427},{"class":1316,"line":1426},13,[1428],{"type":45,"tag":1314,"props":1429,"children":1430},{},[1431],{"type":51,"value":1432},"    dataset_name=\"\u003Cname>\",\n",{"type":45,"tag":1314,"props":1434,"children":1436},{"class":1316,"line":1435},14,[1437],{"type":45,"tag":1314,"props":1438,"children":1439},{},[1440],{"type":51,"value":1441},"    description=\"\u003Coptional>\",\n",{"type":45,"tag":1314,"props":1443,"children":1445},{"class":1316,"line":1444},15,[1446],{"type":45,"tag":1314,"props":1447,"children":1448},{},[1449],{"type":51,"value":1450},"    records=[\n",{"type":45,"tag":1314,"props":1452,"children":1454},{"class":1316,"line":1453},16,[1455],{"type":45,"tag":1314,"props":1456,"children":1457},{},[1458],{"type":51,"value":1459},"        # Per-record `tags` MUST be a list of \"key:value\" strings (e.g. \"env:smoke\"),\n",{"type":45,"tag":1314,"props":1461,"children":1463},{"class":1316,"line":1462},17,[1464],{"type":45,"tag":1314,"props":1465,"children":1466},{},[1467],{"type":51,"value":1468},"        # never bare strings — the SDK rejects malformed tags with a ValueError on append.\n",{"type":45,"tag":1314,"props":1470,"children":1472},{"class":1316,"line":1471},18,[1473],{"type":45,"tag":1314,"props":1474,"children":1475},{},[1476],{"type":51,"value":1477},"        {\"input_data\": {\"\u003Ck>\": \"\u003Cv>\"}, \"expected_output\": \"\u003Cv>\", \"metadata\": {}, \"tags\": [\"env:\u003Cenv>\"]},\n",{"type":45,"tag":1314,"props":1479,"children":1480},{"class":1316,"line":32},[1481],{"type":45,"tag":1314,"props":1482,"children":1483},{},[1484],{"type":51,"value":1485},"        # ...\n",{"type":45,"tag":1314,"props":1487,"children":1489},{"class":1316,"line":1488},20,[1490],{"type":45,"tag":1314,"props":1491,"children":1492},{},[1493],{"type":51,"value":1494},"    ],\n",{"type":45,"tag":1314,"props":1496,"children":1498},{"class":1316,"line":1497},21,[1499],{"type":45,"tag":1314,"props":1500,"children":1501},{},[1502],{"type":51,"value":1377},{"type":45,"tag":1314,"props":1504,"children":1506},{"class":1316,"line":1505},22,[1507],{"type":45,"tag":1314,"props":1508,"children":1509},{},[1510],{"type":51,"value":1511},"# OR\n",{"type":45,"tag":1314,"props":1513,"children":1515},{"class":1316,"line":1514},23,[1516],{"type":45,"tag":1314,"props":1517,"children":1518},{},[1519],{"type":51,"value":1520},"dataset = LLMObs.create_dataset_from_csv(\n",{"type":45,"tag":1314,"props":1522,"children":1524},{"class":1316,"line":1523},24,[1525],{"type":45,"tag":1314,"props":1526,"children":1527},{},[1528],{"type":51,"value":1529},"    csv_path=\"\u003Cpath>\",\n",{"type":45,"tag":1314,"props":1531,"children":1533},{"class":1316,"line":1532},25,[1534],{"type":45,"tag":1314,"props":1535,"children":1536},{},[1537],{"type":51,"value":1432},{"type":45,"tag":1314,"props":1539,"children":1541},{"class":1316,"line":1540},26,[1542],{"type":45,"tag":1314,"props":1543,"children":1544},{},[1545],{"type":51,"value":1546},"    input_data_columns=[\"\u003Ccol1>\", \"\u003Ccol2>\"],\n",{"type":45,"tag":1314,"props":1548,"children":1550},{"class":1316,"line":1549},27,[1551],{"type":45,"tag":1314,"props":1552,"children":1553},{},[1554],{"type":51,"value":1555},"    expected_output_columns=[\"\u003Ccol>\"],\n",{"type":45,"tag":1314,"props":1557,"children":1559},{"class":1316,"line":1558},28,[1560],{"type":45,"tag":1314,"props":1561,"children":1562},{},[1563],{"type":51,"value":1377},{"type":45,"tag":1314,"props":1565,"children":1567},{"class":1316,"line":1566},29,[1568],{"type":45,"tag":1314,"props":1569,"children":1570},{},[1571],{"type":51,"value":1572},"# OR pull an existing Datadog dataset by name (no local file needed)\n",{"type":45,"tag":1314,"props":1574,"children":1576},{"class":1316,"line":1575},30,[1577],{"type":45,"tag":1314,"props":1578,"children":1579},{},[1580],{"type":51,"value":1581},"dataset = LLMObs.pull_dataset(\n",{"type":45,"tag":1314,"props":1583,"children":1585},{"class":1316,"line":1584},31,[1586],{"type":45,"tag":1314,"props":1587,"children":1588},{},[1589],{"type":51,"value":1432},{"type":45,"tag":1314,"props":1591,"children":1593},{"class":1316,"line":1592},32,[1594],{"type":45,"tag":1314,"props":1595,"children":1596},{},[1597],{"type":51,"value":1598},"    project_name=\"\u003Cproject>\",   # optional — defaults to the project on enable()\n",{"type":45,"tag":1314,"props":1600,"children":1602},{"class":1316,"line":1601},33,[1603],{"type":45,"tag":1314,"props":1604,"children":1605},{},[1606],{"type":51,"value":1607},"    version=2,                  # optional — pin a version; omit for the latest\n",{"type":45,"tag":1314,"props":1609,"children":1611},{"class":1316,"line":1610},34,[1612],{"type":45,"tag":1314,"props":1613,"children":1614},{},[1615],{"type":51,"value":1377},{"type":45,"tag":1314,"props":1617,"children":1619},{"class":1316,"line":1618},35,[1620],{"type":45,"tag":1314,"props":1621,"children":1622},{"emptyLinePlaceholder":1411},[1623],{"type":51,"value":1414},{"type":45,"tag":1314,"props":1625,"children":1627},{"class":1316,"line":1626},36,[1628],{"type":45,"tag":1314,"props":1629,"children":1630},{},[1631],{"type":51,"value":1632},"def task_fn(input_data: dict, config: dict):\n",{"type":45,"tag":1314,"props":1634,"children":1636},{"class":1316,"line":1635},37,[1637],{"type":45,"tag":1314,"props":1638,"children":1639},{},[1640],{"type":51,"value":1641},"    # In real output, this is wired to the user's discovered LLM call site via\n",{"type":45,"tag":1314,"props":1643,"children":1645},{"class":1316,"line":1644},38,[1646],{"type":45,"tag":1314,"props":1647,"children":1648},{},[1649],{"type":51,"value":1650},"    # Workflow step 2.5. Only emits as a generic placeholder (with # TODO(user))\n",{"type":45,"tag":1314,"props":1652,"children":1654},{"class":1316,"line":1653},39,[1655],{"type":45,"tag":1314,"props":1656,"children":1657},{},[1658],{"type":51,"value":1659},"    # when --placeholder-task is set or introspection found nothing.\n",{"type":45,"tag":1314,"props":1661,"children":1663},{"class":1316,"line":1662},40,[1664],{"type":45,"tag":1314,"props":1665,"children":1666},{},[1667],{"type":51,"value":1668},"    ...\n",{"type":45,"tag":1314,"props":1670,"children":1672},{"class":1316,"line":1671},41,[1673],{"type":45,"tag":1314,"props":1674,"children":1675},{"emptyLinePlaceholder":1411},[1676],{"type":51,"value":1414},{"type":45,"tag":1314,"props":1678,"children":1680},{"class":1316,"line":1679},42,[1681],{"type":45,"tag":1314,"props":1682,"children":1683},{},[1684],{"type":51,"value":1685},"# Plain function evaluator (default style)\n",{"type":45,"tag":1314,"props":1687,"children":1689},{"class":1316,"line":1688},43,[1690],{"type":45,"tag":1314,"props":1691,"children":1692},{},[1693],{"type":51,"value":1694},"def exact_match(input_data, output_data, expected_output) -> bool:\n",{"type":45,"tag":1314,"props":1696,"children":1698},{"class":1316,"line":1697},44,[1699],{"type":45,"tag":1314,"props":1700,"children":1701},{},[1702],{"type":51,"value":1703},"    return output_data == expected_output\n",{"type":45,"tag":1314,"props":1705,"children":1707},{"class":1316,"line":1706},45,[1708],{"type":45,"tag":1314,"props":1709,"children":1710},{"emptyLinePlaceholder":1411},[1711],{"type":51,"value":1414},{"type":45,"tag":1314,"props":1713,"children":1715},{"class":1316,"line":1714},46,[1716],{"type":45,"tag":1314,"props":1717,"children":1718},{},[1719],{"type":51,"value":1720},"experiment = LLMObs.experiment(\n",{"type":45,"tag":1314,"props":1722,"children":1724},{"class":1316,"line":1723},47,[1725],{"type":45,"tag":1314,"props":1726,"children":1727},{},[1728],{"type":51,"value":1729},"    name=\"\u003Cexperiment_name>\",\n",{"type":45,"tag":1314,"props":1731,"children":1733},{"class":1316,"line":1732},48,[1734],{"type":45,"tag":1314,"props":1735,"children":1736},{},[1737],{"type":51,"value":1738},"    dataset=dataset,\n",{"type":45,"tag":1314,"props":1740,"children":1742},{"class":1316,"line":1741},49,[1743],{"type":45,"tag":1314,"props":1744,"children":1745},{},[1746],{"type":51,"value":1747},"    task=task_fn,\n",{"type":45,"tag":1314,"props":1749,"children":1751},{"class":1316,"line":1750},50,[1752],{"type":45,"tag":1314,"props":1753,"children":1754},{},[1755],{"type":51,"value":1756},"    evaluators=[exact_match],\n",{"type":45,"tag":1314,"props":1758,"children":1760},{"class":1316,"line":1759},51,[1761],{"type":45,"tag":1314,"props":1762,"children":1763},{},[1764],{"type":51,"value":1765},"    config={\n",{"type":45,"tag":1314,"props":1767,"children":1769},{"class":1316,"line":1768},52,[1770],{"type":45,"tag":1314,"props":1771,"children":1772},{},[1773],{"type":51,"value":1774},"        \"model\": \"gpt-4o-mini\",\n",{"type":45,"tag":1314,"props":1776,"children":1778},{"class":1316,"line":1777},53,[1779],{"type":45,"tag":1314,"props":1780,"children":1781},{},[1782],{"type":51,"value":1783},"        \"temperature\": 0.0,\n",{"type":45,"tag":1314,"props":1785,"children":1787},{"class":1316,"line":1786},54,[1788],{"type":45,"tag":1314,"props":1789,"children":1790},{},[1791],{"type":51,"value":1792},"        # Provenance also lives in `config` so it renders in the\n",{"type":45,"tag":1314,"props":1794,"children":1796},{"class":1316,"line":1795},55,[1797],{"type":45,"tag":1314,"props":1798,"children":1799},{},[1800],{"type":51,"value":1801},"        # experiment's Configuration view alongside model\u002Ftemperature.\n",{"type":45,"tag":1314,"props":1803,"children":1805},{"class":1316,"line":1804},56,[1806],{"type":45,"tag":1314,"props":1807,"children":1808},{},[1809],{"type":51,"value":1810},"        # `tags=` below only reaches metadata.tags, which the current UI\n",{"type":45,"tag":1314,"props":1812,"children":1814},{"class":1316,"line":1813},57,[1815],{"type":45,"tag":1314,"props":1816,"children":1817},{},[1818],{"type":51,"value":1819},"        # does not surface as chips — config is what users actually see.\n",{"type":45,"tag":1314,"props":1821,"children":1823},{"class":1316,"line":1822},58,[1824],{"type":45,"tag":1314,"props":1825,"children":1826},{},[1827],{"type":51,"value":1828},"        \"generated_by\": \"claude-code\",\n",{"type":45,"tag":1314,"props":1830,"children":1832},{"class":1316,"line":1831},59,[1833],{"type":45,"tag":1314,"props":1834,"children":1835},{},[1836],{"type":51,"value":1837},"        \"skill\": \"agent-observability-experiment-py-bootstrap\",\n",{"type":45,"tag":1314,"props":1839,"children":1841},{"class":1316,"line":1840},60,[1842],{"type":45,"tag":1314,"props":1843,"children":1844},{},[1845],{"type":51,"value":1846},"        \"purpose\": \"\u003Cone-line purpose from step 2.4>\",\n",{"type":45,"tag":1314,"props":1848,"children":1850},{"class":1316,"line":1849},61,[1851],{"type":45,"tag":1314,"props":1852,"children":1853},{},[1854],{"type":51,"value":1855},"    },\n",{"type":45,"tag":1314,"props":1857,"children":1859},{"class":1316,"line":1858},62,[1860],{"type":45,"tag":1314,"props":1861,"children":1862},{},[1863],{"type":51,"value":1864},"    description=\"\u003Cone-line purpose from step 2.4>\",\n",{"type":45,"tag":1314,"props":1866,"children":1868},{"class":1316,"line":1867},63,[1869],{"type":45,"tag":1314,"props":1870,"children":1871},{},[1872],{"type":51,"value":1873},"    tags={\n",{"type":45,"tag":1314,"props":1875,"children":1877},{"class":1316,"line":1876},64,[1878],{"type":45,"tag":1314,"props":1879,"children":1880},{},[1881],{"type":51,"value":1882},"        # Same provenance, sent to experiment metadata.tags for any future\n",{"type":45,"tag":1314,"props":1884,"children":1886},{"class":1316,"line":1885},65,[1887],{"type":45,"tag":1314,"props":1888,"children":1889},{},[1890],{"type":51,"value":1891},"        # tag-filter UI \u002F API consumers. Always emitted alongside the\n",{"type":45,"tag":1314,"props":1893,"children":1895},{"class":1316,"line":1894},66,[1896],{"type":45,"tag":1314,"props":1897,"children":1898},{},[1899],{"type":51,"value":1900},"        # config copy — never one without the other.\n",{"type":45,"tag":1314,"props":1902,"children":1904},{"class":1316,"line":1903},67,[1905],{"type":45,"tag":1314,"props":1906,"children":1907},{},[1908],{"type":51,"value":1828},{"type":45,"tag":1314,"props":1910,"children":1912},{"class":1316,"line":1911},68,[1913],{"type":45,"tag":1314,"props":1914,"children":1915},{},[1916],{"type":51,"value":1837},{"type":45,"tag":1314,"props":1918,"children":1920},{"class":1316,"line":1919},69,[1921],{"type":45,"tag":1314,"props":1922,"children":1923},{},[1924],{"type":51,"value":1846},{"type":45,"tag":1314,"props":1926,"children":1928},{"class":1316,"line":1927},70,[1929],{"type":45,"tag":1314,"props":1930,"children":1931},{},[1932],{"type":51,"value":1855},{"type":45,"tag":1314,"props":1934,"children":1936},{"class":1316,"line":1935},71,[1937],{"type":45,"tag":1314,"props":1938,"children":1939},{},[1940],{"type":51,"value":1377},{"type":45,"tag":1314,"props":1942,"children":1944},{"class":1316,"line":1943},72,[1945],{"type":45,"tag":1314,"props":1946,"children":1947},{"emptyLinePlaceholder":1411},[1948],{"type":51,"value":1414},{"type":45,"tag":1314,"props":1950,"children":1952},{"class":1316,"line":1951},73,[1953],{"type":45,"tag":1314,"props":1954,"children":1955},{},[1956],{"type":51,"value":1957},"experiment.run(jobs=10)\n",{"type":45,"tag":1314,"props":1959,"children":1961},{"class":1316,"line":1960},74,[1962],{"type":45,"tag":1314,"props":1963,"children":1964},{},[1965],{"type":51,"value":1966},"print(experiment.url)\n",{"type":45,"tag":311,"props":1968,"children":1969},{},[],{"type":45,"tag":46,"props":1971,"children":1973},{"id":1972},"evaluator-styles",[1974],{"type":51,"value":1975},"Evaluator Styles",{"type":45,"tag":54,"props":1977,"children":1978},{},[1979,1981,1986,1988,1993,1995,2001,2003,2008],{"type":51,"value":1980},"Generated code uses ",{"type":45,"tag":58,"props":1982,"children":1983},{},[1984],{"type":51,"value":1985},"one",{"type":51,"value":1987}," of three evaluator surfaces, picked by ",{"type":45,"tag":76,"props":1989,"children":1991},{"className":1990},[],[1992],{"type":51,"value":713},{"type":51,"value":1994},". The detail for each lives in ",{"type":45,"tag":76,"props":1996,"children":1998},{"className":1997},[],[1999],{"type":51,"value":2000},"references\u002Fevaluator-styles\u002F\u003Cstyle>.md",{"type":51,"value":2002}," and is loaded on demand — only the chosen style is consulted at generation time. The single piece of guidance that applies to all three is the ",{"type":45,"tag":76,"props":2004,"children":2006},{"className":2005},[],[2007],{"type":51,"value":1254},{"type":51,"value":2009}," rule below.",{"type":45,"tag":413,"props":2011,"children":2013},{"id":2012},"return-evaluatorresult-not-bare-values",[2014,2016,2021],{"type":51,"value":2015},"Return ",{"type":45,"tag":76,"props":2017,"children":2019},{"className":2018},[],[2020],{"type":51,"value":1254},{"type":51,"value":2022},", not bare values",{"type":45,"tag":54,"props":2024,"children":2025},{},[2026,2028,2034,2035,2041,2042,2048,2050,2056,2058,2064,2066,2071],{"type":51,"value":2027},"Plain functions are allowed to return ",{"type":45,"tag":76,"props":2029,"children":2031},{"className":2030},[],[2032],{"type":51,"value":2033},"bool",{"type":51,"value":934},{"type":45,"tag":76,"props":2036,"children":2038},{"className":2037},[],[2039],{"type":51,"value":2040},"float",{"type":51,"value":934},{"type":45,"tag":76,"props":2043,"children":2045},{"className":2044},[],[2046],{"type":51,"value":2047},"dict",{"type":51,"value":2049},", and ",{"type":45,"tag":76,"props":2051,"children":2053},{"className":2052},[],[2054],{"type":51,"value":2055},"BaseEvaluator.evaluate()",{"type":51,"value":2057}," is allowed to return raw ",{"type":45,"tag":76,"props":2059,"children":2061},{"className":2060},[],[2062],{"type":51,"value":2063},"JSONType",{"type":51,"value":2065},". The SDK accepts both — but ",{"type":45,"tag":76,"props":2067,"children":2069},{"className":2068},[],[2070],{"type":51,"value":1254},{"type":51,"value":2072}," carries fields the Datadog UI surfaces in ways the raw value cannot:",{"type":45,"tag":441,"props":2074,"children":2075},{},[2076,2097],{"type":45,"tag":445,"props":2077,"children":2078},{},[2079],{"type":45,"tag":449,"props":2080,"children":2081},{},[2082,2087,2092],{"type":45,"tag":453,"props":2083,"children":2084},{},[2085],{"type":51,"value":2086},"Field",{"type":45,"tag":453,"props":2088,"children":2089},{},[2090],{"type":51,"value":2091},"Type",{"type":45,"tag":453,"props":2093,"children":2094},{},[2095],{"type":51,"value":2096},"Used by Datadog UI for",{"type":45,"tag":469,"props":2098,"children":2099},{},[2100,2151,2184,2233,2267],{"type":45,"tag":449,"props":2101,"children":2102},{},[2103,2112,2141],{"type":45,"tag":476,"props":2104,"children":2105},{},[2106],{"type":45,"tag":76,"props":2107,"children":2109},{"className":2108},[],[2110],{"type":51,"value":2111},"value",{"type":45,"tag":476,"props":2113,"children":2114},{},[2115,2120,2121,2126,2127,2133,2134,2139],{"type":45,"tag":76,"props":2116,"children":2118},{"className":2117},[],[2119],{"type":51,"value":2033},{"type":51,"value":934},{"type":45,"tag":76,"props":2122,"children":2124},{"className":2123},[],[2125],{"type":51,"value":2040},{"type":51,"value":934},{"type":45,"tag":76,"props":2128,"children":2130},{"className":2129},[],[2131],{"type":51,"value":2132},"str",{"type":51,"value":934},{"type":45,"tag":76,"props":2135,"children":2137},{"className":2136},[],[2138],{"type":51,"value":2047},{"type":51,"value":2140}," (JSONType)",{"type":45,"tag":476,"props":2142,"children":2143},{},[2144,2146],{"type":51,"value":2145},"The score itself — shown on the experiment metric. ",{"type":45,"tag":58,"props":2147,"children":2148},{},[2149],{"type":51,"value":2150},"Required.",{"type":45,"tag":449,"props":2152,"children":2153},{},[2154,2163,2171],{"type":45,"tag":476,"props":2155,"children":2156},{},[2157],{"type":45,"tag":76,"props":2158,"children":2160},{"className":2159},[],[2161],{"type":51,"value":2162},"reasoning",{"type":45,"tag":476,"props":2164,"children":2165},{},[2166],{"type":45,"tag":76,"props":2167,"children":2169},{"className":2168},[],[2170],{"type":51,"value":2132},{"type":45,"tag":476,"props":2172,"children":2173},{},[2174,2176,2182],{"type":51,"value":2175},"Per-record explanation shown in the compare UI; lets reviewers see ",{"type":45,"tag":2177,"props":2178,"children":2179},"em",{},[2180],{"type":51,"value":2181},"why",{"type":51,"value":2183}," an evaluator passed\u002Ffailed without re-running the LLM.",{"type":45,"tag":449,"props":2185,"children":2186},{},[2187,2196,2228],{"type":45,"tag":476,"props":2188,"children":2189},{},[2190],{"type":45,"tag":76,"props":2191,"children":2193},{"className":2192},[],[2194],{"type":51,"value":2195},"assessment",{"type":45,"tag":476,"props":2197,"children":2198},{},[2199,2204,2206,2212,2213,2219,2220,2226],{"type":45,"tag":76,"props":2200,"children":2202},{"className":2201},[],[2203],{"type":51,"value":2132},{"type":51,"value":2205}," (e.g. ",{"type":45,"tag":76,"props":2207,"children":2209},{"className":2208},[],[2210],{"type":51,"value":2211},"\"pass\"",{"type":51,"value":934},{"type":45,"tag":76,"props":2214,"children":2216},{"className":2215},[],[2217],{"type":51,"value":2218},"\"fail\"",{"type":51,"value":934},{"type":45,"tag":76,"props":2221,"children":2223},{"className":2222},[],[2224],{"type":51,"value":2225},"\"partial\"",{"type":51,"value":2227},")",{"type":45,"tag":476,"props":2229,"children":2230},{},[2231],{"type":51,"value":2232},"Determines whether a metric trend going up vs. down is an improvement; the UI uses this to color baseline-vs-candidate comparisons.",{"type":45,"tag":449,"props":2234,"children":2235},{},[2236,2245,2254],{"type":45,"tag":476,"props":2237,"children":2238},{},[2239],{"type":45,"tag":76,"props":2240,"children":2242},{"className":2241},[],[2243],{"type":51,"value":2244},"metadata",{"type":45,"tag":476,"props":2246,"children":2247},{},[2248],{"type":45,"tag":76,"props":2249,"children":2251},{"className":2250},[],[2252],{"type":51,"value":2253},"dict[str, JSONType]",{"type":45,"tag":476,"props":2255,"children":2256},{},[2257,2259,2265],{"type":51,"value":2258},"Free-form per-record context (e.g. ",{"type":45,"tag":76,"props":2260,"children":2262},{"className":2261},[],[2263],{"type":51,"value":2264},"{\"confidence\": 0.95}",{"type":51,"value":2266},"); shown in record drill-down.",{"type":45,"tag":449,"props":2268,"children":2269},{},[2270,2279,2287],{"type":45,"tag":476,"props":2271,"children":2272},{},[2273],{"type":45,"tag":76,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":51,"value":2278},"tags",{"type":45,"tag":476,"props":2280,"children":2281},{},[2282],{"type":45,"tag":76,"props":2283,"children":2285},{"className":2284},[],[2286],{"type":51,"value":2253},{"type":45,"tag":476,"props":2288,"children":2289},{},[2290,2292,2298],{"type":51,"value":2291},"Used to slice experiment results in the UI (e.g. ",{"type":45,"tag":76,"props":2293,"children":2295},{"className":2294},[],[2296],{"type":51,"value":2297},"{\"category\": \"accuracy\"}",{"type":51,"value":1019},{"type":45,"tag":54,"props":2300,"children":2301},{},[2302,2304,2309,2311,2317,2319,2325,2327,2332],{"type":51,"value":2303},"Default to ",{"type":45,"tag":76,"props":2305,"children":2307},{"className":2306},[],[2308],{"type":51,"value":1254},{"type":51,"value":2310}," for any evaluator richer than a one-line equality check. Trivial checks like ",{"type":45,"tag":76,"props":2312,"children":2314},{"className":2313},[],[2315],{"type":51,"value":2316},"exact_match",{"type":51,"value":2318}," and ",{"type":45,"tag":76,"props":2320,"children":2322},{"className":2321},[],[2323],{"type":51,"value":2324},"length_under_500",{"type":51,"value":2326}," are the only cases where a bare ",{"type":45,"tag":76,"props":2328,"children":2330},{"className":2329},[],[2331],{"type":51,"value":2033},{"type":51,"value":2333}," is acceptable.",{"type":45,"tag":413,"props":2335,"children":2337},{"id":2336},"style-router",[2338],{"type":51,"value":2339},"Style router",{"type":45,"tag":54,"props":2341,"children":2342},{},[2343,2345,2350,2352,2357,2359,2365],{"type":51,"value":2344},"When ",{"type":45,"tag":76,"props":2346,"children":2348},{"className":2347},[],[2349],{"type":51,"value":713},{"type":51,"value":2351}," is resolved (default ",{"type":45,"tag":76,"props":2353,"children":2355},{"className":2354},[],[2356],{"type":51,"value":722},{"type":51,"value":2358},"), read the matching reference file from ",{"type":45,"tag":76,"props":2360,"children":2362},{"className":2361},[],[2363],{"type":51,"value":2364},"\u003Cthis-skill-dir>\u002Freferences\u002Fevaluator-styles\u002F",{"type":51,"value":2366}," and emit section 5 of the generated file using the code template it contains:",{"type":45,"tag":441,"props":2368,"children":2369},{},[2370,2396],{"type":45,"tag":445,"props":2371,"children":2372},{},[2373],{"type":45,"tag":449,"props":2374,"children":2375},{},[2376,2386,2391],{"type":45,"tag":453,"props":2377,"children":2378},{},[2379,2384],{"type":45,"tag":76,"props":2380,"children":2382},{"className":2381},[],[2383],{"type":51,"value":713},{"type":51,"value":2385}," value",{"type":45,"tag":453,"props":2387,"children":2388},{},[2389],{"type":51,"value":2390},"Read",{"type":45,"tag":453,"props":2392,"children":2393},{},[2394],{"type":51,"value":2395},"Best for",{"type":45,"tag":469,"props":2397,"children":2398},{},[2399,2431,2456],{"type":45,"tag":449,"props":2400,"children":2401},{},[2402,2417,2426],{"type":45,"tag":476,"props":2403,"children":2404},{},[2405,2410,2412],{"type":45,"tag":76,"props":2406,"children":2408},{"className":2407},[],[2409],{"type":51,"value":722},{"type":51,"value":2411}," ",{"type":45,"tag":2177,"props":2413,"children":2414},{},[2415],{"type":51,"value":2416},"(default)",{"type":45,"tag":476,"props":2418,"children":2419},{},[2420],{"type":45,"tag":76,"props":2421,"children":2423},{"className":2422},[],[2424],{"type":51,"value":2425},"references\u002Fevaluator-styles\u002Ffunction.md",{"type":45,"tag":476,"props":2427,"children":2428},{},[2429],{"type":51,"value":2430},"Most cases — matches the canonical notebook style",{"type":45,"tag":449,"props":2432,"children":2433},{},[2434,2442,2451],{"type":45,"tag":476,"props":2435,"children":2436},{},[2437],{"type":45,"tag":76,"props":2438,"children":2440},{"className":2439},[],[2441],{"type":51,"value":738},{"type":45,"tag":476,"props":2443,"children":2444},{},[2445],{"type":45,"tag":76,"props":2446,"children":2448},{"className":2447},[],[2449],{"type":51,"value":2450},"references\u002Fevaluator-styles\u002Fclass.md",{"type":45,"tag":476,"props":2452,"children":2453},{},[2454],{"type":51,"value":2455},"Evaluators that need persistent state or an async client",{"type":45,"tag":449,"props":2457,"children":2458},{},[2459,2467,2476],{"type":45,"tag":476,"props":2460,"children":2461},{},[2462],{"type":45,"tag":76,"props":2463,"children":2465},{"className":2464},[],[2466],{"type":51,"value":754},{"type":45,"tag":476,"props":2468,"children":2469},{},[2470],{"type":45,"tag":76,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":51,"value":2475},"references\u002Fevaluator-styles\u002Fremote.md",{"type":45,"tag":476,"props":2477,"children":2478},{},[2479],{"type":51,"value":2480},"Server-side LLM-as-Judge configured in the Datadog UI, reused across experiments",{"type":45,"tag":54,"props":2482,"children":2483},{},[2484,2486,2491],{"type":51,"value":2485},"Do ",{"type":45,"tag":58,"props":2487,"children":2488},{},[2489],{"type":51,"value":2490},"not",{"type":51,"value":2492}," load all three. Each reference file is self-contained — code template + when-to-use + when-not-to-use guidance.",{"type":45,"tag":311,"props":2494,"children":2495},{},[],{"type":45,"tag":46,"props":2497,"children":2499},{"id":2498},"generated-file-structure",[2500],{"type":51,"value":2501},"Generated File Structure",{"type":45,"tag":54,"props":2503,"children":2504},{},[2505,2507,2512,2514,2519],{"type":51,"value":2506},"The same section sequence in both formats. In ",{"type":45,"tag":76,"props":2508,"children":2510},{"className":2509},[],[2511],{"type":51,"value":349},{"type":51,"value":2513}," these become comment banners; in ",{"type":45,"tag":76,"props":2515,"children":2517},{"className":2516},[],[2518],{"type":51,"value":357},{"type":51,"value":2520}," each becomes one markdown cell + one code cell.",{"type":45,"tag":396,"props":2522,"children":2525},{"className":2523,"code":2524,"language":51},[399],"0. File header docstring — name, generated_at, purpose (from step 2.4), provider, wired task source.\n1. Env setup           — auto-discover .env files (cwd, app root, parent walk,\n                         ~\u002F.datadog\u002Fcredentials), then os.getenv reads + hard-assert\n                         required keys. NO python-dotenv dependency. Shell env wins\n                         over file-loaded values. Only the provider keys actually\n                         needed by the wired task_fn are asserted (see step 2.5).\n2. LLMObs.enable()     — explicit api_key\u002Fapp_key\u002Fproject_name\u002Fagentless_enabled\n3. Dataset             — inline records OR create_dataset_from_csv\n4. Task function       — REAL task function imported from the user's app (via Workflow step 2.5\n                         introspection) and adapted to the SDK signature. Only falls back to a\n                         placeholder OpenAI call with # TODO(user) if --placeholder-task is set or\n                         no LLM call site was found in the project.\n5. Evaluators          — 2-3 in the requested style, semantics seeded by the purpose (step 3)\n6. Experiment          — LLMObs.experiment(config={..., \"generated_by\": \"claude-code\", \"purpose\": \"...\", ...}, tags={...})\n7. Run                 — experiment.run(jobs=N); print(experiment.url)\n8. Results inspection  — experiment.as_dataframe() if pandas, else print\n",[2526],{"type":45,"tag":76,"props":2527,"children":2528},{"__ignoreMap":404},[2529],{"type":51,"value":2524},{"type":45,"tag":311,"props":2531,"children":2532},{},[],{"type":45,"tag":46,"props":2534,"children":2536},{"id":2535},"workflow",[2537],{"type":51,"value":2538},"Workflow",{"type":45,"tag":66,"props":2540,"children":2541},{},[2542],{"type":45,"tag":70,"props":2543,"children":2544},{},[2545,2550,2552,2558,2560,2565,2567,2572,2573,2577,2579,2584,2586,2591,2593,2700,2703,2705,2710,2712,2718,2720,2726,2728,2734,2736,2741,2743,2749,2751,2756,2758,2763,2765,2768,2770,2776,2778,2784],{"type":45,"tag":58,"props":2546,"children":2547},{},[2548],{"type":51,"value":2549},"Parse arguments",{"type":51,"value":2551},". Default ",{"type":45,"tag":76,"props":2553,"children":2555},{"className":2554},[],[2556],{"type":51,"value":2557},"--format py",{"type":51,"value":2559},". Resolve ",{"type":45,"tag":76,"props":2561,"children":2563},{"className":2562},[],[2564],{"type":51,"value":807},{"type":51,"value":2566}," extension from ",{"type":45,"tag":76,"props":2568,"children":2570},{"className":2569},[],[2571],{"type":51,"value":437},{"type":51,"value":110},{"type":45,"tag":2574,"props":2575,"children":2576},"br",{},[],{"type":51,"value":2578},"If ",{"type":45,"tag":76,"props":2580,"children":2582},{"className":2581},[],[2583],{"type":51,"value":669},{"type":51,"value":2585}," is not provided, resolve a default of the form ",{"type":45,"tag":76,"props":2587,"children":2589},{"className":2588},[],[2590],{"type":51,"value":678},{"type":51,"value":2592}," by walking these sources in order, taking the first match:",{"type":45,"tag":66,"props":2594,"children":2595},{},[2596,2621,2637,2662,2680],{"type":45,"tag":70,"props":2597,"children":2598},{},[2599,2604,2606,2612,2614,2620],{"type":45,"tag":76,"props":2600,"children":2602},{"className":2601},[],[2603],{"type":51,"value":932},{"type":51,"value":2605}," → ",{"type":45,"tag":76,"props":2607,"children":2609},{"className":2608},[],[2610],{"type":51,"value":2611},"[project] name",{"type":51,"value":2613}," (PEP 621) or ",{"type":45,"tag":76,"props":2615,"children":2617},{"className":2616},[],[2618],{"type":51,"value":2619},"[tool.poetry] name",{"type":51,"value":110},{"type":45,"tag":70,"props":2622,"children":2623},{},[2624,2629,2630,2636],{"type":45,"tag":76,"props":2625,"children":2627},{"className":2626},[],[2628],{"type":51,"value":940},{"type":51,"value":2605},{"type":45,"tag":76,"props":2631,"children":2633},{"className":2632},[],[2634],{"type":51,"value":2635},"[metadata] name",{"type":51,"value":110},{"type":45,"tag":70,"props":2638,"children":2639},{},[2640,2645,2647,2653,2655,2661],{"type":45,"tag":76,"props":2641,"children":2643},{"className":2642},[],[2644],{"type":51,"value":947},{"type":51,"value":2646}," → first ",{"type":45,"tag":76,"props":2648,"children":2650},{"className":2649},[],[2651],{"type":51,"value":2652},"name=\"...\"",{"type":51,"value":2654}," argument to ",{"type":45,"tag":76,"props":2656,"children":2658},{"className":2657},[],[2659],{"type":51,"value":2660},"setup(...)",{"type":51,"value":110},{"type":45,"tag":70,"props":2663,"children":2664},{},[2665,2671,2672,2678],{"type":45,"tag":76,"props":2666,"children":2668},{"className":2667},[],[2669],{"type":51,"value":2670},"package.json",{"type":51,"value":2605},{"type":45,"tag":76,"props":2673,"children":2675},{"className":2674},[],[2676],{"type":51,"value":2677},"\"name\"",{"type":51,"value":2679}," (useful when the LLM app lives in a TS\u002FJS monorepo Python package).",{"type":45,"tag":70,"props":2681,"children":2682},{},[2683,2685,2691,2693,2699],{"type":51,"value":2684},"The basename of the current working directory, lowercased and slugified (",{"type":45,"tag":76,"props":2686,"children":2688},{"className":2687},[],[2689],{"type":51,"value":2690},"\u002F^[a-z0-9-]+$\u002F",{"type":51,"value":2692}," — replace non-matching chars with ",{"type":45,"tag":76,"props":2694,"children":2696},{"className":2695},[],[2697],{"type":51,"value":2698},"-",{"type":51,"value":1019},{"type":45,"tag":2574,"props":2701,"children":2702},{},[],{"type":51,"value":2704},"The final project name is ",{"type":45,"tag":76,"props":2706,"children":2708},{"className":2707},[],[2709],{"type":51,"value":678},{"type":51,"value":2711},". Strip a leading ",{"type":45,"tag":76,"props":2713,"children":2715},{"className":2714},[],[2716],{"type":51,"value":2717},"experiment-",{"type":51,"value":2719}," from ",{"type":45,"tag":76,"props":2721,"children":2723},{"className":2722},[],[2724],{"type":51,"value":2725},"\u003Cservice-name>",{"type":51,"value":2727}," if it already starts with one (so a package literally named ",{"type":45,"tag":76,"props":2729,"children":2731},{"className":2730},[],[2732],{"type":51,"value":2733},"experiment-foo",{"type":51,"value":2735}," yields ",{"type":45,"tag":76,"props":2737,"children":2739},{"className":2738},[],[2740],{"type":51,"value":2733},{"type":51,"value":2742},", not ",{"type":45,"tag":76,"props":2744,"children":2746},{"className":2745},[],[2747],{"type":51,"value":2748},"experiment-experiment-foo",{"type":51,"value":2750},"). If none of the five sources resolve to a non-empty string, fall back to ",{"type":45,"tag":76,"props":2752,"children":2754},{"className":2753},[],[2755],{"type":51,"value":686},{"type":51,"value":2757}," and emit a warning in the next-steps output that the user should set ",{"type":45,"tag":76,"props":2759,"children":2761},{"className":2760},[],[2762],{"type":51,"value":669},{"type":51,"value":2764}," explicitly.",{"type":45,"tag":2574,"props":2766,"children":2767},{},[],{"type":51,"value":2769},"Embed the resolved name as a string literal in the generated ",{"type":45,"tag":76,"props":2771,"children":2773},{"className":2772},[],[2774],{"type":51,"value":2775},"PROJECT_NAME = \"...\"",{"type":51,"value":2777}," line — don't emit runtime ",{"type":45,"tag":76,"props":2779,"children":2781},{"className":2780},[],[2782],{"type":51,"value":2783},"os.getcwd()",{"type":51,"value":2785}," lookups, since the user may run the file from a different directory than where the skill resolved it.",{"type":45,"tag":54,"props":2787,"children":2788},{},[2789,2791,2796,2798,2802,2804,2810,2812,2818,2820,2825],{"type":51,"value":2790},"1.5. ",{"type":45,"tag":58,"props":2792,"children":2793},{},[2794],{"type":51,"value":2795},"Startup beacon.",{"type":51,"value":2797}," Per the ",{"type":45,"tag":58,"props":2799,"children":2800},{},[2801],{"type":51,"value":52},{"type":51,"value":2803}," section above: generate an 8-character hex invocation ID, then issue the single beacon call (MCP ",{"type":45,"tag":76,"props":2805,"children":2807},{"className":2806},[],[2808],{"type":51,"value":2809},"list_llmobs_evals",{"type":51,"value":2811}," tagged with ",{"type":45,"tag":76,"props":2813,"children":2815},{"className":2814},[],[2816],{"type":51,"value":2817},"skill:agent-observability-experiment-py-bootstrap:start[\u003Cinv_id>] — Skill startup: ...",{"type":51,"value":2819},", or pup equivalent, or skip if no backend). Surface any error as a one-line ",{"type":45,"tag":76,"props":2821,"children":2823},{"className":2822},[],[2824],{"type":51,"value":231},{"type":51,"value":2826}," and proceed regardless — the beacon is for usage attribution and connectivity validation only; it never blocks codegen.",{"type":45,"tag":66,"props":2828,"children":2829},{"start":1326},[2830],{"type":45,"tag":70,"props":2831,"children":2832},{},[2833,2838,2840,2845,2846,2851,2853,3085,3088,3093,3095,3100,3102,3108,3110,3116,3118,3123,3125,3130],{"type":45,"tag":58,"props":2834,"children":2835},{},[2836],{"type":51,"value":2837},"Resolve the dataset source.",{"type":51,"value":2839}," Error out if both ",{"type":45,"tag":76,"props":2841,"children":2843},{"className":2842},[],[2844],{"type":51,"value":529},{"type":51,"value":2318},{"type":45,"tag":76,"props":2847,"children":2849},{"className":2848},[],[2850],{"type":51,"value":577},{"type":51,"value":2852}," are passed — they're mutually exclusive.",{"type":45,"tag":235,"props":2854,"children":2855},{},[2856,2985,3061],{"type":45,"tag":70,"props":2857,"children":2858},{},[2859,2870,2872],{"type":45,"tag":58,"props":2860,"children":2861},{},[2862,2868],{"type":45,"tag":76,"props":2863,"children":2865},{"className":2864},[],[2866],{"type":51,"value":2867},"--dataset \u003Cpath>",{"type":51,"value":2869}," (local file → inline records or CSV loader)",{"type":51,"value":2871},":",{"type":45,"tag":235,"props":2873,"children":2874},{},[2875,2940,2953],{"type":45,"tag":70,"props":2876,"children":2877},{},[2878,2880,2886,2888,2894,2896,2902,2903,2908,2909,2914,2916,2922,2924,2930,2932,2938],{"type":51,"value":2879},"Read the file. If JSON, validate top-level array of ",{"type":45,"tag":76,"props":2881,"children":2883},{"className":2882},[],[2884],{"type":51,"value":2885},"DatasetRecordRaw",{"type":51,"value":2887}," shape (",{"type":45,"tag":76,"props":2889,"children":2891},{"className":2890},[],[2892],{"type":51,"value":2893},"input_data",{"type":51,"value":2895},", optional ",{"type":45,"tag":76,"props":2897,"children":2899},{"className":2898},[],[2900],{"type":51,"value":2901},"expected_output",{"type":51,"value":431},{"type":45,"tag":76,"props":2904,"children":2906},{"className":2905},[],[2907],{"type":51,"value":2244},{"type":51,"value":431},{"type":45,"tag":76,"props":2910,"children":2912},{"className":2911},[],[2913],{"type":51,"value":2278},{"type":51,"value":2915},"). If CSV, parse header and auto-detect columns using the ",{"type":45,"tag":76,"props":2917,"children":2919},{"className":2918},[],[2920],{"type":51,"value":2921},"dataset-bootstrap",{"type":51,"value":2923}," heuristics: ",{"type":45,"tag":76,"props":2925,"children":2927},{"className":2926},[],[2928],{"type":51,"value":2929},"prompt|input|query|question",{"type":51,"value":2931}," → input, ",{"type":45,"tag":76,"props":2933,"children":2935},{"className":2934},[],[2936],{"type":51,"value":2937},"expected|gold|truth|answer",{"type":51,"value":2939}," → expected.",{"type":45,"tag":70,"props":2941,"children":2942},{},[2943,2945,2951],{"type":51,"value":2944},"Run a PII scrub (email\u002Fphone\u002FSSN\u002FAPI-key regexes) on all string values; replace matches with ",{"type":45,"tag":76,"props":2946,"children":2948},{"className":2947},[],[2949],{"type":51,"value":2950},"\u003CREDACTED:pii-type>",{"type":51,"value":2952}," and surface a warning listing affected indices.",{"type":45,"tag":70,"props":2954,"children":2955},{},[2956,2961,2963,2968,2970,2975,2977,2983],{"type":45,"tag":58,"props":2957,"children":2958},{},[2959],{"type":51,"value":2960},"For JSON datasets",{"type":51,"value":2962},", embed the records inline in the generated file (",{"type":45,"tag":76,"props":2964,"children":2966},{"className":2965},[],[2967],{"type":51,"value":540},{"type":51,"value":2969},") so the user has a single self-contained artifact. ",{"type":45,"tag":58,"props":2971,"children":2972},{},[2973],{"type":51,"value":2974},"For CSV datasets",{"type":51,"value":2976},", emit ",{"type":45,"tag":76,"props":2978,"children":2980},{"className":2979},[],[2981],{"type":51,"value":2982},"LLMObs.create_dataset_from_csv(csv_path=\"\u003Cabsolute path>\", ...)",{"type":51,"value":2984}," and tell the user the CSV needs to be present at runtime.",{"type":45,"tag":70,"props":2986,"children":2987},{},[2988,2999,3000],{"type":45,"tag":58,"props":2989,"children":2990},{},[2991,2997],{"type":45,"tag":76,"props":2992,"children":2994},{"className":2993},[],[2995],{"type":51,"value":2996},"--dataset-name \u003Cname>",{"type":51,"value":2998}," (existing Datadog dataset → runtime pull)",{"type":51,"value":2871},{"type":45,"tag":235,"props":3001,"children":3002},{},[3003,3024,3044,3056],{"type":45,"tag":70,"props":3004,"children":3005},{},[3006,3008,3014,3016,3022],{"type":51,"value":3007},"Emit ",{"type":45,"tag":76,"props":3009,"children":3011},{"className":3010},[],[3012],{"type":51,"value":3013},"LLMObs.pull_dataset(dataset_name=\"\u003Cname>\", project_name=\"\u003Cproject>\"[, version=\u003Cn>])",{"type":51,"value":3015}," in place of any ",{"type":45,"tag":76,"props":3017,"children":3019},{"className":3018},[],[3020],{"type":51,"value":3021},"create_dataset*",{"type":51,"value":3023}," call. The fetch happens when the generated experiment runs — the skill itself does not call Datadog.",{"type":45,"tag":70,"props":3025,"children":3026},{},[3027,3029,3035,3037,3042],{"type":51,"value":3028},"Pass ",{"type":45,"tag":76,"props":3030,"children":3032},{"className":3031},[],[3033],{"type":51,"value":3034},"version=\u003Cn>",{"type":51,"value":3036}," through only if ",{"type":45,"tag":76,"props":3038,"children":3040},{"className":3039},[],[3041],{"type":51,"value":625},{"type":51,"value":3043}," was set; otherwise omit it so the SDK resolves the latest.",{"type":45,"tag":70,"props":3045,"children":3046},{},[3047,3049,3055],{"type":51,"value":3048},"Add a one-line comment above the call documenting what's being pulled, e.g. ",{"type":45,"tag":76,"props":3050,"children":3052},{"className":3051},[],[3053],{"type":51,"value":3054},"# Pulled from Datadog: dataset_name=\"qa_v3\", version=latest",{"type":51,"value":110},{"type":45,"tag":70,"props":3057,"children":3058},{},[3059],{"type":51,"value":3060},"Skip the PII scrub and the inline-records emission — there are no local records to scrub.",{"type":45,"tag":70,"props":3062,"children":3063},{},[3064,3069,3070],{"type":45,"tag":58,"props":3065,"children":3066},{},[3067],{"type":51,"value":3068},"Neither flag given",{"type":51,"value":2871},{"type":45,"tag":235,"props":3071,"children":3072},{},[3073],{"type":45,"tag":70,"props":3074,"children":3075},{},[3076,3078,3083],{"type":51,"value":3077},"Fall back to the inline 3-record sample described under ",{"type":45,"tag":76,"props":3079,"children":3081},{"className":3080},[],[3082],{"type":51,"value":529},{"type":51,"value":3084},"'s default, so the generated file remains runnable as-is.",{"type":45,"tag":2574,"props":3086,"children":3087},{},[],{"type":45,"tag":58,"props":3089,"children":3090},{},[3091],{"type":51,"value":3092},"Note on dataset IDs.",{"type":51,"value":3094}," The public SDK's ",{"type":45,"tag":76,"props":3096,"children":3098},{"className":3097},[],[3099],{"type":51,"value":605},{"type":51,"value":3101}," takes a name, not an ID — so there's no ",{"type":45,"tag":76,"props":3103,"children":3105},{"className":3104},[],[3106],{"type":51,"value":3107},"--dataset-id",{"type":51,"value":3109}," flag. If a user only has a dataset ID from a Datadog UI URL (",{"type":45,"tag":76,"props":3111,"children":3113},{"className":3112},[],[3114],{"type":51,"value":3115},"\u002Fllm\u002Fdatasets\u002F\u003Cid>",{"type":51,"value":3117},"), the workflow is: open that URL in the UI, copy the dataset name, and pass it as ",{"type":45,"tag":76,"props":3119,"children":3121},{"className":3120},[],[3122],{"type":51,"value":577},{"type":51,"value":3124},". The skill must not import ",{"type":45,"tag":76,"props":3126,"children":3128},{"className":3127},[],[3129],{"type":51,"value":1098},{"type":51,"value":3131}," or any other underscore module to work around this.",{"type":45,"tag":54,"props":3133,"children":3134},{},[3135,3137,3142,3144,3149,3151,3155],{"type":51,"value":3136},"2.4. ",{"type":45,"tag":58,"props":3138,"children":3139},{},[3140],{"type":51,"value":3141},"Determine the experiment's purpose.",{"type":51,"value":3143}," Capture a one-sentence statement of what the experiment is meant to validate. This becomes a ",{"type":45,"tag":2177,"props":3145,"children":3146},{},[3147],{"type":51,"value":3148},"reasoning input",{"type":51,"value":3150}," that biases every downstream step (introspection ranking in 2.5, wrapper shape in 2.5d, evaluator selection in step 3, file header in step 4). It is ",{"type":45,"tag":58,"props":3152,"children":3153},{},[3154],{"type":51,"value":2490},{"type":51,"value":3156}," a fixed taxonomy — the user types whatever describes their goal, and Claude reads it and applies judgment per invocation.",{"type":45,"tag":54,"props":3158,"children":3159},{},[3160,3165],{"type":45,"tag":58,"props":3161,"children":3162},{},[3163],{"type":51,"value":3164},"Resolution order",{"type":51,"value":2871},{"type":45,"tag":66,"props":3167,"children":3168},{},[3169,3181,3212],{"type":45,"tag":70,"props":3170,"children":3171},{},[3172,3173,3179],{"type":51,"value":2578},{"type":45,"tag":76,"props":3174,"children":3176},{"className":3175},[],[3177],{"type":51,"value":3178},"--purpose \"\u003Ctext>\"",{"type":51,"value":3180}," was passed → use it verbatim. Skip to 2.5.",{"type":45,"tag":70,"props":3182,"children":3183},{},[3184,3186,3191,3192,3197,3198,3203,3205,3210],{"type":51,"value":3185},"If the user's original invocation message contains a clear purpose statement (e.g. ",{"type":45,"tag":2177,"props":3187,"children":3188},{},[3189],{"type":51,"value":3190},"\"set up an experiment to test my tool selection logic\"",{"type":51,"value":431},{"type":45,"tag":2177,"props":3193,"children":3194},{},[3195],{"type":51,"value":3196},"\"validate that SQL output is always parseable\"",{"type":51,"value":431},{"type":45,"tag":2177,"props":3199,"children":3200},{},[3201],{"type":51,"value":3202},"\"regression-test prompt v3\"",{"type":51,"value":3204},"), extract it and present it back for confirmation via ",{"type":45,"tag":76,"props":3206,"children":3208},{"className":3207},[],[3209],{"type":51,"value":1042},{"type":51,"value":3211}," with the extracted text as a single option labeled \"Use the purpose I described\" plus \"Pick a different purpose\" and \"Other\". If confirmed, use the extracted text. Skip to 2.5.",{"type":45,"tag":70,"props":3213,"children":3214},{},[3215,3217,3222,3224,3229,3231,3237],{"type":51,"value":3216},"Otherwise, prompt with ",{"type":45,"tag":76,"props":3218,"children":3220},{"className":3219},[],[3221],{"type":51,"value":1042},{"type":51,"value":3223},". The options are ",{"type":45,"tag":58,"props":3225,"children":3226},{},[3227],{"type":51,"value":3228},"seed prompts",{"type":51,"value":3230},", not a constrained taxonomy — they exist to give the user something to react to. Each \"label\" maps to a starter sentence that the user can refine; selecting an option uses its starter sentence as ",{"type":45,"tag":76,"props":3232,"children":3234},{"className":3233},[],[3235],{"type":51,"value":3236},"purpose",{"type":51,"value":3238}," unless the user provides notes\u002Ffree text instead.",{"type":45,"tag":54,"props":3240,"children":3241},{},[3242,3252],{"type":45,"tag":58,"props":3243,"children":3244},{},[3245,3250],{"type":45,"tag":76,"props":3246,"children":3248},{"className":3247},[],[3249],{"type":51,"value":1042},{"type":51,"value":3251}," payload",{"type":51,"value":3253}," (use these 5 options verbatim — they cover the common cases without locking the user into them):",{"type":45,"tag":396,"props":3255,"children":3258},{"className":3256,"code":3257,"language":51},[399],"question: \"What is this experiment meant to validate?\"\nheader:   \"Purpose\"\noptions:\n  - label: \"Output accuracy \u002F answer quality\"\n    description: \"Verify the model produces correct answers compared to expected outputs. Most common starting point.\"\n  - label: \"Tool call correctness\"\n    description: \"For agent apps — validate the agent picks the right tool with the right arguments. Useful when tool routing is the failure surface.\"\n  - label: \"Structured output \u002F schema validity\"\n    description: \"Verify the output always conforms to a required shape (JSON, SQL, citation format, etc.).\"\n  - label: \"Retrieval \u002F RAG faithfulness\"\n    description: \"Verify the answer is grounded in retrieved documents — no hallucinations beyond the retrieved context.\"\n  - label: \"Refactor \u002F regression test\"\n    description: \"Check whether a prompt or code change preserves observed behavior. Uses the dataset's expected_output as the ground-truth baseline.\"\n",[3259],{"type":45,"tag":76,"props":3260,"children":3261},{"__ignoreMap":404},[3262],{"type":51,"value":3257},{"type":45,"tag":54,"props":3264,"children":3265},{},[3266,3268,3273],{"type":51,"value":3267},"The user picks an option (which becomes the starter sentence), or picks \"Other\" and writes their own. Either way, the resulting string is the ",{"type":45,"tag":76,"props":3269,"children":3271},{"className":3270},[],[3272],{"type":51,"value":3236},{"type":51,"value":3274}," carried forward.",{"type":45,"tag":54,"props":3276,"children":3277},{},[3278,3283],{"type":45,"tag":58,"props":3279,"children":3280},{},[3281],{"type":51,"value":3282},"What to do with the purpose downstream",{"type":51,"value":3284}," — Claude reads it and reasons about effects, no static mapping:",{"type":45,"tag":235,"props":3286,"children":3287},{},[3288,3345,3370,3423,3441],{"type":45,"tag":70,"props":3289,"children":3290},{},[3291,3296,3298,3304,3305,3311,3313,3319,3321,3327,3329,3335,3337,3343],{"type":45,"tag":58,"props":3292,"children":3293},{},[3294],{"type":51,"value":3295},"In step 2.5 (introspection ranking)",{"type":51,"value":3297},": read the purpose; if it mentions tools \u002F agents → boost candidates with ",{"type":45,"tag":76,"props":3299,"children":3301},{"className":3300},[],[3302],{"type":51,"value":3303},"@LLMObs.agent",{"type":51,"value":431},{"type":45,"tag":76,"props":3306,"children":3308},{"className":3307},[],[3309],{"type":51,"value":3310},"@workflow",{"type":51,"value":3312},", LangChain ",{"type":45,"tag":76,"props":3314,"children":3316},{"className":3315},[],[3317],{"type":51,"value":3318},"ReActAgent",{"type":51,"value":3320},", or tool-using shapes. If it mentions retrieval \u002F RAG \u002F grounding → boost candidates that import vector stores or call ",{"type":45,"tag":76,"props":3322,"children":3324},{"className":3323},[],[3325],{"type":51,"value":3326},"retrieve",{"type":51,"value":3328},"\u002F",{"type":45,"tag":76,"props":3330,"children":3332},{"className":3331},[],[3333],{"type":51,"value":3334},"query_engine",{"type":51,"value":3336},". If it mentions schema \u002F JSON \u002F SQL → boost candidates that use ",{"type":45,"tag":76,"props":3338,"children":3340},{"className":3339},[],[3341],{"type":51,"value":3342},"response_format",{"type":51,"value":3344},", pydantic, or structured-output APIs. Apply judgment — there's no hardcoded mapping.",{"type":45,"tag":70,"props":3346,"children":3347},{},[3348,3353,3355,3360,3362,3368],{"type":45,"tag":58,"props":3349,"children":3350},{},[3351],{"type":51,"value":3352},"In step 2.5d (wrapper generation)",{"type":51,"value":3354},": if the purpose needs richer signal than just the final output (tool calls, retrieved docs, intermediate state), emit a wrapper that captures it ",{"type":45,"tag":2177,"props":3356,"children":3357},{},[3358],{"type":51,"value":3359},"if the user's function exposes it",{"type":51,"value":3361},". Otherwise emit a ",{"type":45,"tag":76,"props":3363,"children":3365},{"className":3364},[],[3366],{"type":51,"value":3367},"# Note:",{"type":51,"value":3369}," comment explaining what the wrapper would ideally return and why a richer return shape would help — do not refactor the user's actual function.",{"type":45,"tag":70,"props":3371,"children":3372},{},[3373,3378,3380,3385,3387,3392,3394,3400,3401,3407,3409,3414,3416,3421],{"type":45,"tag":58,"props":3374,"children":3375},{},[3376],{"type":51,"value":3377},"In step 3 (evaluator template)",{"type":51,"value":3379},": read the purpose and seed the evaluator list to match. Accuracy → standard ",{"type":45,"tag":76,"props":3381,"children":3383},{"className":3382},[],[3384],{"type":51,"value":2316},{"type":51,"value":3386}," + LLMJudge. Tool calls → an LLMJudge with a tool-correctness rubric, or a ",{"type":45,"tag":76,"props":3388,"children":3390},{"className":3389},[],[3391],{"type":51,"value":761},{"type":51,"value":3393}," if the user has one configured. Schema → ",{"type":45,"tag":76,"props":3395,"children":3397},{"className":3396},[],[3398],{"type":51,"value":3399},"JSONEvaluator(required_keys=[...])",{"type":51,"value":835},{"type":45,"tag":76,"props":3402,"children":3404},{"className":3403},[],[3405],{"type":51,"value":3406},"RegexMatchEvaluator",{"type":51,"value":3408},". Retrieval → an LLMJudge for groundedness. Regression → ",{"type":45,"tag":76,"props":3410,"children":3412},{"className":3411},[],[3413],{"type":51,"value":2316},{"type":51,"value":3415}," + a near-match check. The ",{"type":45,"tag":76,"props":3417,"children":3419},{"className":3418},[],[3420],{"type":51,"value":713},{"type":51,"value":3422}," flag (function \u002F class \u002F remote) still picks the SURFACE; the purpose picks the SEMANTICS.",{"type":45,"tag":70,"props":3424,"children":3425},{},[3426,3431,3433,3439],{"type":45,"tag":58,"props":3427,"children":3428},{},[3429],{"type":51,"value":3430},"In step 4 (file emission)",{"type":51,"value":3432},": include the purpose as a ",{"type":45,"tag":76,"props":3434,"children":3436},{"className":3435},[],[3437],{"type":51,"value":3438},"# Purpose:",{"type":51,"value":3440}," comment in the file's docstring header so the user (and reviewers) can see what the experiment was scaffolded for.",{"type":45,"tag":70,"props":3442,"children":3443},{},[3444,3449],{"type":45,"tag":58,"props":3445,"children":3446},{},[3447],{"type":51,"value":3448},"In Output (next-steps)",{"type":51,"value":3450},": surface the resolved purpose in the next-steps block so it's part of the run-summary record.",{"type":45,"tag":54,"props":3452,"children":3453},{},[3454,3459,3461,3466],{"type":45,"tag":58,"props":3455,"children":3456},{},[3457],{"type":51,"value":3458},"If the user is silent \u002F picks the default option without notes",{"type":51,"value":3460},", use the starter sentence as-is. Never block on this step — there should always be a ",{"type":45,"tag":76,"props":3462,"children":3464},{"className":3463},[],[3465],{"type":51,"value":3236},{"type":51,"value":3467}," string moving forward, even if it's a generic \"Output accuracy \u002F answer quality\".",{"type":45,"tag":311,"props":3469,"children":3470},{},[],{"type":45,"tag":54,"props":3472,"children":3473},{},[3474,3476,3481,3483,3488],{"type":51,"value":3475},"2.5. ",{"type":45,"tag":58,"props":3477,"children":3478},{},[3479],{"type":51,"value":3480},"Discover the task function from the user's application code.",{"type":51,"value":3482}," This step replaces the old \"placeholder ",{"type":45,"tag":76,"props":3484,"children":3486},{"className":3485},[],[3487],{"type":51,"value":877},{"type":51,"value":3489},"\" behavior — an onboarding-grade experiment needs to actually exercise the user's real LLM logic. The skill must do the work, not push it onto the user.",{"type":45,"tag":54,"props":3491,"children":3492},{},[3493,3498,3499,3504,3506,3512],{"type":45,"tag":58,"props":3494,"children":3495},{},[3496],{"type":51,"value":3497},"Skip this entire step if",{"type":51,"value":2411},{"type":45,"tag":76,"props":3500,"children":3502},{"className":3501},[],[3503],{"type":51,"value":891},{"type":51,"value":3505}," is set, OR if ",{"type":45,"tag":76,"props":3507,"children":3509},{"className":3508},[],[3510],{"type":51,"value":3511},"--task-source \u003Cmodule>:\u003Cfunction>",{"type":51,"value":3513}," is set (in which case use the provided source directly — jump to substep 2.5d). Otherwise:",{"type":45,"tag":54,"props":3515,"children":3516},{},[3517,3522,3524,3530,3532,3537,3538,3543,3544,3549,3550,3555,3557,3562,3564,3568,3570,3576,3577,3583,3584,3590,3591,3597,3598,3604,3605,3611,3612,3618,3619,3625,3626,3632,3633,3639,3641,3647,3649,3654,3655,3661],{"type":45,"tag":58,"props":3518,"children":3519},{},[3520],{"type":51,"value":3521},"2.5a — Resolve the app root.",{"type":51,"value":3523}," If ",{"type":45,"tag":76,"props":3525,"children":3527},{"className":3526},[],[3528],{"type":51,"value":3529},"--app-root \u003Cpath>",{"type":51,"value":3531}," was supplied, use it. Otherwise use the directory containing whichever of these resolved during step 1 (in order): ",{"type":45,"tag":76,"props":3533,"children":3535},{"className":3534},[],[3536],{"type":51,"value":932},{"type":51,"value":431},{"type":45,"tag":76,"props":3539,"children":3541},{"className":3540},[],[3542],{"type":51,"value":940},{"type":51,"value":431},{"type":45,"tag":76,"props":3545,"children":3547},{"className":3546},[],[3548],{"type":51,"value":947},{"type":51,"value":431},{"type":45,"tag":76,"props":3551,"children":3553},{"className":3552},[],[3554],{"type":51,"value":2670},{"type":51,"value":3556},". If none of those resolved, use the current working directory. ",{"type":45,"tag":58,"props":3558,"children":3559},{},[3560],{"type":51,"value":3561},"Hard cap",{"type":51,"value":3563}," the scan to that directory tree; do ",{"type":45,"tag":58,"props":3565,"children":3566},{},[3567],{"type":51,"value":2490},{"type":51,"value":3569}," traverse ",{"type":45,"tag":76,"props":3571,"children":3573},{"className":3572},[],[3574],{"type":51,"value":3575},"node_modules",{"type":51,"value":431},{"type":45,"tag":76,"props":3578,"children":3580},{"className":3579},[],[3581],{"type":51,"value":3582},".venv",{"type":51,"value":431},{"type":45,"tag":76,"props":3585,"children":3587},{"className":3586},[],[3588],{"type":51,"value":3589},"venv",{"type":51,"value":431},{"type":45,"tag":76,"props":3592,"children":3594},{"className":3593},[],[3595],{"type":51,"value":3596},"__pycache__",{"type":51,"value":431},{"type":45,"tag":76,"props":3599,"children":3601},{"className":3600},[],[3602],{"type":51,"value":3603},".git",{"type":51,"value":431},{"type":45,"tag":76,"props":3606,"children":3608},{"className":3607},[],[3609],{"type":51,"value":3610},"dist",{"type":51,"value":431},{"type":45,"tag":76,"props":3613,"children":3615},{"className":3614},[],[3616],{"type":51,"value":3617},"build",{"type":51,"value":431},{"type":45,"tag":76,"props":3620,"children":3622},{"className":3621},[],[3623],{"type":51,"value":3624},"target",{"type":51,"value":431},{"type":45,"tag":76,"props":3627,"children":3629},{"className":3628},[],[3630],{"type":51,"value":3631},"vendor",{"type":51,"value":431},{"type":45,"tag":76,"props":3634,"children":3636},{"className":3635},[],[3637],{"type":51,"value":3638},"third_party",{"type":51,"value":3640},", or any directory matched by ",{"type":45,"tag":76,"props":3642,"children":3644},{"className":3643},[],[3645],{"type":51,"value":3646},".gitignore",{"type":51,"value":3648}," if present. Refuse to scan if the resolved root is ",{"type":45,"tag":76,"props":3650,"children":3652},{"className":3651},[],[3653],{"type":51,"value":3328},{"type":51,"value":835},{"type":45,"tag":76,"props":3656,"children":3658},{"className":3657},[],[3659],{"type":51,"value":3660},"~",{"type":51,"value":3662}," — that means resolution failed; treat as no candidate found.",{"type":45,"tag":54,"props":3664,"children":3665},{},[3666,3671],{"type":45,"tag":58,"props":3667,"children":3668},{},[3669],{"type":51,"value":3670},"2.5b — Candidate discovery.",{"type":51,"value":3672}," Use Grep \u002F Bash to find Python files inside the app root and identify call sites of these LLM SDKs (the union of common LLM clients):",{"type":45,"tag":441,"props":3674,"children":3675},{},[3676,3692],{"type":45,"tag":445,"props":3677,"children":3678},{},[3679],{"type":45,"tag":449,"props":3680,"children":3681},{},[3682,3687],{"type":45,"tag":453,"props":3683,"children":3684},{},[3685],{"type":51,"value":3686},"Module \u002F call pattern",{"type":45,"tag":453,"props":3688,"children":3689},{},[3690],{"type":51,"value":3691},"Signal",{"type":45,"tag":469,"props":3693,"children":3694},{},[3695,3726,3750,3781,3805,3843,3874,3891,3908],{"type":45,"tag":449,"props":3696,"children":3697},{},[3698,3721],{"type":45,"tag":476,"props":3699,"children":3700},{},[3701,3707,3708,3714,3715],{"type":45,"tag":76,"props":3702,"children":3704},{"className":3703},[],[3705],{"type":51,"value":3706},"openai.ChatCompletion.create",{"type":51,"value":431},{"type":45,"tag":76,"props":3709,"children":3711},{"className":3710},[],[3712],{"type":51,"value":3713},"openai.chat.completions.create",{"type":51,"value":431},{"type":45,"tag":76,"props":3716,"children":3718},{"className":3717},[],[3719],{"type":51,"value":3720},"client.chat.completions.create",{"type":45,"tag":476,"props":3722,"children":3723},{},[3724],{"type":51,"value":3725},"OpenAI chat",{"type":45,"tag":449,"props":3727,"children":3728},{},[3729,3745],{"type":45,"tag":476,"props":3730,"children":3731},{},[3732,3738,3739],{"type":45,"tag":76,"props":3733,"children":3735},{"className":3734},[],[3736],{"type":51,"value":3737},"openai.completions.create",{"type":51,"value":431},{"type":45,"tag":76,"props":3740,"children":3742},{"className":3741},[],[3743],{"type":51,"value":3744},"client.completions.create",{"type":45,"tag":476,"props":3746,"children":3747},{},[3748],{"type":51,"value":3749},"OpenAI text",{"type":45,"tag":449,"props":3751,"children":3752},{},[3753,3776],{"type":45,"tag":476,"props":3754,"children":3755},{},[3756,3762,3763,3769,3770],{"type":45,"tag":76,"props":3757,"children":3759},{"className":3758},[],[3760],{"type":51,"value":3761},"anthropic.messages.create",{"type":51,"value":431},{"type":45,"tag":76,"props":3764,"children":3766},{"className":3765},[],[3767],{"type":51,"value":3768},"client.messages.create",{"type":51,"value":431},{"type":45,"tag":76,"props":3771,"children":3773},{"className":3772},[],[3774],{"type":51,"value":3775},"Anthropic(...).messages.create",{"type":45,"tag":476,"props":3777,"children":3778},{},[3779],{"type":51,"value":3780},"Anthropic",{"type":45,"tag":449,"props":3782,"children":3783},{},[3784,3800],{"type":45,"tag":476,"props":3785,"children":3786},{},[3787,3793,3794],{"type":45,"tag":76,"props":3788,"children":3790},{"className":3789},[],[3791],{"type":51,"value":3792},"litellm.completion",{"type":51,"value":431},{"type":45,"tag":76,"props":3795,"children":3797},{"className":3796},[],[3798],{"type":51,"value":3799},"litellm.acompletion",{"type":45,"tag":476,"props":3801,"children":3802},{},[3803],{"type":51,"value":3804},"LiteLLM (router)",{"type":45,"tag":449,"props":3806,"children":3807},{},[3808,3838],{"type":45,"tag":476,"props":3809,"children":3810},{},[3811,3817,3818,3824,3825,3831,3832],{"type":45,"tag":76,"props":3812,"children":3814},{"className":3813},[],[3815],{"type":51,"value":3816},"langchain.*.invoke",{"type":51,"value":431},{"type":45,"tag":76,"props":3819,"children":3821},{"className":3820},[],[3822],{"type":51,"value":3823},"ChatOpenAI(...)",{"type":51,"value":431},{"type":45,"tag":76,"props":3826,"children":3828},{"className":3827},[],[3829],{"type":51,"value":3830},"ChatAnthropic(...)",{"type":51,"value":431},{"type":45,"tag":76,"props":3833,"children":3835},{"className":3834},[],[3836],{"type":51,"value":3837},"LLMChain(...)",{"type":45,"tag":476,"props":3839,"children":3840},{},[3841],{"type":51,"value":3842},"LangChain",{"type":45,"tag":449,"props":3844,"children":3845},{},[3846,3869],{"type":45,"tag":476,"props":3847,"children":3848},{},[3849,3855,3856,3862,3863],{"type":45,"tag":76,"props":3850,"children":3852},{"className":3851},[],[3853],{"type":51,"value":3854},"from llama_index",{"type":51,"value":431},{"type":45,"tag":76,"props":3857,"children":3859},{"className":3858},[],[3860],{"type":51,"value":3861},"as_query_engine",{"type":51,"value":431},{"type":45,"tag":76,"props":3864,"children":3866},{"className":3865},[],[3867],{"type":51,"value":3868},"as_chat_engine",{"type":45,"tag":476,"props":3870,"children":3871},{},[3872],{"type":51,"value":3873},"LlamaIndex",{"type":45,"tag":449,"props":3875,"children":3876},{},[3877,3886],{"type":45,"tag":476,"props":3878,"children":3879},{},[3880],{"type":45,"tag":76,"props":3881,"children":3883},{"className":3882},[],[3884],{"type":51,"value":3885},"google.generativeai.GenerativeModel(...).generate_content",{"type":45,"tag":476,"props":3887,"children":3888},{},[3889],{"type":51,"value":3890},"Vertex\u002FGemini",{"type":45,"tag":449,"props":3892,"children":3893},{},[3894,3903],{"type":45,"tag":476,"props":3895,"children":3896},{},[3897],{"type":45,"tag":76,"props":3898,"children":3900},{"className":3899},[],[3901],{"type":51,"value":3902},"boto3.client(\"bedrock-runtime\").invoke_model",{"type":45,"tag":476,"props":3904,"children":3905},{},[3906],{"type":51,"value":3907},"AWS Bedrock",{"type":45,"tag":449,"props":3909,"children":3910},{},[3911,3953],{"type":45,"tag":476,"props":3912,"children":3913},{},[3914,3920,3921,3926,3927,3933,3934,3940,3941,3946,3947],{"type":45,"tag":76,"props":3915,"children":3917},{"className":3916},[],[3918],{"type":51,"value":3919},"@LLMObs.llm",{"type":51,"value":431},{"type":45,"tag":76,"props":3922,"children":3924},{"className":3923},[],[3925],{"type":51,"value":3303},{"type":51,"value":431},{"type":45,"tag":76,"props":3928,"children":3930},{"className":3929},[],[3931],{"type":51,"value":3932},"@LLMObs.workflow",{"type":51,"value":431},{"type":45,"tag":76,"props":3935,"children":3937},{"className":3936},[],[3938],{"type":51,"value":3939},"@LLMObs.task",{"type":51,"value":431},{"type":45,"tag":76,"props":3942,"children":3944},{"className":3943},[],[3945],{"type":51,"value":3310},{"type":51,"value":431},{"type":45,"tag":76,"props":3948,"children":3950},{"className":3949},[],[3951],{"type":51,"value":3952},"@agent",{"type":45,"tag":476,"props":3954,"children":3955},{},[3956],{"type":51,"value":3957},"Already instrumented",{"type":45,"tag":54,"props":3959,"children":3960},{},[3961,3963,3968,3970,3976,3977,3983,3985,3990,3992,3998],{"type":51,"value":3962},"For each match, walk ",{"type":45,"tag":58,"props":3964,"children":3965},{},[3966],{"type":51,"value":3967},"up",{"type":51,"value":3969}," to the enclosing function (",{"type":45,"tag":76,"props":3971,"children":3973},{"className":3972},[],[3974],{"type":51,"value":3975},"def",{"type":51,"value":934},{"type":45,"tag":76,"props":3978,"children":3980},{"className":3979},[],[3981],{"type":51,"value":3982},"async def",{"type":51,"value":3984},") — call it the ",{"type":45,"tag":2177,"props":3986,"children":3987},{},[3988],{"type":51,"value":3989},"call-site function",{"type":51,"value":3991},". Record ",{"type":45,"tag":76,"props":3993,"children":3995},{"className":3994},[],[3996],{"type":51,"value":3997},"{file, line, function_name, is_async, signature, enclosing_class}",{"type":51,"value":110},{"type":45,"tag":54,"props":4000,"children":4001},{},[4002,4007,4009,4015,4016,4022,4023,4029,4030,4036,4037,4043,4045,4051,4053,4058],{"type":45,"tag":58,"props":4003,"children":4004},{},[4005],{"type":51,"value":4006},"Skip",{"type":51,"value":4008}," files matching ",{"type":45,"tag":76,"props":4010,"children":4012},{"className":4011},[],[4013],{"type":51,"value":4014},"test_*.py",{"type":51,"value":431},{"type":45,"tag":76,"props":4017,"children":4019},{"className":4018},[],[4020],{"type":51,"value":4021},"*_test.py",{"type":51,"value":431},{"type":45,"tag":76,"props":4024,"children":4026},{"className":4025},[],[4027],{"type":51,"value":4028},"tests\u002F**",{"type":51,"value":431},{"type":45,"tag":76,"props":4031,"children":4033},{"className":4032},[],[4034],{"type":51,"value":4035},"conftest.py",{"type":51,"value":431},{"type":45,"tag":76,"props":4038,"children":4040},{"className":4039},[],[4041],{"type":51,"value":4042},"*_fixtures.py",{"type":51,"value":4044},". Skip private helpers (",{"type":45,"tag":76,"props":4046,"children":4048},{"className":4047},[],[4049],{"type":51,"value":4050},"def _foo",{"type":51,"value":4052}," where the function is the ",{"type":45,"tag":2177,"props":4054,"children":4055},{},[4056],{"type":51,"value":4057},"only",{"type":51,"value":4059}," one in the file and looks like a utility — judge by whether it has a typed string input).",{"type":45,"tag":54,"props":4061,"children":4062},{},[4063,4068],{"type":45,"tag":58,"props":4064,"children":4065},{},[4066],{"type":51,"value":4067},"2.5c — Score and rank candidates.",{"type":51,"value":4069}," Score each candidate; higher is more likely to be the \"core\" entry point:",{"type":45,"tag":441,"props":4071,"children":4072},{},[4073,4088],{"type":45,"tag":445,"props":4074,"children":4075},{},[4076],{"type":45,"tag":449,"props":4077,"children":4078},{},[4079,4083],{"type":45,"tag":453,"props":4080,"children":4081},{},[4082],{"type":51,"value":3691},{"type":45,"tag":453,"props":4084,"children":4085},{},[4086],{"type":51,"value":4087},"Score delta",{"type":45,"tag":469,"props":4089,"children":4090},{},[4091,4213,4240,4284,4300,4321,4371,4397,4433],{"type":45,"tag":449,"props":4092,"children":4093},{},[4094,4205],{"type":45,"tag":476,"props":4095,"children":4096},{},[4097,4099,4105,4106,4112,4113,4119,4120,4126,4127,4133,4134,4140,4141,4147,4148,4154,4155,4161,4162,4168,4169,4175,4176,4182,4183,4189,4190,4196,4197,4203],{"type":51,"value":4098},"Function name in {",{"type":45,"tag":76,"props":4100,"children":4102},{"className":4101},[],[4103],{"type":51,"value":4104},"generate",{"type":51,"value":431},{"type":45,"tag":76,"props":4107,"children":4109},{"className":4108},[],[4110],{"type":51,"value":4111},"chat",{"type":51,"value":431},{"type":45,"tag":76,"props":4114,"children":4116},{"className":4115},[],[4117],{"type":51,"value":4118},"complete",{"type":51,"value":431},{"type":45,"tag":76,"props":4121,"children":4123},{"className":4122},[],[4124],{"type":51,"value":4125},"respond",{"type":51,"value":431},{"type":45,"tag":76,"props":4128,"children":4130},{"className":4129},[],[4131],{"type":51,"value":4132},"answer",{"type":51,"value":431},{"type":45,"tag":76,"props":4135,"children":4137},{"className":4136},[],[4138],{"type":51,"value":4139},"handle_request",{"type":51,"value":431},{"type":45,"tag":76,"props":4142,"children":4144},{"className":4143},[],[4145],{"type":51,"value":4146},"process_query",{"type":51,"value":431},{"type":45,"tag":76,"props":4149,"children":4151},{"className":4150},[],[4152],{"type":51,"value":4153},"process_message",{"type":51,"value":431},{"type":45,"tag":76,"props":4156,"children":4158},{"className":4157},[],[4159],{"type":51,"value":4160},"run",{"type":51,"value":431},{"type":45,"tag":76,"props":4163,"children":4165},{"className":4164},[],[4166],{"type":51,"value":4167},"predict",{"type":51,"value":431},{"type":45,"tag":76,"props":4170,"children":4172},{"className":4171},[],[4173],{"type":51,"value":4174},"infer",{"type":51,"value":431},{"type":45,"tag":76,"props":4177,"children":4179},{"className":4178},[],[4180],{"type":51,"value":4181},"call_llm",{"type":51,"value":431},{"type":45,"tag":76,"props":4184,"children":4186},{"className":4185},[],[4187],{"type":51,"value":4188},"query",{"type":51,"value":431},{"type":45,"tag":76,"props":4191,"children":4193},{"className":4192},[],[4194],{"type":51,"value":4195},"agent_loop",{"type":51,"value":431},{"type":45,"tag":76,"props":4198,"children":4200},{"className":4199},[],[4201],{"type":51,"value":4202},"main",{"type":51,"value":4204},"}",{"type":45,"tag":476,"props":4206,"children":4207},{},[4208],{"type":45,"tag":58,"props":4209,"children":4210},{},[4211],{"type":51,"value":4212},"+5",{"type":45,"tag":449,"props":4214,"children":4215},{},[4216,4232],{"type":45,"tag":476,"props":4217,"children":4218},{},[4219,4221,4226,4227],{"type":51,"value":4220},"Function takes exactly one or two parameters, at least one being a ",{"type":45,"tag":76,"props":4222,"children":4224},{"className":4223},[],[4225],{"type":51,"value":2132},{"type":51,"value":835},{"type":45,"tag":76,"props":4228,"children":4230},{"className":4229},[],[4231],{"type":51,"value":2047},{"type":45,"tag":476,"props":4233,"children":4234},{},[4235],{"type":45,"tag":58,"props":4236,"children":4237},{},[4238],{"type":51,"value":4239},"+3",{"type":45,"tag":449,"props":4241,"children":4242},{},[4243,4268],{"type":45,"tag":476,"props":4244,"children":4245},{},[4246,4248,4254,4255,4260,4261,4266],{"type":51,"value":4247},"Function is decorated with any ",{"type":45,"tag":76,"props":4249,"children":4251},{"className":4250},[],[4252],{"type":51,"value":4253},"@LLMObs.*",{"type":51,"value":934},{"type":45,"tag":76,"props":4256,"children":4258},{"className":4257},[],[4259],{"type":51,"value":3310},{"type":51,"value":934},{"type":45,"tag":76,"props":4262,"children":4264},{"className":4263},[],[4265],{"type":51,"value":3952},{"type":51,"value":4267}," decorator",{"type":45,"tag":476,"props":4269,"children":4270},{},[4271,4275,4277,4282],{"type":45,"tag":58,"props":4272,"children":4273},{},[4274],{"type":51,"value":4212},{"type":51,"value":4276}," (this is the ",{"type":45,"tag":2177,"props":4278,"children":4279},{},[4280],{"type":51,"value":4281},"intended",{"type":51,"value":4283}," instrumentation point)",{"type":45,"tag":449,"props":4285,"children":4286},{},[4287,4292],{"type":45,"tag":476,"props":4288,"children":4289},{},[4290],{"type":51,"value":4291},"Function is at the top of its module (first non-import block)",{"type":45,"tag":476,"props":4293,"children":4294},{},[4295],{"type":45,"tag":58,"props":4296,"children":4297},{},[4298],{"type":51,"value":4299},"+2",{"type":45,"tag":449,"props":4301,"children":4302},{},[4303,4314],{"type":45,"tag":476,"props":4304,"children":4305},{},[4306,4308],{"type":51,"value":4307},"Module path matches ",{"type":45,"tag":76,"props":4309,"children":4311},{"className":4310},[],[4312],{"type":51,"value":4313},"**\u002F{main,app,api,handlers,server,routes,agent,bot,chat}.py",{"type":45,"tag":476,"props":4315,"children":4316},{},[4317],{"type":45,"tag":58,"props":4318,"children":4319},{},[4320],{"type":51,"value":4239},{"type":45,"tag":449,"props":4322,"children":4323},{},[4324,4364],{"type":45,"tag":476,"props":4325,"children":4326},{},[4327,4329,4335,4337,4343,4344,4350,4351,4357,4358],{"type":51,"value":4328},"Function is a class method (",{"type":45,"tag":76,"props":4330,"children":4332},{"className":4331},[],[4333],{"type":51,"value":4334},"self",{"type":51,"value":4336}," first arg) but the class name matches ",{"type":45,"tag":76,"props":4338,"children":4340},{"className":4339},[],[4341],{"type":51,"value":4342},"*Agent",{"type":51,"value":431},{"type":45,"tag":76,"props":4345,"children":4347},{"className":4346},[],[4348],{"type":51,"value":4349},"*Bot",{"type":51,"value":431},{"type":45,"tag":76,"props":4352,"children":4354},{"className":4353},[],[4355],{"type":51,"value":4356},"*Handler",{"type":51,"value":431},{"type":45,"tag":76,"props":4359,"children":4361},{"className":4360},[],[4362],{"type":51,"value":4363},"*Service",{"type":45,"tag":476,"props":4365,"children":4366},{},[4367],{"type":45,"tag":58,"props":4368,"children":4369},{},[4370],{"type":51,"value":4299},{"type":45,"tag":449,"props":4372,"children":4373},{},[4374,4387],{"type":45,"tag":476,"props":4375,"children":4376},{},[4377,4379,4385],{"type":51,"value":4378},"Function name starts with ",{"type":45,"tag":76,"props":4380,"children":4382},{"className":4381},[],[4383],{"type":51,"value":4384},"_",{"type":51,"value":4386}," and module has other non-underscore candidates",{"type":45,"tag":476,"props":4388,"children":4389},{},[4390,4395],{"type":45,"tag":58,"props":4391,"children":4392},{},[4393],{"type":51,"value":4394},"−3",{"type":51,"value":4396}," (likely a helper)",{"type":45,"tag":449,"props":4398,"children":4399},{},[4400,4425],{"type":45,"tag":476,"props":4401,"children":4402},{},[4403,4405,4411,4412,4418,4419],{"type":51,"value":4404},"Function file is under ",{"type":45,"tag":76,"props":4406,"children":4408},{"className":4407},[],[4409],{"type":51,"value":4410},"examples\u002F",{"type":51,"value":431},{"type":45,"tag":76,"props":4413,"children":4415},{"className":4414},[],[4416],{"type":51,"value":4417},"scripts\u002F",{"type":51,"value":431},{"type":45,"tag":76,"props":4420,"children":4422},{"className":4421},[],[4423],{"type":51,"value":4424},"notebooks\u002F",{"type":45,"tag":476,"props":4426,"children":4427},{},[4428],{"type":45,"tag":58,"props":4429,"children":4430},{},[4431],{"type":51,"value":4432},"−2",{"type":45,"tag":449,"props":4434,"children":4435},{},[4436,4441],{"type":45,"tag":476,"props":4437,"children":4438},{},[4439],{"type":51,"value":4440},"Function uses LLM SDK at multiple lines (looks like a multi-step orchestration, not just a single call)",{"type":45,"tag":476,"props":4442,"children":4443},{},[4444],{"type":45,"tag":58,"props":4445,"children":4446},{},[4447],{"type":51,"value":4448},"+1",{"type":45,"tag":54,"props":4450,"children":4451},{},[4452,4457],{"type":45,"tag":58,"props":4453,"children":4454},{},[4455],{"type":51,"value":4456},"Apply the experiment purpose from step 2.4 as an additional soft bias on top of these scores.",{"type":51,"value":4458}," Read the purpose string and bump candidates whose shape matches what the user is trying to test — agent \u002F tool-using functions for tool-correctness purposes; retrieval \u002F vector-store users for RAG \u002F grounding purposes; structured-output users for schema purposes; etc. Use judgment, not a hardcoded mapping. A purpose-aligned candidate should typically beat a generically-better-named one (+3 to +5 effective bias is reasonable). See step 2.4 \"What to do with the purpose downstream\" for examples.",{"type":45,"tag":54,"props":4460,"children":4461},{},[4462,4464,4469,4471,4476],{"type":51,"value":4463},"Pick the ",{"type":45,"tag":58,"props":4465,"children":4466},{},[4467],{"type":51,"value":4468},"top 3",{"type":51,"value":4470}," candidates and present them to the user as a single short prompt (no ",{"type":45,"tag":76,"props":4472,"children":4474},{"className":4473},[],[4475],{"type":51,"value":1042},{"type":51,"value":4477}," — just a checkpoint prompt in chat):",{"type":45,"tag":396,"props":4479,"children":4482},{"className":4480,"code":4481,"language":51},[399],"## Task function discovery\n\nScanned `\u003Capp_root>` and found \u003CN> LLM-calling functions. Top candidates:\n\n1. `\u003Cmodule.path>:\u003Cfunction_name>`   score \u003CS>   (\u003Cfile>:\u003Cline>, args=\u003Csig>)\n2. ...\n3. ...\n\nI'll wire candidate **1** as the experiment's task function unless you say otherwise.\nReply with the number to pick a different candidate, or \"placeholder\" to emit a\ngeneric placeholder task instead.\n",[4483],{"type":45,"tag":76,"props":4484,"children":4485},{"__ignoreMap":404},[4486],{"type":51,"value":4481},{"type":45,"tag":54,"props":4488,"children":4489},{},[4490,4492,4497,4499,4505],{"type":51,"value":4491},"Wait for confirmation. If the user is silent \u002F says \"go\" \u002F says \"1\", use #1. If they pick another number, use that. If they say \"placeholder\", set ",{"type":45,"tag":76,"props":4493,"children":4495},{"className":4494},[],[4496],{"type":51,"value":891},{"type":51,"value":4498}," semantics and skip to step 3. If the user supplies a different ",{"type":45,"tag":76,"props":4500,"children":4502},{"className":4501},[],[4503],{"type":51,"value":4504},"module:function",{"type":51,"value":4506},", validate it exists, then use it.",{"type":45,"tag":54,"props":4508,"children":4509},{},[4510,4515,4517,4523],{"type":45,"tag":58,"props":4511,"children":4512},{},[4513],{"type":51,"value":4514},"If zero candidates were found",{"type":51,"value":4516},", do not block the user — print a one-line note (\"No LLM call sites detected under ",{"type":45,"tag":76,"props":4518,"children":4520},{"className":4519},[],[4521],{"type":51,"value":4522},"\u003Capp_root>",{"type":51,"value":4524},". Emitting a generic placeholder task — replace it before running.\") and fall through with placeholder semantics.",{"type":45,"tag":54,"props":4526,"children":4527},{},[4528,4533,4535,4541,4543,4548,4550,4555,4557,4562],{"type":45,"tag":58,"props":4529,"children":4530},{},[4531],{"type":51,"value":4532},"2.5d — Generate the task function wrapper.",{"type":51,"value":4534}," Once a target ",{"type":45,"tag":76,"props":4536,"children":4538},{"className":4537},[],[4539],{"type":51,"value":4540},"\u003Cmodule>:\u003Cfunction>",{"type":51,"value":4542}," is locked in, emit a real ",{"type":45,"tag":76,"props":4544,"children":4546},{"className":4545},[],[4547],{"type":51,"value":877},{"type":51,"value":4549}," that imports and calls the user's function. This is ",{"type":45,"tag":58,"props":4551,"children":4552},{},[4553],{"type":51,"value":4554},"section 4",{"type":51,"value":4556}," of the generated file (\"Task function\") and must NOT include ",{"type":45,"tag":76,"props":4558,"children":4560},{"className":4559},[],[4561],{"type":51,"value":907},{"type":51,"value":4563}," markers if introspection succeeded — the whole point is to remove that burden.",{"type":45,"tag":54,"props":4565,"children":4566},{},[4567],{"type":51,"value":4568},"Wrapper construction rules:",{"type":45,"tag":235,"props":4570,"children":4571},{},[4572,4612,4752,4799,4879],{"type":45,"tag":70,"props":4573,"children":4574},{},[4575,4579,4581,4587,4589,4594,4596,4602,4604,4610],{"type":45,"tag":58,"props":4576,"children":4577},{},[4578],{"type":51,"value":1114},{"type":51,"value":4580},": emit ",{"type":45,"tag":76,"props":4582,"children":4584},{"className":4583},[],[4585],{"type":51,"value":4586},"from \u003Cmodule> import \u003Cfunction_name>",{"type":51,"value":4588}," at the top of section 4 (NOT at the top of the file — keeping it local makes the section self-contained and easy to swap). If the function is a class method, also import the class and instantiate it lazily inside ",{"type":45,"tag":76,"props":4590,"children":4592},{"className":4591},[],[4593],{"type":51,"value":877},{"type":51,"value":4595}," (default ",{"type":45,"tag":76,"props":4597,"children":4599},{"className":4598},[],[4600],{"type":51,"value":4601},"__init__()",{"type":51,"value":4603},", with a ",{"type":45,"tag":76,"props":4605,"children":4607},{"className":4606},[],[4608],{"type":51,"value":4609},"# TODO: pass constructor args if needed",{"type":51,"value":4611}," comment only if the constructor takes required arguments).",{"type":45,"tag":70,"props":4613,"children":4614},{},[4615,4620,4622,4628,4630],{"type":45,"tag":58,"props":4616,"children":4617},{},[4618],{"type":51,"value":4619},"Signature adaptation",{"type":51,"value":4621},": the SDK's task function signature is ",{"type":45,"tag":76,"props":4623,"children":4625},{"className":4624},[],[4626],{"type":51,"value":4627},"task_fn(input_data: dict, config: dict) -> Any",{"type":51,"value":4629},". The user's function probably has a different signature. Build a small adapter:",{"type":45,"tag":235,"props":4631,"children":4632},{},[4633,4660,4680,4700],{"type":45,"tag":70,"props":4634,"children":4635},{},[4636,4638,4643,4645,4651,4653,4659],{"type":51,"value":4637},"If the user's function takes one parameter and the dataset's ",{"type":45,"tag":76,"props":4639,"children":4641},{"className":4640},[],[4642],{"type":51,"value":2893},{"type":51,"value":4644}," looks like ",{"type":45,"tag":76,"props":4646,"children":4648},{"className":4647},[],[4649],{"type":51,"value":4650},"{\"\u003Ckey>\": \u003Cvalue>}",{"type":51,"value":4652}," with one key, call ",{"type":45,"tag":76,"props":4654,"children":4656},{"className":4655},[],[4657],{"type":51,"value":4658},"\u003Cfunction_name>(input_data[\"\u003Ckey>\"])",{"type":51,"value":110},{"type":45,"tag":70,"props":4661,"children":4662},{},[4663,4665,4670,4672,4678],{"type":51,"value":4664},"If the user's function takes multiple parameters, map them by name from ",{"type":45,"tag":76,"props":4666,"children":4668},{"className":4667},[],[4669],{"type":51,"value":2893},{"type":51,"value":4671}," keys. If a name doesn't match, fall back to positional via ",{"type":45,"tag":76,"props":4673,"children":4675},{"className":4674},[],[4676],{"type":51,"value":4677},"input_data.values()",{"type":51,"value":4679}," order but emit a one-line comment flagging the assumption.",{"type":45,"tag":70,"props":4681,"children":4682},{},[4683,4685,4691,4693,4698],{"type":51,"value":4684},"If the user's function takes ",{"type":45,"tag":76,"props":4686,"children":4688},{"className":4687},[],[4689],{"type":51,"value":4690},"**kwargs",{"type":51,"value":4692}," or a single dict, pass ",{"type":45,"tag":76,"props":4694,"children":4696},{"className":4695},[],[4697],{"type":51,"value":2893},{"type":51,"value":4699}," through unmodified.",{"type":45,"tag":70,"props":4701,"children":4702},{},[4703,4705,4711,4713,4719,4721,4726,4728,4734,4736,4742,4744,4750],{"type":51,"value":4704},"If the user's function is ",{"type":45,"tag":76,"props":4706,"children":4708},{"className":4707},[],[4709],{"type":51,"value":4710},"async",{"type":51,"value":4712},", wrap with ",{"type":45,"tag":76,"props":4714,"children":4716},{"className":4715},[],[4717],{"type":51,"value":4718},"asyncio.run(...)",{"type":51,"value":4720}," inside a sync ",{"type":45,"tag":76,"props":4722,"children":4724},{"className":4723},[],[4725],{"type":51,"value":877},{"type":51,"value":4727}," (simplest path; lets ",{"type":45,"tag":76,"props":4729,"children":4731},{"className":4730},[],[4732],{"type":51,"value":4733},"LLMObs.experiment(...).run()",{"type":51,"value":4735}," stay sync). Alternative: emit an ",{"type":45,"tag":76,"props":4737,"children":4739},{"className":4738},[],[4740],{"type":51,"value":4741},"async def task_fn",{"type":51,"value":4743}," and use ",{"type":45,"tag":76,"props":4745,"children":4747},{"className":4746},[],[4748],{"type":51,"value":4749},"LLMObs.async_experiment(...)",{"type":51,"value":4751}," — only do this if the rest of the experiment is already async.",{"type":45,"tag":70,"props":4753,"children":4754},{},[4755,4760,4762,4768,4770,4775,4776,4782,4783,4789,4791,4797],{"type":45,"tag":58,"props":4756,"children":4757},{},[4758],{"type":51,"value":4759},"Config passthrough",{"type":51,"value":4761},": never silently drop the ",{"type":45,"tag":76,"props":4763,"children":4765},{"className":4764},[],[4766],{"type":51,"value":4767},"config",{"type":51,"value":4769}," argument. If the user's function takes a ",{"type":45,"tag":76,"props":4771,"children":4773},{"className":4772},[],[4774],{"type":51,"value":4767},{"type":51,"value":934},{"type":45,"tag":76,"props":4777,"children":4779},{"className":4778},[],[4780],{"type":51,"value":4781},"model",{"type":51,"value":934},{"type":45,"tag":76,"props":4784,"children":4786},{"className":4785},[],[4787],{"type":51,"value":4788},"temperature",{"type":51,"value":4790}," parameter, wire ",{"type":45,"tag":76,"props":4792,"children":4794},{"className":4793},[],[4795],{"type":51,"value":4796},"config.get(\"...\")",{"type":51,"value":4798}," into it.",{"type":45,"tag":70,"props":4800,"children":4801},{},[4802,4807,4809,4814,4816],{"type":45,"tag":58,"props":4803,"children":4804},{},[4805],{"type":51,"value":4806},"Comment header",{"type":51,"value":4808},": emit a comment block above ",{"type":45,"tag":76,"props":4810,"children":4812},{"className":4811},[],[4813],{"type":51,"value":877},{"type":51,"value":4815}," that names the source function, file, line, the experiment purpose (from step 2.4), and any adaptation choices, e.g.:",{"type":45,"tag":396,"props":4817,"children":4819},{"className":1307,"code":4818,"language":24,"meta":404,"style":404},"# Task function wired to: \u003Cmodule.path>:\u003Cfunction_name>\n#   source:  \u003Cfile>:\u003Cline>\n#   purpose: \u003Cone-line purpose from step 2.4>\n#   adapter: input_data[\"\u003Ckey>\"] -> \u003Cfunction_name>(\u003Ckey>=...)\n#\n# To experiment with prompt \u002F model variants without editing your app, inline\n# the call here instead of importing.\n",[4820],{"type":45,"tag":76,"props":4821,"children":4822},{"__ignoreMap":404},[4823,4831,4839,4847,4855,4863,4871],{"type":45,"tag":1314,"props":4824,"children":4825},{"class":1316,"line":1317},[4826],{"type":45,"tag":1314,"props":4827,"children":4828},{},[4829],{"type":51,"value":4830},"# Task function wired to: \u003Cmodule.path>:\u003Cfunction_name>\n",{"type":45,"tag":1314,"props":4832,"children":4833},{"class":1316,"line":1326},[4834],{"type":45,"tag":1314,"props":4835,"children":4836},{},[4837],{"type":51,"value":4838},"#   source:  \u003Cfile>:\u003Cline>\n",{"type":45,"tag":1314,"props":4840,"children":4841},{"class":1316,"line":1335},[4842],{"type":45,"tag":1314,"props":4843,"children":4844},{},[4845],{"type":51,"value":4846},"#   purpose: \u003Cone-line purpose from step 2.4>\n",{"type":45,"tag":1314,"props":4848,"children":4849},{"class":1316,"line":1344},[4850],{"type":45,"tag":1314,"props":4851,"children":4852},{},[4853],{"type":51,"value":4854},"#   adapter: input_data[\"\u003Ckey>\"] -> \u003Cfunction_name>(\u003Ckey>=...)\n",{"type":45,"tag":1314,"props":4856,"children":4857},{"class":1316,"line":1353},[4858],{"type":45,"tag":1314,"props":4859,"children":4860},{},[4861],{"type":51,"value":4862},"#\n",{"type":45,"tag":1314,"props":4864,"children":4865},{"class":1316,"line":1362},[4866],{"type":45,"tag":1314,"props":4867,"children":4868},{},[4869],{"type":51,"value":4870},"# To experiment with prompt \u002F model variants without editing your app, inline\n",{"type":45,"tag":1314,"props":4872,"children":4873},{"class":1316,"line":1371},[4874],{"type":45,"tag":1314,"props":4875,"children":4876},{},[4877],{"type":51,"value":4878},"# the call here instead of importing.\n",{"type":45,"tag":70,"props":4880,"children":4881},{},[4882,4887,4889,4894,4896,4901,4903],{"type":45,"tag":58,"props":4883,"children":4884},{},[4885],{"type":51,"value":4886},"Richer-return shape when the purpose demands it",{"type":51,"value":4888},": read the purpose from step 2.4. If validating just the final output string is too lossy for what the user is testing (e.g. they're checking tool-call selection — the final string says \"I'll book that for you\" but the ",{"type":45,"tag":2177,"props":4890,"children":4891},{},[4892],{"type":51,"value":4893},"interesting",{"type":51,"value":4895}," signal is which tool was called with which args), emit a wrapper that captures the richer signal ",{"type":45,"tag":58,"props":4897,"children":4898},{},[4899],{"type":51,"value":4900},"only if the user's function exposes it",{"type":51,"value":4902},". Two cases:",{"type":45,"tag":235,"props":4904,"children":4905},{},[4906,4953],{"type":45,"tag":70,"props":4907,"children":4908},{},[4909,4914,4915,4921,4923,4929,4931,4937,4939,4944,4946,4952],{"type":45,"tag":58,"props":4910,"children":4911},{},[4912],{"type":51,"value":4913},"The function already returns richer data",{"type":51,"value":2205},{"type":45,"tag":76,"props":4916,"children":4918},{"className":4917},[],[4919],{"type":51,"value":4920},"{\"output\": str, \"tool_calls\": [...]}",{"type":51,"value":4922},", or a LangChain ",{"type":45,"tag":76,"props":4924,"children":4926},{"className":4925},[],[4927],{"type":51,"value":4928},"AgentExecutor.invoke()",{"type":51,"value":4930}," result with ",{"type":45,"tag":76,"props":4932,"children":4934},{"className":4933},[],[4935],{"type":51,"value":4936},"intermediate_steps",{"type":51,"value":4938},"): wire ",{"type":45,"tag":76,"props":4940,"children":4942},{"className":4941},[],[4943],{"type":51,"value":877},{"type":51,"value":4945}," to return that shape directly. Evaluators in step 3 will receive it as ",{"type":45,"tag":76,"props":4947,"children":4949},{"className":4948},[],[4950],{"type":51,"value":4951},"output_data",{"type":51,"value":110},{"type":45,"tag":70,"props":4954,"children":4955},{},[4956,4961,4963,4968,4970,4975,4977,4997,4999,5004],{"type":45,"tag":58,"props":4957,"children":4958},{},[4959],{"type":51,"value":4960},"The function returns just a string",{"type":51,"value":4962}," but the purpose needs more: emit a ",{"type":45,"tag":76,"props":4964,"children":4966},{"className":4965},[],[4967],{"type":51,"value":3367},{"type":51,"value":4969}," comment above ",{"type":45,"tag":76,"props":4971,"children":4973},{"className":4972},[],[4974],{"type":51,"value":877},{"type":51,"value":4976}," explaining what richer shape would help, and how the user could expose it (e.g. ",{"type":45,"tag":2177,"props":4978,"children":4979},{},[4980,4982,4988,4990,4995],{"type":51,"value":4981},"\"To capture tool calls for evaluation, refactor ",{"type":45,"tag":76,"props":4983,"children":4985},{"className":4984},[],[4986],{"type":51,"value":4987},"respond(query)",{"type":51,"value":4989}," to also return the ",{"type":45,"tag":76,"props":4991,"children":4993},{"className":4992},[],[4994],{"type":51,"value":4936},{"type":51,"value":4996}," from your AgentExecutor.\"",{"type":51,"value":4998},"). ",{"type":45,"tag":58,"props":5000,"children":5001},{},[5002],{"type":51,"value":5003},"Do not refactor the user's function",{"type":51,"value":5005}," — emit a note, ship the simple wrapper, let the user choose.",{"type":45,"tag":54,"props":5007,"children":5008},{},[5009],{"type":51,"value":5010},"Never invent a richer return shape that the user's function doesn't actually provide. The wrapper is plumbing, not surgery.",{"type":45,"tag":235,"props":5012,"children":5013},{},[5014,5182],{"type":45,"tag":70,"props":5015,"children":5016},{},[5017,5022,5024,5030,5032],{"type":45,"tag":58,"props":5018,"children":5019},{},[5020],{"type":51,"value":5021},"Side-effect warning",{"type":51,"value":5023},": scan the chosen function (and its immediate calls within the same module) for these patterns. If any are present, print a ",{"type":45,"tag":76,"props":5025,"children":5027},{"className":5026},[],[5028],{"type":51,"value":5029},"WARNING:",{"type":51,"value":5031}," line in the next-steps output, but do NOT alter the import:",{"type":45,"tag":441,"props":5033,"children":5034},{},[5035,5051],{"type":45,"tag":445,"props":5036,"children":5037},{},[5038],{"type":45,"tag":449,"props":5039,"children":5040},{},[5041,5046],{"type":45,"tag":453,"props":5042,"children":5043},{},[5044],{"type":51,"value":5045},"Pattern",{"type":45,"tag":453,"props":5047,"children":5048},{},[5049],{"type":51,"value":5050},"Warning",{"type":45,"tag":469,"props":5052,"children":5053},{},[5054,5073,5106,5155],{"type":45,"tag":449,"props":5055,"children":5056},{},[5057,5068],{"type":45,"tag":476,"props":5058,"children":5059},{},[5060,5066],{"type":45,"tag":76,"props":5061,"children":5063},{"className":5062},[],[5064],{"type":51,"value":5065},"os.environ[...]",{"type":51,"value":5067}," reads beyond LLM API keys",{"type":45,"tag":476,"props":5069,"children":5070},{},[5071],{"type":51,"value":5072},"\"Task reads env vars beyond LLM credentials — make sure they're set when running the experiment.\"",{"type":45,"tag":449,"props":5074,"children":5075},{},[5076,5101],{"type":45,"tag":476,"props":5077,"children":5078},{},[5079,5085,5086,5092,5093,5099],{"type":45,"tag":76,"props":5080,"children":5082},{"className":5081},[],[5083],{"type":51,"value":5084},"requests.",{"type":51,"value":431},{"type":45,"tag":76,"props":5087,"children":5089},{"className":5088},[],[5090],{"type":51,"value":5091},"httpx.",{"type":51,"value":431},{"type":45,"tag":76,"props":5094,"children":5096},{"className":5095},[],[5097],{"type":51,"value":5098},"aiohttp.",{"type":51,"value":5100}," calls to non-LLM-provider URLs",{"type":45,"tag":476,"props":5102,"children":5103},{},[5104],{"type":51,"value":5105},"\"Task makes external HTTP calls — running the experiment will hit those endpoints.\"",{"type":45,"tag":449,"props":5107,"children":5108},{},[5109,5150],{"type":45,"tag":476,"props":5110,"children":5111},{},[5112,5114,5120,5121,5127,5128,5134,5135,5141,5142,5148],{"type":51,"value":5113},"DB drivers (",{"type":45,"tag":76,"props":5115,"children":5117},{"className":5116},[],[5118],{"type":51,"value":5119},"psycopg2",{"type":51,"value":431},{"type":45,"tag":76,"props":5122,"children":5124},{"className":5123},[],[5125],{"type":51,"value":5126},"sqlalchemy",{"type":51,"value":431},{"type":45,"tag":76,"props":5129,"children":5131},{"className":5130},[],[5132],{"type":51,"value":5133},"pymongo",{"type":51,"value":431},{"type":45,"tag":76,"props":5136,"children":5138},{"className":5137},[],[5139],{"type":51,"value":5140},"redis",{"type":51,"value":431},{"type":45,"tag":76,"props":5143,"children":5145},{"className":5144},[],[5146],{"type":51,"value":5147},"boto3",{"type":51,"value":5149}," ≠ bedrock)",{"type":45,"tag":476,"props":5151,"children":5152},{},[5153],{"type":51,"value":5154},"\"Task hits a database — point at a non-prod instance before running.\"",{"type":45,"tag":449,"props":5156,"children":5157},{},[5158,5177],{"type":45,"tag":476,"props":5159,"children":5160},{},[5161,5163,5169,5170,5176],{"type":51,"value":5162},"File I\u002FO writes (",{"type":45,"tag":76,"props":5164,"children":5166},{"className":5165},[],[5167],{"type":51,"value":5168},"open(..., 'w')",{"type":51,"value":431},{"type":45,"tag":76,"props":5171,"children":5173},{"className":5172},[],[5174],{"type":51,"value":5175},"Path.write_*",{"type":51,"value":2227},{"type":45,"tag":476,"props":5178,"children":5179},{},[5180],{"type":51,"value":5181},"\"Task writes to disk — make sure the path is safe in your experiment env.\"",{"type":45,"tag":70,"props":5183,"children":5184},{},[5185,5190,5192,5197,5199,5204,5206,5212],{"type":45,"tag":58,"props":5186,"children":5187},{},[5188],{"type":51,"value":5189},"Fallback to placeholder",{"type":51,"value":5191},": if ",{"type":45,"tag":76,"props":5193,"children":5195},{"className":5194},[],[5196],{"type":51,"value":891},{"type":51,"value":5198}," is set OR the introspection picked nothing, emit the original placeholder block (generic OpenAI call with ",{"type":45,"tag":76,"props":5200,"children":5202},{"className":5201},[],[5203],{"type":51,"value":907},{"type":51,"value":5205},"). This is the only path where ",{"type":45,"tag":76,"props":5207,"children":5209},{"className":5208},[],[5210],{"type":51,"value":5211},"TODO(user)",{"type":51,"value":5213}," is allowed in the task section.",{"type":45,"tag":54,"props":5215,"children":5216},{},[5217,5219,5224,5226,5231,5233,5238],{"type":51,"value":5218},"2.6. ",{"type":45,"tag":58,"props":5220,"children":5221},{},[5222],{"type":51,"value":5223},"Determine required credentials and emit the env-setup section.",{"type":51,"value":5225}," The generated experiment must work without the user pre-exporting anything in their shell, ",{"type":45,"tag":2177,"props":5227,"children":5228},{},[5229],{"type":51,"value":5230},"as long as",{"type":51,"value":5232}," a discoverable ",{"type":45,"tag":76,"props":5234,"children":5236},{"className":5235},[],[5237],{"type":51,"value":990},{"type":51,"value":5239}," lives in a standard location. Don't push setup work onto the user that the file can do itself.",{"type":45,"tag":54,"props":5241,"children":5242},{},[5243,5248,5250,5256,5258,5264,5266,5272,5273,5279,5281,5287],{"type":45,"tag":58,"props":5244,"children":5245},{},[5246],{"type":51,"value":5247},"Required Datadog keys",{"type":51,"value":5249}," (always): ",{"type":45,"tag":76,"props":5251,"children":5253},{"className":5252},[],[5254],{"type":51,"value":5255},"DD_API_KEY",{"type":51,"value":5257}," AND (",{"type":45,"tag":76,"props":5259,"children":5261},{"className":5260},[],[5262],{"type":51,"value":5263},"DD_APPLICATION_KEY",{"type":51,"value":5265}," OR ",{"type":45,"tag":76,"props":5267,"children":5269},{"className":5268},[],[5270],{"type":51,"value":5271},"DD_APP_KEY",{"type":51,"value":4998},{"type":45,"tag":76,"props":5274,"children":5276},{"className":5275},[],[5277],{"type":51,"value":5278},"DD_SITE",{"type":51,"value":5280}," is optional (defaults to ",{"type":45,"tag":76,"props":5282,"children":5284},{"className":5283},[],[5285],{"type":51,"value":5286},"datadoghq.com",{"type":51,"value":1019},{"type":45,"tag":54,"props":5289,"children":5290},{},[5291,5296,5298,5303],{"type":45,"tag":58,"props":5292,"children":5293},{},[5294],{"type":51,"value":5295},"Required provider keys",{"type":51,"value":5297}," — depend on what step 2.5 picked. Inspect the imported call site to identify the provider, then ",{"type":45,"tag":58,"props":5299,"children":5300},{},[5301],{"type":51,"value":5302},"read the matching reference file",{"type":51,"value":5304}," for the exact assert lines, adapter notes, and gotchas. Only load the one that applies; do not load all of them.",{"type":45,"tag":441,"props":5306,"children":5307},{},[5308,5323],{"type":45,"tag":445,"props":5309,"children":5310},{},[5311],{"type":45,"tag":449,"props":5312,"children":5313},{},[5314,5319],{"type":45,"tag":453,"props":5315,"children":5316},{},[5317],{"type":51,"value":5318},"Detected SDK in task",{"type":45,"tag":453,"props":5320,"children":5321},{},[5322],{"type":51,"value":2390},{"type":45,"tag":469,"props":5324,"children":5325},{},[5326,5357,5389,5420,5449,5483,5499,5524,5548],{"type":45,"tag":449,"props":5327,"children":5328},{},[5329,5348],{"type":45,"tag":476,"props":5330,"children":5331},{},[5332,5334,5340,5341,5346],{"type":51,"value":5333},"OpenAI (",{"type":45,"tag":76,"props":5335,"children":5337},{"className":5336},[],[5338],{"type":51,"value":5339},"openai.*",{"type":51,"value":431},{"type":45,"tag":76,"props":5342,"children":5344},{"className":5343},[],[5345],{"type":51,"value":3720},{"type":51,"value":5347},") — also the placeholder fallback",{"type":45,"tag":476,"props":5349,"children":5350},{},[5351],{"type":45,"tag":76,"props":5352,"children":5354},{"className":5353},[],[5355],{"type":51,"value":5356},"references\u002Fproviders\u002Fopenai.md",{"type":45,"tag":449,"props":5358,"children":5359},{},[5360,5379],{"type":45,"tag":476,"props":5361,"children":5362},{},[5363,5365,5371,5372,5378],{"type":51,"value":5364},"Azure OpenAI (",{"type":45,"tag":76,"props":5366,"children":5368},{"className":5367},[],[5369],{"type":51,"value":5370},"AzureOpenAI(...)",{"type":51,"value":431},{"type":45,"tag":76,"props":5373,"children":5375},{"className":5374},[],[5376],{"type":51,"value":5377},"azure_endpoint=",{"type":51,"value":2227},{"type":45,"tag":476,"props":5380,"children":5381},{},[5382,5387],{"type":45,"tag":76,"props":5383,"children":5385},{"className":5384},[],[5386],{"type":51,"value":5356},{"type":51,"value":5388}," (Azure section)",{"type":45,"tag":449,"props":5390,"children":5391},{},[5392,5411],{"type":45,"tag":476,"props":5393,"children":5394},{},[5395,5397,5403,5404,5410],{"type":51,"value":5396},"Anthropic (",{"type":45,"tag":76,"props":5398,"children":5400},{"className":5399},[],[5401],{"type":51,"value":5402},"anthropic.*",{"type":51,"value":431},{"type":45,"tag":76,"props":5405,"children":5407},{"className":5406},[],[5408],{"type":51,"value":5409},"Anthropic().messages.create",{"type":51,"value":2227},{"type":45,"tag":476,"props":5412,"children":5413},{},[5414],{"type":45,"tag":76,"props":5415,"children":5417},{"className":5416},[],[5418],{"type":51,"value":5419},"references\u002Fproviders\u002Fanthropic.md",{"type":45,"tag":449,"props":5421,"children":5422},{},[5423,5440],{"type":45,"tag":476,"props":5424,"children":5425},{},[5426,5428,5433,5434,5439],{"type":51,"value":5427},"LiteLLM (",{"type":45,"tag":76,"props":5429,"children":5431},{"className":5430},[],[5432],{"type":51,"value":3792},{"type":51,"value":431},{"type":45,"tag":76,"props":5435,"children":5437},{"className":5436},[],[5438],{"type":51,"value":3799},{"type":51,"value":2227},{"type":45,"tag":476,"props":5441,"children":5442},{},[5443],{"type":45,"tag":76,"props":5444,"children":5446},{"className":5445},[],[5447],{"type":51,"value":5448},"references\u002Fproviders\u002Flitellm.md",{"type":45,"tag":449,"props":5450,"children":5451},{},[5452,5472],{"type":45,"tag":476,"props":5453,"children":5454},{},[5455,5457,5463,5464,5470],{"type":51,"value":5456},"LangChain (",{"type":45,"tag":76,"props":5458,"children":5460},{"className":5459},[],[5461],{"type":51,"value":5462},"ChatOpenAI",{"type":51,"value":934},{"type":45,"tag":76,"props":5465,"children":5467},{"className":5466},[],[5468],{"type":51,"value":5469},"ChatAnthropic",{"type":51,"value":5471}," \u002F etc.)",{"type":45,"tag":476,"props":5473,"children":5474},{},[5475,5481],{"type":45,"tag":76,"props":5476,"children":5478},{"className":5477},[],[5479],{"type":51,"value":5480},"references\u002Fproviders\u002Flangchain.md",{"type":51,"value":5482}," (walks one level deeper to the underlying provider)",{"type":45,"tag":449,"props":5484,"children":5485},{},[5486,5490],{"type":45,"tag":476,"props":5487,"children":5488},{},[5489],{"type":51,"value":3873},{"type":45,"tag":476,"props":5491,"children":5492},{},[5493],{"type":45,"tag":76,"props":5494,"children":5496},{"className":5495},[],[5497],{"type":51,"value":5498},"references\u002Fproviders\u002Fllamaindex.md",{"type":45,"tag":449,"props":5500,"children":5501},{},[5502,5515],{"type":45,"tag":476,"props":5503,"children":5504},{},[5505,5507,5513],{"type":51,"value":5506},"Google Gemini (",{"type":45,"tag":76,"props":5508,"children":5510},{"className":5509},[],[5511],{"type":51,"value":5512},"google.generativeai",{"type":51,"value":5514},") \u002F Vertex AI",{"type":45,"tag":476,"props":5516,"children":5517},{},[5518],{"type":45,"tag":76,"props":5519,"children":5521},{"className":5520},[],[5522],{"type":51,"value":5523},"references\u002Fproviders\u002Fgemini.md",{"type":45,"tag":449,"props":5525,"children":5526},{},[5527,5539],{"type":45,"tag":476,"props":5528,"children":5529},{},[5530,5532,5538],{"type":51,"value":5531},"AWS Bedrock (",{"type":45,"tag":76,"props":5533,"children":5535},{"className":5534},[],[5536],{"type":51,"value":5537},"boto3.client(\"bedrock-runtime\")",{"type":51,"value":2227},{"type":45,"tag":476,"props":5540,"children":5541},{},[5542],{"type":45,"tag":76,"props":5543,"children":5545},{"className":5544},[],[5546],{"type":51,"value":5547},"references\u002Fproviders\u002Fbedrock.md",{"type":45,"tag":449,"props":5549,"children":5550},{},[5551,5556],{"type":45,"tag":476,"props":5552,"children":5553},{},[5554],{"type":51,"value":5555},"Custom \u002F not-recognized SDK",{"type":45,"tag":476,"props":5557,"children":5558},{},[5559,5561,5567],{"type":51,"value":5560},"No reference file — emit a ",{"type":45,"tag":76,"props":5562,"children":5564},{"className":5563},[],[5565],{"type":51,"value":5566},"# TODO(user): set the API key(s) your task_fn needs in your .env or shell.",{"type":51,"value":5568}," comment instead of an assert. Do NOT fabricate provider keys.",{"type":45,"tag":54,"props":5570,"children":5571},{},[5572,5585,5587,5593,5595,5599],{"type":45,"tag":58,"props":5573,"children":5574},{},[5575,5577,5583],{"type":51,"value":5576},"Emit section 1 by reading the shipped template at ",{"type":45,"tag":76,"props":5578,"children":5580},{"className":5579},[],[5581],{"type":51,"value":5582},"\u003Cthis-skill-dir>\u002Fscripts\u002Fenv_setup_template.py",{"type":51,"value":5584}," and substituting two placeholders.",{"type":51,"value":5586}," The template ships alongside this SKILL.md (",{"type":45,"tag":76,"props":5588,"children":5590},{"className":5589},[],[5591],{"type":51,"value":5592},"cp -r",{"type":51,"value":5594}," install handles it) and is the canonical source for the loader + assert shape — do ",{"type":45,"tag":58,"props":5596,"children":5597},{},[5598],{"type":51,"value":2490},{"type":51,"value":5600}," re-derive it inline. Read the file, perform the substitutions below, and emit the result verbatim as section 1 of the generated experiment file.",{"type":45,"tag":441,"props":5602,"children":5603},{},[5604,5625],{"type":45,"tag":445,"props":5605,"children":5606},{},[5607],{"type":45,"tag":449,"props":5608,"children":5609},{},[5610,5615,5620],{"type":45,"tag":453,"props":5611,"children":5612},{},[5613],{"type":51,"value":5614},"Placeholder",{"type":45,"tag":453,"props":5616,"children":5617},{},[5618],{"type":51,"value":5619},"Replacement",{"type":45,"tag":453,"props":5621,"children":5622},{},[5623],{"type":51,"value":5624},"Example",{"type":45,"tag":469,"props":5626,"children":5627},{},[5628,5668],{"type":45,"tag":449,"props":5629,"children":5630},{},[5631,5640,5652],{"type":45,"tag":476,"props":5632,"children":5633},{},[5634],{"type":45,"tag":76,"props":5635,"children":5637},{"className":5636},[],[5638],{"type":51,"value":5639},"{{ENV_FILE_OVERRIDE_LIST}}",{"type":45,"tag":476,"props":5641,"children":5642},{},[5643,5645,5650],{"type":51,"value":5644},"A Python list literal of absolute paths from ",{"type":45,"tag":76,"props":5646,"children":5648},{"className":5647},[],[5649],{"type":51,"value":979},{"type":51,"value":5651}," (repeatable; empty list if not passed)",{"type":45,"tag":476,"props":5653,"children":5654},{},[5655,5661,5662],{"type":45,"tag":76,"props":5656,"children":5658},{"className":5657},[],[5659],{"type":51,"value":5660},"[]",{"type":51,"value":835},{"type":45,"tag":76,"props":5663,"children":5665},{"className":5664},[],[5666],{"type":51,"value":5667},"[\"\u002FUsers\u002Fme\u002F.config\u002Fdd\u002Fstaging.env\"]",{"type":45,"tag":449,"props":5669,"children":5670},{},[5671,5680,5693],{"type":45,"tag":476,"props":5672,"children":5673},{},[5674],{"type":45,"tag":76,"props":5675,"children":5677},{"className":5676},[],[5678],{"type":51,"value":5679},"{{PROVIDER_ASSERTS}}",{"type":45,"tag":476,"props":5681,"children":5682},{},[5683,5685,5691],{"type":51,"value":5684},"Zero or more ",{"type":45,"tag":76,"props":5686,"children":5688},{"className":5687},[],[5689],{"type":51,"value":5690},"assert os.getenv(...)",{"type":51,"value":5692}," lines derived from what step 2.5 wired — per the SDK-to-key table above. One line per required provider key. Empty string if the task uses LiteLLM (which auto-routes) or a custom\u002Funrecognized SDK.",{"type":45,"tag":476,"props":5694,"children":5695},{},[5696],{"type":45,"tag":76,"props":5697,"children":5699},{"className":5698},[],[5700],{"type":51,"value":5701},"assert os.getenv(\"ANTHROPIC_API_KEY\"), \"ANTHROPIC_API_KEY is required for the wired task_fn.\"",{"type":45,"tag":54,"props":5703,"children":5704},{},[5705,5707,5712,5714,5720],{"type":51,"value":5706},"The exact assert line(s) to substitute into ",{"type":45,"tag":76,"props":5708,"children":5710},{"className":5709},[],[5711],{"type":51,"value":5679},{"type":51,"value":5713}," live in each provider's reference file under the ",{"type":45,"tag":76,"props":5715,"children":5717},{"className":5716},[],[5718],{"type":51,"value":5719},"## {{PROVIDER_ASSERTS}} substitution",{"type":51,"value":5721}," heading. Read only the file that matches the detected SDK from the table above.",{"type":45,"tag":54,"props":5723,"children":5724},{},[5725,5727,5733,5735,5741],{"type":51,"value":5726},"The template's discovery walk (env file → script dir → cwd → parent dirs → ",{"type":45,"tag":76,"props":5728,"children":5730},{"className":5729},[],[5731],{"type":51,"value":5732},"~\u002F.datadog\u002Fcredentials",{"type":51,"value":5734},") and shell-overrides-file contract are stable — they live in the template, not in this skill body. If you need to change discovery semantics, edit ",{"type":45,"tag":76,"props":5736,"children":5738},{"className":5737},[],[5739],{"type":51,"value":5740},"scripts\u002Fenv_setup_template.py",{"type":51,"value":5742}," directly; do not re-derive them in-prose here.",{"type":45,"tag":66,"props":5744,"children":5745},{"start":1335},[5746,6136,6857,6900],{"type":45,"tag":70,"props":5747,"children":5748},{},[5749,5754,5756,5761,5763,5766,5771,5773,5778,5780,5785,5787,5792,5794,5797,5799,5916,5919,5924,5926,6067,6070,6072,6077,6079,6082,6087,6089,6094,6096,6101,6102,6107,6108,6113,6115,6120,6122,6127,6129,6134],{"type":45,"tag":58,"props":5750,"children":5751},{},[5752],{"type":51,"value":5753},"Pick evaluator template",{"type":51,"value":5755}," based on ",{"type":45,"tag":76,"props":5757,"children":5759},{"className":5758},[],[5760],{"type":51,"value":713},{"type":51,"value":5762}," AND the purpose from step 2.4.",{"type":45,"tag":2574,"props":5764,"children":5765},{},[],{"type":45,"tag":76,"props":5767,"children":5769},{"className":5768},[],[5770],{"type":51,"value":713},{"type":51,"value":5772}," decides the ",{"type":45,"tag":58,"props":5774,"children":5775},{},[5776],{"type":51,"value":5777},"surface",{"type":51,"value":5779}," (function \u002F class \u002F remote — see ",{"type":45,"tag":76,"props":5781,"children":5783},{"className":5782},[],[5784],{"type":51,"value":2000},{"type":51,"value":5786}," for the syntax shape). The purpose decides the ",{"type":45,"tag":58,"props":5788,"children":5789},{},[5790],{"type":51,"value":5791},"semantics",{"type":51,"value":5793}," — what each evaluator actually measures. Both inputs apply together.",{"type":45,"tag":2574,"props":5795,"children":5796},{},[],{"type":51,"value":5798},"Default shape (purpose: accuracy \u002F unspecified):",{"type":45,"tag":235,"props":5800,"children":5801},{},[5802,5866,5898],{"type":45,"tag":70,"props":5803,"children":5804},{},[5805,5810,5812,5817,5819,5824,5826,5831,5832,5837,5839,5844,5846,5851,5853,5858,5860,5865],{"type":45,"tag":76,"props":5806,"children":5808},{"className":5807},[],[5809],{"type":51,"value":722},{"type":51,"value":5811},": 3 plain functions — one trivial boolean (",{"type":45,"tag":76,"props":5813,"children":5815},{"className":5814},[],[5816],{"type":51,"value":2316},{"type":51,"value":5818},"-style, bare ",{"type":45,"tag":76,"props":5820,"children":5822},{"className":5821},[],[5823],{"type":51,"value":2033},{"type":51,"value":5825}," OK), one richer rule-based check returning ",{"type":45,"tag":76,"props":5827,"children":5829},{"className":5828},[],[5830],{"type":51,"value":1254},{"type":51,"value":184},{"type":45,"tag":76,"props":5833,"children":5835},{"className":5834},[],[5836],{"type":51,"value":2162},{"type":51,"value":5838}," + ",{"type":45,"tag":76,"props":5840,"children":5842},{"className":5841},[],[5843],{"type":51,"value":2195},{"type":51,"value":5845},", and one LLM-as-Judge surrogate. If ",{"type":45,"tag":76,"props":5847,"children":5849},{"className":5848},[],[5850],{"type":51,"value":529},{"type":51,"value":5852}," had structured ",{"type":45,"tag":76,"props":5854,"children":5856},{"className":5855},[],[5857],{"type":51,"value":2901},{"type":51,"value":5859},", add a JSON-shape check (also returning ",{"type":45,"tag":76,"props":5861,"children":5863},{"className":5862},[],[5864],{"type":51,"value":1254},{"type":51,"value":1019},{"type":45,"tag":70,"props":5867,"children":5868},{},[5869,5874,5876,5881,5883,5889,5891,5896],{"type":45,"tag":76,"props":5870,"children":5872},{"className":5871},[],[5873],{"type":51,"value":738},{"type":51,"value":5875},": 2 ",{"type":45,"tag":76,"props":5877,"children":5879},{"className":5878},[],[5880],{"type":51,"value":746},{"type":51,"value":5882}," subclasses with ",{"type":45,"tag":76,"props":5884,"children":5886},{"className":5885},[],[5887],{"type":51,"value":5888},"evaluate(self, context: EvaluatorContext) -> EvaluatorResult",{"type":51,"value":5890},". Always return ",{"type":45,"tag":76,"props":5892,"children":5894},{"className":5893},[],[5895],{"type":51,"value":1254},{"type":51,"value":5897}," (never a bare value) — state-bearing evaluators have richer signal to surface.",{"type":45,"tag":70,"props":5899,"children":5900},{},[5901,5906,5908,5914],{"type":45,"tag":76,"props":5902,"children":5904},{"className":5903},[],[5905],{"type":51,"value":754},{"type":51,"value":5907},": 1-2 ",{"type":45,"tag":76,"props":5909,"children":5911},{"className":5910},[],[5912],{"type":51,"value":5913},"RemoteEvaluator(eval_name=...)",{"type":51,"value":5915}," instances with a comment instructing the user to create the judge in the Datadog UI first.",{"type":45,"tag":2574,"props":5917,"children":5918},{},[],{"type":45,"tag":58,"props":5920,"children":5921},{},[5922],{"type":51,"value":5923},"Purpose-driven amendments",{"type":51,"value":5925}," (read the purpose string from step 2.4 and adjust the default seed accordingly — use natural-language judgment, not a hardcoded lookup):",{"type":45,"tag":235,"props":5927,"children":5928},{},[5929,5957,5990,6022,6055],{"type":45,"tag":70,"props":5930,"children":5931},{},[5932,5934,5939,5941,5947,5949,5955],{"type":51,"value":5933},"If the purpose mentions ",{"type":45,"tag":58,"props":5935,"children":5936},{},[5937],{"type":51,"value":5938},"tool calls \u002F agent routing \u002F tool selection",{"type":51,"value":5940},": replace the LLM-as-Judge surrogate with a ",{"type":45,"tag":76,"props":5942,"children":5944},{"className":5943},[],[5945],{"type":51,"value":5946},"tool_calls_correct",{"type":51,"value":5948}," LLMJudge whose rubric checks the right-tool-with-right-args question. The rubric should reference ",{"type":45,"tag":76,"props":5950,"children":5952},{"className":5951},[],[5953],{"type":51,"value":5954},"output_data[\"tool_calls\"]",{"type":51,"value":5956}," if the wrapper from 2.5d captured them; otherwise check the textual output for the tool name.",{"type":45,"tag":70,"props":5958,"children":5959},{},[5960,5961,5966,5968,5974,5976,5981,5983,5988],{"type":51,"value":5933},{"type":45,"tag":58,"props":5962,"children":5963},{},[5964],{"type":51,"value":5965},"groundedness \u002F faithfulness \u002F RAG",{"type":51,"value":5967},": add a ",{"type":45,"tag":76,"props":5969,"children":5971},{"className":5970},[],[5972],{"type":51,"value":5973},"groundedness",{"type":51,"value":5975}," LLMJudge that takes ",{"type":45,"tag":76,"props":5977,"children":5979},{"className":5978},[],[5980],{"type":51,"value":2893},{"type":51,"value":5982}," (the question + retrieved docs) and ",{"type":45,"tag":76,"props":5984,"children":5986},{"className":5985},[],[5987],{"type":51,"value":4951},{"type":51,"value":5989}," (the answer) and checks that every claim in the answer traces to the docs. Make the rubric explicit about hallucinations being a fail.",{"type":45,"tag":70,"props":5991,"children":5992},{},[5993,5994,5999,6001,6006,6007,6013,6015,6020],{"type":51,"value":5933},{"type":45,"tag":58,"props":5995,"children":5996},{},[5997],{"type":51,"value":5998},"structured output \u002F JSON \u002F SQL \u002F schema",{"type":51,"value":6000},": lean into ",{"type":45,"tag":76,"props":6002,"children":6004},{"className":6003},[],[6005],{"type":51,"value":3399},{"type":51,"value":835},{"type":45,"tag":76,"props":6008,"children":6010},{"className":6009},[],[6011],{"type":51,"value":6012},"RegexMatchEvaluator(pattern=r\"...\")",{"type":51,"value":6014}," from the built-in set. Add a richer ",{"type":45,"tag":76,"props":6016,"children":6018},{"className":6017},[],[6019],{"type":51,"value":1254},{"type":51,"value":6021}," check for semantic schema-fit if the user's structured output has constraints beyond shape (e.g. enum values, numeric ranges).",{"type":45,"tag":70,"props":6023,"children":6024},{},[6025,6026,6031,6033,6038,6040,6046,6048,6053],{"type":51,"value":5933},{"type":45,"tag":58,"props":6027,"children":6028},{},[6029],{"type":51,"value":6030},"regression \u002F refactor \u002F preserve behavior",{"type":51,"value":6032},": prefer ",{"type":45,"tag":76,"props":6034,"children":6036},{"className":6035},[],[6037],{"type":51,"value":2316},{"type":51,"value":6039}," as the primary signal, plus a near-match check (",{"type":45,"tag":76,"props":6041,"children":6043},{"className":6042},[],[6044],{"type":51,"value":6045},"difflib.SequenceMatcher.ratio()",{"type":51,"value":6047}," ≥ 0.95 returning ",{"type":45,"tag":76,"props":6049,"children":6051},{"className":6050},[],[6052],{"type":51,"value":1254},{"type":51,"value":6054}," with assessment=\"partial\" between 0.7 and 0.95). Drop the LLMJudge surrogate — regression tests want determinism, not model judgment.",{"type":45,"tag":70,"props":6056,"children":6057},{},[6058,6060,6065],{"type":51,"value":6059},"For purposes the SKILL.md doesn't recognize verbatim: read the user's purpose string, reason about what evaluator shape would best measure it, and write the evaluator from scratch. Cite the purpose in a comment above the evaluator so the user can see the link. Never invent a ",{"type":45,"tag":76,"props":6061,"children":6063},{"className":6062},[],[6064],{"type":51,"value":5913},{"type":51,"value":6066}," for a name that may not exist in their Datadog org — when in doubt emit an LLMJudge with an inline rubric instead.",{"type":45,"tag":2574,"props":6068,"children":6069},{},[],{"type":51,"value":6071},"Whatever you emit, the ",{"type":45,"tag":58,"props":6073,"children":6074},{},[6075],{"type":51,"value":6076},"count of evaluators stays 2–3",{"type":51,"value":6078}," to keep the generated file lean. The user can add more after the first run.",{"type":45,"tag":2574,"props":6080,"children":6081},{},[],{"type":45,"tag":58,"props":6083,"children":6084},{},[6085],{"type":51,"value":6086},"In all styles",{"type":51,"value":6088},": any evaluator with non-trivial logic must return ",{"type":45,"tag":76,"props":6090,"children":6092},{"className":6091},[],[6093],{"type":51,"value":1254},{"type":51,"value":6095}," populating at minimum ",{"type":45,"tag":76,"props":6097,"children":6099},{"className":6098},[],[6100],{"type":51,"value":2111},{"type":51,"value":5838},{"type":45,"tag":76,"props":6103,"children":6105},{"className":6104},[],[6106],{"type":51,"value":2162},{"type":51,"value":5838},{"type":45,"tag":76,"props":6109,"children":6111},{"className":6110},[],[6112],{"type":51,"value":2195},{"type":51,"value":6114}," (see the \"Return ",{"type":45,"tag":76,"props":6116,"children":6118},{"className":6117},[],[6119],{"type":51,"value":1254},{"type":51,"value":6121},", not bare values\" section). The compare UI uses ",{"type":45,"tag":76,"props":6123,"children":6125},{"className":6124},[],[6126],{"type":51,"value":2162},{"type":51,"value":6128}," for per-record drill-downs and ",{"type":45,"tag":76,"props":6130,"children":6132},{"className":6131},[],[6133],{"type":51,"value":2195},{"type":51,"value":6135}," to determine whether a metric trend is an improvement.",{"type":45,"tag":70,"props":6137,"children":6138},{},[6139,6144,6145,6148,6158,6160,6174,6177,6179,6185,6186,6192,6194,6197,6206,6208,6852,6855],{"type":45,"tag":58,"props":6140,"children":6141},{},[6142],{"type":51,"value":6143},"Emit the file",{"type":51,"value":110},{"type":45,"tag":2574,"props":6146,"children":6147},{},[],{"type":45,"tag":58,"props":6149,"children":6150},{},[6151,6153],{"type":51,"value":6152},"For ",{"type":45,"tag":76,"props":6154,"children":6156},{"className":6155},[],[6157],{"type":51,"value":349},{"type":51,"value":6159}," — single file, one blank line between sections, banner comments like:",{"type":45,"tag":396,"props":6161,"children":6163},{"className":1307,"code":6162,"language":24,"meta":404,"style":404},"# ─── 3. Dataset ───────────────────────────────────────────────────────────────\n",[6164],{"type":45,"tag":76,"props":6165,"children":6166},{"__ignoreMap":404},[6167],{"type":45,"tag":1314,"props":6168,"children":6169},{"class":1316,"line":1317},[6170],{"type":45,"tag":1314,"props":6171,"children":6172},{},[6173],{"type":51,"value":6162},{"type":45,"tag":2574,"props":6175,"children":6176},{},[],{"type":51,"value":6178},"Use ",{"type":45,"tag":76,"props":6180,"children":6182},{"className":6181},[],[6183],{"type":51,"value":6184},"from __future__ import annotations",{"type":51,"value":2318},{"type":45,"tag":76,"props":6187,"children":6189},{"className":6188},[],[6190],{"type":51,"value":6191},"from typing import Any, Dict",{"type":51,"value":6193}," at the top. Type-hint task and evaluator function signatures.",{"type":45,"tag":2574,"props":6195,"children":6196},{},[],{"type":45,"tag":58,"props":6198,"children":6199},{},[6200,6201],{"type":51,"value":6152},{"type":45,"tag":76,"props":6202,"children":6204},{"className":6203},[],[6205],{"type":51,"value":357},{"type":51,"value":6207}," — valid Jupyter notebook JSON. Schema:",{"type":45,"tag":396,"props":6209,"children":6213},{"className":6210,"code":6211,"language":6212,"meta":404,"style":404},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"cells\": [\n    {\"cell_type\": \"markdown\", \"metadata\": {}, \"source\": [\"## 1. Env setup\\n\", \"...\"]},\n    {\"cell_type\": \"code\", \"execution_count\": null, \"metadata\": {}, \"outputs\": [], \"source\": [\"...\"]},\n    ...\n  ],\n  \"metadata\": {\n    \"kernelspec\": {\"display_name\": \"Python 3\", \"language\": \"python\", \"name\": \"python3\"},\n    \"language_info\": {\"name\": \"python\", \"version\": \"3.10\"}\n  },\n  \"nbformat\": 4,\n  \"nbformat_minor\": 5\n}\n","json",[6214],{"type":45,"tag":76,"props":6215,"children":6216},{"__ignoreMap":404},[6217,6226,6254,6384,6523,6530,6538,6562,6691,6782,6790,6820,6845],{"type":45,"tag":1314,"props":6218,"children":6219},{"class":1316,"line":1317},[6220],{"type":45,"tag":1314,"props":6221,"children":6223},{"style":6222},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[6224],{"type":51,"value":6225},"{\n",{"type":45,"tag":1314,"props":6227,"children":6228},{"class":1316,"line":1326},[6229,6234,6240,6245,6249],{"type":45,"tag":1314,"props":6230,"children":6231},{"style":6222},[6232],{"type":51,"value":6233},"  \"",{"type":45,"tag":1314,"props":6235,"children":6237},{"style":6236},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[6238],{"type":51,"value":6239},"cells",{"type":45,"tag":1314,"props":6241,"children":6242},{"style":6222},[6243],{"type":51,"value":6244},"\"",{"type":45,"tag":1314,"props":6246,"children":6247},{"style":6222},[6248],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6250,"children":6251},{"style":6222},[6252],{"type":51,"value":6253}," [\n",{"type":45,"tag":1314,"props":6255,"children":6256},{"class":1316,"line":1335},[6257,6262,6266,6272,6276,6280,6285,6291,6295,6300,6304,6308,6312,6316,6321,6325,6330,6334,6338,6343,6347,6352,6358,6362,6366,6370,6375,6379],{"type":45,"tag":1314,"props":6258,"children":6259},{"style":6222},[6260],{"type":51,"value":6261},"    {",{"type":45,"tag":1314,"props":6263,"children":6264},{"style":6222},[6265],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6267,"children":6269},{"style":6268},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[6270],{"type":51,"value":6271},"cell_type",{"type":45,"tag":1314,"props":6273,"children":6274},{"style":6222},[6275],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6277,"children":6278},{"style":6222},[6279],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6281,"children":6282},{"style":6222},[6283],{"type":51,"value":6284}," \"",{"type":45,"tag":1314,"props":6286,"children":6288},{"style":6287},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[6289],{"type":51,"value":6290},"markdown",{"type":45,"tag":1314,"props":6292,"children":6293},{"style":6222},[6294],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6296,"children":6297},{"style":6222},[6298],{"type":51,"value":6299},",",{"type":45,"tag":1314,"props":6301,"children":6302},{"style":6222},[6303],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6305,"children":6306},{"style":6268},[6307],{"type":51,"value":2244},{"type":45,"tag":1314,"props":6309,"children":6310},{"style":6222},[6311],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6313,"children":6314},{"style":6222},[6315],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6317,"children":6318},{"style":6222},[6319],{"type":51,"value":6320}," {},",{"type":45,"tag":1314,"props":6322,"children":6323},{"style":6222},[6324],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6326,"children":6327},{"style":6268},[6328],{"type":51,"value":6329},"source",{"type":45,"tag":1314,"props":6331,"children":6332},{"style":6222},[6333],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6335,"children":6336},{"style":6222},[6337],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6339,"children":6340},{"style":6222},[6341],{"type":51,"value":6342}," [",{"type":45,"tag":1314,"props":6344,"children":6345},{"style":6222},[6346],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6348,"children":6349},{"style":6287},[6350],{"type":51,"value":6351},"## 1. Env setup",{"type":45,"tag":1314,"props":6353,"children":6355},{"style":6354},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[6356],{"type":51,"value":6357},"\\n",{"type":45,"tag":1314,"props":6359,"children":6360},{"style":6222},[6361],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6363,"children":6364},{"style":6222},[6365],{"type":51,"value":6299},{"type":45,"tag":1314,"props":6367,"children":6368},{"style":6222},[6369],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6371,"children":6372},{"style":6287},[6373],{"type":51,"value":6374},"...",{"type":45,"tag":1314,"props":6376,"children":6377},{"style":6222},[6378],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6380,"children":6381},{"style":6222},[6382],{"type":51,"value":6383},"]},\n",{"type":45,"tag":1314,"props":6385,"children":6386},{"class":1316,"line":1344},[6387,6391,6395,6399,6403,6407,6411,6415,6419,6423,6427,6432,6436,6440,6445,6449,6453,6457,6461,6465,6469,6474,6478,6482,6487,6491,6495,6499,6503,6507,6511,6515,6519],{"type":45,"tag":1314,"props":6388,"children":6389},{"style":6222},[6390],{"type":51,"value":6261},{"type":45,"tag":1314,"props":6392,"children":6393},{"style":6222},[6394],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6396,"children":6397},{"style":6268},[6398],{"type":51,"value":6271},{"type":45,"tag":1314,"props":6400,"children":6401},{"style":6222},[6402],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6404,"children":6405},{"style":6222},[6406],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6408,"children":6409},{"style":6222},[6410],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6412,"children":6413},{"style":6287},[6414],{"type":51,"value":76},{"type":45,"tag":1314,"props":6416,"children":6417},{"style":6222},[6418],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6420,"children":6421},{"style":6222},[6422],{"type":51,"value":6299},{"type":45,"tag":1314,"props":6424,"children":6425},{"style":6222},[6426],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6428,"children":6429},{"style":6268},[6430],{"type":51,"value":6431},"execution_count",{"type":45,"tag":1314,"props":6433,"children":6434},{"style":6222},[6435],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6437,"children":6438},{"style":6222},[6439],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6441,"children":6442},{"style":6222},[6443],{"type":51,"value":6444}," null,",{"type":45,"tag":1314,"props":6446,"children":6447},{"style":6222},[6448],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6450,"children":6451},{"style":6268},[6452],{"type":51,"value":2244},{"type":45,"tag":1314,"props":6454,"children":6455},{"style":6222},[6456],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6458,"children":6459},{"style":6222},[6460],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6462,"children":6463},{"style":6222},[6464],{"type":51,"value":6320},{"type":45,"tag":1314,"props":6466,"children":6467},{"style":6222},[6468],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6470,"children":6471},{"style":6268},[6472],{"type":51,"value":6473},"outputs",{"type":45,"tag":1314,"props":6475,"children":6476},{"style":6222},[6477],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6479,"children":6480},{"style":6222},[6481],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6483,"children":6484},{"style":6222},[6485],{"type":51,"value":6486}," [],",{"type":45,"tag":1314,"props":6488,"children":6489},{"style":6222},[6490],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6492,"children":6493},{"style":6268},[6494],{"type":51,"value":6329},{"type":45,"tag":1314,"props":6496,"children":6497},{"style":6222},[6498],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6500,"children":6501},{"style":6222},[6502],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6504,"children":6505},{"style":6222},[6506],{"type":51,"value":6342},{"type":45,"tag":1314,"props":6508,"children":6509},{"style":6222},[6510],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6512,"children":6513},{"style":6287},[6514],{"type":51,"value":6374},{"type":45,"tag":1314,"props":6516,"children":6517},{"style":6222},[6518],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6520,"children":6521},{"style":6222},[6522],{"type":51,"value":6383},{"type":45,"tag":1314,"props":6524,"children":6525},{"class":1316,"line":1353},[6526],{"type":45,"tag":1314,"props":6527,"children":6528},{"style":6354},[6529],{"type":51,"value":1668},{"type":45,"tag":1314,"props":6531,"children":6532},{"class":1316,"line":1362},[6533],{"type":45,"tag":1314,"props":6534,"children":6535},{"style":6222},[6536],{"type":51,"value":6537},"  ],\n",{"type":45,"tag":1314,"props":6539,"children":6540},{"class":1316,"line":1371},[6541,6545,6549,6553,6557],{"type":45,"tag":1314,"props":6542,"children":6543},{"style":6222},[6544],{"type":51,"value":6233},{"type":45,"tag":1314,"props":6546,"children":6547},{"style":6236},[6548],{"type":51,"value":2244},{"type":45,"tag":1314,"props":6550,"children":6551},{"style":6222},[6552],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6554,"children":6555},{"style":6222},[6556],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6558,"children":6559},{"style":6222},[6560],{"type":51,"value":6561}," {\n",{"type":45,"tag":1314,"props":6563,"children":6564},{"class":1316,"line":1380},[6565,6570,6575,6579,6583,6588,6592,6598,6602,6606,6610,6615,6619,6623,6627,6632,6636,6640,6644,6648,6652,6656,6660,6665,6669,6673,6677,6682,6686],{"type":45,"tag":1314,"props":6566,"children":6567},{"style":6222},[6568],{"type":51,"value":6569},"    \"",{"type":45,"tag":1314,"props":6571,"children":6572},{"style":6268},[6573],{"type":51,"value":6574},"kernelspec",{"type":45,"tag":1314,"props":6576,"children":6577},{"style":6222},[6578],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6580,"children":6581},{"style":6222},[6582],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6584,"children":6585},{"style":6222},[6586],{"type":51,"value":6587}," {",{"type":45,"tag":1314,"props":6589,"children":6590},{"style":6222},[6591],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6593,"children":6595},{"style":6594},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[6596],{"type":51,"value":6597},"display_name",{"type":45,"tag":1314,"props":6599,"children":6600},{"style":6222},[6601],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6603,"children":6604},{"style":6222},[6605],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6607,"children":6608},{"style":6222},[6609],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6611,"children":6612},{"style":6287},[6613],{"type":51,"value":6614},"Python 3",{"type":45,"tag":1314,"props":6616,"children":6617},{"style":6222},[6618],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6620,"children":6621},{"style":6222},[6622],{"type":51,"value":6299},{"type":45,"tag":1314,"props":6624,"children":6625},{"style":6222},[6626],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6628,"children":6629},{"style":6594},[6630],{"type":51,"value":6631},"language",{"type":45,"tag":1314,"props":6633,"children":6634},{"style":6222},[6635],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6637,"children":6638},{"style":6222},[6639],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6641,"children":6642},{"style":6222},[6643],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6645,"children":6646},{"style":6287},[6647],{"type":51,"value":24},{"type":45,"tag":1314,"props":6649,"children":6650},{"style":6222},[6651],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6653,"children":6654},{"style":6222},[6655],{"type":51,"value":6299},{"type":45,"tag":1314,"props":6657,"children":6658},{"style":6222},[6659],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6661,"children":6662},{"style":6594},[6663],{"type":51,"value":6664},"name",{"type":45,"tag":1314,"props":6666,"children":6667},{"style":6222},[6668],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6670,"children":6671},{"style":6222},[6672],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6674,"children":6675},{"style":6222},[6676],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6678,"children":6679},{"style":6287},[6680],{"type":51,"value":6681},"python3",{"type":45,"tag":1314,"props":6683,"children":6684},{"style":6222},[6685],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6687,"children":6688},{"style":6222},[6689],{"type":51,"value":6690},"},\n",{"type":45,"tag":1314,"props":6692,"children":6693},{"class":1316,"line":1389},[6694,6698,6703,6707,6711,6715,6719,6723,6727,6731,6735,6739,6743,6747,6751,6756,6760,6764,6768,6773,6777],{"type":45,"tag":1314,"props":6695,"children":6696},{"style":6222},[6697],{"type":51,"value":6569},{"type":45,"tag":1314,"props":6699,"children":6700},{"style":6268},[6701],{"type":51,"value":6702},"language_info",{"type":45,"tag":1314,"props":6704,"children":6705},{"style":6222},[6706],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6708,"children":6709},{"style":6222},[6710],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6712,"children":6713},{"style":6222},[6714],{"type":51,"value":6587},{"type":45,"tag":1314,"props":6716,"children":6717},{"style":6222},[6718],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6720,"children":6721},{"style":6594},[6722],{"type":51,"value":6664},{"type":45,"tag":1314,"props":6724,"children":6725},{"style":6222},[6726],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6728,"children":6729},{"style":6222},[6730],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6732,"children":6733},{"style":6222},[6734],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6736,"children":6737},{"style":6287},[6738],{"type":51,"value":24},{"type":45,"tag":1314,"props":6740,"children":6741},{"style":6222},[6742],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6744,"children":6745},{"style":6222},[6746],{"type":51,"value":6299},{"type":45,"tag":1314,"props":6748,"children":6749},{"style":6222},[6750],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6752,"children":6753},{"style":6594},[6754],{"type":51,"value":6755},"version",{"type":45,"tag":1314,"props":6757,"children":6758},{"style":6222},[6759],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6761,"children":6762},{"style":6222},[6763],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6765,"children":6766},{"style":6222},[6767],{"type":51,"value":6284},{"type":45,"tag":1314,"props":6769,"children":6770},{"style":6287},[6771],{"type":51,"value":6772},"3.10",{"type":45,"tag":1314,"props":6774,"children":6775},{"style":6222},[6776],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6778,"children":6779},{"style":6222},[6780],{"type":51,"value":6781},"}\n",{"type":45,"tag":1314,"props":6783,"children":6784},{"class":1316,"line":1398},[6785],{"type":45,"tag":1314,"props":6786,"children":6787},{"style":6222},[6788],{"type":51,"value":6789},"  },\n",{"type":45,"tag":1314,"props":6791,"children":6792},{"class":1316,"line":1407},[6793,6797,6802,6806,6810,6815],{"type":45,"tag":1314,"props":6794,"children":6795},{"style":6222},[6796],{"type":51,"value":6233},{"type":45,"tag":1314,"props":6798,"children":6799},{"style":6236},[6800],{"type":51,"value":6801},"nbformat",{"type":45,"tag":1314,"props":6803,"children":6804},{"style":6222},[6805],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6807,"children":6808},{"style":6222},[6809],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6811,"children":6812},{"style":6594},[6813],{"type":51,"value":6814}," 4",{"type":45,"tag":1314,"props":6816,"children":6817},{"style":6222},[6818],{"type":51,"value":6819},",\n",{"type":45,"tag":1314,"props":6821,"children":6822},{"class":1316,"line":1417},[6823,6827,6832,6836,6840],{"type":45,"tag":1314,"props":6824,"children":6825},{"style":6222},[6826],{"type":51,"value":6233},{"type":45,"tag":1314,"props":6828,"children":6829},{"style":6236},[6830],{"type":51,"value":6831},"nbformat_minor",{"type":45,"tag":1314,"props":6833,"children":6834},{"style":6222},[6835],{"type":51,"value":6244},{"type":45,"tag":1314,"props":6837,"children":6838},{"style":6222},[6839],{"type":51,"value":2871},{"type":45,"tag":1314,"props":6841,"children":6842},{"style":6594},[6843],{"type":51,"value":6844}," 5\n",{"type":45,"tag":1314,"props":6846,"children":6847},{"class":1316,"line":1426},[6848],{"type":45,"tag":1314,"props":6849,"children":6850},{"style":6222},[6851],{"type":51,"value":6781},{"type":45,"tag":2574,"props":6853,"children":6854},{},[],{"type":51,"value":6856},"One markdown cell + one code cell per section. Keep each code cell self-contained enough that re-running it in isolation makes sense.",{"type":45,"tag":70,"props":6858,"children":6859},{},[6860,6865,6867],{"type":45,"tag":58,"props":6861,"children":6862},{},[6863],{"type":51,"value":6864},"Best-effort syntax check",{"type":51,"value":6866}," via Bash. Don't fail the skill if the toolchain is missing — just report.",{"type":45,"tag":235,"props":6868,"children":6869},{},[6870,6885],{"type":45,"tag":70,"props":6871,"children":6872},{},[6873,6878,6879],{"type":45,"tag":76,"props":6874,"children":6876},{"className":6875},[],[6877],{"type":51,"value":349},{"type":51,"value":828},{"type":45,"tag":76,"props":6880,"children":6882},{"className":6881},[],[6883],{"type":51,"value":6884},"python -m py_compile \u003Cpath>",{"type":45,"tag":70,"props":6886,"children":6887},{},[6888,6893,6894],{"type":45,"tag":76,"props":6889,"children":6891},{"className":6890},[],[6892],{"type":51,"value":357},{"type":51,"value":828},{"type":45,"tag":76,"props":6895,"children":6897},{"className":6896},[],[6898],{"type":51,"value":6899},"python -c \"import json; nb = json.load(open('\u003Cpath>')); assert nb.get('cells'); print(f'cells={len(nb[\\\"cells\\\"])}')\"",{"type":45,"tag":70,"props":6901,"children":6902},{},[6903,6908],{"type":45,"tag":58,"props":6904,"children":6905},{},[6906],{"type":51,"value":6907},"Print next-steps",{"type":51,"value":6909}," (see Output section).",{"type":45,"tag":311,"props":6911,"children":6912},{},[],{"type":45,"tag":46,"props":6914,"children":6916},{"id":6915},"what-the-generated-code-must-not-do",[6917],{"type":51,"value":6918},"What the Generated Code MUST NOT Do",{"type":45,"tag":54,"props":6920,"children":6921},{},[6922,6924,6930],{"type":51,"value":6923},"A reviewer should be able to run these ",{"type":45,"tag":76,"props":6925,"children":6927},{"className":6926},[],[6928],{"type":51,"value":6929},"grep",{"type":51,"value":6931}," checks against the generated file and get zero matches:",{"type":45,"tag":441,"props":6933,"children":6934},{},[6935,6956],{"type":45,"tag":445,"props":6936,"children":6937},{},[6938],{"type":45,"tag":449,"props":6939,"children":6940},{},[6941,6951],{"type":45,"tag":453,"props":6942,"children":6943},{},[6944,6949],{"type":45,"tag":76,"props":6945,"children":6947},{"className":6946},[],[6948],{"type":51,"value":6929},{"type":51,"value":6950}," pattern",{"type":45,"tag":453,"props":6952,"children":6953},{},[6954],{"type":51,"value":6955},"Why it's wrong",{"type":45,"tag":469,"props":6957,"children":6958},{},[6959,6997,7028,7052,7078,7102],{"type":45,"tag":449,"props":6960,"children":6961},{},[6962,6978],{"type":45,"tag":476,"props":6963,"children":6964},{},[6965,6971,6972],{"type":45,"tag":76,"props":6966,"children":6968},{"className":6967},[],[6969],{"type":51,"value":6970},"uuid4",{"type":51,"value":431},{"type":45,"tag":76,"props":6973,"children":6975},{"className":6974},[],[6976],{"type":51,"value":6977},"uuid.uuid4",{"type":45,"tag":476,"props":6979,"children":6980},{},[6981,6987,6989,6995],{"type":45,"tag":76,"props":6982,"children":6984},{"className":6983},[],[6985],{"type":51,"value":6986},"record_id",{"type":51,"value":6988}," is minted by the SDK on ",{"type":45,"tag":76,"props":6990,"children":6992},{"className":6991},[],[6993],{"type":51,"value":6994},"dataset.append()",{"type":51,"value":6996},"; never client-generate.",{"type":45,"tag":449,"props":6998,"children":6999},{},[7000,7023],{"type":45,"tag":476,"props":7001,"children":7002},{},[7003,7009,7010,7016,7017],{"type":45,"tag":76,"props":7004,"children":7006},{"className":7005},[],[7007],{"type":51,"value":7008},"PATCH ",{"type":51,"value":431},{"type":45,"tag":76,"props":7011,"children":7013},{"className":7012},[],[7014],{"type":51,"value":7015},"batch_update",{"type":51,"value":431},{"type":45,"tag":76,"props":7018,"children":7020},{"className":7019},[],[7021],{"type":51,"value":7022},"records\u002Fupload",{"type":45,"tag":476,"props":7024,"children":7025},{},[7026],{"type":51,"value":7027},"Status state machine and dataset diff are SDK responsibilities.",{"type":45,"tag":449,"props":7029,"children":7030},{},[7031,7040],{"type":45,"tag":476,"props":7032,"children":7033},{},[7034],{"type":45,"tag":76,"props":7035,"children":7037},{"className":7036},[],[7038],{"type":51,"value":7039},"from ddtrace.llmobs._",{"type":45,"tag":476,"props":7041,"children":7042},{},[7043,7045,7051],{"type":51,"value":7044},"Private import paths. Always use ",{"type":45,"tag":76,"props":7046,"children":7048},{"className":7047},[],[7049],{"type":51,"value":7050},"from ddtrace.llmobs import ...",{"type":51,"value":110},{"type":45,"tag":449,"props":7053,"children":7054},{},[7055,7073],{"type":45,"tag":476,"props":7056,"children":7057},{},[7058,7064,7065,7071],{"type":45,"tag":76,"props":7059,"children":7061},{"className":7060},[],[7062],{"type":51,"value":7063},"\"record_id\"",{"type":51,"value":431},{"type":45,"tag":76,"props":7066,"children":7068},{"className":7067},[],[7069],{"type":51,"value":7070},"\"canonical_id\"",{"type":51,"value":7072}," (as dict keys in records)",{"type":45,"tag":476,"props":7074,"children":7075},{},[7076],{"type":51,"value":7077},"The SDK owns them.",{"type":45,"tag":449,"props":7079,"children":7080},{},[7081,7090],{"type":45,"tag":476,"props":7082,"children":7083},{},[7084],{"type":45,"tag":76,"props":7085,"children":7087},{"className":7086},[],[7088],{"type":51,"value":7089},"DD_API_KEY = \"\u003Cactual key>\"",{"type":45,"tag":476,"props":7091,"children":7092},{},[7093,7095,7101],{"type":51,"value":7094},"Always read from ",{"type":45,"tag":76,"props":7096,"children":7098},{"className":7097},[],[7099],{"type":51,"value":7100},"os.environ",{"type":51,"value":110},{"type":45,"tag":449,"props":7103,"children":7104},{},[7105,7121],{"type":45,"tag":476,"props":7106,"children":7107},{},[7108,7114,7115],{"type":45,"tag":76,"props":7109,"children":7111},{"className":7110},[],[7112],{"type":51,"value":7113},"requests.post",{"type":51,"value":431},{"type":45,"tag":76,"props":7116,"children":7118},{"className":7117},[],[7119],{"type":51,"value":7120},"httpx.post",{"type":45,"tag":476,"props":7122,"children":7123},{},[7124],{"type":51,"value":7125},"The skill produces SDK-only code. Direct HTTP calls bypass the SDK's lazy creation, push-diff, and bulk-threshold handling.",{"type":45,"tag":54,"props":7127,"children":7128},{},[7129],{"type":51,"value":7130},"If any of those slip into the output, the skill is wrong — re-emit.",{"type":45,"tag":311,"props":7132,"children":7133},{},[],{"type":45,"tag":46,"props":7135,"children":7137},{"id":7136},"output",[7138],{"type":51,"value":7139},"Output",{"type":45,"tag":54,"props":7141,"children":7142},{},[7143],{"type":51,"value":7144},"After writing, print:",{"type":45,"tag":396,"props":7146,"children":7149},{"className":7147,"code":7148,"language":51},[399],"Generated SDK experiment: \u003Cformat>\nPath: \u003Cpath>\nLines: \u003Ccount>   (or Cells: \u003Ccount> for .ipynb)\n\nPurpose:\n  \"\u003Cresolved purpose string from step 2.4>\"\n  (sourced via: --purpose flag | extracted from invocation message | AskUserQuestion default | AskUserQuestion + free text)\n\nSDK calls used:\n  ✓ LLMObs.enable(...)                       (line\u002Fcell ~\u003CN>)\n  ✓ LLMObs.\u003Ccreate_dataset|create_dataset_from_csv|pull_dataset>(...)  (line\u002Fcell ~\u003CN>)\n  ✓ task_fn(input_data, config)              (line\u002Fcell ~\u003CN>)\n  ✓ \u003CN> evaluators (style: \u003Cfunction|class|remote>, semantics seeded by purpose)\n  ✓ LLMObs.experiment(...).run(jobs=\u003CN>)     (line\u002Fcell ~\u003CN>)\n  ✓ Provenance (in config + tags): generated_by=claude-code, skill=agent-observability-experiment-py-bootstrap, purpose=...\n\nTask function source:\n  ✓ Wired to: \u003Cmodule.path>:\u003Cfunction_name>   (source: \u003Cfile>:\u003Cline>)\n  ✓ Adapter: \u003Cone line describing the input_data → call shape mapping>\n  ✓ Return shape: \u003Cplain string | {output, tool_calls} | {answer, retrieved_docs} | etc.>\n  ✓ Sync\u002Fasync: \u003Csync | async (wrapped with asyncio.run)>\n  [WARNING lines from the side-effect scan in Workflow step 2.5d, if any]\n  [Note line if the purpose requested richer return shape but the function only exposes a string]\n\n(If --placeholder-task was used or introspection found nothing:)\nTask function source:\n  ⚠ Placeholder task emitted (no real LLM call site found \u002F opted out).\n    Replace `task_fn` with your actual LLM call before running.\n\nSyntax check: \u003Cpass | skipped: toolchain missing | fail with details>\n\nInstall:\n  pip install \"ddtrace>=4.7\" \u003Cprovider-sdk-if-needed>\n  # python-dotenv is NOT required — the generated file ships its own .env loader.\n\nCredentials:\n  The generated file auto-discovers .env files at runtime. Discovery order\n  (first non-empty value wins per key; shell env always overrides files):\n    1. --env-file path baked in as ENV_FILE_OVERRIDE (if --env-file was passed)\n    2. \u003Coutput-file's-directory>\u002F.env\n    3. \u003Coutput-file's-directory>\u002F.env.local\n    4. \u003Ccwd>\u002F.env  and  \u003Ccwd>\u002F.env.local\n    5. parent-walk from cwd up to \u002F\n    6. ~\u002F.datadog\u002Fcredentials\n\n  Drop a .env at any of those locations with at minimum:\n    DD_API_KEY=...\n    DD_APPLICATION_KEY=...\n    DD_SITE=datadoghq.com           # only if not the US1 prod site\n    \u003CPROVIDER>_API_KEY=...           # the provider key the wired task needs\n  Or override on a per-run basis by exporting them in your shell — the loader\n  never overwrites a value that is already in os.environ.\n\nRun:\n  python \u003Cpath>                  # for --format py\n  jupyter notebook \u003Cpath>        # for --format ipynb\n\nNext steps:\n1. Confirm the wired task_fn matches the entry point you want to evaluate (see \"Task function source\" above).\n   Edit the import in section 4 of the generated file if you'd rather inline the call or pick a different function.\n2. Confirm the purpose (\"\u003Cpurpose string>\") matches what you actually want to measure — section 0 of the\n   file documents it, section 5's evaluators were seeded against it, and section 6's experiment carries\n   it as a config tag. Re-run this skill with `--purpose \"\u003Cnew text>\"` to regenerate against a different\n   target without changing your code.\n3. Adjust the evaluators if needed (or wire up RemoteEvaluator names you created in the Datadog UI).\n4. Run it. The script prints experiment.url at the end.\n5. Watch the experiment: https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Fexperiments\n",[7150],{"type":45,"tag":76,"props":7151,"children":7152},{"__ignoreMap":404},[7153],{"type":51,"value":7148},{"type":45,"tag":311,"props":7155,"children":7156},{},[],{"type":45,"tag":46,"props":7158,"children":7160},{"id":7159},"reference-notebook-patterns-use-as-templates",[7161],{"type":51,"value":7162},"Reference Notebook Patterns (use as templates)",{"type":45,"tag":54,"props":7164,"children":7165},{},[7166,7168,7173],{"type":51,"value":7167},"The canonical set lives at ",{"type":45,"tag":361,"props":7169,"children":7171},{"href":363,"rel":7170},[365],[7172],{"type":51,"value":363},{"type":51,"value":7174}," and serves as the style reference — the generated code should feel like it could have come from this set.",{"type":45,"tag":441,"props":7176,"children":7177},{},[7178,7194],{"type":45,"tag":445,"props":7179,"children":7180},{},[7181],{"type":45,"tag":449,"props":7182,"children":7183},{},[7184,7189],{"type":45,"tag":453,"props":7185,"children":7186},{},[7187],{"type":51,"value":7188},"Notebook",{"type":45,"tag":453,"props":7190,"children":7191},{},[7192],{"type":51,"value":7193},"Pattern demonstrated",{"type":45,"tag":469,"props":7195,"children":7196},{},[7197,7214,7231,7248,7270],{"type":45,"tag":449,"props":7198,"children":7199},{},[7200,7209],{"type":45,"tag":476,"props":7201,"children":7202},{},[7203],{"type":45,"tag":76,"props":7204,"children":7206},{"className":7205},[],[7207],{"type":51,"value":7208},"00-basic-datasets.ipynb",{"type":45,"tag":476,"props":7210,"children":7211},{},[7212],{"type":51,"value":7213},"Dataset create\u002Fappend\u002Fpush lifecycle",{"type":45,"tag":449,"props":7215,"children":7216},{},[7217,7226],{"type":45,"tag":476,"props":7218,"children":7219},{},[7220],{"type":45,"tag":76,"props":7221,"children":7223},{"className":7222},[],[7224],{"type":51,"value":7225},"01-basic-experiments.ipynb",{"type":45,"tag":476,"props":7227,"children":7228},{},[7229],{"type":51,"value":7230},"Minimum viable experiment — inline records, OpenAI task, 2 boolean evaluators",{"type":45,"tag":449,"props":7232,"children":7233},{},[7234,7243],{"type":45,"tag":476,"props":7235,"children":7236},{},[7237],{"type":45,"tag":76,"props":7238,"children":7240},{"className":7239},[],[7241],{"type":51,"value":7242},"02-extra-data.ipynb",{"type":45,"tag":476,"props":7244,"children":7245},{},[7246],{"type":51,"value":7247},"CSV-loaded dataset, multi-value task output, confidence-based evaluators",{"type":45,"tag":449,"props":7249,"children":7250},{},[7251,7260],{"type":45,"tag":476,"props":7252,"children":7253},{},[7254],{"type":45,"tag":76,"props":7255,"children":7257},{"className":7256},[],[7258],{"type":51,"value":7259},"04-multi-span-experiments.ipynb",{"type":45,"tag":476,"props":7261,"children":7262},{},[7263,7265],{"type":51,"value":7264},"Two-step LLM pipelines inside a single ",{"type":45,"tag":76,"props":7266,"children":7268},{"className":7267},[],[7269],{"type":51,"value":877},{"type":45,"tag":449,"props":7271,"children":7272},{},[7273,7282],{"type":45,"tag":476,"props":7274,"children":7275},{},[7276],{"type":45,"tag":76,"props":7277,"children":7279},{"className":7278},[],[7280],{"type":51,"value":7281},"07-remote-evaluators.ipynb",{"type":45,"tag":476,"props":7283,"children":7284},{},[7285,7290,7292],{"type":45,"tag":76,"props":7286,"children":7288},{"className":7287},[],[7289],{"type":51,"value":761},{"type":51,"value":7291}," with custom ",{"type":45,"tag":76,"props":7293,"children":7295},{"className":7294},[],[7296],{"type":51,"value":7297},"transform_fn",{"type":45,"tag":54,"props":7299,"children":7300},{},[7301,7302,7308,7310,7316,7318,7323,7325,7331,7333,7338,7339,7345,7347,7353],{"type":51,"value":2344},{"type":45,"tag":76,"props":7303,"children":7305},{"className":7304},[],[7306],{"type":51,"value":7307},"--evaluator-style remote",{"type":51,"value":7309},", lean toward the ",{"type":45,"tag":76,"props":7311,"children":7313},{"className":7312},[],[7314],{"type":51,"value":7315},"07",{"type":51,"value":7317}," style. When ",{"type":45,"tag":76,"props":7319,"children":7321},{"className":7320},[],[7322],{"type":51,"value":529},{"type":51,"value":7324}," is a CSV, lean toward ",{"type":45,"tag":76,"props":7326,"children":7328},{"className":7327},[],[7329],{"type":51,"value":7330},"02",{"type":51,"value":7332},". Default (no ",{"type":45,"tag":76,"props":7334,"children":7336},{"className":7335},[],[7337],{"type":51,"value":529},{"type":51,"value":431},{"type":45,"tag":76,"props":7340,"children":7342},{"className":7341},[],[7343],{"type":51,"value":7344},"--evaluator-style function",{"type":51,"value":7346},") is the ",{"type":45,"tag":76,"props":7348,"children":7350},{"className":7349},[],[7351],{"type":51,"value":7352},"01",{"type":51,"value":7354}," style.",{"type":45,"tag":311,"props":7356,"children":7357},{},[],{"type":45,"tag":46,"props":7359,"children":7361},{"id":7360},"datadog-documentation",[7362],{"type":51,"value":7363},"Datadog Documentation",{"type":45,"tag":54,"props":7365,"children":7366},{},[7367,7369,7375],{"type":51,"value":7368},"These are the canonical reference pages on ",{"type":45,"tag":361,"props":7370,"children":7373},{"href":7371,"rel":7372},"https:\u002F\u002Fdocs.datadoghq.com\u002F",[365],[7374],{"type":51,"value":7371},{"type":51,"value":7376},". Use them to ground answers about Agent Observability features and to look up details that aren't covered in this skill.",{"type":45,"tag":441,"props":7378,"children":7379},{},[7380,7401],{"type":45,"tag":445,"props":7381,"children":7382},{},[7383],{"type":45,"tag":449,"props":7384,"children":7385},{},[7386,7391,7396],{"type":45,"tag":453,"props":7387,"children":7388},{},[7389],{"type":51,"value":7390},"Topic",{"type":45,"tag":453,"props":7392,"children":7393},{},[7394],{"type":51,"value":7395},"URL",{"type":45,"tag":453,"props":7397,"children":7398},{},[7399],{"type":51,"value":7400},"Use when",{"type":45,"tag":469,"props":7402,"children":7403},{},[7404,7426,7448,7470,7492,7520,7542,7569,7591,7613,7635,7657,7687,7709],{"type":45,"tag":449,"props":7405,"children":7406},{},[7407,7412,7421],{"type":45,"tag":476,"props":7408,"children":7409},{},[7410],{"type":51,"value":7411},"Agent Observability overview",{"type":45,"tag":476,"props":7413,"children":7414},{},[7415],{"type":45,"tag":361,"props":7416,"children":7419},{"href":7417,"rel":7418},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002F",[365],[7420],{"type":51,"value":7417},{"type":45,"tag":476,"props":7422,"children":7423},{},[7424],{"type":51,"value":7425},"Establishing what the product covers, terminology",{"type":45,"tag":449,"props":7427,"children":7428},{},[7429,7434,7443],{"type":45,"tag":476,"props":7430,"children":7431},{},[7432],{"type":51,"value":7433},"Setup",{"type":45,"tag":476,"props":7435,"children":7436},{},[7437],{"type":45,"tag":361,"props":7438,"children":7441},{"href":7439,"rel":7440},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fsetup\u002F",[365],[7442],{"type":51,"value":7439},{"type":45,"tag":476,"props":7444,"children":7445},{},[7446],{"type":51,"value":7447},"API\u002Fapp key creation, project + ml_app setup, region\u002Fsite selection",{"type":45,"tag":449,"props":7449,"children":7450},{},[7451,7456,7465],{"type":45,"tag":476,"props":7452,"children":7453},{},[7454],{"type":51,"value":7455},"Instrumentation overview",{"type":45,"tag":476,"props":7457,"children":7458},{},[7459],{"type":45,"tag":361,"props":7460,"children":7463},{"href":7461,"rel":7462},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Finstrumentation\u002F",[365],[7464],{"type":51,"value":7461},{"type":45,"tag":476,"props":7466,"children":7467},{},[7468],{"type":51,"value":7469},"Auto-instrumentation, manual SDK usage, span model",{"type":45,"tag":449,"props":7471,"children":7472},{},[7473,7478,7487],{"type":45,"tag":476,"props":7474,"children":7475},{},[7476],{"type":51,"value":7477},"Python SDK reference",{"type":45,"tag":476,"props":7479,"children":7480},{},[7481],{"type":45,"tag":361,"props":7482,"children":7485},{"href":7483,"rel":7484},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Finstrumentation\u002Fsdk\u002F",[365],[7486],{"type":51,"value":7483},{"type":45,"tag":476,"props":7488,"children":7489},{},[7490],{"type":51,"value":7491},"Public symbol list, decorator semantics, span kinds, annotate\u002Fenable signatures",{"type":45,"tag":449,"props":7493,"children":7494},{},[7495,7500,7509],{"type":45,"tag":476,"props":7496,"children":7497},{},[7498],{"type":51,"value":7499},"Experiments",{"type":45,"tag":476,"props":7501,"children":7502},{},[7503],{"type":45,"tag":361,"props":7504,"children":7507},{"href":7505,"rel":7506},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fexperiments\u002F",[365],[7508],{"type":51,"value":7505},{"type":45,"tag":476,"props":7510,"children":7511},{},[7512,7518],{"type":45,"tag":76,"props":7513,"children":7515},{"className":7514},[],[7516],{"type":51,"value":7517},"LLMObs.experiment(...)",{"type":51,"value":7519},", dataset lifecycle, eval streaming, status states",{"type":45,"tag":449,"props":7521,"children":7522},{},[7523,7528,7537],{"type":45,"tag":476,"props":7524,"children":7525},{},[7526],{"type":51,"value":7527},"Evaluations",{"type":45,"tag":476,"props":7529,"children":7530},{},[7531],{"type":45,"tag":361,"props":7532,"children":7535},{"href":7533,"rel":7534},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002F",[365],[7536],{"type":51,"value":7533},{"type":45,"tag":476,"props":7538,"children":7539},{},[7540],{"type":51,"value":7541},"Evaluator concepts, managed vs custom evaluators",{"type":45,"tag":449,"props":7543,"children":7544},{},[7545,7550,7559],{"type":45,"tag":476,"props":7546,"children":7547},{},[7548],{"type":51,"value":7549},"Custom LLM-as-a-judge evaluations",{"type":45,"tag":476,"props":7551,"children":7552},{},[7553],{"type":45,"tag":361,"props":7554,"children":7557},{"href":7555,"rel":7556},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fcustom_llm_as_a_judge_evaluations\u002F",[365],[7558],{"type":51,"value":7555},{"type":45,"tag":476,"props":7560,"children":7561},{},[7562,7567],{"type":45,"tag":76,"props":7563,"children":7565},{"className":7564},[],[7566],{"type":51,"value":761},{"type":51,"value":7568}," payload shape and rubric design",{"type":45,"tag":449,"props":7570,"children":7571},{},[7572,7577,7586],{"type":45,"tag":476,"props":7573,"children":7574},{},[7575],{"type":51,"value":7576},"Managed evaluations",{"type":45,"tag":476,"props":7578,"children":7579},{},[7580],{"type":45,"tag":361,"props":7581,"children":7584},{"href":7582,"rel":7583},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fmanaged_evaluations\u002F",[365],[7585],{"type":51,"value":7582},{"type":45,"tag":476,"props":7587,"children":7588},{},[7589],{"type":51,"value":7590},"Pre-built judges (faithfulness, toxicity, etc.)",{"type":45,"tag":449,"props":7592,"children":7593},{},[7594,7599,7608],{"type":45,"tag":476,"props":7595,"children":7596},{},[7597],{"type":51,"value":7598},"Monitoring",{"type":45,"tag":476,"props":7600,"children":7601},{},[7602],{"type":45,"tag":361,"props":7603,"children":7606},{"href":7604,"rel":7605},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fmonitoring\u002F",[365],[7607],{"type":51,"value":7604},{"type":45,"tag":476,"props":7609,"children":7610},{},[7611],{"type":51,"value":7612},"Alerts, dashboards, span-level monitors",{"type":45,"tag":449,"props":7614,"children":7615},{},[7616,7621,7630],{"type":45,"tag":476,"props":7617,"children":7618},{},[7619],{"type":51,"value":7620},"Terms \u002F glossary",{"type":45,"tag":476,"props":7622,"children":7623},{},[7624],{"type":45,"tag":361,"props":7625,"children":7628},{"href":7626,"rel":7627},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fterms\u002F",[365],[7629],{"type":51,"value":7626},{"type":45,"tag":476,"props":7631,"children":7632},{},[7633],{"type":51,"value":7634},"Span kinds, sessions, traces, ml_app",{"type":45,"tag":449,"props":7636,"children":7637},{},[7638,7643,7652],{"type":45,"tag":476,"props":7639,"children":7640},{},[7641],{"type":51,"value":7642},"Evaluation developer guide",{"type":45,"tag":476,"props":7644,"children":7645},{},[7646],{"type":45,"tag":361,"props":7647,"children":7650},{"href":7648,"rel":7649},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fguide\u002Fevaluation_developer_guide\u002F",[365],[7651],{"type":51,"value":7648},{"type":45,"tag":476,"props":7653,"children":7654},{},[7655],{"type":51,"value":7656},"Writing offline evaluators, validation strategy",{"type":45,"tag":449,"props":7658,"children":7659},{},[7660,7665,7674],{"type":45,"tag":476,"props":7661,"children":7662},{},[7663],{"type":51,"value":7664},"Claude Code skills guide",{"type":45,"tag":476,"props":7666,"children":7667},{},[7668],{"type":45,"tag":361,"props":7669,"children":7672},{"href":7670,"rel":7671},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fguide\u002Fclaude_code_skills\u002F",[365],[7673],{"type":51,"value":7670},{"type":45,"tag":476,"props":7675,"children":7676},{},[7677,7679,7685],{"type":51,"value":7678},"How this skill fits alongside the rest of the ",{"type":45,"tag":76,"props":7680,"children":7682},{"className":7681},[],[7683],{"type":51,"value":7684},"agent-observability",{"type":51,"value":7686}," set",{"type":45,"tag":449,"props":7688,"children":7689},{},[7690,7695,7704],{"type":45,"tag":476,"props":7691,"children":7692},{},[7693],{"type":51,"value":7694},"MCP server",{"type":45,"tag":476,"props":7696,"children":7697},{},[7698],{"type":45,"tag":361,"props":7699,"children":7702},{"href":7700,"rel":7701},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fmcp_server\u002F",[365],[7703],{"type":51,"value":7700},{"type":45,"tag":476,"props":7705,"children":7706},{},[7707],{"type":51,"value":7708},"Connecting MCP-compatible clients to Agent Observability data",{"type":45,"tag":449,"props":7710,"children":7711},{},[7712,7717,7725],{"type":45,"tag":476,"props":7713,"children":7714},{},[7715],{"type":51,"value":7716},"Reference notebooks (GitHub)",{"type":45,"tag":476,"props":7718,"children":7719},{},[7720],{"type":45,"tag":361,"props":7721,"children":7723},{"href":363,"rel":7722},[365],[7724],{"type":51,"value":363},{"type":45,"tag":476,"props":7726,"children":7727},{},[7728,7730,7735,7736],{"type":51,"value":7729},"Style-of-life examples for the generated ",{"type":45,"tag":76,"props":7731,"children":7733},{"className":7732},[],[7734],{"type":51,"value":349},{"type":51,"value":934},{"type":45,"tag":76,"props":7737,"children":7739},{"className":7738},[],[7740],{"type":51,"value":357},{"type":45,"tag":413,"props":7742,"children":7744},{"id":7743},"researching-features-the-skill-does-not-cover",[7745],{"type":51,"value":7746},"Researching features the skill does not cover",{"type":45,"tag":54,"props":7748,"children":7749},{},[7750,7752,7758],{"type":51,"value":7751},"If the user asks about an Agent Observability feature the skill's body doesn't address (e.g., specific span kinds, dataset versioning semantics, an evaluator type not covered above), fetch the relevant page from ",{"type":45,"tag":76,"props":7753,"children":7755},{"className":7754},[],[7756],{"type":51,"value":7757},"docs.datadoghq.com",{"type":51,"value":7759}," rather than guessing:",{"type":45,"tag":66,"props":7761,"children":7762},{},[7763,7780,7817,7840],{"type":45,"tag":70,"props":7764,"children":7765},{},[7766,7771,7773,7779],{"type":45,"tag":58,"props":7767,"children":7768},{},[7769],{"type":51,"value":7770},"Pick the most specific URL",{"type":51,"value":7772}," from the table above. Most Agent Observability questions resolve under ",{"type":45,"tag":76,"props":7774,"children":7776},{"className":7775},[],[7777],{"type":51,"value":7778},"\u002Fllm_observability\u002F{experiments,evaluations,instrumentation,monitoring}\u002F",{"type":51,"value":110},{"type":45,"tag":70,"props":7781,"children":7782},{},[7783,7793,7795,7801,7803,7808,7810,7816],{"type":45,"tag":58,"props":7784,"children":7785},{},[7786,7787],{"type":51,"value":6178},{"type":45,"tag":76,"props":7788,"children":7790},{"className":7789},[],[7791],{"type":51,"value":7792},"WebFetch",{"type":51,"value":7794}," on that URL with a focused query (e.g., ",{"type":45,"tag":76,"props":7796,"children":7798},{"className":7797},[],[7799],{"type":51,"value":7800},"\"How does Dataset.push() handle the 5 MB threshold?\"",{"type":51,"value":7802},"). Prefer ",{"type":45,"tag":76,"props":7804,"children":7806},{"className":7805},[],[7807],{"type":51,"value":7792},{"type":51,"value":7809}," over generic web search — the canonical page is almost always under ",{"type":45,"tag":76,"props":7811,"children":7813},{"className":7812},[],[7814],{"type":51,"value":7815},"docs.datadoghq.com\u002Fllm_observability\u002F",{"type":51,"value":110},{"type":45,"tag":70,"props":7818,"children":7819},{},[7820,7831,7832,7838],{"type":45,"tag":58,"props":7821,"children":7822},{},[7823,7825],{"type":51,"value":7824},"Fall back to ",{"type":45,"tag":76,"props":7826,"children":7828},{"className":7827},[],[7829],{"type":51,"value":7830},"WebSearch",{"type":51,"value":184},{"type":45,"tag":76,"props":7833,"children":7835},{"className":7834},[],[7836],{"type":51,"value":7837},"site:docs.datadoghq.com\u002Fllm_observability",{"type":51,"value":7839}," if you don't know which subpage owns the topic.",{"type":45,"tag":70,"props":7841,"children":7842},{},[7843,7848],{"type":45,"tag":58,"props":7844,"children":7845},{},[7846],{"type":51,"value":7847},"Cite the page",{"type":51,"value":7849}," in the answer with its URL so the user can verify and bookmark.",{"type":45,"tag":54,"props":7851,"children":7852},{},[7853,7855,7861],{"type":51,"value":7854},"Never invent symbols or behaviors not present in this skill body or the docs above. If the docs don't cover the question either, say so explicitly and suggest filing an issue on ",{"type":45,"tag":76,"props":7856,"children":7858},{"className":7857},[],[7859],{"type":51,"value":7860},"DataDog\u002Fllm-observability",{"type":51,"value":7862}," rather than fabricating a workaround.",{"type":45,"tag":311,"props":7864,"children":7865},{},[],{"type":45,"tag":46,"props":7867,"children":7869},{"id":7868},"operating-rules",[7870],{"type":51,"value":7871},"Operating Rules",{"type":45,"tag":235,"props":7873,"children":7874},{},[7875,7892,7923,7956,8005,8023,8078,8193,8224,8234,8265,8280,8336,8354,8486,8502,8533,8549,8574],{"type":45,"tag":70,"props":7876,"children":7877},{},[7878,7883,7885,7890],{"type":45,"tag":58,"props":7879,"children":7880},{},[7881],{"type":51,"value":7882},"SDK only.",{"type":51,"value":7884}," No ",{"type":45,"tag":76,"props":7886,"children":7888},{"className":7887},[],[7889],{"type":51,"value":7113},{"type":51,"value":7891},", no manual JSON:API envelope construction, no manual ID generation. If a feature seems to require those, you're solving the wrong problem — the SDK already covers it.",{"type":45,"tag":70,"props":7893,"children":7894},{},[7895,7900,7901,7906,7908,7914,7915,7921],{"type":45,"tag":58,"props":7896,"children":7897},{},[7898],{"type":51,"value":7899},"Public imports only.",{"type":51,"value":2411},{"type":45,"tag":76,"props":7902,"children":7904},{"className":7903},[],[7905],{"type":51,"value":7050},{"type":51,"value":7907},". Never ",{"type":45,"tag":76,"props":7909,"children":7911},{"className":7910},[],[7912],{"type":51,"value":7913},"_experiment",{"type":51,"value":431},{"type":45,"tag":76,"props":7916,"children":7918},{"className":7917},[],[7919],{"type":51,"value":7920},"_llmobs",{"type":51,"value":7922},", or any underscore-prefixed module.",{"type":45,"tag":70,"props":7924,"children":7925},{},[7926,7931,7933,7938,7940,7946,7948,7954],{"type":45,"tag":58,"props":7927,"children":7928},{},[7929],{"type":51,"value":7930},"Env vars, not literals.",{"type":51,"value":7932}," Credentials always read from ",{"type":45,"tag":76,"props":7934,"children":7936},{"className":7935},[],[7937],{"type":51,"value":7100},{"type":51,"value":7939},". The generated ",{"type":45,"tag":76,"props":7941,"children":7943},{"className":7942},[],[7944],{"type":51,"value":7945},"main()",{"type":51,"value":7947}," (or the env-setup cell) must ",{"type":45,"tag":76,"props":7949,"children":7951},{"className":7950},[],[7952],{"type":51,"value":7953},"assert",{"type":51,"value":7955}," they're set with a clear message.",{"type":45,"tag":70,"props":7957,"children":7958},{},[7959,7964,7966,7972,7974,7980,7982,7988,7990,7995,7997,8003],{"type":45,"tag":58,"props":7960,"children":7961},{},[7962],{"type":51,"value":7963},"Auto-discover, don't push setup work onto the user.",{"type":51,"value":7965}," Section 1 always emits the ",{"type":45,"tag":76,"props":7967,"children":7969},{"className":7968},[],[7970],{"type":51,"value":7971},"_load_env_files",{"type":51,"value":7973}," helper (no ",{"type":45,"tag":76,"props":7975,"children":7977},{"className":7976},[],[7978],{"type":51,"value":7979},"python-dotenv",{"type":51,"value":7981}," dependency). It walks the discovery order documented in Workflow step 2.6 and prints which file(s) it loaded. Never substitute ",{"type":45,"tag":76,"props":7983,"children":7985},{"className":7984},[],[7986],{"type":51,"value":7987},"load_dotenv()",{"type":51,"value":7989}," from the third-party ",{"type":45,"tag":76,"props":7991,"children":7993},{"className":7992},[],[7994],{"type":51,"value":7979},{"type":51,"value":7996}," package — the inline helper has zero dependencies and is identical in behavior. Shell env vars always win over file-loaded values so the user can override any auto-discovered value by ",{"type":45,"tag":76,"props":7998,"children":8000},{"className":7999},[],[8001],{"type":51,"value":8002},"export \u003CKEY>=...",{"type":51,"value":8004}," before re-running.",{"type":45,"tag":70,"props":8006,"children":8007},{},[8008,8013,8015,8021],{"type":45,"tag":58,"props":8009,"children":8010},{},[8011],{"type":51,"value":8012},"Provider-key asserts must match the wired task.",{"type":51,"value":8014}," Generate the assert for ",{"type":45,"tag":76,"props":8016,"children":8018},{"className":8017},[],[8019],{"type":51,"value":8020},"OPENAI_API_KEY",{"type":51,"value":8022}," only if the task imports \u002F calls OpenAI; same for Anthropic \u002F Gemini \u002F Bedrock \u002F etc. Per the Workflow step 2.6 table. Never emit asserts for provider keys the task does not actually need — they're confusing and cause spurious \"missing key\" failures.",{"type":45,"tag":70,"props":8024,"children":8025},{},[8026,8046,8048,8054,8056,8061,8063,8069,8070,8076],{"type":45,"tag":58,"props":8027,"children":8028},{},[8029,8031,8037,8039,8045],{"type":51,"value":8030},"Always pass ",{"type":45,"tag":76,"props":8032,"children":8034},{"className":8033},[],[8035],{"type":51,"value":8036},"site=",{"type":51,"value":8038}," to ",{"type":45,"tag":76,"props":8040,"children":8042},{"className":8041},[],[8043],{"type":51,"value":8044},"LLMObs.enable()",{"type":51,"value":110},{"type":51,"value":8047}," Read it from ",{"type":45,"tag":76,"props":8049,"children":8051},{"className":8050},[],[8052],{"type":51,"value":8053},"os.getenv(\"DD_SITE\", \"datadoghq.com\")",{"type":51,"value":8055},". Omitting ",{"type":45,"tag":76,"props":8057,"children":8059},{"className":8058},[],[8060],{"type":51,"value":8036},{"type":51,"value":8062}," silently defaults to US1 prod, which breaks every non-prod org (e.g. staging ",{"type":45,"tag":76,"props":8064,"children":8066},{"className":8065},[],[8067],{"type":51,"value":8068},"datad0g.com",{"type":51,"value":431},{"type":45,"tag":76,"props":8071,"children":8073},{"className":8072},[],[8074],{"type":51,"value":8075},"datadoghq.eu",{"type":51,"value":8077},"). The canonical signature already includes it — never drop it.",{"type":45,"tag":70,"props":8079,"children":8080},{},[8081,8101,8103,8108,8110,8116,8118,8123,8125,8131,8132,8138,8139,8145,8147,8153,8154,8160,8162,8168,8170,8176,8178,8183,8185,8191],{"type":45,"tag":58,"props":8082,"children":8083},{},[8084,8086,8091,8093,8099],{"type":51,"value":8085},"Per-record ",{"type":45,"tag":76,"props":8087,"children":8089},{"className":8088},[],[8090],{"type":51,"value":2278},{"type":51,"value":8092}," are ",{"type":45,"tag":76,"props":8094,"children":8096},{"className":8095},[],[8097],{"type":51,"value":8098},"\"key:value\"",{"type":51,"value":8100}," strings.",{"type":51,"value":8102}," When inlining records (whether from ",{"type":45,"tag":76,"props":8104,"children":8106},{"className":8105},[],[8107],{"type":51,"value":529},{"type":51,"value":8109}," JSON, CSV, or the default sample), each entry in a record's ",{"type":45,"tag":76,"props":8111,"children":8113},{"className":8112},[],[8114],{"type":51,"value":8115},"\"tags\"",{"type":51,"value":8117}," list must be a ",{"type":45,"tag":76,"props":8119,"children":8121},{"className":8120},[],[8122],{"type":51,"value":8098},{"type":51,"value":8124}," string like ",{"type":45,"tag":76,"props":8126,"children":8128},{"className":8127},[],[8129],{"type":51,"value":8130},"\"env:prod\"",{"type":51,"value":431},{"type":45,"tag":76,"props":8133,"children":8135},{"className":8134},[],[8136],{"type":51,"value":8137},"\"source:traces\"",{"type":51,"value":431},{"type":45,"tag":76,"props":8140,"children":8142},{"className":8141},[],[8143],{"type":51,"value":8144},"\"category:geography\"",{"type":51,"value":8146},". Bare strings (",{"type":45,"tag":76,"props":8148,"children":8150},{"className":8149},[],[8151],{"type":51,"value":8152},"\"smoke\"",{"type":51,"value":431},{"type":45,"tag":76,"props":8155,"children":8157},{"className":8156},[],[8158],{"type":51,"value":8159},"\"baseline\"",{"type":51,"value":8161},") trigger ",{"type":45,"tag":76,"props":8163,"children":8165},{"className":8164},[],[8166],{"type":51,"value":8167},"ValueError: Tag '\u003Cname>' is malformed.",{"type":51,"value":8169}," at ",{"type":45,"tag":76,"props":8171,"children":8173},{"className":8172},[],[8174],{"type":51,"value":8175},"Dataset.append()",{"type":51,"value":8177}," time. If the source data has bare-string tags, namespace them — e.g. wrap ",{"type":45,"tag":76,"props":8179,"children":8181},{"className":8180},[],[8182],{"type":51,"value":8152},{"type":51,"value":8184}," as ",{"type":45,"tag":76,"props":8186,"children":8188},{"className":8187},[],[8189],{"type":51,"value":8190},"\"tag:smoke\"",{"type":51,"value":8192}," rather than dropping it.",{"type":45,"tag":70,"props":8194,"children":8195},{},[8196,8201,8203,8208,8210,8215,8217,8222],{"type":45,"tag":58,"props":8197,"children":8198},{},[8199],{"type":51,"value":8200},"Always resolve a purpose before generating.",{"type":51,"value":8202}," Step 2.4 must produce a non-empty ",{"type":45,"tag":76,"props":8204,"children":8206},{"className":8205},[],[8207],{"type":51,"value":3236},{"type":51,"value":8209}," string before steps 2.5 \u002F 3 \u002F 4 run. If the user provides one via ",{"type":45,"tag":76,"props":8211,"children":8213},{"className":8212},[],[8214],{"type":51,"value":1031},{"type":51,"value":8216}," or in their invocation, use it. Otherwise prompt via ",{"type":45,"tag":76,"props":8218,"children":8220},{"className":8219},[],[8221],{"type":51,"value":1042},{"type":51,"value":8223}," with the 5 seed options + Other. Never skip the prompt; never proceed with a blank purpose. A weak purpose (\"test it\") is still better than no purpose — generic accuracy semantics will at least seed reasonable defaults.",{"type":45,"tag":70,"props":8225,"children":8226},{},[8227,8232],{"type":45,"tag":58,"props":8228,"children":8229},{},[8230],{"type":51,"value":8231},"Treat the purpose as reasoning input, not a switch statement.",{"type":51,"value":8233}," There is no hardcoded mapping from purpose strings to evaluator code or wrapper shape. Read the purpose, reason about what's being measured, and emit appropriately. The same purpose string may produce different output for two different apps (a tool-call purpose in a LangChain app generates different wrapper code than in a raw OpenAI app) — that's expected.",{"type":45,"tag":70,"props":8235,"children":8236},{},[8237,8242,8244,8249,8251,8256,8258,8263],{"type":45,"tag":58,"props":8238,"children":8239},{},[8240],{"type":51,"value":8241},"Introspect first, placeholder last.",{"type":51,"value":8243}," The default behavior is Workflow step 2.5 — scan the user's app, find the LLM entry point, wire ",{"type":45,"tag":76,"props":8245,"children":8247},{"className":8246},[],[8248],{"type":51,"value":877},{"type":51,"value":8250}," to it. A ",{"type":45,"tag":76,"props":8252,"children":8254},{"className":8253},[],[8255],{"type":51,"value":907},{"type":51,"value":8257}," marker in the task section is only acceptable when introspection genuinely found nothing or the user passed ",{"type":45,"tag":76,"props":8259,"children":8261},{"className":8260},[],[8262],{"type":51,"value":891},{"type":51,"value":8264},". Never emit a placeholder task when a real candidate exists in the project — that's the failure mode this skill exists to fix.",{"type":45,"tag":70,"props":8266,"children":8267},{},[8268,8278],{"type":45,"tag":58,"props":8269,"children":8270},{},[8271,8276],{"type":45,"tag":76,"props":8272,"children":8274},{"className":8273},[],[8275],{"type":51,"value":907},{"type":51,"value":8277}," markers on at least one evaluator",{"type":51,"value":8279}," so reviewers can't ship un-customized evaluators by accident. (Evaluators stay user-owned even when the task is auto-wired.)",{"type":45,"tag":70,"props":8281,"children":8282},{},[8283,8288,8290,8295,8297,8302,8304,8309,8310,8315,8316,8321,8323,8328,8329,8334],{"type":45,"tag":58,"props":8284,"children":8285},{},[8286],{"type":51,"value":8287},"Introspection is bounded.",{"type":51,"value":8289}," The scan in Workflow step 2.5 must respect ",{"type":45,"tag":76,"props":8291,"children":8293},{"className":8292},[],[8294],{"type":51,"value":921},{"type":51,"value":8296}," (or its default-resolved value), ",{"type":45,"tag":76,"props":8298,"children":8300},{"className":8299},[],[8301],{"type":51,"value":3646},{"type":51,"value":8303}," if present, and the directory blocklist (",{"type":45,"tag":76,"props":8305,"children":8307},{"className":8306},[],[8308],{"type":51,"value":3575},{"type":51,"value":431},{"type":45,"tag":76,"props":8311,"children":8313},{"className":8312},[],[8314],{"type":51,"value":3582},{"type":51,"value":431},{"type":45,"tag":76,"props":8317,"children":8319},{"className":8318},[],[8320],{"type":51,"value":3596},{"type":51,"value":8322},", etc.). Refuse to scan ",{"type":45,"tag":76,"props":8324,"children":8326},{"className":8325},[],[8327],{"type":51,"value":3328},{"type":51,"value":835},{"type":45,"tag":76,"props":8330,"children":8332},{"className":8331},[],[8333],{"type":51,"value":3660},{"type":51,"value":8335},". If a scan would touch more than ~10k Python files, narrow the root or ask the user to point at the relevant subdirectory.",{"type":45,"tag":70,"props":8337,"children":8338},{},[8339,8344,8346,8352],{"type":45,"tag":58,"props":8340,"children":8341},{},[8342],{"type":51,"value":8343},"Match notebook conventions.",{"type":51,"value":8345}," Plain function evaluators by default; class-based only when the user opts in. Print ",{"type":45,"tag":76,"props":8347,"children":8349},{"className":8348},[],[8350],{"type":51,"value":8351},"experiment.url",{"type":51,"value":8353}," at the end of every generated file.",{"type":45,"tag":70,"props":8355,"children":8356},{},[8357,8374,8376,8381,8382,8387,8389,8395,8396,8402,8404,8410,8412,8417,8419,8425,8427,8432,8433,8439,8441,8447,8449,8455,8457,8462,8464,8469,8471,8476,8478,8484],{"type":45,"tag":58,"props":8358,"children":8359},{},[8360,8362,8367,8368,8373],{"type":51,"value":8361},"Tag every experiment with provenance + purpose — in both ",{"type":45,"tag":76,"props":8363,"children":8365},{"className":8364},[],[8366],{"type":51,"value":4767},{"type":51,"value":2318},{"type":45,"tag":76,"props":8369,"children":8371},{"className":8370},[],[8372],{"type":51,"value":2278},{"type":51,"value":110},{"type":51,"value":8375}," Every ",{"type":45,"tag":76,"props":8377,"children":8379},{"className":8378},[],[8380],{"type":51,"value":7517},{"type":51,"value":247},{"type":45,"tag":58,"props":8383,"children":8384},{},[8385],{"type":51,"value":8386},"must",{"type":51,"value":8388}," carry ",{"type":45,"tag":76,"props":8390,"children":8392},{"className":8391},[],[8393],{"type":51,"value":8394},"\"generated_by\": \"claude-code\"",{"type":51,"value":431},{"type":45,"tag":76,"props":8397,"children":8399},{"className":8398},[],[8400],{"type":51,"value":8401},"\"skill\": \"agent-observability-experiment-py-bootstrap\"",{"type":51,"value":8403},", AND ",{"type":45,"tag":76,"props":8405,"children":8407},{"className":8406},[],[8408],{"type":51,"value":8409},"\"purpose\": \"\u003Cstep 2.4 string>\"",{"type":51,"value":8411}," as keys in ",{"type":45,"tag":58,"props":8413,"children":8414},{},[8415],{"type":51,"value":8416},"both",{"type":51,"value":8418}," the ",{"type":45,"tag":76,"props":8420,"children":8422},{"className":8421},[],[8423],{"type":51,"value":8424},"config={...}",{"type":51,"value":8426}," dict (so they render in the experiment's Configuration view, which is where users actually look) ",{"type":45,"tag":58,"props":8428,"children":8429},{},[8430],{"type":51,"value":8431},"and",{"type":51,"value":8418},{"type":45,"tag":76,"props":8434,"children":8436},{"className":8435},[],[8437],{"type":51,"value":8438},"tags={...}",{"type":51,"value":8440}," dict (which the SDK serializes into ",{"type":45,"tag":76,"props":8442,"children":8444},{"className":8443},[],[8445],{"type":51,"value":8446},"metadata.tags",{"type":51,"value":8448}," for future tag-filter consumers). The ",{"type":45,"tag":76,"props":8450,"children":8452},{"className":8451},[],[8453],{"type":51,"value":8454},"tags=",{"type":51,"value":8456}," path alone is not enough: the current LLM Experiments UI does not surface ",{"type":45,"tag":76,"props":8458,"children":8460},{"className":8459},[],[8461],{"type":51,"value":8446},{"type":51,"value":8463}," as filterable chips, so users won't see the values unless they're also in ",{"type":45,"tag":76,"props":8465,"children":8467},{"className":8466},[],[8468],{"type":51,"value":4767},{"type":51,"value":8470},". The ",{"type":45,"tag":76,"props":8472,"children":8474},{"className":8473},[],[8475],{"type":51,"value":3236},{"type":51,"value":8477}," field is what makes future runs of the same experiment discoverable by intent — without it, users see ten experiments with cryptic names and no idea what each was testing. Also set ",{"type":45,"tag":76,"props":8479,"children":8481},{"className":8480},[],[8482],{"type":51,"value":8483},"description=\"\u003Cpurpose>\"",{"type":51,"value":8485}," on the experiment so the UI list view shows it.",{"type":45,"tag":70,"props":8487,"children":8488},{},[8489,8494,8495,8500],{"type":45,"tag":58,"props":8490,"children":8491},{},[8492],{"type":51,"value":8493},"PII scrub at the door.",{"type":51,"value":3523},{"type":45,"tag":76,"props":8496,"children":8498},{"className":8497},[],[8499],{"type":51,"value":529},{"type":51,"value":8501}," is given, scrub before inlining into the generated file. Never embed a record that contains an unmasked email\u002Fphone\u002FSSN\u002FAPI-key pattern.",{"type":45,"tag":70,"props":8503,"children":8504},{},[8505,8523,8525,8531],{"type":45,"tag":58,"props":8506,"children":8507},{},[8508,8510,8516,8517,8522],{"type":51,"value":8509},"Don't generate ",{"type":45,"tag":76,"props":8511,"children":8513},{"className":8512},[],[8514],{"type":51,"value":8515},"requirements.txt",{"type":51,"value":835},{"type":45,"tag":76,"props":8518,"children":8520},{"className":8519},[],[8521],{"type":51,"value":932},{"type":51,"value":110},{"type":51,"value":8524}," Print the ",{"type":45,"tag":76,"props":8526,"children":8528},{"className":8527},[],[8529],{"type":51,"value":8530},"pip install",{"type":51,"value":8532}," command in the next-steps message instead — most users already have a venv.",{"type":45,"tag":70,"props":8534,"children":8535},{},[8536,8541,8542,8547],{"type":45,"tag":58,"props":8537,"children":8538},{},[8539],{"type":51,"value":8540},"No silent fallbacks.",{"type":51,"value":3523},{"type":45,"tag":76,"props":8543,"children":8545},{"className":8544},[],[8546],{"type":51,"value":437},{"type":51,"value":8548}," is unsupported, error out with the valid choices.",{"type":45,"tag":70,"props":8550,"children":8551},{},[8552,8557,8559,8565,8567,8572],{"type":45,"tag":58,"props":8553,"children":8554},{},[8555],{"type":51,"value":8556},"Python only.",{"type":51,"value":8558}," If a user passes ",{"type":45,"tag":76,"props":8560,"children":8562},{"className":8561},[],[8563],{"type":51,"value":8564},"--language typescript",{"type":51,"value":8566}," (or any non-Python language flag), error out — this skill produces Python ",{"type":45,"tag":76,"props":8568,"children":8570},{"className":8569},[],[8571],{"type":51,"value":326},{"type":51,"value":8573}," SDK code only.",{"type":45,"tag":70,"props":8575,"children":8576},{},[8577,8582,8584,8589,8591,8597],{"type":45,"tag":58,"props":8578,"children":8579},{},[8580],{"type":51,"value":8581},"Research, don't invent.",{"type":51,"value":8583}," If the user asks about an Agent Observability feature, span kind, evaluator type, or SDK symbol that is not documented in this skill body, ",{"type":45,"tag":76,"props":8585,"children":8587},{"className":8586},[],[8588],{"type":51,"value":7792},{"type":51,"value":8590}," the relevant ",{"type":45,"tag":76,"props":8592,"children":8594},{"className":8593},[],[8595],{"type":51,"value":8596},"docs.datadoghq.com\u002Fllm_observability\u002F*",{"type":51,"value":8598}," page (see the Datadog Documentation table above for the canonical URLs) before answering. Cite the page URL in the response. If the docs don't cover the topic, say so explicitly — never fabricate symbols, flags, or behaviors.",{"type":45,"tag":8600,"props":8601,"children":8602},"style",{},[8603],{"type":51,"value":8604},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":8606,"total":1618},[8607,8622,8638,8649,8665,8679,8687,8701,8713,8725,8736,8756],{"slug":8608,"name":8608,"fn":8609,"description":8610,"org":8611,"tags":8612,"stars":28,"repoUrl":29,"updatedAt":8621},"agent-install","install Datadog Agent on Kubernetes","Install the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8613,8614,8617,8620],{"name":20,"slug":21,"type":15},{"name":8615,"slug":8616,"type":15},"Deployment","deployment",{"name":8618,"slug":8619,"type":15},"Kubernetes","kubernetes",{"name":13,"slug":14,"type":15},"2026-04-15T04:57:27.489805",{"slug":8623,"name":8623,"fn":8624,"description":8625,"org":8626,"tags":8627,"stars":28,"repoUrl":29,"updatedAt":8637},"agent-observability-auto-experiment","run iterative code improvements using Datadog data","Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says \"run an auto experiment\", \"hill-climb this code\", \"iteratively improve X and measure the delta\", \"optimize this prompt\u002Ffile against my traces\", \"auto-optimize against LLM-Obs\", or wants the local equivalent of the auto_experiments worker. Works from a local dataset file, an ml_app, a dataset_id, or a list of trace_ids.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8628,8629,8632,8635,8636],{"name":20,"slug":21,"type":15},{"name":8630,"slug":8631,"type":15},"Debugging","debugging",{"name":8633,"slug":8634,"type":15},"Evals","evals",{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:52:13.711906",{"slug":8639,"name":8639,"fn":8640,"description":8641,"org":8642,"tags":8643,"stars":28,"repoUrl":29,"updatedAt":8648},"agent-observability-eval-bootstrap","bootstrap evaluators from production traces","Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says \"bootstrap evaluators\", \"generate evaluators\", \"create evals from traces\", \"eval bootstrap\", \"write evaluators\", \"build eval suite\", \"publish evaluators\", or wants to generate BaseEvaluator\u002FLLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8644,8645,8646,8647],{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:19.259734",{"slug":8650,"name":8650,"fn":8651,"description":8652,"org":8653,"tags":8654,"stars":28,"repoUrl":29,"updatedAt":8664},"agent-observability-eval-pipeline","run agent observability and evaluation pipelines","End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a \"continue\" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-py-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says \"run the eval pipeline\", \"go from traces to evals\", \"bootstrap evals end to end\", \"classify then RCA then bootstrap\", \"build an eval set from scratch\", \"onboard me to datasets and experiments\", \"walk me through experiments\", \"I have an ml_app, now what\", \"Agent Observability onboarding\", \"guided experiment setup\", \"from traces to experiments\", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after \u003Cphase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at \u003Cphase>`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8655,8658,8661,8662,8663],{"name":8656,"slug":8657,"type":15},"Agents","agents",{"name":8659,"slug":8660,"type":15},"Data Pipeline","data-pipeline",{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:17.231423",{"slug":8666,"name":8666,"fn":8667,"description":8668,"org":8669,"tags":8670,"stars":28,"repoUrl":29,"updatedAt":8678},"agent-observability-experiment-analyzer","analyze LLM experiment results","Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says \"analyze experiment\", \"compare experiments\", \"analyze against baseline\", or provides one or two experiment IDs for analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8671,8674,8675,8676,8677],{"name":8672,"slug":8673,"type":15},"Analytics","analytics",{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:21.212498",{"slug":4,"name":4,"fn":5,"description":6,"org":8680,"tags":8681,"stars":28,"repoUrl":29,"updatedAt":30},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8682,8683,8684,8685,8686],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"slug":8688,"name":8688,"fn":8689,"description":8690,"org":8691,"tags":8692,"stars":28,"repoUrl":29,"updatedAt":8700},"agent-observability-replay-trace","iterate on LLM traces with local code","Use when a developer wants to iterate on ONE specific Agent Observability \u002F LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as \u002Fagent-observability-replay-trace \u003Ctrace-id> [changes to test]. Signals: \"replay this trace\"; \"iterate on a trace\"; \"this trace's output is wrong, fix it and re-run\"; \"re-run trace \u003Cid> with \u003Cchange>\"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace \u002F LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser \"Replay\" button (that's agent-observability-replay-experiment), building an experiment from a dataset\u002FCSV, writing evaluators, root-causing failed traces, or RUM\u002FHTTP session replay.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8693,8694,8695,8696,8697],{"name":20,"slug":21,"type":15},{"name":8630,"slug":8631,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":8698,"slug":8699,"type":15},"Tracing","tracing","2026-07-31T05:52:08.594182",{"slug":8702,"name":8702,"fn":8703,"description":8704,"org":8705,"tags":8706,"stars":28,"repoUrl":29,"updatedAt":8712},"agent-observability-session-classify","evaluate user intent satisfaction in sessions","Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions\u002Ftraces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8707,8708,8709,8710,8711],{"name":8672,"slug":8673,"type":15},{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:26.341497",{"slug":8714,"name":8714,"fn":8715,"description":8716,"org":8717,"tags":8718,"stars":28,"repoUrl":29,"updatedAt":8724},"agent-observability-trace-rca","diagnose LLM application failures from traces","Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says \"what's wrong with my app\", \"why is my eval failing\", \"analyze errors\", \"root cause analysis\", \"diagnose failures\", or wants to understand production failure patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8719,8720,8721,8722,8723],{"name":20,"slug":21,"type":15},{"name":8630,"slug":8631,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:24.316244",{"slug":8726,"name":8726,"fn":8727,"description":8728,"org":8729,"tags":8730,"stars":28,"repoUrl":29,"updatedAt":8735},"agent-skills","use Datadog observability skills","Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8731,8732,8734],{"name":20,"slug":21,"type":15},{"name":7598,"slug":8733,"type":15},"monitoring",{"name":13,"slug":14,"type":15},"2026-04-06T18:08:33.337476",{"slug":8737,"name":8737,"fn":8738,"description":8739,"org":8740,"tags":8741,"stars":28,"repoUrl":29,"updatedAt":8755},"datadog-app","build and manage Datadog applications","Guides developers building Datadog Apps with TypeScript, React, the @datadog\u002Fapps scaffolder, and @datadog\u002Fvite-plugin. Use when a user wants to scaffold, run, debug, upgrade, build, upload, publish, upload without publishing (draft upload), add an upload-no-publish script, set up CI\u002FCD, use OAuth or API\u002Fapplication key auth, trigger\u002Fpoll Workflow Automation, choose DDSQL or Action Catalog for backend data access, or query app datastores with DDSQL, including backend function troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8742,8743,8746,8749,8752],{"name":20,"slug":21,"type":15},{"name":8744,"slug":8745,"type":15},"Frontend","frontend",{"name":8747,"slug":8748,"type":15},"React","react",{"name":8750,"slug":8751,"type":15},"TypeScript","typescript",{"name":8753,"slug":8754,"type":15},"Vite","vite","2026-06-18T08:01:32.562331",{"slug":8757,"name":8757,"fn":8758,"description":8759,"org":8760,"tags":8761,"stars":28,"repoUrl":29,"updatedAt":8770},"dd-apm","query Datadog APM traces","APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8762,8763,8766,8767],{"name":20,"slug":21,"type":15},{"name":8764,"slug":8765,"type":15},"Distributed Tracing","distributed-tracing",{"name":13,"slug":14,"type":15},{"name":8768,"slug":8769,"type":15},"Performance","performance","2026-04-06T18:08:34.575282",{"items":8772,"total":1610},[8773,8780,8788,8795,8803,8811,8819],{"slug":8608,"name":8608,"fn":8609,"description":8610,"org":8774,"tags":8775,"stars":28,"repoUrl":29,"updatedAt":8621},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8776,8777,8778,8779],{"name":20,"slug":21,"type":15},{"name":8615,"slug":8616,"type":15},{"name":8618,"slug":8619,"type":15},{"name":13,"slug":14,"type":15},{"slug":8623,"name":8623,"fn":8624,"description":8625,"org":8781,"tags":8782,"stars":28,"repoUrl":29,"updatedAt":8637},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8783,8784,8785,8786,8787],{"name":20,"slug":21,"type":15},{"name":8630,"slug":8631,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":8639,"name":8639,"fn":8640,"description":8641,"org":8789,"tags":8790,"stars":28,"repoUrl":29,"updatedAt":8648},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8791,8792,8793,8794],{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":8650,"name":8650,"fn":8651,"description":8652,"org":8796,"tags":8797,"stars":28,"repoUrl":29,"updatedAt":8664},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8798,8799,8800,8801,8802],{"name":8656,"slug":8657,"type":15},{"name":8659,"slug":8660,"type":15},{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":13,"slug":14,"type":15},{"slug":8666,"name":8666,"fn":8667,"description":8668,"org":8804,"tags":8805,"stars":28,"repoUrl":29,"updatedAt":8678},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8806,8807,8808,8809,8810],{"name":8672,"slug":8673,"type":15},{"name":20,"slug":21,"type":15},{"name":8633,"slug":8634,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":8812,"tags":8813,"stars":28,"repoUrl":29,"updatedAt":30},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8814,8815,8816,8817,8818],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"slug":8688,"name":8688,"fn":8689,"description":8690,"org":8820,"tags":8821,"stars":28,"repoUrl":29,"updatedAt":8700},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[8822,8823,8824,8825,8826],{"name":20,"slug":21,"type":15},{"name":8630,"slug":8631,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":8698,"slug":8699,"type":15}]