[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-weights-and-biases-offline-eval-improvement":3,"mdc-torwrd-key":35,"related-repo-weights-and-biases-offline-eval-improvement":2665,"related-org-weights-and-biases-offline-eval-improvement":2700},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":33,"mdContent":34},"offline-eval-improvement","improve prompts using offline evaluation baselines","Guides coding agents through Discovery Forge prompt improvement from a specified Weave offline evaluation baseline. Use `wandb-primary` to fetch evaluation results, failed eval rows, dataset refs, prompt refs, and scorer evidence, then use this workflow when improving researcher.md from failed evaluation rows, comparing eval runs, or iterating on a fixed dataset.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"weights-and-biases","Weights & Biases","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fweights-and-biases.png","wandb",[13,17,19,22],{"name":14,"slug":15,"type":16},"Datasets","datasets","tag",{"name":9,"slug":18,"type":16},"weights-biases",{"name":20,"slug":21,"type":16},"Evals","evals",{"name":23,"slug":24,"type":16},"Agents","agents",2,"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fdiscovery-forge","2026-07-17T06:07:26.501045",null,0,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":28},[],"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fdiscovery-forge\u002Ftree\u002FHEAD\u002Fskills\u002Foffline-eval-improvement","---\nname: offline-eval-improvement\ndescription: Guides coding agents through Discovery Forge prompt improvement from a specified Weave offline evaluation baseline. Use `wandb-primary` to fetch evaluation results, failed eval rows, dataset refs, prompt refs, and scorer evidence, then use this workflow when improving researcher.md from failed evaluation rows, comparing eval runs, or iterating on a fixed dataset.\n---\n\n# Offline Eval Improvement\n\nUse this skill when improving `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md` from Weave Evaluation evidence fetched with the `wandb-primary` skill rather than daily annotation queues.\n\nThe coding agent starts from explicit experiment inputs, inspects the baseline evaluation run, analyzes failed rows, writes a plan, backs up the starting prompt, edits the local prompt, validates the change, reruns the same dataset, and reports the before\u002Fafter result for each iteration.\n\n## Before You Start\n\nFollow `AGENTS.md` setup first, then select and use the installed `wandb-primary` skill for Weave evaluation, dataset, trace, and scorer evidence. This skill only defines the Discovery Forge offline-eval improvement workflow.\n\nFetch all evaluation evidence live from Weave through `wandb-primary` skill guidance, including its Weave SDK \u002F W&B API patterns. Do not use W&B MCP tools, and do not add discovery-forge query wrappers.\n\n## Required Inputs\n\nAsk for or infer these inputs before editing. First check `src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml` and use its `offline_eval_improvement` values when present. If any input is still missing and cannot be inferred from the user's request or latest `evaluate.py` output, stop and ask for it.\n\n- Dataset: a configured dataset key such as `verdict_quality`, or a full published Weave Dataset ref.\n- Start prompt: the local prompt file to improve, or a Weave `StringPrompt` ref to inspect as the starting baseline. Default local file is `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md`.\n- Baseline evaluation ID: the Weave Evaluation parent call ID or URL that represents the score to beat.\n- Max iterations: maximum number of prompt edit + eval cycles to run before reporting back.\n\nKeep these inputs visible in every plan and report so workshop participants can see which dataset, prompt, baseline, and iteration budget they are using.\n\n## Default Project\n\n- Entity: read from `.env` as `WANDB_ENTITY` (required; use your own W&B entity)\n- Project: read from `.env` as `WANDB_PROJECT` (required; `.env.example` uses `discovery-forge`)\n- API key: read from `.env` as `WANDB_API_KEY` (required)\n- Prompt file: `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md`\n- Prompt backups: `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher_backup_\u003CN>.md`\n- Evaluation entrypoint: `uv run python evaluate.py`\n- Evaluation config: `src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml`\n- Improvement history: `src\u002Fdiscovery_forge\u002Fagents\u002Fimprove_history\u002F\u003CYYYY-MM-DD-HHMM-offline-eval>\u002Fplan.md` and `src\u002Fdiscovery_forge\u002Fagents\u002Fimprove_history\u002F\u003CYYYY-MM-DD-HHMM-offline-eval>\u002Fapplied.md`\n- Primary metric: `verdict_quality_scorer.is_correct.true_fraction`\n- Versioning behavior: when `evaluate.py` uses the local prompt, it publishes the current `researcher.md` as a Weave `StringPrompt` and publishes `ResearcherAgentModel` for the run. Use `--researcher-prompt-ref` only when intentionally comparing a fixed published prompt version.\n\n## Prompt Edit Scope\n\nEach improvement iteration should make a focused, policy-level prompt edit that covers one or two related failure clusters. Pair every new reject rule with an explicit accept-protection clause (for example \"missing metadata is a profile limitation, not a scope failure\") so that tightening one cluster does not push correctly-accepted rows into false rejects.\n\n### Rules\n\n- Cover one or two related failure clusters in a focused edit; keep every change **policy-level** (artifact types, loop evidence, output discipline), not candidate-specific cheats.\n- Pair reject rules with accept-protection so tightening one cluster does not create false rejects elsewhere.\n- After each iteration, rerun `evaluate.py`. Because a single run varies by ~±2-3 rows, rerun 2-3 times (or compare per-row stability) before trusting a change. If the metric regresses across runs, revert to the backup and retry.\n- Do not hardcode dataset row IDs or candidate names as a lookup table in the prompt.\n- Record in `plan.md` what you are **not** changing this iteration so the next pass has a clear queue.\n\n### Prompt-edit guardrails\n\nThese keep edits general (no cherry-picking specific candidates) while avoiding the common regression where tightening rejects silently destroys correct accepts:\n\n- Express policy as **distinctions** the candidate either meets or not (artifact type, observable behavior, what it revises). Avoid rules keyed on **evidence completeness** — clauses that say \"reject if the loop is missing, implied, weak, unverified, or not fully proven\" fire on genuine positives too, because real in-scope systems often have incomplete primary sources. They cause broad false rejects.\n- A reject rule should describe **what kind of thing** is out of scope, not **how much proof** you happened to find this session. Missing\u002Flimited evidence is a profile limitation to record, not a reject trigger.\n- Do not add a clause that contradicts an existing accept-protection clause. If a new sentence and an existing one disagree (e.g. \"accept loop runners even with thin metadata\" vs \"reject if evidence is implied\"), the model follows the stricter one and the protection is lost. Resolve the conflict instead of stacking both.\n- Every iteration, check **both** error directions on the rerun: count new false rejects (accepted→rejected) as well as fixed false accepts. A change that fixes one cluster but flips several correct accepts is a net regression even if it \"sounds stricter\".\n- Before keeping an edit, ask: would this sentence change the verdict on candidates I did not intend to touch? If yes, it is too broad — narrow it to the targeted artifact type or behavior.\n\n### Common Failure Clusters\n\n1. Wrong artifact type (survey, cookbook, guide, list-like sources)\n2. Adjacent infrastructure (memory-only, testing gate, repo-automation host, generic framework)\n3. Weak-evidence or over-strict edge cases (protect correctly-accepted loop runners)\n\n## Evidence Workflow\n\n1. Record the required inputs: dataset, start prompt, baseline evaluation ID, and max iterations.\n2. Resolve the dataset:\n   - If the user gave a dataset key, read `src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml` and use that key's `ref`.\n   - If the config defines `offline_eval_improvement.dataset_key`, use that key unless the user supplied a different one.\n   - If the user gave a full Weave Dataset ref, use it directly for this loop.\n3. Resolve the start prompt:\n   - If the config defines `offline_eval_improvement.start_prompt`, use that path unless the user supplied a different prompt.\n   - If the user gave a local path, read that file and treat it as the editable prompt.\n   - If the user gave a Weave `StringPrompt` ref, inspect it through `wandb-primary`, then decide with the user whether to copy its content into the local editable prompt.\n   - If unspecified, use `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md`.\n4. Resolve max iterations from the user's request or `offline_eval_improvement.max_iterations`.\n5. Inspect the baseline Weave Evaluation parent call from the given call ID or URL. If `offline_eval_improvement.baseline_evaluation_id` is set, use that as the default baseline.\n6. If no baseline evaluation run exists yet, run one with the selected dataset and start prompt, then use that new parent call as the baseline:\n\n   ```bash\n   uv run python evaluate.py --verdict-dataset-key '\u003Cdataset-key>'\n   # or: uv run python evaluate.py --verdict-dataset-ref '\u003Cdataset-ref>'\n   ```\n\n7. Use the `wandb-primary` skill to inspect the parent evaluation call.\n8. Use the `wandb-primary` skill to inspect the child row calls.\n9. Pull only the fields needed for diagnosis: row inputs, expected labels, model output, scorer outputs, call status, and prompt\u002Fmodel ref metadata when present. For the current verdict eval, include `expected_scope_status`, output `scope_status`, `verdict`, `verdict_reason`, `final_output`, `profile_review_markdown`, `run.researcher_prompt_ref`, `run.researcher_prompt_hash`, `run.researcher_model_ref`, and `verdict_quality_scorer.is_correct`.\n10. Filter to rows where the primary scorer failed. Keep a small sample of passing rows only when needed to avoid changing behavior that already works.\n11. Group failures into prompt-policy patterns. For verdict eval:\n   - expected `rejected`, observed `accepted`: scope filter is too permissive.\n   - expected `accepted`, observed `rejected`: scope filter is too strict or treats metadata weakness as scope failure.\n   - observed `unknown`: agent failed to call either save tool; improve the decision\u002Foutput discipline.\n12. Read the local prompt that will be edited.\n13. Write `src\u002Fdiscovery_forge\u002Fagents\u002Fimprove_history\u002F\u003CYYYY-MM-DD-HHMM-offline-eval>\u002Fplan.md` before editing. Use the local execution start time for the directory name, for example `2026-06-12-1505-offline-eval`. If the file already exists for that improvement loop, update it instead of creating a new call-ID directory.\n14. Before editing, preserve the current prompt as `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher_backup_\u003CN>.md`, where `\u003CN>` is the next available integer starting from `0`.\n\n### Evaluation Evidence Selection\n\nUse the `wandb-primary` skill to fetch and inspect Weave Evaluation evidence. This skill only defines which Discovery Forge evidence matters:\n\n- Parent evaluation call: the exact eval call ID or link under analysis.\n- Child row calls: the dataset-row-level evaluation results under that parent.\n- Dataset row inputs: candidate name, candidate URL, description, and `expected_scope_status`.\n- Model output: reviewer payload (`profile_review_markdown`, `verdict`, `tool_name`, `primary_url`, `summary`, `search`, `run`) plus eval-compatible fields (`scope_status`, `verdict_reason`, `final_output`, and `profile` when available). For accepted\u002Frejected rows, `scope_status` should mirror `verdict`; scorers read `scope_status` first for compatibility.\n- Scorer output: `verdict_quality_scorer.is_correct` and any supporting scorer detail.\n- Prompt\u002Fmodel metadata: prompt ref\u002Fhash and `ResearcherAgentModel` ref when present.\n\nInspect one child row first to learn the shape, then extract the smallest stable set of fields needed to identify failed verdict rows.\n\n## Plan Format\n\nUse this structure:\n\n```markdown\n# Offline Eval Prompt Improvement Plan for \u003CYYYY-MM-DD-HHMM-offline-eval>\n\n## Source\n- Dataset: \u003Cdataset key or full ref>\n- Dataset ref: \u003Cresolved published Weave Dataset ref>\n- Start prompt: \u003Clocal path or Weave StringPrompt ref>\n- Baseline prompt ref\u002Fhash: \u003Cpublished prompt ref and hash from the run>\n- Baseline ResearcherAgentModel ref: \u003Cpublished model ref from the run>\n- Baseline Weave Evaluation parent call: \u003Ceval-call-id or link>\n- Baseline metric: `verdict_quality_scorer.is_correct` = \u003Cvalue>\n- Max iterations: \u003Cn>\n- Current iteration: \u003Ci>\u002F\u003Cn>\n- Failed rows inspected: \u003Cn>\n- Current `researcher.md` inspected\n\n## Prompt Edit Scope\n- Target clusters this iteration: \u003Cone or two related failure patterns>\n- Accept-protection: \u003Cpassing behavior that must stay stable>\n- Deferred patterns: \u003Cremaining failure patterns, if any>\n\n## Failure Summary\n- \u003Cconcrete failure pattern with candidate examples>\n\n## Proposed Prompt Change\n- \u003Cexact behavior change>\n\n## Expected Effect\n- \u003Cwhich failure patterns should improve and which passing behavior must stay stable>\n\n## Applied Files\n- `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md`\n\n## Out of Scope\n- Dataset rows, labels, scorer logic, evaluation runner, registry, schemas, and search backend behavior are not changed during prompt iteration.\n```\n\n## Apply Rules\n\n- Prompt-only changes may edit only `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md`.\n- Evaluation reruns publish the local `researcher.md` prompt and `ResearcherAgentModel` by design so every run has version refs in Weave. Do not change `evaluate.py`, datasets, labels, or scorers to force a better score.\n- Before each prompt edit, create the next backup file: `researcher_backup_0.md`, then `researcher_backup_1.md`, and so on. Never overwrite an existing backup.\n- Do not change evaluation datasets, labels, scorers, or `evaluate.py` to make results look better.\n- Do not create fallback behavior.\n- Keep scope verdict separate from metadata completeness.\n- If primary sources prove the task -> evaluator -> feedback\u002Fmemory -> improvement loop but metadata is incomplete, record the weakness in `key_limitations` instead of rejecting solely for metadata incompleteness.\n- If `expected_scope_status == accepted` and the agent rejects due to missing metadata, treat the prompt as over-strict unless the dataset row appears wrong.\n- If `expected_scope_status == rejected` and the agent accepts, treat it as a scope-policy issue unless the dataset row appears wrong.\n- If a dataset row, label, or scorer appears wrong, report it as dataset\u002Fscorer maintenance rather than editing those artifacts in this workflow.\n- Use failed candidate names as policy examples in the plan. Do not hardcode them as required outputs.\n\nAfter editing, write `src\u002Fdiscovery_forge\u002Fagents\u002Fimprove_history\u002F\u003CYYYY-MM-DD-HHMM-offline-eval>\u002Fapplied.md`. If the file already exists for that improvement loop, update it with the latest applied summary.\n\n## Local Prompt Iteration\n\nDuring development, evaluate the local prompt file directly. Do not pass `--researcher-prompt-ref`; `evaluate.py` will publish the current local prompt and `ResearcherAgentModel` for each run so the result is versioned. Pass `--researcher-prompt-ref` only when the goal is to compare a fixed published prompt ref.\n\nRun no more than the requested max iterations. Stop early if the primary metric improves enough for the user's goal, if the remaining failures are dataset\u002Fscorer maintenance issues, or if two consecutive focused edits regress the baseline.\n\nFor each iteration:\n\n1. Update `plan.md` with the current iteration number, selected dataset ref, start prompt, baseline eval ID, target cluster, and deferred patterns.\n2. Create the next backup file.\n3. Make a focused edit covering one or two related clusters with paired accept-protection.\n4. Run focused validation.\n5. Run the same selected dataset:\n\n   ```bash\n   uv run python evaluate.py --verdict-dataset-key '\u003Cdataset-key>'\n   # or: uv run python evaluate.py --verdict-dataset-ref '\u003Cdataset-ref>'\n   ```\n\n6. Compare the new run against the baseline evaluation ID and the previous iteration.\n7. Keep the change only if the selected dataset metric improves across runs without unacceptable row-level regressions. Because a single run varies by ~±2-3 rows, rerun 2-3 times or compare per-row stability before trusting a borderline change.\n8. If correctly-accepted rows flipped to rejected, strengthen the accept-protection clause and rerun within the max-iteration budget.\n9. If the metric regresses across runs, revert to the backup and either try the next focused cluster or stop and report the regression.\n\n## Validation\n\nRun focused validation:\n\n```bash\nuv run pytest tests\u002Funit\u002Ftest_researcher.py -q\n```\n\nThen rerun the same selected dataset:\n\n```bash\nuv run python evaluate.py --verdict-dataset-key '\u003Cdataset-key>'\n# or: uv run python evaluate.py --verdict-dataset-ref '\u003Cdataset-ref>'\n```\n\n## Report Back\n\nReport:\n\n- dataset key\u002Fref used\n- start prompt path\u002Fref\n- baseline evaluation ID\n- max iterations requested and iterations completed\n- targeted failure cluster and deferred patterns\n- baseline `verdict_quality_scorer.is_correct`\n- number of failed rows inspected\n- failure patterns found\n- prompt changes made\n- prompt ref\u002Fhash and `ResearcherAgentModel` ref used by each rerun\n- both error directions after rerun: false accepts fixed AND any new false rejects introduced\n- backup file created\n- tests run\n- rerun evaluation ID(s) and `verdict_quality_scorer.is_correct`\n- remaining failed patterns or dataset\u002Fscorer maintenance notes\n",{"data":36,"body":37},{"name":4,"description":6},{"type":38,"children":39},"root",[40,48,71,76,83,103,115,121,150,198,203,209,402,408,413,420,478,484,489,552,558,577,583,901,955,1011,1017,1028,1176,1181,1187,1192,2155,2161,2292,2304,2310,2343,2348,2353,2456,2462,2467,2500,2505,2553,2559,2564,2659],{"type":41,"tag":42,"props":43,"children":44},"element","h1",{"id":4},[45],{"type":46,"value":47},"text","Offline Eval Improvement",{"type":41,"tag":49,"props":50,"children":51},"p",{},[52,54,61,63,69],{"type":46,"value":53},"Use this skill when improving ",{"type":41,"tag":55,"props":56,"children":58},"code",{"className":57},[],[59],{"type":46,"value":60},"src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md",{"type":46,"value":62}," from Weave Evaluation evidence fetched with the ",{"type":41,"tag":55,"props":64,"children":66},{"className":65},[],[67],{"type":46,"value":68},"wandb-primary",{"type":46,"value":70}," skill rather than daily annotation queues.",{"type":41,"tag":49,"props":72,"children":73},{},[74],{"type":46,"value":75},"The coding agent starts from explicit experiment inputs, inspects the baseline evaluation run, analyzes failed rows, writes a plan, backs up the starting prompt, edits the local prompt, validates the change, reruns the same dataset, and reports the before\u002Fafter result for each iteration.",{"type":41,"tag":77,"props":78,"children":80},"h2",{"id":79},"before-you-start",[81],{"type":46,"value":82},"Before You Start",{"type":41,"tag":49,"props":84,"children":85},{},[86,88,94,96,101],{"type":46,"value":87},"Follow ",{"type":41,"tag":55,"props":89,"children":91},{"className":90},[],[92],{"type":46,"value":93},"AGENTS.md",{"type":46,"value":95}," setup first, then select and use the installed ",{"type":41,"tag":55,"props":97,"children":99},{"className":98},[],[100],{"type":46,"value":68},{"type":46,"value":102}," skill for Weave evaluation, dataset, trace, and scorer evidence. This skill only defines the Discovery Forge offline-eval improvement workflow.",{"type":41,"tag":49,"props":104,"children":105},{},[106,108,113],{"type":46,"value":107},"Fetch all evaluation evidence live from Weave through ",{"type":41,"tag":55,"props":109,"children":111},{"className":110},[],[112],{"type":46,"value":68},{"type":46,"value":114}," skill guidance, including its Weave SDK \u002F W&B API patterns. Do not use W&B MCP tools, and do not add discovery-forge query wrappers.",{"type":41,"tag":77,"props":116,"children":118},{"id":117},"required-inputs",[119],{"type":46,"value":120},"Required Inputs",{"type":41,"tag":49,"props":122,"children":123},{},[124,126,132,134,140,142,148],{"type":46,"value":125},"Ask for or infer these inputs before editing. First check ",{"type":41,"tag":55,"props":127,"children":129},{"className":128},[],[130],{"type":46,"value":131},"src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml",{"type":46,"value":133}," and use its ",{"type":41,"tag":55,"props":135,"children":137},{"className":136},[],[138],{"type":46,"value":139},"offline_eval_improvement",{"type":46,"value":141}," values when present. If any input is still missing and cannot be inferred from the user's request or latest ",{"type":41,"tag":55,"props":143,"children":145},{"className":144},[],[146],{"type":46,"value":147},"evaluate.py",{"type":46,"value":149}," output, stop and ask for it.",{"type":41,"tag":151,"props":152,"children":153},"ul",{},[154,168,188,193],{"type":41,"tag":155,"props":156,"children":157},"li",{},[158,160,166],{"type":46,"value":159},"Dataset: a configured dataset key such as ",{"type":41,"tag":55,"props":161,"children":163},{"className":162},[],[164],{"type":46,"value":165},"verdict_quality",{"type":46,"value":167},", or a full published Weave Dataset ref.",{"type":41,"tag":155,"props":169,"children":170},{},[171,173,179,181,186],{"type":46,"value":172},"Start prompt: the local prompt file to improve, or a Weave ",{"type":41,"tag":55,"props":174,"children":176},{"className":175},[],[177],{"type":46,"value":178},"StringPrompt",{"type":46,"value":180}," ref to inspect as the starting baseline. Default local file is ",{"type":41,"tag":55,"props":182,"children":184},{"className":183},[],[185],{"type":46,"value":60},{"type":46,"value":187},".",{"type":41,"tag":155,"props":189,"children":190},{},[191],{"type":46,"value":192},"Baseline evaluation ID: the Weave Evaluation parent call ID or URL that represents the score to beat.",{"type":41,"tag":155,"props":194,"children":195},{},[196],{"type":46,"value":197},"Max iterations: maximum number of prompt edit + eval cycles to run before reporting back.",{"type":41,"tag":49,"props":199,"children":200},{},[201],{"type":46,"value":202},"Keep these inputs visible in every plan and report so workshop participants can see which dataset, prompt, baseline, and iteration budget they are using.",{"type":41,"tag":77,"props":204,"children":206},{"id":205},"default-project",[207],{"type":46,"value":208},"Default Project",{"type":41,"tag":151,"props":210,"children":211},{},[212,233,268,287,297,308,319,329,348,359],{"type":41,"tag":155,"props":213,"children":214},{},[215,217,223,225,231],{"type":46,"value":216},"Entity: read from ",{"type":41,"tag":55,"props":218,"children":220},{"className":219},[],[221],{"type":46,"value":222},".env",{"type":46,"value":224}," as ",{"type":41,"tag":55,"props":226,"children":228},{"className":227},[],[229],{"type":46,"value":230},"WANDB_ENTITY",{"type":46,"value":232}," (required; use your own W&B entity)",{"type":41,"tag":155,"props":234,"children":235},{},[236,238,243,244,250,252,258,260,266],{"type":46,"value":237},"Project: read from ",{"type":41,"tag":55,"props":239,"children":241},{"className":240},[],[242],{"type":46,"value":222},{"type":46,"value":224},{"type":41,"tag":55,"props":245,"children":247},{"className":246},[],[248],{"type":46,"value":249},"WANDB_PROJECT",{"type":46,"value":251}," (required; ",{"type":41,"tag":55,"props":253,"children":255},{"className":254},[],[256],{"type":46,"value":257},".env.example",{"type":46,"value":259}," uses ",{"type":41,"tag":55,"props":261,"children":263},{"className":262},[],[264],{"type":46,"value":265},"discovery-forge",{"type":46,"value":267},")",{"type":41,"tag":155,"props":269,"children":270},{},[271,273,278,279,285],{"type":46,"value":272},"API key: read from ",{"type":41,"tag":55,"props":274,"children":276},{"className":275},[],[277],{"type":46,"value":222},{"type":46,"value":224},{"type":41,"tag":55,"props":280,"children":282},{"className":281},[],[283],{"type":46,"value":284},"WANDB_API_KEY",{"type":46,"value":286}," (required)",{"type":41,"tag":155,"props":288,"children":289},{},[290,292],{"type":46,"value":291},"Prompt file: ",{"type":41,"tag":55,"props":293,"children":295},{"className":294},[],[296],{"type":46,"value":60},{"type":41,"tag":155,"props":298,"children":299},{},[300,302],{"type":46,"value":301},"Prompt backups: ",{"type":41,"tag":55,"props":303,"children":305},{"className":304},[],[306],{"type":46,"value":307},"src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher_backup_\u003CN>.md",{"type":41,"tag":155,"props":309,"children":310},{},[311,313],{"type":46,"value":312},"Evaluation entrypoint: ",{"type":41,"tag":55,"props":314,"children":316},{"className":315},[],[317],{"type":46,"value":318},"uv run python evaluate.py",{"type":41,"tag":155,"props":320,"children":321},{},[322,324],{"type":46,"value":323},"Evaluation config: ",{"type":41,"tag":55,"props":325,"children":327},{"className":326},[],[328],{"type":46,"value":131},{"type":41,"tag":155,"props":330,"children":331},{},[332,334,340,342],{"type":46,"value":333},"Improvement history: ",{"type":41,"tag":55,"props":335,"children":337},{"className":336},[],[338],{"type":46,"value":339},"src\u002Fdiscovery_forge\u002Fagents\u002Fimprove_history\u002F\u003CYYYY-MM-DD-HHMM-offline-eval>\u002Fplan.md",{"type":46,"value":341}," and ",{"type":41,"tag":55,"props":343,"children":345},{"className":344},[],[346],{"type":46,"value":347},"src\u002Fdiscovery_forge\u002Fagents\u002Fimprove_history\u002F\u003CYYYY-MM-DD-HHMM-offline-eval>\u002Fapplied.md",{"type":41,"tag":155,"props":349,"children":350},{},[351,353],{"type":46,"value":352},"Primary metric: ",{"type":41,"tag":55,"props":354,"children":356},{"className":355},[],[357],{"type":46,"value":358},"verdict_quality_scorer.is_correct.true_fraction",{"type":41,"tag":155,"props":360,"children":361},{},[362,364,369,371,377,379,384,386,392,394,400],{"type":46,"value":363},"Versioning behavior: when ",{"type":41,"tag":55,"props":365,"children":367},{"className":366},[],[368],{"type":46,"value":147},{"type":46,"value":370}," uses the local prompt, it publishes the current ",{"type":41,"tag":55,"props":372,"children":374},{"className":373},[],[375],{"type":46,"value":376},"researcher.md",{"type":46,"value":378}," as a Weave ",{"type":41,"tag":55,"props":380,"children":382},{"className":381},[],[383],{"type":46,"value":178},{"type":46,"value":385}," and publishes ",{"type":41,"tag":55,"props":387,"children":389},{"className":388},[],[390],{"type":46,"value":391},"ResearcherAgentModel",{"type":46,"value":393}," for the run. Use ",{"type":41,"tag":55,"props":395,"children":397},{"className":396},[],[398],{"type":46,"value":399},"--researcher-prompt-ref",{"type":46,"value":401}," only when intentionally comparing a fixed published prompt version.",{"type":41,"tag":77,"props":403,"children":405},{"id":404},"prompt-edit-scope",[406],{"type":46,"value":407},"Prompt Edit Scope",{"type":41,"tag":49,"props":409,"children":410},{},[411],{"type":46,"value":412},"Each improvement iteration should make a focused, policy-level prompt edit that covers one or two related failure clusters. Pair every new reject rule with an explicit accept-protection clause (for example \"missing metadata is a profile limitation, not a scope failure\") so that tightening one cluster does not push correctly-accepted rows into false rejects.",{"type":41,"tag":414,"props":415,"children":417},"h3",{"id":416},"rules",[418],{"type":46,"value":419},"Rules",{"type":41,"tag":151,"props":421,"children":422},{},[423,436,441,453,458],{"type":41,"tag":155,"props":424,"children":425},{},[426,428,434],{"type":46,"value":427},"Cover one or two related failure clusters in a focused edit; keep every change ",{"type":41,"tag":429,"props":430,"children":431},"strong",{},[432],{"type":46,"value":433},"policy-level",{"type":46,"value":435}," (artifact types, loop evidence, output discipline), not candidate-specific cheats.",{"type":41,"tag":155,"props":437,"children":438},{},[439],{"type":46,"value":440},"Pair reject rules with accept-protection so tightening one cluster does not create false rejects elsewhere.",{"type":41,"tag":155,"props":442,"children":443},{},[444,446,451],{"type":46,"value":445},"After each iteration, rerun ",{"type":41,"tag":55,"props":447,"children":449},{"className":448},[],[450],{"type":46,"value":147},{"type":46,"value":452},". Because a single run varies by ~±2-3 rows, rerun 2-3 times (or compare per-row stability) before trusting a change. If the metric regresses across runs, revert to the backup and retry.",{"type":41,"tag":155,"props":454,"children":455},{},[456],{"type":46,"value":457},"Do not hardcode dataset row IDs or candidate names as a lookup table in the prompt.",{"type":41,"tag":155,"props":459,"children":460},{},[461,463,469,471,476],{"type":46,"value":462},"Record in ",{"type":41,"tag":55,"props":464,"children":466},{"className":465},[],[467],{"type":46,"value":468},"plan.md",{"type":46,"value":470}," what you are ",{"type":41,"tag":429,"props":472,"children":473},{},[474],{"type":46,"value":475},"not",{"type":46,"value":477}," changing this iteration so the next pass has a clear queue.",{"type":41,"tag":414,"props":479,"children":481},{"id":480},"prompt-edit-guardrails",[482],{"type":46,"value":483},"Prompt-edit guardrails",{"type":41,"tag":49,"props":485,"children":486},{},[487],{"type":46,"value":488},"These keep edits general (no cherry-picking specific candidates) while avoiding the common regression where tightening rejects silently destroys correct accepts:",{"type":41,"tag":151,"props":490,"children":491},{},[492,511,530,535,547],{"type":41,"tag":155,"props":493,"children":494},{},[495,497,502,504,509],{"type":46,"value":496},"Express policy as ",{"type":41,"tag":429,"props":498,"children":499},{},[500],{"type":46,"value":501},"distinctions",{"type":46,"value":503}," the candidate either meets or not (artifact type, observable behavior, what it revises). Avoid rules keyed on ",{"type":41,"tag":429,"props":505,"children":506},{},[507],{"type":46,"value":508},"evidence completeness",{"type":46,"value":510}," — clauses that say \"reject if the loop is missing, implied, weak, unverified, or not fully proven\" fire on genuine positives too, because real in-scope systems often have incomplete primary sources. They cause broad false rejects.",{"type":41,"tag":155,"props":512,"children":513},{},[514,516,521,523,528],{"type":46,"value":515},"A reject rule should describe ",{"type":41,"tag":429,"props":517,"children":518},{},[519],{"type":46,"value":520},"what kind of thing",{"type":46,"value":522}," is out of scope, not ",{"type":41,"tag":429,"props":524,"children":525},{},[526],{"type":46,"value":527},"how much proof",{"type":46,"value":529}," you happened to find this session. Missing\u002Flimited evidence is a profile limitation to record, not a reject trigger.",{"type":41,"tag":155,"props":531,"children":532},{},[533],{"type":46,"value":534},"Do not add a clause that contradicts an existing accept-protection clause. If a new sentence and an existing one disagree (e.g. \"accept loop runners even with thin metadata\" vs \"reject if evidence is implied\"), the model follows the stricter one and the protection is lost. Resolve the conflict instead of stacking both.",{"type":41,"tag":155,"props":536,"children":537},{},[538,540,545],{"type":46,"value":539},"Every iteration, check ",{"type":41,"tag":429,"props":541,"children":542},{},[543],{"type":46,"value":544},"both",{"type":46,"value":546}," error directions on the rerun: count new false rejects (accepted→rejected) as well as fixed false accepts. A change that fixes one cluster but flips several correct accepts is a net regression even if it \"sounds stricter\".",{"type":41,"tag":155,"props":548,"children":549},{},[550],{"type":46,"value":551},"Before keeping an edit, ask: would this sentence change the verdict on candidates I did not intend to touch? If yes, it is too broad — narrow it to the targeted artifact type or behavior.",{"type":41,"tag":414,"props":553,"children":555},{"id":554},"common-failure-clusters",[556],{"type":46,"value":557},"Common Failure Clusters",{"type":41,"tag":559,"props":560,"children":561},"ol",{},[562,567,572],{"type":41,"tag":155,"props":563,"children":564},{},[565],{"type":46,"value":566},"Wrong artifact type (survey, cookbook, guide, list-like sources)",{"type":41,"tag":155,"props":568,"children":569},{},[570],{"type":46,"value":571},"Adjacent infrastructure (memory-only, testing gate, repo-automation host, generic framework)",{"type":41,"tag":155,"props":573,"children":574},{},[575],{"type":46,"value":576},"Weak-evidence or over-strict edge cases (protect correctly-accepted loop runners)",{"type":41,"tag":77,"props":578,"children":580},{"id":579},"evidence-workflow",[581],{"type":46,"value":582},"Evidence Workflow",{"type":41,"tag":559,"props":584,"children":585},{},[586,591,636,691,703,716,790,802,813,891,896],{"type":41,"tag":155,"props":587,"children":588},{},[589],{"type":46,"value":590},"Record the required inputs: dataset, start prompt, baseline evaluation ID, and max iterations.",{"type":41,"tag":155,"props":592,"children":593},{},[594,596],{"type":46,"value":595},"Resolve the dataset:",{"type":41,"tag":151,"props":597,"children":598},{},[599,618,631],{"type":41,"tag":155,"props":600,"children":601},{},[602,604,609,611,617],{"type":46,"value":603},"If the user gave a dataset key, read ",{"type":41,"tag":55,"props":605,"children":607},{"className":606},[],[608],{"type":46,"value":131},{"type":46,"value":610}," and use that key's ",{"type":41,"tag":55,"props":612,"children":614},{"className":613},[],[615],{"type":46,"value":616},"ref",{"type":46,"value":187},{"type":41,"tag":155,"props":619,"children":620},{},[621,623,629],{"type":46,"value":622},"If the config defines ",{"type":41,"tag":55,"props":624,"children":626},{"className":625},[],[627],{"type":46,"value":628},"offline_eval_improvement.dataset_key",{"type":46,"value":630},", use that key unless the user supplied a different one.",{"type":41,"tag":155,"props":632,"children":633},{},[634],{"type":46,"value":635},"If the user gave a full Weave Dataset ref, use it directly for this loop.",{"type":41,"tag":155,"props":637,"children":638},{},[639,641],{"type":46,"value":640},"Resolve the start prompt:",{"type":41,"tag":151,"props":642,"children":643},{},[644,656,661,680],{"type":41,"tag":155,"props":645,"children":646},{},[647,648,654],{"type":46,"value":622},{"type":41,"tag":55,"props":649,"children":651},{"className":650},[],[652],{"type":46,"value":653},"offline_eval_improvement.start_prompt",{"type":46,"value":655},", use that path unless the user supplied a different prompt.",{"type":41,"tag":155,"props":657,"children":658},{},[659],{"type":46,"value":660},"If the user gave a local path, read that file and treat it as the editable prompt.",{"type":41,"tag":155,"props":662,"children":663},{},[664,666,671,673,678],{"type":46,"value":665},"If the user gave a Weave ",{"type":41,"tag":55,"props":667,"children":669},{"className":668},[],[670],{"type":46,"value":178},{"type":46,"value":672}," ref, inspect it through ",{"type":41,"tag":55,"props":674,"children":676},{"className":675},[],[677],{"type":46,"value":68},{"type":46,"value":679},", then decide with the user whether to copy its content into the local editable prompt.",{"type":41,"tag":155,"props":681,"children":682},{},[683,685,690],{"type":46,"value":684},"If unspecified, use ",{"type":41,"tag":55,"props":686,"children":688},{"className":687},[],[689],{"type":46,"value":60},{"type":46,"value":187},{"type":41,"tag":155,"props":692,"children":693},{},[694,696,702],{"type":46,"value":695},"Resolve max iterations from the user's request or ",{"type":41,"tag":55,"props":697,"children":699},{"className":698},[],[700],{"type":46,"value":701},"offline_eval_improvement.max_iterations",{"type":46,"value":187},{"type":41,"tag":155,"props":704,"children":705},{},[706,708,714],{"type":46,"value":707},"Inspect the baseline Weave Evaluation parent call from the given call ID or URL. If ",{"type":41,"tag":55,"props":709,"children":711},{"className":710},[],[712],{"type":46,"value":713},"offline_eval_improvement.baseline_evaluation_id",{"type":46,"value":715}," is set, use that as the default baseline.",{"type":41,"tag":155,"props":717,"children":718},{},[719,721],{"type":46,"value":720},"If no baseline evaluation run exists yet, run one with the selected dataset and start prompt, then use that new parent call as the baseline:",{"type":41,"tag":722,"props":723,"children":728},"pre",{"className":724,"code":725,"language":726,"meta":727,"style":727},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv run python evaluate.py --verdict-dataset-key '\u003Cdataset-key>'\n# or: uv run python evaluate.py --verdict-dataset-ref '\u003Cdataset-ref>'\n","bash","",[729],{"type":41,"tag":55,"props":730,"children":731},{"__ignoreMap":727},[732,781],{"type":41,"tag":733,"props":734,"children":737},"span",{"class":735,"line":736},"line",1,[738,744,750,755,760,765,771,776],{"type":41,"tag":733,"props":739,"children":741},{"style":740},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[742],{"type":46,"value":743},"uv",{"type":41,"tag":733,"props":745,"children":747},{"style":746},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[748],{"type":46,"value":749}," run",{"type":41,"tag":733,"props":751,"children":752},{"style":746},[753],{"type":46,"value":754}," python",{"type":41,"tag":733,"props":756,"children":757},{"style":746},[758],{"type":46,"value":759}," evaluate.py",{"type":41,"tag":733,"props":761,"children":762},{"style":746},[763],{"type":46,"value":764}," --verdict-dataset-key",{"type":41,"tag":733,"props":766,"children":768},{"style":767},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[769],{"type":46,"value":770}," '",{"type":41,"tag":733,"props":772,"children":773},{"style":746},[774],{"type":46,"value":775},"\u003Cdataset-key>",{"type":41,"tag":733,"props":777,"children":778},{"style":767},[779],{"type":46,"value":780},"'\n",{"type":41,"tag":733,"props":782,"children":783},{"class":735,"line":25},[784],{"type":41,"tag":733,"props":785,"children":787},{"style":786},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[788],{"type":46,"value":789},"# or: uv run python evaluate.py --verdict-dataset-ref '\u003Cdataset-ref>'\n",{"type":41,"tag":155,"props":791,"children":792},{},[793,795,800],{"type":46,"value":794},"Use the ",{"type":41,"tag":55,"props":796,"children":798},{"className":797},[],[799],{"type":46,"value":68},{"type":46,"value":801}," skill to inspect the parent evaluation call.",{"type":41,"tag":155,"props":803,"children":804},{},[805,806,811],{"type":46,"value":794},{"type":41,"tag":55,"props":807,"children":809},{"className":808},[],[810],{"type":46,"value":68},{"type":46,"value":812}," skill to inspect the child row calls.",{"type":41,"tag":155,"props":814,"children":815},{},[816,818,824,826,832,834,840,841,847,848,854,855,861,862,868,869,875,876,882,884,890],{"type":46,"value":817},"Pull only the fields needed for diagnosis: row inputs, expected labels, model output, scorer outputs, call status, and prompt\u002Fmodel ref metadata when present. For the current verdict eval, include ",{"type":41,"tag":55,"props":819,"children":821},{"className":820},[],[822],{"type":46,"value":823},"expected_scope_status",{"type":46,"value":825},", output ",{"type":41,"tag":55,"props":827,"children":829},{"className":828},[],[830],{"type":46,"value":831},"scope_status",{"type":46,"value":833},", ",{"type":41,"tag":55,"props":835,"children":837},{"className":836},[],[838],{"type":46,"value":839},"verdict",{"type":46,"value":833},{"type":41,"tag":55,"props":842,"children":844},{"className":843},[],[845],{"type":46,"value":846},"verdict_reason",{"type":46,"value":833},{"type":41,"tag":55,"props":849,"children":851},{"className":850},[],[852],{"type":46,"value":853},"final_output",{"type":46,"value":833},{"type":41,"tag":55,"props":856,"children":858},{"className":857},[],[859],{"type":46,"value":860},"profile_review_markdown",{"type":46,"value":833},{"type":41,"tag":55,"props":863,"children":865},{"className":864},[],[866],{"type":46,"value":867},"run.researcher_prompt_ref",{"type":46,"value":833},{"type":41,"tag":55,"props":870,"children":872},{"className":871},[],[873],{"type":46,"value":874},"run.researcher_prompt_hash",{"type":46,"value":833},{"type":41,"tag":55,"props":877,"children":879},{"className":878},[],[880],{"type":46,"value":881},"run.researcher_model_ref",{"type":46,"value":883},", and ",{"type":41,"tag":55,"props":885,"children":887},{"className":886},[],[888],{"type":46,"value":889},"verdict_quality_scorer.is_correct",{"type":46,"value":187},{"type":41,"tag":155,"props":892,"children":893},{},[894],{"type":46,"value":895},"Filter to rows where the primary scorer failed. Keep a small sample of passing rows only when needed to avoid changing behavior that already works.",{"type":41,"tag":155,"props":897,"children":898},{},[899],{"type":46,"value":900},"Group failures into prompt-policy patterns. For verdict eval:",{"type":41,"tag":151,"props":902,"children":903},{},[904,925,942],{"type":41,"tag":155,"props":905,"children":906},{},[907,909,915,917,923],{"type":46,"value":908},"expected ",{"type":41,"tag":55,"props":910,"children":912},{"className":911},[],[913],{"type":46,"value":914},"rejected",{"type":46,"value":916},", observed ",{"type":41,"tag":55,"props":918,"children":920},{"className":919},[],[921],{"type":46,"value":922},"accepted",{"type":46,"value":924},": scope filter is too permissive.",{"type":41,"tag":155,"props":926,"children":927},{},[928,929,934,935,940],{"type":46,"value":908},{"type":41,"tag":55,"props":930,"children":932},{"className":931},[],[933],{"type":46,"value":922},{"type":46,"value":916},{"type":41,"tag":55,"props":936,"children":938},{"className":937},[],[939],{"type":46,"value":914},{"type":46,"value":941},": scope filter is too strict or treats metadata weakness as scope failure.",{"type":41,"tag":155,"props":943,"children":944},{},[945,947,953],{"type":46,"value":946},"observed ",{"type":41,"tag":55,"props":948,"children":950},{"className":949},[],[951],{"type":46,"value":952},"unknown",{"type":46,"value":954},": agent failed to call either save tool; improve the decision\u002Foutput discipline.",{"type":41,"tag":559,"props":956,"children":958},{"start":957},12,[959,964,984],{"type":41,"tag":155,"props":960,"children":961},{},[962],{"type":46,"value":963},"Read the local prompt that will be edited.",{"type":41,"tag":155,"props":965,"children":966},{},[967,969,974,976,982],{"type":46,"value":968},"Write ",{"type":41,"tag":55,"props":970,"children":972},{"className":971},[],[973],{"type":46,"value":339},{"type":46,"value":975}," before editing. Use the local execution start time for the directory name, for example ",{"type":41,"tag":55,"props":977,"children":979},{"className":978},[],[980],{"type":46,"value":981},"2026-06-12-1505-offline-eval",{"type":46,"value":983},". If the file already exists for that improvement loop, update it instead of creating a new call-ID directory.",{"type":41,"tag":155,"props":985,"children":986},{},[987,989,994,996,1002,1004,1010],{"type":46,"value":988},"Before editing, preserve the current prompt as ",{"type":41,"tag":55,"props":990,"children":992},{"className":991},[],[993],{"type":46,"value":307},{"type":46,"value":995},", where ",{"type":41,"tag":55,"props":997,"children":999},{"className":998},[],[1000],{"type":46,"value":1001},"\u003CN>",{"type":46,"value":1003}," is the next available integer starting from ",{"type":41,"tag":55,"props":1005,"children":1007},{"className":1006},[],[1008],{"type":46,"value":1009},"0",{"type":46,"value":187},{"type":41,"tag":414,"props":1012,"children":1014},{"id":1013},"evaluation-evidence-selection",[1015],{"type":46,"value":1016},"Evaluation Evidence Selection",{"type":41,"tag":49,"props":1018,"children":1019},{},[1020,1021,1026],{"type":46,"value":794},{"type":41,"tag":55,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":46,"value":68},{"type":46,"value":1027}," skill to fetch and inspect Weave Evaluation evidence. This skill only defines which Discovery Forge evidence matters:",{"type":41,"tag":151,"props":1029,"children":1030},{},[1031,1036,1041,1052,1152,1164],{"type":41,"tag":155,"props":1032,"children":1033},{},[1034],{"type":46,"value":1035},"Parent evaluation call: the exact eval call ID or link under analysis.",{"type":41,"tag":155,"props":1037,"children":1038},{},[1039],{"type":46,"value":1040},"Child row calls: the dataset-row-level evaluation results under that parent.",{"type":41,"tag":155,"props":1042,"children":1043},{},[1044,1046,1051],{"type":46,"value":1045},"Dataset row inputs: candidate name, candidate URL, description, and ",{"type":41,"tag":55,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":46,"value":823},{"type":46,"value":187},{"type":41,"tag":155,"props":1053,"children":1054},{},[1055,1057,1062,1063,1068,1069,1075,1076,1082,1083,1089,1090,1096,1097,1103,1105,1110,1111,1116,1117,1122,1123,1129,1131,1136,1138,1143,1145,1150],{"type":46,"value":1056},"Model output: reviewer payload (",{"type":41,"tag":55,"props":1058,"children":1060},{"className":1059},[],[1061],{"type":46,"value":860},{"type":46,"value":833},{"type":41,"tag":55,"props":1064,"children":1066},{"className":1065},[],[1067],{"type":46,"value":839},{"type":46,"value":833},{"type":41,"tag":55,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":46,"value":1074},"tool_name",{"type":46,"value":833},{"type":41,"tag":55,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":46,"value":1081},"primary_url",{"type":46,"value":833},{"type":41,"tag":55,"props":1084,"children":1086},{"className":1085},[],[1087],{"type":46,"value":1088},"summary",{"type":46,"value":833},{"type":41,"tag":55,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":46,"value":1095},"search",{"type":46,"value":833},{"type":41,"tag":55,"props":1098,"children":1100},{"className":1099},[],[1101],{"type":46,"value":1102},"run",{"type":46,"value":1104},") plus eval-compatible fields (",{"type":41,"tag":55,"props":1106,"children":1108},{"className":1107},[],[1109],{"type":46,"value":831},{"type":46,"value":833},{"type":41,"tag":55,"props":1112,"children":1114},{"className":1113},[],[1115],{"type":46,"value":846},{"type":46,"value":833},{"type":41,"tag":55,"props":1118,"children":1120},{"className":1119},[],[1121],{"type":46,"value":853},{"type":46,"value":883},{"type":41,"tag":55,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":46,"value":1128},"profile",{"type":46,"value":1130}," when available). For accepted\u002Frejected rows, ",{"type":41,"tag":55,"props":1132,"children":1134},{"className":1133},[],[1135],{"type":46,"value":831},{"type":46,"value":1137}," should mirror ",{"type":41,"tag":55,"props":1139,"children":1141},{"className":1140},[],[1142],{"type":46,"value":839},{"type":46,"value":1144},"; scorers read ",{"type":41,"tag":55,"props":1146,"children":1148},{"className":1147},[],[1149],{"type":46,"value":831},{"type":46,"value":1151}," first for compatibility.",{"type":41,"tag":155,"props":1153,"children":1154},{},[1155,1157,1162],{"type":46,"value":1156},"Scorer output: ",{"type":41,"tag":55,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":46,"value":889},{"type":46,"value":1163}," and any supporting scorer detail.",{"type":41,"tag":155,"props":1165,"children":1166},{},[1167,1169,1174],{"type":46,"value":1168},"Prompt\u002Fmodel metadata: prompt ref\u002Fhash and ",{"type":41,"tag":55,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":46,"value":391},{"type":46,"value":1175}," ref when present.",{"type":41,"tag":49,"props":1177,"children":1178},{},[1179],{"type":46,"value":1180},"Inspect one child row first to learn the shape, then extract the smallest stable set of fields needed to identify failed verdict rows.",{"type":41,"tag":77,"props":1182,"children":1184},{"id":1183},"plan-format",[1185],{"type":46,"value":1186},"Plan Format",{"type":41,"tag":49,"props":1188,"children":1189},{},[1190],{"type":46,"value":1191},"Use this structure:",{"type":41,"tag":722,"props":1193,"children":1197},{"className":1194,"code":1195,"language":1196,"meta":727,"style":727},"language-markdown shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Offline Eval Prompt Improvement Plan for \u003CYYYY-MM-DD-HHMM-offline-eval>\n\n## Source\n- Dataset: \u003Cdataset key or full ref>\n- Dataset ref: \u003Cresolved published Weave Dataset ref>\n- Start prompt: \u003Clocal path or Weave StringPrompt ref>\n- Baseline prompt ref\u002Fhash: \u003Cpublished prompt ref and hash from the run>\n- Baseline ResearcherAgentModel ref: \u003Cpublished model ref from the run>\n- Baseline Weave Evaluation parent call: \u003Ceval-call-id or link>\n- Baseline metric: `verdict_quality_scorer.is_correct` = \u003Cvalue>\n- Max iterations: \u003Cn>\n- Current iteration: \u003Ci>\u002F\u003Cn>\n- Failed rows inspected: \u003Cn>\n- Current `researcher.md` inspected\n\n## Prompt Edit Scope\n- Target clusters this iteration: \u003Cone or two related failure patterns>\n- Accept-protection: \u003Cpassing behavior that must stay stable>\n- Deferred patterns: \u003Cremaining failure patterns, if any>\n\n## Failure Summary\n- \u003Cconcrete failure pattern with candidate examples>\n\n## Proposed Prompt Change\n- \u003Cexact behavior change>\n\n## Expected Effect\n- \u003Cwhich failure patterns should improve and which passing behavior must stay stable>\n\n## Applied Files\n- `src\u002Fdiscovery_forge\u002Fagents\u002Fresearcher.md`\n\n## Out of Scope\n- Dataset rows, labels, scorer logic, evaluation runner, registry, schemas, and search backend behavior are not changed during prompt iteration.\n","markdown",[1198],{"type":41,"tag":55,"props":1199,"children":1200},{"__ignoreMap":727},[1201,1230,1239,1253,1302,1347,1395,1454,1500,1535,1579,1605,1648,1673,1703,1711,1724,1774,1825,1870,1878,1891,1937,1945,1958,1988,1996,2009,2078,2086,2099,2121,2129,2142],{"type":41,"tag":733,"props":1202,"children":1203},{"class":735,"line":736},[1204,1209,1214,1219,1225],{"type":41,"tag":733,"props":1205,"children":1206},{"style":767},[1207],{"type":46,"value":1208},"# ",{"type":41,"tag":733,"props":1210,"children":1211},{"style":740},[1212],{"type":46,"value":1213},"Offline Eval Prompt Improvement Plan for ",{"type":41,"tag":733,"props":1215,"children":1216},{"style":767},[1217],{"type":46,"value":1218},"\u003C",{"type":41,"tag":733,"props":1220,"children":1222},{"style":1221},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[1223],{"type":46,"value":1224},"YYYY-MM-DD-HHMM-offline-eval",{"type":41,"tag":733,"props":1226,"children":1227},{"style":767},[1228],{"type":46,"value":1229},">\n",{"type":41,"tag":733,"props":1231,"children":1232},{"class":735,"line":25},[1233],{"type":41,"tag":733,"props":1234,"children":1236},{"emptyLinePlaceholder":1235},true,[1237],{"type":46,"value":1238},"\n",{"type":41,"tag":733,"props":1240,"children":1242},{"class":735,"line":1241},3,[1243,1248],{"type":41,"tag":733,"props":1244,"children":1245},{"style":767},[1246],{"type":46,"value":1247},"## ",{"type":41,"tag":733,"props":1249,"children":1250},{"style":740},[1251],{"type":46,"value":1252},"Source\n",{"type":41,"tag":733,"props":1254,"children":1256},{"class":735,"line":1255},4,[1257,1262,1268,1272,1277,1283,1288,1293,1298],{"type":41,"tag":733,"props":1258,"children":1259},{"style":767},[1260],{"type":46,"value":1261},"-",{"type":41,"tag":733,"props":1263,"children":1265},{"style":1264},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1266],{"type":46,"value":1267}," Dataset: ",{"type":41,"tag":733,"props":1269,"children":1270},{"style":767},[1271],{"type":46,"value":1218},{"type":41,"tag":733,"props":1273,"children":1274},{"style":1221},[1275],{"type":46,"value":1276},"dataset",{"type":41,"tag":733,"props":1278,"children":1280},{"style":1279},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1281],{"type":46,"value":1282}," key",{"type":41,"tag":733,"props":1284,"children":1285},{"style":1279},[1286],{"type":46,"value":1287}," or",{"type":41,"tag":733,"props":1289,"children":1290},{"style":1279},[1291],{"type":46,"value":1292}," full",{"type":41,"tag":733,"props":1294,"children":1295},{"style":1279},[1296],{"type":46,"value":1297}," ref",{"type":41,"tag":733,"props":1299,"children":1300},{"style":767},[1301],{"type":46,"value":1229},{"type":41,"tag":733,"props":1303,"children":1305},{"class":735,"line":1304},5,[1306,1310,1315,1319,1324,1329,1334,1339,1343],{"type":41,"tag":733,"props":1307,"children":1308},{"style":767},[1309],{"type":46,"value":1261},{"type":41,"tag":733,"props":1311,"children":1312},{"style":1264},[1313],{"type":46,"value":1314}," Dataset ref: ",{"type":41,"tag":733,"props":1316,"children":1317},{"style":767},[1318],{"type":46,"value":1218},{"type":41,"tag":733,"props":1320,"children":1321},{"style":1221},[1322],{"type":46,"value":1323},"resolved",{"type":41,"tag":733,"props":1325,"children":1326},{"style":1279},[1327],{"type":46,"value":1328}," published",{"type":41,"tag":733,"props":1330,"children":1331},{"style":1279},[1332],{"type":46,"value":1333}," Weave",{"type":41,"tag":733,"props":1335,"children":1336},{"style":1279},[1337],{"type":46,"value":1338}," Dataset",{"type":41,"tag":733,"props":1340,"children":1341},{"style":1279},[1342],{"type":46,"value":1297},{"type":41,"tag":733,"props":1344,"children":1345},{"style":767},[1346],{"type":46,"value":1229},{"type":41,"tag":733,"props":1348,"children":1350},{"class":735,"line":1349},6,[1351,1355,1360,1364,1369,1374,1378,1382,1387,1391],{"type":41,"tag":733,"props":1352,"children":1353},{"style":767},[1354],{"type":46,"value":1261},{"type":41,"tag":733,"props":1356,"children":1357},{"style":1264},[1358],{"type":46,"value":1359}," Start prompt: ",{"type":41,"tag":733,"props":1361,"children":1362},{"style":767},[1363],{"type":46,"value":1218},{"type":41,"tag":733,"props":1365,"children":1366},{"style":1221},[1367],{"type":46,"value":1368},"local",{"type":41,"tag":733,"props":1370,"children":1371},{"style":1279},[1372],{"type":46,"value":1373}," path",{"type":41,"tag":733,"props":1375,"children":1376},{"style":1279},[1377],{"type":46,"value":1287},{"type":41,"tag":733,"props":1379,"children":1380},{"style":1279},[1381],{"type":46,"value":1333},{"type":41,"tag":733,"props":1383,"children":1384},{"style":1279},[1385],{"type":46,"value":1386}," StringPrompt",{"type":41,"tag":733,"props":1388,"children":1389},{"style":1279},[1390],{"type":46,"value":1297},{"type":41,"tag":733,"props":1392,"children":1393},{"style":767},[1394],{"type":46,"value":1229},{"type":41,"tag":733,"props":1396,"children":1398},{"class":735,"line":1397},7,[1399,1403,1408,1412,1417,1422,1426,1431,1436,1441,1446,1450],{"type":41,"tag":733,"props":1400,"children":1401},{"style":767},[1402],{"type":46,"value":1261},{"type":41,"tag":733,"props":1404,"children":1405},{"style":1264},[1406],{"type":46,"value":1407}," Baseline prompt ref\u002Fhash: ",{"type":41,"tag":733,"props":1409,"children":1410},{"style":767},[1411],{"type":46,"value":1218},{"type":41,"tag":733,"props":1413,"children":1414},{"style":1221},[1415],{"type":46,"value":1416},"published",{"type":41,"tag":733,"props":1418,"children":1419},{"style":1279},[1420],{"type":46,"value":1421}," prompt",{"type":41,"tag":733,"props":1423,"children":1424},{"style":1279},[1425],{"type":46,"value":1297},{"type":41,"tag":733,"props":1427,"children":1428},{"style":1279},[1429],{"type":46,"value":1430}," and",{"type":41,"tag":733,"props":1432,"children":1433},{"style":1279},[1434],{"type":46,"value":1435}," hash",{"type":41,"tag":733,"props":1437,"children":1438},{"style":1279},[1439],{"type":46,"value":1440}," from",{"type":41,"tag":733,"props":1442,"children":1443},{"style":1279},[1444],{"type":46,"value":1445}," the",{"type":41,"tag":733,"props":1447,"children":1448},{"style":1279},[1449],{"type":46,"value":749},{"type":41,"tag":733,"props":1451,"children":1452},{"style":767},[1453],{"type":46,"value":1229},{"type":41,"tag":733,"props":1455,"children":1457},{"class":735,"line":1456},8,[1458,1462,1467,1471,1475,1480,1484,1488,1492,1496],{"type":41,"tag":733,"props":1459,"children":1460},{"style":767},[1461],{"type":46,"value":1261},{"type":41,"tag":733,"props":1463,"children":1464},{"style":1264},[1465],{"type":46,"value":1466}," Baseline ResearcherAgentModel ref: ",{"type":41,"tag":733,"props":1468,"children":1469},{"style":767},[1470],{"type":46,"value":1218},{"type":41,"tag":733,"props":1472,"children":1473},{"style":1221},[1474],{"type":46,"value":1416},{"type":41,"tag":733,"props":1476,"children":1477},{"style":1279},[1478],{"type":46,"value":1479}," model",{"type":41,"tag":733,"props":1481,"children":1482},{"style":1279},[1483],{"type":46,"value":1297},{"type":41,"tag":733,"props":1485,"children":1486},{"style":1279},[1487],{"type":46,"value":1440},{"type":41,"tag":733,"props":1489,"children":1490},{"style":1279},[1491],{"type":46,"value":1445},{"type":41,"tag":733,"props":1493,"children":1494},{"style":1279},[1495],{"type":46,"value":749},{"type":41,"tag":733,"props":1497,"children":1498},{"style":767},[1499],{"type":46,"value":1229},{"type":41,"tag":733,"props":1501,"children":1503},{"class":735,"line":1502},9,[1504,1508,1513,1517,1522,1526,1531],{"type":41,"tag":733,"props":1505,"children":1506},{"style":767},[1507],{"type":46,"value":1261},{"type":41,"tag":733,"props":1509,"children":1510},{"style":1264},[1511],{"type":46,"value":1512}," Baseline Weave Evaluation parent call: ",{"type":41,"tag":733,"props":1514,"children":1515},{"style":767},[1516],{"type":46,"value":1218},{"type":41,"tag":733,"props":1518,"children":1519},{"style":1221},[1520],{"type":46,"value":1521},"eval-call-id",{"type":41,"tag":733,"props":1523,"children":1524},{"style":1279},[1525],{"type":46,"value":1287},{"type":41,"tag":733,"props":1527,"children":1528},{"style":1279},[1529],{"type":46,"value":1530}," link",{"type":41,"tag":733,"props":1532,"children":1533},{"style":767},[1534],{"type":46,"value":1229},{"type":41,"tag":733,"props":1536,"children":1538},{"class":735,"line":1537},10,[1539,1543,1548,1553,1557,1561,1566,1570,1575],{"type":41,"tag":733,"props":1540,"children":1541},{"style":767},[1542],{"type":46,"value":1261},{"type":41,"tag":733,"props":1544,"children":1545},{"style":1264},[1546],{"type":46,"value":1547}," Baseline metric: ",{"type":41,"tag":733,"props":1549,"children":1550},{"style":767},[1551],{"type":46,"value":1552},"`",{"type":41,"tag":733,"props":1554,"children":1555},{"style":746},[1556],{"type":46,"value":889},{"type":41,"tag":733,"props":1558,"children":1559},{"style":767},[1560],{"type":46,"value":1552},{"type":41,"tag":733,"props":1562,"children":1563},{"style":1264},[1564],{"type":46,"value":1565}," = ",{"type":41,"tag":733,"props":1567,"children":1568},{"style":767},[1569],{"type":46,"value":1218},{"type":41,"tag":733,"props":1571,"children":1572},{"style":1221},[1573],{"type":46,"value":1574},"value",{"type":41,"tag":733,"props":1576,"children":1577},{"style":767},[1578],{"type":46,"value":1229},{"type":41,"tag":733,"props":1580,"children":1582},{"class":735,"line":1581},11,[1583,1587,1592,1596,1601],{"type":41,"tag":733,"props":1584,"children":1585},{"style":767},[1586],{"type":46,"value":1261},{"type":41,"tag":733,"props":1588,"children":1589},{"style":1264},[1590],{"type":46,"value":1591}," Max iterations: ",{"type":41,"tag":733,"props":1593,"children":1594},{"style":767},[1595],{"type":46,"value":1218},{"type":41,"tag":733,"props":1597,"children":1598},{"style":1221},[1599],{"type":46,"value":1600},"n",{"type":41,"tag":733,"props":1602,"children":1603},{"style":767},[1604],{"type":46,"value":1229},{"type":41,"tag":733,"props":1606,"children":1607},{"class":735,"line":957},[1608,1612,1617,1621,1626,1631,1636,1640,1644],{"type":41,"tag":733,"props":1609,"children":1610},{"style":767},[1611],{"type":46,"value":1261},{"type":41,"tag":733,"props":1613,"children":1614},{"style":1264},[1615],{"type":46,"value":1616}," Current iteration: ",{"type":41,"tag":733,"props":1618,"children":1619},{"style":767},[1620],{"type":46,"value":1218},{"type":41,"tag":733,"props":1622,"children":1623},{"style":1221},[1624],{"type":46,"value":1625},"i",{"type":41,"tag":733,"props":1627,"children":1628},{"style":767},[1629],{"type":46,"value":1630},">",{"type":41,"tag":733,"props":1632,"children":1633},{"style":1264},[1634],{"type":46,"value":1635},"\u002F",{"type":41,"tag":733,"props":1637,"children":1638},{"style":767},[1639],{"type":46,"value":1218},{"type":41,"tag":733,"props":1641,"children":1642},{"style":1221},[1643],{"type":46,"value":1600},{"type":41,"tag":733,"props":1645,"children":1646},{"style":767},[1647],{"type":46,"value":1229},{"type":41,"tag":733,"props":1649,"children":1651},{"class":735,"line":1650},13,[1652,1656,1661,1665,1669],{"type":41,"tag":733,"props":1653,"children":1654},{"style":767},[1655],{"type":46,"value":1261},{"type":41,"tag":733,"props":1657,"children":1658},{"style":1264},[1659],{"type":46,"value":1660}," Failed rows inspected: ",{"type":41,"tag":733,"props":1662,"children":1663},{"style":767},[1664],{"type":46,"value":1218},{"type":41,"tag":733,"props":1666,"children":1667},{"style":1221},[1668],{"type":46,"value":1600},{"type":41,"tag":733,"props":1670,"children":1671},{"style":767},[1672],{"type":46,"value":1229},{"type":41,"tag":733,"props":1674,"children":1676},{"class":735,"line":1675},14,[1677,1681,1686,1690,1694,1698],{"type":41,"tag":733,"props":1678,"children":1679},{"style":767},[1680],{"type":46,"value":1261},{"type":41,"tag":733,"props":1682,"children":1683},{"style":1264},[1684],{"type":46,"value":1685}," Current ",{"type":41,"tag":733,"props":1687,"children":1688},{"style":767},[1689],{"type":46,"value":1552},{"type":41,"tag":733,"props":1691,"children":1692},{"style":746},[1693],{"type":46,"value":376},{"type":41,"tag":733,"props":1695,"children":1696},{"style":767},[1697],{"type":46,"value":1552},{"type":41,"tag":733,"props":1699,"children":1700},{"style":1264},[1701],{"type":46,"value":1702}," inspected\n",{"type":41,"tag":733,"props":1704,"children":1706},{"class":735,"line":1705},15,[1707],{"type":41,"tag":733,"props":1708,"children":1709},{"emptyLinePlaceholder":1235},[1710],{"type":46,"value":1238},{"type":41,"tag":733,"props":1712,"children":1714},{"class":735,"line":1713},16,[1715,1719],{"type":41,"tag":733,"props":1716,"children":1717},{"style":767},[1718],{"type":46,"value":1247},{"type":41,"tag":733,"props":1720,"children":1721},{"style":740},[1722],{"type":46,"value":1723},"Prompt Edit Scope\n",{"type":41,"tag":733,"props":1725,"children":1727},{"class":735,"line":1726},17,[1728,1732,1737,1741,1746,1750,1755,1760,1765,1770],{"type":41,"tag":733,"props":1729,"children":1730},{"style":767},[1731],{"type":46,"value":1261},{"type":41,"tag":733,"props":1733,"children":1734},{"style":1264},[1735],{"type":46,"value":1736}," Target clusters this iteration: ",{"type":41,"tag":733,"props":1738,"children":1739},{"style":767},[1740],{"type":46,"value":1218},{"type":41,"tag":733,"props":1742,"children":1743},{"style":1221},[1744],{"type":46,"value":1745},"one",{"type":41,"tag":733,"props":1747,"children":1748},{"style":1279},[1749],{"type":46,"value":1287},{"type":41,"tag":733,"props":1751,"children":1752},{"style":1279},[1753],{"type":46,"value":1754}," two",{"type":41,"tag":733,"props":1756,"children":1757},{"style":1279},[1758],{"type":46,"value":1759}," related",{"type":41,"tag":733,"props":1761,"children":1762},{"style":1279},[1763],{"type":46,"value":1764}," failure",{"type":41,"tag":733,"props":1766,"children":1767},{"style":1279},[1768],{"type":46,"value":1769}," patterns",{"type":41,"tag":733,"props":1771,"children":1772},{"style":767},[1773],{"type":46,"value":1229},{"type":41,"tag":733,"props":1775,"children":1777},{"class":735,"line":1776},18,[1778,1782,1787,1791,1796,1801,1806,1811,1816,1821],{"type":41,"tag":733,"props":1779,"children":1780},{"style":767},[1781],{"type":46,"value":1261},{"type":41,"tag":733,"props":1783,"children":1784},{"style":1264},[1785],{"type":46,"value":1786}," Accept-protection: ",{"type":41,"tag":733,"props":1788,"children":1789},{"style":767},[1790],{"type":46,"value":1218},{"type":41,"tag":733,"props":1792,"children":1793},{"style":1221},[1794],{"type":46,"value":1795},"passing",{"type":41,"tag":733,"props":1797,"children":1798},{"style":1279},[1799],{"type":46,"value":1800}," behavior",{"type":41,"tag":733,"props":1802,"children":1803},{"style":1279},[1804],{"type":46,"value":1805}," that",{"type":41,"tag":733,"props":1807,"children":1808},{"style":1279},[1809],{"type":46,"value":1810}," must",{"type":41,"tag":733,"props":1812,"children":1813},{"style":1279},[1814],{"type":46,"value":1815}," stay",{"type":41,"tag":733,"props":1817,"children":1818},{"style":1279},[1819],{"type":46,"value":1820}," stable",{"type":41,"tag":733,"props":1822,"children":1823},{"style":767},[1824],{"type":46,"value":1229},{"type":41,"tag":733,"props":1826,"children":1828},{"class":735,"line":1827},19,[1829,1833,1838,1842,1847,1851,1856,1861,1866],{"type":41,"tag":733,"props":1830,"children":1831},{"style":767},[1832],{"type":46,"value":1261},{"type":41,"tag":733,"props":1834,"children":1835},{"style":1264},[1836],{"type":46,"value":1837}," Deferred patterns: ",{"type":41,"tag":733,"props":1839,"children":1840},{"style":767},[1841],{"type":46,"value":1218},{"type":41,"tag":733,"props":1843,"children":1844},{"style":1221},[1845],{"type":46,"value":1846},"remaining",{"type":41,"tag":733,"props":1848,"children":1849},{"style":1279},[1850],{"type":46,"value":1764},{"type":41,"tag":733,"props":1852,"children":1853},{"style":1279},[1854],{"type":46,"value":1855}," patterns,",{"type":41,"tag":733,"props":1857,"children":1858},{"style":1279},[1859],{"type":46,"value":1860}," if",{"type":41,"tag":733,"props":1862,"children":1863},{"style":1279},[1864],{"type":46,"value":1865}," any",{"type":41,"tag":733,"props":1867,"children":1868},{"style":767},[1869],{"type":46,"value":1229},{"type":41,"tag":733,"props":1871,"children":1873},{"class":735,"line":1872},20,[1874],{"type":41,"tag":733,"props":1875,"children":1876},{"emptyLinePlaceholder":1235},[1877],{"type":46,"value":1238},{"type":41,"tag":733,"props":1879,"children":1881},{"class":735,"line":1880},21,[1882,1886],{"type":41,"tag":733,"props":1883,"children":1884},{"style":767},[1885],{"type":46,"value":1247},{"type":41,"tag":733,"props":1887,"children":1888},{"style":740},[1889],{"type":46,"value":1890},"Failure Summary\n",{"type":41,"tag":733,"props":1892,"children":1894},{"class":735,"line":1893},22,[1895,1899,1904,1909,1913,1918,1923,1928,1933],{"type":41,"tag":733,"props":1896,"children":1897},{"style":767},[1898],{"type":46,"value":1261},{"type":41,"tag":733,"props":1900,"children":1901},{"style":767},[1902],{"type":46,"value":1903}," \u003C",{"type":41,"tag":733,"props":1905,"children":1906},{"style":1221},[1907],{"type":46,"value":1908},"concrete",{"type":41,"tag":733,"props":1910,"children":1911},{"style":1279},[1912],{"type":46,"value":1764},{"type":41,"tag":733,"props":1914,"children":1915},{"style":1279},[1916],{"type":46,"value":1917}," pattern",{"type":41,"tag":733,"props":1919,"children":1920},{"style":1279},[1921],{"type":46,"value":1922}," with",{"type":41,"tag":733,"props":1924,"children":1925},{"style":1279},[1926],{"type":46,"value":1927}," candidate",{"type":41,"tag":733,"props":1929,"children":1930},{"style":1279},[1931],{"type":46,"value":1932}," examples",{"type":41,"tag":733,"props":1934,"children":1935},{"style":767},[1936],{"type":46,"value":1229},{"type":41,"tag":733,"props":1938,"children":1940},{"class":735,"line":1939},23,[1941],{"type":41,"tag":733,"props":1942,"children":1943},{"emptyLinePlaceholder":1235},[1944],{"type":46,"value":1238},{"type":41,"tag":733,"props":1946,"children":1948},{"class":735,"line":1947},24,[1949,1953],{"type":41,"tag":733,"props":1950,"children":1951},{"style":767},[1952],{"type":46,"value":1247},{"type":41,"tag":733,"props":1954,"children":1955},{"style":740},[1956],{"type":46,"value":1957},"Proposed Prompt Change\n",{"type":41,"tag":733,"props":1959,"children":1961},{"class":735,"line":1960},25,[1962,1966,1970,1975,1979,1984],{"type":41,"tag":733,"props":1963,"children":1964},{"style":767},[1965],{"type":46,"value":1261},{"type":41,"tag":733,"props":1967,"children":1968},{"style":767},[1969],{"type":46,"value":1903},{"type":41,"tag":733,"props":1971,"children":1972},{"style":1221},[1973],{"type":46,"value":1974},"exact",{"type":41,"tag":733,"props":1976,"children":1977},{"style":1279},[1978],{"type":46,"value":1800},{"type":41,"tag":733,"props":1980,"children":1981},{"style":1279},[1982],{"type":46,"value":1983}," change",{"type":41,"tag":733,"props":1985,"children":1986},{"style":767},[1987],{"type":46,"value":1229},{"type":41,"tag":733,"props":1989,"children":1991},{"class":735,"line":1990},26,[1992],{"type":41,"tag":733,"props":1993,"children":1994},{"emptyLinePlaceholder":1235},[1995],{"type":46,"value":1238},{"type":41,"tag":733,"props":1997,"children":1999},{"class":735,"line":1998},27,[2000,2004],{"type":41,"tag":733,"props":2001,"children":2002},{"style":767},[2003],{"type":46,"value":1247},{"type":41,"tag":733,"props":2005,"children":2006},{"style":740},[2007],{"type":46,"value":2008},"Expected Effect\n",{"type":41,"tag":733,"props":2010,"children":2012},{"class":735,"line":2011},28,[2013,2017,2021,2026,2030,2034,2039,2044,2048,2053,2058,2062,2066,2070,2074],{"type":41,"tag":733,"props":2014,"children":2015},{"style":767},[2016],{"type":46,"value":1261},{"type":41,"tag":733,"props":2018,"children":2019},{"style":767},[2020],{"type":46,"value":1903},{"type":41,"tag":733,"props":2022,"children":2023},{"style":1221},[2024],{"type":46,"value":2025},"which",{"type":41,"tag":733,"props":2027,"children":2028},{"style":1279},[2029],{"type":46,"value":1764},{"type":41,"tag":733,"props":2031,"children":2032},{"style":1279},[2033],{"type":46,"value":1769},{"type":41,"tag":733,"props":2035,"children":2036},{"style":1279},[2037],{"type":46,"value":2038}," should",{"type":41,"tag":733,"props":2040,"children":2041},{"style":1279},[2042],{"type":46,"value":2043}," improve",{"type":41,"tag":733,"props":2045,"children":2046},{"style":1279},[2047],{"type":46,"value":1430},{"type":41,"tag":733,"props":2049,"children":2050},{"style":1279},[2051],{"type":46,"value":2052}," which",{"type":41,"tag":733,"props":2054,"children":2055},{"style":1279},[2056],{"type":46,"value":2057}," passing",{"type":41,"tag":733,"props":2059,"children":2060},{"style":1279},[2061],{"type":46,"value":1800},{"type":41,"tag":733,"props":2063,"children":2064},{"style":1279},[2065],{"type":46,"value":1810},{"type":41,"tag":733,"props":2067,"children":2068},{"style":1279},[2069],{"type":46,"value":1815},{"type":41,"tag":733,"props":2071,"children":2072},{"style":1279},[2073],{"type":46,"value":1820},{"type":41,"tag":733,"props":2075,"children":2076},{"style":767},[2077],{"type":46,"value":1229},{"type":41,"tag":733,"props":2079,"children":2081},{"class":735,"line":2080},29,[2082],{"type":41,"tag":733,"props":2083,"children":2084},{"emptyLinePlaceholder":1235},[2085],{"type":46,"value":1238},{"type":41,"tag":733,"props":2087,"children":2089},{"class":735,"line":2088},30,[2090,2094],{"type":41,"tag":733,"props":2091,"children":2092},{"style":767},[2093],{"type":46,"value":1247},{"type":41,"tag":733,"props":2095,"children":2096},{"style":740},[2097],{"type":46,"value":2098},"Applied Files\n",{"type":41,"tag":733,"props":2100,"children":2102},{"class":735,"line":2101},31,[2103,2107,2112,2116],{"type":41,"tag":733,"props":2104,"children":2105},{"style":767},[2106],{"type":46,"value":1261},{"type":41,"tag":733,"props":2108,"children":2109},{"style":767},[2110],{"type":46,"value":2111}," `",{"type":41,"tag":733,"props":2113,"children":2114},{"style":746},[2115],{"type":46,"value":60},{"type":41,"tag":733,"props":2117,"children":2118},{"style":767},[2119],{"type":46,"value":2120},"`\n",{"type":41,"tag":733,"props":2122,"children":2124},{"class":735,"line":2123},32,[2125],{"type":41,"tag":733,"props":2126,"children":2127},{"emptyLinePlaceholder":1235},[2128],{"type":46,"value":1238},{"type":41,"tag":733,"props":2130,"children":2132},{"class":735,"line":2131},33,[2133,2137],{"type":41,"tag":733,"props":2134,"children":2135},{"style":767},[2136],{"type":46,"value":1247},{"type":41,"tag":733,"props":2138,"children":2139},{"style":740},[2140],{"type":46,"value":2141},"Out of Scope\n",{"type":41,"tag":733,"props":2143,"children":2145},{"class":735,"line":2144},34,[2146,2150],{"type":41,"tag":733,"props":2147,"children":2148},{"style":767},[2149],{"type":46,"value":1261},{"type":41,"tag":733,"props":2151,"children":2152},{"style":1264},[2153],{"type":46,"value":2154}," Dataset rows, labels, scorer logic, evaluation runner, registry, schemas, and search backend behavior are not changed during prompt iteration.\n",{"type":41,"tag":77,"props":2156,"children":2158},{"id":2157},"apply-rules",[2159],{"type":46,"value":2160},"Apply Rules",{"type":41,"tag":151,"props":2162,"children":2163},{},[2164,2175,2201,2222,2234,2239,2244,2257,2270,2282,2287],{"type":41,"tag":155,"props":2165,"children":2166},{},[2167,2169,2174],{"type":46,"value":2168},"Prompt-only changes may edit only ",{"type":41,"tag":55,"props":2170,"children":2172},{"className":2171},[],[2173],{"type":46,"value":60},{"type":46,"value":187},{"type":41,"tag":155,"props":2176,"children":2177},{},[2178,2180,2185,2187,2192,2194,2199],{"type":46,"value":2179},"Evaluation reruns publish the local ",{"type":41,"tag":55,"props":2181,"children":2183},{"className":2182},[],[2184],{"type":46,"value":376},{"type":46,"value":2186}," prompt and ",{"type":41,"tag":55,"props":2188,"children":2190},{"className":2189},[],[2191],{"type":46,"value":391},{"type":46,"value":2193}," by design so every run has version refs in Weave. Do not change ",{"type":41,"tag":55,"props":2195,"children":2197},{"className":2196},[],[2198],{"type":46,"value":147},{"type":46,"value":2200},", datasets, labels, or scorers to force a better score.",{"type":41,"tag":155,"props":2202,"children":2203},{},[2204,2206,2212,2214,2220],{"type":46,"value":2205},"Before each prompt edit, create the next backup file: ",{"type":41,"tag":55,"props":2207,"children":2209},{"className":2208},[],[2210],{"type":46,"value":2211},"researcher_backup_0.md",{"type":46,"value":2213},", then ",{"type":41,"tag":55,"props":2215,"children":2217},{"className":2216},[],[2218],{"type":46,"value":2219},"researcher_backup_1.md",{"type":46,"value":2221},", and so on. Never overwrite an existing backup.",{"type":41,"tag":155,"props":2223,"children":2224},{},[2225,2227,2232],{"type":46,"value":2226},"Do not change evaluation datasets, labels, scorers, or ",{"type":41,"tag":55,"props":2228,"children":2230},{"className":2229},[],[2231],{"type":46,"value":147},{"type":46,"value":2233}," to make results look better.",{"type":41,"tag":155,"props":2235,"children":2236},{},[2237],{"type":46,"value":2238},"Do not create fallback behavior.",{"type":41,"tag":155,"props":2240,"children":2241},{},[2242],{"type":46,"value":2243},"Keep scope verdict separate from metadata completeness.",{"type":41,"tag":155,"props":2245,"children":2246},{},[2247,2249,2255],{"type":46,"value":2248},"If primary sources prove the task -> evaluator -> feedback\u002Fmemory -> improvement loop but metadata is incomplete, record the weakness in ",{"type":41,"tag":55,"props":2250,"children":2252},{"className":2251},[],[2253],{"type":46,"value":2254},"key_limitations",{"type":46,"value":2256}," instead of rejecting solely for metadata incompleteness.",{"type":41,"tag":155,"props":2258,"children":2259},{},[2260,2262,2268],{"type":46,"value":2261},"If ",{"type":41,"tag":55,"props":2263,"children":2265},{"className":2264},[],[2266],{"type":46,"value":2267},"expected_scope_status == accepted",{"type":46,"value":2269}," and the agent rejects due to missing metadata, treat the prompt as over-strict unless the dataset row appears wrong.",{"type":41,"tag":155,"props":2271,"children":2272},{},[2273,2274,2280],{"type":46,"value":2261},{"type":41,"tag":55,"props":2275,"children":2277},{"className":2276},[],[2278],{"type":46,"value":2279},"expected_scope_status == rejected",{"type":46,"value":2281}," and the agent accepts, treat it as a scope-policy issue unless the dataset row appears wrong.",{"type":41,"tag":155,"props":2283,"children":2284},{},[2285],{"type":46,"value":2286},"If a dataset row, label, or scorer appears wrong, report it as dataset\u002Fscorer maintenance rather than editing those artifacts in this workflow.",{"type":41,"tag":155,"props":2288,"children":2289},{},[2290],{"type":46,"value":2291},"Use failed candidate names as policy examples in the plan. Do not hardcode them as required outputs.",{"type":41,"tag":49,"props":2293,"children":2294},{},[2295,2297,2302],{"type":46,"value":2296},"After editing, write ",{"type":41,"tag":55,"props":2298,"children":2300},{"className":2299},[],[2301],{"type":46,"value":347},{"type":46,"value":2303},". If the file already exists for that improvement loop, update it with the latest applied summary.",{"type":41,"tag":77,"props":2305,"children":2307},{"id":2306},"local-prompt-iteration",[2308],{"type":46,"value":2309},"Local Prompt Iteration",{"type":41,"tag":49,"props":2311,"children":2312},{},[2313,2315,2320,2322,2327,2329,2334,2336,2341],{"type":46,"value":2314},"During development, evaluate the local prompt file directly. Do not pass ",{"type":41,"tag":55,"props":2316,"children":2318},{"className":2317},[],[2319],{"type":46,"value":399},{"type":46,"value":2321},"; ",{"type":41,"tag":55,"props":2323,"children":2325},{"className":2324},[],[2326],{"type":46,"value":147},{"type":46,"value":2328}," will publish the current local prompt and ",{"type":41,"tag":55,"props":2330,"children":2332},{"className":2331},[],[2333],{"type":46,"value":391},{"type":46,"value":2335}," for each run so the result is versioned. Pass ",{"type":41,"tag":55,"props":2337,"children":2339},{"className":2338},[],[2340],{"type":46,"value":399},{"type":46,"value":2342}," only when the goal is to compare a fixed published prompt ref.",{"type":41,"tag":49,"props":2344,"children":2345},{},[2346],{"type":46,"value":2347},"Run no more than the requested max iterations. Stop early if the primary metric improves enough for the user's goal, if the remaining failures are dataset\u002Fscorer maintenance issues, or if two consecutive focused edits regress the baseline.",{"type":41,"tag":49,"props":2349,"children":2350},{},[2351],{"type":46,"value":2352},"For each iteration:",{"type":41,"tag":559,"props":2354,"children":2355},{},[2356,2368,2373,2378,2383,2436,2441,2446,2451],{"type":41,"tag":155,"props":2357,"children":2358},{},[2359,2361,2366],{"type":46,"value":2360},"Update ",{"type":41,"tag":55,"props":2362,"children":2364},{"className":2363},[],[2365],{"type":46,"value":468},{"type":46,"value":2367}," with the current iteration number, selected dataset ref, start prompt, baseline eval ID, target cluster, and deferred patterns.",{"type":41,"tag":155,"props":2369,"children":2370},{},[2371],{"type":46,"value":2372},"Create the next backup file.",{"type":41,"tag":155,"props":2374,"children":2375},{},[2376],{"type":46,"value":2377},"Make a focused edit covering one or two related clusters with paired accept-protection.",{"type":41,"tag":155,"props":2379,"children":2380},{},[2381],{"type":46,"value":2382},"Run focused validation.",{"type":41,"tag":155,"props":2384,"children":2385},{},[2386,2388],{"type":46,"value":2387},"Run the same selected dataset:",{"type":41,"tag":722,"props":2389,"children":2390},{"className":724,"code":725,"language":726,"meta":727,"style":727},[2391],{"type":41,"tag":55,"props":2392,"children":2393},{"__ignoreMap":727},[2394,2429],{"type":41,"tag":733,"props":2395,"children":2396},{"class":735,"line":736},[2397,2401,2405,2409,2413,2417,2421,2425],{"type":41,"tag":733,"props":2398,"children":2399},{"style":740},[2400],{"type":46,"value":743},{"type":41,"tag":733,"props":2402,"children":2403},{"style":746},[2404],{"type":46,"value":749},{"type":41,"tag":733,"props":2406,"children":2407},{"style":746},[2408],{"type":46,"value":754},{"type":41,"tag":733,"props":2410,"children":2411},{"style":746},[2412],{"type":46,"value":759},{"type":41,"tag":733,"props":2414,"children":2415},{"style":746},[2416],{"type":46,"value":764},{"type":41,"tag":733,"props":2418,"children":2419},{"style":767},[2420],{"type":46,"value":770},{"type":41,"tag":733,"props":2422,"children":2423},{"style":746},[2424],{"type":46,"value":775},{"type":41,"tag":733,"props":2426,"children":2427},{"style":767},[2428],{"type":46,"value":780},{"type":41,"tag":733,"props":2430,"children":2431},{"class":735,"line":25},[2432],{"type":41,"tag":733,"props":2433,"children":2434},{"style":786},[2435],{"type":46,"value":789},{"type":41,"tag":155,"props":2437,"children":2438},{},[2439],{"type":46,"value":2440},"Compare the new run against the baseline evaluation ID and the previous iteration.",{"type":41,"tag":155,"props":2442,"children":2443},{},[2444],{"type":46,"value":2445},"Keep the change only if the selected dataset metric improves across runs without unacceptable row-level regressions. Because a single run varies by ~±2-3 rows, rerun 2-3 times or compare per-row stability before trusting a borderline change.",{"type":41,"tag":155,"props":2447,"children":2448},{},[2449],{"type":46,"value":2450},"If correctly-accepted rows flipped to rejected, strengthen the accept-protection clause and rerun within the max-iteration budget.",{"type":41,"tag":155,"props":2452,"children":2453},{},[2454],{"type":46,"value":2455},"If the metric regresses across runs, revert to the backup and either try the next focused cluster or stop and report the regression.",{"type":41,"tag":77,"props":2457,"children":2459},{"id":2458},"validation",[2460],{"type":46,"value":2461},"Validation",{"type":41,"tag":49,"props":2463,"children":2464},{},[2465],{"type":46,"value":2466},"Run focused validation:",{"type":41,"tag":722,"props":2468,"children":2470},{"className":724,"code":2469,"language":726,"meta":727,"style":727},"uv run pytest tests\u002Funit\u002Ftest_researcher.py -q\n",[2471],{"type":41,"tag":55,"props":2472,"children":2473},{"__ignoreMap":727},[2474],{"type":41,"tag":733,"props":2475,"children":2476},{"class":735,"line":736},[2477,2481,2485,2490,2495],{"type":41,"tag":733,"props":2478,"children":2479},{"style":740},[2480],{"type":46,"value":743},{"type":41,"tag":733,"props":2482,"children":2483},{"style":746},[2484],{"type":46,"value":749},{"type":41,"tag":733,"props":2486,"children":2487},{"style":746},[2488],{"type":46,"value":2489}," pytest",{"type":41,"tag":733,"props":2491,"children":2492},{"style":746},[2493],{"type":46,"value":2494}," tests\u002Funit\u002Ftest_researcher.py",{"type":41,"tag":733,"props":2496,"children":2497},{"style":746},[2498],{"type":46,"value":2499}," -q\n",{"type":41,"tag":49,"props":2501,"children":2502},{},[2503],{"type":46,"value":2504},"Then rerun the same selected dataset:",{"type":41,"tag":722,"props":2506,"children":2507},{"className":724,"code":725,"language":726,"meta":727,"style":727},[2508],{"type":41,"tag":55,"props":2509,"children":2510},{"__ignoreMap":727},[2511,2546],{"type":41,"tag":733,"props":2512,"children":2513},{"class":735,"line":736},[2514,2518,2522,2526,2530,2534,2538,2542],{"type":41,"tag":733,"props":2515,"children":2516},{"style":740},[2517],{"type":46,"value":743},{"type":41,"tag":733,"props":2519,"children":2520},{"style":746},[2521],{"type":46,"value":749},{"type":41,"tag":733,"props":2523,"children":2524},{"style":746},[2525],{"type":46,"value":754},{"type":41,"tag":733,"props":2527,"children":2528},{"style":746},[2529],{"type":46,"value":759},{"type":41,"tag":733,"props":2531,"children":2532},{"style":746},[2533],{"type":46,"value":764},{"type":41,"tag":733,"props":2535,"children":2536},{"style":767},[2537],{"type":46,"value":770},{"type":41,"tag":733,"props":2539,"children":2540},{"style":746},[2541],{"type":46,"value":775},{"type":41,"tag":733,"props":2543,"children":2544},{"style":767},[2545],{"type":46,"value":780},{"type":41,"tag":733,"props":2547,"children":2548},{"class":735,"line":25},[2549],{"type":41,"tag":733,"props":2550,"children":2551},{"style":786},[2552],{"type":46,"value":789},{"type":41,"tag":77,"props":2554,"children":2556},{"id":2555},"report-back",[2557],{"type":46,"value":2558},"Report Back",{"type":41,"tag":49,"props":2560,"children":2561},{},[2562],{"type":46,"value":2563},"Report:",{"type":41,"tag":151,"props":2565,"children":2566},{},[2567,2572,2577,2582,2587,2592,2602,2607,2612,2617,2629,2634,2639,2644,2654],{"type":41,"tag":155,"props":2568,"children":2569},{},[2570],{"type":46,"value":2571},"dataset key\u002Fref used",{"type":41,"tag":155,"props":2573,"children":2574},{},[2575],{"type":46,"value":2576},"start prompt path\u002Fref",{"type":41,"tag":155,"props":2578,"children":2579},{},[2580],{"type":46,"value":2581},"baseline evaluation ID",{"type":41,"tag":155,"props":2583,"children":2584},{},[2585],{"type":46,"value":2586},"max iterations requested and iterations completed",{"type":41,"tag":155,"props":2588,"children":2589},{},[2590],{"type":46,"value":2591},"targeted failure cluster and deferred patterns",{"type":41,"tag":155,"props":2593,"children":2594},{},[2595,2597],{"type":46,"value":2596},"baseline ",{"type":41,"tag":55,"props":2598,"children":2600},{"className":2599},[],[2601],{"type":46,"value":889},{"type":41,"tag":155,"props":2603,"children":2604},{},[2605],{"type":46,"value":2606},"number of failed rows inspected",{"type":41,"tag":155,"props":2608,"children":2609},{},[2610],{"type":46,"value":2611},"failure patterns found",{"type":41,"tag":155,"props":2613,"children":2614},{},[2615],{"type":46,"value":2616},"prompt changes made",{"type":41,"tag":155,"props":2618,"children":2619},{},[2620,2622,2627],{"type":46,"value":2621},"prompt ref\u002Fhash and ",{"type":41,"tag":55,"props":2623,"children":2625},{"className":2624},[],[2626],{"type":46,"value":391},{"type":46,"value":2628}," ref used by each rerun",{"type":41,"tag":155,"props":2630,"children":2631},{},[2632],{"type":46,"value":2633},"both error directions after rerun: false accepts fixed AND any new false rejects introduced",{"type":41,"tag":155,"props":2635,"children":2636},{},[2637],{"type":46,"value":2638},"backup file created",{"type":41,"tag":155,"props":2640,"children":2641},{},[2642],{"type":46,"value":2643},"tests run",{"type":41,"tag":155,"props":2645,"children":2646},{},[2647,2649],{"type":46,"value":2648},"rerun evaluation ID(s) and ",{"type":41,"tag":55,"props":2650,"children":2652},{"className":2651},[],[2653],{"type":46,"value":889},{"type":41,"tag":155,"props":2655,"children":2656},{},[2657],{"type":46,"value":2658},"remaining failed patterns or dataset\u002Fscorer maintenance notes",{"type":41,"tag":2660,"props":2661,"children":2662},"style",{},[2663],{"type":46,"value":2664},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2666,"total":1241},[2667,2680,2693],{"slug":2668,"name":2668,"fn":2669,"description":2670,"org":2671,"tags":2672,"stars":25,"repoUrl":26,"updatedAt":2679},"annotation-improvement","improve prompts using Weave research traces","Guides coding agents through Discovery Forge prompt improvement after selecting the `wandb-primary` skill. Use `wandb-primary` to fetch W&B Weave research_run traces, human annotations, runnable feedback, and evaluations, then use this workflow when improving researcher.md from annotation queues, reviewed research traces, or human feedback.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2673,2674,2675,2678],{"name":23,"slug":24,"type":16},{"name":20,"slug":21,"type":16},{"name":2676,"slug":2677,"type":16},"Prompt Engineering","prompt-engineering",{"name":9,"slug":18,"type":16},"2026-07-17T06:06:36.008062",{"slug":2681,"name":2681,"fn":2682,"description":2683,"org":2684,"tags":2685,"stars":25,"repoUrl":26,"updatedAt":2692},"build-verdict-dataset","build verdict datasets from Weave annotations","Guides coding agents through building the verdict_quality_dataset from W&B Weave research_annotation evidence — querying annotated research_run calls, mapping human QualitySelector verdicts to gold labels, refining row inputs per the rubric, and publishing a new versioned Weave Dataset. Use when the user asks to (re)generate the verdict dataset from annotations, seed a new eval dataset version, or rebuild verdict_quality_dataset.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2686,2689,2690,2691],{"name":2687,"slug":2688,"type":16},"Data Analysis","data-analysis",{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},"2026-07-17T06:07:59.30763",{"slug":4,"name":4,"fn":5,"description":6,"org":2694,"tags":2695,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2696,2697,2698,2699],{"name":23,"slug":24,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},{"items":2701,"total":1502},[2702,2719,2738,2751,2764,2774,2781,2788,2795],{"slug":2703,"name":2703,"fn":2704,"description":2705,"org":2706,"tags":2707,"stars":2716,"repoUrl":2717,"updatedAt":2718},"weave-instrument","instrument LLM codebases with Weave","Add Weave (Weights & Biases) observability to an LLM or agent codebase. This covers calling `weave.init()`, setting up authentication, and choosing between OTEL auto-instrumentation and the explicit Session SDK agent-logging APIs (Turn, LLM, Tool, SubAgent), based on the libraries the code already uses. Works for Python and TypeScript\u002FNode. Use this whenever the user wants to instrument, trace, or add observability, logging, or monitoring to an agent, chatbot, RAG pipeline, or LLM app. This includes phrasings like \"log my agent to weave\", \"add agent tracing\", \"get my agent into the Weave Agents tab\", \"instrument this with the weave session sdk\", \"trace my tool calls\", or \"set up weave logging\", even when the user does not name a specific API.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2708,2709,2712,2715],{"name":23,"slug":24,"type":16},{"name":2710,"slug":2711,"type":16},"LLM","llm",{"name":2713,"slug":2714,"type":16},"Observability","observability",{"name":9,"slug":18,"type":16},1104,"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fweave","2026-07-17T06:07:25.81154",{"slug":68,"name":68,"fn":2720,"description":2721,"org":2722,"tags":2723,"stars":2735,"repoUrl":2736,"updatedAt":2737},"manage Weights & Biases projects and runs","Primary W&B skill for broad or mixed Weights & Biases work: project overviews, W&B runs and artifacts, Weave traces and evaluations, Reports, and Launch workflows. Use when the task spans multiple W&B surfaces or the user asks generally what is happening in a W&B project.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2724,2725,2726,2729,2732],{"name":2687,"slug":2688,"type":16},{"name":20,"slug":21,"type":16},{"name":2727,"slug":2728,"type":16},"MLOps","mlops",{"name":2730,"slug":2731,"type":16},"Monitoring","monitoring",{"name":2733,"slug":2734,"type":16},"Tracing","tracing",60,"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fskills","2026-07-17T06:06:35.194929",{"slug":2739,"name":2739,"fn":2740,"description":2741,"org":2742,"tags":2743,"stars":1502,"repoUrl":2749,"updatedAt":2750},"weave-config","configure Weave project and settings","This skill should be used when the user wants to \"configure weave\", \"set weave project\", \"change weave project\", \"set wandb api key\", \"update weave settings\", \"show weave config\", \"change weave configuration\", \"restart the weave daemon\", \"apply weave config changes\", \"restart weave to pick up changes\", or needs to read or update any Weave Claude Code plugin settings.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2744,2747,2748],{"name":2745,"slug":2746,"type":16},"Configuration","configuration",{"name":2713,"slug":2714,"type":16},{"name":9,"slug":18,"type":16},"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fweave-claude-code","2026-07-17T06:07:38.107392",{"slug":2752,"name":2752,"fn":2753,"description":2754,"org":2755,"tags":2756,"stars":1502,"repoUrl":2749,"updatedAt":2763},"weave-install","install and configure Weave observability","This skill should be used when the user wants to \"install the weave plugin\", \"set up weave\", \"install weave-claude-code\", \"configure weave for the first time\", \"get started with weave tracing\", or needs to complete the initial setup of the Weave Claude Code plugin including dependency installation and project configuration.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2757,2760,2761,2762],{"name":2758,"slug":2759,"type":16},"Claude Code","claude-code",{"name":2745,"slug":2746,"type":16},{"name":2713,"slug":2714,"type":16},{"name":9,"slug":18,"type":16},"2026-07-17T06:06:35.660384",{"slug":2765,"name":2765,"fn":2766,"description":2767,"org":2768,"tags":2769,"stars":1502,"repoUrl":2749,"updatedAt":2773},"weave-status","verify Weave plugin status and health","This skill should be used when the user wants to \"check weave status\", \"verify the weave plugin is running\", \"see if weave is set up correctly\", \"check weave configuration\", \"is weave working\", \"weave is running an older config\", \"the daemon is on an old config\", or needs to diagnose why Claude Code sessions are not appearing in Weave.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2770,2771,2772],{"name":2730,"slug":2731,"type":16},{"name":2713,"slug":2714,"type":16},{"name":9,"slug":18,"type":16},"2026-07-17T06:07:58.961737",{"slug":2668,"name":2668,"fn":2669,"description":2670,"org":2775,"tags":2776,"stars":25,"repoUrl":26,"updatedAt":2679},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2777,2778,2779,2780],{"name":23,"slug":24,"type":16},{"name":20,"slug":21,"type":16},{"name":2676,"slug":2677,"type":16},{"name":9,"slug":18,"type":16},{"slug":2681,"name":2681,"fn":2682,"description":2683,"org":2782,"tags":2783,"stars":25,"repoUrl":26,"updatedAt":2692},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2784,2785,2786,2787],{"name":2687,"slug":2688,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":2789,"tags":2790,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2791,2792,2793,2794],{"name":23,"slug":24,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},{"slug":2796,"name":2796,"fn":2797,"description":2798,"org":2799,"tags":2800,"stars":25,"repoUrl":2807,"updatedAt":2808},"weave-integration","integrate Weave into applications","Comprehensive skill for adding W&B Weave to existing applications. Covers trace-first instrumentation, evaluation only after trace verification, documentation-first implementation, CLI-based Weave data access, and validation workflows. Activate this skill only when the user explicitly mentions the skill by name, such as `weave-integration`, `@skills weave-integration`, or another direct skill reference. Do not auto-trigger from generic Weave or W&B questions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2801,2804,2805,2806],{"name":2802,"slug":2803,"type":16},"Instrumentation","instrumentation",{"name":2713,"slug":2714,"type":16},{"name":2733,"slug":2734,"type":16},{"name":9,"slug":18,"type":16},"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fweave-integration-skills","2026-07-17T06:07:59.666449"]