[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-weights-and-biases-build-verdict-dataset":3,"mdc-absu88-key":35,"related-org-weights-and-biases-build-verdict-dataset":1410,"related-repo-weights-and-biases-build-verdict-dataset":1532},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":33,"mdContent":34},"build-verdict-dataset","build verdict datasets from Weave annotations","Guides coding agents through building the verdict_quality_dataset from W&B Weave research_annotation evidence — querying annotated research_run calls, mapping human QualitySelector verdicts to gold labels, refining row inputs per the rubric, and publishing a new versioned Weave Dataset. Use when the user asks to (re)generate the verdict dataset from annotations, seed a new eval dataset version, or rebuild verdict_quality_dataset.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"weights-and-biases","Weights & Biases","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fweights-and-biases.png","wandb",[13,17,19,22],{"name":14,"slug":15,"type":16},"Datasets","datasets","tag",{"name":9,"slug":18,"type":16},"weights-biases",{"name":20,"slug":21,"type":16},"Evals","evals",{"name":23,"slug":24,"type":16},"Data Analysis","data-analysis",2,"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fdiscovery-forge","2026-07-17T06:07:59.30763",null,0,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":28},[],"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fdiscovery-forge\u002Ftree\u002FHEAD\u002Fskills\u002Fbuild-verdict-dataset","---\nname: build-verdict-dataset\ndescription: Guides coding agents through building the verdict_quality_dataset from W&B Weave research_annotation evidence — querying annotated research_run calls, mapping human QualitySelector verdicts to gold labels, refining row inputs per the rubric, and publishing a new versioned Weave Dataset. Use when the user asks to (re)generate the verdict dataset from annotations, seed a new eval dataset version, or rebuild verdict_quality_dataset.\n---\n\n# Build Verdict Dataset\n\nUse this skill to create or refresh `verdict_quality_dataset` from human annotations in Weave, then publish it as a versioned Weave Dataset and pin the ref.\n\nThe coding agent queries annotated `research_run_\u003Ci>` calls, extracts candidate fields, maps the human verdict to a gold label, refines each row against the rubric, writes a local JSONL plus an audit report, publishes the dataset, and updates the pinned ref in `evaluation_config.yaml`.\n\nThis skill **does not change the human annotation results themselves**. It only transforms annotation evidence into a clean eval dataset and records provenance.\n\n## Before You Start\n\nFollow `AGENTS.md` W&B Skills setup first. Use W&B Skills for Weave trace, feedback, annotation, and dataset access; this skill only defines the Discovery Forge dataset-building workflow.\n\nFetch all trace and feedback evidence live from Weave through W&B Skills. Do not use W&B MCP tools, and do not add discovery-forge query wrappers.\n\n## Default Project\n\n- Entity: read from `.env` as `WANDB_ENTITY` (required; use your own W&B entity)\n- Project: read from `.env` as `WANDB_PROJECT` (required; `.env.example` uses `discovery-forge`)\n- API key: read from `.env` as `WANDB_API_KEY` (required)\n- Annotation queue: `research_annotation`\n- Root trace unit: one `research_run_\u003Ci>` \u002F `openai_agent_trace` call per reviewed candidate\n- Dataset name: `verdict_quality_dataset`\n- Pinned ref: `src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml` `datasets.verdict_quality.ref`\n- Publish helper: `discovery_forge.evaluation.datasets.publish_eval_dataset`\n- Labeling standard: `src\u002Fdiscovery_forge\u002Fevaluation\u002Fverdict_dataset_rubric.md`\n- Local outputs: `src\u002Fdiscovery_forge\u002Fevaluation\u002Fdatasets\u002Fverdict_quality_dataset_clean_\u003CYYYY-MM-DD>.jsonl` and `verdict_quality_dataset_audit_\u003CYYYY-MM-DD>.md`\n\n## Row Schema\n\nEach dataset row must contain:\n\n- `id`: stable slug for the candidate (kebab-case of the tool name)\n- `input_tool_name`: candidate name\n- `input_candidate_url`: candidate primary URL\n- `input_candidate_description`: neutral candidate description (see hygiene rules)\n- `expected_scope_status`: `accepted` or `rejected` (the only field the scorer reads)\n- `expected_issue_category`: for rejects, one of `out_of_scope`, `missing_url`, `duplicate_known_tool`; otherwise `null`\n- `label_reason`: human-readable rationale (not scored)\n- `annotation_source`: `{queue_name, call_id, quality_selector}` provenance — copy verbatim from the source annotation, never invent or alter\n\n## Workflow\n\n1. Identify the source scope: a set of `research_run` call IDs (from the user or a Weave UI link), a run day, or \"all reviewed items in `research_annotation`\". Prefer an explicit call-ID list or run day over \"every historical annotation\". Get call IDs from Weave, not from local files.\n2. Use W&B Skills to fetch the annotated root calls and feedback **from Weave**.\n3. For each call, separate human annotations (`wandb.annotation.QualitySelector`, `wandb.annotation.QualityReviewer`) from runnable scorer feedback. Use only `research_annotation` human annotations as the verdict seed; do not seed labels from runnable scorers.\n4. Inspect one call first to learn the exact `output` shape, then extract the candidate name, primary URL, and a pre-verdict description plus the human reviewer's rationale.\n5. Map the human `QualitySelector` to a draft gold label:\n   - `Good` -> `accepted`\n   - `Bad` -> `rejected`\n   - `Neutral` -> **ask the user how to handle it.** Do not silently auto-assign. Present each neutral candidate with its `QualityReviewer` rationale and a recommended verdict (`accepted` with `key_limitations`, `rejected`, or `drop`), then let the user decide per row or give a blanket rule. Only fall back to a provisional label + `needs_review` flag if the user explicitly defers or says to proceed without them.\n6. Refine each row against `verdict_dataset_rubric.md` (see Labeling and Hygiene). Record `keep` \u002F `relabel` \u002F `drop` \u002F `needs_review` with a reason.\n7. Write the clean JSONL and the audit report.\n8. Publish the dataset and update `datasets.verdict_quality.ref` in `src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml`.\n9. Validate.\n\n### Evidence Selection\n\nUse W&B Skills to fetch and inspect Weave evidence. This skill only defines which Discovery Forge evidence matters:\n\n- Root trace unit: one `research_run_\u003Ci>` \u002F `openai_agent_trace` call per reviewed candidate.\n- Inspect only calls in the requested source scope.\n- Keep only human annotations from `research_annotation` (`QualitySelector`, `QualityReviewer`) as verdict seeds.\n- Do not seed labels from runnable scorer feedback.\n- Deduplicate annotations by feedback ID.\n- Inspect one representative call first to learn where candidate name, primary URL, description, reviewer rationale, and annotation payload live.\n\n## Labeling and Hygiene\n\nApply `verdict_dataset_rubric.md`. Key rules:\n\n- **Accept** when the candidate itself runs a loop: task -> evaluation -> feedback\u002Fstate -> revision of its own artifact.\n- **Reject** lists\u002Fsurveys\u002Fcookbooks\u002Fguides, GUI\u002Fcomputer-use frameworks, testing\u002Fevaluation-only gates, repository-automation hosts, memory-only components, generic frameworks, and weak-evidence candidates.\n- **Metadata vs scope**: missing stars\u002Flicense\u002Fdates\u002FGitHub fetch is a profile limitation, not a scope reject. Do not encode metadata gaps as `missing_url` when a primary URL exists.\n- **Input hygiene**: `input_candidate_description` must be neutral (no leaked verdict) but must carry enough factual evidence to support the verdict from the row input alone. A description with only `Autonomy: X Domains: ...` is too sparse — add the neutral artifact type and, for accepts, the concrete loop, verified against primary sources.\n- **Verify with primary sources** before relabeling away from the human verdict. Record the source (repo, paper, docs) in the audit.\n\n## Provenance Rules\n\n- Never change the human annotation result. Copy `quality_selector` and the originating `call_id` verbatim into `annotation_source`.\n- When you relabel `expected_scope_status` away from the raw `QualitySelector` mapping, keep the original `annotation_source` and explain the rubric-based reason in the audit. The provenance must still point to the same human annotation.\n- Do not invent annotations, call IDs, URLs, or dates.\n\n## Audit Report\n\nWrite `verdict_quality_dataset_audit_\u003CYYYY-MM-DD>.md` with:\n\n- Source scope (day \u002F call IDs \u002F queue) and the rubric path\n- Row count and accepted\u002Frejected distribution\n- Per-row action table: `id | action (keep\u002Frelabel\u002Fdrop\u002Fneeds_review) | expected | issue | reason`\n- Primary sources consulted for any relabel\n- The published dataset ref after publishing\n\n## Publish And Pin\n\n```bash\nuv run python - \u003C\u003C'PY'\nfrom pathlib import Path\nfrom dotenv import load_dotenv\nload_dotenv(\".env\")\nimport weave\nfrom discovery_forge.observability import weave_project_path\nfrom discovery_forge.evaluation.datasets import publish_eval_dataset, VERDICT_DATASET_NAME\n\nweave.init(weave_project_path())\nresult = publish_eval_dataset(\n    Path(\"src\u002Fdiscovery_forge\u002Fevaluation\u002Fdatasets\u002Fverdict_quality_dataset_clean_\u003CYYYY-MM-DD>.jsonl\"),\n    name=VERDICT_DATASET_NAME,\n)\nprint(result)\nPY\n```\n\nThen update `datasets.verdict_quality.ref` in `src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml` to the new digest. Preserve older versions; publish a new one rather than overwriting.\n\n## Validation\n\n```bash\nuv run pytest tests\u002Funit\u002Ftest_eval_datasets.py tests\u002Funit\u002Ftest_researcher_evaluation.py -q\nuv run python evaluate.py\n```\n\nConfirm the new ref resolves on the Weave server and that the eval links to `verdict_quality_dataset` (not an anonymous `Dataset`).\n\n## Apply Rules\n\n- Build the dataset from `research_annotation` human annotations only; do not seed labels from runnable scorers.\n- For `Neutral` annotations, ask the user how to handle each one before finalizing. Do not silently auto-assign neutral verdicts; only use a provisional label with a `needs_review` flag if the user explicitly defers.\n- Do not change the human annotation results; only transform and refine into eval rows.\n- Do not tune labels or descriptions to make a specific prompt score higher. Refine for correctness and clarity per the rubric, and record every change in the audit.\n- Keep the scorer, evaluation runner, and registry unchanged in this workflow.\n\n## Report Back\n\nReport:\n\n- source scope (day \u002F call IDs \u002F queue) and number of annotations used\n- row count and accepted\u002Frejected distribution\n- rows relabeled or dropped, with rubric reasons and primary sources\n- published dataset ref and the updated `evaluation_config.yaml` entry\n- validation results (tests, eval run, ref resolves)\n- rows left as `needs_review`\n",{"data":36,"body":37},{"name":4,"description":6},{"type":38,"children":39},"root",[40,48,63,84,97,104,117,122,128,309,315,320,463,469,721,728,733,798,804,816,890,896,957,963,975,1009,1015,1191,1209,1215,1273,1293,1299,1348,1354,1359,1404],{"type":41,"tag":42,"props":43,"children":44},"element","h1",{"id":4},[45],{"type":46,"value":47},"text","Build Verdict Dataset",{"type":41,"tag":49,"props":50,"children":51},"p",{},[52,54,61],{"type":46,"value":53},"Use this skill to create or refresh ",{"type":41,"tag":55,"props":56,"children":58},"code",{"className":57},[],[59],{"type":46,"value":60},"verdict_quality_dataset",{"type":46,"value":62}," from human annotations in Weave, then publish it as a versioned Weave Dataset and pin the ref.",{"type":41,"tag":49,"props":64,"children":65},{},[66,68,74,76,82],{"type":46,"value":67},"The coding agent queries annotated ",{"type":41,"tag":55,"props":69,"children":71},{"className":70},[],[72],{"type":46,"value":73},"research_run_\u003Ci>",{"type":46,"value":75}," calls, extracts candidate fields, maps the human verdict to a gold label, refines each row against the rubric, writes a local JSONL plus an audit report, publishes the dataset, and updates the pinned ref in ",{"type":41,"tag":55,"props":77,"children":79},{"className":78},[],[80],{"type":46,"value":81},"evaluation_config.yaml",{"type":46,"value":83},".",{"type":41,"tag":49,"props":85,"children":86},{},[87,89,95],{"type":46,"value":88},"This skill ",{"type":41,"tag":90,"props":91,"children":92},"strong",{},[93],{"type":46,"value":94},"does not change the human annotation results themselves",{"type":46,"value":96},". It only transforms annotation evidence into a clean eval dataset and records provenance.",{"type":41,"tag":98,"props":99,"children":101},"h2",{"id":100},"before-you-start",[102],{"type":46,"value":103},"Before You Start",{"type":41,"tag":49,"props":105,"children":106},{},[107,109,115],{"type":46,"value":108},"Follow ",{"type":41,"tag":55,"props":110,"children":112},{"className":111},[],[113],{"type":46,"value":114},"AGENTS.md",{"type":46,"value":116}," W&B Skills setup first. Use W&B Skills for Weave trace, feedback, annotation, and dataset access; this skill only defines the Discovery Forge dataset-building workflow.",{"type":41,"tag":49,"props":118,"children":119},{},[120],{"type":46,"value":121},"Fetch all trace and feedback evidence live from Weave through W&B Skills. Do not use W&B MCP tools, and do not add discovery-forge query wrappers.",{"type":41,"tag":98,"props":123,"children":125},{"id":124},"default-project",[126],{"type":46,"value":127},"Default Project",{"type":41,"tag":129,"props":130,"children":131},"ul",{},[132,154,189,208,219,239,249,268,279,290],{"type":41,"tag":133,"props":134,"children":135},"li",{},[136,138,144,146,152],{"type":46,"value":137},"Entity: read from ",{"type":41,"tag":55,"props":139,"children":141},{"className":140},[],[142],{"type":46,"value":143},".env",{"type":46,"value":145}," as ",{"type":41,"tag":55,"props":147,"children":149},{"className":148},[],[150],{"type":46,"value":151},"WANDB_ENTITY",{"type":46,"value":153}," (required; use your own W&B entity)",{"type":41,"tag":133,"props":155,"children":156},{},[157,159,164,165,171,173,179,181,187],{"type":46,"value":158},"Project: read from ",{"type":41,"tag":55,"props":160,"children":162},{"className":161},[],[163],{"type":46,"value":143},{"type":46,"value":145},{"type":41,"tag":55,"props":166,"children":168},{"className":167},[],[169],{"type":46,"value":170},"WANDB_PROJECT",{"type":46,"value":172}," (required; ",{"type":41,"tag":55,"props":174,"children":176},{"className":175},[],[177],{"type":46,"value":178},".env.example",{"type":46,"value":180}," uses ",{"type":41,"tag":55,"props":182,"children":184},{"className":183},[],[185],{"type":46,"value":186},"discovery-forge",{"type":46,"value":188},")",{"type":41,"tag":133,"props":190,"children":191},{},[192,194,199,200,206],{"type":46,"value":193},"API key: read from ",{"type":41,"tag":55,"props":195,"children":197},{"className":196},[],[198],{"type":46,"value":143},{"type":46,"value":145},{"type":41,"tag":55,"props":201,"children":203},{"className":202},[],[204],{"type":46,"value":205},"WANDB_API_KEY",{"type":46,"value":207}," (required)",{"type":41,"tag":133,"props":209,"children":210},{},[211,213],{"type":46,"value":212},"Annotation queue: ",{"type":41,"tag":55,"props":214,"children":216},{"className":215},[],[217],{"type":46,"value":218},"research_annotation",{"type":41,"tag":133,"props":220,"children":221},{},[222,224,229,231,237],{"type":46,"value":223},"Root trace unit: one ",{"type":41,"tag":55,"props":225,"children":227},{"className":226},[],[228],{"type":46,"value":73},{"type":46,"value":230}," \u002F ",{"type":41,"tag":55,"props":232,"children":234},{"className":233},[],[235],{"type":46,"value":236},"openai_agent_trace",{"type":46,"value":238}," call per reviewed candidate",{"type":41,"tag":133,"props":240,"children":241},{},[242,244],{"type":46,"value":243},"Dataset name: ",{"type":41,"tag":55,"props":245,"children":247},{"className":246},[],[248],{"type":46,"value":60},{"type":41,"tag":133,"props":250,"children":251},{},[252,254,260,262],{"type":46,"value":253},"Pinned ref: ",{"type":41,"tag":55,"props":255,"children":257},{"className":256},[],[258],{"type":46,"value":259},"src\u002Fdiscovery_forge\u002Fevaluation\u002Fevaluation_config.yaml",{"type":46,"value":261}," ",{"type":41,"tag":55,"props":263,"children":265},{"className":264},[],[266],{"type":46,"value":267},"datasets.verdict_quality.ref",{"type":41,"tag":133,"props":269,"children":270},{},[271,273],{"type":46,"value":272},"Publish helper: ",{"type":41,"tag":55,"props":274,"children":276},{"className":275},[],[277],{"type":46,"value":278},"discovery_forge.evaluation.datasets.publish_eval_dataset",{"type":41,"tag":133,"props":280,"children":281},{},[282,284],{"type":46,"value":283},"Labeling standard: ",{"type":41,"tag":55,"props":285,"children":287},{"className":286},[],[288],{"type":46,"value":289},"src\u002Fdiscovery_forge\u002Fevaluation\u002Fverdict_dataset_rubric.md",{"type":41,"tag":133,"props":291,"children":292},{},[293,295,301,303],{"type":46,"value":294},"Local outputs: ",{"type":41,"tag":55,"props":296,"children":298},{"className":297},[],[299],{"type":46,"value":300},"src\u002Fdiscovery_forge\u002Fevaluation\u002Fdatasets\u002Fverdict_quality_dataset_clean_\u003CYYYY-MM-DD>.jsonl",{"type":46,"value":302}," and ",{"type":41,"tag":55,"props":304,"children":306},{"className":305},[],[307],{"type":46,"value":308},"verdict_quality_dataset_audit_\u003CYYYY-MM-DD>.md",{"type":41,"tag":98,"props":310,"children":312},{"id":311},"row-schema",[313],{"type":46,"value":314},"Row Schema",{"type":41,"tag":49,"props":316,"children":317},{},[318],{"type":46,"value":319},"Each dataset row must contain:",{"type":41,"tag":129,"props":321,"children":322},{},[323,334,345,356,367,394,434,445],{"type":41,"tag":133,"props":324,"children":325},{},[326,332],{"type":41,"tag":55,"props":327,"children":329},{"className":328},[],[330],{"type":46,"value":331},"id",{"type":46,"value":333},": stable slug for the candidate (kebab-case of the tool name)",{"type":41,"tag":133,"props":335,"children":336},{},[337,343],{"type":41,"tag":55,"props":338,"children":340},{"className":339},[],[341],{"type":46,"value":342},"input_tool_name",{"type":46,"value":344},": candidate name",{"type":41,"tag":133,"props":346,"children":347},{},[348,354],{"type":41,"tag":55,"props":349,"children":351},{"className":350},[],[352],{"type":46,"value":353},"input_candidate_url",{"type":46,"value":355},": candidate primary URL",{"type":41,"tag":133,"props":357,"children":358},{},[359,365],{"type":41,"tag":55,"props":360,"children":362},{"className":361},[],[363],{"type":46,"value":364},"input_candidate_description",{"type":46,"value":366},": neutral candidate description (see hygiene rules)",{"type":41,"tag":133,"props":368,"children":369},{},[370,376,378,384,386,392],{"type":41,"tag":55,"props":371,"children":373},{"className":372},[],[374],{"type":46,"value":375},"expected_scope_status",{"type":46,"value":377},": ",{"type":41,"tag":55,"props":379,"children":381},{"className":380},[],[382],{"type":46,"value":383},"accepted",{"type":46,"value":385}," or ",{"type":41,"tag":55,"props":387,"children":389},{"className":388},[],[390],{"type":46,"value":391},"rejected",{"type":46,"value":393}," (the only field the scorer reads)",{"type":41,"tag":133,"props":395,"children":396},{},[397,403,405,411,413,419,420,426,428],{"type":41,"tag":55,"props":398,"children":400},{"className":399},[],[401],{"type":46,"value":402},"expected_issue_category",{"type":46,"value":404},": for rejects, one of ",{"type":41,"tag":55,"props":406,"children":408},{"className":407},[],[409],{"type":46,"value":410},"out_of_scope",{"type":46,"value":412},", ",{"type":41,"tag":55,"props":414,"children":416},{"className":415},[],[417],{"type":46,"value":418},"missing_url",{"type":46,"value":412},{"type":41,"tag":55,"props":421,"children":423},{"className":422},[],[424],{"type":46,"value":425},"duplicate_known_tool",{"type":46,"value":427},"; otherwise ",{"type":41,"tag":55,"props":429,"children":431},{"className":430},[],[432],{"type":46,"value":433},"null",{"type":41,"tag":133,"props":435,"children":436},{},[437,443],{"type":41,"tag":55,"props":438,"children":440},{"className":439},[],[441],{"type":46,"value":442},"label_reason",{"type":46,"value":444},": human-readable rationale (not scored)",{"type":41,"tag":133,"props":446,"children":447},{},[448,454,455,461],{"type":41,"tag":55,"props":449,"children":451},{"className":450},[],[452],{"type":46,"value":453},"annotation_source",{"type":46,"value":377},{"type":41,"tag":55,"props":456,"children":458},{"className":457},[],[459],{"type":46,"value":460},"{queue_name, call_id, quality_selector}",{"type":46,"value":462}," provenance — copy verbatim from the source annotation, never invent or alter",{"type":41,"tag":98,"props":464,"children":466},{"id":465},"workflow",[467],{"type":46,"value":468},"Workflow",{"type":41,"tag":470,"props":471,"children":472},"ol",{},[473,493,504,531,544,653,693,698,716],{"type":41,"tag":133,"props":474,"children":475},{},[476,478,484,486,491],{"type":46,"value":477},"Identify the source scope: a set of ",{"type":41,"tag":55,"props":479,"children":481},{"className":480},[],[482],{"type":46,"value":483},"research_run",{"type":46,"value":485}," call IDs (from the user or a Weave UI link), a run day, or \"all reviewed items in ",{"type":41,"tag":55,"props":487,"children":489},{"className":488},[],[490],{"type":46,"value":218},{"type":46,"value":492},"\". Prefer an explicit call-ID list or run day over \"every historical annotation\". Get call IDs from Weave, not from local files.",{"type":41,"tag":133,"props":494,"children":495},{},[496,498,503],{"type":46,"value":497},"Use W&B Skills to fetch the annotated root calls and feedback ",{"type":41,"tag":90,"props":499,"children":500},{},[501],{"type":46,"value":502},"from Weave",{"type":46,"value":83},{"type":41,"tag":133,"props":505,"children":506},{},[507,509,515,516,522,524,529],{"type":46,"value":508},"For each call, separate human annotations (",{"type":41,"tag":55,"props":510,"children":512},{"className":511},[],[513],{"type":46,"value":514},"wandb.annotation.QualitySelector",{"type":46,"value":412},{"type":41,"tag":55,"props":517,"children":519},{"className":518},[],[520],{"type":46,"value":521},"wandb.annotation.QualityReviewer",{"type":46,"value":523},") from runnable scorer feedback. Use only ",{"type":41,"tag":55,"props":525,"children":527},{"className":526},[],[528],{"type":46,"value":218},{"type":46,"value":530}," human annotations as the verdict seed; do not seed labels from runnable scorers.",{"type":41,"tag":133,"props":532,"children":533},{},[534,536,542],{"type":46,"value":535},"Inspect one call first to learn the exact ",{"type":41,"tag":55,"props":537,"children":539},{"className":538},[],[540],{"type":46,"value":541},"output",{"type":46,"value":543}," shape, then extract the candidate name, primary URL, and a pre-verdict description plus the human reviewer's rationale.",{"type":41,"tag":133,"props":545,"children":546},{},[547,549,555,557],{"type":46,"value":548},"Map the human ",{"type":41,"tag":55,"props":550,"children":552},{"className":551},[],[553],{"type":46,"value":554},"QualitySelector",{"type":46,"value":556}," to a draft gold label:\n",{"type":41,"tag":129,"props":558,"children":559},{},[560,576,591],{"type":41,"tag":133,"props":561,"children":562},{},[563,569,571],{"type":41,"tag":55,"props":564,"children":566},{"className":565},[],[567],{"type":46,"value":568},"Good",{"type":46,"value":570}," -> ",{"type":41,"tag":55,"props":572,"children":574},{"className":573},[],[575],{"type":46,"value":383},{"type":41,"tag":133,"props":577,"children":578},{},[579,585,586],{"type":41,"tag":55,"props":580,"children":582},{"className":581},[],[583],{"type":46,"value":584},"Bad",{"type":46,"value":570},{"type":41,"tag":55,"props":587,"children":589},{"className":588},[],[590],{"type":46,"value":391},{"type":41,"tag":133,"props":592,"children":593},{},[594,600,601,606,608,614,616,621,623,629,630,635,637,643,645,651],{"type":41,"tag":55,"props":595,"children":597},{"className":596},[],[598],{"type":46,"value":599},"Neutral",{"type":46,"value":570},{"type":41,"tag":90,"props":602,"children":603},{},[604],{"type":46,"value":605},"ask the user how to handle it.",{"type":46,"value":607}," Do not silently auto-assign. Present each neutral candidate with its ",{"type":41,"tag":55,"props":609,"children":611},{"className":610},[],[612],{"type":46,"value":613},"QualityReviewer",{"type":46,"value":615}," rationale and a recommended verdict (",{"type":41,"tag":55,"props":617,"children":619},{"className":618},[],[620],{"type":46,"value":383},{"type":46,"value":622}," with ",{"type":41,"tag":55,"props":624,"children":626},{"className":625},[],[627],{"type":46,"value":628},"key_limitations",{"type":46,"value":412},{"type":41,"tag":55,"props":631,"children":633},{"className":632},[],[634],{"type":46,"value":391},{"type":46,"value":636},", or ",{"type":41,"tag":55,"props":638,"children":640},{"className":639},[],[641],{"type":46,"value":642},"drop",{"type":46,"value":644},"), then let the user decide per row or give a blanket rule. Only fall back to a provisional label + ",{"type":41,"tag":55,"props":646,"children":648},{"className":647},[],[649],{"type":46,"value":650},"needs_review",{"type":46,"value":652}," flag if the user explicitly defers or says to proceed without them.",{"type":41,"tag":133,"props":654,"children":655},{},[656,658,664,666,672,673,679,680,685,686,691],{"type":46,"value":657},"Refine each row against ",{"type":41,"tag":55,"props":659,"children":661},{"className":660},[],[662],{"type":46,"value":663},"verdict_dataset_rubric.md",{"type":46,"value":665}," (see Labeling and Hygiene). Record ",{"type":41,"tag":55,"props":667,"children":669},{"className":668},[],[670],{"type":46,"value":671},"keep",{"type":46,"value":230},{"type":41,"tag":55,"props":674,"children":676},{"className":675},[],[677],{"type":46,"value":678},"relabel",{"type":46,"value":230},{"type":41,"tag":55,"props":681,"children":683},{"className":682},[],[684],{"type":46,"value":642},{"type":46,"value":230},{"type":41,"tag":55,"props":687,"children":689},{"className":688},[],[690],{"type":46,"value":650},{"type":46,"value":692}," with a reason.",{"type":41,"tag":133,"props":694,"children":695},{},[696],{"type":46,"value":697},"Write the clean JSONL and the audit report.",{"type":41,"tag":133,"props":699,"children":700},{},[701,703,708,710,715],{"type":46,"value":702},"Publish the dataset and update ",{"type":41,"tag":55,"props":704,"children":706},{"className":705},[],[707],{"type":46,"value":267},{"type":46,"value":709}," in ",{"type":41,"tag":55,"props":711,"children":713},{"className":712},[],[714],{"type":46,"value":259},{"type":46,"value":83},{"type":41,"tag":133,"props":717,"children":718},{},[719],{"type":46,"value":720},"Validate.",{"type":41,"tag":722,"props":723,"children":725},"h3",{"id":724},"evidence-selection",[726],{"type":46,"value":727},"Evidence Selection",{"type":41,"tag":49,"props":729,"children":730},{},[731],{"type":46,"value":732},"Use W&B Skills to fetch and inspect Weave evidence. This skill only defines which Discovery Forge evidence matters:",{"type":41,"tag":129,"props":734,"children":735},{},[736,753,758,783,788,793],{"type":41,"tag":133,"props":737,"children":738},{},[739,740,745,746,751],{"type":46,"value":223},{"type":41,"tag":55,"props":741,"children":743},{"className":742},[],[744],{"type":46,"value":73},{"type":46,"value":230},{"type":41,"tag":55,"props":747,"children":749},{"className":748},[],[750],{"type":46,"value":236},{"type":46,"value":752}," call per reviewed candidate.",{"type":41,"tag":133,"props":754,"children":755},{},[756],{"type":46,"value":757},"Inspect only calls in the requested source scope.",{"type":41,"tag":133,"props":759,"children":760},{},[761,763,768,770,775,776,781],{"type":46,"value":762},"Keep only human annotations from ",{"type":41,"tag":55,"props":764,"children":766},{"className":765},[],[767],{"type":46,"value":218},{"type":46,"value":769}," (",{"type":41,"tag":55,"props":771,"children":773},{"className":772},[],[774],{"type":46,"value":554},{"type":46,"value":412},{"type":41,"tag":55,"props":777,"children":779},{"className":778},[],[780],{"type":46,"value":613},{"type":46,"value":782},") as verdict seeds.",{"type":41,"tag":133,"props":784,"children":785},{},[786],{"type":46,"value":787},"Do not seed labels from runnable scorer feedback.",{"type":41,"tag":133,"props":789,"children":790},{},[791],{"type":46,"value":792},"Deduplicate annotations by feedback ID.",{"type":41,"tag":133,"props":794,"children":795},{},[796],{"type":46,"value":797},"Inspect one representative call first to learn where candidate name, primary URL, description, reviewer rationale, and annotation payload live.",{"type":41,"tag":98,"props":799,"children":801},{"id":800},"labeling-and-hygiene",[802],{"type":46,"value":803},"Labeling and Hygiene",{"type":41,"tag":49,"props":805,"children":806},{},[807,809,814],{"type":46,"value":808},"Apply ",{"type":41,"tag":55,"props":810,"children":812},{"className":811},[],[813],{"type":46,"value":663},{"type":46,"value":815},". Key rules:",{"type":41,"tag":129,"props":817,"children":818},{},[819,829,839,856,880],{"type":41,"tag":133,"props":820,"children":821},{},[822,827],{"type":41,"tag":90,"props":823,"children":824},{},[825],{"type":46,"value":826},"Accept",{"type":46,"value":828}," when the candidate itself runs a loop: task -> evaluation -> feedback\u002Fstate -> revision of its own artifact.",{"type":41,"tag":133,"props":830,"children":831},{},[832,837],{"type":41,"tag":90,"props":833,"children":834},{},[835],{"type":46,"value":836},"Reject",{"type":46,"value":838}," lists\u002Fsurveys\u002Fcookbooks\u002Fguides, GUI\u002Fcomputer-use frameworks, testing\u002Fevaluation-only gates, repository-automation hosts, memory-only components, generic frameworks, and weak-evidence candidates.",{"type":41,"tag":133,"props":840,"children":841},{},[842,847,849,854],{"type":41,"tag":90,"props":843,"children":844},{},[845],{"type":46,"value":846},"Metadata vs scope",{"type":46,"value":848},": missing stars\u002Flicense\u002Fdates\u002FGitHub fetch is a profile limitation, not a scope reject. Do not encode metadata gaps as ",{"type":41,"tag":55,"props":850,"children":852},{"className":851},[],[853],{"type":46,"value":418},{"type":46,"value":855}," when a primary URL exists.",{"type":41,"tag":133,"props":857,"children":858},{},[859,864,865,870,872,878],{"type":41,"tag":90,"props":860,"children":861},{},[862],{"type":46,"value":863},"Input hygiene",{"type":46,"value":377},{"type":41,"tag":55,"props":866,"children":868},{"className":867},[],[869],{"type":46,"value":364},{"type":46,"value":871}," must be neutral (no leaked verdict) but must carry enough factual evidence to support the verdict from the row input alone. A description with only ",{"type":41,"tag":55,"props":873,"children":875},{"className":874},[],[876],{"type":46,"value":877},"Autonomy: X Domains: ...",{"type":46,"value":879}," is too sparse — add the neutral artifact type and, for accepts, the concrete loop, verified against primary sources.",{"type":41,"tag":133,"props":881,"children":882},{},[883,888],{"type":41,"tag":90,"props":884,"children":885},{},[886],{"type":46,"value":887},"Verify with primary sources",{"type":46,"value":889}," before relabeling away from the human verdict. Record the source (repo, paper, docs) in the audit.",{"type":41,"tag":98,"props":891,"children":893},{"id":892},"provenance-rules",[894],{"type":46,"value":895},"Provenance Rules",{"type":41,"tag":129,"props":897,"children":898},{},[899,926,952],{"type":41,"tag":133,"props":900,"children":901},{},[902,904,910,912,918,920,925],{"type":46,"value":903},"Never change the human annotation result. Copy ",{"type":41,"tag":55,"props":905,"children":907},{"className":906},[],[908],{"type":46,"value":909},"quality_selector",{"type":46,"value":911}," and the originating ",{"type":41,"tag":55,"props":913,"children":915},{"className":914},[],[916],{"type":46,"value":917},"call_id",{"type":46,"value":919}," verbatim into ",{"type":41,"tag":55,"props":921,"children":923},{"className":922},[],[924],{"type":46,"value":453},{"type":46,"value":83},{"type":41,"tag":133,"props":927,"children":928},{},[929,931,936,938,943,945,950],{"type":46,"value":930},"When you relabel ",{"type":41,"tag":55,"props":932,"children":934},{"className":933},[],[935],{"type":46,"value":375},{"type":46,"value":937}," away from the raw ",{"type":41,"tag":55,"props":939,"children":941},{"className":940},[],[942],{"type":46,"value":554},{"type":46,"value":944}," mapping, keep the original ",{"type":41,"tag":55,"props":946,"children":948},{"className":947},[],[949],{"type":46,"value":453},{"type":46,"value":951}," and explain the rubric-based reason in the audit. The provenance must still point to the same human annotation.",{"type":41,"tag":133,"props":953,"children":954},{},[955],{"type":46,"value":956},"Do not invent annotations, call IDs, URLs, or dates.",{"type":41,"tag":98,"props":958,"children":960},{"id":959},"audit-report",[961],{"type":46,"value":962},"Audit Report",{"type":41,"tag":49,"props":964,"children":965},{},[966,968,973],{"type":46,"value":967},"Write ",{"type":41,"tag":55,"props":969,"children":971},{"className":970},[],[972],{"type":46,"value":308},{"type":46,"value":974}," with:",{"type":41,"tag":129,"props":976,"children":977},{},[978,983,988,999,1004],{"type":41,"tag":133,"props":979,"children":980},{},[981],{"type":46,"value":982},"Source scope (day \u002F call IDs \u002F queue) and the rubric path",{"type":41,"tag":133,"props":984,"children":985},{},[986],{"type":46,"value":987},"Row count and accepted\u002Frejected distribution",{"type":41,"tag":133,"props":989,"children":990},{},[991,993],{"type":46,"value":992},"Per-row action table: ",{"type":41,"tag":55,"props":994,"children":996},{"className":995},[],[997],{"type":46,"value":998},"id | action (keep\u002Frelabel\u002Fdrop\u002Fneeds_review) | expected | issue | reason",{"type":41,"tag":133,"props":1000,"children":1001},{},[1002],{"type":46,"value":1003},"Primary sources consulted for any relabel",{"type":41,"tag":133,"props":1005,"children":1006},{},[1007],{"type":46,"value":1008},"The published dataset ref after publishing",{"type":41,"tag":98,"props":1010,"children":1012},{"id":1011},"publish-and-pin",[1013],{"type":46,"value":1014},"Publish And Pin",{"type":41,"tag":1016,"props":1017,"children":1022},"pre",{"className":1018,"code":1019,"language":1020,"meta":1021,"style":1021},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv run python - \u003C\u003C'PY'\nfrom pathlib import Path\nfrom dotenv import load_dotenv\nload_dotenv(\".env\")\nimport weave\nfrom discovery_forge.observability import weave_project_path\nfrom discovery_forge.evaluation.datasets import publish_eval_dataset, VERDICT_DATASET_NAME\n\nweave.init(weave_project_path())\nresult = publish_eval_dataset(\n    Path(\"src\u002Fdiscovery_forge\u002Fevaluation\u002Fdatasets\u002Fverdict_quality_dataset_clean_\u003CYYYY-MM-DD>.jsonl\"),\n    name=VERDICT_DATASET_NAME,\n)\nprint(result)\nPY\n","bash","",[1023],{"type":41,"tag":55,"props":1024,"children":1025},{"__ignoreMap":1021},[1026,1065,1073,1082,1091,1100,1109,1118,1128,1137,1146,1155,1164,1173,1182],{"type":41,"tag":1027,"props":1028,"children":1031},"span",{"class":1029,"line":1030},"line",1,[1032,1038,1044,1049,1054,1060],{"type":41,"tag":1027,"props":1033,"children":1035},{"style":1034},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1036],{"type":46,"value":1037},"uv",{"type":41,"tag":1027,"props":1039,"children":1041},{"style":1040},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1042],{"type":46,"value":1043}," run",{"type":41,"tag":1027,"props":1045,"children":1046},{"style":1040},[1047],{"type":46,"value":1048}," python",{"type":41,"tag":1027,"props":1050,"children":1051},{"style":1040},[1052],{"type":46,"value":1053}," -",{"type":41,"tag":1027,"props":1055,"children":1057},{"style":1056},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1058],{"type":46,"value":1059}," \u003C\u003C",{"type":41,"tag":1027,"props":1061,"children":1062},{"style":1056},[1063],{"type":46,"value":1064},"'PY'\n",{"type":41,"tag":1027,"props":1066,"children":1067},{"class":1029,"line":25},[1068],{"type":41,"tag":1027,"props":1069,"children":1070},{"style":1040},[1071],{"type":46,"value":1072},"from pathlib import Path\n",{"type":41,"tag":1027,"props":1074,"children":1076},{"class":1029,"line":1075},3,[1077],{"type":41,"tag":1027,"props":1078,"children":1079},{"style":1040},[1080],{"type":46,"value":1081},"from dotenv import load_dotenv\n",{"type":41,"tag":1027,"props":1083,"children":1085},{"class":1029,"line":1084},4,[1086],{"type":41,"tag":1027,"props":1087,"children":1088},{"style":1040},[1089],{"type":46,"value":1090},"load_dotenv(\".env\")\n",{"type":41,"tag":1027,"props":1092,"children":1094},{"class":1029,"line":1093},5,[1095],{"type":41,"tag":1027,"props":1096,"children":1097},{"style":1040},[1098],{"type":46,"value":1099},"import weave\n",{"type":41,"tag":1027,"props":1101,"children":1103},{"class":1029,"line":1102},6,[1104],{"type":41,"tag":1027,"props":1105,"children":1106},{"style":1040},[1107],{"type":46,"value":1108},"from discovery_forge.observability import weave_project_path\n",{"type":41,"tag":1027,"props":1110,"children":1112},{"class":1029,"line":1111},7,[1113],{"type":41,"tag":1027,"props":1114,"children":1115},{"style":1040},[1116],{"type":46,"value":1117},"from discovery_forge.evaluation.datasets import publish_eval_dataset, VERDICT_DATASET_NAME\n",{"type":41,"tag":1027,"props":1119,"children":1121},{"class":1029,"line":1120},8,[1122],{"type":41,"tag":1027,"props":1123,"children":1125},{"emptyLinePlaceholder":1124},true,[1126],{"type":46,"value":1127},"\n",{"type":41,"tag":1027,"props":1129,"children":1131},{"class":1029,"line":1130},9,[1132],{"type":41,"tag":1027,"props":1133,"children":1134},{"style":1040},[1135],{"type":46,"value":1136},"weave.init(weave_project_path())\n",{"type":41,"tag":1027,"props":1138,"children":1140},{"class":1029,"line":1139},10,[1141],{"type":41,"tag":1027,"props":1142,"children":1143},{"style":1040},[1144],{"type":46,"value":1145},"result = publish_eval_dataset(\n",{"type":41,"tag":1027,"props":1147,"children":1149},{"class":1029,"line":1148},11,[1150],{"type":41,"tag":1027,"props":1151,"children":1152},{"style":1040},[1153],{"type":46,"value":1154},"    Path(\"src\u002Fdiscovery_forge\u002Fevaluation\u002Fdatasets\u002Fverdict_quality_dataset_clean_\u003CYYYY-MM-DD>.jsonl\"),\n",{"type":41,"tag":1027,"props":1156,"children":1158},{"class":1029,"line":1157},12,[1159],{"type":41,"tag":1027,"props":1160,"children":1161},{"style":1040},[1162],{"type":46,"value":1163},"    name=VERDICT_DATASET_NAME,\n",{"type":41,"tag":1027,"props":1165,"children":1167},{"class":1029,"line":1166},13,[1168],{"type":41,"tag":1027,"props":1169,"children":1170},{"style":1040},[1171],{"type":46,"value":1172},")\n",{"type":41,"tag":1027,"props":1174,"children":1176},{"class":1029,"line":1175},14,[1177],{"type":41,"tag":1027,"props":1178,"children":1179},{"style":1040},[1180],{"type":46,"value":1181},"print(result)\n",{"type":41,"tag":1027,"props":1183,"children":1185},{"class":1029,"line":1184},15,[1186],{"type":41,"tag":1027,"props":1187,"children":1188},{"style":1056},[1189],{"type":46,"value":1190},"PY\n",{"type":41,"tag":49,"props":1192,"children":1193},{},[1194,1196,1201,1202,1207],{"type":46,"value":1195},"Then update ",{"type":41,"tag":55,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":46,"value":267},{"type":46,"value":709},{"type":41,"tag":55,"props":1203,"children":1205},{"className":1204},[],[1206],{"type":46,"value":259},{"type":46,"value":1208}," to the new digest. Preserve older versions; publish a new one rather than overwriting.",{"type":41,"tag":98,"props":1210,"children":1212},{"id":1211},"validation",[1213],{"type":46,"value":1214},"Validation",{"type":41,"tag":1016,"props":1216,"children":1218},{"className":1018,"code":1217,"language":1020,"meta":1021,"style":1021},"uv run pytest tests\u002Funit\u002Ftest_eval_datasets.py tests\u002Funit\u002Ftest_researcher_evaluation.py -q\nuv run python evaluate.py\n",[1219],{"type":41,"tag":55,"props":1220,"children":1221},{"__ignoreMap":1021},[1222,1253],{"type":41,"tag":1027,"props":1223,"children":1224},{"class":1029,"line":1030},[1225,1229,1233,1238,1243,1248],{"type":41,"tag":1027,"props":1226,"children":1227},{"style":1034},[1228],{"type":46,"value":1037},{"type":41,"tag":1027,"props":1230,"children":1231},{"style":1040},[1232],{"type":46,"value":1043},{"type":41,"tag":1027,"props":1234,"children":1235},{"style":1040},[1236],{"type":46,"value":1237}," pytest",{"type":41,"tag":1027,"props":1239,"children":1240},{"style":1040},[1241],{"type":46,"value":1242}," tests\u002Funit\u002Ftest_eval_datasets.py",{"type":41,"tag":1027,"props":1244,"children":1245},{"style":1040},[1246],{"type":46,"value":1247}," tests\u002Funit\u002Ftest_researcher_evaluation.py",{"type":41,"tag":1027,"props":1249,"children":1250},{"style":1040},[1251],{"type":46,"value":1252}," -q\n",{"type":41,"tag":1027,"props":1254,"children":1255},{"class":1029,"line":25},[1256,1260,1264,1268],{"type":41,"tag":1027,"props":1257,"children":1258},{"style":1034},[1259],{"type":46,"value":1037},{"type":41,"tag":1027,"props":1261,"children":1262},{"style":1040},[1263],{"type":46,"value":1043},{"type":41,"tag":1027,"props":1265,"children":1266},{"style":1040},[1267],{"type":46,"value":1048},{"type":41,"tag":1027,"props":1269,"children":1270},{"style":1040},[1271],{"type":46,"value":1272}," evaluate.py\n",{"type":41,"tag":49,"props":1274,"children":1275},{},[1276,1278,1283,1285,1291],{"type":46,"value":1277},"Confirm the new ref resolves on the Weave server and that the eval links to ",{"type":41,"tag":55,"props":1279,"children":1281},{"className":1280},[],[1282],{"type":46,"value":60},{"type":46,"value":1284}," (not an anonymous ",{"type":41,"tag":55,"props":1286,"children":1288},{"className":1287},[],[1289],{"type":46,"value":1290},"Dataset",{"type":46,"value":1292},").",{"type":41,"tag":98,"props":1294,"children":1296},{"id":1295},"apply-rules",[1297],{"type":46,"value":1298},"Apply Rules",{"type":41,"tag":129,"props":1300,"children":1301},{},[1302,1314,1333,1338,1343],{"type":41,"tag":133,"props":1303,"children":1304},{},[1305,1307,1312],{"type":46,"value":1306},"Build the dataset from ",{"type":41,"tag":55,"props":1308,"children":1310},{"className":1309},[],[1311],{"type":46,"value":218},{"type":46,"value":1313}," human annotations only; do not seed labels from runnable scorers.",{"type":41,"tag":133,"props":1315,"children":1316},{},[1317,1319,1324,1326,1331],{"type":46,"value":1318},"For ",{"type":41,"tag":55,"props":1320,"children":1322},{"className":1321},[],[1323],{"type":46,"value":599},{"type":46,"value":1325}," annotations, ask the user how to handle each one before finalizing. Do not silently auto-assign neutral verdicts; only use a provisional label with a ",{"type":41,"tag":55,"props":1327,"children":1329},{"className":1328},[],[1330],{"type":46,"value":650},{"type":46,"value":1332}," flag if the user explicitly defers.",{"type":41,"tag":133,"props":1334,"children":1335},{},[1336],{"type":46,"value":1337},"Do not change the human annotation results; only transform and refine into eval rows.",{"type":41,"tag":133,"props":1339,"children":1340},{},[1341],{"type":46,"value":1342},"Do not tune labels or descriptions to make a specific prompt score higher. Refine for correctness and clarity per the rubric, and record every change in the audit.",{"type":41,"tag":133,"props":1344,"children":1345},{},[1346],{"type":46,"value":1347},"Keep the scorer, evaluation runner, and registry unchanged in this workflow.",{"type":41,"tag":98,"props":1349,"children":1351},{"id":1350},"report-back",[1352],{"type":46,"value":1353},"Report Back",{"type":41,"tag":49,"props":1355,"children":1356},{},[1357],{"type":46,"value":1358},"Report:",{"type":41,"tag":129,"props":1360,"children":1361},{},[1362,1367,1372,1377,1389,1394],{"type":41,"tag":133,"props":1363,"children":1364},{},[1365],{"type":46,"value":1366},"source scope (day \u002F call IDs \u002F queue) and number of annotations used",{"type":41,"tag":133,"props":1368,"children":1369},{},[1370],{"type":46,"value":1371},"row count and accepted\u002Frejected distribution",{"type":41,"tag":133,"props":1373,"children":1374},{},[1375],{"type":46,"value":1376},"rows relabeled or dropped, with rubric reasons and primary sources",{"type":41,"tag":133,"props":1378,"children":1379},{},[1380,1382,1387],{"type":46,"value":1381},"published dataset ref and the updated ",{"type":41,"tag":55,"props":1383,"children":1385},{"className":1384},[],[1386],{"type":46,"value":81},{"type":46,"value":1388}," entry",{"type":41,"tag":133,"props":1390,"children":1391},{},[1392],{"type":46,"value":1393},"validation results (tests, eval run, ref resolves)",{"type":41,"tag":133,"props":1395,"children":1396},{},[1397,1399],{"type":46,"value":1398},"rows left as ",{"type":41,"tag":55,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":46,"value":650},{"type":41,"tag":1405,"props":1406,"children":1407},"style",{},[1408],{"type":46,"value":1409},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1411,"total":1130},[1412,1431,1451,1464,1477,1487,1500,1507,1518],{"slug":1413,"name":1413,"fn":1414,"description":1415,"org":1416,"tags":1417,"stars":1428,"repoUrl":1429,"updatedAt":1430},"weave-instrument","instrument LLM codebases with Weave","Add Weave (Weights & Biases) observability to an LLM or agent codebase. This covers calling `weave.init()`, setting up authentication, and choosing between OTEL auto-instrumentation and the explicit Session SDK agent-logging APIs (Turn, LLM, Tool, SubAgent), based on the libraries the code already uses. Works for Python and TypeScript\u002FNode. Use this whenever the user wants to instrument, trace, or add observability, logging, or monitoring to an agent, chatbot, RAG pipeline, or LLM app. This includes phrasings like \"log my agent to weave\", \"add agent tracing\", \"get my agent into the Weave Agents tab\", \"instrument this with the weave session sdk\", \"trace my tool calls\", or \"set up weave logging\", even when the user does not name a specific API.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1418,1421,1424,1427],{"name":1419,"slug":1420,"type":16},"Agents","agents",{"name":1422,"slug":1423,"type":16},"LLM","llm",{"name":1425,"slug":1426,"type":16},"Observability","observability",{"name":9,"slug":18,"type":16},1104,"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fweave","2026-07-17T06:07:25.81154",{"slug":1432,"name":1432,"fn":1433,"description":1434,"org":1435,"tags":1436,"stars":1448,"repoUrl":1449,"updatedAt":1450},"wandb-primary","manage Weights & Biases projects and runs","Primary W&B skill for broad or mixed Weights & Biases work: project overviews, W&B runs and artifacts, Weave traces and evaluations, Reports, and Launch workflows. Use when the task spans multiple W&B surfaces or the user asks generally what is happening in a W&B project.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1437,1438,1439,1442,1445],{"name":23,"slug":24,"type":16},{"name":20,"slug":21,"type":16},{"name":1440,"slug":1441,"type":16},"MLOps","mlops",{"name":1443,"slug":1444,"type":16},"Monitoring","monitoring",{"name":1446,"slug":1447,"type":16},"Tracing","tracing",60,"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fskills","2026-07-17T06:06:35.194929",{"slug":1452,"name":1452,"fn":1453,"description":1454,"org":1455,"tags":1456,"stars":1130,"repoUrl":1462,"updatedAt":1463},"weave-config","configure Weave project and settings","This skill should be used when the user wants to \"configure weave\", \"set weave project\", \"change weave project\", \"set wandb api key\", \"update weave settings\", \"show weave config\", \"change weave configuration\", \"restart the weave daemon\", \"apply weave config changes\", \"restart weave to pick up changes\", or needs to read or update any Weave Claude Code plugin settings.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1457,1460,1461],{"name":1458,"slug":1459,"type":16},"Configuration","configuration",{"name":1425,"slug":1426,"type":16},{"name":9,"slug":18,"type":16},"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fweave-claude-code","2026-07-17T06:07:38.107392",{"slug":1465,"name":1465,"fn":1466,"description":1467,"org":1468,"tags":1469,"stars":1130,"repoUrl":1462,"updatedAt":1476},"weave-install","install and configure Weave observability","This skill should be used when the user wants to \"install the weave plugin\", \"set up weave\", \"install weave-claude-code\", \"configure weave for the first time\", \"get started with weave tracing\", or needs to complete the initial setup of the Weave Claude Code plugin including dependency installation and project configuration.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1470,1473,1474,1475],{"name":1471,"slug":1472,"type":16},"Claude Code","claude-code",{"name":1458,"slug":1459,"type":16},{"name":1425,"slug":1426,"type":16},{"name":9,"slug":18,"type":16},"2026-07-17T06:06:35.660384",{"slug":1478,"name":1478,"fn":1479,"description":1480,"org":1481,"tags":1482,"stars":1130,"repoUrl":1462,"updatedAt":1486},"weave-status","verify Weave plugin status and health","This skill should be used when the user wants to \"check weave status\", \"verify the weave plugin is running\", \"see if weave is set up correctly\", \"check weave configuration\", \"is weave working\", \"weave is running an older config\", \"the daemon is on an old config\", or needs to diagnose why Claude Code sessions are not appearing in Weave.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1483,1484,1485],{"name":1443,"slug":1444,"type":16},{"name":1425,"slug":1426,"type":16},{"name":9,"slug":18,"type":16},"2026-07-17T06:07:58.961737",{"slug":1488,"name":1488,"fn":1489,"description":1490,"org":1491,"tags":1492,"stars":25,"repoUrl":26,"updatedAt":1499},"annotation-improvement","improve prompts using Weave research traces","Guides coding agents through Discovery Forge prompt improvement after selecting the `wandb-primary` skill. Use `wandb-primary` to fetch W&B Weave research_run traces, human annotations, runnable feedback, and evaluations, then use this workflow when improving researcher.md from annotation queues, reviewed research traces, or human feedback.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1493,1494,1495,1498],{"name":1419,"slug":1420,"type":16},{"name":20,"slug":21,"type":16},{"name":1496,"slug":1497,"type":16},"Prompt Engineering","prompt-engineering",{"name":9,"slug":18,"type":16},"2026-07-17T06:06:36.008062",{"slug":4,"name":4,"fn":5,"description":6,"org":1501,"tags":1502,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1503,1504,1505,1506],{"name":23,"slug":24,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},{"slug":1508,"name":1508,"fn":1509,"description":1510,"org":1511,"tags":1512,"stars":25,"repoUrl":26,"updatedAt":1517},"offline-eval-improvement","improve prompts using offline evaluation baselines","Guides coding agents through Discovery Forge prompt improvement from a specified Weave offline evaluation baseline. Use `wandb-primary` to fetch evaluation results, failed eval rows, dataset refs, prompt refs, and scorer evidence, then use this workflow when improving researcher.md from failed evaluation rows, comparing eval runs, or iterating on a fixed dataset.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1513,1514,1515,1516],{"name":1419,"slug":1420,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},"2026-07-17T06:07:26.501045",{"slug":1519,"name":1519,"fn":1520,"description":1521,"org":1522,"tags":1523,"stars":25,"repoUrl":1530,"updatedAt":1531},"weave-integration","integrate Weave into applications","Comprehensive skill for adding W&B Weave to existing applications. Covers trace-first instrumentation, evaluation only after trace verification, documentation-first implementation, CLI-based Weave data access, and validation workflows. Activate this skill only when the user explicitly mentions the skill by name, such as `weave-integration`, `@skills weave-integration`, or another direct skill reference. Do not auto-trigger from generic Weave or W&B questions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1524,1527,1528,1529],{"name":1525,"slug":1526,"type":16},"Instrumentation","instrumentation",{"name":1425,"slug":1426,"type":16},{"name":1446,"slug":1447,"type":16},{"name":9,"slug":18,"type":16},"https:\u002F\u002Fgithub.com\u002Fwandb\u002Fweave-integration-skills","2026-07-17T06:07:59.666449",{"items":1533,"total":1075},[1534,1541,1548],{"slug":1488,"name":1488,"fn":1489,"description":1490,"org":1535,"tags":1536,"stars":25,"repoUrl":26,"updatedAt":1499},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1537,1538,1539,1540],{"name":1419,"slug":1420,"type":16},{"name":20,"slug":21,"type":16},{"name":1496,"slug":1497,"type":16},{"name":9,"slug":18,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":1542,"tags":1543,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1544,1545,1546,1547],{"name":23,"slug":24,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16},{"slug":1508,"name":1508,"fn":1509,"description":1510,"org":1549,"tags":1550,"stars":25,"repoUrl":26,"updatedAt":1517},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1551,1552,1553,1554],{"name":1419,"slug":1420,"type":16},{"name":14,"slug":15,"type":16},{"name":20,"slug":21,"type":16},{"name":9,"slug":18,"type":16}]