[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-rag-eval":3,"mdc-r6vv7u-key":34,"related-repo-nvidia-rag-eval":1666,"related-org-nvidia-rag-eval":1772},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"rag-eval","benchmark RAG quality with RAGAS","Filesystem RAG benchmarks: corpus\u002F, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency\u002Fthroughput benchmarking (use rag-perf), or evals outside this repo layout.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"RAG","rag",{"name":20,"slug":21,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:27:02.591431","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Frag-eval","---\nname: rag-eval\nversion: \"2.6.0\"\ndescription: >-\n  Filesystem RAG benchmarks: corpus\u002F, train.json, evaluate_rag.py (RAGAS quality). Not for prod\n  monitoring, latency\u002Fthroughput benchmarking (use rag-perf), or evals outside this repo layout.\nlicense: Apache-2.0\ncompatibility: Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts\u002Feval (eval deps live in scripts\u002Feval\u002Fpyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai\u002Fmixtral-8x22b-instruct-v0.1).\nmetadata:\n  author: NVIDIA RAG \u003Cfoundational-rag-dev@exchange.nvidia.com>\n  github-url: \"https:\u002F\u002Fgithub.com\u002FNVIDIA-AI-Blueprints\u002Frag\"\n  endpoint-openapi-schemas:\n    - docs\u002Fapi_reference\u002Fopenapi_schema_rag_server.json\n    - docs\u002Fapi_reference\u002Fopenapi_schema_ingestor_server.json\n  argument-hint: RAGAS eval | evaluate_rag | train.json | corpus | results json | error triage | uv run --project scripts\u002Feval | enable_reranker | query_rewriting | temperature | skip_ingestion\n  tags:\n    - nvidia\n    - blueprint\n    - rag\n    - evaluation\n    - ragas\n    - benchmarking\n    - nvidia-rag-blueprint\n  languages:\n    - python\n    - shell\n  frameworks:\n    - ragas\n    - fastapi\n  domain: ai-ml\nallowed-tools: Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Write Edit\n---\n\n# On-disk RAG evaluation (`corpus\u002F` + `train.json`)\n\n## Purpose\n\nGuide agents through NVIDIA RAG Blueprint **filesystem** benchmarks: preparing `corpus\u002F` and `train.json`, running `scripts\u002Feval\u002Fevaluate_rag.py`, tuning retrieval and generation flags for **quality** comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP\u002Fstream errors, empty contexts, collection mismatch, judge API).\n\nFor **latency, throughput, and load testing**, use the **rag-perf** skill (`scripts\u002Frag-perf`, `docs\u002Fperformance-benchmarking.md`) — not this skill.\n\n## When not to use\n\nDo **not** use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the `corpus\u002F` + `train.json` layout; general ML experimentation unrelated to this evaluator; production monitoring\u002Falerting; or latency\u002Fthroughput benchmarking (use **rag-perf**).\n\n## Prerequisites\n\n- Repo cloned; **run commands from repo root** (imports and paths assume this).\n- Python **3.11+** and **uv**; eval deps: `uv sync --project scripts\u002Feval`.\n- Reachable **RAG server** and **ingestor** (defaults often `localhost:8081` \u002F `8082`).\n- **`NVIDIA_API_KEY`** for RAGAS (see [credential hygiene](references\u002Fbenchmark-execution.md#credential-hygiene-nvidia_api_key)); optional **`RAG_EVAL_JUDGE_MODEL`**.\n- Dataset roots passed to `--dataset-paths` each contain **`corpus\u002F`** and **`train.json`**.\n\n## Instructions\n\n1. **Prepare data** — Ensure each dataset directory matches the layout and `train.json` rules in [`references\u002Fdataset-and-conversion.md`](references\u002Fdataset-and-conversion.md). When sources arrive as public links (sites or dataset pages), materialize documents under `corpus\u002F`—prefer **PDF** for multimodal content so **images stay embedded**; convert CSV\u002FJSONL\u002Fetc. using the patterns there.\n2. **Run eval** — `uv run --project scripts\u002Feval python scripts\u002Feval\u002Fevaluate_rag.py` with `--dataset-paths`, `--host`, and `--port`. See [`references\u002Fbenchmark-execution.md`](references\u002Fbenchmark-execution.md) for command examples, outputs, and errors. Use [`references\u002Fevaluate-rag-cli.md`](references\u002Fevaluate-rag-cli.md) for flag-level detail.\n3. **Tune quality** — Adjust `--top_k` \u002F `--vdb_top_k`, reranker and query-rewriting toggles, and generation overrides (`--temperature`, `--top-p`, `--max-tokens`) as documented in [`references\u002Fbenchmark-execution.md`](references\u002Fbenchmark-execution.md) when comparing retrieval\u002Fgeneration configs for RAGAS scores.\n4. **Analyze results** — Use [`references\u002Fresult-analysis.md`](references\u002Fresult-analysis.md) for scripts; scan `rag_*_evaluation_summary.json` for headline RAGAS metrics.\n5. **Triage errors** — Use the [error signal table](references\u002Fbenchmark-execution.md#common-error-cases-and-signals) and the **Troubleshooting** section below.\n\n## Examples\n\n**Set API key without putting secrets in shell history (preferred patterns):** load from a gitignored env file or secrets manager; avoid committing `.env`; rotate keys if exposed. Details: [`references\u002Fbenchmark-execution.md#credential-hygiene-nvidia_api_key`](references\u002Fbenchmark-execution.md#credential-hygiene-nvidia_api_key).\n\n**Minimal eval (key already in environment):**\n\n```bash\nuv sync --project scripts\u002Feval\nuv run --project scripts\u002Feval python scripts\u002Feval\u002Fevaluate_rag.py \\\n  --dataset-paths \u002Fpath\u002Fto\u002Fmy_dataset \\\n  --host localhost \\\n  --port 8081\n```\n\n**Pretty-print summary JSON:**\n\n```bash\npython3 -m json.tool results\u002Fmy_dataset\u002Frag_my_dataset_evaluation_summary.json\n```\n\nMore examples (skip ingestion, quality sweeps): [`references\u002Fbenchmark-execution.md`](references\u002Fbenchmark-execution.md).\n\n## Limitations\n\n- Evaluator behavior is fixed to the **filesystem contract** and `evaluate_rag.py`; it does not substitute for custom offline judges or non-RAG benchmarks.\n- **Vector DB \u002F embedding** choices follow deployed ingestor and RAG env — not overridden by this CLI alone.\n- **Scores depend on** retrieval quality, judge model availability, and `NVIDIA_API_KEY`; empty contexts yield partial RAGAS metrics (see references).\n- Large procedural detail lives under **`references\u002F`** to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.\n\n## Troubleshooting\n\n| Error \u002F signal | Likely cause | What to do |\n|----------------|--------------|------------|\n| Immediate exit mentioning `NVIDIA_API_KEY` | Missing or invalid key | Set key via secure channel; see credential hygiene in [`references\u002Fbenchmark-execution.md`](references\u002Fbenchmark-execution.md). |\n| `train.json must be a JSON array` | Wrong JSON shape | Top-level array of objects; validate per [`references\u002Fdataset-and-conversion.md`](references\u002Fdataset-and-conversion.md). |\n| Fewer rows in `evaluation_data.json` than `train.json` | Per-query failures | Check stderr: network or stream JSON errors; see error table in benchmark-execution. |\n| Empty `generated_contexts` everywhere | Retrieval gap | Verify collection, ingestion, `top_k` \u002F `vdb_top_k`, and `ingestor_server_url` **without** `\u002Fv1` suffix. |\n| Ingestor 404 on upload | Bad ingestor base URL | Pass `http:\u002F\u002Fhost:port` only — code appends `\u002Fv1\u002F`. |\n\nFull signal table: [`references\u002Fbenchmark-execution.md#common-error-cases-and-signals`](references\u002Fbenchmark-execution.md#common-error-cases-and-signals).\n\n## Gotchas\n\n- **Run from repo root**: paths and imports in `scripts\u002Feval\u002Fevaluate_rag.py` assume this; a wrong directory silently breaks imports.\n- **`--ingestor_server_url`**: pass `http:\u002F\u002Fhost:port` without `\u002Fv1`—the code appends `\u002Fv1\u002F` automatically. Including `\u002Fv1` causes 404s on ingestor calls.\n- **Vector DB \u002F embedding settings**: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. `APP_VECTORSTORE_URL`, embedding model).\n- **`--model` \u002F `--llm_endpoint`**: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.\n- **Stale collections**: a previous run's ingested data persists unless you use `--force_ingestion`. Use `--collection` with a unique name when comparing quality across isolated runs.\n- **Empty context metrics**: if all `generated_contexts` are empty, RAGAS scores only `nv_accuracy` and leaves the other two metrics blank—this is not a silent success.\n\n## Source of truth\n\n| Piece | Location |\n|-------|----------|\n| Driver | `scripts\u002Feval\u002Fevaluate_rag.py` (`CORPUS_DIRECTORY` = `corpus`, `EVAL_DATA` = `train.json`) |\n| Human README (always in-repo) | `scripts\u002Feval\u002FREADME.md` |\n| Full CLI (flags, defaults) | `scripts\u002Feval\u002Fevaluate_rag.py --help`; [`references\u002Fevaluate-rag-cli.md`](references\u002Fevaluate-rag-cli.md) |\n| Dataset \u002F conversion | [`references\u002Fdataset-and-conversion.md`](references\u002Fdataset-and-conversion.md) |\n| Runs, outputs, errors | [`references\u002Fbenchmark-execution.md`](references\u002Fbenchmark-execution.md) |\n| Result analysis scripts | [`references\u002Fresult-analysis.md`](references\u002Fresult-analysis.md) |\n| Latency \u002F throughput | **rag-perf** skill, `docs\u002Fperformance-benchmarking.md` |\n\n## Agent playbook\n\n1. **Run eval** — `uv sync --project scripts\u002Feval` then `uv run --project scripts\u002Feval python scripts\u002Feval\u002Fevaluate_rag.py` with required `--dataset-paths`, `--host`, and `--port` (and env `NVIDIA_API_KEY`). Argument `--ingestor_server_url` is optional (defaults to `http:\u002F\u002Flocalhost:8082`); pass it only when overriding the ingestor endpoint.\n2. **Quality tuning** — See [`references\u002Fbenchmark-execution.md`](references\u002Fbenchmark-execution.md): `--top_k`\u002F`--vdb_top_k`, reranker and query-rewriting toggles, `--temperature`, `--top-p`, `--max-tokens`.\n3. **Data conversion** — Follow [`references\u002Fdataset-and-conversion.md`](references\u002Fdataset-and-conversion.md).\n4. **Analyze results** — [`references\u002Fresult-analysis.md`](references\u002Fresult-analysis.md); quick scan: `python3 -m json.tool results\u002F\u003Cdataset>\u002Frag_\u003Cdataset>_evaluation_summary.json`.\n5. **Error triage** — [`references\u002Fbenchmark-execution.md#common-error-cases-and-signals`](references\u002Fbenchmark-execution.md#common-error-cases-and-signals).\n",{"data":35,"body":58},{"name":4,"version":36,"description":6,"license":26,"compatibility":37,"metadata":38,"allowed-tools":57},"2.6.0","Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts\u002Feval (eval deps live in scripts\u002Feval\u002Fpyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai\u002Fmixtral-8x22b-instruct-v0.1).",{"author":39,"github-url":40,"endpoint-openapi-schemas":41,"argument-hint":44,"tags":45,"languages":51,"frameworks":54,"domain":56},"NVIDIA RAG \u003Cfoundational-rag-dev@exchange.nvidia.com>","https:\u002F\u002Fgithub.com\u002FNVIDIA-AI-Blueprints\u002Frag",[42,43],"docs\u002Fapi_reference\u002Fopenapi_schema_rag_server.json","docs\u002Fapi_reference\u002Fopenapi_schema_ingestor_server.json","RAGAS eval | evaluate_rag | train.json | corpus | results json | error triage | uv run --project scripts\u002Feval | enable_reranker | query_rewriting | temperature | skip_ingestion",[8,46,18,47,48,49,50],"blueprint","evaluation","ragas","benchmarking","nvidia-rag-blueprint",[52,53],"python","shell",[48,55],"fastapi","ai-ml","Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Write Edit",{"type":59,"children":60},"root",[61,87,94,137,172,178,209,215,355,361,587,593,620,628,755,763,793,807,813,878,883,1105,1119,1125,1277,1283,1474,1480,1660],{"type":62,"tag":63,"props":64,"children":66},"element","h1",{"id":65},"on-disk-rag-evaluation-corpus-trainjson",[67,70,77,79,85],{"type":68,"value":69},"text","On-disk RAG evaluation (",{"type":62,"tag":71,"props":72,"children":74},"code",{"className":73},[],[75],{"type":68,"value":76},"corpus\u002F",{"type":68,"value":78}," + ",{"type":62,"tag":71,"props":80,"children":82},{"className":81},[],[83],{"type":68,"value":84},"train.json",{"type":68,"value":86},")",{"type":62,"tag":88,"props":89,"children":91},"h2",{"id":90},"purpose",[92],{"type":68,"value":93},"Purpose",{"type":62,"tag":95,"props":96,"children":97},"p",{},[98,100,106,108,113,115,120,122,128,130,135],{"type":68,"value":99},"Guide agents through NVIDIA RAG Blueprint ",{"type":62,"tag":101,"props":102,"children":103},"strong",{},[104],{"type":68,"value":105},"filesystem",{"type":68,"value":107}," benchmarks: preparing ",{"type":62,"tag":71,"props":109,"children":111},{"className":110},[],[112],{"type":68,"value":76},{"type":68,"value":114}," and ",{"type":62,"tag":71,"props":116,"children":118},{"className":117},[],[119],{"type":68,"value":84},{"type":68,"value":121},", running ",{"type":62,"tag":71,"props":123,"children":125},{"className":124},[],[126],{"type":68,"value":127},"scripts\u002Feval\u002Fevaluate_rag.py",{"type":68,"value":129},", tuning retrieval and generation flags for ",{"type":62,"tag":101,"props":131,"children":132},{},[133],{"type":68,"value":134},"quality",{"type":68,"value":136}," comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP\u002Fstream errors, empty contexts, collection mismatch, judge API).",{"type":62,"tag":95,"props":138,"children":139},{},[140,142,147,149,154,156,162,164,170],{"type":68,"value":141},"For ",{"type":62,"tag":101,"props":143,"children":144},{},[145],{"type":68,"value":146},"latency, throughput, and load testing",{"type":68,"value":148},", use the ",{"type":62,"tag":101,"props":150,"children":151},{},[152],{"type":68,"value":153},"rag-perf",{"type":68,"value":155}," skill (",{"type":62,"tag":71,"props":157,"children":159},{"className":158},[],[160],{"type":68,"value":161},"scripts\u002Frag-perf",{"type":68,"value":163},", ",{"type":62,"tag":71,"props":165,"children":167},{"className":166},[],[168],{"type":68,"value":169},"docs\u002Fperformance-benchmarking.md",{"type":68,"value":171},") — not this skill.",{"type":62,"tag":88,"props":173,"children":175},{"id":174},"when-not-to-use",[176],{"type":68,"value":177},"When not to use",{"type":62,"tag":95,"props":179,"children":180},{},[181,183,188,190,195,196,201,203,207],{"type":68,"value":182},"Do ",{"type":62,"tag":101,"props":184,"children":185},{},[186],{"type":68,"value":187},"not",{"type":68,"value":189}," use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the ",{"type":62,"tag":71,"props":191,"children":193},{"className":192},[],[194],{"type":68,"value":76},{"type":68,"value":78},{"type":62,"tag":71,"props":197,"children":199},{"className":198},[],[200],{"type":68,"value":84},{"type":68,"value":202}," layout; general ML experimentation unrelated to this evaluator; production monitoring\u002Falerting; or latency\u002Fthroughput benchmarking (use ",{"type":62,"tag":101,"props":204,"children":205},{},[206],{"type":68,"value":153},{"type":68,"value":208},").",{"type":62,"tag":88,"props":210,"children":212},{"id":211},"prerequisites",[213],{"type":68,"value":214},"Prerequisites",{"type":62,"tag":216,"props":217,"children":218},"ul",{},[219,232,258,291,324],{"type":62,"tag":220,"props":221,"children":222},"li",{},[223,225,230],{"type":68,"value":224},"Repo cloned; ",{"type":62,"tag":101,"props":226,"children":227},{},[228],{"type":68,"value":229},"run commands from repo root",{"type":68,"value":231}," (imports and paths assume this).",{"type":62,"tag":220,"props":233,"children":234},{},[235,237,242,243,248,250,256],{"type":68,"value":236},"Python ",{"type":62,"tag":101,"props":238,"children":239},{},[240],{"type":68,"value":241},"3.11+",{"type":68,"value":114},{"type":62,"tag":101,"props":244,"children":245},{},[246],{"type":68,"value":247},"uv",{"type":68,"value":249},"; eval deps: ",{"type":62,"tag":71,"props":251,"children":253},{"className":252},[],[254],{"type":68,"value":255},"uv sync --project scripts\u002Feval",{"type":68,"value":257},".",{"type":62,"tag":220,"props":259,"children":260},{},[261,263,268,269,274,276,282,284,290],{"type":68,"value":262},"Reachable ",{"type":62,"tag":101,"props":264,"children":265},{},[266],{"type":68,"value":267},"RAG server",{"type":68,"value":114},{"type":62,"tag":101,"props":270,"children":271},{},[272],{"type":68,"value":273},"ingestor",{"type":68,"value":275}," (defaults often ",{"type":62,"tag":71,"props":277,"children":279},{"className":278},[],[280],{"type":68,"value":281},"localhost:8081",{"type":68,"value":283}," \u002F ",{"type":62,"tag":71,"props":285,"children":287},{"className":286},[],[288],{"type":68,"value":289},"8082",{"type":68,"value":208},{"type":62,"tag":220,"props":292,"children":293},{},[294,303,305,312,314,323],{"type":62,"tag":101,"props":295,"children":296},{},[297],{"type":62,"tag":71,"props":298,"children":300},{"className":299},[],[301],{"type":68,"value":302},"NVIDIA_API_KEY",{"type":68,"value":304}," for RAGAS (see ",{"type":62,"tag":306,"props":307,"children":309},"a",{"href":308},"references\u002Fbenchmark-execution.md#credential-hygiene-nvidia_api_key",[310],{"type":68,"value":311},"credential hygiene",{"type":68,"value":313},"); optional ",{"type":62,"tag":101,"props":315,"children":316},{},[317],{"type":62,"tag":71,"props":318,"children":320},{"className":319},[],[321],{"type":68,"value":322},"RAG_EVAL_JUDGE_MODEL",{"type":68,"value":257},{"type":62,"tag":220,"props":325,"children":326},{},[327,329,335,337,345,346,354],{"type":68,"value":328},"Dataset roots passed to ",{"type":62,"tag":71,"props":330,"children":332},{"className":331},[],[333],{"type":68,"value":334},"--dataset-paths",{"type":68,"value":336}," each contain ",{"type":62,"tag":101,"props":338,"children":339},{},[340],{"type":62,"tag":71,"props":341,"children":343},{"className":342},[],[344],{"type":68,"value":76},{"type":68,"value":114},{"type":62,"tag":101,"props":347,"children":348},{},[349],{"type":62,"tag":71,"props":350,"children":352},{"className":351},[],[353],{"type":68,"value":84},{"type":68,"value":257},{"type":62,"tag":88,"props":356,"children":358},{"id":357},"instructions",[359],{"type":68,"value":360},"Instructions",{"type":62,"tag":362,"props":363,"children":364},"ol",{},[365,414,476,533,562],{"type":62,"tag":220,"props":366,"children":367},{},[368,373,375,380,382,391,393,398,400,405,407,412],{"type":62,"tag":101,"props":369,"children":370},{},[371],{"type":68,"value":372},"Prepare data",{"type":68,"value":374}," — Ensure each dataset directory matches the layout and ",{"type":62,"tag":71,"props":376,"children":378},{"className":377},[],[379],{"type":68,"value":84},{"type":68,"value":381}," rules in ",{"type":62,"tag":306,"props":383,"children":385},{"href":384},"references\u002Fdataset-and-conversion.md",[386],{"type":62,"tag":71,"props":387,"children":389},{"className":388},[],[390],{"type":68,"value":384},{"type":68,"value":392},". When sources arrive as public links (sites or dataset pages), materialize documents under ",{"type":62,"tag":71,"props":394,"children":396},{"className":395},[],[397],{"type":68,"value":76},{"type":68,"value":399},"—prefer ",{"type":62,"tag":101,"props":401,"children":402},{},[403],{"type":68,"value":404},"PDF",{"type":68,"value":406}," for multimodal content so ",{"type":62,"tag":101,"props":408,"children":409},{},[410],{"type":68,"value":411},"images stay embedded",{"type":68,"value":413},"; convert CSV\u002FJSONL\u002Fetc. using the patterns there.",{"type":62,"tag":220,"props":415,"children":416},{},[417,422,424,430,432,437,438,444,446,452,454,463,465,474],{"type":62,"tag":101,"props":418,"children":419},{},[420],{"type":68,"value":421},"Run eval",{"type":68,"value":423}," — ",{"type":62,"tag":71,"props":425,"children":427},{"className":426},[],[428],{"type":68,"value":429},"uv run --project scripts\u002Feval python scripts\u002Feval\u002Fevaluate_rag.py",{"type":68,"value":431}," with ",{"type":62,"tag":71,"props":433,"children":435},{"className":434},[],[436],{"type":68,"value":334},{"type":68,"value":163},{"type":62,"tag":71,"props":439,"children":441},{"className":440},[],[442],{"type":68,"value":443},"--host",{"type":68,"value":445},", and ",{"type":62,"tag":71,"props":447,"children":449},{"className":448},[],[450],{"type":68,"value":451},"--port",{"type":68,"value":453},". See ",{"type":62,"tag":306,"props":455,"children":457},{"href":456},"references\u002Fbenchmark-execution.md",[458],{"type":62,"tag":71,"props":459,"children":461},{"className":460},[],[462],{"type":68,"value":456},{"type":68,"value":464}," for command examples, outputs, and errors. Use ",{"type":62,"tag":306,"props":466,"children":468},{"href":467},"references\u002Fevaluate-rag-cli.md",[469],{"type":62,"tag":71,"props":470,"children":472},{"className":471},[],[473],{"type":68,"value":467},{"type":68,"value":475}," for flag-level detail.",{"type":62,"tag":220,"props":477,"children":478},{},[479,484,486,492,493,499,501,507,508,514,515,521,523,531],{"type":62,"tag":101,"props":480,"children":481},{},[482],{"type":68,"value":483},"Tune quality",{"type":68,"value":485}," — Adjust ",{"type":62,"tag":71,"props":487,"children":489},{"className":488},[],[490],{"type":68,"value":491},"--top_k",{"type":68,"value":283},{"type":62,"tag":71,"props":494,"children":496},{"className":495},[],[497],{"type":68,"value":498},"--vdb_top_k",{"type":68,"value":500},", reranker and query-rewriting toggles, and generation overrides (",{"type":62,"tag":71,"props":502,"children":504},{"className":503},[],[505],{"type":68,"value":506},"--temperature",{"type":68,"value":163},{"type":62,"tag":71,"props":509,"children":511},{"className":510},[],[512],{"type":68,"value":513},"--top-p",{"type":68,"value":163},{"type":62,"tag":71,"props":516,"children":518},{"className":517},[],[519],{"type":68,"value":520},"--max-tokens",{"type":68,"value":522},") as documented in ",{"type":62,"tag":306,"props":524,"children":525},{"href":456},[526],{"type":62,"tag":71,"props":527,"children":529},{"className":528},[],[530],{"type":68,"value":456},{"type":68,"value":532}," when comparing retrieval\u002Fgeneration configs for RAGAS scores.",{"type":62,"tag":220,"props":534,"children":535},{},[536,541,543,552,554,560],{"type":62,"tag":101,"props":537,"children":538},{},[539],{"type":68,"value":540},"Analyze results",{"type":68,"value":542}," — Use ",{"type":62,"tag":306,"props":544,"children":546},{"href":545},"references\u002Fresult-analysis.md",[547],{"type":62,"tag":71,"props":548,"children":550},{"className":549},[],[551],{"type":68,"value":545},{"type":68,"value":553}," for scripts; scan ",{"type":62,"tag":71,"props":555,"children":557},{"className":556},[],[558],{"type":68,"value":559},"rag_*_evaluation_summary.json",{"type":68,"value":561}," for headline RAGAS metrics.",{"type":62,"tag":220,"props":563,"children":564},{},[565,570,572,578,580,585],{"type":62,"tag":101,"props":566,"children":567},{},[568],{"type":68,"value":569},"Triage errors",{"type":68,"value":571}," — Use the ",{"type":62,"tag":306,"props":573,"children":575},{"href":574},"references\u002Fbenchmark-execution.md#common-error-cases-and-signals",[576],{"type":68,"value":577},"error signal table",{"type":68,"value":579}," and the ",{"type":62,"tag":101,"props":581,"children":582},{},[583],{"type":68,"value":584},"Troubleshooting",{"type":68,"value":586}," section below.",{"type":62,"tag":88,"props":588,"children":590},{"id":589},"examples",[591],{"type":68,"value":592},"Examples",{"type":62,"tag":95,"props":594,"children":595},{},[596,601,603,609,611,619],{"type":62,"tag":101,"props":597,"children":598},{},[599],{"type":68,"value":600},"Set API key without putting secrets in shell history (preferred patterns):",{"type":68,"value":602}," load from a gitignored env file or secrets manager; avoid committing ",{"type":62,"tag":71,"props":604,"children":606},{"className":605},[],[607],{"type":68,"value":608},".env",{"type":68,"value":610},"; rotate keys if exposed. Details: ",{"type":62,"tag":306,"props":612,"children":613},{"href":308},[614],{"type":62,"tag":71,"props":615,"children":617},{"className":616},[],[618],{"type":68,"value":308},{"type":68,"value":257},{"type":62,"tag":95,"props":621,"children":622},{},[623],{"type":62,"tag":101,"props":624,"children":625},{},[626],{"type":68,"value":627},"Minimal eval (key already in environment):",{"type":62,"tag":629,"props":630,"children":635},"pre",{"className":631,"code":632,"language":633,"meta":634,"style":634},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv sync --project scripts\u002Feval\nuv run --project scripts\u002Feval python scripts\u002Feval\u002Fevaluate_rag.py \\\n  --dataset-paths \u002Fpath\u002Fto\u002Fmy_dataset \\\n  --host localhost \\\n  --port 8081\n","bash","",[636],{"type":62,"tag":71,"props":637,"children":638},{"__ignoreMap":634},[639,666,704,722,740],{"type":62,"tag":640,"props":641,"children":644},"span",{"class":642,"line":643},"line",1,[645,650,656,661],{"type":62,"tag":640,"props":646,"children":648},{"style":647},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[649],{"type":68,"value":247},{"type":62,"tag":640,"props":651,"children":653},{"style":652},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[654],{"type":68,"value":655}," sync",{"type":62,"tag":640,"props":657,"children":658},{"style":652},[659],{"type":68,"value":660}," --project",{"type":62,"tag":640,"props":662,"children":663},{"style":652},[664],{"type":68,"value":665}," scripts\u002Feval\n",{"type":62,"tag":640,"props":667,"children":669},{"class":642,"line":668},2,[670,674,679,683,688,693,698],{"type":62,"tag":640,"props":671,"children":672},{"style":647},[673],{"type":68,"value":247},{"type":62,"tag":640,"props":675,"children":676},{"style":652},[677],{"type":68,"value":678}," run",{"type":62,"tag":640,"props":680,"children":681},{"style":652},[682],{"type":68,"value":660},{"type":62,"tag":640,"props":684,"children":685},{"style":652},[686],{"type":68,"value":687}," scripts\u002Feval",{"type":62,"tag":640,"props":689,"children":690},{"style":652},[691],{"type":68,"value":692}," python",{"type":62,"tag":640,"props":694,"children":695},{"style":652},[696],{"type":68,"value":697}," scripts\u002Feval\u002Fevaluate_rag.py",{"type":62,"tag":640,"props":699,"children":701},{"style":700},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[702],{"type":68,"value":703}," \\\n",{"type":62,"tag":640,"props":705,"children":707},{"class":642,"line":706},3,[708,713,718],{"type":62,"tag":640,"props":709,"children":710},{"style":652},[711],{"type":68,"value":712},"  --dataset-paths",{"type":62,"tag":640,"props":714,"children":715},{"style":652},[716],{"type":68,"value":717}," \u002Fpath\u002Fto\u002Fmy_dataset",{"type":62,"tag":640,"props":719,"children":720},{"style":700},[721],{"type":68,"value":703},{"type":62,"tag":640,"props":723,"children":725},{"class":642,"line":724},4,[726,731,736],{"type":62,"tag":640,"props":727,"children":728},{"style":652},[729],{"type":68,"value":730},"  --host",{"type":62,"tag":640,"props":732,"children":733},{"style":652},[734],{"type":68,"value":735}," localhost",{"type":62,"tag":640,"props":737,"children":738},{"style":700},[739],{"type":68,"value":703},{"type":62,"tag":640,"props":741,"children":743},{"class":642,"line":742},5,[744,749],{"type":62,"tag":640,"props":745,"children":746},{"style":652},[747],{"type":68,"value":748},"  --port",{"type":62,"tag":640,"props":750,"children":752},{"style":751},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[753],{"type":68,"value":754}," 8081\n",{"type":62,"tag":95,"props":756,"children":757},{},[758],{"type":62,"tag":101,"props":759,"children":760},{},[761],{"type":68,"value":762},"Pretty-print summary JSON:",{"type":62,"tag":629,"props":764,"children":766},{"className":631,"code":765,"language":633,"meta":634,"style":634},"python3 -m json.tool results\u002Fmy_dataset\u002Frag_my_dataset_evaluation_summary.json\n",[767],{"type":62,"tag":71,"props":768,"children":769},{"__ignoreMap":634},[770],{"type":62,"tag":640,"props":771,"children":772},{"class":642,"line":643},[773,778,783,788],{"type":62,"tag":640,"props":774,"children":775},{"style":647},[776],{"type":68,"value":777},"python3",{"type":62,"tag":640,"props":779,"children":780},{"style":652},[781],{"type":68,"value":782}," -m",{"type":62,"tag":640,"props":784,"children":785},{"style":652},[786],{"type":68,"value":787}," json.tool",{"type":62,"tag":640,"props":789,"children":790},{"style":652},[791],{"type":68,"value":792}," results\u002Fmy_dataset\u002Frag_my_dataset_evaluation_summary.json\n",{"type":62,"tag":95,"props":794,"children":795},{},[796,798,806],{"type":68,"value":797},"More examples (skip ingestion, quality sweeps): ",{"type":62,"tag":306,"props":799,"children":800},{"href":456},[801],{"type":62,"tag":71,"props":802,"children":804},{"className":803},[],[805],{"type":68,"value":456},{"type":68,"value":257},{"type":62,"tag":88,"props":808,"children":810},{"id":809},"limitations",[811],{"type":68,"value":812},"Limitations",{"type":62,"tag":216,"props":814,"children":815},{},[816,835,845,862],{"type":62,"tag":220,"props":817,"children":818},{},[819,821,826,827,833],{"type":68,"value":820},"Evaluator behavior is fixed to the ",{"type":62,"tag":101,"props":822,"children":823},{},[824],{"type":68,"value":825},"filesystem contract",{"type":68,"value":114},{"type":62,"tag":71,"props":828,"children":830},{"className":829},[],[831],{"type":68,"value":832},"evaluate_rag.py",{"type":68,"value":834},"; it does not substitute for custom offline judges or non-RAG benchmarks.",{"type":62,"tag":220,"props":836,"children":837},{},[838,843],{"type":62,"tag":101,"props":839,"children":840},{},[841],{"type":68,"value":842},"Vector DB \u002F embedding",{"type":68,"value":844}," choices follow deployed ingestor and RAG env — not overridden by this CLI alone.",{"type":62,"tag":220,"props":846,"children":847},{},[848,853,855,860],{"type":62,"tag":101,"props":849,"children":850},{},[851],{"type":68,"value":852},"Scores depend on",{"type":68,"value":854}," retrieval quality, judge model availability, and ",{"type":62,"tag":71,"props":856,"children":858},{"className":857},[],[859],{"type":68,"value":302},{"type":68,"value":861},"; empty contexts yield partial RAGAS metrics (see references).",{"type":62,"tag":220,"props":863,"children":864},{},[865,867,876],{"type":68,"value":866},"Large procedural detail lives under ",{"type":62,"tag":101,"props":868,"children":869},{},[870],{"type":62,"tag":71,"props":871,"children":873},{"className":872},[],[874],{"type":68,"value":875},"references\u002F",{"type":68,"value":877}," to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.",{"type":62,"tag":88,"props":879,"children":881},{"id":880},"troubleshooting",[882],{"type":68,"value":584},{"type":62,"tag":884,"props":885,"children":886},"table",{},[887,911],{"type":62,"tag":888,"props":889,"children":890},"thead",{},[891],{"type":62,"tag":892,"props":893,"children":894},"tr",{},[895,901,906],{"type":62,"tag":896,"props":897,"children":898},"th",{},[899],{"type":68,"value":900},"Error \u002F signal",{"type":62,"tag":896,"props":902,"children":903},{},[904],{"type":68,"value":905},"Likely cause",{"type":62,"tag":896,"props":907,"children":908},{},[909],{"type":68,"value":910},"What to do",{"type":62,"tag":912,"props":913,"children":914},"tbody",{},[915,948,979,1010,1072],{"type":62,"tag":892,"props":916,"children":917},{},[918,929,934],{"type":62,"tag":919,"props":920,"children":921},"td",{},[922,924],{"type":68,"value":923},"Immediate exit mentioning ",{"type":62,"tag":71,"props":925,"children":927},{"className":926},[],[928],{"type":68,"value":302},{"type":62,"tag":919,"props":930,"children":931},{},[932],{"type":68,"value":933},"Missing or invalid key",{"type":62,"tag":919,"props":935,"children":936},{},[937,939,947],{"type":68,"value":938},"Set key via secure channel; see credential hygiene in ",{"type":62,"tag":306,"props":940,"children":941},{"href":456},[942],{"type":62,"tag":71,"props":943,"children":945},{"className":944},[],[946],{"type":68,"value":456},{"type":68,"value":257},{"type":62,"tag":892,"props":949,"children":950},{},[951,960,965],{"type":62,"tag":919,"props":952,"children":953},{},[954],{"type":62,"tag":71,"props":955,"children":957},{"className":956},[],[958],{"type":68,"value":959},"train.json must be a JSON array",{"type":62,"tag":919,"props":961,"children":962},{},[963],{"type":68,"value":964},"Wrong JSON shape",{"type":62,"tag":919,"props":966,"children":967},{},[968,970,978],{"type":68,"value":969},"Top-level array of objects; validate per ",{"type":62,"tag":306,"props":971,"children":972},{"href":384},[973],{"type":62,"tag":71,"props":974,"children":976},{"className":975},[],[977],{"type":68,"value":384},{"type":68,"value":257},{"type":62,"tag":892,"props":980,"children":981},{},[982,1000,1005],{"type":62,"tag":919,"props":983,"children":984},{},[985,987,993,995],{"type":68,"value":986},"Fewer rows in ",{"type":62,"tag":71,"props":988,"children":990},{"className":989},[],[991],{"type":68,"value":992},"evaluation_data.json",{"type":68,"value":994}," than ",{"type":62,"tag":71,"props":996,"children":998},{"className":997},[],[999],{"type":68,"value":84},{"type":62,"tag":919,"props":1001,"children":1002},{},[1003],{"type":68,"value":1004},"Per-query failures",{"type":62,"tag":919,"props":1006,"children":1007},{},[1008],{"type":68,"value":1009},"Check stderr: network or stream JSON errors; see error table in benchmark-execution.",{"type":62,"tag":892,"props":1011,"children":1012},{},[1013,1026,1031],{"type":62,"tag":919,"props":1014,"children":1015},{},[1016,1018,1024],{"type":68,"value":1017},"Empty ",{"type":62,"tag":71,"props":1019,"children":1021},{"className":1020},[],[1022],{"type":68,"value":1023},"generated_contexts",{"type":68,"value":1025}," everywhere",{"type":62,"tag":919,"props":1027,"children":1028},{},[1029],{"type":68,"value":1030},"Retrieval gap",{"type":62,"tag":919,"props":1032,"children":1033},{},[1034,1036,1042,1043,1049,1050,1056,1058,1063,1064,1070],{"type":68,"value":1035},"Verify collection, ingestion, ",{"type":62,"tag":71,"props":1037,"children":1039},{"className":1038},[],[1040],{"type":68,"value":1041},"top_k",{"type":68,"value":283},{"type":62,"tag":71,"props":1044,"children":1046},{"className":1045},[],[1047],{"type":68,"value":1048},"vdb_top_k",{"type":68,"value":445},{"type":62,"tag":71,"props":1051,"children":1053},{"className":1052},[],[1054],{"type":68,"value":1055},"ingestor_server_url",{"type":68,"value":1057}," ",{"type":62,"tag":101,"props":1059,"children":1060},{},[1061],{"type":68,"value":1062},"without",{"type":68,"value":1057},{"type":62,"tag":71,"props":1065,"children":1067},{"className":1066},[],[1068],{"type":68,"value":1069},"\u002Fv1",{"type":68,"value":1071}," suffix.",{"type":62,"tag":892,"props":1073,"children":1074},{},[1075,1080,1085],{"type":62,"tag":919,"props":1076,"children":1077},{},[1078],{"type":68,"value":1079},"Ingestor 404 on upload",{"type":62,"tag":919,"props":1081,"children":1082},{},[1083],{"type":68,"value":1084},"Bad ingestor base URL",{"type":62,"tag":919,"props":1086,"children":1087},{},[1088,1090,1096,1098,1104],{"type":68,"value":1089},"Pass ",{"type":62,"tag":71,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":68,"value":1095},"http:\u002F\u002Fhost:port",{"type":68,"value":1097}," only — code appends ",{"type":62,"tag":71,"props":1099,"children":1101},{"className":1100},[],[1102],{"type":68,"value":1103},"\u002Fv1\u002F",{"type":68,"value":257},{"type":62,"tag":95,"props":1106,"children":1107},{},[1108,1110,1118],{"type":68,"value":1109},"Full signal table: ",{"type":62,"tag":306,"props":1111,"children":1112},{"href":574},[1113],{"type":62,"tag":71,"props":1114,"children":1116},{"className":1115},[],[1117],{"type":68,"value":574},{"type":68,"value":257},{"type":62,"tag":88,"props":1120,"children":1122},{"id":1121},"gotchas",[1123],{"type":68,"value":1124},"Gotchas",{"type":62,"tag":216,"props":1126,"children":1127},{},[1128,1145,1187,1205,1226,1252],{"type":62,"tag":220,"props":1129,"children":1130},{},[1131,1136,1138,1143],{"type":62,"tag":101,"props":1132,"children":1133},{},[1134],{"type":68,"value":1135},"Run from repo root",{"type":68,"value":1137},": paths and imports in ",{"type":62,"tag":71,"props":1139,"children":1141},{"className":1140},[],[1142],{"type":68,"value":127},{"type":68,"value":1144}," assume this; a wrong directory silently breaks imports.",{"type":62,"tag":220,"props":1146,"children":1147},{},[1148,1157,1159,1164,1166,1171,1173,1178,1180,1185],{"type":62,"tag":101,"props":1149,"children":1150},{},[1151],{"type":62,"tag":71,"props":1152,"children":1154},{"className":1153},[],[1155],{"type":68,"value":1156},"--ingestor_server_url",{"type":68,"value":1158},": pass ",{"type":62,"tag":71,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":68,"value":1095},{"type":68,"value":1165}," without ",{"type":62,"tag":71,"props":1167,"children":1169},{"className":1168},[],[1170],{"type":68,"value":1069},{"type":68,"value":1172},"—the code appends ",{"type":62,"tag":71,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":68,"value":1103},{"type":68,"value":1179}," automatically. Including ",{"type":62,"tag":71,"props":1181,"children":1183},{"className":1182},[],[1184],{"type":68,"value":1069},{"type":68,"value":1186}," causes 404s on ingestor calls.",{"type":62,"tag":220,"props":1188,"children":1189},{},[1190,1195,1197,1203],{"type":62,"tag":101,"props":1191,"children":1192},{},[1193],{"type":68,"value":1194},"Vector DB \u002F embedding settings",{"type":68,"value":1196},": not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. ",{"type":62,"tag":71,"props":1198,"children":1200},{"className":1199},[],[1201],{"type":68,"value":1202},"APP_VECTORSTORE_URL",{"type":68,"value":1204},", embedding model).",{"type":62,"tag":220,"props":1206,"children":1207},{},[1208,1224],{"type":62,"tag":101,"props":1209,"children":1210},{},[1211,1217,1218],{"type":62,"tag":71,"props":1212,"children":1214},{"className":1213},[],[1215],{"type":68,"value":1216},"--model",{"type":68,"value":283},{"type":62,"tag":71,"props":1219,"children":1221},{"className":1220},[],[1222],{"type":68,"value":1223},"--llm_endpoint",{"type":68,"value":1225},": forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.",{"type":62,"tag":220,"props":1227,"children":1228},{},[1229,1234,1236,1242,1244,1250],{"type":62,"tag":101,"props":1230,"children":1231},{},[1232],{"type":68,"value":1233},"Stale collections",{"type":68,"value":1235},": a previous run's ingested data persists unless you use ",{"type":62,"tag":71,"props":1237,"children":1239},{"className":1238},[],[1240],{"type":68,"value":1241},"--force_ingestion",{"type":68,"value":1243},". Use ",{"type":62,"tag":71,"props":1245,"children":1247},{"className":1246},[],[1248],{"type":68,"value":1249},"--collection",{"type":68,"value":1251}," with a unique name when comparing quality across isolated runs.",{"type":62,"tag":220,"props":1253,"children":1254},{},[1255,1260,1262,1267,1269,1275],{"type":62,"tag":101,"props":1256,"children":1257},{},[1258],{"type":68,"value":1259},"Empty context metrics",{"type":68,"value":1261},": if all ",{"type":62,"tag":71,"props":1263,"children":1265},{"className":1264},[],[1266],{"type":68,"value":1023},{"type":68,"value":1268}," are empty, RAGAS scores only ",{"type":62,"tag":71,"props":1270,"children":1272},{"className":1271},[],[1273],{"type":68,"value":1274},"nv_accuracy",{"type":68,"value":1276}," and leaves the other two metrics blank—this is not a silent success.",{"type":62,"tag":88,"props":1278,"children":1280},{"id":1279},"source-of-truth",[1281],{"type":68,"value":1282},"Source of truth",{"type":62,"tag":884,"props":1284,"children":1285},{},[1286,1302],{"type":62,"tag":888,"props":1287,"children":1288},{},[1289],{"type":62,"tag":892,"props":1290,"children":1291},{},[1292,1297],{"type":62,"tag":896,"props":1293,"children":1294},{},[1295],{"type":68,"value":1296},"Piece",{"type":62,"tag":896,"props":1298,"children":1299},{},[1300],{"type":68,"value":1301},"Location",{"type":62,"tag":912,"props":1303,"children":1304},{},[1305,1351,1368,1395,1414,1433,1452],{"type":62,"tag":892,"props":1306,"children":1307},{},[1308,1313],{"type":62,"tag":919,"props":1309,"children":1310},{},[1311],{"type":68,"value":1312},"Driver",{"type":62,"tag":919,"props":1314,"children":1315},{},[1316,1321,1323,1329,1331,1337,1338,1344,1345,1350],{"type":62,"tag":71,"props":1317,"children":1319},{"className":1318},[],[1320],{"type":68,"value":127},{"type":68,"value":1322}," (",{"type":62,"tag":71,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":68,"value":1328},"CORPUS_DIRECTORY",{"type":68,"value":1330}," = ",{"type":62,"tag":71,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":68,"value":1336},"corpus",{"type":68,"value":163},{"type":62,"tag":71,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":68,"value":1343},"EVAL_DATA",{"type":68,"value":1330},{"type":62,"tag":71,"props":1346,"children":1348},{"className":1347},[],[1349],{"type":68,"value":84},{"type":68,"value":86},{"type":62,"tag":892,"props":1352,"children":1353},{},[1354,1359],{"type":62,"tag":919,"props":1355,"children":1356},{},[1357],{"type":68,"value":1358},"Human README (always in-repo)",{"type":62,"tag":919,"props":1360,"children":1361},{},[1362],{"type":62,"tag":71,"props":1363,"children":1365},{"className":1364},[],[1366],{"type":68,"value":1367},"scripts\u002Feval\u002FREADME.md",{"type":62,"tag":892,"props":1369,"children":1370},{},[1371,1376],{"type":62,"tag":919,"props":1372,"children":1373},{},[1374],{"type":68,"value":1375},"Full CLI (flags, defaults)",{"type":62,"tag":919,"props":1377,"children":1378},{},[1379,1385,1387],{"type":62,"tag":71,"props":1380,"children":1382},{"className":1381},[],[1383],{"type":68,"value":1384},"scripts\u002Feval\u002Fevaluate_rag.py --help",{"type":68,"value":1386},"; ",{"type":62,"tag":306,"props":1388,"children":1389},{"href":467},[1390],{"type":62,"tag":71,"props":1391,"children":1393},{"className":1392},[],[1394],{"type":68,"value":467},{"type":62,"tag":892,"props":1396,"children":1397},{},[1398,1403],{"type":62,"tag":919,"props":1399,"children":1400},{},[1401],{"type":68,"value":1402},"Dataset \u002F conversion",{"type":62,"tag":919,"props":1404,"children":1405},{},[1406],{"type":62,"tag":306,"props":1407,"children":1408},{"href":384},[1409],{"type":62,"tag":71,"props":1410,"children":1412},{"className":1411},[],[1413],{"type":68,"value":384},{"type":62,"tag":892,"props":1415,"children":1416},{},[1417,1422],{"type":62,"tag":919,"props":1418,"children":1419},{},[1420],{"type":68,"value":1421},"Runs, outputs, errors",{"type":62,"tag":919,"props":1423,"children":1424},{},[1425],{"type":62,"tag":306,"props":1426,"children":1427},{"href":456},[1428],{"type":62,"tag":71,"props":1429,"children":1431},{"className":1430},[],[1432],{"type":68,"value":456},{"type":62,"tag":892,"props":1434,"children":1435},{},[1436,1441],{"type":62,"tag":919,"props":1437,"children":1438},{},[1439],{"type":68,"value":1440},"Result analysis scripts",{"type":62,"tag":919,"props":1442,"children":1443},{},[1444],{"type":62,"tag":306,"props":1445,"children":1446},{"href":545},[1447],{"type":62,"tag":71,"props":1448,"children":1450},{"className":1449},[],[1451],{"type":68,"value":545},{"type":62,"tag":892,"props":1453,"children":1454},{},[1455,1460],{"type":62,"tag":919,"props":1456,"children":1457},{},[1458],{"type":68,"value":1459},"Latency \u002F throughput",{"type":62,"tag":919,"props":1461,"children":1462},{},[1463,1467,1469],{"type":62,"tag":101,"props":1464,"children":1465},{},[1466],{"type":68,"value":153},{"type":68,"value":1468}," skill, ",{"type":62,"tag":71,"props":1470,"children":1472},{"className":1471},[],[1473],{"type":68,"value":169},{"type":62,"tag":88,"props":1475,"children":1477},{"id":1476},"agent-playbook",[1478],{"type":68,"value":1479},"Agent playbook",{"type":62,"tag":362,"props":1481,"children":1482},{},[1483,1546,1598,1617,1642],{"type":62,"tag":220,"props":1484,"children":1485},{},[1486,1490,1491,1496,1498,1503,1505,1510,1511,1516,1517,1522,1524,1529,1531,1536,1538,1544],{"type":62,"tag":101,"props":1487,"children":1488},{},[1489],{"type":68,"value":421},{"type":68,"value":423},{"type":62,"tag":71,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":68,"value":255},{"type":68,"value":1497}," then ",{"type":62,"tag":71,"props":1499,"children":1501},{"className":1500},[],[1502],{"type":68,"value":429},{"type":68,"value":1504}," with required ",{"type":62,"tag":71,"props":1506,"children":1508},{"className":1507},[],[1509],{"type":68,"value":334},{"type":68,"value":163},{"type":62,"tag":71,"props":1512,"children":1514},{"className":1513},[],[1515],{"type":68,"value":443},{"type":68,"value":445},{"type":62,"tag":71,"props":1518,"children":1520},{"className":1519},[],[1521],{"type":68,"value":451},{"type":68,"value":1523}," (and env ",{"type":62,"tag":71,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":68,"value":302},{"type":68,"value":1530},"). Argument ",{"type":62,"tag":71,"props":1532,"children":1534},{"className":1533},[],[1535],{"type":68,"value":1156},{"type":68,"value":1537}," is optional (defaults to ",{"type":62,"tag":71,"props":1539,"children":1541},{"className":1540},[],[1542],{"type":68,"value":1543},"http:\u002F\u002Flocalhost:8082",{"type":68,"value":1545},"); pass it only when overriding the ingestor endpoint.",{"type":62,"tag":220,"props":1547,"children":1548},{},[1549,1554,1556,1564,1566,1571,1573,1578,1580,1585,1586,1591,1592,1597],{"type":62,"tag":101,"props":1550,"children":1551},{},[1552],{"type":68,"value":1553},"Quality tuning",{"type":68,"value":1555}," — See ",{"type":62,"tag":306,"props":1557,"children":1558},{"href":456},[1559],{"type":62,"tag":71,"props":1560,"children":1562},{"className":1561},[],[1563],{"type":68,"value":456},{"type":68,"value":1565},": ",{"type":62,"tag":71,"props":1567,"children":1569},{"className":1568},[],[1570],{"type":68,"value":491},{"type":68,"value":1572},"\u002F",{"type":62,"tag":71,"props":1574,"children":1576},{"className":1575},[],[1577],{"type":68,"value":498},{"type":68,"value":1579},", reranker and query-rewriting toggles, ",{"type":62,"tag":71,"props":1581,"children":1583},{"className":1582},[],[1584],{"type":68,"value":506},{"type":68,"value":163},{"type":62,"tag":71,"props":1587,"children":1589},{"className":1588},[],[1590],{"type":68,"value":513},{"type":68,"value":163},{"type":62,"tag":71,"props":1593,"children":1595},{"className":1594},[],[1596],{"type":68,"value":520},{"type":68,"value":257},{"type":62,"tag":220,"props":1599,"children":1600},{},[1601,1606,1608,1616],{"type":62,"tag":101,"props":1602,"children":1603},{},[1604],{"type":68,"value":1605},"Data conversion",{"type":68,"value":1607}," — Follow ",{"type":62,"tag":306,"props":1609,"children":1610},{"href":384},[1611],{"type":62,"tag":71,"props":1612,"children":1614},{"className":1613},[],[1615],{"type":68,"value":384},{"type":68,"value":257},{"type":62,"tag":220,"props":1618,"children":1619},{},[1620,1624,1625,1633,1635,1641],{"type":62,"tag":101,"props":1621,"children":1622},{},[1623],{"type":68,"value":540},{"type":68,"value":423},{"type":62,"tag":306,"props":1626,"children":1627},{"href":545},[1628],{"type":62,"tag":71,"props":1629,"children":1631},{"className":1630},[],[1632],{"type":68,"value":545},{"type":68,"value":1634},"; quick scan: ",{"type":62,"tag":71,"props":1636,"children":1638},{"className":1637},[],[1639],{"type":68,"value":1640},"python3 -m json.tool results\u002F\u003Cdataset>\u002Frag_\u003Cdataset>_evaluation_summary.json",{"type":68,"value":257},{"type":62,"tag":220,"props":1643,"children":1644},{},[1645,1650,1651,1659],{"type":62,"tag":101,"props":1646,"children":1647},{},[1648],{"type":68,"value":1649},"Error triage",{"type":68,"value":423},{"type":62,"tag":306,"props":1652,"children":1653},{"href":574},[1654],{"type":62,"tag":71,"props":1655,"children":1657},{"className":1656},[],[1658],{"type":68,"value":574},{"type":68,"value":257},{"type":62,"tag":1661,"props":1662,"children":1663},"style",{},[1664],{"type":68,"value":1665},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1667,"total":1771},[1668,1685,1699,1713,1725,1742,1757],{"slug":1669,"name":1669,"fn":1670,"description":1671,"org":1672,"tags":1673,"stars":23,"repoUrl":24,"updatedAt":1684},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1674,1677,1680,1681],{"name":1675,"slug":1676,"type":15},"Data Analysis","data-analysis",{"name":1678,"slug":1679,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":1682,"slug":1683,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":1686,"name":1686,"fn":1687,"description":1688,"org":1689,"tags":1690,"stars":23,"repoUrl":24,"updatedAt":1698},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1691,1694,1697],{"name":1692,"slug":1693,"type":15},"Deployment","deployment",{"name":1695,"slug":1696,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":1700,"name":1700,"fn":1701,"description":1702,"org":1703,"tags":1704,"stars":23,"repoUrl":24,"updatedAt":1712},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1705,1708,1709],{"name":1706,"slug":1707,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1710,"slug":1711,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":1714,"name":1714,"fn":1715,"description":1716,"org":1717,"tags":1718,"stars":23,"repoUrl":24,"updatedAt":1724},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1719,1720,1721],{"name":1675,"slug":1676,"type":15},{"name":9,"slug":8,"type":15},{"name":1722,"slug":1723,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":1726,"name":1726,"fn":1727,"description":1728,"org":1729,"tags":1730,"stars":23,"repoUrl":24,"updatedAt":1741},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1731,1734,1737,1738],{"name":1732,"slug":1733,"type":15},"Automation","automation",{"name":1735,"slug":1736,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":1739,"slug":1740,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":1743,"name":1743,"fn":1744,"description":1745,"org":1746,"tags":1747,"stars":23,"repoUrl":24,"updatedAt":1756},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1748,1749,1752,1753],{"name":1692,"slug":1693,"type":15},{"name":1750,"slug":1751,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":1754,"slug":1755,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":1758,"name":1758,"fn":1759,"description":1760,"org":1761,"tags":1762,"stars":23,"repoUrl":24,"updatedAt":1770},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1763,1764,1767],{"name":9,"slug":8,"type":15},{"name":1765,"slug":1766,"type":15},"Quantum Computing","quantum-computing",{"name":1768,"slug":1769,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":1773,"total":1923},[1774,1792,1807,1818,1830,1844,1857,1871,1882,1891,1905,1914],{"slug":1775,"name":1775,"fn":1776,"description":1777,"org":1778,"tags":1779,"stars":1789,"repoUrl":1790,"updatedAt":1791},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1780,1783,1786],{"name":1781,"slug":1782,"type":15},"Documentation","documentation",{"name":1784,"slug":1785,"type":15},"MCP","mcp",{"name":1787,"slug":1788,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1793,"name":1793,"fn":1794,"description":1795,"org":1796,"tags":1797,"stars":1804,"repoUrl":1805,"updatedAt":1806},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1798,1801,1802],{"name":1799,"slug":1800,"type":15},"Containers","containers",{"name":1692,"slug":1693,"type":15},{"name":1803,"slug":52,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1808,"name":1808,"fn":1809,"description":1810,"org":1811,"tags":1812,"stars":1804,"repoUrl":1805,"updatedAt":1817},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1813,1816],{"name":1814,"slug":1815,"type":15},"CI\u002FCD","ci-cd",{"name":1692,"slug":1693,"type":15},"2026-07-14T05:25:59.97109",{"slug":1819,"name":1819,"fn":1820,"description":1821,"org":1822,"tags":1823,"stars":1804,"repoUrl":1805,"updatedAt":1829},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1824,1825,1826],{"name":1814,"slug":1815,"type":15},{"name":1692,"slug":1693,"type":15},{"name":1827,"slug":1828,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1831,"name":1831,"fn":1832,"description":1833,"org":1834,"tags":1835,"stars":1804,"repoUrl":1805,"updatedAt":1843},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1836,1839,1840],{"name":1837,"slug":1838,"type":15},"Debugging","debugging",{"name":1827,"slug":1828,"type":15},{"name":1841,"slug":1842,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1845,"name":1845,"fn":1846,"description":1847,"org":1848,"tags":1849,"stars":1804,"repoUrl":1805,"updatedAt":1856},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1850,1853],{"name":1851,"slug":1852,"type":15},"Best Practices","best-practices",{"name":1854,"slug":1855,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1858,"name":1858,"fn":1859,"description":1860,"org":1861,"tags":1862,"stars":1804,"repoUrl":1805,"updatedAt":1870},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1863,1866,1869],{"name":1864,"slug":1865,"type":15},"Machine Learning","machine-learning",{"name":1867,"slug":1868,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1872,"name":1872,"fn":1873,"description":1874,"org":1875,"tags":1876,"stars":1804,"repoUrl":1805,"updatedAt":1881},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1877,1880],{"name":1878,"slug":1879,"type":15},"QA","qa",{"name":1722,"slug":1723,"type":15},"2026-07-14T05:25:53.673039",{"slug":1883,"name":1883,"fn":1884,"description":1885,"org":1886,"tags":1887,"stars":1804,"repoUrl":1805,"updatedAt":1890},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1888,1889],{"name":1692,"slug":1693,"type":15},{"name":1695,"slug":1696,"type":15},"2026-07-14T05:25:49.362534",{"slug":1892,"name":1892,"fn":1893,"description":1894,"org":1895,"tags":1896,"stars":1804,"repoUrl":1805,"updatedAt":1904},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1897,1900,1901],{"name":1898,"slug":1899,"type":15},"Code Review","code-review",{"name":1827,"slug":1828,"type":15},{"name":1902,"slug":1903,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1906,"name":1906,"fn":1907,"description":1908,"org":1909,"tags":1910,"stars":1804,"repoUrl":1805,"updatedAt":1913},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1911,1912],{"name":1878,"slug":1879,"type":15},{"name":1722,"slug":1723,"type":15},"2026-07-14T05:25:54.928983",{"slug":1915,"name":1915,"fn":1916,"description":1917,"org":1918,"tags":1919,"stars":1804,"repoUrl":1805,"updatedAt":1922},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1920,1921],{"name":1732,"slug":1733,"type":15},{"name":1814,"slug":1815,"type":15},"2026-07-30T05:29:03.275638",496]