[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-rag-perf":3,"mdc--gaptrk-key":34,"related-repo-nvidia-rag-perf":2509,"related-org-nvidia-rag-perf":2613},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"rag-perf","benchmark NVIDIA RAG Blueprint server performance","Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy \u002F RAGAS scoring (use rag-eval) or for deploying \u002F repairing services (use rag-blueprint).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Performance","performance",{"name":20,"slug":21,"type":15},"RAG","rag",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:26:20.18577","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Frag-perf","---\nname: rag-perf\nversion: \"2.6.0\"\ndescription: >-\n  Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf\n  load test driven by a single YAML config. Not for accuracy \u002F RAGAS scoring (use rag-eval) or\n  for deploying \u002F repairing services (use rag-blueprint).\nlicense: Apache-2.0\ncompatibility: Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts\u002Frag-perf (perf deps live in scripts\u002Frag-perf\u002Fpyproject.toml); reachable RAG server (default http:\u002F\u002Flocalhost:8081); for synthetic queries an OpenAI-compatible chat-completions endpoint is required (default http:\u002F\u002Flocalhost:8999\u002Fv1\u002Fchat\u002Fcompletions); aiperf load-test phase uses the bundled nvidia_rag endpoint plugin, registered automatically when rag-perf is installed editable.\nmetadata:\n  tool-version: \"0.1.0\"\n  author: NVIDIA RAG \u003Cfoundational-rag-dev@exchange.nvidia.com>\n  github-url: \"https:\u002F\u002Fgithub.com\u002FNVIDIA-AI-Blueprints\u002Frag\"\n  endpoint-openapi-schemas:\n    - docs\u002Fapi_reference\u002Fopenapi_schema_rag_server.json\n  argument-hint: rag-perf | aiperf | TTFT | latency | throughput | concurrency sweep | bottleneck | retrieval \u002F reranker tuning | profile-only | synthetic queries | quick_profile.yaml | single_run.yaml | sweep.yaml | uv run --project scripts\u002Frag-perf\n  tags:\n    - nvidia\n    - blueprint\n    - rag\n    - performance\n    - benchmarking\n    - aiperf\n    - nvidia-rag-blueprint\n  languages:\n    - python\n    - shell\n  frameworks:\n    - aiperf\n    - fastapi\n  domain: ai-ml\nallowed-tools: Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Bash(cat *) Bash(curl *) Write Edit\n---\n\n# RAG-Perf — config-driven perf benchmark CLI\n\n## Purpose\n\nDrive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side **profiling pass** (per-stage timing, citation quality, bottleneck inference) and an optional **aiperf load test** (TTFT \u002F E2E \u002F token & request throughput \u002F error rate), and write a unified report. The CLI is intentionally minimal: `rag-perf -c \u003Cconfig>` plus `--help` \u002F `--version`. Behaviour is *fully* config-driven; field variations belong in YAML.\n\n## Scope\n\n- **Accuracy \u002F RAGAS** scoring of answer quality → use the **rag-eval** skill.\n- **Deploying, repairing, or configuring services** (compose, helm, NIM env vars) → use the **rag-blueprint** skill.\n- **Production monitoring \u002F alerting** — rag-perf is a one-shot benchmark tool.\n- **Runtime requirement:** a deployed RAG server reachable on the network.\n\n## Prerequisites\n\n- Repo cloned; **run commands from the repo root** (config paths in the presets are repo-root-relative).\n- Python **3.11+** and **uv** on PATH.\n- Install rag-perf into its own uv-managed venv: `uv sync --project scripts\u002Frag-perf`.\n- For unit tests: install dev extras as well — `uv sync --project scripts\u002Frag-perf --extra dev` (otherwise `pytest-asyncio` is missing and async tests error out at collection time).\n- A reachable RAG server (default `http:\u002F\u002Flocalhost:8081`). For the aiperf phase, the bundled `nvidia_rag` endpoint plugin must be installed — `pip install -e .\u002Fscripts\u002Frag-perf` registers it via the `aiperf.plugins` entry point.\n- For **synthetic** queries: an OpenAI-compatible chat-completions endpoint reachable at `synthetic.llm_url` (default `http:\u002F\u002Flocalhost:8999\u002Fv1\u002Fchat\u002Fcompletions`).\n- rag-perf itself runs without `NVIDIA_API_KEY` (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.\n\n## Instructions\n\n1. **Pick a preset.** The three under [`scripts\u002Frag-perf\u002Fconfigs\u002F`](..\u002F..\u002Fscripts\u002Frag-perf\u002Fconfigs) are:\n   - `quick_profile.yaml` — profile-only, ~30 s. Skips load test. For fast iteration on retrieval \u002F reranker tuning.\n   - `single_run.yaml` — one concurrency level, profiling + aiperf, ~2 min. Regression checks.\n   - `sweep.yaml` — multi-axis sweep. `load.concurrency`, `rag.vdb_top_k`, `rag.reranker_top_k` are all `int | list[int]`; any of them as a list becomes a sweep axis (Cartesian product).\n\n2. **Edit the preset.** **Required:** replace `rag.collection_names: [\"\u003Ccollection_name>\"]` with a real collection on the deployed ingestor server. Verify the collection exists via `GET \u002Fv1\u002Fcollections` on the ingestor. The placeholder `\u003Ccollection_name>` validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.\n\n3. **Run.** From repo root:\n   ```bash\n   uv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fsingle_run.yaml\n   ```\n   Same form for the other presets. The CLI accepts only `-c \u002F --config` (required), `--help`, `--version`.\n\n4. **Read stdout.** Every invocation prints, in order: a startup banner, a one-line summary, the **fully resolved config as YAML** (so the run is reproducible from terminal output), per-grid-point progress with the **shlex-joined aiperf command** in copy-pastable form, a **rich per-point summary table** (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a **side-by-side comparison table** auto-labelled by whichever axis varied. See [`references\u002Foutput-and-analysis.md`](references\u002Foutput-and-analysis.md).\n\n5. **Inspect artifacts.** Layout depends on run shape — flat for single-point + `iterations=1`, nested under `iter_\u003Ci>\u002F\u003Cpoint>\u002F...` otherwise. See [`references\u002Foutput-and-analysis.md`](references\u002Foutput-and-analysis.md) for the full directory tree, file purposes, and how to parse `results.json` \u002F `results.csv` \u002F `report.md`.\n\n6. **Summarise for the user.** When reporting back, follow the playbook in [`references\u002Foutput-and-analysis.md#summarising-results-to-the-user`](references\u002Foutput-and-analysis.md#summarising-results-to-the-user): pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, **always flag** zero citations \u002F non-zero error rate \u002F suspect `llm_ttft_ms` \u002F small-sample p99, and propose a concrete next-experiment YAML.\n\n7. **Tune.** Schema is fully documented in [`docs\u002Fperformance-benchmarking.md`](..\u002F..\u002Fdocs\u002Fperformance-benchmarking.md) and the deeper-dive references below. Common knobs: turn `aiperf.enabled: false` for profile-only mode, increase `load.iterations` for variance estimation, set `load.sleep_between_points_s: 60` for overnight Cartesian sweeps.\n\n## Examples\n\n**Profile-only (quickest signal on retrieval \u002F reranker tuning):**\n\n```bash\nuv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fquick_profile.yaml\n```\n\nOutput: `rag-perf-results\u002Fquick_profile\u002Frun_\u003Cts>\u002F{profile_report.md, profile_results.json, profiling\u002F}`. The `aiperf_rag_on\u002F` directory is omitted. Filenames are `profile_*` because `aiperf.enabled: false`.\n\n**Single benchmark point with full report:**\n\n```bash\nuv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fsingle_run.yaml\n```\n\nOutput: flat `run_\u003Cts>\u002F{report.md, results.json, results.csv, profiling\u002F, aiperf_rag_on\u002F}`.\n\n**Concurrency sweep:**\n\n```bash\nuv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fsweep.yaml\n```\n\nOutput: nested `run_\u003Cts>\u002Fiter_1\u002F\u003CCR:_VDB-K:_RERANKER-K:_…>\u002F{profiling,aiperf_rag_on}\u002F` per point, plus aggregate `report.md` \u002F `results.json` \u002F `results.csv` at the run root.\n\n**Run unit tests:**\n\n```bash\nuv sync --project scripts\u002Frag-perf --extra dev   # one-time, installs pytest-asyncio\nuv run --project scripts\u002Frag-perf python -m pytest tests\u002Funit\u002Ftest_rag_perf\u002F\n```\n\n## Limitations\n\n- The CLI is **config-only**: author or copy YAML to vary a parameter.\n- `load.concurrency` \u002F `rag.vdb_top_k` \u002F `rag.reranker_top_k` accept `int | list[int]`; the validator requires unique list values because each value names a unique point dir.\n- `input.file` and `input.synthetic` follow an XOR rule — both set fails validation. When neither is set, `synthetic` auto-fills with defaults so a bare config still validates.\n- File-based input format is **inferred from extension only** (`.jsonl` or `.csv`); other extensions are rejected.\n- Synthetic generation streams each query to disk as it completes (failure-resilient) but **fails fast on the first LLM error** — partial JSONL is preserved. Re-run after fixing the endpoint.\n- Reasoning models (Nemotron Omni, Qwen-Reasoning) require `synthetic.disable_thinking: true` (the default). Without it the model exhausts the token budget on chain-of-thought and `content` returns empty — the generator now raises with a clear message instead of substituting `reasoning_content` for the answer.\n- aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing `AiperfRunner._base_aiperf_cmd` in `scripts\u002Frag-perf\u002Frag_perf\u002Frunner.py`.\n- Procedural detail lives under **`references\u002F`** to keep this file concise.\n\n## Troubleshooting\n\n| Error \u002F signal | Likely cause | What to do |\n|---|---|---|\n| `Configuration errors in \u003Cyaml>:  •  input  —  ... XOR rule` | Both `input.file` and `input.synthetic` set | Pick one. The XOR validator runs at YAML load time. |\n| `input.file must end in .jsonl or .csv` | Extension other than `.jsonl` \u002F `.csv` | Rename or convert. |\n| `load.concurrency has duplicate values` | e.g. `[2, 2, 4]` | Each concurrency maps to a unique point dir; dedupe. |\n| `warmup_requests must be >= 1` | YAML had `warmup_requests: 0` | aiperf rejects warmup=0; minimum is 1. |\n| `LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true).` | Reasoning model used CoT and ran out of tokens | Set `synthetic.disable_thinking: true` (the default) or raise `min_query_tokens`. |\n| `✗ All N profiling requests failed across M point(s).` + exit 1 | Bad URL, server down, wrong collection | Verify `target.url`, `rag.collection_names` (the `\u003Ccollection_name>` placeholder will hit this). |\n| Per-iteration `⚠ N profiling requests failed` warning, run continues | Some requests timed out \u002F errored mid-run | Check rag-server logs, raise `target.timeout_s`, drop concurrency. |\n| `RuntimeError: Random synthetic query generation failed at query N: ...` | LLM endpoint rejected a request mid-generation | Partial JSONL is at `synthetic.jsonl_output_path`; fix endpoint and re-run with reduced `num_queries`, or point `input.file` at the partial file. |\n| `Citation count (mean): 0` and `Citation relevance score: N\u002FA` for a non-empty deployment | Collection mismatch between `rag.collection_names` and what's actually ingested | Run `curl -s http:\u002F\u002F\u003Cingestor>:8082\u002Fv1\u002Fcollections` to list real collections. |\n| Tests error with `ModuleNotFoundError: No module named 'pytest_asyncio'` | Dev extras missing | `uv sync --project scripts\u002Frag-perf --extra dev`. |\n| CI: `ModuleNotFoundError: No module named 'ruamel'` from `tests\u002Funit\u002Ftest_rag_perf\u002F` | rag-perf package missing from CI venv | Add `uv pip install -e .\u002Fscripts\u002Frag-perf` after the top-level install in the unit-tests job. |\n\n## Gotchas\n\n- **Run from repo root.** Preset configs reference `scripts\u002Frag-perf\u002Fexamples\u002Fqueries.jsonl` and `scripts\u002Frag-perf\u002Fprompts\u002Fdefault_prompts.yaml` with repo-root-relative paths. Running from inside `scripts\u002Frag-perf\u002F` will fail those file lookups.\n- **CLI is config-only.** Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.\n- **Always edit `rag.collection_names` before the first run.** The presets ship with `[\"\u003Ccollection_name>\"]` as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as `Citation count (mean): 0` everywhere.\n- **`load.concurrency_list`, `rag.vdb_top_k_list`, `rag.reranker_top_k_list`** are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.\n- **`aiperf.enabled: false` changes filenames.** The top-level outputs become `profile_report.md` \u002F `profile_results.json` \u002F `profile_results.csv`. The aggregate sweep table also suppresses load-test rows and the \"Optimal throughput\" footer.\n- **Resolved-config dump is verbose** (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.\n- **The aiperf shell command is logged before each subprocess.** Look for `\\n  $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ...` in stdout — copy-paste runnable for reproducing a single point outside rag-perf.\n- **`--endpoint-type nvidia_rag`** comes from the bundled plugin at `scripts\u002Frag-perf\u002Frag_perf\u002Fplugin\u002Fnvidia_rag.py`. It teaches aiperf about the RAG `\u002Fv1\u002Fgenerate` request shape and parses citations + per-stage `metrics` out of the SSE stream. If aiperf can't resolve `nvidia_rag`, rag-perf needs editable installation in the venv — re-run `uv sync --project scripts\u002Frag-perf` (or `uv pip install -e .\u002Fscripts\u002Frag-perf`).\n- **Sweep-mode point-name collision.** When two points differ only in concurrency (e.g. `[1, 4]` × single `vdb_top_k`), the dir name encodes everything: `CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:...`. Cluster \u002F GPU \u002F experiment_name (`output.cluster`, `output.gpu`, `output.experiment_name`) are appended too — useful for diff-friendly artifact paths across machines.\n- **`load.iterations > 1` repeats the entire grid**. Each repetition writes to its own `iter_\u003Ci>\u002F`. Aggregate CSV row count = `n_points × iterations`.\n\n## Source of truth\n\n| Piece | Location |\n|---|---|\n| Driver | [`scripts\u002Frag-perf\u002Frag_perf\u002Fcli.py`](..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Fcli.py) (`main` is the single Click command) |\n| Schema | [`scripts\u002Frag-perf\u002Frag_perf\u002Fconfig.py`](..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Fconfig.py) (`RunConfig` and sub-models) |\n| Orchestrator | [`scripts\u002Frag-perf\u002Frag_perf\u002Frunner.py`](..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Frunner.py) (`BenchmarkRunner.run`, `RagProfiler`, `AiperfRunner`) |\n| aiperf plugin | [`scripts\u002Frag-perf\u002Frag_perf\u002Fplugin\u002Fnvidia_rag.py`](..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Fplugin\u002Fnvidia_rag.py) |\n| User-facing doc | [`docs\u002Fperformance-benchmarking.md`](..\u002F..\u002Fdocs\u002Fperformance-benchmarking.md) |\n| Presets | [`scripts\u002Frag-perf\u002Fconfigs\u002F{quick_profile,single_run,sweep}.yaml`](..\u002F..\u002Fscripts\u002Frag-perf\u002Fconfigs\u002F) |\n| Sample queries | [`scripts\u002Frag-perf\u002Fexamples\u002Fqueries.jsonl`](..\u002F..\u002Fscripts\u002Frag-perf\u002Fexamples\u002Fqueries.jsonl) |\n| Synthetic prompts | [`scripts\u002Frag-perf\u002Fprompts\u002Fdefault_prompts.yaml`](..\u002F..\u002Fscripts\u002Frag-perf\u002Fprompts\u002Fdefault_prompts.yaml) |\n| Config schema details | [`references\u002Fconfig-schema.md`](references\u002Fconfig-schema.md) |\n| Synthetic-query generation | [`references\u002Fsynthetic-generation.md`](references\u002Fsynthetic-generation.md) |\n| Output layout & metric semantics | [`references\u002Foutput-and-analysis.md`](references\u002Foutput-and-analysis.md) |\n\n## Agent playbook\n\n1. **Sync deps:** `uv sync --project scripts\u002Frag-perf` (one-time per checkout).\n2. **Pick & customise a preset:** copy `scripts\u002Frag-perf\u002Fconfigs\u002F\u003Cpreset>.yaml` if you want a variant; always set `rag.collection_names` to a real collection.\n3. **Run:** `uv run --project scripts\u002Frag-perf rag-perf -c \u003Cconfig>` from repo root.\n4. **Read the per-point + aggregate tables on stdout.** Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table.\n5. **Parse artifacts** under `output.dir\u002Frun_\u003Cts>\u002F` — see [`references\u002Foutput-and-analysis.md`](references\u002Foutput-and-analysis.md). For multi-point runs, `results.csv` has one row per (point × iteration).\n6. **Summarise for the user** using the playbook in [`references\u002Foutput-and-analysis.md#summarising-results-to-the-user`](references\u002Foutput-and-analysis.md#summarising-results-to-the-user) — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations \u002F non-zero errors \u002F suspect `llm_ttft_ms` \u002F low sample size, and a concrete next-experiment YAML.\n7. **Tune retrieval \u002F reranker:** flip to `quick_profile.yaml` or `aiperf.enabled: false` for fast iteration, then return to `single_run.yaml` \u002F `sweep.yaml` when characterising under load.\n8. **Triage failures:** see Troubleshooting above and [`references\u002Foutput-and-analysis.md`](references\u002Foutput-and-analysis.md) for empty-citation \u002F bottleneck=N\u002FA patterns.\n",{"data":35,"body":57},{"name":4,"version":36,"description":6,"license":26,"compatibility":37,"metadata":38,"allowed-tools":56},"2.6.0","Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts\u002Frag-perf (perf deps live in scripts\u002Frag-perf\u002Fpyproject.toml); reachable RAG server (default http:\u002F\u002Flocalhost:8081); for synthetic queries an OpenAI-compatible chat-completions endpoint is required (default http:\u002F\u002Flocalhost:8999\u002Fv1\u002Fchat\u002Fcompletions); aiperf load-test phase uses the bundled nvidia_rag endpoint plugin, registered automatically when rag-perf is installed editable.",{"tool-version":39,"author":40,"github-url":41,"endpoint-openapi-schemas":42,"argument-hint":44,"tags":45,"languages":50,"frameworks":53,"domain":55},"0.1.0","NVIDIA RAG \u003Cfoundational-rag-dev@exchange.nvidia.com>","https:\u002F\u002Fgithub.com\u002FNVIDIA-AI-Blueprints\u002Frag",[43],"docs\u002Fapi_reference\u002Fopenapi_schema_rag_server.json","rag-perf | aiperf | TTFT | latency | throughput | concurrency sweep | bottleneck | retrieval \u002F reranker tuning | profile-only | synthetic queries | quick_profile.yaml | single_run.yaml | sweep.yaml | uv run --project scripts\u002Frag-perf",[8,46,21,18,47,48,49],"blueprint","benchmarking","aiperf","nvidia-rag-blueprint",[51,52],"python","shell",[48,54],"fastapi","ai-ml","Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Bash(cat *) Bash(curl *) Write Edit",{"type":58,"children":59},"root",[60,69,76,130,136,194,200,346,352,763,769,777,816,851,859,896,908,916,955,987,995,1078,1084,1258,1264,1696,1702,2018,2024,2308,2314,2503],{"type":61,"tag":62,"props":63,"children":65},"element","h1",{"id":64},"rag-perf-config-driven-perf-benchmark-cli",[66],{"type":67,"value":68},"text","RAG-Perf — config-driven perf benchmark CLI",{"type":61,"tag":70,"props":71,"children":73},"h2",{"id":72},"purpose",[74],{"type":67,"value":75},"Purpose",{"type":61,"tag":77,"props":78,"children":79},"p",{},[80,82,88,90,95,97,104,106,112,114,120,122,128],{"type":67,"value":81},"Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side ",{"type":61,"tag":83,"props":84,"children":85},"strong",{},[86],{"type":67,"value":87},"profiling pass",{"type":67,"value":89}," (per-stage timing, citation quality, bottleneck inference) and an optional ",{"type":61,"tag":83,"props":91,"children":92},{},[93],{"type":67,"value":94},"aiperf load test",{"type":67,"value":96}," (TTFT \u002F E2E \u002F token & request throughput \u002F error rate), and write a unified report. The CLI is intentionally minimal: ",{"type":61,"tag":98,"props":99,"children":101},"code",{"className":100},[],[102],{"type":67,"value":103},"rag-perf -c \u003Cconfig>",{"type":67,"value":105}," plus ",{"type":61,"tag":98,"props":107,"children":109},{"className":108},[],[110],{"type":67,"value":111},"--help",{"type":67,"value":113}," \u002F ",{"type":61,"tag":98,"props":115,"children":117},{"className":116},[],[118],{"type":67,"value":119},"--version",{"type":67,"value":121},". Behaviour is ",{"type":61,"tag":123,"props":124,"children":125},"em",{},[126],{"type":67,"value":127},"fully",{"type":67,"value":129}," config-driven; field variations belong in YAML.",{"type":61,"tag":70,"props":131,"children":133},{"id":132},"scope",[134],{"type":67,"value":135},"Scope",{"type":61,"tag":137,"props":138,"children":139},"ul",{},[140,158,174,184],{"type":61,"tag":141,"props":142,"children":143},"li",{},[144,149,151,156],{"type":61,"tag":83,"props":145,"children":146},{},[147],{"type":67,"value":148},"Accuracy \u002F RAGAS",{"type":67,"value":150}," scoring of answer quality → use the ",{"type":61,"tag":83,"props":152,"children":153},{},[154],{"type":67,"value":155},"rag-eval",{"type":67,"value":157}," skill.",{"type":61,"tag":141,"props":159,"children":160},{},[161,166,168,173],{"type":61,"tag":83,"props":162,"children":163},{},[164],{"type":67,"value":165},"Deploying, repairing, or configuring services",{"type":67,"value":167}," (compose, helm, NIM env vars) → use the ",{"type":61,"tag":83,"props":169,"children":170},{},[171],{"type":67,"value":172},"rag-blueprint",{"type":67,"value":157},{"type":61,"tag":141,"props":175,"children":176},{},[177,182],{"type":61,"tag":83,"props":178,"children":179},{},[180],{"type":67,"value":181},"Production monitoring \u002F alerting",{"type":67,"value":183}," — rag-perf is a one-shot benchmark tool.",{"type":61,"tag":141,"props":185,"children":186},{},[187,192],{"type":61,"tag":83,"props":188,"children":189},{},[190],{"type":67,"value":191},"Runtime requirement:",{"type":67,"value":193}," a deployed RAG server reachable on the network.",{"type":61,"tag":70,"props":195,"children":197},{"id":196},"prerequisites",[198],{"type":67,"value":199},"Prerequisites",{"type":61,"tag":137,"props":201,"children":202},{},[203,215,234,247,268,305,333],{"type":61,"tag":141,"props":204,"children":205},{},[206,208,213],{"type":67,"value":207},"Repo cloned; ",{"type":61,"tag":83,"props":209,"children":210},{},[211],{"type":67,"value":212},"run commands from the repo root",{"type":67,"value":214}," (config paths in the presets are repo-root-relative).",{"type":61,"tag":141,"props":216,"children":217},{},[218,220,225,227,232],{"type":67,"value":219},"Python ",{"type":61,"tag":83,"props":221,"children":222},{},[223],{"type":67,"value":224},"3.11+",{"type":67,"value":226}," and ",{"type":61,"tag":83,"props":228,"children":229},{},[230],{"type":67,"value":231},"uv",{"type":67,"value":233}," on PATH.",{"type":61,"tag":141,"props":235,"children":236},{},[237,239,245],{"type":67,"value":238},"Install rag-perf into its own uv-managed venv: ",{"type":61,"tag":98,"props":240,"children":242},{"className":241},[],[243],{"type":67,"value":244},"uv sync --project scripts\u002Frag-perf",{"type":67,"value":246},".",{"type":61,"tag":141,"props":248,"children":249},{},[250,252,258,260,266],{"type":67,"value":251},"For unit tests: install dev extras as well — ",{"type":61,"tag":98,"props":253,"children":255},{"className":254},[],[256],{"type":67,"value":257},"uv sync --project scripts\u002Frag-perf --extra dev",{"type":67,"value":259}," (otherwise ",{"type":61,"tag":98,"props":261,"children":263},{"className":262},[],[264],{"type":67,"value":265},"pytest-asyncio",{"type":67,"value":267}," is missing and async tests error out at collection time).",{"type":61,"tag":141,"props":269,"children":270},{},[271,273,279,281,287,289,295,297,303],{"type":67,"value":272},"A reachable RAG server (default ",{"type":61,"tag":98,"props":274,"children":276},{"className":275},[],[277],{"type":67,"value":278},"http:\u002F\u002Flocalhost:8081",{"type":67,"value":280},"). For the aiperf phase, the bundled ",{"type":61,"tag":98,"props":282,"children":284},{"className":283},[],[285],{"type":67,"value":286},"nvidia_rag",{"type":67,"value":288}," endpoint plugin must be installed — ",{"type":61,"tag":98,"props":290,"children":292},{"className":291},[],[293],{"type":67,"value":294},"pip install -e .\u002Fscripts\u002Frag-perf",{"type":67,"value":296}," registers it via the ",{"type":61,"tag":98,"props":298,"children":300},{"className":299},[],[301],{"type":67,"value":302},"aiperf.plugins",{"type":67,"value":304}," entry point.",{"type":61,"tag":141,"props":306,"children":307},{},[308,310,315,317,323,325,331],{"type":67,"value":309},"For ",{"type":61,"tag":83,"props":311,"children":312},{},[313],{"type":67,"value":314},"synthetic",{"type":67,"value":316}," queries: an OpenAI-compatible chat-completions endpoint reachable at ",{"type":61,"tag":98,"props":318,"children":320},{"className":319},[],[321],{"type":67,"value":322},"synthetic.llm_url",{"type":67,"value":324}," (default ",{"type":61,"tag":98,"props":326,"children":328},{"className":327},[],[329],{"type":67,"value":330},"http:\u002F\u002Flocalhost:8999\u002Fv1\u002Fchat\u002Fcompletions",{"type":67,"value":332},").",{"type":61,"tag":141,"props":334,"children":335},{},[336,338,344],{"type":67,"value":337},"rag-perf itself runs without ",{"type":61,"tag":98,"props":339,"children":341},{"className":340},[],[342],{"type":67,"value":343},"NVIDIA_API_KEY",{"type":67,"value":345}," (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.",{"type":61,"tag":70,"props":347,"children":349},{"id":348},"instructions",[350],{"type":67,"value":351},"Instructions",{"type":61,"tag":353,"props":354,"children":355},"ol",{},[356,446,487,576,624,681,717],{"type":61,"tag":141,"props":357,"children":358},{},[359,364,366,377,379],{"type":61,"tag":83,"props":360,"children":361},{},[362],{"type":67,"value":363},"Pick a preset.",{"type":67,"value":365}," The three under ",{"type":61,"tag":367,"props":368,"children":370},"a",{"href":369},"..\u002F..\u002Fscripts\u002Frag-perf\u002Fconfigs",[371],{"type":61,"tag":98,"props":372,"children":374},{"className":373},[],[375],{"type":67,"value":376},"scripts\u002Frag-perf\u002Fconfigs\u002F",{"type":67,"value":378}," are:",{"type":61,"tag":137,"props":380,"children":381},{},[382,393,404],{"type":61,"tag":141,"props":383,"children":384},{},[385,391],{"type":61,"tag":98,"props":386,"children":388},{"className":387},[],[389],{"type":67,"value":390},"quick_profile.yaml",{"type":67,"value":392}," — profile-only, ~30 s. Skips load test. For fast iteration on retrieval \u002F reranker tuning.",{"type":61,"tag":141,"props":394,"children":395},{},[396,402],{"type":61,"tag":98,"props":397,"children":399},{"className":398},[],[400],{"type":67,"value":401},"single_run.yaml",{"type":67,"value":403}," — one concurrency level, profiling + aiperf, ~2 min. Regression checks.",{"type":61,"tag":141,"props":405,"children":406},{},[407,413,415,421,423,429,430,436,438,444],{"type":61,"tag":98,"props":408,"children":410},{"className":409},[],[411],{"type":67,"value":412},"sweep.yaml",{"type":67,"value":414}," — multi-axis sweep. ",{"type":61,"tag":98,"props":416,"children":418},{"className":417},[],[419],{"type":67,"value":420},"load.concurrency",{"type":67,"value":422},", ",{"type":61,"tag":98,"props":424,"children":426},{"className":425},[],[427],{"type":67,"value":428},"rag.vdb_top_k",{"type":67,"value":422},{"type":61,"tag":98,"props":431,"children":433},{"className":432},[],[434],{"type":67,"value":435},"rag.reranker_top_k",{"type":67,"value":437}," are all ",{"type":61,"tag":98,"props":439,"children":441},{"className":440},[],[442],{"type":67,"value":443},"int | list[int]",{"type":67,"value":445},"; any of them as a list becomes a sweep axis (Cartesian product).",{"type":61,"tag":141,"props":447,"children":448},{},[449,454,456,461,463,469,471,477,479,485],{"type":61,"tag":83,"props":450,"children":451},{},[452],{"type":67,"value":453},"Edit the preset.",{"type":67,"value":455}," ",{"type":61,"tag":83,"props":457,"children":458},{},[459],{"type":67,"value":460},"Required:",{"type":67,"value":462}," replace ",{"type":61,"tag":98,"props":464,"children":466},{"className":465},[],[467],{"type":67,"value":468},"rag.collection_names: [\"\u003Ccollection_name>\"]",{"type":67,"value":470}," with a real collection on the deployed ingestor server. Verify the collection exists via ",{"type":61,"tag":98,"props":472,"children":474},{"className":473},[],[475],{"type":67,"value":476},"GET \u002Fv1\u002Fcollections",{"type":67,"value":478}," on the ingestor. The placeholder ",{"type":61,"tag":98,"props":480,"children":482},{"className":481},[],[483],{"type":67,"value":484},"\u003Ccollection_name>",{"type":67,"value":486}," validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.",{"type":61,"tag":141,"props":488,"children":489},{},[490,495,497,550,554,556,562,564,569,570,575],{"type":61,"tag":83,"props":491,"children":492},{},[493],{"type":67,"value":494},"Run.",{"type":67,"value":496}," From repo root:",{"type":61,"tag":498,"props":499,"children":504},"pre",{"className":500,"code":501,"language":502,"meta":503,"style":503},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fsingle_run.yaml\n","bash","",[505],{"type":61,"tag":98,"props":506,"children":507},{"__ignoreMap":503},[508],{"type":61,"tag":509,"props":510,"children":513},"span",{"class":511,"line":512},"line",1,[514,519,525,530,535,540,545],{"type":61,"tag":509,"props":515,"children":517},{"style":516},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[518],{"type":67,"value":231},{"type":61,"tag":509,"props":520,"children":522},{"style":521},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[523],{"type":67,"value":524}," run",{"type":61,"tag":509,"props":526,"children":527},{"style":521},[528],{"type":67,"value":529}," --project",{"type":61,"tag":509,"props":531,"children":532},{"style":521},[533],{"type":67,"value":534}," scripts\u002Frag-perf",{"type":61,"tag":509,"props":536,"children":537},{"style":521},[538],{"type":67,"value":539}," rag-perf",{"type":61,"tag":509,"props":541,"children":542},{"style":521},[543],{"type":67,"value":544}," -c",{"type":61,"tag":509,"props":546,"children":547},{"style":521},[548],{"type":67,"value":549}," scripts\u002Frag-perf\u002Fconfigs\u002Fsingle_run.yaml\n",{"type":61,"tag":551,"props":552,"children":553},"br",{},[],{"type":67,"value":555},"Same form for the other presets. The CLI accepts only ",{"type":61,"tag":98,"props":557,"children":559},{"className":558},[],[560],{"type":67,"value":561},"-c \u002F --config",{"type":67,"value":563}," (required), ",{"type":61,"tag":98,"props":565,"children":567},{"className":566},[],[568],{"type":67,"value":111},{"type":67,"value":422},{"type":61,"tag":98,"props":571,"children":573},{"className":572},[],[574],{"type":67,"value":119},{"type":67,"value":246},{"type":61,"tag":141,"props":577,"children":578},{},[579,584,586,591,593,598,600,605,607,612,614,623],{"type":61,"tag":83,"props":580,"children":581},{},[582],{"type":67,"value":583},"Read stdout.",{"type":67,"value":585}," Every invocation prints, in order: a startup banner, a one-line summary, the ",{"type":61,"tag":83,"props":587,"children":588},{},[589],{"type":67,"value":590},"fully resolved config as YAML",{"type":67,"value":592}," (so the run is reproducible from terminal output), per-grid-point progress with the ",{"type":61,"tag":83,"props":594,"children":595},{},[596],{"type":67,"value":597},"shlex-joined aiperf command",{"type":67,"value":599}," in copy-pastable form, a ",{"type":61,"tag":83,"props":601,"children":602},{},[603],{"type":67,"value":604},"rich per-point summary table",{"type":67,"value":606}," (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a ",{"type":61,"tag":83,"props":608,"children":609},{},[610],{"type":67,"value":611},"side-by-side comparison table",{"type":67,"value":613}," auto-labelled by whichever axis varied. See ",{"type":61,"tag":367,"props":615,"children":617},{"href":616},"references\u002Foutput-and-analysis.md",[618],{"type":61,"tag":98,"props":619,"children":621},{"className":620},[],[622],{"type":67,"value":616},{"type":67,"value":246},{"type":61,"tag":141,"props":625,"children":626},{},[627,632,634,640,642,648,650,658,660,666,667,673,674,680],{"type":61,"tag":83,"props":628,"children":629},{},[630],{"type":67,"value":631},"Inspect artifacts.",{"type":67,"value":633}," Layout depends on run shape — flat for single-point + ",{"type":61,"tag":98,"props":635,"children":637},{"className":636},[],[638],{"type":67,"value":639},"iterations=1",{"type":67,"value":641},", nested under ",{"type":61,"tag":98,"props":643,"children":645},{"className":644},[],[646],{"type":67,"value":647},"iter_\u003Ci>\u002F\u003Cpoint>\u002F...",{"type":67,"value":649}," otherwise. See ",{"type":61,"tag":367,"props":651,"children":652},{"href":616},[653],{"type":61,"tag":98,"props":654,"children":656},{"className":655},[],[657],{"type":67,"value":616},{"type":67,"value":659}," for the full directory tree, file purposes, and how to parse ",{"type":61,"tag":98,"props":661,"children":663},{"className":662},[],[664],{"type":67,"value":665},"results.json",{"type":67,"value":113},{"type":61,"tag":98,"props":668,"children":670},{"className":669},[],[671],{"type":67,"value":672},"results.csv",{"type":67,"value":113},{"type":61,"tag":98,"props":675,"children":677},{"className":676},[],[678],{"type":67,"value":679},"report.md",{"type":67,"value":246},{"type":61,"tag":141,"props":682,"children":683},{},[684,689,691,700,702,707,709,715],{"type":61,"tag":83,"props":685,"children":686},{},[687],{"type":67,"value":688},"Summarise for the user.",{"type":67,"value":690}," When reporting back, follow the playbook in ",{"type":61,"tag":367,"props":692,"children":694},{"href":693},"references\u002Foutput-and-analysis.md#summarising-results-to-the-user",[695],{"type":61,"tag":98,"props":696,"children":698},{"className":697},[],[699],{"type":67,"value":693},{"type":67,"value":701},": pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, ",{"type":61,"tag":83,"props":703,"children":704},{},[705],{"type":67,"value":706},"always flag",{"type":67,"value":708}," zero citations \u002F non-zero error rate \u002F suspect ",{"type":61,"tag":98,"props":710,"children":712},{"className":711},[],[713],{"type":67,"value":714},"llm_ttft_ms",{"type":67,"value":716}," \u002F small-sample p99, and propose a concrete next-experiment YAML.",{"type":61,"tag":141,"props":718,"children":719},{},[720,725,727,737,739,745,747,753,755,761],{"type":61,"tag":83,"props":721,"children":722},{},[723],{"type":67,"value":724},"Tune.",{"type":67,"value":726}," Schema is fully documented in ",{"type":61,"tag":367,"props":728,"children":730},{"href":729},"..\u002F..\u002Fdocs\u002Fperformance-benchmarking.md",[731],{"type":61,"tag":98,"props":732,"children":734},{"className":733},[],[735],{"type":67,"value":736},"docs\u002Fperformance-benchmarking.md",{"type":67,"value":738}," and the deeper-dive references below. Common knobs: turn ",{"type":61,"tag":98,"props":740,"children":742},{"className":741},[],[743],{"type":67,"value":744},"aiperf.enabled: false",{"type":67,"value":746}," for profile-only mode, increase ",{"type":61,"tag":98,"props":748,"children":750},{"className":749},[],[751],{"type":67,"value":752},"load.iterations",{"type":67,"value":754}," for variance estimation, set ",{"type":61,"tag":98,"props":756,"children":758},{"className":757},[],[759],{"type":67,"value":760},"load.sleep_between_points_s: 60",{"type":67,"value":762}," for overnight Cartesian sweeps.",{"type":61,"tag":70,"props":764,"children":766},{"id":765},"examples",[767],{"type":67,"value":768},"Examples",{"type":61,"tag":77,"props":770,"children":771},{},[772],{"type":61,"tag":83,"props":773,"children":774},{},[775],{"type":67,"value":776},"Profile-only (quickest signal on retrieval \u002F reranker tuning):",{"type":61,"tag":498,"props":778,"children":780},{"className":500,"code":779,"language":502,"meta":503,"style":503},"uv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fquick_profile.yaml\n",[781],{"type":61,"tag":98,"props":782,"children":783},{"__ignoreMap":503},[784],{"type":61,"tag":509,"props":785,"children":786},{"class":511,"line":512},[787,791,795,799,803,807,811],{"type":61,"tag":509,"props":788,"children":789},{"style":516},[790],{"type":67,"value":231},{"type":61,"tag":509,"props":792,"children":793},{"style":521},[794],{"type":67,"value":524},{"type":61,"tag":509,"props":796,"children":797},{"style":521},[798],{"type":67,"value":529},{"type":61,"tag":509,"props":800,"children":801},{"style":521},[802],{"type":67,"value":534},{"type":61,"tag":509,"props":804,"children":805},{"style":521},[806],{"type":67,"value":539},{"type":61,"tag":509,"props":808,"children":809},{"style":521},[810],{"type":67,"value":544},{"type":61,"tag":509,"props":812,"children":813},{"style":521},[814],{"type":67,"value":815}," scripts\u002Frag-perf\u002Fconfigs\u002Fquick_profile.yaml\n",{"type":61,"tag":77,"props":817,"children":818},{},[819,821,827,829,835,837,843,845,850],{"type":67,"value":820},"Output: ",{"type":61,"tag":98,"props":822,"children":824},{"className":823},[],[825],{"type":67,"value":826},"rag-perf-results\u002Fquick_profile\u002Frun_\u003Cts>\u002F{profile_report.md, profile_results.json, profiling\u002F}",{"type":67,"value":828},". The ",{"type":61,"tag":98,"props":830,"children":832},{"className":831},[],[833],{"type":67,"value":834},"aiperf_rag_on\u002F",{"type":67,"value":836}," directory is omitted. Filenames are ",{"type":61,"tag":98,"props":838,"children":840},{"className":839},[],[841],{"type":67,"value":842},"profile_*",{"type":67,"value":844}," because ",{"type":61,"tag":98,"props":846,"children":848},{"className":847},[],[849],{"type":67,"value":744},{"type":67,"value":246},{"type":61,"tag":77,"props":852,"children":853},{},[854],{"type":61,"tag":83,"props":855,"children":856},{},[857],{"type":67,"value":858},"Single benchmark point with full report:",{"type":61,"tag":498,"props":860,"children":861},{"className":500,"code":501,"language":502,"meta":503,"style":503},[862],{"type":61,"tag":98,"props":863,"children":864},{"__ignoreMap":503},[865],{"type":61,"tag":509,"props":866,"children":867},{"class":511,"line":512},[868,872,876,880,884,888,892],{"type":61,"tag":509,"props":869,"children":870},{"style":516},[871],{"type":67,"value":231},{"type":61,"tag":509,"props":873,"children":874},{"style":521},[875],{"type":67,"value":524},{"type":61,"tag":509,"props":877,"children":878},{"style":521},[879],{"type":67,"value":529},{"type":61,"tag":509,"props":881,"children":882},{"style":521},[883],{"type":67,"value":534},{"type":61,"tag":509,"props":885,"children":886},{"style":521},[887],{"type":67,"value":539},{"type":61,"tag":509,"props":889,"children":890},{"style":521},[891],{"type":67,"value":544},{"type":61,"tag":509,"props":893,"children":894},{"style":521},[895],{"type":67,"value":549},{"type":61,"tag":77,"props":897,"children":898},{},[899,901,907],{"type":67,"value":900},"Output: flat ",{"type":61,"tag":98,"props":902,"children":904},{"className":903},[],[905],{"type":67,"value":906},"run_\u003Cts>\u002F{report.md, results.json, results.csv, profiling\u002F, aiperf_rag_on\u002F}",{"type":67,"value":246},{"type":61,"tag":77,"props":909,"children":910},{},[911],{"type":61,"tag":83,"props":912,"children":913},{},[914],{"type":67,"value":915},"Concurrency sweep:",{"type":61,"tag":498,"props":917,"children":919},{"className":500,"code":918,"language":502,"meta":503,"style":503},"uv run --project scripts\u002Frag-perf rag-perf -c scripts\u002Frag-perf\u002Fconfigs\u002Fsweep.yaml\n",[920],{"type":61,"tag":98,"props":921,"children":922},{"__ignoreMap":503},[923],{"type":61,"tag":509,"props":924,"children":925},{"class":511,"line":512},[926,930,934,938,942,946,950],{"type":61,"tag":509,"props":927,"children":928},{"style":516},[929],{"type":67,"value":231},{"type":61,"tag":509,"props":931,"children":932},{"style":521},[933],{"type":67,"value":524},{"type":61,"tag":509,"props":935,"children":936},{"style":521},[937],{"type":67,"value":529},{"type":61,"tag":509,"props":939,"children":940},{"style":521},[941],{"type":67,"value":534},{"type":61,"tag":509,"props":943,"children":944},{"style":521},[945],{"type":67,"value":539},{"type":61,"tag":509,"props":947,"children":948},{"style":521},[949],{"type":67,"value":544},{"type":61,"tag":509,"props":951,"children":952},{"style":521},[953],{"type":67,"value":954}," scripts\u002Frag-perf\u002Fconfigs\u002Fsweep.yaml\n",{"type":61,"tag":77,"props":956,"children":957},{},[958,960,966,968,973,974,979,980,985],{"type":67,"value":959},"Output: nested ",{"type":61,"tag":98,"props":961,"children":963},{"className":962},[],[964],{"type":67,"value":965},"run_\u003Cts>\u002Fiter_1\u002F\u003CCR:_VDB-K:_RERANKER-K:_…>\u002F{profiling,aiperf_rag_on}\u002F",{"type":67,"value":967}," per point, plus aggregate ",{"type":61,"tag":98,"props":969,"children":971},{"className":970},[],[972],{"type":67,"value":679},{"type":67,"value":113},{"type":61,"tag":98,"props":975,"children":977},{"className":976},[],[978],{"type":67,"value":665},{"type":67,"value":113},{"type":61,"tag":98,"props":981,"children":983},{"className":982},[],[984],{"type":67,"value":672},{"type":67,"value":986}," at the run root.",{"type":61,"tag":77,"props":988,"children":989},{},[990],{"type":61,"tag":83,"props":991,"children":992},{},[993],{"type":67,"value":994},"Run unit tests:",{"type":61,"tag":498,"props":996,"children":998},{"className":500,"code":997,"language":502,"meta":503,"style":503},"uv sync --project scripts\u002Frag-perf --extra dev   # one-time, installs pytest-asyncio\nuv run --project scripts\u002Frag-perf python -m pytest tests\u002Funit\u002Ftest_rag_perf\u002F\n",[999],{"type":61,"tag":98,"props":1000,"children":1001},{"__ignoreMap":503},[1002,1038],{"type":61,"tag":509,"props":1003,"children":1004},{"class":511,"line":512},[1005,1009,1014,1018,1022,1027,1032],{"type":61,"tag":509,"props":1006,"children":1007},{"style":516},[1008],{"type":67,"value":231},{"type":61,"tag":509,"props":1010,"children":1011},{"style":521},[1012],{"type":67,"value":1013}," sync",{"type":61,"tag":509,"props":1015,"children":1016},{"style":521},[1017],{"type":67,"value":529},{"type":61,"tag":509,"props":1019,"children":1020},{"style":521},[1021],{"type":67,"value":534},{"type":61,"tag":509,"props":1023,"children":1024},{"style":521},[1025],{"type":67,"value":1026}," --extra",{"type":61,"tag":509,"props":1028,"children":1029},{"style":521},[1030],{"type":67,"value":1031}," dev",{"type":61,"tag":509,"props":1033,"children":1035},{"style":1034},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1036],{"type":67,"value":1037},"   # one-time, installs pytest-asyncio\n",{"type":61,"tag":509,"props":1039,"children":1041},{"class":511,"line":1040},2,[1042,1046,1050,1054,1058,1063,1068,1073],{"type":61,"tag":509,"props":1043,"children":1044},{"style":516},[1045],{"type":67,"value":231},{"type":61,"tag":509,"props":1047,"children":1048},{"style":521},[1049],{"type":67,"value":524},{"type":61,"tag":509,"props":1051,"children":1052},{"style":521},[1053],{"type":67,"value":529},{"type":61,"tag":509,"props":1055,"children":1056},{"style":521},[1057],{"type":67,"value":534},{"type":61,"tag":509,"props":1059,"children":1060},{"style":521},[1061],{"type":67,"value":1062}," python",{"type":61,"tag":509,"props":1064,"children":1065},{"style":521},[1066],{"type":67,"value":1067}," -m",{"type":61,"tag":509,"props":1069,"children":1070},{"style":521},[1071],{"type":67,"value":1072}," pytest",{"type":61,"tag":509,"props":1074,"children":1075},{"style":521},[1076],{"type":67,"value":1077}," tests\u002Funit\u002Ftest_rag_perf\u002F\n",{"type":61,"tag":70,"props":1079,"children":1081},{"id":1080},"limitations",[1082],{"type":67,"value":1083},"Limitations",{"type":61,"tag":137,"props":1085,"children":1086},{},[1087,1099,1128,1153,1181,1193,1222,1242],{"type":61,"tag":141,"props":1088,"children":1089},{},[1090,1092,1097],{"type":67,"value":1091},"The CLI is ",{"type":61,"tag":83,"props":1093,"children":1094},{},[1095],{"type":67,"value":1096},"config-only",{"type":67,"value":1098},": author or copy YAML to vary a parameter.",{"type":61,"tag":141,"props":1100,"children":1101},{},[1102,1107,1108,1113,1114,1119,1121,1126],{"type":61,"tag":98,"props":1103,"children":1105},{"className":1104},[],[1106],{"type":67,"value":420},{"type":67,"value":113},{"type":61,"tag":98,"props":1109,"children":1111},{"className":1110},[],[1112],{"type":67,"value":428},{"type":67,"value":113},{"type":61,"tag":98,"props":1115,"children":1117},{"className":1116},[],[1118],{"type":67,"value":435},{"type":67,"value":1120}," accept ",{"type":61,"tag":98,"props":1122,"children":1124},{"className":1123},[],[1125],{"type":67,"value":443},{"type":67,"value":1127},"; the validator requires unique list values because each value names a unique point dir.",{"type":61,"tag":141,"props":1129,"children":1130},{},[1131,1137,1138,1144,1146,1151],{"type":61,"tag":98,"props":1132,"children":1134},{"className":1133},[],[1135],{"type":67,"value":1136},"input.file",{"type":67,"value":226},{"type":61,"tag":98,"props":1139,"children":1141},{"className":1140},[],[1142],{"type":67,"value":1143},"input.synthetic",{"type":67,"value":1145}," follow an XOR rule — both set fails validation. When neither is set, ",{"type":61,"tag":98,"props":1147,"children":1149},{"className":1148},[],[1150],{"type":67,"value":314},{"type":67,"value":1152}," auto-fills with defaults so a bare config still validates.",{"type":61,"tag":141,"props":1154,"children":1155},{},[1156,1158,1163,1165,1171,1173,1179],{"type":67,"value":1157},"File-based input format is ",{"type":61,"tag":83,"props":1159,"children":1160},{},[1161],{"type":67,"value":1162},"inferred from extension only",{"type":67,"value":1164}," (",{"type":61,"tag":98,"props":1166,"children":1168},{"className":1167},[],[1169],{"type":67,"value":1170},".jsonl",{"type":67,"value":1172}," or ",{"type":61,"tag":98,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":67,"value":1178},".csv",{"type":67,"value":1180},"); other extensions are rejected.",{"type":61,"tag":141,"props":1182,"children":1183},{},[1184,1186,1191],{"type":67,"value":1185},"Synthetic generation streams each query to disk as it completes (failure-resilient) but ",{"type":61,"tag":83,"props":1187,"children":1188},{},[1189],{"type":67,"value":1190},"fails fast on the first LLM error",{"type":67,"value":1192}," — partial JSONL is preserved. Re-run after fixing the endpoint.",{"type":61,"tag":141,"props":1194,"children":1195},{},[1196,1198,1204,1206,1212,1214,1220],{"type":67,"value":1197},"Reasoning models (Nemotron Omni, Qwen-Reasoning) require ",{"type":61,"tag":98,"props":1199,"children":1201},{"className":1200},[],[1202],{"type":67,"value":1203},"synthetic.disable_thinking: true",{"type":67,"value":1205}," (the default). Without it the model exhausts the token budget on chain-of-thought and ",{"type":61,"tag":98,"props":1207,"children":1209},{"className":1208},[],[1210],{"type":67,"value":1211},"content",{"type":67,"value":1213}," returns empty — the generator now raises with a clear message instead of substituting ",{"type":61,"tag":98,"props":1215,"children":1217},{"className":1216},[],[1218],{"type":67,"value":1219},"reasoning_content",{"type":67,"value":1221}," for the answer.",{"type":61,"tag":141,"props":1223,"children":1224},{},[1225,1227,1233,1235,1241],{"type":67,"value":1226},"aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing ",{"type":61,"tag":98,"props":1228,"children":1230},{"className":1229},[],[1231],{"type":67,"value":1232},"AiperfRunner._base_aiperf_cmd",{"type":67,"value":1234}," in ",{"type":61,"tag":98,"props":1236,"children":1238},{"className":1237},[],[1239],{"type":67,"value":1240},"scripts\u002Frag-perf\u002Frag_perf\u002Frunner.py",{"type":67,"value":246},{"type":61,"tag":141,"props":1243,"children":1244},{},[1245,1247,1256],{"type":67,"value":1246},"Procedural detail lives under ",{"type":61,"tag":83,"props":1248,"children":1249},{},[1250],{"type":61,"tag":98,"props":1251,"children":1253},{"className":1252},[],[1254],{"type":67,"value":1255},"references\u002F",{"type":67,"value":1257}," to keep this file concise.",{"type":61,"tag":70,"props":1259,"children":1261},{"id":1260},"troubleshooting",[1262],{"type":67,"value":1263},"Troubleshooting",{"type":61,"tag":1265,"props":1266,"children":1267},"table",{},[1268,1292],{"type":61,"tag":1269,"props":1270,"children":1271},"thead",{},[1272],{"type":61,"tag":1273,"props":1274,"children":1275},"tr",{},[1276,1282,1287],{"type":61,"tag":1277,"props":1278,"children":1279},"th",{},[1280],{"type":67,"value":1281},"Error \u002F signal",{"type":61,"tag":1277,"props":1283,"children":1284},{},[1285],{"type":67,"value":1286},"Likely cause",{"type":61,"tag":1277,"props":1288,"children":1289},{},[1290],{"type":67,"value":1291},"What to do",{"type":61,"tag":1293,"props":1294,"children":1295},"tbody",{},[1296,1332,1365,1393,1421,1457,1503,1537,1582,1628,1656],{"type":61,"tag":1273,"props":1297,"children":1298},{},[1299,1309,1327],{"type":61,"tag":1300,"props":1301,"children":1302},"td",{},[1303],{"type":61,"tag":98,"props":1304,"children":1306},{"className":1305},[],[1307],{"type":67,"value":1308},"Configuration errors in \u003Cyaml>:  •  input  —  ... XOR rule",{"type":61,"tag":1300,"props":1310,"children":1311},{},[1312,1314,1319,1320,1325],{"type":67,"value":1313},"Both ",{"type":61,"tag":98,"props":1315,"children":1317},{"className":1316},[],[1318],{"type":67,"value":1136},{"type":67,"value":226},{"type":61,"tag":98,"props":1321,"children":1323},{"className":1322},[],[1324],{"type":67,"value":1143},{"type":67,"value":1326}," set",{"type":61,"tag":1300,"props":1328,"children":1329},{},[1330],{"type":67,"value":1331},"Pick one. The XOR validator runs at YAML load time.",{"type":61,"tag":1273,"props":1333,"children":1334},{},[1335,1344,1360],{"type":61,"tag":1300,"props":1336,"children":1337},{},[1338],{"type":61,"tag":98,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":67,"value":1343},"input.file must end in .jsonl or .csv",{"type":61,"tag":1300,"props":1345,"children":1346},{},[1347,1349,1354,1355],{"type":67,"value":1348},"Extension other than ",{"type":61,"tag":98,"props":1350,"children":1352},{"className":1351},[],[1353],{"type":67,"value":1170},{"type":67,"value":113},{"type":61,"tag":98,"props":1356,"children":1358},{"className":1357},[],[1359],{"type":67,"value":1178},{"type":61,"tag":1300,"props":1361,"children":1362},{},[1363],{"type":67,"value":1364},"Rename or convert.",{"type":61,"tag":1273,"props":1366,"children":1367},{},[1368,1377,1388],{"type":61,"tag":1300,"props":1369,"children":1370},{},[1371],{"type":61,"tag":98,"props":1372,"children":1374},{"className":1373},[],[1375],{"type":67,"value":1376},"load.concurrency has duplicate values",{"type":61,"tag":1300,"props":1378,"children":1379},{},[1380,1382],{"type":67,"value":1381},"e.g. ",{"type":61,"tag":98,"props":1383,"children":1385},{"className":1384},[],[1386],{"type":67,"value":1387},"[2, 2, 4]",{"type":61,"tag":1300,"props":1389,"children":1390},{},[1391],{"type":67,"value":1392},"Each concurrency maps to a unique point dir; dedupe.",{"type":61,"tag":1273,"props":1394,"children":1395},{},[1396,1405,1416],{"type":61,"tag":1300,"props":1397,"children":1398},{},[1399],{"type":61,"tag":98,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":67,"value":1404},"warmup_requests must be >= 1",{"type":61,"tag":1300,"props":1406,"children":1407},{},[1408,1410],{"type":67,"value":1409},"YAML had ",{"type":61,"tag":98,"props":1411,"children":1413},{"className":1412},[],[1414],{"type":67,"value":1415},"warmup_requests: 0",{"type":61,"tag":1300,"props":1417,"children":1418},{},[1419],{"type":67,"value":1420},"aiperf rejects warmup=0; minimum is 1.",{"type":61,"tag":1273,"props":1422,"children":1423},{},[1424,1433,1438],{"type":61,"tag":1300,"props":1425,"children":1426},{},[1427],{"type":61,"tag":98,"props":1428,"children":1430},{"className":1429},[],[1431],{"type":67,"value":1432},"LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true).",{"type":61,"tag":1300,"props":1434,"children":1435},{},[1436],{"type":67,"value":1437},"Reasoning model used CoT and ran out of tokens",{"type":61,"tag":1300,"props":1439,"children":1440},{},[1441,1443,1448,1450,1456],{"type":67,"value":1442},"Set ",{"type":61,"tag":98,"props":1444,"children":1446},{"className":1445},[],[1447],{"type":67,"value":1203},{"type":67,"value":1449}," (the default) or raise ",{"type":61,"tag":98,"props":1451,"children":1453},{"className":1452},[],[1454],{"type":67,"value":1455},"min_query_tokens",{"type":67,"value":246},{"type":61,"tag":1273,"props":1458,"children":1459},{},[1460,1471,1476],{"type":61,"tag":1300,"props":1461,"children":1462},{},[1463,1469],{"type":61,"tag":98,"props":1464,"children":1466},{"className":1465},[],[1467],{"type":67,"value":1468},"✗ All N profiling requests failed across M point(s).",{"type":67,"value":1470}," + exit 1",{"type":61,"tag":1300,"props":1472,"children":1473},{},[1474],{"type":67,"value":1475},"Bad URL, server down, wrong collection",{"type":61,"tag":1300,"props":1477,"children":1478},{},[1479,1481,1487,1488,1494,1496,1501],{"type":67,"value":1480},"Verify ",{"type":61,"tag":98,"props":1482,"children":1484},{"className":1483},[],[1485],{"type":67,"value":1486},"target.url",{"type":67,"value":422},{"type":61,"tag":98,"props":1489,"children":1491},{"className":1490},[],[1492],{"type":67,"value":1493},"rag.collection_names",{"type":67,"value":1495}," (the ",{"type":61,"tag":98,"props":1497,"children":1499},{"className":1498},[],[1500],{"type":67,"value":484},{"type":67,"value":1502}," placeholder will hit this).",{"type":61,"tag":1273,"props":1504,"children":1505},{},[1506,1519,1524],{"type":61,"tag":1300,"props":1507,"children":1508},{},[1509,1511,1517],{"type":67,"value":1510},"Per-iteration ",{"type":61,"tag":98,"props":1512,"children":1514},{"className":1513},[],[1515],{"type":67,"value":1516},"⚠ N profiling requests failed",{"type":67,"value":1518}," warning, run continues",{"type":61,"tag":1300,"props":1520,"children":1521},{},[1522],{"type":67,"value":1523},"Some requests timed out \u002F errored mid-run",{"type":61,"tag":1300,"props":1525,"children":1526},{},[1527,1529,1535],{"type":67,"value":1528},"Check rag-server logs, raise ",{"type":61,"tag":98,"props":1530,"children":1532},{"className":1531},[],[1533],{"type":67,"value":1534},"target.timeout_s",{"type":67,"value":1536},", drop concurrency.",{"type":61,"tag":1273,"props":1538,"children":1539},{},[1540,1549,1554],{"type":61,"tag":1300,"props":1541,"children":1542},{},[1543],{"type":61,"tag":98,"props":1544,"children":1546},{"className":1545},[],[1547],{"type":67,"value":1548},"RuntimeError: Random synthetic query generation failed at query N: ...",{"type":61,"tag":1300,"props":1550,"children":1551},{},[1552],{"type":67,"value":1553},"LLM endpoint rejected a request mid-generation",{"type":61,"tag":1300,"props":1555,"children":1556},{},[1557,1559,1565,1567,1573,1575,1580],{"type":67,"value":1558},"Partial JSONL is at ",{"type":61,"tag":98,"props":1560,"children":1562},{"className":1561},[],[1563],{"type":67,"value":1564},"synthetic.jsonl_output_path",{"type":67,"value":1566},"; fix endpoint and re-run with reduced ",{"type":61,"tag":98,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":67,"value":1572},"num_queries",{"type":67,"value":1574},", or point ",{"type":61,"tag":98,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":67,"value":1136},{"type":67,"value":1581}," at the partial file.",{"type":61,"tag":1273,"props":1583,"children":1584},{},[1585,1603,1615],{"type":61,"tag":1300,"props":1586,"children":1587},{},[1588,1594,1595,1601],{"type":61,"tag":98,"props":1589,"children":1591},{"className":1590},[],[1592],{"type":67,"value":1593},"Citation count (mean): 0",{"type":67,"value":226},{"type":61,"tag":98,"props":1596,"children":1598},{"className":1597},[],[1599],{"type":67,"value":1600},"Citation relevance score: N\u002FA",{"type":67,"value":1602}," for a non-empty deployment",{"type":61,"tag":1300,"props":1604,"children":1605},{},[1606,1608,1613],{"type":67,"value":1607},"Collection mismatch between ",{"type":61,"tag":98,"props":1609,"children":1611},{"className":1610},[],[1612],{"type":67,"value":1493},{"type":67,"value":1614}," and what's actually ingested",{"type":61,"tag":1300,"props":1616,"children":1617},{},[1618,1620,1626],{"type":67,"value":1619},"Run ",{"type":61,"tag":98,"props":1621,"children":1623},{"className":1622},[],[1624],{"type":67,"value":1625},"curl -s http:\u002F\u002F\u003Cingestor>:8082\u002Fv1\u002Fcollections",{"type":67,"value":1627}," to list real collections.",{"type":61,"tag":1273,"props":1629,"children":1630},{},[1631,1642,1647],{"type":61,"tag":1300,"props":1632,"children":1633},{},[1634,1636],{"type":67,"value":1635},"Tests error with ",{"type":61,"tag":98,"props":1637,"children":1639},{"className":1638},[],[1640],{"type":67,"value":1641},"ModuleNotFoundError: No module named 'pytest_asyncio'",{"type":61,"tag":1300,"props":1643,"children":1644},{},[1645],{"type":67,"value":1646},"Dev extras missing",{"type":61,"tag":1300,"props":1648,"children":1649},{},[1650,1655],{"type":61,"tag":98,"props":1651,"children":1653},{"className":1652},[],[1654],{"type":67,"value":257},{"type":67,"value":246},{"type":61,"tag":1273,"props":1657,"children":1658},{},[1659,1678,1683],{"type":61,"tag":1300,"props":1660,"children":1661},{},[1662,1664,1670,1672],{"type":67,"value":1663},"CI: ",{"type":61,"tag":98,"props":1665,"children":1667},{"className":1666},[],[1668],{"type":67,"value":1669},"ModuleNotFoundError: No module named 'ruamel'",{"type":67,"value":1671}," from ",{"type":61,"tag":98,"props":1673,"children":1675},{"className":1674},[],[1676],{"type":67,"value":1677},"tests\u002Funit\u002Ftest_rag_perf\u002F",{"type":61,"tag":1300,"props":1679,"children":1680},{},[1681],{"type":67,"value":1682},"rag-perf package missing from CI venv",{"type":61,"tag":1300,"props":1684,"children":1685},{},[1686,1688,1694],{"type":67,"value":1687},"Add ",{"type":61,"tag":98,"props":1689,"children":1691},{"className":1690},[],[1692],{"type":67,"value":1693},"uv pip install -e .\u002Fscripts\u002Frag-perf",{"type":67,"value":1695}," after the top-level install in the unit-tests job.",{"type":61,"tag":70,"props":1697,"children":1699},{"id":1698},"gotchas",[1700],{"type":67,"value":1701},"Gotchas",{"type":61,"tag":137,"props":1703,"children":1704},{},[1705,1738,1748,1780,1808,1845,1855,1873,1931,1987],{"type":61,"tag":141,"props":1706,"children":1707},{},[1708,1713,1715,1721,1722,1728,1730,1736],{"type":61,"tag":83,"props":1709,"children":1710},{},[1711],{"type":67,"value":1712},"Run from repo root.",{"type":67,"value":1714}," Preset configs reference ",{"type":61,"tag":98,"props":1716,"children":1718},{"className":1717},[],[1719],{"type":67,"value":1720},"scripts\u002Frag-perf\u002Fexamples\u002Fqueries.jsonl",{"type":67,"value":226},{"type":61,"tag":98,"props":1723,"children":1725},{"className":1724},[],[1726],{"type":67,"value":1727},"scripts\u002Frag-perf\u002Fprompts\u002Fdefault_prompts.yaml",{"type":67,"value":1729}," with repo-root-relative paths. Running from inside ",{"type":61,"tag":98,"props":1731,"children":1733},{"className":1732},[],[1734],{"type":67,"value":1735},"scripts\u002Frag-perf\u002F",{"type":67,"value":1737}," will fail those file lookups.",{"type":61,"tag":141,"props":1739,"children":1740},{},[1741,1746],{"type":61,"tag":83,"props":1742,"children":1743},{},[1744],{"type":67,"value":1745},"CLI is config-only.",{"type":67,"value":1747}," Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.",{"type":61,"tag":141,"props":1749,"children":1750},{},[1751,1763,1765,1771,1773,1778],{"type":61,"tag":83,"props":1752,"children":1753},{},[1754,1756,1761],{"type":67,"value":1755},"Always edit ",{"type":61,"tag":98,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":67,"value":1493},{"type":67,"value":1762}," before the first run.",{"type":67,"value":1764}," The presets ship with ",{"type":61,"tag":98,"props":1766,"children":1768},{"className":1767},[],[1769],{"type":67,"value":1770},"[\"\u003Ccollection_name>\"]",{"type":67,"value":1772}," as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as ",{"type":61,"tag":98,"props":1774,"children":1776},{"className":1775},[],[1777],{"type":67,"value":1593},{"type":67,"value":1779}," everywhere.",{"type":61,"tag":141,"props":1781,"children":1782},{},[1783,1806],{"type":61,"tag":83,"props":1784,"children":1785},{},[1786,1792,1793,1799,1800],{"type":61,"tag":98,"props":1787,"children":1789},{"className":1788},[],[1790],{"type":67,"value":1791},"load.concurrency_list",{"type":67,"value":422},{"type":61,"tag":98,"props":1794,"children":1796},{"className":1795},[],[1797],{"type":67,"value":1798},"rag.vdb_top_k_list",{"type":67,"value":422},{"type":61,"tag":98,"props":1801,"children":1803},{"className":1802},[],[1804],{"type":67,"value":1805},"rag.reranker_top_k_list",{"type":67,"value":1807}," are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.",{"type":61,"tag":141,"props":1809,"children":1810},{},[1811,1821,1823,1829,1830,1836,1837,1843],{"type":61,"tag":83,"props":1812,"children":1813},{},[1814,1819],{"type":61,"tag":98,"props":1815,"children":1817},{"className":1816},[],[1818],{"type":67,"value":744},{"type":67,"value":1820}," changes filenames.",{"type":67,"value":1822}," The top-level outputs become ",{"type":61,"tag":98,"props":1824,"children":1826},{"className":1825},[],[1827],{"type":67,"value":1828},"profile_report.md",{"type":67,"value":113},{"type":61,"tag":98,"props":1831,"children":1833},{"className":1832},[],[1834],{"type":67,"value":1835},"profile_results.json",{"type":67,"value":113},{"type":61,"tag":98,"props":1838,"children":1840},{"className":1839},[],[1841],{"type":67,"value":1842},"profile_results.csv",{"type":67,"value":1844},". The aggregate sweep table also suppresses load-test rows and the \"Optimal throughput\" footer.",{"type":61,"tag":141,"props":1846,"children":1847},{},[1848,1853],{"type":61,"tag":83,"props":1849,"children":1850},{},[1851],{"type":67,"value":1852},"Resolved-config dump is verbose",{"type":67,"value":1854}," (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.",{"type":61,"tag":141,"props":1856,"children":1857},{},[1858,1863,1865,1871],{"type":61,"tag":83,"props":1859,"children":1860},{},[1861],{"type":67,"value":1862},"The aiperf shell command is logged before each subprocess.",{"type":67,"value":1864}," Look for ",{"type":61,"tag":98,"props":1866,"children":1868},{"className":1867},[],[1869],{"type":67,"value":1870},"\\n  $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ...",{"type":67,"value":1872}," in stdout — copy-paste runnable for reproducing a single point outside rag-perf.",{"type":61,"tag":141,"props":1874,"children":1875},{},[1876,1885,1887,1893,1895,1901,1903,1909,1911,1916,1918,1923,1925,1930],{"type":61,"tag":83,"props":1877,"children":1878},{},[1879],{"type":61,"tag":98,"props":1880,"children":1882},{"className":1881},[],[1883],{"type":67,"value":1884},"--endpoint-type nvidia_rag",{"type":67,"value":1886}," comes from the bundled plugin at ",{"type":61,"tag":98,"props":1888,"children":1890},{"className":1889},[],[1891],{"type":67,"value":1892},"scripts\u002Frag-perf\u002Frag_perf\u002Fplugin\u002Fnvidia_rag.py",{"type":67,"value":1894},". It teaches aiperf about the RAG ",{"type":61,"tag":98,"props":1896,"children":1898},{"className":1897},[],[1899],{"type":67,"value":1900},"\u002Fv1\u002Fgenerate",{"type":67,"value":1902}," request shape and parses citations + per-stage ",{"type":61,"tag":98,"props":1904,"children":1906},{"className":1905},[],[1907],{"type":67,"value":1908},"metrics",{"type":67,"value":1910}," out of the SSE stream. If aiperf can't resolve ",{"type":61,"tag":98,"props":1912,"children":1914},{"className":1913},[],[1915],{"type":67,"value":286},{"type":67,"value":1917},", rag-perf needs editable installation in the venv — re-run ",{"type":61,"tag":98,"props":1919,"children":1921},{"className":1920},[],[1922],{"type":67,"value":244},{"type":67,"value":1924}," (or ",{"type":61,"tag":98,"props":1926,"children":1928},{"className":1927},[],[1929],{"type":67,"value":1693},{"type":67,"value":332},{"type":61,"tag":141,"props":1932,"children":1933},{},[1934,1939,1941,1947,1949,1955,1957,1963,1965,1971,1972,1978,1979,1985],{"type":61,"tag":83,"props":1935,"children":1936},{},[1937],{"type":67,"value":1938},"Sweep-mode point-name collision.",{"type":67,"value":1940}," When two points differ only in concurrency (e.g. ",{"type":61,"tag":98,"props":1942,"children":1944},{"className":1943},[],[1945],{"type":67,"value":1946},"[1, 4]",{"type":67,"value":1948}," × single ",{"type":61,"tag":98,"props":1950,"children":1952},{"className":1951},[],[1953],{"type":67,"value":1954},"vdb_top_k",{"type":67,"value":1956},"), the dir name encodes everything: ",{"type":61,"tag":98,"props":1958,"children":1960},{"className":1959},[],[1961],{"type":67,"value":1962},"CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:...",{"type":67,"value":1964},". Cluster \u002F GPU \u002F experiment_name (",{"type":61,"tag":98,"props":1966,"children":1968},{"className":1967},[],[1969],{"type":67,"value":1970},"output.cluster",{"type":67,"value":422},{"type":61,"tag":98,"props":1973,"children":1975},{"className":1974},[],[1976],{"type":67,"value":1977},"output.gpu",{"type":67,"value":422},{"type":61,"tag":98,"props":1980,"children":1982},{"className":1981},[],[1983],{"type":67,"value":1984},"output.experiment_name",{"type":67,"value":1986},") are appended too — useful for diff-friendly artifact paths across machines.",{"type":61,"tag":141,"props":1988,"children":1989},{},[1990,2001,2003,2009,2011,2017],{"type":61,"tag":83,"props":1991,"children":1992},{},[1993,1999],{"type":61,"tag":98,"props":1994,"children":1996},{"className":1995},[],[1997],{"type":67,"value":1998},"load.iterations > 1",{"type":67,"value":2000}," repeats the entire grid",{"type":67,"value":2002},". Each repetition writes to its own ",{"type":61,"tag":98,"props":2004,"children":2006},{"className":2005},[],[2007],{"type":67,"value":2008},"iter_\u003Ci>\u002F",{"type":67,"value":2010},". Aggregate CSV row count = ",{"type":61,"tag":98,"props":2012,"children":2014},{"className":2013},[],[2015],{"type":67,"value":2016},"n_points × iterations",{"type":67,"value":246},{"type":61,"tag":70,"props":2019,"children":2021},{"id":2020},"source-of-truth",[2022],{"type":67,"value":2023},"Source of truth",{"type":61,"tag":1265,"props":2025,"children":2026},{},[2027,2043],{"type":61,"tag":1269,"props":2028,"children":2029},{},[2030],{"type":61,"tag":1273,"props":2031,"children":2032},{},[2033,2038],{"type":61,"tag":1277,"props":2034,"children":2035},{},[2036],{"type":67,"value":2037},"Piece",{"type":61,"tag":1277,"props":2039,"children":2040},{},[2041],{"type":67,"value":2042},"Location",{"type":61,"tag":1293,"props":2044,"children":2045},{},[2046,2076,2106,2149,2169,2188,2209,2229,2249,2269,2289],{"type":61,"tag":1273,"props":2047,"children":2048},{},[2049,2054],{"type":61,"tag":1300,"props":2050,"children":2051},{},[2052],{"type":67,"value":2053},"Driver",{"type":61,"tag":1300,"props":2055,"children":2056},{},[2057,2067,2068,2074],{"type":61,"tag":367,"props":2058,"children":2060},{"href":2059},"..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Fcli.py",[2061],{"type":61,"tag":98,"props":2062,"children":2064},{"className":2063},[],[2065],{"type":67,"value":2066},"scripts\u002Frag-perf\u002Frag_perf\u002Fcli.py",{"type":67,"value":1164},{"type":61,"tag":98,"props":2069,"children":2071},{"className":2070},[],[2072],{"type":67,"value":2073},"main",{"type":67,"value":2075}," is the single Click command)",{"type":61,"tag":1273,"props":2077,"children":2078},{},[2079,2084],{"type":61,"tag":1300,"props":2080,"children":2081},{},[2082],{"type":67,"value":2083},"Schema",{"type":61,"tag":1300,"props":2085,"children":2086},{},[2087,2097,2098,2104],{"type":61,"tag":367,"props":2088,"children":2090},{"href":2089},"..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Fconfig.py",[2091],{"type":61,"tag":98,"props":2092,"children":2094},{"className":2093},[],[2095],{"type":67,"value":2096},"scripts\u002Frag-perf\u002Frag_perf\u002Fconfig.py",{"type":67,"value":1164},{"type":61,"tag":98,"props":2099,"children":2101},{"className":2100},[],[2102],{"type":67,"value":2103},"RunConfig",{"type":67,"value":2105}," and sub-models)",{"type":61,"tag":1273,"props":2107,"children":2108},{},[2109,2114],{"type":61,"tag":1300,"props":2110,"children":2111},{},[2112],{"type":67,"value":2113},"Orchestrator",{"type":61,"tag":1300,"props":2115,"children":2116},{},[2117,2126,2127,2133,2134,2140,2141,2147],{"type":61,"tag":367,"props":2118,"children":2120},{"href":2119},"..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Frunner.py",[2121],{"type":61,"tag":98,"props":2122,"children":2124},{"className":2123},[],[2125],{"type":67,"value":1240},{"type":67,"value":1164},{"type":61,"tag":98,"props":2128,"children":2130},{"className":2129},[],[2131],{"type":67,"value":2132},"BenchmarkRunner.run",{"type":67,"value":422},{"type":61,"tag":98,"props":2135,"children":2137},{"className":2136},[],[2138],{"type":67,"value":2139},"RagProfiler",{"type":67,"value":422},{"type":61,"tag":98,"props":2142,"children":2144},{"className":2143},[],[2145],{"type":67,"value":2146},"AiperfRunner",{"type":67,"value":2148},")",{"type":61,"tag":1273,"props":2150,"children":2151},{},[2152,2157],{"type":61,"tag":1300,"props":2153,"children":2154},{},[2155],{"type":67,"value":2156},"aiperf plugin",{"type":61,"tag":1300,"props":2158,"children":2159},{},[2160],{"type":61,"tag":367,"props":2161,"children":2163},{"href":2162},"..\u002F..\u002Fscripts\u002Frag-perf\u002Frag_perf\u002Fplugin\u002Fnvidia_rag.py",[2164],{"type":61,"tag":98,"props":2165,"children":2167},{"className":2166},[],[2168],{"type":67,"value":1892},{"type":61,"tag":1273,"props":2170,"children":2171},{},[2172,2177],{"type":61,"tag":1300,"props":2173,"children":2174},{},[2175],{"type":67,"value":2176},"User-facing doc",{"type":61,"tag":1300,"props":2178,"children":2179},{},[2180],{"type":61,"tag":367,"props":2181,"children":2182},{"href":729},[2183],{"type":61,"tag":98,"props":2184,"children":2186},{"className":2185},[],[2187],{"type":67,"value":736},{"type":61,"tag":1273,"props":2189,"children":2190},{},[2191,2196],{"type":61,"tag":1300,"props":2192,"children":2193},{},[2194],{"type":67,"value":2195},"Presets",{"type":61,"tag":1300,"props":2197,"children":2198},{},[2199],{"type":61,"tag":367,"props":2200,"children":2202},{"href":2201},"..\u002F..\u002Fscripts\u002Frag-perf\u002Fconfigs\u002F",[2203],{"type":61,"tag":98,"props":2204,"children":2206},{"className":2205},[],[2207],{"type":67,"value":2208},"scripts\u002Frag-perf\u002Fconfigs\u002F{quick_profile,single_run,sweep}.yaml",{"type":61,"tag":1273,"props":2210,"children":2211},{},[2212,2217],{"type":61,"tag":1300,"props":2213,"children":2214},{},[2215],{"type":67,"value":2216},"Sample queries",{"type":61,"tag":1300,"props":2218,"children":2219},{},[2220],{"type":61,"tag":367,"props":2221,"children":2223},{"href":2222},"..\u002F..\u002Fscripts\u002Frag-perf\u002Fexamples\u002Fqueries.jsonl",[2224],{"type":61,"tag":98,"props":2225,"children":2227},{"className":2226},[],[2228],{"type":67,"value":1720},{"type":61,"tag":1273,"props":2230,"children":2231},{},[2232,2237],{"type":61,"tag":1300,"props":2233,"children":2234},{},[2235],{"type":67,"value":2236},"Synthetic prompts",{"type":61,"tag":1300,"props":2238,"children":2239},{},[2240],{"type":61,"tag":367,"props":2241,"children":2243},{"href":2242},"..\u002F..\u002Fscripts\u002Frag-perf\u002Fprompts\u002Fdefault_prompts.yaml",[2244],{"type":61,"tag":98,"props":2245,"children":2247},{"className":2246},[],[2248],{"type":67,"value":1727},{"type":61,"tag":1273,"props":2250,"children":2251},{},[2252,2257],{"type":61,"tag":1300,"props":2253,"children":2254},{},[2255],{"type":67,"value":2256},"Config schema details",{"type":61,"tag":1300,"props":2258,"children":2259},{},[2260],{"type":61,"tag":367,"props":2261,"children":2263},{"href":2262},"references\u002Fconfig-schema.md",[2264],{"type":61,"tag":98,"props":2265,"children":2267},{"className":2266},[],[2268],{"type":67,"value":2262},{"type":61,"tag":1273,"props":2270,"children":2271},{},[2272,2277],{"type":61,"tag":1300,"props":2273,"children":2274},{},[2275],{"type":67,"value":2276},"Synthetic-query generation",{"type":61,"tag":1300,"props":2278,"children":2279},{},[2280],{"type":61,"tag":367,"props":2281,"children":2283},{"href":2282},"references\u002Fsynthetic-generation.md",[2284],{"type":61,"tag":98,"props":2285,"children":2287},{"className":2286},[],[2288],{"type":67,"value":2282},{"type":61,"tag":1273,"props":2290,"children":2291},{},[2292,2297],{"type":61,"tag":1300,"props":2293,"children":2294},{},[2295],{"type":67,"value":2296},"Output layout & metric semantics",{"type":61,"tag":1300,"props":2298,"children":2299},{},[2300],{"type":61,"tag":367,"props":2301,"children":2302},{"href":616},[2303],{"type":61,"tag":98,"props":2304,"children":2306},{"className":2305},[],[2307],{"type":67,"value":616},{"type":61,"tag":70,"props":2309,"children":2311},{"id":2310},"agent-playbook",[2312],{"type":67,"value":2313},"Agent playbook",{"type":61,"tag":353,"props":2315,"children":2316},{},[2317,2333,2358,2375,2385,2420,2447,2483],{"type":61,"tag":141,"props":2318,"children":2319},{},[2320,2325,2326,2331],{"type":61,"tag":83,"props":2321,"children":2322},{},[2323],{"type":67,"value":2324},"Sync deps:",{"type":67,"value":455},{"type":61,"tag":98,"props":2327,"children":2329},{"className":2328},[],[2330],{"type":67,"value":244},{"type":67,"value":2332}," (one-time per checkout).",{"type":61,"tag":141,"props":2334,"children":2335},{},[2336,2341,2343,2349,2351,2356],{"type":61,"tag":83,"props":2337,"children":2338},{},[2339],{"type":67,"value":2340},"Pick & customise a preset:",{"type":67,"value":2342}," copy ",{"type":61,"tag":98,"props":2344,"children":2346},{"className":2345},[],[2347],{"type":67,"value":2348},"scripts\u002Frag-perf\u002Fconfigs\u002F\u003Cpreset>.yaml",{"type":67,"value":2350}," if you want a variant; always set ",{"type":61,"tag":98,"props":2352,"children":2354},{"className":2353},[],[2355],{"type":67,"value":1493},{"type":67,"value":2357}," to a real collection.",{"type":61,"tag":141,"props":2359,"children":2360},{},[2361,2366,2367,2373],{"type":61,"tag":83,"props":2362,"children":2363},{},[2364],{"type":67,"value":2365},"Run:",{"type":67,"value":455},{"type":61,"tag":98,"props":2368,"children":2370},{"className":2369},[],[2371],{"type":67,"value":2372},"uv run --project scripts\u002Frag-perf rag-perf -c \u003Cconfig>",{"type":67,"value":2374}," from repo root.",{"type":61,"tag":141,"props":2376,"children":2377},{},[2378,2383],{"type":61,"tag":83,"props":2379,"children":2380},{},[2381],{"type":67,"value":2382},"Read the per-point + aggregate tables on stdout.",{"type":67,"value":2384}," Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table.",{"type":61,"tag":141,"props":2386,"children":2387},{},[2388,2393,2395,2401,2403,2411,2413,2418],{"type":61,"tag":83,"props":2389,"children":2390},{},[2391],{"type":67,"value":2392},"Parse artifacts",{"type":67,"value":2394}," under ",{"type":61,"tag":98,"props":2396,"children":2398},{"className":2397},[],[2399],{"type":67,"value":2400},"output.dir\u002Frun_\u003Cts>\u002F",{"type":67,"value":2402}," — see ",{"type":61,"tag":367,"props":2404,"children":2405},{"href":616},[2406],{"type":61,"tag":98,"props":2407,"children":2409},{"className":2408},[],[2410],{"type":67,"value":616},{"type":67,"value":2412},". For multi-point runs, ",{"type":61,"tag":98,"props":2414,"children":2416},{"className":2415},[],[2417],{"type":67,"value":672},{"type":67,"value":2419}," has one row per (point × iteration).",{"type":61,"tag":141,"props":2421,"children":2422},{},[2423,2428,2430,2438,2440,2445],{"type":61,"tag":83,"props":2424,"children":2425},{},[2426],{"type":67,"value":2427},"Summarise for the user",{"type":67,"value":2429}," using the playbook in ",{"type":61,"tag":367,"props":2431,"children":2432},{"href":693},[2433],{"type":61,"tag":98,"props":2434,"children":2436},{"className":2435},[],[2437],{"type":67,"value":693},{"type":67,"value":2439}," — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations \u002F non-zero errors \u002F suspect ",{"type":61,"tag":98,"props":2441,"children":2443},{"className":2442},[],[2444],{"type":67,"value":714},{"type":67,"value":2446}," \u002F low sample size, and a concrete next-experiment YAML.",{"type":61,"tag":141,"props":2448,"children":2449},{},[2450,2455,2457,2462,2463,2468,2470,2475,2476,2481],{"type":61,"tag":83,"props":2451,"children":2452},{},[2453],{"type":67,"value":2454},"Tune retrieval \u002F reranker:",{"type":67,"value":2456}," flip to ",{"type":61,"tag":98,"props":2458,"children":2460},{"className":2459},[],[2461],{"type":67,"value":390},{"type":67,"value":1172},{"type":61,"tag":98,"props":2464,"children":2466},{"className":2465},[],[2467],{"type":67,"value":744},{"type":67,"value":2469}," for fast iteration, then return to ",{"type":61,"tag":98,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":67,"value":401},{"type":67,"value":113},{"type":61,"tag":98,"props":2477,"children":2479},{"className":2478},[],[2480],{"type":67,"value":412},{"type":67,"value":2482}," when characterising under load.",{"type":61,"tag":141,"props":2484,"children":2485},{},[2486,2491,2493,2501],{"type":61,"tag":83,"props":2487,"children":2488},{},[2489],{"type":67,"value":2490},"Triage failures:",{"type":67,"value":2492}," see Troubleshooting above and ",{"type":61,"tag":367,"props":2494,"children":2495},{"href":616},[2496],{"type":61,"tag":98,"props":2497,"children":2499},{"className":2498},[],[2500],{"type":67,"value":616},{"type":67,"value":2502}," for empty-citation \u002F bottleneck=N\u002FA patterns.",{"type":61,"tag":2504,"props":2505,"children":2506},"style",{},[2507],{"type":67,"value":2508},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2510,"total":2612},[2511,2526,2540,2554,2566,2583,2598],{"slug":2512,"name":2512,"fn":2513,"description":2514,"org":2515,"tags":2516,"stars":23,"repoUrl":24,"updatedAt":2525},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2517,2520,2523,2524],{"name":2518,"slug":2519,"type":15},"Data Analysis","data-analysis",{"name":2521,"slug":2522,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},"2026-07-14T05:28:43.176466",{"slug":2527,"name":2527,"fn":2528,"description":2529,"org":2530,"tags":2531,"stars":23,"repoUrl":24,"updatedAt":2539},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2532,2535,2538],{"name":2533,"slug":2534,"type":15},"Deployment","deployment",{"name":2536,"slug":2537,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2541,"name":2541,"fn":2542,"description":2543,"org":2544,"tags":2545,"stars":23,"repoUrl":24,"updatedAt":2553},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2546,2549,2550],{"name":2547,"slug":2548,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2551,"slug":2552,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2555,"name":2555,"fn":2556,"description":2557,"org":2558,"tags":2559,"stars":23,"repoUrl":24,"updatedAt":2565},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2560,2561,2562],{"name":2518,"slug":2519,"type":15},{"name":9,"slug":8,"type":15},{"name":2563,"slug":2564,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2567,"name":2567,"fn":2568,"description":2569,"org":2570,"tags":2571,"stars":23,"repoUrl":24,"updatedAt":2582},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2572,2575,2578,2579],{"name":2573,"slug":2574,"type":15},"Automation","automation",{"name":2576,"slug":2577,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2580,"slug":2581,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2584,"name":2584,"fn":2585,"description":2586,"org":2587,"tags":2588,"stars":23,"repoUrl":24,"updatedAt":2597},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2589,2590,2593,2594],{"name":2533,"slug":2534,"type":15},{"name":2591,"slug":2592,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2595,"slug":2596,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2599,"name":2599,"fn":2600,"description":2601,"org":2602,"tags":2603,"stars":23,"repoUrl":24,"updatedAt":2611},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2604,2605,2608],{"name":9,"slug":8,"type":15},{"name":2606,"slug":2607,"type":15},"Quantum Computing","quantum-computing",{"name":2609,"slug":2610,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2614,"total":2764},[2615,2633,2648,2659,2671,2685,2698,2712,2723,2732,2746,2755],{"slug":2616,"name":2616,"fn":2617,"description":2618,"org":2619,"tags":2620,"stars":2630,"repoUrl":2631,"updatedAt":2632},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2621,2624,2627],{"name":2622,"slug":2623,"type":15},"Documentation","documentation",{"name":2625,"slug":2626,"type":15},"MCP","mcp",{"name":2628,"slug":2629,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2634,"name":2634,"fn":2635,"description":2636,"org":2637,"tags":2638,"stars":2645,"repoUrl":2646,"updatedAt":2647},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2639,2642,2643],{"name":2640,"slug":2641,"type":15},"Containers","containers",{"name":2533,"slug":2534,"type":15},{"name":2644,"slug":51,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2649,"name":2649,"fn":2650,"description":2651,"org":2652,"tags":2653,"stars":2645,"repoUrl":2646,"updatedAt":2658},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2654,2657],{"name":2655,"slug":2656,"type":15},"CI\u002FCD","ci-cd",{"name":2533,"slug":2534,"type":15},"2026-07-14T05:25:59.97109",{"slug":2660,"name":2660,"fn":2661,"description":2662,"org":2663,"tags":2664,"stars":2645,"repoUrl":2646,"updatedAt":2670},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2665,2666,2667],{"name":2655,"slug":2656,"type":15},{"name":2533,"slug":2534,"type":15},{"name":2668,"slug":2669,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2672,"name":2672,"fn":2673,"description":2674,"org":2675,"tags":2676,"stars":2645,"repoUrl":2646,"updatedAt":2684},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2677,2680,2681],{"name":2678,"slug":2679,"type":15},"Debugging","debugging",{"name":2668,"slug":2669,"type":15},{"name":2682,"slug":2683,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2686,"name":2686,"fn":2687,"description":2688,"org":2689,"tags":2690,"stars":2645,"repoUrl":2646,"updatedAt":2697},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2691,2694],{"name":2692,"slug":2693,"type":15},"Best Practices","best-practices",{"name":2695,"slug":2696,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2699,"name":2699,"fn":2700,"description":2701,"org":2702,"tags":2703,"stars":2645,"repoUrl":2646,"updatedAt":2711},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2704,2707,2710],{"name":2705,"slug":2706,"type":15},"Machine Learning","machine-learning",{"name":2708,"slug":2709,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2713,"name":2713,"fn":2714,"description":2715,"org":2716,"tags":2717,"stars":2645,"repoUrl":2646,"updatedAt":2722},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2718,2721],{"name":2719,"slug":2720,"type":15},"QA","qa",{"name":2563,"slug":2564,"type":15},"2026-07-14T05:25:53.673039",{"slug":2724,"name":2724,"fn":2725,"description":2726,"org":2727,"tags":2728,"stars":2645,"repoUrl":2646,"updatedAt":2731},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2729,2730],{"name":2533,"slug":2534,"type":15},{"name":2536,"slug":2537,"type":15},"2026-07-14T05:25:49.362534",{"slug":2733,"name":2733,"fn":2734,"description":2735,"org":2736,"tags":2737,"stars":2645,"repoUrl":2646,"updatedAt":2745},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2738,2741,2742],{"name":2739,"slug":2740,"type":15},"Code Review","code-review",{"name":2668,"slug":2669,"type":15},{"name":2743,"slug":2744,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2747,"name":2747,"fn":2748,"description":2749,"org":2750,"tags":2751,"stars":2645,"repoUrl":2646,"updatedAt":2754},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2752,2753],{"name":2719,"slug":2720,"type":15},{"name":2563,"slug":2564,"type":15},"2026-07-14T05:25:54.928983",{"slug":2756,"name":2756,"fn":2757,"description":2758,"org":2759,"tags":2760,"stars":2645,"repoUrl":2646,"updatedAt":2763},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2761,2762],{"name":2573,"slug":2574,"type":15},{"name":2655,"slug":2656,"type":15},"2026-07-30T05:29:03.275638",496]